Title: Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Foundation Model; VLM; Image and Video Caption;
Abstract: We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2$-$2.4$\times$ faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Perceive Anything Model (PAM), a vision-language model designed for comprehensive region-level understanding in images and videos. PAM builds on the Segment Anything Model 2 (SAM 2) and integrates Large Language Models (LLMs) to achieve simultaneous object segmentation and diverse semantic output generation.  Compared to previous methods, PAM improvements in inference efficiency and GPU memory usage.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The integration of SAM 2 with LLMs via the Semantic Perceiver is a successful attempt to combining segmentation with rich semantic understanding, enabling multi-dimensional outputs like region-level captions and streaming video captioning. The performance and ablations show the power of the proposed method.\n- The video data refinement pipeline, storyboard-driven caption expansion, addresses a critical need and efficient way for fine-grained semantic annotations.\n\nWeakness:\n\nSee questions.\n\nQUESTIONS:\n- Data Processing using External Models\n\nThe authors mention using advanced VLMs such as qwen2.5-vl-72B, gpt4o, and deepseek v3 for data labeling and processing. The performance gains of PAM may not be attributed to its architectural design, but could be significantly determined by the power of these external models and the extensive labeled data utilized. Thus, the true effectiveness of PAM's own structure remains a question. \n\n- Comparison with Existing VLMs\n\nThe authors seem to avoid direct comparison of PAM with those existing VLMs such as open-sourced Qwen2.5-VL series. Although the author mentioned the differences between region-level VLMs and image-level VLMs, I think that these VLMs can be adapted to perform region-level tasks with the help of system prompts and few-shot examples. A fair comparison with these VLMs may be important. Since if existing VLMs are already capable of handling these tasks, the necessity of distilling a new region-level VLM appears to be questionable. \n\n- Explanation of Efficiency Gains\n\nThe authors highlight PAM's superior inference efficiency and reduced GPU memory usage for image and video processing. However, they do not provide a sufficiently detailed explanation of the sources of these efficiency improvements. It is unclear whether the faster speed is primarily due to the smaller model size (e.g., 3B vs. 7B), different model architecture, or other factors such as more efficient processing pipelines. A clear analysis may be better.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the vision language model area and introduces an end-to-end vision-language model, PAM, to bridge the gap between object segmentation and deep semantic understanding in both images and videos. This work also introduces a large-scale, high-quality dataset comprising 1.8M image and 0.6M video samples, which features multi-level semantic granularity. The authors conducted comprehensive experiments and validate the effectiveness of the proposed architecture and dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors successfully integrate segmentation with deep and multi-level semantic understanding, which is a significant step towards more holistic machine perception.\n2. The authors have made a substantial contribution by curating a large and high-quality dataset. The dataset's multi-dimensional annotations (category, definition, function, caption) and its bilingual support are valuable.\n3. The model's capabilities are thoroughly validated across a wide array of demanding benchmarks for both images (LVIS, PACO, RefCOCOg, Ferret-Bench) and videos (Elysium, BensMOT, HC-STVG). The authors also propose a new, relevant metric, G-STDC, for evaluating the continuity of streaming captions, demonstrating a thoughtful approach to assessment.\n\nWeaknesses\n1. The authors transparently acknowledge that PAM is specialized for a specific set of region-level understanding tasks. It currently lacks support for more general vision-language tasks like open-ended Visual Question Answering (VQA) or region-level dialogue, which limits its scope compared to some other large multimodal models.\n2. The experiments and ablation studies in this work were primarily conducted on Qwen2.5-1.5B/3B. This relatively limited selection raises concerns about the generalizability and scalability of the method.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Percieve Anything Model (PAM), which integreates the visual understanding ability of SAM2 into LLMs. Specifically, a frozen SAM2 is connected with LLM with trainable semantic perciever. While the segmentation ability of PAM is fully inherented from SAM2, such integretion provides detailed visual information for LLMs to perform recognition, explaination and caption. To train such integreted model, the authors builds a billingual visual instruction tunning dataset, which mixes recgnition, explanation and caption on images with multi-scale video caption. After training on such dataset, the PAM shows superior performance on classification, OCR, region caption, and video caption tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of using SAM2 as visual encoder for LLMs to provide better capability on specific understanding tasks is novel.\n2. The paper is well written and easy to follow.\n3. The experiments shows that PAM can outperform existing MLLMs on various tasks.\n\nWeaknesses\n1. Does the model lost its instruction following ability after such training? According to Figure 2 and Section 4, it seems the LLM takes aligned SAM2 outputs and gives the label, discription, and/or caption directly during training. If so, the LLM could lost its instruction following ability and cannot give perception result for text beased prompts (like \"the animal behind the door\"). Also, it will lost the LLM's ability as fundation model, which highly limits the application of this framework compared to methods like DAM [1].\n2. The reason why SAM2 is frozen during the training is not clear. Typically, when applying visual alignment with a vision encoder and LLM, we will (in some stages) finetune the vision encoder for better performance [2,3]. However, in this framework, the SAM2 is completely forzen, and the reason of such conter-intuative design is not well illustrated.\n3. Compared to standard pretrained ViTs, representations from SAM2 naturally contains more information on segmentation, therefore is reasonable to have better performance on specific tasks like recognition, explaination, and caption. However, such structure and representation seems not effective on other understanding tasks (e.g., visual question answering), limiting the generalization of this framework.\n\nOverall, although the framework shows superior performance on multiple perception tasks, such gain seems sacrifice the instruction following and generalization ability of LLM, which significantly limits the application of both framework and the trained model compared to similar prior approaches [1].\n\nQUESTIONS:\n1. Does the model lost its instruction following ability after such training? \n2. Why is SAM2 frozen during training? \n3. Can this framework generalize to other understanding tasks? Does SAM2 representations still outperform other models?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}