Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Authors: Weifeng Lin
Xinyu Wei
Ruichuan An
Tianhe Ren
Tingwei Chen
Renrui Zhang
Ziyu Guo
Wentao Zhang
Lei Zhang
Hongsheng Li
Venue: NeurIPS 2025 poster ()
Keywords: Vision Foundation Model; VLM; Image and Video Caption;
Primary area: deep_learning

Abstract: We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2$-$2.4$\times$ faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding.

Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Weifeng Lin'*
Xinyu Wei
Ruichuan An
Tianhe Ren
Tingwei Chen' Hongsheng Li
Renrui Zhang!
Ziyu Guo!
Wentao Zhang
Lei Zhang'
CUHK
HKU
PolyU
*Peking University
Labet Akmruum alsy lanker
Teei
Label: Hasa
Befretion Fenetianalyy
Defiustion Fanctionalty:
Arrat as yelrue
lgteoeggt, durabl lnaiee
calk desigeed
plastic ottached abject
IMAGE
rade bin ziminer ally slto varbuss
ta castes, lae captaral
opticall bearine information
about soject owace
requre renching heights this
in motion on oil sood, leeh
they
aloy werter latter heing rod pue accos the
providig prinaty purpone
poatting assertl iniomattion porents
amaato the cat's mane
adiiddeale need
riciti front wiedellete The
yellow,
cytiedrical rungs and panilel side ralls festurng slght
econdsse
with cligity coreed hood
toxi har dast, maidem dodge
sppeer
tep The ug black tod that reals "Thrnyy" and "Bay" ideatifer
Promptable Video Caption: The boy wearing blue hooded top, seating at wuoden table. In the viden, he is hulding dark-colored plaything which he appears protending to shoot something focused object that direction. He is looking romain fairly head body tarpot adjusting his aims then hid hehind chair.
VIDEO
Promptable Video Caption: The large house in the cester of the background cartoon-style building The main structure and roof appeor be predominantly blue It set colorful environment with what looks like grassy or sandy fureground, surrounded by styliadd trees tall plants and other smaller ofjects
Promptable actively presenting, changes hand gestures and facial expressions. She has dark baic.
She is not visually present within these scenes the game the video's end, she reappears in the studio, apparently to discuss the
Streaming
Perceive Anything Model (PAM): PAM accepts various visual prompts (such clicks, boxes, and masks) to produce region-specific information for images and videos, including masks, category, label definition, contextual function, and detailed captions. The model also handles demanding region-level streaming video captioning.
Figure 1:
Abstract
We present Perceive Anything Model (PAM), conceptually straightforward and efficient framework for comprehensive region-level visual understanding im- ages and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LL.Ms), enabling simultaneous object
"Core Contributor
'Correspondigg Authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

segmentation with the generation of diverse, region-specines semantic outputs, in- cluding categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support ro- bust multi- golliiiganllarity understanding, we also develop dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region under- standing tasks. It runs x faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that effective approach will serve strong baseline for future search in region-level visual understanding. Code, model and data are available at: https: /tetpp:/Peeeeivveeeee Anything worggiiiii git
Introduction
The vision community has rapidly witnessed advances in vision foundation models, such as SAM [34] and SAM [52], which have dramatically improved interactive object segmentation performance in images and videos. These models offer remarkable precision in localizing arbitrary objects based on various visual prompts. However, they typically lack deep semantic understanding of the segmented regions, elucidating what these regions mean or how they function in context remains a challenging problem.
Recent studies seek to endow Vision Aasion-Language Models (VLMs) with region-level understanding capability through visual prompts. As illustrated in Fig. 2, current methods can be grouped into three paradigms: (1) textual encoding [63, 78, 86, 44], which encode 2-D bounding-box coordinates as natural-language strings inside the prompt, thereby supplying no explicit region prior; (2) visual- prompt encoding (VPE) [41, 5 which introduce extra module to embed regional image features and positional features; (3) Rol/segmentation based encoding [38, 77, 83, 80, 29]. which utilize an external mask generator to concatenate image embedding and mask embedding. While these methods show promise, they often present several limitations: (i) they usually generate only limited semantic outputs ooteutt-ftten just category labels or short captions [26, 88, 69, 67]; (ii) their designs are modality-specified focusing on one single visual modality (image or video), offering limited generality [63, 78, 77, 80, 81]. (iii) they rely on external segmentation models to supply masks, serial design that adds computational overhead and makes overall performance sensitive to mask quality [80, 81, 38).
To address these challenges, we introduce the Perceive Anything Model (PAM), an end-to-end region-level lision-language model designed for fast and comprehensive fine-grained visual under- standing across both images and vidcos, encompassing capabilities such as predicting categories, explaining the definition and contextual function of identified regional elements, and generating detailed descriptions of specific regions. Rather than redesigning model architecture from scratch, our approach efficiently extends the SAM 2 framework with Large Language Models (LLMs) to support semantic understanding. Specifically, we introduce Semantic Perceiver that acts as essential bridge, effectively leveraging rich intermediate visual features from the SAM 2 backbone to integrate general vision, localization, and semantic priors into visual tokens. These tokens are subsequently processed by the LLM to generate a diverse semantic outputs. Furthermore, PAM features parallel design for its mask and semantic decoders, enabling simultancous generation of region masks and semantic content, thereby improving computational efficiency.
To ensure PAM's robustness in understanding region-level multi-dimensional semantic granularity, high-quality training data is an essential component. While multiple existing datasets [6, 32, 36, 43, 29, 68] provide aegion-semantic annotations, we noticed that they are often overly coarse, limiting their utility for fine-grained understanding tasks. Therefore, to construct high- training data, we develop an advanced data refinement and augmentation pipeline that leverages leading VLMS (e.g., GPT-4o [27]) and human expert validation to refine and augment existing region-level annotated datasets. For images, we generate annotations at multiple distinct semantic granularities for each specific region: fine-grained category label, a context-aware definition that clarifies the region's 's role

---

Semonnices
Semantics
obg
L.M
LLM
LLM
Sog. Model (Vision Encodar)
LLM
f
Vision Fnender
Toleu bexr
Secoder
Viston
Toleentzer
Vision Brrader
VIEE
Tukenioor
1-N mace
f Task
f
Sog.
Visual promplx
Taloeizer
promges
Visual
Task
prompts
Visual
Task
f Trak
(a) Testual
(b) Previous VPE Methods
Rol-based
(d) PAM (ours)
Figure 2: Previous Paradigms vs. Our Paradigm (PAM). (a b) Textual/VPE methods provide region understanding using positional embeddings but typically lack simultaneous object masks. methods use external segmenter for object masks, subsequently fusing
(c) Rol//egmentation-bannn image and mask embeddings. (d) In contrast to previous paradigms, our method directly treats the Seg. model as vision encoder. It effectively leverages the rich visual embeddings from the robust segmentation model and features parallel design for its mask and semantic decoders.
or function within the scene, and detailed descriptions. For videos, we refined original coarse-level annotations from referring video detection and segmentation dataset [64, 58, 18, 71, 17] into detailed, temporally- region-level captions. Furthermore, we pioneered the development of weent-based. region-level streaming video caption data. To the best of our knowledge, this is the first work to construct such dataset, enabling the model to support streaming video region captioning. Notably, we also generate bilingual (English and Chinese) versions of each data annotation to equip the model with multilingual response capabilities. This process yields a final high-quality dataset comprising 1.5M maag-rregion-semamee semantics triples and 0.6M video-region-sente siteorregoon-semanticss
semantics triples.
Our experimental results demonstrate that PAM delivers robust performance across diverse range of regional understanding tasks for both images and videos, while operating 1.2-2.4X faster and consuming less GPU memory compared to prior models. We believe our model, dataset, and insights will significantly advance research in this domain and broadly benefit the rision-language community.
2 Related Work
Interactive Image and Video Object Segmentation. Interactive object segmentation has pro- gressed rapidly in recent years. Early methods-such as Graph Cut [5] and Active Contours [9]-relied on manual annotations (c.g., foreground/bucckrround clicks). Inspired by the paradigm of pre-training autoregressive Transformer architectures large-scale data in language modeling, the Segment Anything Model (SAM) [34] revolutionized user-mode interaction by ingesting multiple visual prompts and segmenting arbitrary objects in class-agnoste manner. SAM 2 [52] extended this capability to video, enabling real-time processing of arbitrarily long sequences with strong generalization. Subsequent research [33, 82, 87, 54, 13, 74]. while retaining the core architecture, has further improved the accuracy and efficiency of this model family.
Region-level Vision- Vision-Language Models (VLMs). Region-level understanding tasks, such as region classification and captioning, are fundamental in computer vision. Regarding image-based VLMs, recent researches [46, 12, 70, 84, 77, 80, 8, 41, 1] have demonstrated notable trend towards enabling region-level understanding capabilities through spatial visual prompts. Furthermore, research has extended regional understanding to the video domain [78, 63, 81, 48], focusing on identifying and interpreting speciieee regions across temporal intervals. However, these approaches typically operate within a single modality (cither image or video), and more complex tasks, such as region-leve streaming video captioning which requires continuously generating textual descriptions for specific regions as a video premae
remain largely unaddressed.
Streaming Video Captioning. Streaming video captioning demands per-frame processing and rapid response times. Recent online video understanding models [16, 20] aim to identify the current action at each timestamp. Streaming video caption [89] propose to incorporate memory modules and develop specialized streaming decoding algorithms support streaming captioning. VideoLLM online [11]
3

---

Mask Decoder
Image Encoder
Memory Enmnde Banle
Prompe Encoder
Semantic Perceiver
tine
DUSAM
Dnazen
Rich Visual Semantic Content
Catageey
Triinall
52
Semantic Perceiver
large Language
2. Detinition Functionality
Brrid Caption
Concut
Descriplian
Streaming Video Caption (Milingaal Eingish 4:33
Structural
oupy
Nak dend
Decoder
Semantic
Figure 3: Overall Architecture of PAM.
further pioneered the use of LLMs to achieve free-form dialogue synchronized with the online video stream. However, these approaches predominantly focus general event comprehension, leaving the continuous tracking and description of specific regions within a video stream as a significant unresolved challenge.
3 Perceive Anything Model (PAM)
Given visual prompts such as points, boxes, or masks to specify a region interest, Perceive Anything Model (PAM) can simultaneously: (I) Segment: Generate precise segmentation masks for the indicated region within an image or throughout a video. (2) Recognize: Identify the category of the designated region object. (3) Explain: Provide clear explanations of the region's 's or object's definition, attributes, and functionality within its given context. (4) Caption: Generate concise or detailed captions for the region within images, videos, and video streams.
3.1 Model Architecture
As illustrated in Fig. 3, our PAM can be divided into two parts. The first part is the SAM 2 framework, which comprises image encoder, a prompt encoder, memory modules, and mask decoder. This framework provides robust spatio-temport visual feature extraction and segmentation capabilities. The second part is semantic decoder, which is based large language model (LLM). Crucially, our proposed Semantic Perceiver acts as a bridge, effectively leverages intermediate visual features from the SAM 2 backhone and results in visual tokens. These tokens subsequently processed by the LLM to generate diverse semantic outputs. For decoding, PAM features parallel design for its mask and semantic decoders, enabling the simultaneous segmentation of objects while generating diverse semantic outputs of them. The design of components and training process are detailed below.
Semantic Perceiver. As shown in Fig. 3(b) and Fig. 4, the architecture of Semantic Perceiver mirrors the SAM Feature Fusing module (S2-FFM), employing a lightweight two-layer transformer with self-attention, cross-attention, and point-wise MLP. Specifically, receives primary inputs: enhanced mask tokens from S2-FFM, which incorporate IoU and prompt tokens information and serve as unique identifiers for precise mask generation; and updated image embeddings after S2-FFM, capturing general visual context and implicit features enriched through interaction with mask tokens. Next, following [26, 28), we concatenate N, learnable semantic tokens with the enhanced mask tokens. Finally, through further attention mechanisms within the Semantic Perceiver, we can fetch visual tokens rich in both general visual and object-level localization information. Given an input of N frames (where N for single image), Semantic Perceiver outputs two sets of 256-dimensional vectors: 24
N visual tokens and N₂ x N semantic tokens (N, 16 by default).
Projector. The projector preceding the LLM comprises two layers: a pixel shuffle operation and an MLP projector. For image inputs, we apply the pixel shuffle operation over adjacent 2x2 feature patches to downsample the number of visual tokens. For video inputs, the prompted frame is processed similarly with single image, while the remaining frames in the video clip undergo a more aggressive 4x4 pixel shuffle operation to significantly reduce visual tokens and further improve processing efficiency for semantic decoder. Subsequently, we use two distinct ML.Ps [45] to project visual and semantic tokens separately.

---

12.FFM
for Semoorile Peredver
22
Large Language Model (Qwen 2.5)
caboking
Image
Insge lokes
image
Token to image sits.
Suag Neloes
Wikeo Talar (ols
Video Tolker (dg sjolt)))
Tokera
Mall Tokes atm
-Mai Tokuu
Iange Phg.
Scx. Prdi
Tokenierr
Figure 4: Detailed illustration of our PAM workflow. Semantic Perceiver first receives enhanced image embeddings and mask tokens from the S2-FFM and outputs enriched visual tokens and semantic tokens. These are subsequently fed into the semantic decoder for decoding.
Semantic Decoder. We adopt the pre-trained Qwen2.5 LLM [72] as semantic decoder, lever- aging its strong language processing capabilities. This decoder responsible for interpreting the processed visual tokens and semantic tokens alongside task instructions to generate the desired semantic outputs.
Streaming Video Encode and Decode. Building upon the progressive introduction of historical information per frame via memory modules SAM 2, we propose a straightforward strategy for region-level streaming video captioning without adding complex components. Specifically, an additional 2x2 pixel shuffle operation is applied to the last frame each video clip. This leads to greater density of visual tokens, improving the preservation of historical visual information. These tokens subsequently act as the initial frame for the next video clip and are processed by the LLM together with the remaining frames of that clip. This approach ensures that each clip processed consistently and effectively passes crucial historical information from the previous clip into the next video clip. Additionally, incorporate the previous textual description into the prompt further augment contextual history, enhancing the model's comprehension and descriptive accuracy for ongoing events. In practice, framework allows users to flexibly specify decode timestamps. Upon reaching a designated timestamp, the model describes the specified region within the temporal interval between the current timestamp and the previous one.
Training Strategies. We structure our training process using a three-stage curriculum learning approach, progressively enhancing the PAM's region-level visual understanding capabilities from images to video. In all training stage, the parameters of SAM 2 are frozen. The each training stage are summarized in Appendix A.
Stage 1: Image Pretraining and Alignment. The initial training stage focuses on establishing robust alignment among visual tokens, semantic tokens and the language model's embedding space. The primary objective to enable the model to effectively understand region-level image content. To this end, utilize a large dataset of region-level image classification and captioning. During this stage, only the semantic perceiver and the projector are trained.
Stage 1.5: Video-Enhanced Pretraining and Alignment. In this stage, we extend the initial image- based training by incorporating region-level video captions. This inclusion enables the model to comprehend dynamic scenes through the integration of spatio-temporal visual information. The trainable modules are the same as in Stage 1.
Stage 2: Multimodal Fine-Tuming The final stage employs supervised fine-tuning (SFT) to enable
the model to perform diverse tasks and generate desired responses. This stage utilizes high-quality dataset, which has been refined and augmented via our pipeline (Sec. 4). Training in this phase jointly involves the semantic perceiver, the projector, and the semantic decoder.
Data
To enhance PAM's comprehensive visual perception capabilities, we develop robust data refinement and augmentation pipeline to curate high-quality training dataset. This dataset distinguished by three key features: (1) Broad-ranging Semantic Granularities. It provides diverse visual semantic
5

---

Onig. Ame
Orig.
black
green bottle
Nos-stream Acto. Ruline
Trace-Uini
Orig. deg le playing with another dog.
The prkkn noranes adarned
padiermed untilt inaumatts
nighly turning head
zere thk lel.
Labet butix
netirttion ligulte
is centiket syreds
ooo
vol svragg in this conee the with appeoss to
csstan mortthonsh unaty the
aruand
then the dag the uriped cutto masee frscande the plalo with dhosage 1i6
Caprien it traeutoen plrt betts. hac buttl agpnus sightty laudd bad
saitle co Yeur
sprght
Figure 5: Illustrative examples of dataset construction pipeline. The left panel displays image annotations; the right panel details annotations for non-streaming and streaming video.
annotations spanning from coarse-level (categories, definitions, contextual functionalities) to fine- grained (detailed descriptions) (Sec. 4.1). (2) Regional Streaming Caption Annotations. The first dataset to curate annotations specifically for streaming video region captioning (Sec. 4.2). (3) Bilingual Annotations, supporting both English and Chinese (App. B.2). The pipeline is detailed below, and additional information are available in Appendix B.
4.1 Image Dataset
Regional Recognition, Explanation, and Caption. For regional recognition, we utilize multiple instance detection and segmentation datasets [55, 35, 40, 23, 50, 66], along with scene text recognition datasets [56, 31, 30, 19, 24, 14,7 76, 57, 4]. In this context, the bounding box or mask serves as the visual prompt input, and the label treated as the output.
To achieve deep, fine-grained visual understanding beyond simple classification, we propose an enhanced pipeline that generates: clear conceptual explanations, contextual functional roles, and detailed descriptions for each specific region. This multi calti-dimensional information aims to significantly improve user comprehension, particularly for uncommon terms or unfamiliar subjects. To implement this, we utilize the latest VLMs for their extensive world knowledge and powerful visual understanding capabilities to assist refinement. Specifically, we apply the Set of Mask (SoM) method [75] to identify regions of interest, and use original annotations as context to guide models to produce desired responses, which then undergo manual quality assurance. An illustrative example presented in Fig. 5(left). We present more details in Appendix B.
4.2 Video Dataset
Region-level Video Caption. To extend the model's regional captioning capabilities to video, we collected and analyzed several existing video datasets, including referring detection and segmentation datasets [71, 47, 18, 62, 58, 17, 85, 64], as well as the recent Sa2VA [79] annotations for the SA- [53] dataset. These datasets, designed for detecting, segmenting, and captioning specific objects in videos based on textual descriptions, often contain descriptions that are overly coarse, simplistic, inaccurate, or predominantly static, neglecting essential temporal details such as object motion, interactions, and state changes throughout the video.
To address the existing limitations, we propose the storyboard-driven caption expansion method. This process involves several key stages: (1) Keyframe Sampling: Six keyframes are uniformly extracted from each video. (2) Storybourd Synthesis: These extracted keyframes are combined to form high-resolution composite image, presented in storyboard format (as illustrated in Fig. 5). (3) Object-Centte Highlighting: Within this composite image, each individual frame specifically highlights the target object using colored bounding box mask, implemented by

---

Classification
OCR
Model
LVIS
PACO
COCO Text
Total-Text
Semantic Sim.
Semantic IoUl
Semantic Sim.
Semantic IoU
Acc. (cc.(%)
Acc.(%)
Shikra-7B [12]
49.7
19.8
43.6
11.4
GPT-Ro1-7B [84]
51.3
12.0
48.0
12.1
Osprey-7B [80)
63.2
38.2
73.1
52.7
Femret11B [77]
65.0
37.8
VP-L.AVA 8B [41]
86.7
61.5
75.7
50.0
44.8
46.9
VP-SPHINX-I3B
87.1
62.9
76.8
DAM-SB [38]
89.0
77.7
84.2
73.2
PAM-1.5B (Ours)
87.4
76.5
85.1
73.5
39.4
PAM-3B (Ours)
88.6
78.3
87.4
74.9
42.2
52.3
Table Results of region-level image recognition on LVIS, PACO, COCO Text, and Total-Text.
Refeocog
Ref-L4
Ferrt
MDVP Bench
Model
METEOR
CIDEr
METEOR
CIDEr
ROUGE-L
METEOR
CIDEr
Refer. Desc.
Avg.
[51]
104.0
Osprey-7B [80]
16.6
108.3
72.2
44.3
Ferret-7B [77]
VP-LLaVA-88E 8B [41]
22.4
153.6
75.2
70.6
VP-SPHINX-13B [+1]
20.6
162.5
74.3
Omni-RGPT-7B [25]
17.0
139.3
17.0
109.7
RegionGPT-77
17.0
145.6
16.9
109.9
25.3
12.2
42.0
DAM-SB [38]
37.1
19.4
70.0
PAM-115B (Ours)
19.2
132.9
24.7
135.0
29.6
15.9
55.8
75.4
69.4
PAM-3B (Ours)
20.8
142.3
26.9
143.1
31.3
17.2
59.7
78.3
72.2
Table 2: Performance comparison on region-level image captioning across multiple benchmarks.
SoM. (4) LLM-Powered Elaboration: Then, using the original annotations as condition, we prompt GPT-4o to generate descriptions that both refined, detailed and temporally aware. This multi- frame consolidation is critical as it enhances GPT-4o's contextual comprehension, yielding superior descriptions compared individual frame analysis.
Region-level Streaming Video Caption. Beyond describing the entire video, we aim extend the model's s capabilities to a streaming manner. To achieve this, we perform additional augmentation on our refined region-level video caption data. Specifically, we first employ the TRACE-Uni model [22] to segment the input video into multiple distinct events, each demarcated by its temporal boundaries. Subsequently, for each segmented video clip, we apply the same "storyboard-driven method. To generate precise and continuous event descriptions, the GPT-4o input prompt was redesigned to iteratively incorporate the description from the preceding video clip as contextual information for processing the current clip. The entire workflow is illustrated in Fig. 5(right).
processing
Experiments
5.1 Implementation Details
We employ Qwen2.5-1.5BBl [72] as our semantic decoder, and utilize the pre-trained hierarchical SAM 2-Large* the base vision foundation model. By default, we use 16 learnable semantic tokens and uniformly sample 16 frames per video clip. All training is conducted on 8 NVIDIA A100 GPUs with 80GB. For all evaluation experiments, adopt zero-shot test manner without fine-tuning on specific datasets. The best and the second best results are indicated in bold and with underline
5.2 Image Benchmarks
Regional Recognition and Explanation. This task requires the model to identify either the object category or scene text within a specified image region. Recognition performance is assessed on the validation sets of the LVIS (object-level) [23] and PACO (part-level) [50] datasets, alongside the test sets of COCO-Tes -ext [61] and Total-Text [14]. Standard evaluation metrics include Semantic Similarity [80], Semantic Intersection over Union (Sem. IoU) [15], and accuracy.
htng:://l.bbippbiiiiiiie.ccmmmmp anything 2/092824/sam2.1 hiera
large.pt larre_

---

Label
Definition & Functionality
Caption
Pace Mask
protatt the weane's mouth and from surounddngg unironeet
work offere protection. ausriin
the person likely weating a for reduxing the spread of droplets
that the persun can innreese thersshhes in the desiled
bruchwork
3.4.- *+ 1 A ad5- 12 [01 1t. a.m. A.B. F. B...
SAINTT 2.
F R.S. Re. dAa 122081155
Label
Definition Functionality
Caption
plasis is unsician who plass the plana in the orchestia
"The pimiie pootioned the conter of stogs under the bright stage lighes, sarves as the central figure this profermncs liee dedled paying createe profensunall inmentee stmruphers
contest, the platist is petfie ming
Ponist
appeas be caeet lunge veaue
piano
aa P.e. 4.8
Figure 6: PAM provides various semantic granularities informantion and support bilingual outputs.
As shown in Table 1, both our PAM-1.5B and PAM-3B demonstrate strong performance. Notably, PAM-3B significantly outperforms other competing methods. It achieves optimal performance on the PACO benchmark, exceeding the previous best model by over 3.2%, and surpasses the current SOTA model, DAM-SB, on the LVIS benchmark in terms of semantic IoU. Furthermore, indicated in the right column of Table 1, our PAM-3B outperforms VP-SPHINX- by over 3.5% on Total-Text and achieves comparable performance on COCO-Text. These results demonstrate its promising capabilities in scene text recognition. We further showcase qualitative visualizations in Fig. 6, illustrating PAM's effectiveness in generating insightful explanations that cover both the general definition and the contextual role of prompted objects.
Regional Caption. We evaluate the model's capability to generate both concise and detailed region descriptions on multiple benchmarks. For concise region captioning, we evaluate on the validation splits of RefCOCOg [32] and Visual Genome (VG)[36]. For more expressive descriptions, assessments are conducted on the challenging Ref-L4[10] dataset. Caption quality is measured using ROUGE-L [39], METEOR [3], and CIDEr [60]. Additionally, we benchmark referring descriptions via Ferret- Bench [77] and MDVP- [41], where GPT-40 is employed gauge the quality of the generated responses.
As the results shown in Table 2, PAM-3B surpasses existing methods the VG, RefCOCOg, and Ferret henchmarks. On MDVP-Bench, it achieves performance comparable to the current Furthermore, on the Ref-L4 benchmark, PAM-3B demonstrates
SOTA method, VP-SPHINX- outstanding performance, surpassing all models except the top-performing DAM-SB. Notably, these competitive results are achieved with fewer parameters and reduced computational cost, highlighting PAM's excellent balance of performance and efficiency.
5.3
Video Benchmarks
Model
Elysium
METDOR
BenaMOT
METEOR
BCSTVV
CIDCr
SC
VideoReler BebeWeele-Beuch-DD
TD
Besch-D
HID
Avg.
CIDEr
ActivityNei
METFOR
G-STDC
Merin [78]
Clysum 70 [63]
19.1
1.1
11.3
10.5
2.35
0.30
0.02
3.59
1.77
VideoRefe--BB [8]]
DAM-1R [18]
11.3
14.7
Omni RGPT-R [25]
9.3
14.6
GIT [65]
29.8
VideoRater 7B [S1]
Artemis-79 [49]
18.7
18.0
68.5
4.45
3.42
3.10
1.39
3.46
2.26
Straming Vid2Seg [73]
Vis2Sog [73]
37.8
30.2
10.0
DAM-3B [58]
21.0
4.69
3,34
3,68
PAM-1.5B lours)
28.6
24.8
2.43
PAM-1.5H (ours)
22.7
20.1
19.5
3.73
2.77
3.03
PAM-NB (ours)
30.1
27.3
2.07
(omrs)
23.3
2.88
Table 4: Performance for streaming Table 3: Performance comparison on video region captioning. region captioning on ActivityNet.
Video Region Caption. This task requires the model to generate an accurate and temporally- aware description for a prompted region within the video's context. We primarily evaluate on four public benchmarks: Elysium [63], BensMOT [37], HC-STVG [59], and indeoRefer-Benche [81]. As shown in Table 3. our PAM-1.5B and PAM-3B achieve the SOTA performance on both the Elysium and BensMOT benchmarks. Furthermore, our PAM-3B surpasses the current SOTA method, DAM- 8B, by 2.3% in terms of METEOR on the HC-STVG benchmark. On the VideoRefer- Bench, our
8

---

Non Non-Streal Stre-ttmmm
dhsplaynng
the viden, hand halding small but mailt muved.
0. dim &
Stream
wabe and dornt stande baskethall the trandiions las dyaani
Ti actens practiing drmahing lix the Beal than
The run is sow postionnd the nurt It starts tring up
-0.8 Ta 0.0.......
SLa S.M.A 2 9. &t 14 (4 P.BA 14-6 0RRRE R.
8427514. NLO 4960
SRAR URRAA
Figure 7: Qualitative visualization examples of PAM for region-level non-streaming and streaming video caption.
models exhibit marginally lower performance compared to VideoRefer-7B and DAM-8B, indicating potential for further improvement.
Streaming Video Region Caption. This task requires the model to generate continuous descriptions for a prompted region in streaming manner. For evaluation, primarily utilize the validation set of the ActivityNet dataset [7]. To ensure a fair comparison and to accurately assess region-level streaming captioning capabilities, manually curated subset of 400 samples. This selection process adhered to two key criteria:
(/) each annotated event within given video is temporally continu- ous and non-overlapping, and (2) all annotated event descriptions for that video pertain to the same subject. Subsequently, we manu- ally annotated bounding boxes for the target subject in each selected video. We initially employ two standard dense captioning metrics for evaluation: CIDEr and ME- TEOR. To further assess the con- tinuity and entity consistency of descriptions for sequential events, propose new metric: the GPT-4o-evaluated Spatio- Temporal Description Continuity Score (G-STDC), which ranges from 0 to (Details in App. C). The results in Table indicate that recent region-level video caption models. including VideoRefer and DAM, exhibit limited capability in the streaming caption task. Compared to general streaming caption approaches such Streaming Vid2Seq. our PAM-3B outperforms on the METEOR metric. Furthermore, PAM-3B achieves optimal performance on G-STDC, indicating its excellent apatio-tempore continuity and ability to maintain consistent subject descriptions.
Image
Runtiine (msiltent)
Nemory Use (GB) Runime (xlkam))
2210
1.2-2.4x faster
memory
2012
Memory Use (GB)
Figure 8: Comparison of GPU memory usage and inference efficiency on an A6000 GPU,
5.4 Efficiency
As shown in Fig. 8, compared to existing works, our PAM demonstrates superior inference efficiency and requires less GPU memory for both image and video processing, highlighting its suitability for efficient deployment in real-world applications.
5.5 Ablations
We study the effectiveness of the proposed key techniques below.

---

RecCOCOs
(METSOR)
Method
(5.ioU)
LVTS
ReCCOCOz (METEUR)
ITC (METEOR)
Method
LVIS (5.1o1))
RarCOCOs (METTRAR)
(MFTTOR))
STVG
T4een
10 =
scauT
80.0
78.9
79.5
2700
76.9
26.1
24.6
253
23.5
225
1343
980
572
S1 S1-2
SI 1.55 -22
All onz
79.6
79.7
78.7
26.7
26.9
25.8
23,3
21.6
22.4
mark -.m.7
pro ST-TEM
senil
79.6
78.6
26.0
25.0
23.3
Table 5: Number of Sem.T.
Table 6: Different training stage. Table 7: Impact of different in-
termediate embeddings.
In Table 5, present the impact of adjusting the number of Icarnable semantic tokens (sem.T). It is observed that using an insufficient number of sem, leads to drop in performance. Conversely, using an excessive number of sem.T T results in diminishing gains, while also increasing the computational cost. Therefore, we select 16 sem.T to achieve a favorable performance-efffciene. trade-off.
In Table 6, we compare different training strategies. It is seen that initialization from the image- video model checkpoint (from Stage 1.5) consistently leads to enhanced performance compared to either initializing directly from Stage model checkpoint or training directly in an all-in-one stage.
Table 7 compares the impact of different intermediate features from SAM 2. The results show that embeddings updated by S2-FFM enhance our model's performance, which further underscore the critical role of the feature selection approach.
Conclusion
We present Perceive Anything Model (PAM), a region-level vision-language model extended from SAM 2, designed for simultaneous segmentation of objects while generating diverse semantic outputs of them across both images and videos. PAM demonstrates robust performance multiple region-level understanding tasks while achieving high computational efficiency. The simplicity and efficiency of our approach make it well-suitable for real-world applications, enabling a fine-grained, multi-dimensional understanding of visual content from a single interaction.
10

---

Acknowledgments
This study was supported in part by National Key R&D Program of China Project 2022ZD0161 100. in part by the Centre for Perceptual and Interactive Intelligence, a CUHK-led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government, in part by NSFC-RGC Project N CUHK498/24, and in part by Guangdong Basic and Applied Basic Research Foundation (No.2023B1515130000, XW).
References
[1] Ruichuan An, Sihan Yang. Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411. 11161 2024.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Siho Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.1392. 2025.
[3] Satanjeev Banerjee and Alon Lavie. Metcor: An automatic metrie for mt evaluation with improved correlation with human judgments. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarizaaion,, pages 65-72, 2005.
[4] Chandra Bhagavatula. Ronan Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning, 2020.
[5] Yuri Boykov and Olga Veksler. Graph cuts in vision and graphics: Theories and applications. In Handbook of mathematical models computer vision, pages 79-96. Springer, 2006.
[6] Yuqi Bu, Liuwu Li. Jiayuan Xie, Qiong Liu, Yi Cai, Qingbao Huang, and Qing Li. Scene-text oriented referring expression comprehension. IEEE Transactions on Multimedia, 25:7208 7221, 2023.
[7] Fabian Caba Heilbron. Victor Escorcia. Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961 970, 2015.
[8] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jac Lee. Vip-Ilava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12914-12923, 2024.
[9] Tony F Chan and Luminita. A Vese, Active contours without edges. IEEE Transactions image processing. 10(2):266-277, 2001.
[10] Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S. H. Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the of large multimodal models, 2024.
[11] Joya Chen. Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18407-18418, 2024.
[12] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal l1m's referential dialogue magic. arXiv preprint arXiv:2306.1512 2023.
[13] Yangming Cheng, Liulei Li. Yuanyou Xu, Xiaodi Li. Zongxin Yang. Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06558 2023.
[14] Chee Kheng Chng and Chee Seng Chan. Total-text: A comprehensive dataset for seene text detection and recogmition, 2017.
[15] Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary-fiee free image classification, 2024.
[16] Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snock, and Tinne Tuytelaars. Online action detection. In Computer EiCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 269-284. Springer, 2016.
[17] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang. and Chen Change Lay. Mevis: A large-scale benchmark for video segmentation with motion expressions, 2023.

---

[18] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang, Li, and Cees GM Snoek. Actor and action video segmentation from sentence. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5958-5966, 2018.
[19] Raul Gomez. Baoguang Shi. Lluis Gomez, Lukas Numann, Andreas Veit, Jiri Matas, Serge Belongie, and Dimosthenis Karatzas. 1edar2017 robust reading challenge coco-text. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 1435-1443 2017.
[20] Kristen Grauman, Andrew Westhury, Fugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022.
[21] Qiushan Guo, Shalini De Mello, Hongxu Yin. Wonmin Byeon, Ka Chun Cheung. Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model, 2024.
[22] Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. Trace: Temporal grounding video Ilm via causal event modeling. arXiv preprint arXiv:2410.0544 2024.
[23] Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. 2019.
[24] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images, 2016.
[25] Miran Heo, Min-Hung Chen, De-An Huang. Sifei Liu, Subhashree Radhakrishnan, Seon Joo Kim, Yu- Chiang Frank Wang, and Ryo Hachiuma. Omni-rgpt: Unifying image and video region-level understanding via token marks, 2025.
[26] Xiaoke Huang, Jianfeng Wang, Yansong Tang, Zheng Zhang, Han Hu, Jiwen Lu, Lijuan Wang, and Zicheng Liu. Segment and caption anything. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 13405-13417, 2024.
[27] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- system card. arXiv preprint 21X1v:4400.1176. 2024.
[28] Menglin Jia, Luming Tang, Bor- Ch-C Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709 727. Springer, 2022.
[29] Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. Chatrex: Taming multimodal 1lm for joint perception and understanding. arXiv preprint 2rX 1816663, 2024.
[30] Dimosthenis Karatzas, Lluis Gomez Gomez-Bigorda. Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition robust reading. In 2015 13th international conference document analysis and recognition (ICDAR), pages 1156-1160. ICEE, 2015.
[31] Dimosthenis Karatzas, Faisal Shafait, Seiihhi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazán Almazán, and Lluís Pere de las Heras. ledar 2013 robust reading competition. In 2013 12th International Conference on Document Analysis and Recognition, pages 1484-1493, 2013.
[32] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[33] Lei Ke, Mingqiao Yc, Martin Danelljan, Yu- Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36:29914-299934 2023.
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643. 2023.
[35] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El El-Haija Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale labti-label and multi- class image classification. Dataset available from https://githuu. com/openinages. 2(3):18, 2017.
12

---

[36] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
[37] Yunhao Li, Qin Li, Hao Wang, Xue Ma, Jiali Yao, Shaohua Dong. Heng Fan, and Liho Zhang. Beyond mot: Semantic multi-object tracking. In European Conference on Computer Vision, pages 276-293. Springer, 2024.
[38] Long Lian, Yifan Ding, Yunhao Ge, Sifci Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint 2rXiv:2504. 16072, 2025.
[39] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summurization branches out, pages 74-81, 2004.
[40] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[41] Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. araddaad-understand: Leveraging visual prompts to enable milms to comprehend what you want. arXiv preprint 2.XX:::::333222... 2024.
[42] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[43] Chang Liu, Henghui Ding, and Xudong Jiang. Gres; Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23592-23601, 2023.
[44] Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402. 059355, 2024.
[45] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[46] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[47] Jordi Pont- Tont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arheláez, Alex Korkine-Hornung and Lue Van Gool. The 2017 davis challenge video object segmentation. 2018
[48] Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in complex videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[49] Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma. Pengyu Yan, David Doermann, Qixiang Ye, and Yunjic Tian. Artemis: Towards referential understanding in complex vidcos, 2024,
[50] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. Paco: Parts and attributes of common objects. 2023.
[51] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model, 2024.
[52] Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and vidcos. arXiv preprint arXiv:2408. 00714, 2024.
[53] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos, 2024.
13

---

[54] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin. Kunchang Li. He Cao. Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, al. Grounded sam: Assembling open-world models for diverse visual tasks, arXiv preprint 4rXiv:2401. 14159, 2024.
[55] Shuai Shao, Zeming Li, Tianyuan Zhang. Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ohjects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[56] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Textocr: Towards scage-scale end-to-end reasoning for arbitrary-shaped scene text, 2021.
[57] Yipeng Sun, Zihan Ni, Chee- Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng. Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, and Lianwen Jin. Icdar 2019 competition on large-scale street view text with partial laheling - rrelsstt, 2019.
[58] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal vidco grounding with visual transformers, 2021.
[59] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang. Qian Yu, and Dong Xu. Human- centric ppatio-tempone video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Techmology. 32(12)88238 8249, 2021.
[60] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basd image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566 0666-457,2222 2015.
[61] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images, 2016.
[62] Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via milm, 2024.
[63] Han Wang, Yongjic Ye, Yanjie Wang, Yuxiang Nic, and Can Huang. Elysium: Exploring leject-level perception in videos via mllm. In European Conference on Computer Vision, pages 166-185. Springer, 2024.
[64] Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, XU Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open-vocabulary video instance segmentation, 2023.
[65] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. Git: A generative image-to-ter text transformer for vision and language, 2022.
[66] Jiaqi Wang. Pan Zhang, Tao Chu, Yuhang Cao. Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset, 2023.
[67] Teng Wang. Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li. Minggi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint arXiv:2305.02677 2023.
[68] Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint 27X1v2230..01907. 2023.
[69] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-lext transformer for object understanding. In European Conference on Computer Vision, pages 207-224. Springer, 2024.
[70] Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei La, Tong Lu, et al. Visionlim v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks, Advances Neural Information Processing Systems, 37:69925-69975. 2024.
[71] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark, 2018.
[72] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint 15115 2024.
14

---

[73] Antoine Yang. Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tusel Ivan Laptev, Josel' Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning, 2023.
[74] Cheng- Yen Yang. Hsiang- Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-awar memory, 2024.
[75] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding gpt-4v, 2023.
[76] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu. Detecting texts of arbitrary orientations in natural images. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1083-1090. 2012.
[77] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704. 2023.
[78] En Yu, Liang Zhao, Yana Wei, Jinrong Yang. Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge. Xiangyu Zhang, et al. Merlin: Empowering multimodal Ilms with foresight minds. In European Conference on Computer Vision, pages 425 443. Springer, 2024.
[79] Haobo Yuan. Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi. Jiashi Feng, and images and videos, 2025.
Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of
[80] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202-28211, 2024.
[81] Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yucting Zhuang, et al. Videorefer suite: Advancing spatial temporll object understanding with video Ilm. arXiv preprint arXiv:2501.000 .05999, 2024.
[82] Chaoning Zhang, Dongshen Han, Qiao, Jung Uk Kim, Sung-Ho Bac, Scungkyu Lec, and Choong Scon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:23066. 14289, 2023.
[83] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, 7he Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint 07499 2024.
[84] Shilong Zhang, Peize Sun, Shouta Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model region-offimterest, 2025.
[85] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In CVPR, 2020.
[86] Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunnui Han, et al. Chatspot: Hootstrapping multimodal llms via precise referring instruction tuning. 2023.
arXiv preprint arxiv:2.307.044
[87] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306. 12156, 2023.
[88] Yuzhong Zhao, Yue Liu, Zonghao Guo, Weijia Wu, Chen Gong, Qixiang Ye, and Fang Wan. Controleap: Controllable region-level captioning. In European Conference on Computer Vision, pages 21-38. Springer, 2024.
[89] Xingyi Zhou, Anurag Arnah, Shyamal Buch, Shen Yan, Austin Myers, Xuchan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pages 18243-18252, 2024.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The contributions and scope of the paper are included in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
The limitations section is included in the appendix (Section D)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We introduce the details of the experiment, such as the information on hardware, in the implementation detail section (Section 5.1) and Appendix (Section A)
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Begllii/guides/CoeeStte sot.eeuuubsssiioooooocoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cea.ps..ccpuubiccuuuuuu gesds///////eeeeee/e (CodeSubmian ss cesssonPolllo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the details of the experimental settings, such as the data split, optimizer, etc., in the Table.8
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We provide the details of compute resources the implementation section (Section 5.1) and efficiency analysis section (Section 5.4)
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /ttpps::///uurss.... wttpiiss..cpppppppe attpss//aaurrss.ccpppb. httpc//////tttttttttccleeeeeeeee csGuidelines: delis
Answer:
[Yes]
Justification:
We have made sure that our paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive impacts that our method will bring in the Introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
20

---

Justification:
There is no risk of misuse of the proposed method.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited original paper or attached the link to the existing assets used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
21

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https httpps://neuriai htt///eurip...// wct.oooeeeeeccc///000000 for what should or should not be described.
22

---

Appendix
A
Configuration for Each Training Stage
Table 8 details the configurations for each training stage of the Perceive Anything Model (PAM). It outlines the vision parameters, dataset characteristics, model specifications, and training hyperparam- eters throughout the curriculum learning stages. The maximum number of visual tokens varies by input modality: single images are represented using 1024 tokens, while for videos, we sample up to 16 frames, leading to a maximum of 4864 visual tokens. A global batch size of 1024 is used for stages and 1.5, and 256 for stage 2.
Stage
Stage
Stage 2
Visual Tokens
1024
1024 + N56 MAX 1024 15x256=
2x6 MAX 1024 15x 4864
Sem. Tokens
1x16 (MAX 256)
1x (MAX 256)
image classification
image classification
image classification image caption
image explanation
Dataset
image caption video caption
video caption streaming vidco caption
Trainable components 1.5B
Sem. Perceiver Projector
Sem. Perceiver Projector 7.6M
Sem. Perceiver, Projector, LLM
7.6M
1.6B
3B
7.7M
7.7M
3.1B
Batch Size Learning Rate
1024
1024
256
1x
4x10
xx10
Epoch
1
Warmup Ratio
0.03
0.03
0.03
Optimizer
AdamW
AdamW
AdamW
Table 8: Detailed configuration for each training stage of the PAM.
B Dataset
Recognition & Esplanation
Reziun Recupaitin Explanation (35%)
Opentmages
0bject365
Vddeo Caption
Syntext
ReCTs
Saecitd Amethnnis
The Preeiision
TextOC
TotalTost
ICDAR 13/15/17
Whote
Bogion Image Captios (60%)
8M
A5-100M
Chattlex
Refcoco
Ref-Text
Refcoco+
Refcocog
MDVP Data
Osprey-724K
gRefonco
Caption Streaming
Image Caption
VidSTG
HC-STVG
Youtube-vOS
Mevis
SA2VA
Elysium
A2D
DAVI517
LV-VIS
Figure 9: All Public Dataset Collection. Datasets highlighted in bold selected for further refinement and augmentation pipeline, aimed at generating a high-quality training dataset.
B.1 More Details of Image Data Construction
This section details image data construction process. To generate data encompassing clear conceptual explanations, contextual functional roles, and detailed descriptions of specific regional capabilities, primarily leveraged the extensive world knowledge of leading VLMS.
Our approach involved several stages. Initially, we collected data from public referring object detection, segmentation, and captioning datasets [32, 6, 43, 36, 29, 68]. While these sources provide
23

---

Image
DAM-30
VP-SPIINX--13
PAM-3B
white cst sitting upeight with its front pars slightyy sport. The cat has black putch left side of its fece and anotther black patch its right side of botty and upright, and alert.
The
The predominantly
shite parts sttiing down
body
black patches
upright and looking forward
The blcck patch covers
accuss
maybe
head octends
back,
made of plastic: The has some darfeer black spoes or
tall
carted
right
pointed
sidas The cat has semll, alaet sarts
patches on its bads, head and
The eyes are not dearty detalled but
on its sides. Its cars are
are looldng forward. The for testure
sticking up, like it's litening
suggests a steck, short cost.
A camel with alender elangated neck and small, rounded head barty is coverne in smouth, tielk-inted cooi, with alightly darkeer stade slong spine, camel's logs
The camel on the is tall and has a stngle humny Itss stundy body is covered darks brows far All four of its long, slender logs are vasile. Theee lugs und in browe, padded reet, good for wallinng on sand. short tall hange down at badk. This comnel port of lorger group and sowms to be moving along with them.
The camel stand ing still its head held up a little and it has a long, curved neck There is lump its back, lis body looks strong and sturcly It has legs than plinte the ground.
with visile and slight the legs.
hangs
vertical, pale featuring a smorth surfaox The pole equipped a sertes of eventy spaced,
This sylindrial post, wood. likely from being repooed to wloments over time, giving a somewhat rough surface. The pole set upright, sticking straight out the ground.
Much particularty lower faded, ghring rough and dry appearance, band weogpeed amound this pols. wondeen post shurter than white ring while helping hold the large circular ats uctuore in place.
hit
pears be
hortzontal hands that encircle it, which could serve as grip points or decorstioe live is At the top, the pole sstends into perpanddaauler, flat-togped elem ent that appears to be cap or flsture.
Figure 10: Qualitative comparison between PAM and prior models.
unique descriptions for each region, they often lack comprehensive semantic details. Therefore. our work focused on refining and augmenting these data to achieve richer semantic granularity. Specifically, the Bet-of-Mark (SoM) prompting method [75] was initially employed to identify regions of interest within the images. To generate high-quality conceptual explanations and contextual functionalities for these identified regions, we manually refined the prompts and then utilized strong dlsed-ssourced model, GPT-4o [27]. to produce the desired textual responses. For crafting detailed descriptions of these regions, a powerful open-sourced model, Swee2.55VL-772 [2], was employed to expand and supplement existing textual information. Following this automated expansion, two-stage cleaning process was implemented. First, rule-based methods were applied for preliminary filtering, addressing issues such output format inconsistencies. Subsequently, manual review was conducted to identify and isolate remaining inaccurate or low-quality data, which were then re-annotated.
B.2 Bilingual Annotations
To support bilingual (English and Chinese) output, we extended our refined datasets by generating corresponding Chinese versions. For the majority of these datasets, Chinese annotations were directly created using Chinese prompts. cases where data was initially generated with GPT-4o, the existing English content was translated into Chinese utilizing the DeepSeek- V3 model [42].
C Implement Details of G-STDC
In Sec. 5.3, we introduce the GPT-4o-evaluated Spatio-Temports Description Continuity Score (G-STDC) to assess the continuity and entity consistency of descriptions for sequential events. Specif- ically, for given video, the predicted descriptions and the corresponding ground truth descriptions for its multiple events are first sorted chronologically. Both sequences are then provided to GPT-4o for evaluation. During this process, GPT-4o assesses the predicted descriptions based on temporal continuity and entity consistency (in relation to the ground truths), assigning a G-STDC score on a scale of to 5, where represents the optimal score and 0 the poorest. Results are presented in Table 4.
24

---

D More Analysis of PAM
D.1 Performance for Background
This a stroight path with smooth, paved surface likelt concrete. it starts the foreground and leads off to the right, into the distance. The path itseff appeans clean and has lines
pattern many dhoppy appcarance they reflect larg waves signifiantt turbulence.
relattrely
This gry wall sumredat mugh surfee,, charattetstic building's unique facade This wall surrondss rectangular window with dark framn.
ripples, giving
Figure 11: PAM can accurately describe specific background areas, such as roads, ground surfaces, sea, walls, the sky, and more.
D.2 Failure Cases
The posittoned third from the left, tranelurent, desely the other pentle the
This small, spherical container holding and with metallic color glossy surbsce brightly reflects the letters delicuncly isseribed polished roundacu, possibly indicuting handmade.
dreallr
round
flesh.
sign's
dign
on stardy
acting clear marker sidowalk.
Figure 12: Failure Cases of PAM in Images.
Fig. 12 illustrates several of PAM's failure cases. As these examples show, PAM sometimes makes
errors in its descriptions: (1) For instance, in the first image, the orange slice is the second one, but PAM describes as the third. (2) In the third image, the actual text is 'Tack', but PAM reads present in the image, such describing engraved letters in the second image, which has no such features.
'Teet', (3) PAM occasionally describes elements
As shown in Fig. 13, PAM also exhibits certain limitations in video contexts. Specifically, if user-specified object occupics a minor portion of the frame and temporarily disappears from view, PAM may default to describing the most salient object in the scene instead. We attribute this behavior to potential biases the training data: the GPT-assisted method employed for annotation during data construction might favor describing the most salient object over the specific region, thereby introducing label inaccuracies. Additionally, in streaming captioning, PAM might be influenced by historical descriptions, leading its output for the current video clip be overly similar that of the preceding clip.
We expect these errors be mitigated by broader data coverage and further reinforcement training.
25

---

woman walking down a relatively empty street while holding her phone, She is wearing a blue jacket and appears to be taking selfie or recording a video with her phone, as her head turns from time to time. Her expression is quite calm.
Figure Failure Cases of PAM in Videos.
D.3 Performance on Long Videos
PAM's performance long videos depends the number of video frames processed. A larger number of frames enables PAM to generate more information-rich descriptions, but this incurs an exponential increase computational cost. With its default setting sampling frames, PAM can only provide broad and coarse descriptions of the states and changes of specific subjects in long videos. However, PAM features region-level streaming captioning capability that can improve its handling of long videos. This method involves segmenting a long video into multiple shorter clips; descriptions are then generated for each clip sequentially and subsequently merged create a single, detailed description of the entire video.
E Potential Limitations and Discussions
Limited Capability for General Understanding Tasks. PAM is currently trained for a specific set of four region-level understanding tasks: category prediction, brief and detailed regional captioning, video captioning, and streaming region captioning. Therefore, it presently lacks support for other general vision-language tasks, such Visual Question Answering (VQA). However, the architecture and training strategy of PAM are inherently well-suited to accommodate these broader functionalities. Looking ahead, plan develop additional high-quality conversational datasets extend PAM's capabilities to encompass both region-level image and video dialogue.
Limitations in Real-Time Streaming Video Region Captioning. Despite being 1.2-2.4x faster than existing models, PAM's capability for real-time streaming video region captioning is currently hindered by the excessive number of visual tokens requiring processing by the L.M.. In the future, we aim to further identify methods for substantially reducing the number of visual tokens, with the goal of achieving real-time efficiency while maintain the robust understanding performance.
26


Reviews:
[{"review_id": "EAhnlrsLRU", "reviewer": "Reviewer_H65M", "summary": "This paper proposes Percieve Anything Model (PAM), which integreates the visual understanding ability of SAM2 into LLMs. Specifically, a frozen SAM2 is connected with LLM with trainable semantic perciever. While the segmentation ability of PAM is fully inherented from SAM2, such integretion provides detailed visual information for LLMs to perform recognition, explaination and caption. To train such integreted model, the authors builds a billingual visual instruction tunning dataset, which mixes recgnition, explanation and caption on images with multi-scale video caption. After training on such dataset, the PAM shows superior performance on classification, OCR, region caption, and video caption tasks.", "questions": "1. Does the model lost its instruction following ability after such training? \n2. Why is SAM2 frozen during training? \n3. Can this framework generalize to other understanding tasks? Does SAM2 representations still outperform other models?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of using SAM2 as visual encoder for LLMs to provide better capability on specific understanding tasks is novel.\n2. The paper is well written and easy to follow.\n3. The experiments shows that PAM can outperform existing MLLMs on various tasks.\n\nWeaknesses\n1. Does the model lost its instruction following ability after such training? According to Figure 2 and Section 4, it seems the LLM takes aligned SAM2 outputs and gives the label, discription, and/or caption directly during training. If so, the LLM could lost its instruction following ability and cannot give perception result for text beased prompts (like \"the animal behind the door\"). Also, it will lost the LLM's ability as fundation model, which highly limits the application of this framework compared to methods like DAM [1].\n2. The reason why SAM2 is frozen during the training is not clear. Typically, when applying visual alignment with a vision encoder and LLM, we will (in some stages) finetune the vision encoder for better performance [2,3]. However, in this framework, the SAM2 is completely forzen, and the reason of such conter-intuative design is not well illustrated.\n3. Compared to standard pretrained ViTs, representations from SAM2 naturally contains more information on segmentation, therefore is reasonable to have better performance on specific tasks like recognition, explaination, and caption. However, such structure and representation seems not effective on other understanding tasks (e.g., visual question answering), limiting the generalization of this framework.\n\nOverall, although the framework shows superior performance on multiple perception tasks, such gain seems sacrifice the instruction following and generalization ability of LLM, which significantly limits the application of both framework and the trained model compared to similar prior approaches [1].", "was_revised": true, "final_justification": "I have carefully considered the paper and the authors' rebuttal. The authors have addressed my concerns. After thorough evaluation, I have decided to raise my recommendation to Borderline Accept (4).", "rebuttal": "## **Response to W1 and Q1**\n\nThanks for your comments. However, we think this concern is based on a misunderstanding of our research. Here are the reasons:\n\n> #### **1. Does the model lost its instruction following ability after such training?**\n\nOur goal was to align the text-only LLM (Qwen2.5) with SAM 2 to create a VLM that excels at powerful region-level visual caption. After training, PAM is endowed with **new visual instruction-following abilities**, including open-set classification, functional explanation, brief and detailed captioning, and streaming video captioning. Therefore, the LLM’s original ability to follow text instructions is **not a key factor in PAM.**\n\n> #### **2. The LLM could lost its instruction following ability and cannot give perception result for text beased prompts.**\n\nUsing text-based prompts to identify objects is a feature of general VLMs; the Qwen2.5 LLM, however, never had this capability, as it is unable to process multimodal data. Moreover, we designed PAM to use visual prompts (points, boxes, masks) from SAM 2 because they offer higher precision in complex scenes where describing an object with text can be difficult and ambiguous.\n\n> #### **3. It will lost the LLM's ability as foundation model, which highly limits the application of this framework compared to methods like DAM.**\n\nWe think that by integrating two powerful foundation models (SAM 2 and Qwen2.5), we can create a new and robust region-level vision-language foundation model, excelling in fine-grained visual understanding. Our results show this specialized model achieves superior performance on multiple tasks compared to other methods. \n\nFor DAM, it is also specialized, focusing primarily on captioning tasks while not supporting others like VQA. Furthermore, DAM is less flexible, as it only accepts mask-based visual inputs. In our experiments, PAM-3B outperforms DAM-3B on multiple benchmarks, demonstrating its effectiveness.\n\n| | LVIS (Img Cls.) | | Ref-L4 (Img Cap.) | | | VideoRefer (Video Cap.) |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| | Sem. Sim. | Sem. IoU | ROUGE-L | METEOR | CIDEr | Avg. |\n| DAM-3B | 87.2 | 75.3 | 30.8 | 17.2 | 56.4 | 2.99 |\n| PAM-3B | **88.6** | **78.3** | **31.1** | 17.2 | **59.7** | **3.14** |\n\n\n## **Response to W2 and Q2: Why is SAM2 frozen?**\n\nHere are the reasons:\n\n1.\tThe design of PAM, has two main goals: (1) to generate precise masks based on visual prompts, and (2) to provide rich semantic information for those masks. If it was unfrozen and retrained, the SAM2’s existing segmentation capability could be degraded. While it’s possible to train both SAM2’s segmentation and the LLM’s visual understanding simultaneously, this approach would not only require significantly more computational resources but also introduce a more complex multi-task learning challenge, making the model more difficult to optimize. Therefore, freezing SAM2 is a far more efficient strategy for the overall training process.\n\n2.\tSAM2 inherently contains rich pixel-level information. A key part of our work was to figure out how to best select latent features from SAM2 to efficiently train a region-level VLM. In our ablation studies, we investigated how choosing different latent features impacts model performance, and we analyze the potential reasons in the Methods section. Our experiments demonstrate that it is possible to achieve a high-performing PAM model without retraining SAM2.\n\n3.\tTo further validate this, we ran an additional experiment. We trained PAM-1.5B from stage 1.5 checkpoint on 0.5M region-level classification data under two conditions: once with SAM2's parameters frozen, and once with them unfrozen. The results showed that unfreezing SAM2 offered no significant performance boost.\n\n| | **LVIS** | | **PACO** | |\n| :--- | :---: | :---: | :---: | :---: |\n| PAM-1.5B | Sem. Sim. | Sem. IoU | Sem. Sim. | Sem. IoU |\n| -SAM2 unfrozen | **85.9** | 74.2 | 81.9 | **71.9** |\n| -SAM2 frozen | 85.7 | **74.3** | **82.0** | 71.6 |\n\n\n## **Response to W3 and Q3: Can this framework generalize to other understanding tasks?**\n\nAlthough PAM was designed to specialize in accurate and detailed region-level descriptions, its architecture inherently supports other understanding tasks (e.g., visual question answering) without requiring any modifications.\n\nTo demonstrate this, we collected and trained our model on several open-source, region-level VQA datasets, such as MDVP-Instruct, VCR, Visual7W, and Osprey-724K. The training data were constructed in a format like, \"What kind of clothes does the man wear in this \\<region\\>”\n\nThe experimental results are as follows:\n\n| | **Ferret Bench** |  | **VCR** | |\n| :--- | :---: | :---: | :---: | :---: |\n| | Reasoning | Q -> A (%) | QA -> R (%) | Q -> AR (%) |\n| Osprey-7B | 67.8 | - | - | - |\n| Ferret-7B | 67.3 | - | - | - |\n| VILLA-L | - | 78.4 | 82.5 | 65.1 |\n| GPT4RoI-7B | - | 87.4 | **89.6** | 78.6 |\n| **PAM-3B (ours)**| **68.1** | **87.9** | 89.3 | **79.1** |"}, {"review_id": "HPO82Y8DKV", "reviewer": "Reviewer_y2LG", "summary": "This paper focuses on the vision language model area and introduces an end-to-end vision-language model, PAM, to bridge the gap between object segmentation and deep semantic understanding in both images and videos. This work also introduces a large-scale, high-quality dataset comprising 1.8M image and 0.6M video samples, which features multi-level semantic granularity. The authors conducted comprehensive experiments and validate the effectiveness of the proposed architecture and dataset.", "questions": "See weaknesses", "limitations": "None", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The authors successfully integrate segmentation with deep and multi-level semantic understanding, which is a significant step towards more holistic machine perception.\n2. The authors have made a substantial contribution by curating a large and high-quality dataset. The dataset's multi-dimensional annotations (category, definition, function, caption) and its bilingual support are valuable.\n3. The model's capabilities are thoroughly validated across a wide array of demanding benchmarks for both images (LVIS, PACO, RefCOCOg, Ferret-Bench) and videos (Elysium, BensMOT, HC-STVG). The authors also propose a new, relevant metric, G-STDC, for evaluating the continuity of streaming captions, demonstrating a thoughtful approach to assessment.\n\nWeaknesses\n1. The authors transparently acknowledge that PAM is specialized for a specific set of region-level understanding tasks. It currently lacks support for more general vision-language tasks like open-ended Visual Question Answering (VQA) or region-level dialogue, which limits its scope compared to some other large multimodal models.\n2. The experiments and ablation studies in this work were primarily conducted on Qwen2.5-1.5B/3B. This relatively limited selection raises concerns about the generalizability and scalability of the method.", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to W1: Lacks support for more general vision-language tasks like open-ended (VQA)**\n\nAlthough PAM specializes in providing accurate and detailed region-level descriptions, its core architecture inherently supports other understanding tasks, such as visual question answering (VQA), without any modification.\n\nTo demonstrate this capability, we trained our model on several open-source, region-level VQA datasets, including MDVP-Instruct, VCR, Visual7W, and Osprey-724K. The training data was formatted as question-answer pairs targeting a specific area, for example: “What kind of clothes is the man wearing in this \\<region\\>?”\n\nThe experimental results are as follows:\n\n| | **Ferret Bench** |  | **VCR** | |\n| :--- | :---: | :---: | :---: | :---: |\n| | Reasoning | Q -> A (%) | QA -> R (%) | Q -> AR (%) |\n| Osprey-7B | 67.8 | - | - | - |\n| Ferret-7B | 67.3 | - | - | - |\n| VILLA-L | - | 78.4 | 82.5 | 65.1 |\n| GPT4RoI-7B | - | 87.4 | **89.6** | 78.6 |\n| **PAM-3B (ours)**| **68.1** | **87.9** | 89.3 | **79.1** |\n\n## **Response to W2: Concerns about the generalizability and scalability**\n\nThank you for pointing this out. Our initial objective was to train a region-level VLM that is fast, compact, and high-performing, making it easier to deploy in practical, real-world scenarios. At the same time, we recognize that scalability is crucial. Therefore, we dedicated additional time and resources to train PAM-7B. The experimental results are as follows. As can be observed, PAM-7B exhibits outstanding performance, which demonstrates that our method possesses robust generalizability and scalability. Looking ahead, if sufficient computational resources are available, we will involve testing a 32B model to further evaluate its scalability.\n\n| | LVIS (Img Cls.) | | Ref-L4 (Img Cap.) | | | VideoRefer (Video Cap.) |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| | Sem. Sim. | Sem. IoU | ROUGE-L | METEOR | CIDEr | Avg. |\n| Ferret-7B | 65.0 | 37.8 | 22.3 | 10.7 | 39.7 | - |\n| VP-SPHINX-13B | 87.1 | 62.9 | 22.6 | 10.7 | 32.4 | - |\n| VideoRefer-7B | - | - | - | - | - | 3.46 |\n| **PAM-3B** | 88.6 | 78.3 | 31.1 | 17.2 | 59.7 | 3.14 |\n| **PAM-7B** | **88.9** | **80.1** | **36.5** | **19.1** | **65.9** | **3.50** |"}, {"review_id": "54NfEvJctI", "reviewer": "Reviewer_R2WL", "summary": "This paper introduces the Perceive Anything Model (PAM), a vision-language model designed for comprehensive region-level understanding in images and videos. PAM builds on the Segment Anything Model 2 (SAM 2) and integrates Large Language Models (LLMs) to achieve simultaneous object segmentation and diverse semantic output generation.  Compared to previous methods, PAM improvements in inference efficiency and GPU memory usage.", "questions": "- Data Processing using External Models\n\nThe authors mention using advanced VLMs such as qwen2.5-vl-72B, gpt4o, and deepseek v3 for data labeling and processing. The performance gains of PAM may not be attributed to its architectural design, but could be significantly determined by the power of these external models and the extensive labeled data utilized. Thus, the true effectiveness of PAM's own structure remains a question. \n\n- Comparison with Existing VLMs\n\nThe authors seem to avoid direct comparison of PAM with those existing VLMs such as open-sourced Qwen2.5-VL series. Although the author mentioned the differences between region-level VLMs and image-level VLMs, I think that these VLMs can be adapted to perform region-level tasks with the help of system prompts and few-shot examples. A fair comparison with these VLMs may be important. Since if existing VLMs are already capable of handling these tasks, the necessity of distilling a new region-level VLM appears to be questionable. \n\n- Explanation of Efficiency Gains\n\nThe authors highlight PAM's superior inference efficiency and reduced GPU memory usage for image and video processing. However, they do not provide a sufficiently detailed explanation of the sources of these efficiency improvements. It is unclear whether the faster speed is primarily due to the smaller model size (e.g., 3B vs. 7B), different model architecture, or other factors such as more efficient processing pipelines. A clear analysis may be better.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The integration of SAM 2 with LLMs via the Semantic Perceiver is a successful attempt to combining segmentation with rich semantic understanding, enabling multi-dimensional outputs like region-level captions and streaming video captioning. The performance and ablations show the power of the proposed method.\n- The video data refinement pipeline, storyboard-driven caption expansion, addresses a critical need and efficient way for fine-grained semantic annotations.\n\nWeakness:\n\nSee questions.", "was_revised": true, "final_justification": "The author's response solves my concern. I keep my rating as 4: Borderline accept.", "rebuttal": "## **Response to W1: Performance gains of PAM may not be attributed to its architectural design，but rather from its utilization of extensive labeled data**\n\nData is one of the key contributions of our work. We identified shortcomings in open-source data annotations and performed a comprehensive refinement. We acknowledge that this refined data provided a significant boost to PAM’s performance.\n\nHowever, for the classification and OCR tasks in Table 1, we used the original open-source data. On these benchmarks, our PAM-3B model achieves SOTA results on LVIS (semantic IoU), PACO, and Total-Text. Notably, all competing models we compared against are 7B or larger in scale. This demonstrates the effectiveness of the PAM architecture itself, independent of the data enhancements.\n\n## **Response to W2: Comparison with Existing general VLMs.**\n\nThank you for pointing this out. To demonstrate the necessity of region-level VLMs, we compared them against general VLMs on the DLC-Bench[1], which evaluates accuracy on detailed localized captioning. The experimental results are as follows:\n\n| Method              | Pos (%) | Neg (%) | Avg(%) |\n|:--------------------|:-------:|:-------:|:------:|\n| **General VLMs**    |         |         |        |\n| Llama-3.2-V-11B     | 30.7    | 63.8    | 47.3   |\n| InternVL2.5-8B      | 15.9    | 42.0    | 28.9   |\n| Qwen2.5-VL-7B       | 20.3    | 62.2    | 41.2   |\n| **Region-level VLMs** |         |         |        |\n| VP-SPHINX-13B       | 26.3    | **71.6**    | 49.0   |\n| PAM-3B (ours)          | **34.5**    | 63.1    | 48.8   |\n\nAs the results indicate, Region-level VLMs significantly outperform general VLMs on fine-grained detailed captioning tasks. This validates the necessity of research into region-level VLMs. \n\n[1] Describe Anything: Detailed Localized Image and Video Captioning\n\n\n## **Response to W3: Explanation of Efficiency Gains**\n\n| Model/Img. | Runtime (ms/item) | Memory Use (GB) |\n|:-----------------|:-----------------:|:---------------:|\n| Ferret-7B          | 1845             | 18.7            |\n| Omni-RGPT-7B     | 1970             | 20.8            |\n| PAM-7B           | **1615**          |  **17.1**      |\n\nThank you for your insightful comments. To better demonstrate the efficiency of our model, we trained a PAM-7B to compare it directly with other 7B models. This approach allows us to rule out the influence of model size and focus on architectural advantages.\n\nThe experimental results show that PAM-7B still outperforms others in terms of efficiency. This superiority can be attributed to three key factors:\n\n1.\tSAM 2, which serves as PAM’s vision encoder, is more lightweight compared to the encoders used in other VLMs.\n\n2.\tFewer Visual Tokens: The number of visual tokens processed from SAM 2 and fed into the LLM is significantly lower (1024 vs. more than 2048) than in other models.\n\n3.\tPAM utilizes the latent features from SAM 2, so it does not need to wait for the entire SAM 2 inference process to complete. Once the latent features are obtained, mask decoding and semantic decoding can proceed in parallel."}]

Author rebuttal:

Meta-review: The paper initially received 2 Borderline Accepts and 1 Borderline Reject ratings. While reviewers praised the presentation, effectiveness, experiments and interesting ideas, there were concerns on limitations for general visual-language tasks, unclear performance gains and missing experimental comparisons. The rebuttal helped significantly and assuaged most concerns, leading the negative reviewer to upgrade its score to favor acceptance as well, leading to all reviewers favoring acceptance. The AC agrees with the unanimous assessment and believes that the paper has merit, recommending it to be published at NeurIPS. We require that the clarifications, experiments and additional information provided in the rebuttal be incorporated in the camera-ready version.
Decision: Accept (poster)