UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
Authors: Ye Liu
Zongyang Ma
Junfu Pu
Zhongang Qi
Yang Wu
Ying Shan
Chang Wen Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Multimodal Models
Pixel-Level Reasoning
Primary area: deep_learning

Abstract: Recent advances in Large Multi-modal Models (LMMs) have demonstrated their remarkable success as general-purpose multi-modal assistants, with particular focuses on holistic image- and video-language understanding. Conversely, less attention has been given to scaling fine-grained pixel-level understanding capabilities, where the models are expected to realize pixel-level alignment between visual signals and language semantics. Some previous studies have applied LMMs to related tasks such as region-level captioning and referring expression segmentation. However, these models are limited to performing either referring or segmentation tasks independently and fail to integrate these fine-grained perception capabilities into visual reasoning. To bridge this gap, we propose UniPixel, a large multi-modal model capable of flexibly comprehending visual prompt inputs and generating mask-grounded responses. Our model distinguishes itself by seamlessly integrating pixel-level perception with general visual understanding capabilities. Specifically, UniPixel processes visual prompts and generates relevant masks on demand, and performs subsequent reasoning conditioning on these intermediate pointers during inference, thereby enabling fine-grained pixel-level reasoning. The effectiveness of our approach has been verified on 10 benchmarks across a diverse set of tasks, including pixel-level referring/segmentation and object-centric understanding in images/videos. A novel PixelQA task that jointly requires referring, segmentation, and question answering is also designed to verify the flexibility of our method.

UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
Ye Liu 1.2 Zongyang Ma..3 Junfu Pu?, Zhongang Qi', Yang Wu',
Ying Shan", Chang Wen Chen
The Hong Kong Polytechnic University 2 ARC Lab, Tencent PCG Institute of Automation, Chinese Academy of Sciences 4 vivo Mobile Communication Co.
MindWingman Technology (Shenzhen) Co., Ltd.
coco ye. www.ooneencccc....cmmmmmmmm wwwp:/ooonnnntt.ll......om. polyu. hk
https:/ppolyu-cheullae gi github... 10/ uni pixel/
Itaferring Expression Segmentation (RES)
Reasoning Segmantation (ReasonSeg)
Interactive Segmentation (15)
Par Souter (ettattche
Suaer Ue cameehe
Pessrapmen capplets
Refeering Video Object Segmentation (IVOS]
Reasoning Video Object Segmentation (BeVOS)
Matio--Gmunded Video Beasoning
Firer wcaer madilie Supping of deneleereee
Mor wreeeno
Referred Description
Video Queddon-Anmeetti
cleak plocas Aupura a Buoe tage une rallr arww ahii made
raulin whoti Maly Juin rveit?
Placl Level Video Question Answering (PixcIDA)
Joint Referning Sepmentatiinn QA Videos
Hour fers Neww of Afte jove Vet attree??
Añe gpeeing foadra (i and miget broour deran't for foor
Figure 1: UniPixel flexibly supports a large variety of fine-grained image and video understanding tasks, including referiig/rassonnn//neerccwc segmentation, motion-grounded video reasoning, and referred video description & question answering. can also handle a novel PixelQA task that jointly requires object-centric referring, segmentation, and question answering in videos.
Abstract
Recent advances in Large Multi-modal Models (LMMs) have demonstrated their remarkable success general-purpose multi-modal assistants, with particular focuses on holistic image- and video-language understanding. Conversely, less attention has been given to scaling fine- fine-grained pixel-level understanding capabili- ties, where the models are expected to realize pixel-level aligmment between visual
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

signals and language semantics. Some previous studies have applied LMMs to related tasks such as region-level captioning and referring expression segmentation. However, these models are limited to performing either referring or segmentation tasks independently and fail to integrate these fine-grained perception capabilities into visual reasoning. To bridge this gap, we propose UniPixel, a large multi-modal model capable of flexibly comprehending visual prompt inputs and generating mask-grounded responses. Our model distinguishes itself by seamlessly integrating pixel-level perception with general visual understanding capabilities. Specifically, UniPixel processes visual prompts and generates relevant masks on demand, and performs subsequent reasoning conditioning on these intermediate pointers during inference, thereby enabling aineiirieed pixel-level reasoning. The effectiveness of our approach has been verified on 10 benchmarks across a diverse set of tasks, and object-centric understanding in
including pixel-level referring/segmentaine images/videos. A novel PixelQA task that jointly requires referring, segmentation, and question answering is also designed to verify the flexibility of our method.
1 Introduction
Large Multi-modal Models (LMMs) have been the de facto standard for developing general-purpose assistants. By effectively aligning multi-modalittes with language, their significance has been demonstrated across various applications, including multi-modal analysis (59 19 107
108 48).
autonomous driving (AD) [6 79 [
and Embodied Al [11 22 30 99
In the field of visual-language understanding, efforts have been dedicated to developing holistic understanding models, where simple projection layers between visual encoders and LLMs are utilized to bridge vision and language modalities. Supported by large-scale alignment pre-training and visual instruction tuning. such a straightforward paradigm achieves strong performance in holistic
understanding tasks such as captioning [40 [14] and general question answering [36] 24 [4]
49.. However, these models exhibit two fundamental limitations in fine-grained scenarios. First, their interactions with users are limited to text format, lacking support for more intuitive forms of communication such as drawing points/boxes as references or grounding model responses with key regions represented by masks. Second, the internal reasoning process of these models predominantly operates at a coarse level, directly perceiving the entire content rather than reasoning over specific objects/regions, making them hard to understand fine-grained details. Some previous studies have explored the application of LMMs to related tasks such as region-level captioning [2] T02 103),
referring expression segmentation 229 55 21
62). and reasoning segmentation 32 27
96 4 (12). Nevertheless, their models are limited to performing either referring or segmentation
tasks independently via rigidly defined input/output templates (e.g.. "It's <SEG>." " in LISA (32).
lacking the flexibility to comprehend user-referred concepts and generate mask- gaos--groudde responses simultaneously. More importantly, these methods cannot integrate such fine- ane-grained perception capabilities with their original human-like 190, 89, 88 92, 91] multi-modal reasoning abilities, resulting in degraded performance on general visual understanding benchmarks [100 93 28
In this work, we seek to bridge this gap by introducing UniPixel, a large multi-modal model that can flexibly comprehend visual prompt inputs (i.e. points, boxes, and masks) and generate mask- grounded responses. Our model significantly differenniates itself from existing ones by unifying the internal representations of referred and segmented objects via a novel object memory bank, which is a hashmap storing the spatial-temporg information of object-of-interest During inference, UniPixel initializes the object memory bank and updates on demand by adding oeject-centric information according to the context. The model responses are then generated conditioning on the fine- mee-grained object memory. Benefits from such unification, UniPixel is able to perform not only basic seferring/segmenaatioo tasks, but also flexible pixel-level reasoning tasks that require simultaneous visual prompt comprehension and mask prediction. As illustrated in Fig. (the last row), given a V question, and optionally a visual prompt (e.g., point specified by click on an object in any frame), UniPixel can (1) infer the mask for the referred object in the corresponding frame, (2) propagate it to all video frames containing the same instance, (3) extract the mask-gru maskk-grounded object features, and finally (4) answer the question conditioning on both the video-level and object-centric
Images treated single-frame videos, thus do explicitly differentiate them in this work.

---

Image/Video
Point/Bax//Mass
Mosks
Bank
Now dons the behavior of [S] differ (4)P
Deseriñe this region.
Segmout the bunny.
B
Mow/Mask
Mask Decader
Mask Decoder
Encoder
Encoder
IE shows
Swre. It's kSEG>.
[S] is less interested in companed with (4).
(a) Referring-anly Models (e.g, Diprey, GPTARoL VideoRafor)
[b) Segmentation-anl Models
(c) Unified Pixel-level Reasoning Model
(egg USA, RlettMM VSAJ
Figure 2: Schematic comparison between UniPixel and its counterparts. To the best of our knowl- edge, UniPixel is the first unified method supporting simultaneous object referring and segmentation.
information. All these operations are seamlessly conducted within a single model, eliminating the need for external frame samplers [96], mask generators [122 [03]. or object trackers [4]
We evaluate the effectiveness of UniPixel from two aspects, i.e., basic referring/segmentte pabilities and flexible pixel-level reasoning capabilities. For the first aspect, we conduct extensive experiments on 10 public benchmarks across 9 image/video referring/semmenttionn tasks. Our method achieves state-of safe-oof-the of-thc-art art performance in diverse scenarios. Notably, on the challenging video reason- ing segmentation and referred video QA tasks, our 3B model obtains 62.1 J J&F on ReVOS (96) and 72.8% % Acc on VideoRefer- ReleoRefer-Benchh [103], surpassing strong counterparts with 7B ~ 13B parameters. Further ablation studies also demonstrate the mutual reinforcement effect of referring and segmenta- tion. For the second aspect, we introduce novel PixelQA task that jointly requires object-centric referring, segmentation, and QA in videos, which cannot be handled by existing methods. UniPixel establishes a strong baseline for this novel setting. Our contributions are summarized below:
ca-
1. We propose UniPixel, unified large multi-modal model that supports flexible object referring and segmentation in images and videos, via a novel object memory bank design.
2. Our model achieves state-of-the-at performance 10 public benchmarks across 9 refer- ring/segmentation tasks, verifying the mutual reinforcement effect of such unification.
We also introduce a novel PixelQA task that jointly requires object-centric referring, segmen- tation, and QA in videos, where UniPixel establishes a strong baseline for this setting.
2 Related Work
The remarkable success of large multi-modal models (LMMs) has shifted the paradigm of visual-language understanding from close-ended experts open-ended task solvers. Early attempts [43] 42
Large Multi-modal Models
[7 [15) involve an MLP projector or Q-Former [34] to align visual encoders LLMs, enabling open-ended tasks such as visual question answering. With advanced designs such as dynamic resolution and data augmentation, open-source models, e.g., Qwen-V series, have narrowed the gap with advanced proprietary models like the GPT 58 (59 and Gemini families [8]. Recent studies [60] 25 51
77 and Intern VL 14
74
I3
37, 48) also explore
understanding. However, these methods are spatially coarse- grained. UniPixel can also be regarded as object-centric test-time scaling approach, where key objects are first segmented then encoded to facilitate the subsequent reasoning process.
test-time scaling on visual-languas
Visual Referring and Segmentation To meet the growing demand for fine-grained visual under- standing (50 46 47 45 (84, recent efforts have focused on enhancing LMMs with object referring LISA [32) is a representative model that enables as its decoder. They also introduced novel reasoning segmentation task, requiring models to perform segmentation based on implicit queries.
and segmentation capabilities, as compared in Fig. 2
LMM-based segmentation by integrating SAM 3
65 27 have explored advanced mask decoders, more flexible tasks, and larger-scale datasets. Recent studies have also extended these capabilities to videos 4 96 [01). Additionally, some research has examined regional understanding through boxes [12] and masks [02 103) While recent approaches attempt to unify these two capabilities, they either support only images (65) or rely on sub-optimal, tool-based pipelines [2]] To the best of knowledge,
Other works in this direction [104 69
100
UniPixel is the first end-to-end method unifying object referring and mask prediction.
3

---

Visual Encoder
Mask
Decader
Update Menory
0000000
Point Encoder
LLM
Takenn
Box
<IEG>
Mask Encodee
Vauel Token
Prompt Encoder
Memory Injection
Object Memory Bank
behavior [3] differ from [2] and [3]P
[5]] appears disinterested focuses on nibbling
Takenizer
on the ground, while [2] is engaging with [4],
who is offering some food to [2] and [3]]
Figure 3: The architecture of UniPixel. Given a video, a question, and visual prompts, the model
encodes them into tokens via the visual encoder, prompt encoder, and tokenizer, respectively, then predicts spatial-temoral mask for each visual prompt via the mask decoder. The masks are updated into the object memory bank, and subsequently injected into the prompt for pixel-level reasoning.
Method
Problem Formulation We provide unified definition for pixel-level reasoning tasks. Formally, the inputs are an image or video X' text prompt T, and N optional visual prompts (P:) where each P. could point, box, or mask on a specific frame. The outputs are textual responses to the prompt with K grounded (degenerating to normal visual understanding task) and K is not necessarily equal to N, as the model may segment extra objects/regions that are not specified by the visual prompts.
masks {M,H,K K Here, both N and K could be zero
Overview Fig. 3 presents an overview of UniPixel. It is built upon the Qwen2.5-VL framework, consisting of an LLM backbone and ViT-based visual encoder that supports dynamic resolution inputs. Given video and text prompt, the model first tokenizes them via the visual encoder and text tokenizer, then sends them into the LLM for response generation. To boost this framework from holistic-level pixel-level, we introduce (1) a prompt encoder (Sec 3.1 supporting three types of visual prompts, (2) an object memory bank (Sec 3.2 for storing object information and injecting it into the response generation process, and (3) a mask decoder (33) for generating spatial-temporal masks. We also extend the LLM's vocabulary by adding <REF>, <MEM>, and <SEG> tokens. The former two serve as placeholders the input prompt that would replaced by visual prompt and memory tokens, respectively, while the <SEG> token is utilized to trigger and guide the mask decoding process. Detailed designs and interactions among these components are illustrated as follows.
3.1 Prompt Encoder
This module aims effectively encode each visual prompt into single token that can be processed by the LLM. We denote a point prompt as tuple (x., y, i) containing its spa- tial coordinates (z: y) and the corresponding frame index t. For box prompts, it is extended to (I1, V/, x2, //2 t) containing the positions of top-left and bottom-right corners. A mask prompt is densely represented by 2D binary mask m,j E {0, 1} with the same shape as the encoded target frame.
T (0.3)
Linear
GELU + Linear
20
X.Y, (0.5, 0.2)
Linear
Remperst
Pasiional Dnd
Type
20
Reamor Bret
X-Ys (a.s,
① Channdisehe
Figure 4: Joint positional temporal encoding for
point (X, T) and box (X, Y,x, Y2T) prompts.
For sparse prompts (points and boxes), as shown in Fig. we encode each position (æx, y:) the sum of 2D Fourier embedding (73) and learnable type embedding (indicating whether it a single point, tep-left corner, or bottom-right corner.. For box prompts, merge the two positional embeddings

---

by concatenating them along the channel dimension and linearly projecting them back to the original size. Frame indices are also encoded similarly with ID Fourier embeddings. The resulting positional and temporal embeddings are concatenated again, and then projected to the LLM's embedding space via a CELU -> Linear block, such that the sparse coordinates in a point/box are encoded into a compact high-dimensional token. This design is inspired by [31] [66] with two key differences: (1) the spatial-only embeddings are extended to include temporal information, and (2) the negative points are discarded. For dense prompts (masks), we directly resize the binary masks and apply masked pooling on the outputs the visual encoder. An M->L projector (Linear + GELU + Linear) is leveraged to project the pooled visual features to the LLM's embedding space.
3.2 Object Memory Bank
Although sparse prompts contain rich positional and temporal information indicating the objects that users are referring to, it still hard for the model to focus on these important regions. Previous studies [02 [03] also confirm that direct region cropping can generally provide better object awareness compared to positional pointers. To seamlessly integrate such a mechanism while preserving the flexibility of visual prompts (e.g., allow pointing on single frame instead of drawing complete masks on all frames), we propose an object memory bank to bridge sparse visual prompts and dense object masks. This is a hashmap where the keys are object IDs and the values are the corresponding spatial-tenn spatial-temporal masks. It is initialized as an empty storage for every chat session, and is dynamically updated on demand. We define two operations for the object memory bank, namely memory pre-filling and memory injection. Below is an example of memory-enhancee multi-round conversation.
Prompt 1: How does behavior
differ from [2] <REF> and [3] <REF>?
detected, enhuncing the prompi with ohject memory.
Memory Pre-filligg Response:
The relevant regions for this question are [I] <SBC> [2] <SRC> [3] <SRC> [4] <SEC>, 4 objects saved buto the memory
Memory Injected Prompt:
Here a video with 4 frames denoted us <>> to <4>, The highlighted regions ure as follows:
<I> <2> <NEW> <3> <VEN>
This object cannot be seen in the last frame
<l> <1> <TN <>> <MEM> <3>
<EN
</> <ME>>
[3]: <l>
<2>
<3>
<db <ME>>
[4]: <1> <EE <>> <HE>> <3> CEE <4> <ME>>>
How does the behavior [1] differ from [2] and [3]?
Response 1: [1] appears disinterested and focuses nibbling the ground, while [2] engaging with [4), who is offering some food to [2] and [3].
Prompt 2: What food [4] offering? Users can directly refer objects memory
Response 2: [4] is offering carrots.
Memory Pre-filling This operation is triggered upon the detection of <REF> tokens in the input prompt, aiming to thoroughly analyze the referred objects and predict their corresponding masks. In this stage, the model responds with object IDs and <SEG> tokens for the relevant objects according to the context, and predicts their spatial-temporal masks accordingly. These object-mask pairs are then saved into the object memory bank.
Memory Injection We inject the features of the saved objects into the prompt to enhance object- awareness. Similar to the mask prompt encoder described in Sec. 3.1 each frame-level object mask downsampled to match the resolution of visual tokens. We then apply masked pooling to aggregate object-centric features. Each frame-level mask is condensed into single feature token, projected through the mask projector, and subsequently utilized to replace the corresponding <MEM> token in the memory-injected prompt. Through this pre-filling and injection mechanism, object-centric information is effectively integrated into the model inference process.
Why using object memory bank? An alternative is directly appending <SEG> token to each <REF> token, followed by masked pooled features obtained during inference. However, we do not adopt this approach for two reasons: (1) During mask prediction, the <SEG> tokens, due to the unidirectional nature causal self-attention, are unable to aggregate the full context of the prompt, thereby compromising the quality of predicted masks. (2) By utilizing the object memory bank, can effectively decouple regional understanding and mask prediction, allowing each to benefit from referring and segmentation data during training, thus enhancing both capabilities.
5

---

Table Comparison with state-of thoofth-eaat methods on ReVOS [96 split. The best and second- best results are marked bold and underlined, respectively.
Referring
Reasoning
Overall
Method
Size
R
J
F
JKF
F J&.F
JT
F
J&FF
Now-LLM-bossed Specialists
MITR
29.8
30.2
30.0
20.4
21.5
21.0
25.1
25.9
LMPM
21
29.0
39.1
34.1
13.3
24.3
18.8
21.2
31.7
26.4
3.2
ReferFormer
31.2
34.3
32.7
21.3
25.6
23.4
26.2
29.9
28.
8.8
L.M-hased Generalists
LISA 3
13B
45.2
47.9
46.6
34,3
39.1
36,7
39.8
43.5
41.6
8.6
TrackGT
F2
13B
48.3
50.6
49..5
38.1
42.9
40.55
43.2
46.8
45.0
12.8
VISA
96
13B
55.6
59.1
57.4
42.0
46.7
44.3
48.8
52.9
50.9
14.5
HyperSeg
81
3B
56.0
60.9
58.5
50.2
55.8
53.0
53.1
58.4
55.7
SostructSog
3B
54.8
59.2
57.0
49.2
54.7
51,9
52.0
54.5
GLUS 30
7B
56.0
60.7
58.3
48.8
53.9
51.4
52.4
57.3
$4,9
17.9
54.9
59.1
Sa2VA [0]
4B
53.2
UniPixel (Ours)
3B
62.3
66.7
64.5
57.1
62.1
59.6
59.7
64.4
62.1
19.0
UniPixel (Ours)
7B
63.9
67.8
65.8
59.4
63.7
61.5
61.7
65.7
63.7
19.4
3.3 Mask Decoder
We adopt SAM 2.1 [66] as the mask decoder to disentangle the discrete language modeling and continuous mask prediction capabilities. For each <SEG> token, we extract its last-layer hidden states, downsample them via an L-+M projector (architecturally identical to the M-+ projector), and reshape them into two tokens. Using two tokens ensures better preservation of object information when downsampling from high- to oow-dimensional channel space. These tokens prompt the mask decoder to predict the mask the first frame, which is then propagated to the other frames.
3.4 Model Training
The training loss for UniPixel a linear combination of language modeling loss and mask decoding losses [66], including a focal loss and dice loss for mask prediction, mean-absolute--mme (MAE) loss for loU prediction, and a cross-entrops loss for objectness prediction. The loss weights are set to 1, 100, 5, 5, and 5, respectively. We train the model through three-stage progressive alignment recipe. The datasets are listed in Tab. 12 In the first stage, we pre-train the sparse prompt encoder using 851k K regional captioning data. Then, align the L.LM and mask decoder by training the I--M projector on 87K referring segmentation data. In the last stage, we further unfreeze the M- projector and mask decoder, and apply LoRA (26) the visual encoder and LLM. The model is jointly trained on a large-scale corpus with around IM samples for diverse tasks.
Experiments
We evaluate the effectiveness UniPixel by conducting extensive experiments across diverse of benchmarks. Specifically, we study the following research questions.
Q1. Whether UniPixel is flexible and effective basic image/video referring and segmentation tasks compared to the corresponding representative methods?
Q2. How does perform on the challenging PixeIQA task, which requires joint referring, segmentation, and question answering in videos?
Q3. What effects does each architectural design contribute? More importantly, does the unified modeling of referring and segmentation lead to a mutual reinforcement effect?
Detailed information about the benchmarks, evaluation metrics, implementation details, and more experimental results can be found in the appendix.
4.1 Q1: Comparison with State-of-the-Arts on Referring and Segmentation Tasks
Reasoning Video Object Segmentation We begin with the most challenging ReVOS (96] dataset, which requires models to predict masks based on implicit text queries demanding complex reasoning abilities based on world knowledge. The results are shown in Tab. [ Our 3B variant outperforms all

---

Table Comparison with state-of-the methods on referring video object segmentation (RVOS) motion-grounded (val), Ref-DAVIS17 [62] (val), and GroundMoRe (00 (test). The best and second-best results are marked bold and underlined, respectively.
and motion grounded video reasoning datasets, including MeViS [2] (val), Ref- YouTube-VOSS [7]]
MeViS
Ref-YoTTube-VOS
Ref-DAVISI7
GroundMoRe
Method
Size
F
F&F
J
F&FF
F
I&F
F
F
Non-LLM-bosed Specialists
ReferFormer S
29.8
32.2
31.0
61.3
64.6
62.9
64.1
61.1
14.3
LMPM 21
34.2
40.2
37.2
12.7
14.0
13.3
OnlineRefer
61.6
65.5
63.5
61.6
67.7
64.8
JJM based Gemeralizs
PixeIL
69
7B
36.3
41.1
38.7
54.3
55.7
55.0
63.4
70.0
66.7
9.9
10.0
10.0
LISA
138
35.8
40.0
37.9
54.0
54.4
63.2
68.8
66.0
6.7
VISA
9M
13B
41.8
47.1
44.5
61.4
64.7
63.0
67.0
73.8
70.4
5.3
4.7
5.9
VSdcoLISA
3.88
41.3
47.6
44.4
61.7
65.7
63.7
64.9
72.7
68.8
VedcoGLaMMM
48.2
65.6
ViL.a
12
6B
46.5
52.3
49.4
64.6
70.4
67.5
70.6
78.0
74.3
67.3
Sa2VA 10T
4B
46.2
70.0
73.8
MoRA
20
27.4
26.9
21.2
UniPixel (Ours)
3B
50.4
55.7
53.1
68.6
72.3
70.5
70.7
77.8
74.2
36.0
38.7
37.4
UniPixel (Ours)
58.3
Table 3: Comparison with state-of the-art methods on image referring expression segmentation (RES) and reasoning segmentation datasets, including RefCOCO/+/g [2]|55] and ReasonSeg [2] (val). The best and second-best results are marked bold and underlined, respectively.
ReCCOCO
RefCOCO+
ReCCOCOg
ReasonSeg
Method
Size
testA
testB
testA
test8
val(U)
test(U)
gloU
cloU
Non-LLMMbussed Specialists
ReLA 41
73.8
76.5
70.2
66.0
71.0
51.7
65.0
66.0
X-Decoder
116
69.6
22.6
17.9
SEEM L17
65.7
25.5
21.2
I.M--axeed Image Generalists
NExT-Char [04
7B
74.7
78.9
69.5
65.
71.9
56.7
67.0
67.0
PixeILM 6
73.0
76.5
68.2
66.3
71.7
58.3
69.3
70.5
LISA 32
7B
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
61.3
62.9
Grounhoog
110
7B
78.5
79.9
75.7
70.5
75.0
64.9
74.1
74.6
56.2
LaSagnA
M 27
13B
74.6
77.6
71.0
64.0
68.1
57.6
69.0
69.3
LLM--based Video Generallisty
VideoLISA
61.4
VISA [96)
7B
72.4
75.5
68.1
59.8
64.8
53.1
65.5
66.4
52.7
$7.8
79.5
72.5
Sa2VA 101
4B
78.9
71.7
74.1
UniPixel (Ours)
78.9
64.0
UniPisel (Ouns)
7B
80.8
83.0
77.4
75.3
80.1
70.0
76.4
77.1
60.5
58.7
existing methods with larger LLMs (including Sa2VA-4B [00 also with SAM 2 decoder), achieving 62.1 overall J&F. The 7B model further boosts the performance to 64.0 J &&F an improvement of 12% over the previous state-of site--ofthe-arttt demonstrating that UniPixel can effectively understand implicit queries based on its world knowledge, and accurately generate masks as responses.
Referring Video Object Segmentation The performance comparisons on MeViS 21 Ref- YouTube- VOS (71), and Ref-DAVISI7 [62] datasets are presented in Tab. 2 UniPixel consistently achieves the best performance among its counterparts. Its advantage is particularly evident the more challenging MeViS dataset, where our model outperforms GLUS-7B as well the similarly sized VideoGLaMM-3..8 (56) by 17%. More experimental results on MeViS 21 val" set and Ref-SAV 101 val set are provided in Tab. 4 and Tab. 5] respectively. Ref-SAV features long referring descriptions, large object motion, large camera motion, and heavy occlusion compared with existing datasets. Given these complex descriptions and video content, our method consistently performs better than counterparts, including those fine-tuned on the target dataset.
39 by around 3.5%,
Motion-Grounded Video Reasoning We also evaluate our method on GroundMoRe (0) dataset (results shown in Tab. 2 which highlights visual answer generation that requires joint spatial and

---

Table 4: Experimental results on MeViS [2] val" Table Comparison on Ref-SAV [0o val set.
set. Post means applying post optimization.
FT means fine-tuning after pre-/co -oaooottrainggg...
Method
Size
FT
J
F
J&F
Method
Size
FT
J
F
J&FF
LMPM
21
43.9
UniRef++ 86
11.6
10.5
UNINEXT
95
8.8
6.4
7.6
LISA
39,9
46.5
43.2
12.2
LISA
XMem 15
7B
41.9
49.3
45.6
VISA [X]
7B
13.2
11.3
11.8
VideoLISA
7B
48.4
54,9
51.7
Sa2VA 101
8B
39.6
43.0
41.3
VidcoLISA
Post
58.1
Sa2VA
101
4B
52.1
UniRef++ 66
15.8
13.4
14.6
48.3
UniPixel (Ours)
3B
56. I
63.2
59.7
UniPixel (Ours)
3B
66.9
67.6
67.2
UniPixel (Ours)
65.0
UniPixel (Ours)
Table 6: Fine-tuned performance on referring expression segmentation (RES) datasets, including Ref- The best and second- best results are marked bold and underlined, respectively.
COCO/+/g 229 (5).
RefCOCO
RefCOCO+
RefCOCOg
Method
Size
val
testA
tes:
val
testA
testB
val(U))
test(U)
LISA
32
7B
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
GSVA
7R
77.2
73.5
59.8
72.7
73.3
OMG-ELAY VA T0%
7B
78.0
80.3
74.1
69.1
73.1
63.0
72.9
72.9
GLaMM
KS
7B
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
Sa2VA
4B
80.4
74,3
75.7
UniPixel (Ours)
81.9
78.6
70.6
77.2
78.5
UniPixel (Ours)
7B
83.0
84.9
80.4
77.8
82.3
72.7
78.7
79.7
Table Experimental results on referring expression comprchension (REC) datasets, including Ref- COCO/4/g 229 55). The best and second-bee -best results are marked bold and underlined, respectively.
ReCCOCO
ReCCOCO+
ReCOOCO
Method
Size
testA
lestB
lestA
lestB
val(U)
test(U)
OFA
76.4
61.8
Shikra
7B
87.0
90.6
80.2
81.6
87.4
72.1
82.3
82.2
MiniGiPT-v2 g
88.7
91.6
85.3
79.9
85.1
74.4
84.4
84.6
Vitron
23
90.9
93.2
89.3
83.7
89.1
76.9
86.4
87.0
UniPixel
87.5
90.8
80.3
UniPixel (Ours)
7B
92.0
94.4
88.1
87.2
91.9
82.1
88.6
88.7
temporal grounding. Note that mainly compare the results with MoRA 20). which fine-tuned on GroundMoRe while other methods are evaluated under the zero-shot setting. Benefit from the strong pixel-level reasoning capability, UniPixel significantly performs better than the baseline.
Referring Expression Segmentation and Reasoning Segmentation Tab. compares the image segmentation capabilities using explicit and implicit queries. We evaluate co-trained model on RefCOCO/+/g 22 (55) and ReasonSeg (32] While state-of-the-art performance has been achieved on RES datasets, observe that the reasoning segmentation data (239 samples) can be easily overwhelmed by the other samples during training due to its limited size. Tab. prresentss the RES performance after fine-tuning, We follow the common practice that jointly fine-tunes the model on RefCOCO/+/g datasets 229 (55 and then evaluate on them separately. These results demonstrate the generalizability of UniPixel when facing both explicit and implicit queries.
Referring Expression Comprehension Our method also supports referring expression compre- hension by inferring the bounding boxes from predicted masks. Its performance (accuracy with IoU > 0.5) is compared with representative methods in Tab. [ Benefiting from the high-quality mask prediction, UniPixel can also achieve very competitive performance on this simpler task.
Referred Video Description and Question Answering We study UniPixel's 's regional understand- ing capabilities on VideoRefer- [103] which contains two subsets for description and question answering tasks. The comparisons are in Tab. X and Tab. 9 BQ, SQ, RQ, CQ, and FP denote basic questions, sequential questions, relational questions, reasoning questions, and future predictions, respectively. Both tasks leverage mask prompts as inputs, where single-frant and multi-frame modes denote applying the masks only specific frame and on all frames, respectively. UniPixel can
8

---

Table 8: Comparison with state-of -fe----tthhttte the-art art methods on VideoRefer- P 103 becond-best results marked bold and underlined, respectively.
The best and
Single-Frame
Multi-Frame
Method
Size
Avg.
Avg.
General LMMs
LLaVA-OV
883
7B
2.62
1.58
2.19
2.07
2.12
3,09
1.94
2.50
2.41
2.48
QwenZ-VL
2.39
3.30
2.54
In2erVLL22
74
26B
3.55
2.99
2.57
2.25
2.84
4.08
3.35
3.08
2.28
3.20
mi-m-i
59
3.56
2.92
3.18
2.62
GPT-4o
50
3,34
2.96
3.01
2.50
2.95
4.15
3.31
3.11
2.43
3.25
Imagee Referring LMMs
2.19
2.38
1.97
Osprey
102
7B
3.19
2. 16
1.54
2.45
2.34
3.30
2.66
2.10
1,58
2.41
Video Referring 7MMss
Elysium 75
7B
2.35
0.30
0.02
3.59
1.57
Artemis 5
7B
3.42
1.34
1.39
2.90
2.26
VideoReler
103
3.42
3.27
Unillixel (Ours)
3B
4.04
3.15
3.10
3.37
3.42
4.08
3.13
3.13
3.42
3.44
UniPixel (Ours)
7B
3.83
3.07
2.96
3.62
3.37
3.82
3.05
3.01
3.57
3.36
Table 9: Comparison with state-of-the-art methods on VideoRefer- 8 [033] (mask prompts). MF denotes multi-fram mode. Full question types are in Sec, 4.1
Table 10: Evaluation results on our newly introduced PixelQA task. All the visual prompts are applied in a single frame. See Sec. 4.2 for detailed settings.
Method
Size
MF
BQ
SQ
RQ
CQ
FP
Avg.
General LMMSs
Method
Size
F
J&F Acc
LLaVA-OV
7B
x
58.7
62.9
64.7
87.4
76.3
67.4
Point Prompts
Qwea2-VL
X
62.0
69.6
54,9
87.3
74.6
66.0
InterVL22 74
2615
60.8
InterVVL2
74
26B
X
58.5
63.5
53.4
88.0
78.9
65.0
GPT-to-mini
59
x
57.6
67.1
56.5
85.9
75.4
65.8
Qwen2.
UniPixel (Ours)
72B
3B
57.3
64.4
60.9
71.1
GPT-4o 59
UniPixel (Ours)
7B
42.1
47.1
44.6
Image Referring LMMs
Bax Prompty
Ferret [X
x
35.2
44.7
41.9
70.4
74.6
48.8
Osprey 102
45.9
47.1
30.0
48.6
23.7
39.9
InterVL2 F4
26B
61.3
Qwen2 77
72B
69.0
Reforring
UniPixel (Ours)
3B
57.8
64.7
61.3
70.3
VideoRefer 103
68.6
78.
71.9
UniPixel (Ours)
7B
41.1
46.4
43.8
71.4
UniPixel (Ours)
3B
X
23.6
70,3
60.7
88.8
78.0
72.2
Mixed (30% Points
50% Botes)
UniPivel (Ours)
X
68.9
73.1
64.7
83.8
83.3
73.4
IntemVL2
26B
VideoRefer 103
7B
/
70.6
60.5
72.1
Qwen2-Vi VL 77
72B
69.1
UniPixel (Ours)
70.7
87.4
72.8
UniPixel (Ours)
UniPixel (Ours)
7B
/
20.6
74.6
64.7
88.8
82.5
74.1
UniPivel (Ours)
7B
42.3
47.5
44.9
71.4
effectively comprehend both types of prompts, and accurately respond with object- descriptions or answers, surpassing strong models including GPT-4o [59] and VideoRefer [103].
4.2 Q2: Pixel-Level Video Question Answering (PixeIQA)
We design the new PixeIQA task based VideoRefer- prompts are replaced with more challenging point or box prompts. Given these ambiguous visual cues, models are expected to correctly identify the target object according the question and the visual prompt, then respond with both the textual answer and the corresponding object masks. We report the mask prediction J&F and MCQ accuracy in Tab. TO Note that none of the existing methods supports this scenario. Thus, we apply set-of-mark prompts (97) directly on video frames, and evaluate the QA accuracies of two strong LMMs [77 74 as our baselines. Aside from point- or box-only prompts, we also explore a more flexible setting that randomly chooses different prompts for different objects. The results verify that our memory pre-filling & injection paradigm effectively enhances the model's reasoning capabilities. Visualizations of this task are shown in Fig.
103 where the original mask
4.3
Q3: Key Ablation Studies
Effect of Task Unification We study the effect of task unification in Tab. II (a). Unifying referring and segmentation capabilities into single model and training them jointly leads to better results

---

if [A] continues moving forward, what likely fiture erent?
The bear will endpunter other animals (0) The bear will find piace (C) The bear will start ranning (P) The óear will climb the stone wall
what likely future with
(A) will stop rest (8) will start walking slowly (C) rie continue to navigets through mores obstacies (D) down
Hew [4] roleted to [A]]
kolding condrolling [R] witk leask (C) [4] walking from
[4] ignaring
Figure 5: Visualization of the outputs from UniPixel on PixeIQA task. Star marks and boxes refer point and box prompts, respectively. The boxed frames denote where the visual prompts applied. Given different types of visual prompts on single frame, our method can flexibly infer the relevant object, track it across the entire video, and involve its features in reasoning.
Table 11: Key ablation studies with UniPixel-3B on PixelQA (mixed). See Sec. 4.3
for explanations.
(a) Task Unification
(b) Object Memory Bank
(c) Prompt Encoder & Mask Decoder
Refer Segment
Memory
J&F
Acc
Referring Method
ITF Acc
Encoder
Decoder
J&FF
Acc
64.6
① <REF>
4688
64,5
wio Time
44.3
63.7
47.5
<REF><SBG>
47.8
64.9
w/ Time
49.0
48.2
67.4
<REP><SDD> Pooling
47.5
66.3
Independent
46.1
66.2
Object Memory
Propagation
on both tasks (first three rows), demonstrating the mutual reinforcement effect of such unification. Incorporating memory pre-filling as an auxiliary task (last row) brings extra improvements.
Effeet of Object Memory Bank Tab. I1 (b) verifies the effectiveness of object memory bank. @ means using a single token for each referred object. means adding an extra segmentation token to segment it auxiliary task. @ further appends massked-pooled visual tokens after it. The results show that (1) both adding auxiliary segmentation task and masked-pooled features help regional understanding, and (2) decoupling them via object memory bank can further boost the performance.
Design Space of Prompt Encoder Mask Decoder We compare different prompt encoder and mask decoder designs in Tab. (c). The performance significantly drops when the temporal encoding in the prompt encoder is removed (first two rows). For the mask decoder (last two rows), explore alternative strategy that treats video frames independently (as batched images), which could largely accelerate inference but lead to sub-optimal accuracies. We hypothesize that this because the LLM LLM-eeeerated <SEG> token cannot well-capture the object information in all frames, thus disentangling the segmentation and tracking capabilities to an external module reasonable.
Conclusion
In this work, proposed UniPixel, large multi-modal model that supports flexible pixel-level visual reasoning. It unifies the internal representations of referred and segmented objects through novel object memory bank. We observe that by such unification, the performance of object referring and segmentation can be jointly enhanced. Extensive experiments on diverse pixel-level understanding tasks, including the PixelQA task, demonstrate the significance of the proposed method. We hope this work inspires future advancements in pixel-level visual understanding.
Acknowledgements
This study was supported by The Hong Kong RGC Grant (15229423) and a financial support from ARC Lab, Tencent PCG (ZGG9). We also acknowledge The University Research Facility in Big Data Analyties (UBDA) at The Hong Kong Polytechnie University for providing computing resources that have contributed to the research results reported within this paper.
10

---

References
[1] Ali Athar, Xueqing Deng, and Liang-Chiee Chen. Vicas: dataset for combining holistic and pixel-level video understanding using captions with grounded segmentation. arXiv:2412.09754, 2024.
[2] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
[3] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv:2502. 2025.
[4] Zechen Bai, Tong He, Haiyang Mei, Pichao Wang. Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos. In NeurIPS, pages 6833-6859, 2024.
[5] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In CVPR, pages 4985-4995, 2022.
[6] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding, In CVPR, pages 961-970, 2015.
[7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pages 1209-1218, 2018.
[8] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, 7echun Liu, Pengchuan Zhang, Raghura- man Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt- v2: large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09477 2023.
[9] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Iim's referential dialogue magic. arXiv:2306.15 15/55, 2023.
[10] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pages 1971-1978, 2014.
[11] Yuan Chen, Zi-han Ding, Ziqin Wang, Yan Wang, Lijun Zhang, and Si Liu. Asynchronous large language model enhanced planner for autonomous driving. In ECCV, pages 22 22-38, 2024.
[12] Zewen Chen, Juan Wang, Wen Wang. Sunhan Xu, Hang Xiong, Yun Zeng, Jian Guo. Shuxun Wang, Chunfeng Yuan, Bing Li, othersShilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on eagion-offinteeree ar X3v1:23077 2023.
[13] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Yc, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling. arXiv:2412.05277, 2024.
[14] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic -sngu--inguistic tasks. 11X1:22312.14238, 2023.
[15] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with aukinson-shiffrin memory model. In ECCV, pages 640-658, 2022.
[16] Can Cui, Yunsheng Ma, Xu Cao, Wengian Ye, Yang Zhou, Kaizhao Liang. Jintai Chen, Juanwu Lu, Zichong Yang, Kuci-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In WACV, pages 958-979, 2024.
[17] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose langunelanguage models with instruction tuning. In NeurIPS, 2024.
11

---

[18] Google DeepMind. Introducing gemini 2.0: our new ai model for the agentic era, 2024.
[19] Google DeepMind. Gemini 2.5: Our most intelligent ai model, 2025.
[20] Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, and Chen Chen. Motion-groundee video reasoning: Under- standing and perceiving motion pixel level. arXiv:24/ 2024.
[21] Henghui Ding, Chang L Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In ICCV, pages 2694-2703, 2023.
[22] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. arXiv:2303.03377, 2023.
[23] Hao Fei, Shengqiong Wu, Hanwang Zhang. Tat-Seng Chua, and Shuicheng Yan. Vit- ron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. arXiv:2412.19806, 2024.
[24] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv:2405.21075 2024.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv:2501. 12948, 2025.
[26] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
[27] Donggon Jang, Yucheol Cho, Suin Lee, Tachyeon Kim, and Dae-Shik Kim. Mmr: A large-scale benchmark dataset for multi-target and multi-granularity reasoning segmentation. arXiv:2503. 13881, 2025.
[28] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning visual question answering. In CVPR, pages 2758 2766, 2017.
[29] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787-798, 2014.
[30] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv:2406.09246, 2024.
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. In ICCV, pages 4015 2023.
[32] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In CVPR, pages 9579-9589, 2024.
[33] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. TrXiv:2408.03320 2024.
[34] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, pages 19730-19742, 2023.
[35] Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06355, 2023.
12

---

[36] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, al. Mvbench: A comprehensive multi-moda video understanding benchmark. In CVPR, pages 22195-222000 2024.
[37] Zongzhao Li, Zongyang Ma, Mingze Li, Songyou Li, Yu Rong, Tingyang Xu, Ziqi Zhang,
Deli Zhao, and Wenbing Huang. Star-rl: Spacial transformation reasoning by reinforcing multimodal llms. arXiv:2505. 15804, 2025.
[38] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv:2311.10122, 2023.
[39] Lang Lin, Xueyang Yu, Ziqi Pang. and Yu-Xiong Wang. Glus: Global-local reasoning unified into single large language model for video segmentation. arXiv:2504.07662, 2025.
[40] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[41] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. CVPR, pages 23592-23601, 2023.
[42] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296-26306, 2024.
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, pages 14892-34916, 2023.
[44] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In ECCV, pages 1-18, 2024.
[45] Ye Liu, Jixuan He, Wanhua Li, Junsik Kim, Donglai Wei, Hanspeter Pfister, and Chang Wen Chen. r²-tuning: Efficient toage-to-video transfer learning for video temporal grounding. In
[46] Ye Liu, Huifang Li, Chao Hu, Shuang Luo, Yan Luo, and Chang Wen Chen. Learning to aggregate multi-scale context for instance segmentation in remote sensing images. IEEE Transactions on Neural Networks and Learning Systems, 36(1):595-60) 2024.
[47] Ye Liu, Siyuan Li, Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In CVPR, pages 3042-3051, 2022.
[48] Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Videomind: A chain-of-lora agent for long video reasoning. arXiv:2503. 13444444...344, 2025.
[49] Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang W Chen. E.t. bench: Towards open-ended event-level video-language understanding. In NeurlPS, pages 32076- 32 2024.
[50] Ye Liu, Junsong Yuan, and Chang Wen Chen. Consnet: Learning consistency graph for zero-shot human-object interaction detection. In ACM MM, pages 4235-4243, 2020.
[51] Zongyang Ma, Yuxin Chen, Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Shaojie Zhu, Chengxi- ang Zhuo, Bing Li, Ye Liu, Zang Li, Ying Shan, and Weiming Hu. Visionmath: Vision-form mathematical problem-solving. In ICCV, 2025.
[52] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv:2406.094/ 09418, 2024.
[53] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXh::2306.05444, 2023.
[54] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egosehema: A diagnostic benchmark for very long-form video language understanding. In NeurlPS, 2024.
13

---

[55] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. CVPR, pages 11-20, 2016.
[56] Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: A large multimodal model for pixel-level visual grounding in videos. arXiv:2411.0449 0.4923,, 2024.
[57] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pages 4990-4999, 2017.
[58] OpenAL. Gpt-4v(ision) system card, 2023.
[59] OpenAL. Gpt-4o system card, 2024.
[60]
OpenAL Openai ol system card, 2024.
[61] Wujian Peng, Lingchen Meng, Yitong Chen, Yiweng Xie, Yang Liu, Tao Gui, Hang Xu, Xipeng Qiu, Zuxuan Wu, and Yu-Gang Jiang. Inst-it: Boosting multimodal instance understanding via explicit visual prompt instruction tuning. arXiv:2412.03565, 2024.
[62] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Borkine-Hornune and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv: 1XXiv:/04.006755 2017.
[63] Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Yc, and Yunjie Tian, Artemis: Towards referential understanding in complex videos. In NeurIPS, 2024.
[64] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In CVPR, pages 7141-7151, 2023.
[65] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming- Yang. and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR, pages 13009-13018, 2024.
[66] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv:2408.00714 2024.
[67] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. D.Xiv:2403.05532 2024.
[68] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, pages 14313-14323, 2024.
[69] Zhongwei Ren, Zhicheng Huang. Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, pages 26374- 26383, 2024.
[70] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hier- archical vision transformer without the bells-and-whistles. In ICML, pages 2023.
[71] Sconguk Seo, Joon-Youre Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with large-scale benchmark. In ECCV, pages 208-223, 2020.
[72] Nicholas Stroh. Trackgpt-a generative pre-trained transformer for cross-domain entity trajec- tory forecasting. arXiv:2402.00000 2024.
14

---

[73] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kein Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurlPS, pages 7537-7547, 2020.
[74] OpenGVLab Team. Internv12: Better than the best expandin performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024.
[75] Han Wang, Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm. In ECCV, pages 166-185, 2024.
[76] Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang. Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards open vecabvocabulary video instance segmentation. In ICCV, pages 4057-4066, 2023.
[77] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: vl: Enhancing vision- lasoun-anguuag model's perception of the world at any resolution. arXiv:2409. 2tt:::299..11111,, /2191, 2024.
[78] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequenee
learning framework. In ICML, pages
2022.
[79] Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning.
htt::2405001333, 2024.
[80] Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-base segmentation assistant for complex queries. arXiv:2404.08506, 2024.
[81] Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyper- seg: Towards universal visual segmentation with large language model. 1tfi::2411.166000 2024.
[82] Cong Wei, Yujie Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, and Yujiu Yang. Instructseg: Unifying instructed visual segmentation with multi-modal large language models. arXiv:2412. 14006, 2024.
[83] Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: A simple online baseline for referring video object segmentation. ICCV, pages 2761-2770, 2023.
[84] Jianlong Wu, Wei Liu, Ye Liu, Meng Liu, Liqiang Nie, Zhouchen Lin, and Chang Wen Chen. survey on video temporal grounding with multimodal large language model. arXiv:2508. 10922, 2025.
[85] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, pages 4974-4984, 2022.
[86] Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Segment every reference object in spatial and temporal spaces. In ICCV, pages 2538-2550, 2023.
[87] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In CVPR, pages 3858-3869, 2024.
[88] Linshan Xie, Xuehong Lin. Xunda Wang, Junjian Wen, Teng Ma, Jiahao Hu, Peng Cao, Alex TI. Leong, and Ed X Wu. Simultaneous eeg-fmri reveals spontaneous neural oscillatory activity in cingulate cortex underlying transient rsfmri network dynamics. In ISMRM, 2024.
[89] Linshan Xie, Xunda Wang, Xuehong Lin, Teng Ma, Junjian Wen, Peng Cao, Alex TL Leong, and Ed Wu. Single-pulse optogenetic perturbation of thalamo-cortical networks reveals functional architecture of rsfmri networks. In ISMRM, 2024.
15

---

[90] Linshan Xie, Xunda Wang, Xuehong Lin. Junjian Wen, Teng Ma, Alex TL Leong, and Ed X Wu. Brain-wide resting-state fmri network dynamics elicited by activation of single thalamic input. Nature Communications, 2025.
[91] Linshan Xie, Xunda Wang, Teng Ma, Pit Shan Chong, Lee Wei Lim, Peng Cao, Pek-Lan Khong, Ed X Wu, and Alex TI. Leong. Are topographically segregated excitatory neurons in visual thalamus functionally diverse? an optogenetic fmri study. In ISMRM, 2022.
[92] Linshan Xie, Xunda Wang, Teng Ma, Hang Zeng, Junjian Wen, Peng Cao, Ed X Wu, and Alex TL. Leong. Short single pulse optogenetic fmri mapping of downstream targets in thalamo-cortical pathways. In ISMRM, 2023.
[93] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yucting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, pages 1645-1653, 2017.
[94] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijic Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv:2404. 16994, 2024.
[95] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In CVPR, pages 15325-15336, 2023.
[96] Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In ECCV, pages 98 115, 2024.
[97] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv:2310.11441, 2023.
[98] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv:2310.07704 2023.
[99] Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile-language models. arXiv:2405.02794, 2024.
[100] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activityner-qa: A dataset for understanding complex web videos via question answering. In AAAL, pages 9127-9134, 2019.
[101] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuai Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv:2501.0m 000:2500..4401, 2025.
[102] Yuqian Yuan, Wentong Li. Jian Liu. Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, pages 28202-28211, 2024.
[103] Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video Ilm. axXiv:2501.0059) 2024.
[104] Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat: Imm for chat, detection and segmentation. 2r1 1.04498, 2023.
[105] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: instruction-tunat audio-visual language model for video understanding. 2xXiv:2306.02858 2023.
[106] Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledge-enabled safety-critical scenario generation for autonomous vehicles. In CVPR. pages 15459-15469, 2024.
16

---

[107] Peirong Zhang, Haowei Xu. Jiaxin Zhang, Guitao Xu, Xuhan Zheng, Zhenhua Yang, Junle Liu, Yuyi Zhang, and Lianwen Jin. Aesthetics cheap, show me the text: An empirical evaluation of state-of-the-arr generative models for ocr. arXiv:2507.
2025.
[108] Peirong Zhang, Jiaxin Zhang, Jiahuan Cao, Hongliang Li, and Lianwen Jin. Smaller but better: Unifying layout generation with smaller large language models. International Journal of Computer Vision, 133:3891-3911 2025.
[109] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Chen Change Loy. and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurIPS, pages 71737-71767, 2024.
[110] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai Groundhog: Grounding large language models to holistic segmentation. In CVPR, pages 14227- 14238, 2024.
[111] Xufeng Zhao, Mengdi Li, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter. Chat with the environment: Interactive multimodal perception using large language models. In IROS, pages 3590-3596, 2023.
[112] Rongkun Zheng, Lu Qi, Chen, Yi Wang. Kun Wang. Yu Qiao, and Hengshuang Zhao. Villa: Video reasoning segmentation with large language model. arXiv:2407. (ttv:2477//500 2024.
[113] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633-641, 2017.
[114] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
[115] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023.
[116] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, al. Generalized decoding for pixel, image, and language. In CVPR, pages 15116-15127, 2023.
[117] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li. Linjie Li, Jianfeng Wang, Lijuan Wang. Jianfeng Gao, and Yong Jac Lee. Segment everything everywhere all once. In NeurIPS, pages 19769-19782, 2023.
17

---

Appendix
In this appendix, we provide more details about the training data, model implementation, and experi- mental settings to complement the main paper. Additional analysis, ablation studies, visualizations, and discussions are also incorporated. Below is the table of contents.
A. Model
1. Implementation Details
Training Recipe
B. Experiments
1. Tasks and Benchmarks
2. Evaluation Metrics
3. More Experimental Results
4. Ablation Studies Qualitative Results
C. Discussions
1. Limitations & Future Work
Potential Societal Impacts
D. Licenses
A Model
A.1
Implementation Details
We instantiate our base models with 3B and versions of Qwen2.5-VL. (3] Both variants employ pre-trained SAM 2.1 (66] with Hiera Base+ (70 backbone as the mask decoder. The M-+L projector is initialized with the weights from the V-L projector of Qwen2.5-VL. The hidden size inside the prompt encoder is 256. To reduce GPU memory and accelerate training, we randomly sample 8 frames per video, with each frame resized to 316 448² pixels (128 ~ 256 tokens per frame). The frame sampling strategies follow the specifications of each benchmark during inference. The mask decoder has a fixed resolution of 768 x 768. For each segmentation sample, up to 5 objects are randomly selected to compute the mask prediction losses. During training. LoRA adapters (26) with rank 128 and al alpha=255 256 are applied to all QKVO layers in the visual encoder and LLM. The input sequences are restricted 4K tokens. We train the model with 8 RTX A6000 Ada (48G) GPUs, with a global batch size of 256 for stages and 2, and 32 for stage 3. In the first two stages, the learning rates set 1e-3. In the last stage, it is set to 5e-6 for the mask decoder and 2e-5 for all the other parameters, respectively. A linear warmup in the first 3% steps followed by cosine decay is adopted all stages. The configurations of datasets are introduced in the following section.
A.2 Training Recipe
The detailed distribution of training datasets for UniPixel is shown in Tab, 12 Within the three-stage training recipe, we first pre-train the sparse prompt encoder using short caption samples from Inst-IT 61 and VideoRefer [103]. For each sample, we randomly select a point inside the ground truth mask (50%) or generate augmented box from it (50%). This stage aims to enable the model with simple visual prompt comprehension and regional captioning capabilities on images and videos. In the second stage, we align the LLM and mask decoder using referring object segmentation datasets
29 555 71. use short caption/query samples for the first two stages to focus on alignment rather than knowledge learning. For the last stage, we collect a large-scale, high-quality corpus called MmiPixe--SFT-110 jointly train the model on diverse pixel-level tasks. The original annotations have been rewritten using task-specific templates to incorporate instructions. All the repurposed datasets and pre-processing pipelines will be publicly available to facilitate future research.
18

---

Table 12: The distribution of training datasets for UniPixel. We use different background colors to denote object referring object segmentation, regional understanding memory pre-fill pre-filling and
general video understanding data, respectively.
Inputs
Outputs
Stage
Dataset
#Samples
#Repeat
Ratio
Text
Image
Video
Point
Box
Mask
Text
Mask
Inst-IT- must-IT-nnuge-- Shori- Caption
Shori-Caplion
[6]
/
351K
41.2%
VideoRefer Short Caption 103
500K
58.8%
RefCOCO 20
17K
20.8%
RefCOCO+
17K
20.8%
RetCOCOg
22K
26.8%
ReiClef 29
18K
22.0%
Ref-YouTube--OS
13K
Osprey-Conversation
14K
0.1%
Ospey-DetaillDeee DesellDDeeiiiiiin
102
29K
2.5%
Osprey-Pos Pos:
Pao---osNNeggg Neg
102
20K
1.7%
VidooRRffr-Dettllceee
10.1%
VideoReeee--2
103
69K
5.8%
QA
VideoRefer-QA-MMe
103
69K
3.5%
Int-II--QAMomoory [0]
158K
8.0%
RefCOCO 29
17K
10
2.9%
ReICOCO+ 29
17K
10
2.9%
RefCOCO9
Reitlef 29
18K
10
3.0%
ReasonSeg
ADE20K [
20K
1.0%
COCOStult ?
118K
6.0%
Mapillary Vistus
18K
0.9%
PACO-IVIS
-
46K
2.3%
PASCAL-Part
4,4K
0.2%
Ref You/Tbbe VOS
71
13K
1.1%
Ref-DAVIS17
10
Ref-SAV [00]
56K
2.8%
MeViS 21
23K
1.9%
LV.VIS 76
11K
0.6%
ViCaS 0
41K
2.1%
ReVOS
29K
2.5%
GrundMMRe 200
5.6K
0.3%
LLaVA--.5-Mix-666K
VideoGPT+ Instruct 52
573K
9.7%
B Experiments
B.1 Tasks and Benchmarks
Our method is extensively evaluated across 9 fine-grainee image/video understanding tasks. The benchmark(s) used for each task are listed as follows:
1. Reasoning Video Object Segmentation: ReVOS
2. Referring Video Object Segmentation: MeVIS 21 Ref- YouTube- VOS Ref-DAVISI7 62 Ref-SAV [00]
3. Motion-Grounded Video Reasoning: GroundMoRe
20
Referring Expression Segmentation: RefCOCo
20 RefCOCO+
29
RefCOCOg
Reasoning Segmentation: ReasonSeg 3
6. Referring Expression Comprchension: RefCOCO
29
ReCCOCO+
29
RefCOCOg
55
7. Referred Video Description: VideoReffer-Benchh
103
Referred Video Question Answering: CecoRefer-Benchh
103
9. Flexible Pixel-Level Understanding: PixelQA (Ours)
B.2 Evaluation Metrics
For video segmentation tasks, we adopt J &FF as the main metric to jointly consider region similarity I and contour accuracy F. Image segmentation evaluated using cloU (the cumulative intersection over the cumulative union) and gloU (the average of all per-image loUs) following existing work. For referred video description and question answering tasks, we follow the official evaluation protocols to report GPT-4o [59) scores and MCQ accuracy, respectively. For referring expression comprehension,
19

---

Table 13: Performance comparison on general video question answering (VideoQA) on MVBench Note that UniPixel is the only model supporting pixel-level referring & segmentation.
36
Model
[Size|
AS
AP
AA
FA
UA
OF
OF
OS
MD
AL
ST
AC
MC
MA
SC
FP
CO
EN
ER
CT
Avg.
GPT-4V
58
[5..5
63.5
72.0
46.5
73.5
18.5
59.0
29.5
12.0
40.5
83,5
39.0
12.0
22.5
45.0
47,5
52.0
31.0
59.0
11.0
43,5
Video-ChatGPT
35
7B
23.5
26.0
62.0
22.5
26.5
54,0
28.0
40.0
23.0
20.0
31.0
30.5
25.5
39.5
48.5
29.0
33
29.5
26.0
35,5
32.7
Vidco-LLaMA
108
27.5
25.5
29.0
39.0
48.0
40.5
3800
22.5
22.5
43.0
34,0
22.5
32
45.5
32.5
40.0
30.0
21.0
37.0
34.1
VidenChat
33
7B
31.5
26.5
56.0
33.5
40.5
53.0
40.5
30.0
25.5
27.0
48.5
35.0
20.5
42.5
46.0
26.5
41.
23.5
23
3600
35.5
18
53.5
48.0
41.0
29.0
82.5
33.5
41.5
27.5
38.5
43.0
TimeChat 68
7B
40.5
36.0
61.0
32.5
53.0
53.5
41.5
29.0
19.5
26.5
66.5
34,0
20.0
43.5
42.0
36,5
36
29.0
35.0
35.0
38.5
PLLaVA [4
7B
58.0
49.0
55.5
61.0
56.0
61.0
360
23.5
26.0
82.0
39.5
42.0
52.0
45.0
42.0
53.5
30.5
48.0
31,0
46.6
ST-LLM [4
7B
66.0
53.5
84.0
44.0
58.5
80.5
73.5
38.5
42.5
31.0
86.5
36.5
56.5
78.5
43.0
44.5
46.5
34,5
41.5
58.5
54.9
VideoGPT+ 52
4B
69.0
60.0
83.0
48.5
66.5
85,5
75.5
36.0
44.0
34.0
89.5
39.5
71.0
90.5
45.0
53.0
50.0
29.5
44,0
60.0
58.7
VideoChat2
36
75.5
58.0
83.5
50.5
60.5
87.5
74.5
45.0
47.5
44.0
82.5
37.0
87.5
51.0
06.5
47.0
35.0
37.0
72.5
60.4
UniPixel (Ours)
3B
69.5
62.5
83.0
48.5
76.5
86.5
66.5
38.0
49.0
40.5
87.0
49.0
74.0
95.0
49.0
45.0
63.5
34.5
58.0
73.5
62.5
UniPixel (Ours)
7B
71.0
68.0
84.0
45.0
78.0
91.5
66.5
35.5
57.5
43.0
91.5
47.0
73.5
92.5
58.0
53.0
74,0
37.5
49,0
69.0 64.3
Table 14: Effectiveness justification of multi-stage training. The best and second-best results are marked bold and underlined, respectively. The three-stage recipe leads to optimal performance.
ReVOS
MeViS (val')
VideoRefer Bench
Stage
Stage
Stage
J&J
J
F
J&FF
Single-Frame Framee
Multi Frame
58.3
63.6
61.0
54,8
61.9
58.4
71.1
71.5
59.0
63.4
55.2
62.1
58.7
71.8
22.3
v
59.6
63.5
61.6
55.7
62.5
59.1
71.2
71.6
39.7
59.7
we leverage mean accuracies, where a predicted bounding box is considered correct when it has the intersection over union (IoU) with the ground truth less than 0.5.
B.3 More Experimental Results
General Video Question Answering We also evaluate UniPixel on MVBench (36 to compare its general video understanding capabilities with existing methods. The results are illustrated in Tab. [3] Note that our method is the only one in the table that supports referring and segmentation. By jointly training on holistic-level and pixel-level data, UniPixel can effectively balance the capabilities under both scenarios, demonstrated by the strong performance compared with holistic-level models.
B.4 Ablation Studies
Effeet of Multi-stage Training We investigate the effectiveness multi-stage training in Tab. 14 As shown in the first line, directly training the model using large-scale data only leads to sub-optimal performance, due to the unaligned representations among prompt encoder, LLM, and mask decoder. We observe that pre-training cither the sparse prompt encoder or the L-*M projector (the second and third lines) brings performance gains on both tasks (referring and segmentation). We hypothesize that this is because aligning either of them can alleviate the burden of joint-task learning in stage 3. The last row verifies that the performance can be further boosted by pre-aligning both of them.
Number of Hidden Tokens for Mask Decoder As mentioned in the main paper, there is huge gap between the feature dimensions of the LLM and the mask decoder, thus splitting the <SEG> token into more hidden tokens can better preserve the object information from the LLM. We ablate this mechanism in Tab. [5] According to the results, using only hidden token cannot fully preserve the object information, the mask prediction performance is sub-optimal. However, also observe that using more than hidden tokens (e.g., 4 or 8) only brings negligible performance gain. Therefore, we choose 2 hidden tokens per object in our final model.
Training Strategy for the M- projector The M- projector aims to project the masked-pooled object-centric features to the LLM's embedding space. Since the object features originate from the visual encoder, possible to re-use the pre-trained weights of the original V->L projector in Qwen2.5-VI Its effects are studied in Tab. 16 We investigated two strategies: 1) re-using the weights and 2) adding an extra pre-training stage for better alignment. The comparison shows that directly re-using weights without extra pre-training achieve the best results.
20

---

Table 13: Ablation study on the number of hid- den tokens for each <SEG>. Performance gains are negligible with more than 2 tokens/object.
Table 16: Ablation study on M- projector, Init and PT denote weight initialization from V-+ projector and extra pre-training, respectively.
ReVOS
MeViS (val")
FideoRefer-Bencee ?
PixelQA
#Tokens
Init
PT
J
F
JkF
J
F
J&F
Single-Frame
Mult-Prame
Mixed Ace
99.6
63.5
61.6
55.8
62.5
59.2
71.4
71.9
67.7
59.7
64.4
62.1
56.1
63.2
59.7
71.5
71.7
67.4
59.8
63.9
61.9
56.8
63.1
89.9
72.6
8
59.5
64.0
61.8
56.4
62.8
59.6
72.2
72.8
68.5
Table 17: Ablation study on training data used stage 3. The best and second-best results are marked
bold and underlined, respectively. Gradually adding more pixel-level data brings performance gains.
Re
MeViS (val")
VideoRefer- Benchh 2
Regional
Segmentation
Memory
General
J&F
J
J&F
Single-Frame
Multi-Framo
72.1
72.0
39.2
63.7
61.5
55.8
63.1
59.5
72.3
72.6
59.6
64.5
62.1
56.3
63.5
59.9
72.4
72.5
59.7
64.4
62.1
56.1
63.2
59.7
72.2
72.8
Combination of Training Data Tab. 17 studies the effect of the combination of multi-task co- training data in stage 3. Compared with training only on the regional or segmentation data, leveraging both of them leads to considerable performance on both tasks. Incorporating memory pre-filling data (requiring both referring and segmentation) can further boost the performance. We also mix some general holistic-level video understanding data to preserve the original capabilities of the pre-trained model, while it slightly affects the performance on pixel-level tasks.
B.5
Qualitative Results
Fig. 6 present more visualizations of outputs from UniPixel on different pixel-level understanding tasks. Our method can effectively handle flexible visual prompts [103) implicit queries [32 96 long queries [00], and motion-grounded questions 2
C Discussion
C.1 Limitations & Future Work
Due to the limited computing resources, we did not further scale up the training data incorporate more pixel-level tasks such as grounded caption generation (GCG) on images [65] or videos [56). which are interesting scenarios and their data may bring more performance gains. Besides, the mask decoder currently predicts the first mask on the first frame and propagates it to the following frames, while it potentially supports predicting on the best frame (defined the frame with the best view of the target) and propagates it to both sides of the video. We will focus in our future work to explore more pixel-level understanding tasks and more flexible mechanisms for the mask decoder.
C.2 Potential Societal Impacts
This work introduces new framework for pixel-level visual-language understanding, which could potentially be used in education, surveillance, and healtheare industries, where flexible interactions with the users and fine-grained understanding of images & videos are required. In other scenarios requiring multi-modal assistants, our method can also serve as a more advanced alternative. To the best of our knowledge, there are no potential negative societal impacts to declare.
D Licenses
Our model built based on the pre-trained Qwen2.5-VI. K and SAM 2.1 66 models. They are both licensed under the Apache License 2.0 lhttps://www.t apache httg://icnnnes/LINNSS-222222
21

---

What action does [4] perforen that involves [s]?
extends
hands sometting
igmores
what weoring?
(A) Blke sweatshirt black jeans (8) Red sweatshire and light Blue jeans (C) Green t-skirt white pants (P) Yellow Aoodie and dark blue jeans
f [4] continues to move forward, what likely future evand invalving [2]P
away (9) and stop moving
will walking wheefchair
<eljert1><negion>> (A) [*] will stop (8) [2] will start running (C) [1] will change different outfit (D) [3] will continus to challengs different high difficulty movements
Figure 6: Visualization of the predictions from UniPixel PixeIQA.
Please segmmet the sow that is the furthest from the camera in this video.
Which goldfish left side the beginning of video? provide segmentation
Where is the instrument that serves to shield from the sun or protect from rain and snow?
you find skunk that black body tuft of white fur head tip of
Which ferret(s) is/are being licked by consistently? Please provide the segmentation mask.
Can you segment the insect(ss belonging to the superfamily Papilionaides of the Lepidoptera order in this video?
segmant younger
Figure 7: Visualization of the predictions from UniPixel on Re ReVOS 166].
22

---

segmantt black video.
Where the man wearing cap and shorts this video? Respond with the segmentation mask
Can you wooden car frames?
Segment and track the motorbike in this video.
where rope? Give segmentation results directly.
Figure 8: Visualization of the predictions from UniPixel on Ref-DAVISI7 62
might not open. feeding fish?
woman.
Q: What might not be given the woman by the man if not by himself?
A: bag.
Q: who opons the ziploc bag to transfer the crushed Oreo cookies into the bow!?
A: The girl.
Q: Who driibies the ball before he shoots it?
A: The man the black shorts.
Q: Who kicks the ball into the goal?
A: The boy.
Q: who asked if the little girl could carry the box before she picked up?
A: The man.
Figure 9: Visualization of the predictions from UniPixel on GroundMoRe 20
23

---

Find the object according to the description: The object is dark-colored backpack with light-colored accents, featuring muitiple compartments and pockets, sacurely fastaned individuals back. person dressed dark clothing and ascending escalator public setting, likely functional for carrying various items. The individual moves steadily up the escalator, indicating a purposeful journey.
transportation hub. backpack has adjustable straps top handle, appearing
Analyse the following sentences and provide corresponding segmentation mask: object dark-colored sedan, likely blue or black, parked on an unpaved surface, passibly dirt road or an area with loose soil. has four doors, a visible rear spoiler on the trunk, silver wheels, and tinted windows. The car is slightly tiited, suggesting it might be parked on uneven ground or experiencing some form of imbalance. Throughout the video, the sedan remains stationary, with indication of movemaend being performed by vehicle.
segment object according
desoription: The object person long wearing top
patterned skirt with geometric designs. This individual is stationary or moving very slowly in the background a retail store, possibly a furniture or home goods store. The person remains in close proximity to another shopper pushing shopping cart, suggesting they might be together interacting. The captures typical shopping experience.
Figure 10: Visualization the predictions from UniPixel on Ref-SAV 101
Whkrre goat nearest to the bottom stone? Give me the segmentation mask.
Please localize place where piano players should sit in this image.
Segmant the place the patient lies down to receive examination in this image.
suitable for photographing nearby objects.
In some rural areas, horse- drram carts are still used for transportation and carrying goods. What source of power that drives the cart in the picture?
Which part of the vehiele be used display identifying information required by law? Segment the target directly.
What item in the picture can provide information guide travelers through rugged terrain that can challenging to navigate?
where place where garbage should put? Please respond with the segmentation mask.
main
Figure 11: Visualization of the predictions from UniPixel on ReasonSeg 32
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To the best of our knowledge, all the claims made in the abstract and introduc- tion are verified either by referring to previous studies or by conducting experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are carefully discussed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
The paper does not include theoretical results. All the claims shall be verified through ablation studies.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly present the list of datasets used for training, the implementation details, and the evaluation metrics used to produce the experimental results in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We open-source all the code, checkpoints, data, and training logs to ensure full reproducibility.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Codeebbbmsss cosssiiinolyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/cccppppgguuuiiaeoeaec
Cttelee//ooSSSSSeSSS.oeaa www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the detailed settings, hyperparameters, and other necessary information are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
We are not able to fully report the error bars given the limited computing resources.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is included in the implementation details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We confirm that the research conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential societal impacts are discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provided citations the original papers for all the used datasets and base models.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "Qn1kXJQakd", "reviewer": "Reviewer_HVUw", "summary": "This paper proposes UniPixel, a unified multi-modal model that jointly handles object referring and segmentation across images and videos. A key innovation is the Object Memory Bank, which enables fine-grained, region-specific reasoning by storing spatial-temporal object features. The model supports diverse visual prompts and introduces a new PixelQA task combining segmentation and question answering. Built on Qwen2-VL with tailored encoders and a SAM-based decoder, UniPixel achieves state-of-the-art results on 10 benchmarks, outperforming larger models.", "questions": "1. In Lines 152–153, for objects [1]–[4], are the <MEM> tokens constructed using a single <SEG> token decoded by SAM2 to obtain masks across all frames? How is the multi-frame memory for each object built from a single <SEG> token? Additionally, how does the model determine that <4><MEM> does not exist in the last frame?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, the work is novel, well-motivated, and advances pixel-level vision-language understanding in a unified and efficient manner.\n\n**Major Weakness:**\n\nI have some concerns regarding the reported performance gains. It appears that UniPixel may largely build upon incremental improvements over Sa2VA, particularly due to two architecture upgrades: the transition from SAM2 to SAM2.1 and from InternVL2.5 to Qwen2-VL. These changes alone could contribute significantly to the performance boost. Furthermore, it is unclear why Qwen2.5-VL, a more recent and potentially stronger backbone, was not adopted. While I acknowledge the importance of engineering innovations, it is essential to disentangle how much of the performance gain comes from the proposed architectural design versus backbone/module upgrades. Unfortunately, the current version of the paper does not provide sufficient ablations or analysis to clarify this.\n\n**Minor Weakness:**\n1. In Table 3, under the RefCOCO benchmark, the comparison with non-LLM-based methods is somewhat limited. I suggest including more recent and competitive baselines such as C3VG [1], which represents a strong non-LLM approach to multi-task visual grounding.\n2. Similarly, in Appendix Table 5, which compares REC-related methods, it would strengthen the empirical evaluation to include recent non-LLM SOTA methods like OneRef [2] and SimVG [3], both published at NeurIPS 2024. These additions would provide a more comprehensive comparison and better contextualize the performance of UniPixel.\n\n[1] Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints, AAAI 2025. \n\n[2] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeurIPS 2024. \n\n[3] SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion, NeurIPS 2024.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed responses. After reviewing the other reviewers’ suggestions, I concur with Reviewer D9pB’s concern regarding the time consumption of the two-stage inference process, which also aligns with my own reservations.\n\nAdditionally, I believe the description of UniPixel’s memory bank mechanism requires further elaboration to enhance clarity.\n\nAt this stage, I maintain my initial assessment as a borderline accept.", "rebuttal": "We sincerely thank you for the detailed and constructive comments. Below we provide point-by-point responses to address your concerns.\n\n### **Q1: Attribution of Performance Gains**\n\nWe highlight the differences between Sa2VA and UniPixel in terms of `architectural design`, `supported input/output formats`, and `frame sampling strategy` below.\n\n| Method | Base Model | Mask Head | Temporal Prompt Encoding | Object Memory Bank | Point$_{in}$ | Box$_{in}$ | Mask$_{in}$ | Mask$_{out}$ | Frame Sampling Strategy |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| Sa2VA | InternVL2.5 | SAM2 (Large) | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | First 5 Frames |\n| UniPixel | Qwen2-VL | SAM2.1 (Base+) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Uniform or Fixed FPS |\n\n- **Architectural Design:** Sa2VA uses `SAM2-Large` (224M) while UniPixel utilizes `SAM2.1-Base+` (81M). These mask heads exhibit similar segmentation capability. Ablation studies on both the base model and mask head will be presented below. Compared with Sa2VA, our key architectual innovations are the `Viusal Prompt Encoder w. Joint Positional & Temporal Encoding` (Sec. 3.1) and the `Object Memory Bank` (Sec. 3.2), both effectively enable unified object referring and segmentation.\n- **Supported Input/Output Formats:** Our method supports flexible visual prompt inputs (*i.e,* points, boxes, and masks) **at any frame(s)**. This is achieved by our visual prompt encoder that jointly encodes spatial position and temporal frame index of each visual prompt. Although Sa2VA's preprint also claims support for point and box prompts, their actual code, data, and experiments do not reflect this (Sa2VA GitHub Issue #19). Only mask-pooling-style region caption experiments are presented in their Table 15 and Figure 7.\n- **Frame Sampling Strategy:** Sa2VA samples the first 5 video frames as keyframe inputs to the MLLM, after which SAM2 is employed to track the object in the remaining frames. This is sub-optimal as the MLLM cannot perceive the entire video. Instead, UniPixel leverages **more flexible uniform or fixed FPS sampling strategies**, allowing the LLM to access more temporal context and thereby enhancing overall performance.\n\nWe also provide ablation studies to clarify the attribution of performance gains. The metrics are `J&F` (RVOS datasets) and `Acc` (VideoRefer$^Q$).\n\n| ID | Method | Base Model | Mask Head | ReVOS$_{val}$ | MeViS$_{val(u)}$ | Ref-DAVIS17$_{val}$ | VideoRefer$^Q$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| A | Sa2VA | InternVL2-4B | SAM2 (Large) | 53.2 | 52.1 | 73.8 | – |\n| B | Sa2VA | InternVL2.5-4B | SAM2 (Large) | – | 55.9 | 73.7 | – |\n| C | Sa2VA | Qwen2-VL-2B | SAM2 (Large) | 40.0 | 49.4 | 72.0 | – |\n| D | UniPixel | Qwen2-VL-2B | SAM2.1 (Base+) | 60.2 | **59.8** | 74.5 | 72.5 |\n| E | UniPixel | Qwen2-VL-2B | SAM2 (Large) | 60.4 | 59.6 | 73.9 | 72.6 |\n| F | UniPixel | InternVL2.5-4B | SAM2 (Large) | 60.9 | 61.2 | **74.6** | 72.5 |\n| G | UniPixel | Qwen2.5-VL-3B | SAM2.1 (Base+) | **61.4** | **59.8** | 74.1 | **72.8** |\n\n- **A, B, C**: Sa2VA's comparison presents the ranking of base MLLMs for RVOS tasks: `InternVL2.5-4B > InternVL2-4B > Qwen2-VL-2B`. This is aligned with our ablation study (**E** *vs.* **F**).\n- **D, E**: Changing from `SAM2.1 (Base+)` to `SAM2 (Large)` has minimal effect on model performance.\n- **B, F**: When using the same base model `InternVL2.5-4B` and mask head `SAM2 (Large)`, UniPixel clearly outperforms Sa2VA on MeViS (+9.4%) and Ref-DAVIS17 (+1.2%).\n- **D, G**: Upgrading from `Qwen2-VL-2B` to `Qwen2.5-VL-3B` improves the performance on ReVOS and VideoRefer$^Q$, while slightly downgrades the results on Ref-DAVIS17.\n\nThe above ablation study clarifies that the performance gains of UniPixel largely come from **architectural innovations and learning strategies**, rather than simple backbone/module upgrades. We will include more detailed discussions and the table above in our revision. Qwen2.5-VL based UniPixel will also be open-sourced.\n\n### **Q2: In Table 3, add C3VG [1] as a competitive non-LLM-based baseline.**\n\nThank you for highlighting this strong and important baseline! We have incorporated its evaluation results on RES task in both `Table 3` and `Appendix Table 4`, and included detailed discussions in `Sec. 2` (Related Work). Comparisons between C3VG and UniPixel-7B are presented below. Note that the `cIoU` (cumulative IoU) metric used in our work is equivalent to the `oIoU` in C3VG's paper. We compare results under both `mIoU` and `oIoU` metrics for comprehensive evaluations.\n\nUsing `mIoU` as metric:\n\n| Method | LLM | FT | RefCOCO$_{val}$ | RefCOCO$_{testA}$ | RefCOCO$_{testB}$ | RefCOCO+$_{val}$ | RefCOCO+$_{testA}$ | RefCOCO+$_{testB}$ | RefCOCOg$_{val(U)}$ | RefCOCOg$_{test(U)}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| C3VG [1] | – | – | 81.4 | 82.9 | 79.1 | **77.1** | 79.6 | 72.4 | 76.3 | 77.1 |\n| UniPixel | 7B | ✗ | 80.5 | 81.9 | 77.8 | 75.1 | 79.0 | 70.3 | 75.4 | 76.3 |\n| UniPixel | 7B | ✓ | **81.9** | **83.4** | **80.0** | 76.8 | **80.7** | **72.6** | **77.4** | **78.6** |\n\nUsing `oIoU` as metric:\n\n| Method | LLM | FT | RefCOCO$_{val}$ | RefCOCO$_{testA}$ | RefCOCO$_{testB}$ | RefCOCO+$_{val}$ | RefCOCO+$_{testA}$ | RefCOCO+$_{testB}$ | RefCOCOg$_{val(U)}$ | RefCOCOg$_{test(U)}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| C3VG [1] | – | – | 80.9 | **83.2** | 77.9 | **74.7** | 78.0 | **69.0** | 74.4 | 76.4 |\n| UniPixel | 7B | ✗ | 79.5 | 81.1 | 76.2 | 72.5 | 77.0 | 66.6 | 74.4 | 75.3 |\n| UniPixel | 7B | ✓ | **81.2** | 82.5 | **78.1** | 74.5 | **79.0** | 68.3 | **76.3** | **77.5** |\n\nHere, `FT` denotes fine-tuning on RefCOCO/+/g datasets after multi-task co-training. Without such fine-tuning, our model outperforms all LLM-based methods but slightly lags behind the non-LLM-based SOTA (C3VG [1]). Such a gap can be effectively closed by fine-tuning on these downstream datasets, as can be seen from the competitive post-FT results. The comparisons suggest that **UniPixel can achieve better or comparable results to non-LLM-based SOTAs like C3VG on RES**.\n\n### **Q3: In Appendix Table 5, add OneRef [2] and SimVG [3] as strong non-LLM SOTA methods.**\n\nThanks for this valuable insight! We have added the REC performance of OneRef-L (BEiT3-L) [2] and SimVG-TB (ViT-L/32) [3] in `Appendix Table 5`. The comparison between these methods and UniPixel is shown below.\n\n| Method | LLM | RefCOCO$_{val}$ | RefCOCO$_{testA}$ | RefCOCO$_{testB}$ | RefCOCO+$_{val}$ | RefCOCO+$_{testA}$ | RefCOCO+$_{testB}$ | RefCOCOg$_{val(U)}$ | RefCOCOg$_{test(U)}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| OneRef [2] | – | **93.2** | **95.4** | **90.1** | **88.4** | **92.1** | **82.7** | **87.8** | **88.8** |\n| SimVG [3] | – | 90.6 | 92.5 | 87.7 | 85.4 | 89.6 | 79.7 | 79.3 | 86.0 | 86.8 |\n| UniPixel | 2B | 89.0 | 92.1 | 86.8 | 81.9 | 86.9 | 76.5 | 85.3 | 85.7 |\n| UniPixel | 7B | 91.5 | 93.7 | 87.9 | 85.9 | 90.9 | 80.8 | 87.7 | 88.6 |\n\nOur 7B model slightly outperforms SimVG, but falls short of surpassing OneRef. We attribute it to the sub-optimal bounding box prediction, where the boxes are derived from masks rather than regressed by task-specific heads. This could potentially be improved by introducing dedicated bounding box heads following [1, 3], suggesting a promising direction for our future work. Nevertheless, to our best knowledge, UniPixel is currently **the best-performing LLM-based method (7B-level) on REC**.\n\n### **Q4: Details about the Object Memory Bank.**\n\nIt seems that OpenReview failed to render some parts of your questions. Based on the available content, we have attempted to reconstruct them as follows. Please kindly correct us if our interpretation is inaccurate.\n\n> In Lines 152–153, for objects [1]–[4], are the \\<MEM\\> tokens constructed using a single \\<SEG\\> token decoded by SAM2 to obtain masks across all frames?\n\nYes. For each object, one \\<SEG\\> token is decoded by SAM2 and be used to construct multiple \\<MEM\\> tokens. In this case, there are four objects, thus four \\<SEG\\> tokens are generated during memory pre-filling.\n\n> How is the multi-frame memory for each object built from a single \\<SEG\\> token?\n\nDuring memory pre-filling, the model predicts one \\<SEG\\> token for each object. This token is passed to SAM2 and serves as a \"hidden prompt\" on the first frame. SAM2 then predicts the object mask for the first frame and propagates it across the remaining frames. The output from SAM2 is a **multi-frame mask** (a binary mask with shape [$T$, $H$, $W$]), which would be saved into the object memory bank.\n\nIn the memory injection stage, the saved multi-frame mask is split into $T$ single-frame masks (each with shape [$H$, $W$]). We then construct the memory-enhanced prompt according to these masks. For example, for a video with 4 frames, if the mask for object [1] at frame <2> is all-zero (*i.e.,* the object is not visible in that frame), the prompt would be `[1]: <1> <MEM> <3> <MEM> <4> <MEM>` (<2> is omitted since the mask is empty). In this case, only the masks at frames <1>, <3>, and <4> are valid and are used to crop the corresponding visual features. The cropped 2D features are then average-pooled into 1D vectors, projected, and used to replace the \\<MEM\\> tokens in the prompt.\n\n> How does the model determine that <4> \\<MEM\\> does not exist in the last frame?\n\nThis is determined by the mask head. If the predicted mask in a frame is all-zero, it means that the object does not exist in that frame.\n\nWe will carefully revise `Sec. 3.2` to provide a clearer explanation of how the object memory bank works.\n\nWe hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!\n\n### **References:**\n\n[1] Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints, AAAI 2025.\n\n[2] OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling, NeurIPS 2024.\n\n[3] SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion, NeurIPS 2024."}, {"review_id": "M0roK3jM3n", "reviewer": "Reviewer_D9pB", "summary": "This paper proposes UniPixel, a large multimodal model (MLLM) designed to address the limitations of existing models in achieving pixel-level fine-grained understanding when processing images and videos. The authors present the UniPixel framework: a unified model capable of flexibly interpreting various visual prompts (points, boxes, masks), which inherently combines referring, segmentation, and reasoning capabilities. Through extensive experiments conducted across 10 different public benchmark datasets, UniPixel achieves state-of-the-art performance on seven distinct tasks, including referring/video object segmentation and region-based question answering.", "questions": "1. The authors propose the Object Memory Bank, but it requires a two-stage inference process (first extracting mask features, then semantic information). What are the advantages of this design compared to simply plugging in an external segmentation model? Given that the two-stage approach inevitably increases computational overhead and inference latency, how did you balance this trade-off?Were there attempts to achieve effective single-pass inference?\n\n2. At a high level, UniPixel’s framework resembles Draw-and-Understand [1], with three encoders and three token types for modeling. What are the key architectural innovations or distinctions here? \n\n3. For referring segmentation and referring understanding tasks, is the observed mutual improvement during training solely attributable to UniPixel’s decoupled design? Would this conclusion hold with a single one-turn inference model? (Intuitively, these are fundamentally distinct tasks. The finding that they can mutually enhance each other is intriguing)\n\n[1] Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want, ICLR 2025", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The ​Object Memory Bank​ is a novel and effective innovation. Its two-stage \"pre-fill → inject\" reasoning paradigm decouples segmentation/localization from high-level reasoning while ensuring the latter fully leverages the former’s fine-grained outputs.\n\n2. The work successfully integrates ​referring object segmentation, ​video QA, and ​region-based reasoning​ into a unified, end-to-end model. Experiments demonstrate state-of-the-art (SOTA) performance across multiple tasks.\nThe paper is well-structured, logically coherent, and highly readable.\n\n​Weaknesses:​​\n1. UniPixel’s strong performance may largely stem from its powerful base model (e.g., Qwen2-VL). The paper inadequately discusses the contribution of this foundational model. For example, whether comparable results could be achieved using only Qwen2-LLM.\n\n2. UniPixel’s design requires ​two forward passes​ per inference, introducing notable latency. The authors neither justify this design choice nor analyze its impact on inference efficiency.", "was_revised": true, "final_justification": "The authors have adequately addressed most of my concerns, and I recommend acceptance, and they should incorporate the related discussion content in the final version.", "rebuttal": "Many thanks for your careful review and insightful comments! We are encouraged by your recognition of our technical novelty, experiments, and writing. Below we provide responses to your concerns in detail.\n\n### **Q1: More Ablation Studies on the Base Model**\n\nThanks for your suggestion! We provide the following ablation study to clarify the attribution of UniPixel's performance gains. The evaluation metrics are `J&F` (RVOS datasets) and `Acc` (VideoRefer$^Q$).\n\n| ID | Method | Base Model | Mask Head | ReVOS$_{val}$ | MeViS$_{val(u)}$ | Ref-DAVIS17$_{val}$ | VideoRefer$^Q$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| A | Sa2VA | InternVL2.5-4B | SAM2 (Large) | – | 55.9 | 73.7 | – |\n| B | UniPixel | InternVL2.5-4B | SAM2 (Large) | 60.9 | 61.2 | **74.6** | 72.5 |\n| C | UniPixel | SigLIP-SO-400M + Qwen2-1.5B  | SAM2.1 (Base+) | 58.6 | 57.5 | 73.5 | 71.1 |\n| D | UniPixel | Qwen2-VL-2B | SAM2.1 (Base+) | 60.2 | **59.8** | 74.5 | 72.5 |\n| E | UniPixel | Qwen2.5-VL-3B | SAM2.1 (Base+) | **61.4** | **59.8** | 74.1 | **72.8** |\n\nSeveral conclusions can be drawn from the above results:\n\n- **A *vs.* B**: When using the same base model `InternVL2.5-4B` and mask head `SAM2 (Large)`, UniPixel clearly outperforms Sa2VA on MeViS (+9.4%) and Ref-DAVIS17 (+1.2%), suggesting that the performance gains of UniPixel largely come from **architectural innovations and learning strategies** rather than simple backbone/module upgrades.\n- **C *vs.* D**: Following your suggestion (*whether comparable results could be achieved using only Qwen2-LLM*), we trained an MLLM with `SigLIP-SO-400M + Qwen2-1.5B` with the same training recipe as VideoGPT+ [2], and adopted it as the base model. The results show that **slightly lower but comparable results** could still be achieved by a weaker base model. Notably, this variant remains **competitive with Sa2VA (InternVL2.5-4B)** on MeViS (+2.9%) and Ref-DAVIS17 (-0.3%).\n- **D *vs.* E**: We also tried to upgrade the base model from `Qwen2-VL-2B` to `Qwen2.5-VL-3B`. It improves the performance on ReVOS and VideoRefer$^Q$, but slightly downgrades the results on Ref-DAVIS17.\n\nMore detailed discussions on the contributions of base models will be included in our revision.\n\n### **Q2: Inference Efficiency when Using Object Memory Bank**\n\nThanks for bringing up this important point! We agree that the object memory bank design inevitably introduces additional latency. However, we clarify that:\n\n- This design affects efficiency **only when the inputs contain points or boxes**. Scenarios involving masks as visual prompts (*e.g.,* region captioning) or segmentation-only tasks (*e.g.,* referring video object segmentation) remain unaffected.\n- Predicting masks for point and box prompts before providing responses can help **better interpret the model’s underlying reasoning process**.\n- This is a reasonable trade-off considering the **improved region-level understanding performance**. Moreover, the extra computational overhead can be **effectively minimized using proper inference techniques**.\n\nBelow, we compare the efficiency and performance of different strategies on PixelQA$_{mixed}$. The inference speed was tested on a single RTX 6000 Ada GPU with Flash Attention 2.\n\n| ID | Object Memory Bank | Vision Encoder Cache | Inference Time (s/video) | $J$ | $F$ | $J$ & $F$ | Acc |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| A | ✗ | ✗ | 0.39 | 45.8 | 50.6 | 48.2 | 67.4 |\n| B | ✓ | ✗ | 0.81 | **46.4** | **51.7** | **49.0** | **68.5** |\n| C | ✓ | ✓ | 0.52 | **46.4** | **51.7** | **49.0** | **68.5** |\n\n> What are the advantages of this design compared to simply plugging in an external segmentation model?\n\nAdopting the object memory bank forces the model to predict masks (and leverage them to crop visual features) before actual reasoning. This enables the disentanglement of **object localization** and **regional understanding**, allowing both to benefit from training on related tasks (*e.g.,* referring object segmentation and region-based QA). The comparison between **A** and **B** in the above table demonstrates that this design can effectively enhance both segmentation and regional understanding capabilities.\n\n> Given that the two-stage approach inevitably increases computational overhead and inference latency, how did you balance this trade-off?\n\nThe extra computational overhead can be **minimized using proper inference techniques**. For example, given that the two inference stages share the same visual inputs, the `vision encoder outputs` and the `KV cache` for these visual tokens can be cached and reused across both stages. As shown in the above table (**B** *vs.* **C**), this technique improves the inference speed by **56%**, bringing it very close to that of single-pass inference.\n\n> Were there attempts to achieve effective single-pass inference?\n\nYes. In fact, **A** in the above table represents our simplified implementation of `segmentation + QA`. This is achieved by manually appending a \\<SEG\\> token after each visual prompt token. During inference, we decode these \\<SEG\\> tokens using the mask head to obtain object masks corresponding to the visual prompts, while treating the LLM outputs as textual responses. However, this paradigm does not allow the model to **\"re-watch\" key objects by mask-pooling important regions**, leading to sub-optimal performance in both segmentation and QA.\n\n### **Q3: Key Architectural Innovations or Distinctions Compared to Draw-and-Understand [1]**\n\nThanks for highlighting this highly related work! We acknowledge its contribution to visual prompting understanding, and will include more discussions in `Sec. 2` (Related Work). However, UniPixel is significantly different from their proposed VP-MLLM [1] in several aspects.\n\n| Method | Image Input | Video Input | Visual Prompt Input | Mask Output |\n|:-:|:-:|:-:|:-:|:-:|\n| VP-MLLM [1] | ✓ | ✗ | ✓ | ✗ |\n| UniPixel | ✓ | ✓ | ✓ (At Any Frames) | ✓ |\n\n- UniPixel supports **both image and video inputs**, whereas VP-MLLM can only perceive images.\n- Through joint positional and temporal encoding, UniPixel can accept visual prompt inputs **at any frames**, while VP-MLLM is limited to prompting on only one frame.\n- UniPixel integrates referring and segmentation into a **unified framework**, while VP-MLLM is a **referring-only** model and cannot perform segmentation tasks.\n\nConceptually, UniPixel can be regarded as a superset of VP-MLLM, as it supports unified object referring and segmentation in both images and videos.\n\n### **Q4: The Mutual Improvement of Referring and Segmentation Tasks**\n\nGood question! We also thought that these are fundamentally different tasks, but the experimental results in `Table 7(a)` demonstrate that they can be mutually improved through **multi-task co-training**. We hypothesize that this is because training on both tasks helps the model focus more on fine-grained semantics rather than solely on global information. Further investigation into this phenomenon is left for our future work.\n\n> Is the observed mutual improvement during training solely attributable to UniPixel's decoupled design?\n\nNo. It is also affected by **the ratio of the two dataset types**. During model training, the proportions of samples for `regional understanding` and `object segmentation` were 32.3% and 33.9%, respectively, which is approximately a 1:1 ratio. We observe that the mutual improvement effect disappears when the data distribution is imbalanced.\n\n> Would this conclusion hold with a single one-turn inference model?\n\nYes. According to `Table 7(a)` (the 3rd line), training on both tasks without the object memory bank can **also jointly improve segmentation and regional QA performance**.\n\nWe hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!\n\n### **References:**\n\n[1] Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want, ICLR 2025.\n\n[2] VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding, arXiv 2024."}, {"review_id": "yo11SFOyoD", "reviewer": "Reviewer_NW7Z", "summary": "Large multimodal models (LMMs) have achieved remarkable success as general-purpose multimodal assistants. However, the authors point out that existing LMMs have not yet been extended to **pixel-level understanding**. While some recent works have applied LMMs to tasks such as **region-level captioning** and **referring expression segmentation**, these tasks do not fully incorporate fine-grained perceptual capabilities.\n\nTo address this gap, the authors propose **UniPixel**, a unified large multimodal model that seamlessly integrates **pixel-level perception** with general **visual understanding**. The model demonstrates strong performance not only across multiple baselines but also on a newly introduced benchmark task, **PixelQA**, highlighting its ability to perform detailed and comprehensive multimodal reasoning.", "questions": "Please address the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This work is well-structured and easy to follow.\n\n2. Unipixel can unify the internal representations through the designed object memory bank. The effectiveness of the model is validated through extensive ablation studies.\n\n\n3. The authors demonstrate the superiority of the proposed model across numerous benchmark datasets, as well as on the newly introduced task, PixelQA.\n\n\n**Weaknesses**\n\n1. Most of the LMMs compared in this work are 2B or 7B in scale. While the proposed model demonstrates superior performance compared to similarly sized models, it remains unclear whether it can consistently outperform **larger-scale LMMs**. Comparing against such larger models would strengthen the credibility of the claimed superiority and provide a more comprehensive evaluation of the proposed approach.\n\n2. The authors could consider including the GPU requirements for training the proposed model. This information would be valuable for researchers who wish to reproduce the work and better understand the computational resources needed.\n\n3. The authors have not released the code or the pretrained weights for community use.", "was_revised": true, "final_justification": "The authors' response have addressed my major concerns. I still maintain a positive attitude toward this paper.", "rebuttal": "Thank you for recognizing the value of our work and for providing the constructive feedback! Below we provide detailed responses to your concerns and suggestions.\n\n### **Q1: Whether the proposed model can consistently outperform larger-scale LMMs?**\n\nThanks for the insight! We agree that comparisons with larger models enable a more comprehensive evaluation of the proposed method. In our work, we have already included models larger than 7B such as `LISA-13B` [1], `VISA-13B` [2], `M2SA-13B` [3], and `InternVL-26B` [4]. Below, we supplement our analysis with additional larger models.\n\n| Method | LLM Size | ReVOS$_{val}$ | MeViS$_{val(u)}$ | Ref-DAVIS17$_{val}$ | VideoRefer$^Q$ | PixelQA$_{mixed}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| GPT-4o | – | – | – | – | 71.3 | 69.3 |\n| Qwen2-VL [5] | 72B | – | – | – | 70.8 | 69.1 |\n| Sa2VA [6] | 26B | 58.4 | 57.3 | **77.0** | – | – |\n| UniPixel | 2B | 60.2 | 59.8 | 74.5 | 72.5 | 68.5 |\n| UniPixel | 7B | **63.8** | **59.9** | 76.1 | **76.4** | **69.9** |\n\nCompared with larger models such as `GPT-4o`, `Qwen2-VL-72B`, and `Sa2VA-26B`, the segmentation and referring capabilities of UniPixel-7B are still competitive, as demonstrated by the **SOTA performance on ReVOS, MeViS, VideoRefer$^Q$, and PixelQA datasets**.\n\n### **Q2: The authors could consider including the GPU requirements for training the proposed model.**\n\nThanks for the suggestion! We trained our model with 8 RTX 6000 Ada (48GB) GPUs. Training the 2B and 7B variants took around 2.5 days and 4 days, respectively. This information will be included in our revision.\n\n### **Q3: The authors have not released the code or the pretrained weights for community use.**\n\nWe clarify that all the code, model checkpoints, training logs, and online demos for this project will be **fully open-sourced** upon acceptance of the paper. We also welcome and will support the community to develop stronger unified referring and segmentation models based on our codebase.\n\nWe hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!\n\n### **References:**\n\n[1] LISA: Reasoning Segmentation via Large Language Model, CVPR 2024.\n\n[2] VISA: Reasoning Video Object Segmentation via Large Language Models, ECCV 2024.\n\n[3] MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation, ICLR 2025.\n\n[4] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks, CVPR 2024.\n\n[5] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, arXiv 2024.\n\n[6] Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos, arXiv 2025."}, {"review_id": "YrhlW1CCQw", "reviewer": "Reviewer_EraF", "summary": "This paper proposes a model called UniPixel, which is designed for pixel-level visual reasoning tasks. The model has the ability to flexibly comprehend visual prompt inputs (such as points, boxes, and masks) and generate mask-grounded responses. The motivation is clear. The writing is good. Extensive experiments are conducted to validate the effectiveness of the proposed method.", "questions": "Please refer to [Weaknesses].", "limitations": "Please refer to [Weaknesses].", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n* UniPixel introduces an object memory bank, which effectively integrates the internal representations of referred and segmented objects. This allows the model to perform object referring and segmentation tasks more effectively than previous methods.\n* The authors introduce a new PixelQA task that requires joint object-centric referring, segmentation, and QA in videos.\n\n[Weaknesses]\n* The main contribution of this paper is constructing a model that can perform both referring and segmentation tasks. However, I think the authors overstate their claim here. Visionllm can perform both referring and segmentation tasks. It also has a region encoder, which can encode the input prompt.\n1. Wu J, Zhong M, Xing S, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks[J]. Advances in Neural Information Processing Systems, 2024, 37: 69925-69975.\n\n*  In Object Memory Bank, when performing memory pre-filling, we need to estimate the object mask. Therefore, two forward passes are needed for one question (One for mask prediction and one for answer generation). This process increases the computational complexity. In addition, how to train the model with this mechanism?\n\n* The authors compare many methods in the experiments. It is important to point out the base model used by the method. For example, LISA uses the LLaMA while the proposed method uses the Qwen-2, which is a more powerful model.\n\n* What is the performance of the proposed model on the general VQA tasks, such as VQAv2, MME?", "was_revised": true, "final_justification": "The performance is good. The method is well-motivated.", "rebuttal": "Thank you for your careful review and for the insightful feedback! We now provide detailed responses to address your concerns.\n\n### **Q1: VisionLLM v2 [1] can also perform both referring and segmentation tasks.**\n\nThanks for highlighting this important baseline! We clarify that our statement of **\"Unified\"** refers to the unification of **(1) referring & segmentation** and **(2) processing images & videos**. Although VisionLLM v2 [1] can also jointly perform object referring and segmentation, it is an **image-only** model and thus cannot achieve pixel-level video understanding. We will carefully revise the relevant statements in our paper to ensure no overclaim. Below, we also provide performance comparisons between VisionLLM v2 and UniPixel.\n\nComparison on `Referring Expression Segmentation (RES)` and `Reasoning Segmentation` tasks:\n\n| Method | Base Model | RefCOCO$_{val}$ | RefCOCO$_{testA}$ | RefCOCO$_{testB}$ | RefCOCO+$_{val}$ | RefCOCO+$_{testA}$ | RefCOCO+$_{testB}$ | RefCOCOg$_{val(U)}$ | RefCOCOg$_{test(U)}$ | ReasonSeg$_{gIoU}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| VisionLLM v2 [1] | Vicuna-7B | 76.6 | 79.3 | 74.3 | 64.5 | 69.8 | 61.5 | 70.7 | 71.2 | 51.0 |\n| UniPixel | Qwen2-VL-2B | 76.1 | 79.2 | 72.3 | 67.5 | 72.2 | 60.6 | 70.6 | 71.4 | 59.6 |\n| UniPixel | Qwen2-VL-7B | **79.5** | **81.1** | **76.2** | **72.5** | **77.0** | **66.6** | **74.4** | **75.3** | **63.0** |\n\nComparison on `Referring Expression Comprehension (REC)` task:\n\n| Method | Base Model | RefCOCO$_{val}$ | RefCOCO$_{testA}$ | RefCOCO$_{testB}$ | RefCOCO+$_{val}$ | RefCOCO+$_{testA}$ | RefCOCO+$_{testB}$ | RefCOCOg$_{val(U)}$ | RefCOCOg$_{test(U)}$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| VisionLLM v2 [1] | Vicuna-7B | 87.9 | 91.2 | 84.3 | 77.6 | 83.8 | 70.2 | 82.9 | 84.1 |\n| UniPixel | Qwen2-VL-2B | 89.0 | 92.1 | 86.8 | 81.9 | 86.9 | 76.5 | 85.3 | 85.7 |\n| UniPixel | Qwen2-VL-7B | **91.5** | **93.7** | **87.9** | **85.9** | **90.9** | **80.8** | **87.7** | **88.6** |\n\nResults in both tables demonstrate that **UniPixel significantly outperforms VisionLLM v2** in RES, ReasonSeg, and REC tasks. We will add the above comparisons into `Table 3` and `Appendix Table 4`, and include more detailed discussions in `Sec. 2` (Related Work).\n\n### **Q2: The two-stage inference strategy increases the computational complexity.**\n\nThanks for mentioning this important point! We agree that the object memory bank design inevitably introduces additional latency. However, we clarify that:\n\n- This design affects efficiency **only when the inputs contain points or boxes**. Scenarios involving masks as visual prompts (*e.g.,* region captioning) or segmentation-only tasks (*e.g.,* referring video object segmentation) remain unaffected.\n- Predicting masks for point and box prompts before providing responses can help **better interpret the model’s underlying reasoning process**.\n- This is a reasonable trade-off considering the **improved region-level understanding performance**. Moreover, the extra computational overhead can be **effectively minimized using proper inference techniques**.\n\nBelow we compare the efficiency and performance of different strategies on PixelQA$_{mixed}$. The inference speed was tested on a single RTX 6000 Ada GPU with Flash Attention 2.\n\n| ID | Object Memory Bank | Vision Encoder Cache | Inference Time (s/video) | $J$ | $F$ | $J$ & $F$ | Acc |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| A | ✗ | ✗ | 0.39 | 45.8 | 50.6 | 48.2 | 67.4 |\n| B | ✓ | ✗ | 0.81 | **46.4** | **51.7** | **49.0** | **68.5** |\n| C | ✓ | ✓ | 0.52 | **46.4** | **51.7** | **49.0** | **68.5** |\n\n- **A *vs.* B**: Adopting the object memory bank can **effectively enhance both segmentation and regional understanding capabilities**. This is due to the disentanglement of **object localization** and **regional understanding**, allowing both to benefit from training on related tasks (*e.g.,* referring object segmentation and region-based QA).\n- **B *vs.* C**: Given that the two inference stages share the same visual inputs, the `vision encoder outputs` and the `KV cache` for these visual tokens can be cached and reused across both stages. This technique improves the inference speed by **56%**, bringing it very close to that of single-pass inference.\n\n> In addition, how to train the model with this mechanism?\n\nWe adopt specialized training samples and design a rule-based pipeline to achieve this mechanism:\n\n- During training, as shown in `Appendix Table 1`, we repurposed video QA samples from `VideoRefer-700K` [2] and `Inst-IT` [3] as **memory pre-filling samples**, in which the model is expected to predict masks for a question with visual prompts. An example is shown below.\n\n```\n Input: What are [1] <REF> and [2] <REF> doing in the video?\nOutput: The relevant regions in the question are [1] <SEG> [2] <SEG>.\n```\n\n- During inference, when \\<REF\\> tokens are detected in the input prompt, we manually construct the corresponding model response using the output template above, ensuring it contains the same number of \\<SEG\\> tokens. We then apply **teacher forcing** to guide the model in predicting masks for these visual prompts. The predicted masks are subsequently used to construct \\<MEM\\> tokens to enable regional understanding.\n\nWe will carefully revise `Sec. 3.2` to provide a clearer explanation of how the object memory bank works.\n\n### **Q3: It is important to point out the base model used by the method.**\n\nGood suggestion! We will revise the `Size` column to `Base Model` in all the tables to clearly indicate the underlying LLMs/MLLMs of different methods.\n\n### **Q4: What is the performance of the proposed model on the general VQA tasks, such as VQAv2, MME?**\n\nWe compare UniPixel with a strong counterpart `Sa2VA` and its baseline `Qwen2-VL` on these benchmarks below. The evaluations were conducted using `lmms-eval`.\n\n| Method | Base Model | VQAv2$_{testdev}$ | MME (perception/cognition) |\n|:-:|:-:|:-:|:-:|\n| Sa2VA | InternVL2-4B | – | 1553/540 |\n| – | Qwen2-VL-2B | 61.7 | 1495/366 |\n| – | Qwen2-VL-7B | **78.0** | 1688/628 |\n| UniPixel | Qwen2-VL-2B | 61.9 | 1564/580 |\n| UniPixel | Qwen2-VL-7B | 77.5 | **1695/643** |\n\nThe comparison shows that UniPixel **outperforms Sa2VA** while is **comparable to Qwen2-VL** on general image QA tasks, demonstrating the effectiveness of the proposed fine-grained understanding designs.\n\nWe hope our responses above can address your concerns. More discussions are welcome if you have any further questions. Thank you!\n\n### **References:**\n\n[1] VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks, NeurIPS 2024.\n\n[2] VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM, CVPR 2025.\n\n[3] Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning, arXiv 2024."}]

Author rebuttal:

Meta-review: This paper introduces UniPixel, a unified multimodal model for pixel-level visual reasoning that integrates object referring, segmentation, and reasoning across images and videos. Its key contribution is the Object Memory Bank, enabling region-specific reasoning and supporting the new PixelQA task. The work is clearly written, well-motivated, and demonstrates strong results across 10 benchmarks. Reviewers appreciated the novelty, unified framework, and extensive evaluation. Main concerns centered on whether performance gains are primarily due to backbone upgrades, and the efficiency trade-offs of the two-stage inference. The rebuttal provided detailed ablations, clarified efficiency mitigations, and expanded comparisons to stronger baselines and larger LMMs. Overall, while some reservations remain, the consensus is that the paper is technically solid and novel, with scores of 4, 4, 4, and 5. It makes a valuable contribution to unified pixel-level vision-language reasoning. The authors are encouraged to provide more detailed explanations of the memory bank mechanism and include additional ablation studies in the camera-ready version.
Decision: Accept (poster)