UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Authors: Hao Tang
Chen-Wei Xie
Haiyang Wang
Xiaoyi Bao
Tingyu Weng
Pandeng Li
Yun Zheng
Liwei Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: fine-grained visual perception
unified modeling
multimodal large language models
Primary area: applications

Abstract: Generalist models have achieved remarkable success in both language and vision-language tasks, showcasing the potential of unified modeling. However, effectively integrating fine-grained perception tasks like detection and segmentation into these models remains a significant challenge. This is primarily because these tasks often rely heavily on task-specific designs and architectures that can complicate the modeling process. To address this challenge, we present UFO, a framework that unifies fine-grained visual perception tasks through an open-ended language interface. By transforming all perception targets into the language space, UFO unifies object-level detection, pixel-level segmentation, and image-level vision-language tasks into a single model. Additionally, we introduce a novel embedding retrieval approach that relies solely on the language interface to support segmentation tasks. Our framework bridges the gap between fine-grained perception and vision-language tasks, significantly simplifying architectural design and training strategies while achieving comparable or superior performance to methods with intricate task-specific designs. After multi-task training on five standard visual perception datasets, UFO outperforms the previous state-of-the-art generalist models by 12.3 mAP on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation. Furthermore, our method seamlessly integrates with existing MLLMs, effectively combining fine-grained perception capabilities with their advanced language abilities, thereby achieving superior performance on the challenging reasoning segmentation. Code and models are available at https://github.com/nnnth/UFO.

Xiaoyi Bao UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Hao Tang!. 1.2
Chenwei Xie²
Haiyang Wang
Tingyu Weng? Pandeng Li
Yun Zheng Liwei Wang" 1,4,5t
Center for Data Science, Peking University Alibbba Group
3 CASIA Center for Machine Learning Research, Peking University
State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China (tanghao@stu, wanghalyangâstu, wanglwecis) pku. edu. baoxiaoy12021eia, ac.............................. cn {eniac.xcs, vengtingyu. lipandeng. 1pd, thengyun.yy)@alibenn inc
Abstract
Generalist models have achieved remarkable success in both language and vision- language tasks, showcasing the potential of unified modeling. However, effectively integrating fine-graing een-igrined perception tasks like detection and segmentation into these models remains significant challenge. This primarily because these tasks often rely heavily on task-specific designs and architectures that can complicate the modeling process. To address this challenge, present UFO, framework that Unifies Vine-graines visual perception tasks through an Open-ended language in- terface. By transforming all perception targets into the language space, UFO unifies object-level detection, pixel-level segmentation, and image-level lision-language tasks into a single model. Additionally, we introduce novel embedding retrieval approach that relies solely on the language interface to support segmentation tasks. Our framework bridges the gap between fine-grat fine-griieed perception and vision- language tasks, significantly simplifying architectural design and training strategies while achieving comparable or superior performance to methods with intricate task-specific designs. After multi-task training on five standard visual perception datasets, UFO outperforms the previous state-of-the-art generalist models by 12.3 mAP on COCO instance segmentation and 3.3 mIoU on ADE20K semantic seg- mentation. Furthermore, our method seamlessly integrates with existing MLLMs, effectively combining ain-igrined perception capabilities with their advanced language abilities, thereby enabling more challenging tasks such as reasoning seg- mentation. Code and models available attps://e //t///ggittt comccoo/////////e wnntt/nntttr
1 Introduction
Multimodal large language models (MLLMs) [51, 92, 40, 2, 12, 46] have made significant progress, exhibiting outstanding performance on various visual tasks. Despite these achievements, their scopes are largely confined to image-level vision-language tasks, leaving fine-grained perception (e.g., detection detection and segmentation) as critical weakness. Recent studies have shown that enabling MLLMs to collaborate with off-the-shelf detectors and segmenters understanding [80, 18] and facilitate advanced applications such as mobile agents [71, 70, 82, 41], indicating that endowing MLLMs with fine-grained perception capabilities is beneficial. However, seamlessly integrating these tasks into MLLMs poses challenges because traditional specialized methods heavily rely on complex and task-specifie designs, such RPN [59] and mask decoders [34].
enhance precise visual
"Corresonndigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Polygon
Box Decodor
Mask Decoder
retrieval
<DFT>,
<SFG>
<36m4H96H122
diga..
shheepo...
<MASK
Multi-Modal Transfarmer
Molti-MMdaallaa Transfomer
Mult-Moddl Transformer
Please
Please
locate/segment
locate kaggent the bottom goat.
locate/segment
the bottom
the bottom
(a) Task-
Text-based
[c] Ours
Figure 1: Methods to augment MLLMs with fine- time-graied perception tasks. (a) Relying on task
decoders [37, 77], (b) Previous text-based methods represent boxes with location tokens [52] and represent masks with suboptimal polygons [74, 69] or textual classes [69, 38), (c) Ours: predicting open-ended text sequences while using a simple yet effective embedding retrieval approach for masks.
Many existing works [37, 77, 57, 60, 53, 89, 11] augment MLLMs with task-specific decoders, such as LISA [37] using SAM for segmentation VisionLLM v2 [77] adding box and mask decoders. However, this combination introduces several limitations. First, task decoders add architectural complexity, necessitating compatibility among multiple components whenever the LLM scaled up or new task decoders are introduced. Second, it complicates the end-to-end training. For example, the last stage VisionLLM v2 [77] is dedicated
finetuning the task decoders because they fail to converge in earlier stages. These issues create a significant discrepancy with traditional vision- language modeling, limiting their broader application in general-purpos MLLMs. To remove task decoders, another line of research [52, 7, 81, 74, 54] converts boxes into location tokens or textual numbers and transform masks into polygon vertices. However, using a limited number of vertices for masks introduces quantization errors, especially for masks with complex shapes and multiple regions.
To address the above limitations, GiT [69] uses two mask representations: for instance segmentation, it uses polygons, and for semantic segmentation, it predicts textual classes for each pixel, as shown in Figure (b). However, it still falls short of unifying fine-grained perception tasks into MILMs. Firstly, although textual class can represent masks with any shape, it results in overly long sequences and slow inference. Secondly, achieve better performance. GiT sets specific vocabularies for each task (c.g.. detection can only output location tokens), which is incompatible with open-ended text generation. Finally, GiT does not scale MLLMs and uses Vision Transformer (ViT [21]) for multimodal tasks, resulting in poor language abilities. Hence, it is essential to develop more effective approach unify fine-graine graiind perception into MLLMs. This method should effortlessly align with open-ended language interfaces, involve minimal structural complexity and deliver excellent performance.
In this paper, we present UFO, which unifies fine-grained perception tasks through the same open- ended language interface as vision-language tasks, without any task decoders. By carefully organizing and translating all task outputs into open-ended text sequences, we demonstrate that competitive performance can be achieved without complex task-specific designs. As illustrated in Figure I (c), we reformulate segmentation as an embedding retrieval problem, where the mask token embedding computes similarity with image features by dot product, retrieving high-similarity positions to generate the mask. This design effectively leverages the output image features processed by MILMM, which are often overlooked in previous methods. Our intuition is that since MLLMs achieve strong visual understanding, the mask information is already in the image features and we just need to retrieve it. Furthermore, we introduce a novel method that upsamples output masks by predicting multiple mask tokens, resulting in more refined masks and improved performance. Thanks to this strategy, we can efficiently and accurately represent masks of any shape using only 16 tokens.
We first validate our method following GiT [69], which uses a lightweight ViT but can share the same formulation as MLLMs (see Table 1), allowing efficient validation. GiT constructs a comprehensive multi-task benchmark, which covers various granularity fine-grained perception tasks. Under the same evaluation protocols, UFO outperforms GiT by 12.3 mAP and 3.3 mloU in COCO instance segmentation and ADE20K semantic segmentation (see Table 2). We then scale our method to MLLMs to integrate language abilities with fine titeegrained perception. As shown in Table 3, UFO achieves competitive results in visual grounding without decoders or polygon approximations. Benefiting from the shared representations of the open language interface, UFO can deeply unify textual reasoning and image segmentation, surpassing the previous state-of-the-a method on the challenging ReasonSeg [37] benchmarks by 6.2 gloU (see Table 4).

---

In summary, our contributions are listed as follows:
(1) We introduce UFO, unified framework for diverse fine-grained perception tasks through the same open-ended language interface as vision-language tasks, without task-specific decoders.
(2) We reformulate segmentation as embedding retrieval problem, exploring both text generation and image representation abilities of the language interface, significantly outperforming previous text-based methods instance and semantic segmentation tasks.
(3) Our framework seamlessly integrates with MLLMs, delivering better performance than previous ofateoffthhe-ar methods on the ReasonSeg benchmarks.
Related Work
2.1 Multimodal Large Language Models
Inspired by the success of large language models (LLMs), multimodal large language models (MLLMs) have rapidly advanced in recent years. Early efforts [43, 92, 17] finetune LLMs with instruction datasets, demonstrating strong multimodal understanding. More advanced MLLMs like Qwen2.5-VL. [2], and VL2255 [12] have emerged recently, offering superior multimodal comprehension through larger model sizes and extensive training data. However, these models mainly focus on imaee-level vision-language tasks, with less exploration of fine-grained visual perception.
2.2 Extend MILMs with Fine-grained Perception
Extend MLLMs with Task Decoders. Recent works [37. 77, 57, 60, 3, 87, 53, 78, 88, 84, 89, 76] introduce task decoders to extend MILMs with tasks like detection and segmentation. These models treat the MILMM as coarse proposal extractor, passing the task-relevant embeddings to specialized decoders. The decoders then manage task-specific details, such as regressing boxes or generating masks. Although this approach yields strong performance, extra task decoders complicate architectures and training, undermining the unified design of MLLMs and limiting their potential. Recently, HiMTok [73] reformulates segmentation as mask image generation. However, this approach still requires training a specialized VQ decoder, increasing both training and structural complexity.
Extend MLLMs with Text Outputs. For object-level tasks, previous methods [52, 7, 81, 68, 74, 6, 54] have employed location tokens or textual numbers to represent boxes. For pixel-level tasks such as segmentation, common format is polygonal approximation [74, 54]. Although Vistal.LM [54] reduces the errors of polygons through adaptive sampling, it is inadequate for general segmentation tasks. First, polygons struggle to accurately represent "stuff" categories with amorphous regions (e.g.,
roads with parked cars), which are common in real world [90, 4, 16]. Second, polygons inherently cause information loss, especially for detailed structures like retinal vessels [64]. Text4Seg [38] directly predicts textual labels for image patches but still requires an additional refiner (e.g.. SAM [34]) to achieve better performance. In contrast, UFO leverages the multimodal outputs of MLLMs to generate precise masks for any shape, offering greater expressiveness and improved performance.
2.3 Vision Generalist Models
Vision generalist models aim to establish a unified framework supporting various vision-centric tasks. Inspired by the seq2seq framework in NLP, previous generalist models [72, 69, 9, 47] transform visual tasks into sequence generation problems. Notably, GiT [69] unifies five core visual tasks by language interface, supporting box, mask, and text outputs. However, these models typically focus solely on visual tasks and lack the advanced language capabiliities required for complex reasoning [37].
Methods
As our method applicable to various multimodal architectures, we first present a unified architectural
abstraction in Section 3.1. Then, in Sections 3.2 and 3.3, we explain how to integrate box and mask representations into the open-ended language interface. Finally, in Section 3.4, we describe our multi-task data template for joint training.
3

---

Table 1: We abstract current multimodal architectures into three components: (1) Image tokenizer, converting images into visual tokens; (2) Text tokenizer, outputting text tokens; (3) Multimodal transformer, jointly processing visual and text tokens. We construct three variants by this formulation.
Model
Image Tokenizer
Text Tokenizer
Multimodal Transformer
LLaVA [43]
CLIP MLP
Llama Tokenizer [67]
Vicuna [14]
EVE [20]
Patch embedding
Llama Tokenizer [67]
Vicuna [14]
GiT [69]
Patch embedding
Bert Tokenizer [ [19]
VIT [21]
UPO-VI VIT
Patch Embedding
Bert Tokenizer [19]
VIT [21]
UPO-LLaVA-1.5-78
CLIP [56).MLP
Llama Tokenizer [67]
Vicuna 1.5 [14]
UFO VL2.5-88
IntervViT M12)
IntemLM2.55 Tokenizer [86]
InternL [86]
3.1
Preliminary
Our goal is to unify fine-grained perception tasks into the open-ended language interface, thereby ensuring compatibility with any multimodal architecture that supports the same interface. We abstract existing multimodal architecture into three components based on the modalities they process: image tokenizer, text tokenizer and multimodal transformer, as shown in Table 1. For example, in LLaVA [43], the image tokenizer includes a vision encoder and MLP connector that extract visual features and map them into the LLM's input space, while the multimodal transformer corresponds to the LLM. This abstraction applics not only to MLLMs with various image tokenizers [43, 40, 20] but also to vision generalist models with similar architectures [69], significantly broadening the scope of our method. To avoid confusion, we will refer to MLLMs by default the following sections.
3.2 Bounding Box Representation
To align with the open-ended language interface while avoiding the addition of extra location tokens, we directly translate boxes into textual numbers. Each box is represented by the coordinates of its top-left (æ1; V1) and bottom-right (z2, U2) corners. The continuous values of these coordinates are discretized into integers within [0, range], enclosed by <box> and </box> tokens. a class label is required, simply prepend the textual class before the <box> token. For example, a box of a person can be represented as: person, <box>465, 268, 589, 344</box>. This method converts boxes to open-ended sequences, effectively aligning with vision- languaee tasks.
Segmentation
Image Tokenizer
Donn
dot product
Interpolate
Image
Multi-Modal Transformer
DOOM
is
Output Mask At (f6x1()
Final Mask (448x643)
(b) Seg with multiple mask tokens
Input Image Xy
nAnO
dot product
Image Featwre h.
MAOR
Interpolate
you sogment// locate
Text Tokenizer
It <MAASE--MAS WAAASMMMMAASS------------
Mask. R., (38x32)
(448x448)
goat to the bottom
stone?
Detection
Input Texts
It located <Boc56 4,96,122</Boo..
Outputs
Figure Overview of our approach. (a) Segmentation modeling: the mask token embedding retrieves similar image features to generate masks (shown with matching colors). (b) Upsampling masks by multiple mask tokens, retrieving more details by more tokens. We use N=2 to illustrate while using N= in implementation. (c) We output open-ended text sequences with textual numbers for detection.
3.3 Mask Representation
Representing masks via the language interface more challenging because masks contain more detailed information than boxes. Previous methods either use polygon formats, which sacrifice details,

---

or assign textual classes to each pixel, resulting in overly long sequences. Therefore, a more efficient method to represent detailed masks is needed.
We observe that in MILMs, the language interface is actually multimodal, where projected image features and text features are combined and jointly processed by the LLM. However, most existing methods ignore the output image features processed by the LLM. We argue that since MLLMs can express where and what objects are in text form, the mask information is already encoded in the image features. We just need to teach the model to decode this information. Therefore, we design a representation method based on image features and text embeddings. Instead of storing mask information in text embeddings, we use the text embeddings as query embeddings to extract mask information from the image features. The detailed approach is described below.
Segmentation by Embedding Retrieval. To incorporate the segmentation task using only the language interface, we reformulate it as an embedding retrieval problem. We first augment the basic vocabulary of the model with a <MASK> token, which serves as the indicator for mask generation. When performing segmentation, the model is trained to output the <MASK> token, as shown in Figure (a). Formally, given an input image Xv and segmentation prompt Xx, the model F generates the text response Yt and corresponding output embeddings hG, image features hy as:
hv: Yt, h, F(xv, Xt).
We extract the mask token embedding em corresponding to the <MASK> token from h,- To generate the segmentation mask, we compute the similarity between the mask token embedding om and the image features hy via a scaled dot product. Positive scores are retrieved form the binary mask M. This process is expressed as:
8 em hy
M 1(s > 0),
where d is the embedding dimension, s represents the similarity scores, and I the indicator function that converts the similarity scores into binary mask. By computing the dot product similarity between the mask token embedding and image features, we retrieve the most relevant image features corresponding to the mask token, thereby producing a mask aligned with the original image.
Our approach leverages MILMS' inherent capabilities for segmentation without task decoders. We hypothesize that, in well-encoded image features, features with the same semantics will group into clusters. Therefore, generating mask token embedding equates to identifying the center of the relevant image feature cluster, while computing the similarity reflects this relationship. This approach can easily apply to other pixel-level tasks, such as depth estimation (see Table 20 in the appendix).
Upsampling by Multiple Mask Tokens. Due to the redundancy in visual information, it is common to process visual features at reduced resolutions. For example, the CLIP-L/14 [56] downsamples image features by factor of 14. In above method, similarities are computed using downsampled image features, resulting in low-resolution masks. However, directly upsampling by interpolation leads to coarse results and suboptimal performance due to the high interpolation factor.
To address this issuc, we propose upsampling method by predicting multiple mask tokens. For an image Xy E # x3 we obtain image features hy E ,,,,,,,,** xW, downsampled by the patch size p. where d represents the feature dimension. Our target is to upsample the generated mask by N times, producing Map (IM,,N))(W,, from image features h, E BR x,,, This requires decoding an N x N mask for each position in the image features. To achieve this, we train the model to autoregressively predict 2 <MASK> tokens with embeddings (Cm, Jim1- Each token corresponds single position in the N x N upsampling grid, as illustrated Figure (b). For each mask token embedding C., we compute the similarity with the visual features h,:
N
8; emp h Vd
(3)
where em, E Rixd hy E H, and 8; E XlxH, These similarity scores {s:) are then concatenated and reshaped into an upsampled similarity map:
Sconcar = concat( (
Sconcat E Nx,,w,,
(4)
Sup reshape(s
Sup E E (H,N))(W,,,)
(5)
5

---

Box Output
Mask Output
Multi Prediction
<baxo.... </box
Sure, It <M <MASIo..sMAKC.
56,4,66,,,,,,12227 56,4,,61,2222//boo
duck, <boxo. of booe>
Multi-Modal Transformer
Multi-Modal Transformer
Multi-Modal Transformer
Can you locate the goat nearest to the stone?
Can segment the goat nearest to the stone?
Detect COW, personyduck.
<Text Prompt> Figure 3: Multi-task data template examples. Red dots represent sampled grid point features, acting as local visual prompts for generating text sequences for nearby objects.
<Text Prompt>
<Image>
<Image>
<Text Prompt> <lmage> <Local Feature>
Finally, we retrieve positive scores in Sup to generate the upsampled binary mask Mup- By default, set N 4, predicting 16 <MASK> tokens, which upsamples the output mask by factor of 4. The mask is then aligned with the original image resolution through interpolation.
Our method effectively leverages mask token embeddings upsampling parameters, offering greater flexibility than traditional methods like bilinear interpolation and transposed convolution. Bilinear interpolation
non-leamable parameters, while transposed convolution allows for learnable pa- rameters, the same parameters are applied to all images after training. In contrast, we use embeddings generated by the network the parameters, which can be customized for each image. This approach enables the model to generate optimal upsampling parameters dynamically, enhancing flexibility while achieving better performance (see Table 6).
Note that our method is fully compatible with open-ended language interfaces. We refer "open-endee" as the capability to generate variable-length text sequences terminated by end-of-sequence token. In our approach, while we require fixed-length <MASK> tokens for segmentation, the text generation itself remains open-ended. Only after the generation is complete, we check if the output contains <MASK> segments of the required length. Furthermore, the interaction with image features is also performed after text generation is finished.
3.4 Multi-Task Data Template
Based on the above designs, we construct multi-task data templates for joint training. We classify tasks into categories based on prediction number: single-prediction tasks like grounding produce one box or mask, and multi-prediction tasks like object detection generate several boxes. Merging multiple outputs into a long sequence is inefficient and the order among them is hard define, making autoregressive learning of the sequence difficult [8]. Therefore, we adopt a parallel decoding approach that splits nulti-prediction tasks into independent subtasks, each handling one prediction in parallel. This strategy effectively accelerates inference and enhances task scalability.
Single prediction. For tasks only require single prediction, our task template is: <Text Prompt>< <ert>><mmmme<<x> <Text Response>. As shown in Figure 3, we follow previous methods to con- struct text prompts and use our unified box and mask representation for text responses.
Multiple predictions. To efficiently support multi multi-prediction tasks, we split complex tasks into independent subtasks with single prediction, enabling parallel decoding within batch. The key to achieving parallelism is to ensure all subtasks are independent. Typically, multiple boxes and masks correspond to different locations. Therefore, we introduce local image features in the input to differentiate these sub-tasks, serving as visual prompts. The template is structured follows: <Text Prompt><< < Image><Local>><T] terpolated by grids sampled the image. The core idea is that each grid point is responsible for detecting its spatially nearest objects. During training, each ground- -truh object is assigned to its nearest grid point, while the remaining grid points are assigned to predict end-of-sequence tokens. During inference, as illustrated in Figure 3, we sample points over the image, typically of size K x K, resulting in total of M K² points. We then interpolate the image features at each of the M grid locations to extract distinct grid features. These grid features, along with the global image features and the text prompt, are fed into the LLM. The example input sequence is structured as follows:
Response>, where <Local> refers to local image features in-
Detect cow, person, duck. < Immaee >< Local, >< Localg ... Localy >
(6)

---

Table 2: Results GIT [69]'s multi-task benchmark. "*" denotes the model is capable of the task but no number is reported. "_" means incapability. We highlight joint training improvements with bold font and follow [69] to list modules for specific functions.
Medhods
Specini Modiles
Nam
AParams
Object Detection APsi
APrs
Instance APS)
APrs
Semantic mloUiSS)
BLBU CiDEr
Captioninn
REC Ace
Speeiattrr Models
Detormable DETR
Regressionllend
40M
45.4
64.7
49.0
Misk R-CNN
FPN RPN
46M1
584
Polar
CeneraessHeall
55M
FoemeFe [1:5]
PiseDDrooder
R-CNN
440M
1165
MOGTT
RoBERTi.DFTR
Generolist Model (MuliTTask-Triine Training)
Una Perceiver [95]
None
124M
32,0
Un- Can-Peroiive--ooe [03]
None
167M
33.2
3 M-R50 [14]
Deform-DETR
6
7B
44.6
64.0
48.1
25.1
50.0
22.4
31.0
112.5
80
GIT Beegosut [69]
None
131M
45.1
02.7
49.1
51.4
54.8
31.2
47.7
33.7
107.9
83.3
GiT oa [69]
None
131M
05.7
64.2
59.7
31.9
56.4
31.4
47.8
25.4
112.6
85.3
GiI-Laaaicu [69]
None
387M
51.3
69.2
55.9
35.1
61.4
34.7
50.6
35.7
116.0
88.4
GiT Gil-Haaiiia 1691
None
756M
52.9
71.0
57.8
35.8
62.6
35.6
52.4
36.2
115.2
89.1
URD-VIE-B
Nore
131M
47.8
65.7
52.0
42.6
65.8
46.1
49.5
34.2
111.1
83.6
URD-VII-B.ve.nt
None
131M
48.3
66.6
52.6
43.5
66.2
47.0
50.2
35.3
114.2
85.8
Reprrsoonnnt
+4.5
40.9
44.6
+0.9
40.4
44.9
+6.7
+7.1
43.77
+2.2
1/RO-VT-
None
387M
12.9
71.3
57.9
47.1
70.9
51.6
$4.0
35.9
118.6
88.5
UTD VIT Heathae
None
756M
54.1
724
58.9
48.1
71.6
53.0
58.7
37.6
123.6
89.2
1000IInterVV1.1 5.8
None
80
52.3
71.7
45.8
69.5
49.7
54.6
39,6
131.6
90.4
To enforce the independence of predictions for each grid point, modify the self-attention mask to isolate each grid feature from the others. This ensures that the generation for one point does not influence another. Then we start generating in autoregressive manner, with the difference that we predict M tokens at each forward step instead of just one. The generation process looks like this:
< Local1 >... Localm > < > Th >
T > >
(7)
where TVd denotes the j-th generated token for the i-th grid sequence, and distinguishes the tokens produced in each forward pass. This decoding strategy shares the same philosophy as blockwise prediction [65] in LLMs, accelerating inference by generating multiple tokens simultaneously.
After decoding, we obtain M output sequences. For detection, some might identify objects (Duck, <box>... or Cow, <box>...> .), while others corresponding to empty regions will predict end-of- sequence tokens. For instance segmentation the process is identical, with the textual box representa-
tions (<box>... being replaced by mask tokens (<MASK> .). This approach not only enhances efficiency but also simplifies the problem by breaking down into simple, localized prediction tasks.
Training
To ensure efficient validation and fair comparison, we first follow GIT [69], using a smaller ViT as the multimodal transformer for multi-task training across five standard visual perception tasks. We then scale to MLLMs, validating on the same multi-task benchmark. Finally, we enrich the data by incorporating more diverse datasets, enabling fine-grained instruction tuning of MLLMs. After instruction tuning, the fine-grained perception capabilities are seamlessly integrated with the robust language abilities of MLLMs, thereby applying to perception tasks that require advanced language capabilities,
capabilities, such as reasoning segmentation.
4.1 Multi-Tase Training
Architecture. To ensure fair comparison and validate our effectiveness across various architectures. we conduct multi-task training using two variants: UFO- VIT and Into- VL2.5-8B. UFO-VIT strictly follows GIT [69], employing a SAM (3et-prrtd ViT [21] and a text tokenizer from BERT [19]. It is available in three sizes: ViT-B, ViT-L, and ViT-H. UFO-Intem VL2.5-8B utilizes the pretraining weight of Intern Datasets. We use the same multi-task dataset GiT: COCO 2017 [42] for object detection and instance segmentation, COCO Caption [10] for image captioning the RefCOCO series [48, 83) for referring expression comprehension (REC), and ADE20K [90] for semantic segmentation.
[12], with detailed model specifications provided in Table
7

---

Table 3: Comparison of referring expression comprehension (REC) and segmentation (RES) per- formance. Results on REC are reported based on P@0.5. Results for RES are reported based on cumulative IoU (cloU). denotes the model is specifically finetuned on the dataset.
Reterring Expression Comprehension (REC)
Reterning Expression Segmentation (RES)
Methodk
Refon0o Iss
RefCOnD-
ReiCOCOg
Avg
RefCOC0
ReCEnCO+
0000000g
lestA
test
GLaMMM778 [57]
72.6
78.7
75.6
SAMAMLL M-88 [1]]
79.8
82.7
74.7
74.6
80.0
67.3
75.5
76.4
76.4
HiMTok 8B [23]
$1.1
81.2
79.2
77.1
78.8
71.
75.8
76.7
22.7
PercejticcCiP1-338 [53]
88.6
92.5
84.6
82.1
88.6
74.2
84.1
85.2
K5.0
75.
7885
71.7
68.5
73.9
61.3
70.3
71.7
71.4
VisionLLM x (77)
90.0
93.1
87.1
61.1
$7.3
74.5
55.0
86.4
85.6
79.2
82.3
77.0
68.9
75.8
61.8
73.3
74.8
74.1
MLLMy Tack Decoders
Shirka-7B [7]
Mrs:CPTT<2.78B [6]
91.3
84.3
79.6
85.5
83.8
Ferret-2-711 [85]
92.8
94.7
88.7
87.4
92.8
79.3
$9.4
89.3
89.3
Vi-uLLM-77 [54]
88.1
91.5
85.0
82.9
89.88
74.8
83.6
84.4
84.3
74.5
76.0
72.7
69.1
73.7
64.0
69.0
70,9
71.2
URO-LL-VA-115-78
90.2
93.5
N7.3
84.4
90.3
78.7
56.4
86.8
K7.2
77.2
80.1
76.4
21.8
77.9
70.2
74.1
73.5
75.2
UFO-LLaVA-1.5-20
91.1
93.7
88.6
85.5
90.5
79.9
87.3
87.3
83.0
77.9
81.1
77.0
72.5
78.5
71.4
75.6
74.1
76.0
UF0-ImernVL.255-88
91.8
94.3
N7.5
86.9
91.3
50.6
57.9
88.6
88.6
52.0
81.6
78.1
76.7
79.9
72.3
75.5
76.3
77.6
UPO-ImirrVI.25588B
93.1
94.8
89.2
N7.7
92.1
82.3
58.2
89.3
89.6
51.0
82.6
78.6
77.1
80.4
72.6
76.7
77.3
78.3
Table 4: Results (gloU) on ReasonSeg test set.
Table 5: Results on vision-language benchmarks.
* using reasoning segmentation data in training.
Models
GQA
MMBeach
MMVP
HallBeach
Methods
overall
ReasenSeg quertqqurry
long query
122.mVL2.5-8B UPO.IntemVL...5.8
60.6
84.6
76.3
50.1
60.8
84.2
76.3
50.7
X- X-Deodeer [96]
21.7
20.4
22.2
SEEM [97]
24.3
20.1
25.6
Table 6: Mask token num- Table 7: Ablation of
LISA-7B [37]
36.8
37.6
36.6
LISA-78 [37]*
47.3
40.6
49.4
ber ablation on UFO-ViT- open-ended decoding on
Cona-7B
41.0
Bsingle-task for instance seg- ViT--BBis task for detection.
Cores-78 [3]*
52.4
44.2
55.0
HiMTok-SB [73]
60.8
mentation.
UFO-LLaWA-1.5-78
54.4
41.2
58.5
UFO-LLaVA-1.5-78-
58.8
46.5
62.7
N²
4
16
25
Decoding
Bean
Pontire
Decetton
UFO-Intem
60.0
48.7
63.6
41.3
42.6
9.1
43.0
UFO-IntemVL225-88
67.0
56.2
70.4
FPS
7.0
5.9
3.6
2.8
4.2 Fine-grained Instruction Tuning
Architecture. To demonstrate that method is applicable to various MLLMs, not only VL2.55V [12] but also the L.LVVA-1.--BB [44] for pretraining, specifically UFO-LLaVA-I.5- 7B. Architecture details are in Table 1.
Datasets. To enhance the model's versatility, we enrich the training data to 2.5M across 6 tasks, including VQA data from [39], COCO-Stuff [4], LVIS [25], etc. We additionally add RES task on the basis of five tasks in multi-task training. More details of data composition are in Table 8
5 Experiments
5.1 Experimental Settings
Multi-Task Training Details. To facilitate comparison with specialist models, we also conduct single-task training independently on five selected tasks. For both single-task and multi-task training, we use a batch size of 24 and employ the AdamW [33] optimizer with a cosine annealing schedule, setting the initial learning rate to 0.0002. More details are in the appendix.
Fine-grained Instruction Tuning Details. In training, we use batch size of 32 with gradient accumulation set to 16, running on 8 NVIDIA A100 GPUs for 120K iterations. The Adam [33] optimizer and a cosine annealing schedule are employed, with a learning rate of 0.0002 and weight decay of 0.01. For efficient training, employ LoRA [27] with a rank of 8, freezing the image tokenizer while keeping only the LLM trainable. More details are in Appendix Table 12.
Training Objectives. All tasks utilize CrossEntropy Loss as they are unified under the open-ene open-ended language interface. For segmentation tasks, we additionally apply focal loss [61] and dice loss to supervise the mask output. The final loss for segmentation tasks is expressed as:
Lug ACELCE + Afocal Grocal + >dice
We find that setting all weights to 1 offers better overall performance. See appendix for more details.
8

---

5.2 Multi- Multi-Task Evaluation
We evaluate performance in both single-task and multi-task settings across five vision-centric tasks, benchmarking it against specialized and generalist models. Without task decoders, our model adapts to various tasks by the open-ended language interface and achieves outstanding performance.
Comparison with Specialist Models. As shown in Table 2, our single-task model effectively bridges the performance gap with specialized models, achieving superior performance. For example, we achieve 47.8 mAP in detection compared to 45.4 mAP with Deformable- -DETR [94] and 49.5 mloU in semantic segmentation against 47.2 mloU with Mask2Former [13]. In instance segmentation, we also outperform specialized methods like Mask R-CNN [26] while matching Mask2Former [13]. Comparison with Generalist Models. To facilitate comparison with GiT [69], we adopt its one-stage training without task-specific tuning. This involves jointly training on mixed dataset of the five tasks and directly testing on their respective validation or test sets. Table 2 shows that our model outperforms the previous leading generalist model, GiT, across all tasks, with the same pretraining and data. Notably, in the largest ViT size, we outperform GiT by 12.3 mAP on COCO instance segmentation and 3.3 mloU on ADE20K semantic segmentation, demonstrating the superiority of our segmentation modeling. We also surpass GiT 5.3 CIDEr in captioning, primarily due to our shared vocabulary across all tasks, while GiT uses task-specific vocabularies, hindering the task synergy.
We also observe a multi-task synergy effect like GiT, with performance on instance segmentation improved by 0.9 mAP and captioning increased by 3.1 CIDEr Our multi-task improvements on segmentation also outperform GiT (0.7 vs. 0.1 mloU). We attribute this to unified modeling across segmentation tasks, whereas GiT employs separate methods for instance and semantic segmentation.
After scaling to MLLMs, we observe improved performance on captioning and REC, while other tasks remain comparable to the UFO-ViT-L. We speculate that this performance difference primarily arises from different pretraining. For UTT, we use SAM [34] pretraining, making more aligned with detection and segmentation. In contrast, Intern VL2.5-8B is mainly pre-trained on image-level vision- lssoon-anguage tasks, which better suit captioning and REC.
5.3 Fine-grained Instruction Tuning Results
Visual Grounding can be categorized into referring expression comprehension (REC) and segmen- tation (RES). We comprehensively list the results for the two tasks in Table 3. We report results in two settings: direct evaluation after joint training and specifically finetuning. Without using box decoders, our best model can surpass the VisionLLM v2 [77] by an average of 3.0%. After specific finetuning, our model achieves comparable performance with the state-of-the-ar the--rt method Ferret- v2-7B [85]. While all previous approaches rely on mask decoders polygon approximations for segmentation, our method delivers superior or comparable performance without them. For instance, our Intern 12.55 variant outperforms the SAM4MLLM [11] by an average of 1.9 cloU and matches with HiMTok [73]. These outcomes validate the effectiveness of our method, demonstrating that with proper task modeling, MLLMs can handle fine-grained perception tasks without task decoders.
Reasoning Segmentation (ReasonSeg) is a challenging benchmark introduced by LISA [37], which presents more sophisticated and nuanced instructions, requiring models to leverage world knowledge and engage in deeper logical reasoning. We report both zero-shot and finetuned results. As shown in Table 4, with the same pretraining, our Intern VL2.5 variant outperforms HiMTok [73] by 6.2 gloU in finetuned settings. Notably, both Cores [3] and HiMTok [73] design a CoT strategy to generate segmentation masks progressively: answer the question with text first and then perform segmentation on the answered objects. In contrast, our method achieves better performance without this strategy, which implies that we can effectively perform reasoning while generating mask embeddings.
Since ReasonSeg requires both reasoning and precise segmentation, we attribute our improvement to better task integration through unified modeling. In decoder-based methods, the MLLM handles only language reasoning and generates coarse segmentation prompts, relying on an additional mask decoder for finer segmentation. This leads to information loss and insufficient synergy. In our unified modeling, the MLLM manages both language reasoning and precise segmentation, allowing different task capabilities to fully integrate within a shared parameter space, thereby enhancing synergy.
Visual Question Answering. Table 5 presents the performance on four VQA benchmarks (GQA [28], MMBench [45], MMVP [66], HallusionBench [24]). Thanks to our unification with the language

---

interface, the model's original performance is essentially maintained. Notably, we achieved 0.6 improvement on [24]. which may indicate that fine-grained fine-tuning helps reduce hallucinations.
5.4 Ablation Study
Number of Mask tokens. We ablate the number of mask tokens on the COCO instance segmentation task. As shown in Table 6, using multiple tokens significantly improves performance compared to single token, but the gains plateau after 16. Considering the increased training and inference costs, we set N 16 by default for balance. The visualization of mask tokens is in Appendix Figure 8.
Open-ended decoding. We explore the impact of our open-ended decoding on single-task detection. As noted in Section 3.4, split object detection into sub-tasks for each grid point (c.g.. 625 grid points for a 1 1120x 120 image), which leads to an imbalance between positive and negative samples due to more grid points than objects. Our method utilizes standard vocabulary (e.g.. BERT's s 30.524 tokens) for open-ended decoding. This output space is much bigger than the range of positive classes (e.g., 80 in COCO), worsening class imbalance and reducing positive predictions in inference. As shown in Table 7, while using decoding rules like removing negative classes from the vocabulary [69] could force all outputs be positive, this compromises our generality. Therefore, we use beam search [58], which allows the model to explore multiple potential sequences. This approach effectively increases positive predictions and improves performance. By default, we only apply beam search for COCO detection, instance segmentation, and image captioning.
Conclusion
In this paper, we present UFO, unified approach for various fine-grained visual perception tasks with an open-ended language interface. We translate all perception targets into open-ended text sequences and introduce novel embedding retrieval method for segmentation. Experiments show that our method can achieve excellent performance on MLLMs without requiring architecture modifications. Our unification fully aligns with vision-language tasks, providing a flexible, effective, and scalable solution to enhance the fine-grained perception capabilities of MLLMs, paving the way to build stronger and more general multimodal models.
Acknowledgments
LW is supported by National Science and Technology Major Project (2022ZD01 14902) and National Science Foundation of China (NSFC92470123, NSFC62276005).
References
[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- langun-a model with versatile abilitics. arXiv:2308. /2966, 2023.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[3] Xiaoyi Bao. Siyang Sun, Shuailei Ma, Kecheng Zheng. Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance of reasoning and segmentation. In ECCV, 2024.
[4] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[5] Holger Cacsar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancario Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[6] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. 10.04223 2023.
[7] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal iim's 's referential dialogue magic. MxXi:2206.151955 2023.
10

---

[8] Ting Chen, Saurabh Saxena, Lala Li. David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022.
[9] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoftrey E Hinton. A unified sequence interface for vision tasks, NeurIPS, 2022.
[10] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.0032 2015.
[11] Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu- Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi-modal modal large language model for referring expression segmentation. In ECCV, 2024.
[12] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Yc, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05227, 2024.
[13] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. CVPR, 2022.
[14] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://vicuaa Imsys. org, 2023.
[15] Jaemin Cho. Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- and--angan-eee tasks via text generation. In ICML, 2021.
[16] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[17] Wenliang Dai. Junnan Li, Dongxu Li, Authony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
[18] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. wxXhv2409.17144 2024.
[19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXi::1810.08805 2018.
[20] Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. arXiv:2406. /182, 2024.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[22] Xiao Fu, Wei Yin, Mu Hu. Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision, pages 241-258 Springer, 2024.
[23] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quac V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
[24] Tianrui Guan, Fuxiao Liu. Xiyang Wu, Ruiqi Xian, Zongxia Li. Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, 2024.
[25] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[26] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[27] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhr Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
11

---

[28] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[29] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, 2023.
[30] Aishwarya Kamath, Mannat Singh, Yann LcCun, Gabriel Synnacve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi multi-modal understanding. In ICCV, 2021.
[31] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
[32] Bingxin Ke, Anton Ohukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9492-9502. 2024.
[33] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 2014.
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Ye Yen Lo, al. Segment anything. In ICCV, 2023.
[35] Harold W Kuhn. The hungarian method for the assignment problem. In NRL, 1955.
[36] Alina Kuznetsova, Hassan Rom. Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Puset, Shahab Kamali, Stefan Popov, Mattoo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. In IJV,, 2020.
[37] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In CVPR, 2024.
[38] Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxing Xu. Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Tex44seg: Reimagining image segmentation text generation. In ICLR, 2025.
[39] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024.
[40] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping tunguage-image pre-training with frozen image encoders and large language models. In /CML, 2023.
[41] Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui 2: Mastering universal user interface understanding across platforms. arXiv:2410. 18967, 2024.
[42] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014.
[43] Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[44] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[45] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi- mooal model an all-around player? In ECCV, 2024.
[46] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv:2403.05525 2024.
[47] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO:
A unified model for vision, language, and multi-modal tasks. In ICLR, 2023.
[48] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
[49] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017.
12

---

[50] Dianwen Ng, Yunqi Chen, Biao Tian, Qiang Fu, and Eng Siong Chng.. Convmixer: Feature interactive convolution with curriculum learning for small footprint and noisy far-field keyword spotting. In ICASSP. 2022.
[51] OpenAL. Gpt-4 technical report. 2023.
[52] Zhiliang Peng. Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306. 14824, 2023.
[53] Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into I1m. In CVPR, 2024.
[54] Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag. Ser-Nam Lim, Nicolas Ballas, Qifan Wang- Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Designing general-purpose coarse-to-fine fine vision-language model. In CVPR, 2024.
[55] Xiaojuan Qi, Zhengzhe Liu, Renjie Liao, Philip HS Torr, Raquel Urtasun, and Jiaya Jia. Geonct++: Iterative geometric neural network with edge-aware refinement for joint depth and surface normal estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(2):999-984, 2020.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[57] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024.
[58] Raj Reddy. Speech understanding systems: A summary of results of the five-year research effort at carnegie mellon university. Techical Report, 1977.
[59] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn; Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[60] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024.
[61] T-YLPG Ross and GKHP Dollár. Focal loss for dense object detection. In CVPR, 2017.
[62] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
[63] Nathan Silberman, Derek Hoiem, Pushmect Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[64] Joes Staal, Michael D Abråmoff. Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken. Ridge- based vessel segmentation in color images of the retina. TMI, 2004.
[65] Mitchell Stem, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018
[66] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In CVPR, 2024.
[67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ane Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023.
[68] Haiyang Wang, Yue Fan, Muhammad Ferjad Naeem, Yonggin Xian, Jan Eric Lenssen, Liwei Wang, Federico Tombari, and Bernt Schiele. Tokenformer: Rethinking transformer scaling with tokenized model parameters. arXiv:2410.2316 2024.
[69] Haiyang Wang. Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naeem, Hongsheng Li, Bernt Schiele, and Liwei Wang. Git: Towards generalist vision transformer through universal language interface. In ECCV, 2024.
[70] Junyang Wang. Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2 Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv:2406.0 010144 2024.
13

---

[71] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv:2401.166 161588 2024.
[72] Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-ta- sequence learning framework. In ICML, 2022.
[73] Tao Wang, Changxu Cheng, Lingfeng Wang, Senda Chen, and Wuyue Zhao, Himtok: Learning hicrarchical mask tokens for image segmentation with large multimodal model. arXiv preprint 1xXi:2553313022 2025.
[74] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng. Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv:2305. /1.0005.11.. /155,11711000/ 2023.
[75] Xinlong Wang. Wen Wang, Yuc Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023.
[76] Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards universal visual segmentation with large language model. arXiv preprint arXiv:2411.17600 2024.
[77] Jiannan Wu, Muyan Zhong. Sen Xing. Zeqiang Lai, Zhaoyang Liu, Wenhai Wang. Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In Neur/PS, 2024.
[78] Zhuofan Xia, Dongehen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. CVPR, 2024.
[79] Enze Xie, Peize Sun, Xiange Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In CVPR, 2020.
[80] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Sct-of prompting unleashes extraordinary visual grounding in gpt-4v, arXiv:2310. /44444 2023.
[81] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du. Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. 2rX00:2310.07700, 2023.
[82] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweamgin, Jeffrey Nichols, Yinfci Yang. and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal lims. arXiv:2404.05719, 2024.
[83] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
[84] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi. Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint (xXiv:5501.04000 2025.
[85] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang. Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 2024.
[86] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer22..5 A versatile large vision language model supporting long-contextual input and output. arXiv:2407.03320. 2024.
[87] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-Ilava: Bridging image-level, objectleeel,..... pixel-level arXiv:2406. 19389, 2024.
pixell-ll reasoning and understanding.
[88] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai Groundhog: Grounding large language models to holistic segmentation. In CVPR, 2024.
[89] Zheng Zhang, Yeyao Ma. Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi modal model. In ECCV, 2024.
14

---

[90] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[91] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet+++: A nested u-net architecture for medical image segmentation. In MICCAL, 2018.
[92] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. SEtI:23004.05922 2023.
[93] Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiierr-moe: Learning sparse generalist models with conditional moes. In NeurIPS, 2022.
[94] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
[95] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, 2022.
[96] Xueyan Zou, 7i-Yi Dou, Jianwei Yang. 7he Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, et al. Generalized decoding for pixel, image, and language. CVPR, 2023.
[97] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Lee. Segment everything everywhere once. In NeurlPS, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions of the paper are outlined in bullet points the end of the Introduction section.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We provide failure cases in the Appendix $1 and discuss the corresponding limitations.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer:[Yes)
Justification:
We have elaborated on the training setup in Section and provided detailed specifications of the dataset and training parameters in the Appendix Table 12.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our training data comprised public datasets, which are detailed in Table Training and testing code is provided in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce hopollic/giieeee Codeeubbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental details in Section 5.1 and Table 12.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Calculating statistical significance necessitates multiple training runs of multi- modal large models, incurring high computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list the computational resources used in the Table 12.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: : /neurps.....p// oc cc/ppulic////c/cccccccccccaccctt htttaee/eehhesheeeeelllaeeeee
didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We have reviewed and adhered to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss border impacts in the Appendix $J.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: Our model is tailored for specific low-risk task in detection and segmentation, and its architecture inherits safety mitigations from the base MI.LMs.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the models used in detail in Table 1.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: material.
We have included the code and detailed instructions in the supplementary
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: We only use LLM for paper writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: https:///neurte http////pps.cc/ooeeeeeeeeeeeeeee /LLM) for what should or should not be described.
22

---

A Technical Appendices
Our technical appendices provides detailed information, including implementation details (SB), dataset descriptions ($C), and training specifics ($D). Inference speed and additional ablation studies covered in ŞE and $F, respectively. Experiments work are included in &G and SH. Qualitative results from two training setups and visualizations of multiple mask tokens presented in şI. Broader impacts are discussed in ŞJ.
tasks and discussions with previous
B Implementation Details
masked
unmasked
(a)
Text
Image
It
is <box> 56
(b)
Text
Image
<laabbattbor>>
<Loeildogg-bo->>
Text
Text
Image
Image
<Local>
cat
shox>
<bex>
<Local>
dog
<box>>
Single-prede pagglepreddition
Multi ceati-prediction
Figure 4: Attention mask visualizations. (a) We apply bidirectional attention for image features. (b) For multi-prediction tasks, we mask each subsequence from seeing others.
VIT Architecture. Our VIT architecture follows the same design GiT [69], using the SAM variant of VIT. We also follow GiT to add 6 newly initialized transformer layers upon the original ViT, leading to better performance. For example,
consists of 18 layers.
Grid Generation. As mentioned the multi-task template (Section 3.4), we divide mueti-prediction tasks into sub-tasks, each corresponding to the nearest grid point. The number of grid points is roughly proportional to the image resolution, as shown Table 9. Calculating the loss for all sub-tasks computationally expensive. Therefore, sample grid points and compute the loss presented in Table 9. When sampling, we prioritize positive samples. If the number of positive samples is less than the target size, then sample from the negative samples. During training MLLMs, we adjust the resolution based on the default resolution of the image tokenizer (448² for In2em and 336² for LLaVA-1.55 and modify the grid point configurations accordingly. For the default resolution of the two MILMs, we use 100 grids and sample 40 them.
subset, as
Attention Mask. To accurately model multi-modal features and manage relationships among sub- tasks in multi-prediction tasks, we customize the attention mask based on autoregressive attention. As shown in Figure (a), for tasks with a single prediction, we use autoregressive attention but apply bidirectional attention to the image features to better capture inter-image relationships For concatenate multiple sub-task sequences into
tasks with multiple predictions, during training, a long sequence but use attention masks to prevent different sub-tasks from seeing each other, as illustrated in Figure 4 (b). This approach enables different sub-tasks to be decoded in parallel during inference. Assuming there are M sub-tasks, we first process the <Text Prompt> and <Image> and their key-value (KV) caches. These KV caches are then duplicated M times to create caches for M subsequences. By batching these subsequences together, the model can decode them in parallel, thereby accelerating the inference speed.
Label Assignment. In prlti-prediction tasks, we use the Hungarian algorithm [35] to match sub- tasks with grid points, specifically to associate boxes and masks with grid points. For boxes, the
23

---

matching is based on the distance between the center of each box and the grid points. For masks, we first convert them into box format before matching.
Post Processings. Post-processing involves three steps. First, class names and box coordinates are extracted from the text sequence via pattern matching. Second, the confidence score is derived from the softmax probability of the first token predicted. For example, for the sequence "Duck, <box>...", we use the score associated with the "Duck" token. Finally, NMS applied to filter the predictions.
Coordinate Discretization. We follow GiT [69] to convert continuous coordinates into discrete num- bers within the range [0, rangel. Specifically, we define the range twice the image resolution. For example, for 448 x 448 image, we convert coordinate values into integers within [0, 896].
Data Augmentation. For object detection, instance and semantic segmentation, we use RandomFl SindooFFipp ip and RandomResizedCropp We also use CopyPaste for object detection and instance segmentation. For other tasks, we simply resize the images to the required resolution.
Indicator. In segmentation, we use sigmoid for indicator and then get a mask with 0.5 threshold.
C Dataset Details
Fine-grained Instruction Tuning Datasets. The details of the datasets used for fine- sree-graind fine- tuning are presented in Table For LLaVA-1.55 we utilize the VQA data from the original paper [43], and for InternVL22, we employ the VQA data from LLaVA-OneVir Vision [39]. Due to limited training iterations, we partially sampled some datasets, such as Objects365 and Openlmages, resulting in final training dataset size of approximately 2.5M.
D Training Details
Multi-Task Training Details. Multi-task training setting is in Table 12, For single-task training, we simply reduce the training iterations 120k. For multi-task training Intern VL2.558BB we keep all parameters trainable and reduce the iterations to 400k because of its faster convergence.
Progressive High-resolution Training of MLLMs. In multi-task training, object detection, instance segmentation, and semantic segmentation require predicting a large number of targets, including many small objects, making performance highly sensitive resolution. For the relatively small UFO-ViT, we train directly using high resolution. However, for multi-task training on MILMMs high resolution significantly increases training costs. Therefore, we adopt progressive high-resolution training strategy: first training at a resolution of 4482 for 300k itcrations, then at 8962 for 60k iterations, and finally at 13442 for 40k iterations. We utilize InternVL..52 support high-resolution inputs. As shown in Table 10, increasing the resolution leads to substantial improvements in detection and segmentation performance, even with fewer iterations.
dynamic resolution to
Fine-grained Instruction Tuning Details. Training settings are in Table 12. The training data for the MLLM includes six tasks, each containing multiple datasets. We use sampling probability of 1/6 for REC, RES, Detection, and Instance Segmentation. For VQA and semantic segmentation, we apply sampling probabilities of 1/4 and 1/12, respectively, based on their data volumes. Within each task, sampling is conducted according to the size the dataset. Additionally, inspired by high-resolution training in multi-task training, we first train with a low resolution (c.g.. 4482 for Intern VL2.5) for 90k iterations, and then switch to a high resolution (e.g., 8962 for Intern VL2.5) for 30k iterations. When fine-tuning specific dataset, we maintain the same training setup but train for only 20k iterations.
LoRA Configurations. We use LoRA fine ane-grained instruction tuning. Our trainable parameters include both the LoRA layers and text embeddings. As shown in the Table 13, UFO's LoRA parameter count is comparable to other models and much smaller than that of SAM4MLLM [1]]. Unlike other methods, which also require training extra mask decoder or even the entire LLM (e.g., HiMTok-8B [73], VisionI..M22 [77], VistaLLLM [54]), UFO achieves superior performance with fewer parameters. This highlights our parameter efficiency.
24

---

Table 8: Fine-grained instruction tuning datasets.
Table Resolution, grid number and sample grid number for the five tasks in multi-task training on UFO-ViT. Speed is measured on UFO-ViT-B, single A100 with batch size 1.
Task
LLaVA-vl v1.3-miin65S [43] or
0.7M
VQA
LLa
(39f(1M)
RejCOCO [83], Re/COC0= 1831,
RFC & RES
RefCOCOg [83 RefCCEF [31]
85K
Task
Resolation
Grid
Sample Grid
Speed
Object Detection
Objects365 (22))000), COCO [42].
LVIS [25], nulmmage [5]
0.6M
Object Detection
Instance Seg
1120
1120"
625
250
3,6
Openlmages [36](200K) LVIS J251.
Instance Seg
COCO [42].nabnages 15].
Semantic Seg
225
90
COCOSsuff M.lMiaplll [49]
Image Captioning
221"
Semantic
milmuges [3), (90]
0.3M
REC
224
9.1
Table 10: Performance of progressive high- resolution training on UFO-InterN VL2.5-8.
Table 11: Inference speed MLLMs. Speed is
measured on an A100 GPU with batch size 1.
Resolution
Iters
Detection
Ins Seg
Sem Seg
Task
UFO.IntemVL2.5588
UPO-LLaVA-I.5-7
448²
300k
44.3
37.4
53.9
REC
1.10
0.78
8962
51.7
44.1
54.6
RES ReasonSeg
0.58
0.67
1344
52.3
0.57
0.65
Table 12: Multi-task training and instruction tuning settings.
config
Multi-task (ViT)
Multi-task (MLLM)
Instruction tuning
optimizer
AdamW
AdamW
Adam
learning rate
2e-4
2e-4
2e-4
weight decay
0.05
0.01
0.01
layer-wise Ir decay
0.85
0.85
schedule
cosine
cosine
cosine
gradient norm clip
0.1
1.0
1.0
warmup iters
lk
1k
1k
training iters
640k
400k
90k+30k
batch size
24
24
32
gradient accumulation
16
LoRA rank
8
LoRA alpha
16
LoRA dropout
0.05
LoRA modules
LLMs
drop path
O.I(B), 0.4(L,H)
precision
FP16 24 x V100
BF16 8x A100
BF16
GPUS
8x A100
Table 13: Comparison of trainable parameters when applying LoRA.
Method
Base (M)LLM
LoRA Rank
LoRA Parameters
Text Embedding
Cores- 7B [3]
LLaVA-7BB [43]
20M
262M
GLaMM-7B [57]
Vicuna- 7B [14]
8
20M
262M
SAM4MILL-- [11]
Qwen- VL--BB [1]
256
693M
1.24B
UFO-LLaVA-LL5-7B
LLaVA-1.5-7B [43]
8
20M
262M
UFO-Intem
IntemVL2.5-8BB [12]
19M
758M
25

---

Table 14: Ablation of beam search number on
Table 15: Ablation on COCO detection and
UFO- ViT-B singleesss task.
instance segmentation.
Beam Number
Detection mAP
Instunce Seg
Captioning
Open Fided
Beam Sanch
Copy Paste
Repeal GT
mAP
BLEU-4
CIDEr
Retrieval
45.1
31,4
45.6
40.9
33.0
108.5
43.0
30.4
47.4
42.1
34.2
111.1
45.1
31.5
47.8
42.6
34.0
110.8
45.1
40.4
39.2
5
47.9
42.6
33,9
111.0
Table 16: Loss weight ablation.
Table 17: Ablation of direct box prediction.
CE:Focal
1:1
1:3
1:5
3:1
Method
P@0.5
FPS
Instance Seg
43.5
43.7
43,8
43.4
box mask2box
91.8
1.10
Captioning
35.3
34.9
34.8
35.3
90.5
0.57
E
Inference Speed
Table 9 shows the speed of UFO-ViT-B. By using parallel decoding for multi-prediction tasks, we achieve inference speeds comparable single-prediction tasks, despite higher resolutions (11202 vs. ) and more predictions. Table shows the speed on MLLMs. Our VA-1.5 variant is slower than InterVVL2, on REC because its tokenizer converts textual numbers into longer token sequences. In embedding retrieval, the extra scaled-dot product operation only costs a negligible 0.17 for VL2.5SB on A100.
F
More ablation studies
Beam Search. In Table 14, we present ablation studies on the beam number. As the beam number increases, performance initially improves and then stabilizes, but further increases cause slight performance drop in the captioning. Since larger beam numbers increase inference time, we select the optimal beam number: 3 for object detection and instance segmentation and for captioning.
Loss weights. In Table 16, we ablate loss weights on UFO-ViT-B. We jointly train both instance segmentation and image captioning tasks. Although increasing the focal loss weight is slightly better for segmentation, leads to a drop in captioning. Since our goal is better overall multi-task performance, we all weights to avoid adding task bias.
Advanced training strategies. The sparsity of positive samples in multi multi-prediction tasks hampers effective learning. To mitigate this, samples. First, copy-paste data augmentation [23], where objects from other images are pasted onto the target image. Second, we repeat ground truth & times [29], defaulting to 3. As seen in Table 15, copy-paste boosts mAP by 1.1 for detection and 1.2 for instance segmentation, while repeating ground truth further boosts mAP by 1.6 and 2.2. This demonstrates that the sparsity of is key bottleneck, and our performance can be effectively improved with these strategies. By default, we only use these strategies for COCO detection and instance segmentation.
two advanced strategies increase the ratio of positive
Comparisons with baseline MLLMs. In Table 18, we provide comparisons between UFO and baseline MILMM. Firstly, UFO significantly expands the task range, enabling the model to handle all types of segmentation. Secondly, although InternVL2.5-88 can support REC and perform detection by breaking it into multiple single-category prediction tasks, its performance is markedly inferior to ours, especially in detection. This is primarily because Intern cannot predict multiple boxes for single category nor model the relationships among multiple categories, leading to insufficient and contradictory predictions. In contrast, UFO effectively supports multi- multi-prediction tasks through local prompts, allowing it accommodate any prediction number.
Mask2Box. A simple way to output box based on segmentation mask2box, which uses bounding rectangle of masks. Table 17 shows the performance of mask2box, which is slightly lower than directly predicting boxes. This is mainly because some masks have outlier predictions, distorting the converted boxes. Moreover, boxes are generally shorter than masks, resulting faster speed
26

---

Table 18: Comparisons with bascline MLLMs.
Model
Detection
Instance Seg
Semantic Seg
REC
RES
MMVP
HallBench
VL.255 [12]
12.5
90.3
76.3
50.1
UFO-IntemVL225-8D
52.3
45.8
54.6
93.1
81.0
76.3
50.7
Table 19: DRIVE [64] Segmentation.
Table 20: Depth estimation on NYUv2 Depth [63].
Methods
S-shot
Full fine-tuning
Mcthods
RMSE
51 f
REL1
log10
UNete+ [91]
79.6
ConvMixer [50]
82.2
Painter [75]
0.327
0.930
0.090
GiT-H [69]
57,9
Unified-IO 2 [47]
0.423
JFO-IitemVLL2.5588
UPO-VIT VIT-H
77.4
78.1
82.0
82,4
UFO-Intem 51.2.588B
0.305
0.936
0.087
0.035
(see Table 11). Notably, our box and mask representations are unified through the language interface. The only difference that for boxes, textual numbers are converted into coordinates, and for masks, embedding retrieval is used. These operations task-specific details compared to methods that use task decoders.
simple as mask2box, which greatly reduces
G Extended Experiments
Retinal Vessel Segmentation. Our embedding retrieval method offers superior expressive capability compared to polygons, particularly in highly complex and detailed masks, which require a large number of vertices when using polygons. To further illustrate this, we fine-tune our model on the retinal vessel segmentation, where the vessels possess very irregular and narrow shapes, which are hard to represent as polygons. We follow the few-shot settings in GiT [69], fine-tuning both UFO- ViT-H and UFO-Intem VL2.5-8B on the DRIVE [64] training set for only 100 steps. In performance, UFO-VIT-H achieves 77.4 F1 score, outperforming GiT-H with 57.9 score. UFO-Intern LL255-8B also achieves competitive 78.1 F1 score. After fine-tuning with the entire training set, our performance can surpass strong specialized models such as UNet++ [91] and ConvMixer [50]. As shown in Figure 5, UFO accurately segments the retinal vessels. This result validates the effectiveness of method on extremely fine-grained structures, enabling support for more general segmentation.
Original
UFO-VIT-H
UFO-InterVL22558B
Ground Truth
Figure Visualizations of retinal vessel segmentation. Depth Estimation. Thanks to the flexibility of our method, we can easily extend it to depth estimation similar segmentation. We can apply sigmoid the dot product result interpret relative depth r, which can be then mapped into absolute depth. For depth D within [D D Dmaal we can predict it as follows:
r Ea hy
) D D Dmaz + (1 - r) Dmin
(8)
Vd
Ca is the embedding of <DEPTH> token, As shown in Table 20, we can achieve competitive results.
The above approach essentially shares the same modeling segmentation, differing only on how to interpret the model output. segmentation, dot product results serve as confidence scores that are thresholded to create masks, whereas in depth estimation, they represent relative depth and are then converted to absolute depth. This process can be seen as simple post-processing. which very common in general-purge purepoe models [75, 9]. For example, Painter [75] converts RGB values to
27

---

Table 21: Surface normal prediction on NYUv2 Depth [63]
Method
Mean
Median
11.25
22.5"
30
GeoNet++ [55]
18.5
11.2
9.502
0.732
0.907
Marigold [32]
18.8
0,559
GeoWizard [22]
17.0
0.565
1.00-InemmVL2255-800
17.8
10.4
0.543
0.733
0.800
categories and depth using task-specific rules. Our unification lies in modeling all tasks through the standard language interface. When specific outputs (c.g., masks, depth) needed, the corresponding post-processing is then performed. This design unifies the core understanding capabilities across tasks while requiring only minimal, learning-free ffrenng--reeeeeo post-processing for various formats.
Surface Normal Prediction. Similar to depth estimation, we can extend UFO surface normal prediction. We introduce three task task-specific tokens for normal vectors: <NORMAL_K x>, <NORMAL_ Y>; and <NORMAL_22 z>. Given the image feature h, and the three directional embeddings exs By, and e.. the normal components (e.g., (..)) are computed as follows:
c h
fg =
vd
where (-) denotes the sigmoid function and d is the feature vector dimension. Finally, the predicted values are normalized obtain unit surface normal vector:
(fzg Rly ftg.
ñ
[[(zz Ny. ft=)ll
We conduct the evaluation on NYU v2 Normal Benchmark, and the performance shown in the Table 21. It can be seen that our method achieves comparable performance to specialized models.
H Discussions
Comparisons with GiT. GiT [69] also aims to build a generalist model for fine ane-grained perception tasks. Compared with GIT, provide six key improvements: 1) Segmentation by embedding retrieval, a simple yet intuitive way to support segmentation by language interface. GiT uses polygons and textual classes, leading information loss lengthy sequences. In contrast, UFO accurately segment using only 16 mask tokens. 2) Alignment with the open-ended language interface: unlike GÝT, which requires separate vocabularies and fixed output lengths per task, UFO uses shared vocabulary and outputs arbitrary-lengie sequences. Tables 7 and 18 demonstrate that open-ended detection is challenging due severe class imbalance. We address this issue with text-aligned beam search and achieve enhanced performance. 3) Scalability to MLLMs: while GiT only experiments on relatively small ViTs, UFO can easily scale to larger MLLMS thanks the aligned language interface. 4) Exploring the image representation capabilities of the language interface. GiT is a purely text-based method, while UFO can effectively extract mask information from image features. 5) Better task universality: GiT uses different methods for instance and semantic segmentation (polygon and textual class), while we adopt a unified approach for both tasks because UFO can support masks with any shape. As UFO is aligned with vision-language tasks, we can effortlessly combine VQA reasoning and segmentation, enabling ReasonSeg. 6) Significant performance improvements. In Table 2, UFO-ViT-H outperforms GiT-H by 1.2 mAP and 12.3 mAP COCO detection and instance segmentation, and 3.3 mloU on ADE20K semantic segmentation.
I Visualization
Multi-task Training Results. In Figure 6, we visualize the multi-task training results of UFO-ViT-HE The model can not only handle simple perception tasks also accurately detect and segment multiple objects in complex scenarios.
28

---

Instruction Tuning Results. We present qualitative results of UFO-Intern VL2.5-8B in Figure 7. Leveraging the language capabilities of MLLMs. the model can accurately locate and segment based on both simple phrases and complex queries.
Multiple Mask Tokens. In Figure 8, visualize the masks corresponding each of multiple mask tokens. Each token captures specific details, such as different legs of a horse or the tail of dog. Therefore, combining all the mask tokens results in a higher-resolution, more detailed mask. In Figure 9, we visualize the results for different numbers of mask tokens. Using only one mask token results in rough edges, while increasing the number of mask tokens produces more refined masks, leading to better performance.
Failure cases. We visualize the failure cases of UIOO-Intem VL2.5-8B on REC, RES and ReasonSeg in Figure 10. In the first image, the arm that needs to be localized is very blurry and only appears in small portion at the edge of the image, indicating that the model has deficiencies in localizing objects with low visibility. In the second image, multiple zebras overlap and their patterns are very similar, resulting incorrect segmentation locations, demonstrating the model's shortcomings in segmenting clustered objects. In the third example, the model fails to identify the answer as "river" due to limited knowledge, leading segmentation errors.
Broader Impacts
Our approach achieves compact model design by removing task-specific decoders and integrating diverse fine-grained perception tasks into a unified architecture. This structural efficiency reduces computational demands during training, leading to a decrease in carbon footprint. We do not identify negative social impacts currently.
objects [List COCO Classes).
Detect objects [List COCO Clusses].
Porson, <box>... box-
Kite, <box>...
Chair, <boxp.,. bbbxx
Person, <hox" b/bop
Segment objects in [List of COCO Classes).
Segment objects in [List of COCO Classes).
Sheep, >Mask>..=MASK> -MASK>
<Mask -MASK>
Person, <aikk <ASSK>
Segment categoorie [List ADE20K Classes).
Segment categorics [List of ADE2UK Classes].
Flawer, -Mask>...<MAS <MAK>>
<Mask
Floor, <Maskes -MASS -MAKK>
Pillow; <Mask -AAKK
Please describe this image briefly.
third bottle from the left.
A dog playing with hall the floor.
<box-21 44,245, 198-:bxx
CED
Figure Qualitative results of multi-task training. The first three rows correspond to object detection, instance segmentation, and semantic segmentation, while the last row shows results on captioning and referring expression comprehension.
29

---

Please locate the man holding child in the image.
Pleuse locate the book numed That's not my puppy.
<box--320,47, 402, 274-cbxx....
<box=229, 25445/66ob-
Please segment the man flopping around the right.
Please segiment the cat laying on spotted bed.
Sure, <MASK>.. MASK> SMASSS
Ir is -MSK MASKSS.
Pleuse segment the the objects protect prevent it from getting injured
segment the food containing proceins, carbohydrates and nutrients.
Sure, -MASK>..
-AAK
is <MASK> MASK>
Figure 7: Qualitative results of Fine- emniinained Instruction Tuning. The three rows correspond to REC, RES, and reasoning segmentation in order.
Segment objocts in [List of COCO Classes).
Horse, ~MASK- MASK> -MAKK> <MASK>
<MAK>>
<MAK>>
M
Input Image
<MASK>>
~MASK>
Final Mask
Segment objects in isist COCO Classes).
Dog <MAKK> MASK> <MASK> <MASK>.
<MAK
<AAK
Map
Input Image
<MASK>
<MAK
Final Mask
Figure 8: Visualization of multiple mask tokens. We illustrate with four mask tokens (with 2=2). Employing multiple mask tokens allows for capturing finer details, such the horse leg and the dog tail, resulting in more precise and refined masks.
30

---

Input Image
N
N²=4
N²=16
Figure 9: Segmentation results with different mask token number (N2) on UFO-Vi DIFO-ViT-Biingeee singleakkk
Original
Predicted
Ground Truth
Please locate the blurry arm of person near grill.
Original
Predicted
Ground Truth
Please segment the baby zebra in back of the other two baby zebras.
Original
Predicted
Ground Truth
What the key feature of the rainforest that supports the growth of various types of plants, creating rich habitat for many animals? Please output segmentation mask.
Figure 10: Failure case visualizations of 1120-IInemm REC, RES and ReasonSeg.
31


Reviews:
[{"review_id": "VYH7D8o88P", "reviewer": "Reviewer_nJnP", "summary": "This paper proposes a unified framework for fine-grained perception tasks (like detection and segmentation) by expressing any perception task’s output as open-ended texts. The goal is to remove the reliance on task-specific modules like mask decoders. For detection, the boxes are expressed as textual numbers, while segmentation is formulated as embedding retrieval problem. To do so, a $<$MASK$>$ token is added to the vocabulary to indicate the segmentation task, and its embedding is used as query to retrieve the image features corresponding to the mask token. Also for segmentation, instead of using a single mask token and then upsample by interpolation, the authors propose predicting multiple mask tokens and using each of them to query visual features. Experiments are conducted on multi-task training and fine-grained instruction tuning.", "questions": "* From Fig.1, it’s not clear how detection is different between (b) and (c). Could the authors please explain what is the advantage of predicting box location as text in UFO compared to predicting location tokens as in (b)?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* Casting perception targets as open-ended language sequences is intriguing as an idea to unify visual tasks and integrate more complicated fine-grained tasks like segmentation. This unification is an important step to get closer to the ultimate goal of a generalist model. \n\n* In multi-task training, UFO outperforms both specialist models and previous generalist models, and can seamlessly integrate to MLLMs.\n\n* UFO achieves SOTA results on visual grounding and reasoning segmentation. \n\n\n**Weaknesses**\n\n* Integrating fine-grained perception tasks like detection and segmentation to generalist models under a unified open-ended text generation framework is very interesting, however it doesn't include other visual tasks. For instance, is it possible to integrate under the same framework tasks like estimating scene flow, optical flow or normals?\n\n* The paper lacks a comparison of the methods in terms of the number of trainable parameters when applying LoRA to train UFO. \n\n* While UFO outperforms specialist models, the latter have less parameters, and in practical cases one might prefer using a specialist model for inference speed considerations.\n\nMinor:                           \n\n* Figure 1(a) illustrates VisionLLM v2 while the text mentions LISA (Lines 33-34).\n* Lines 82-83: (3) is not a contribution but a consequence of the proposed method. \n* Table 1 caption: “We construct three variants by this formulation.” $\\rightarrow$ this sentence is unclear\n\nTypos:\n* Line 126: Tabel -> Table\n* Line 245: entrich -> enrich\n* Lines 247: Tabel -> Table", "was_revised": true, "final_justification": "The authors fully addressed my concerns, therefore I raise my score to 5. I encourage the authors to include the normal prediction experiments, as the results look strong compared to state-of-the-art methods like Marigold. I also encourage the authors including the trainable parameters comparison.", "rebuttal": "Dear Reviewer nJnP,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n> [W1] Is it possible to integrate other visual tasks like estimating scene flow, optical flow or normals under the same framework?\n\nThanks for this insightful question regarding the generalizability of UFO. Our framework can naturally extend to more tasks, such as depth estimation and surface normal prediction. The core insight is to **regard embedding retrieval as a general approach for extracting information from image features**. While we focus on mask information extraction in the main paper, the same approach readily applies to other modalities (e.g., depth or surface normals) by introducing corresponding special tokens to interact with image tokens. We have discussed this extension to **depth estimation in Appendix G**, along with performance on NYUv2 Depth. For your convenience, the comparison table is also included below.\n\n| Method             | RMSE ↓    | δ1↑       | REL ↓     |\n| ------------------ | --------- | --------- | --------- |\n| Painter [1]          | 0.327     | 0.930     | 0.090     |\n| Unified-IO 2 [2]       | 0.423     | -         | -         |\n| UFO-InternVL2.5-8B | **0.305** | **0.936** | **0.087** |\n\nFor the **surface normal** prediction task, we can train the model to predict three task-specific tokens:  `<NORMAL_X>`, `<NORMAL_Y>`, and `<NORMAL_Z>`.  Given the image feature $\\mathbf{h}_v$ and three embeddings $\\mathbf{e}_x$, $\\mathbf{e}_y$, and $\\mathbf{e}_z$ for each direction, the normal value (e.g., $\\hat{n}_x$) is computed as follows:\n\n$$\n\\begin{align*}\nn_x &= \\sigma\\left(\\frac{\\mathbf{e}_x^\\top \\mathbf{h}_v}{\\sqrt{d}}\\right), \\hat{n}_x =2\\cdot n_x-1\n\\end{align*}\n$$\n\nwhere $\\sigma(\\cdot)$ denotes the sigmoid function and $d$ is the feature dimension. Finally, the predicted values are normalized to obtain a unit surface normal vector:\n\n$$\n\\mathbf{\\hat{n}} = \\frac{(\\hat{n}_x,\\, \\hat{n}_y,\\, \\hat{n}_z)}{\\left\\|(\\hat{n}_x,\\, \\hat{n}_y,\\, \\hat{n}_z)\\right\\|}\n$$\nWe conduct the evaluation on NYUv2 Normal Benchmark, and the performance is shown in the table below. UFO can achieve comparable performance to specialized models.\n\n| Method             | Mean↓ | Median↓ | 11.25°↑ | 22.5°↑ | 30°↑  |\n| ------------------ | ----- | ------- | ------- | ------ | ----- |\n| GeoNet++ [3]       | 18.5  | 11.2    | 0.502   | 0.732  | 0.807 |\n| Marigold [4]       | 18.8  | -       | 0.559   | -      | -     |\n| GeoWizard [5]      | 17.0  | -       | 0.565   |-    |-      |\n| UFO-InternVL2.5-8B | 17.8  | 10.4    | 0.543   | 0.733  | 0.800 |\n\n\nA possible approach for scene flow and optical flow is to use special tokens to capture motion information from video features. However, as video training is more computationally intensive, we leave this for future work. Extending UFO to videos is highly valuable, and we will highlight this in the future work section and actively explore it!\n\n[1] Painter: Images speak in images: A generalist painter for in-context visual learning. CVPR 2023\n\n[2] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. CVPR 2024\n\n[3] GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation. TPAMI 2020\n\n[4] Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. CVPR 2024\n\n[5] GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image. ECCV 2024\n\n&nbsp;\n\n> [W2] Comparison of trainable parameters when applying LoRA to train UFO.\n\n| Method             | Base (M)LLM    | LoRA Rank | LoRA Parameters | Text Embedding |\n| ------------------ | -------------- | --------- | --------------- | -------------- | \n| Cores-7B           | LLaVA-7B       | 8         | 20M             | 262M           | 641M |\n| GLaMM-7B           | Vicuna-7B      | 8         | 20M             | 262M           | 641M |\n| SAM4MLLM-8B        | Qwen-VL-7B     | 256       | 693M            | 1.24B          | 641M |\n| UFO-LLaVA-1.5-7B   | LLaVA-1.5-7B   | 8         | 20M             | 262M           | /    |\n| UFO-InternVL2.5-8B | InternVL2.5-8B | 8         | 19M             | 758M           | /    |\n\nDuring LoRA training, our trainable parameters include both the LoRA parameters and text embeddings. As shown in the table, UFO’s LoRA parameter count is comparable to other models and much smaller than that of SAM4MLLM. Unlike other methods, which also require training a mask decoder or even the entire LLM (e.g., HiMTok-8B, VisionLLMv2, VistaLLM), UFO achieves **superior performance with fewer trainable parameters**. We will add this comparison to the revision to further emphasize our advantages.\n\n\n\n\n\n> [W3] Although UFO outperforms specialist models, they have fewer parameters and may be preferred for faster inference.\n\nThanks for this insightful comment. The choice between UFO and specialist models depends on scenario requirements.\n\nSpecialist models are ideal for single, well-defined tasks (e.g., closed-set detection) where complex textual understanding isn't needed, as they can offer both strong performance and fast inference. In contrast, generalist models like UFO excel in complex or dynamic scenarios that require handling **multiple tasks** or **advanced reasoning over textual instructions**. As shown in Table 4 of the main paper, specialist models perform poorly on reasoning segmentation tasks compared to MLLM-based approaches.\n\nIn summary, UFO and specialist models are complementary: UFO is suited for scenarios requiring general capabilities and complex reasoning, while specialist models are preferable for straightforward, single-task problems without the need for intricate textual understanding.\n\n\n\n> [W4] Unclear writing and typos.\n\nThanks for these detailed observations! In Table 1, “We construct three variants by this formulation” refers to the three UFO model variants: UFO-ViT, UFO-LLaVA-1.5, and UFO-InternVL2.5-8B, each consisting of an Image Tokenizer, Text Tokenizer, and Multimodal Transformer. We have revised the manuscript to clarify this and correct all ambiguities and typos.\n\n\n\n> [Q1] Advantage of predicting box location as text compared to predicting location tokens.\n\nThanks for this interesting question. The main advantage of representing box coordinates with text is its simplicity and flexibility.\n\n1. Using text representation can avoid adding thousands of new tokens to the text tokenizer, which is easier to implement and better aligned with the original MLLM.\n2. If a finer coordinate granularity is needed (e.g., increasing the discrete intervals from 1,000 to 2,000), text representation allows direct reuse of prior pre-training, while location tokens would need re-initialization, making adaptation less efficient.\n\n***\n\nThanks again for helping us improve the paper and hope our response can resolve your concerns! Please let us know if you have any further questions. We will be actively available until the end of rebuttal period. Looking forward to hearing from you :-) !"}, {"review_id": "4F9myuAHry", "reviewer": "Reviewer_ZRJs", "summary": "The paper presents UFO, a novel framework that integrates fine-grained visual perception tasks such as object detection, segmentation, and image-level vision-language tasks into a single model using an open-ended language interface. The key innovation is the embedding retrieval approach for segmentation tasks, which leverages the language interface to generate masks without task-specific decoders. The authors demonstrate significant improvements over previous state-of-the-art models, particularly in instance and semantic segmentation tasks.", "questions": "See Strengths And Weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper introduces a unified approach to handle diverse fine-grained visual perception tasks through a single model, which is a significant step towards more general and versatile multimodal models. This approach simplifies architectural design and training strategies.\n\n2. The embedding retrieval method for segmentation is effective. It leverages the inherent capabilities of MLLMs to generate precise masks without relying on complex task-specific architectures, leading to improved performance and reduced computational complexity.\n\n3. The proposed method outperforms previous state-of-the-art models on multiple benchmarks, including COCO instance segmentation and ADE20K semantic segmentation. The results demonstrate the effectiveness of the unified approach in achieving high accuracy and robustness.\n\n4. The framework is shown to be scalable and easily integrable with existing MLLMs, such as InternVL2.5 and LLaVA-1.5. This scalability allows the model to benefit from advanced language capabilities, enhancing its performance on tasks that require reasoning and fine-grained perception.\n\nWeaknesses\n\nOne potential concern is the impact of integrating fine-grained perception tasks on the model's general question-answering (Q&A) performance. While the paper demonstrates strong performance on specific tasks, it is unclear how these specialized tasks might conflict with or affect the model's ability to handle broader, more general Q&A tasks.", "was_revised": true, "final_justification": "The authors have addressed my concerns, I keep my rating.", "rebuttal": "Dear Reviewer ZRJs, \n\nMany thanks for recognizing the **novelty, versatility, effectiveness, strong performance and scalability** of UFO. We address your questions as follows.\n\n> [W1] How these specialized tasks might conflict with or affect the model's ability to handle broader, more general Q&A tasks?\n\nThanks for pointing out this important concern. We have included the performance on general QA in Table 5 of the main paper, and we also paste the table here for convenience. \n\n| Models             | GQA  | MMBench | MMVP | HallusionBench |\n| ------------------ | ---- | ------- | ---- | -------------- |\n| InternVL2.5-8B     | 60.6 | 84.6    | 76.3 | 50.1           |\n| UFO-InternVL2.5-8B | 60.8 | 84.2    | 76.3 | 50.7           |\n\nAs shown in the table, UFO achieves performance on par with the baseline MLLM on VQA benchmarks. We attribute this to three key factors: \n1. We employed a relatively small LoRA rank (set to 8), which minimizes modifications with the pretrained MLLM parameters. \n2. Our training corpus contains a substantial proportion (40%) of high-quality VQA data from LLaVA-OneVision [1], helping to preserve and reinforce general QA performance. Additionally, we observed a 0.6-point improvement on HallusionBench, which may indicate that our fine-grained fine-tuning helps reduce hallucinations. \n3. UFO is highly aligned with general QA tasks, both adopting an open-ended language interface. As discussed in Lines 147-149 of the main paper, we argue that since the model can already answer where and what objects are in general QA, the mask information is already encoded in the image features. Our method can decode this information and further enhance it through training, thus essentially sharing the same capabilities as general QA tasks.\n\n[1] LLava-Onevision: Easy visual task transfer. Arxiv 2024.\n\n***\nWe hope the above response can help solve your questions. Thanks again for your thorough review and looking forward to your reply :-) !"}, {"review_id": "M87DoPMOp0", "reviewer": "Reviewer_99r1", "summary": "This paper introduces UFO, a framework designed to unify fine-grained visual perception tasks (object detection, segmentation) and standard vision-language tasks within a single model, using an open-ended language interface. The key problem it addresses is the architectural and training complexity that arises from adding task-specific decoders to Multimodal Large Language Models (MLLMs). UFO's core contributions are twofold: 1) It formulates all perception outputs as text sequences, representing boxes as textual coordinates and, more notably, 2) It introduces a novel \"embedding retrieval\" mechanism for segmentation.", "questions": "1.Considering that UFO's training includes interactions with image tokens, how does this additional supervision impact the model's perception and understanding capabilities? Could further ablation studies be conducted to explore this? For instance, what would happen if a detach operation were applied to the image tokens?\n\n2. Can the embedding retrieval approach also be applied to detection tasks? Specifically, could box embeddings be obtained through interactions with image tokens, similar to traditional anchor-based methods?", "limitations": "See Weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The paper tackles a timely and significant problem: the integration of fine-grained perception into general-purpose MLLMs. The proposed decoder-free approach is a notable departure from prevailing methods that rely on adding complex, task-specific heads , which simplifies the overall architecture and training pipeline.\n2. The \"segmentation by embedding retrieval\" is a particularly clever and intuitive idea. Leveraging the model's own output embeddings to query rich image features is a powerful concept that sidesteps the information loss issues of polygon approximations or the inefficiency of per-pixel text classification.\n3. The experiments are comprehensive, covering a wide range of tasks and datasets, which strengthens the paper's contributions and demonstrates the framework's versatility.\n4. The paper is well-written and structured, making it accessible to a broad audience, including those who may not be specialists in the field.\n\n\nWeaknesses:\n1. About 'open-ended language interface': The paper does not provide a clear definition or explanation of what constitutes an \"open-ended language interface.\" \nAfter all, the current method still requires learning to output implicit tokens and involves additional interactions with image tokens. In my view, methods like SAM4MLLM[1] or SegAgent[6], which only require outputting pure text, might better align with the definition of an \"open-ended language interface.\"\n\n\n2. The paper does not provide sufficient details on the inference and evaluation processes for object detection and instance segmentation tasks. Specifically, how does the method handle multiple objects and categories? How many forward passes are required? Are there any additional post-processing steps? These details are crucial for understanding the practical applicability and scalability of the method.\n\n3. The comparison methods are relatively old and may not fully represent the current state-of-the-art in the field. Taking the RES task as an example, some newer and stronger works have not been included in the comparison, such as:\nGround-V[2], OMG-LLAVA[3], POPEN[4], F-LMM[5], SegAgent[6], HyperSeg[7].\n\n[1]SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation.      ECCV2024\n\n[2] Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels.  CVPR2025\n\n[3] Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding.       Neurips2024\n\n[4]Popen: Preference-based optimization and ensemble for lvlm-based reasoning segmentation.        CVPR2025\n\n[5]F-LMM: Grounding Frozen Large Multimodal Models.    CVPR2025\n\n[6]Segagent: Exploring pixel understanding capabilities in mllms by imitating human annotator trajectories.     CVPR2025\n\n[7]HyperSeg: Hybrid Segmentation Assistant with Fine-grained Visual Perceiver.    CVPR2025", "was_revised": true, "final_justification": "Thank you to the authors for the detailed and thoughtful rebuttal. I appreciate the clear clarifications provided regarding the open-ended language interface, the inference and evaluation process, and the post-processing pipeline. I encourage the authors to incorporate the clarifications provided in the rebuttal into the final version of the paper for greater clarity.\n Based on the improved understanding from the rebuttal, I am raising my score to a 5.", "rebuttal": "Dear Reviewer 99r1,\n\nWe sincerely appreciate your acknowledgment of the **significance, intuition, thorough experiments, and clarity** of our paper. We will respond to your questions one by one.\n\n> [W1] Explanation of open-ended language interface.\n\nThanks for pointing out this important concept. In the paper, we refer to the \"open-ended language interface\" as the capability to **output variable-length text sequences, with termination indicated by the `<END>` token.** This stands in contrast to prior works such as GiT [1], which are limited to fixed-length outputs. \n\nOur method does not conflict with open-ended language interfaces.\n\n1. Although we require fixed-length `<MASK>` tokens for segmentation, we still perform text generation in an open-ended manner. **Only after text generation is complete**, we check whether the output contains `<MASK>` segments of the required length.\n2. The interaction with image features is also carried out only after the text generation is finished.\n\nNotably, after the open-ended text generation ends, **our segmentation mask is determined by the image features and mask embeddings**. In contrast,  SAM4MLLM and SegAgent can only produce coarse prompts by text generation, which requires further refinement by SAM. We will clarify this term in our revision.\n\n\n[1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. ECCV 2024\n\n> [W2] Details on the inference and evaluation processes for object detection and instance segmentation tasks.\n\n> [W2.1]  How does the method handle multiple objects and categories?\n\nWe handle this by the **parallel decoding described in Section 3.4**, which is also illustrated in Figure 3.  During inference,  we sample grid points over the image of size $K\\times K$, resulting in a total of $M = K^2$ points. **Each grid point serves as a local prompt and is responsible for detecting its nearby object**. We interpolate the image features at each grid location to extract the grid features: `grid_feats = torch.nn.functional.grid_sample(image_feats, grid_locations)`\n\nWe then input the text prompt, the image features, and the grid features into the LLM. Take the rightmost image in Figure 3 as an example:\n$$\n\\text{Detect cow, person, duck.}<\\mathrm{Image}><\\mathrm{Grid}_1><\\mathrm{Grid}_2>\\ldots<\\mathrm{Grid}_M>\n$$\n\n`<Grid>` refers to the `<Local>` token in Section 3.4. To ensure independence among $M$ predictions, we adjust the attention mask so that each grid feature is isolated from the others. Then we start generating in an autoregressive manner, with the difference that **we predict $M$ tokens at each forward** instead of just one.\n\n$$\n<\\mathrm{Grid}_1><\\mathrm{Grid}_2>\\ldots<\\mathrm{Grid}_M> | <T^1_1><T^1_2>\\ldots<T^1_M> | <T^2_1><T^2_2>\\ldots<T^2_M> | \\ldots\n$$\n\nwhere $T^j_i$ denotes the $j$-th token in the $i$-th sequence and `|`  distinguishes the tokens of each forward pass. When all $M$ sequences have predicted the `<END>` token, decoding ends. Finally, we obtain M sequences with different objects and categories, which might look like: (1) Duck, `<box>`... (2) Background $\\cdot \\cdot \\cdot$ (M) Cow, `<box>`...\n\nFor instance segmentation, the process remains the same except that the textual boxes are replaced with `<MASK>`  tokens.\n\n\n\n> [W2.2]   How many forward passes are required?\n\nAs explained above, only $1+L$ forward passes are required, where $L$ is the maximum length of the output sequence. Typically, for detection, $L$ is less than 20, and for segmentation, $L$ is less than 30. Thanks to our parallel decoding strategy, the number of forward passes is **independent of the number of objects in the image**, which greatly accelerates inference. For example, in the semantic segmentation, UFO-ViT-B achieves 4.8 FPS, which is 3$\\times$ faster than SAM-B (1.6 FPS). More speed results are in Appendix Table 9.\n\n\n\n> [W2.3]  Are there any additional post-processing steps?\n\nYes, the post-processing mainly consists of the following steps:\n\n1. Pattern matching:  We need to extract class names and box coordinates from text sequences. \n2. Extract the confidence scores. We use the softmax logit of the first token predicted as the confidence score for the prediction. For example, for the sequence \"Duck, `<box>`...\", we use the score associated with the \"Duck\" token.\n3. Finally, we apply NMS to filter boxes and masks.\n\n\n\n> [W3] Some newer and stronger works are not included.\n\nMany thanks for listing these outstanding works. We compare our method with these approaches on the RES and ReasonSeg below. UFO achieves a highly competitive performance on RES, outperforming all methods except HyperSeg on average. On ReasonSeg, UFO achieves the **best** performance, surpassing HyperSeg by **4.3** gIoU on the validation set and Ground-V by **14.9** gIoU on the test set. These results show that UFO unifies reasoning and segmentation more effectively. We will add these comparisons in the revision.\n\n|  Methods  |      | RefCOCO |       |      | RefCOCO+ |       | RefCOCOg |      | Average | ReasonSeg |      |\n| :-------: | :--: | :-----: | :---: | :--: | :------: | :---: | :------: | :--: | :-----: | :-------: | :--: |\n|           | val  |  testA  | testB | val  |  testA   | testB |   val    | test |         |    val    | test |\n| Ground-V  | 83.9 |  85.0   | 82.1  | 73.1 |   75.8   | 69.8  |   74.8   | 74.2 |  77.3   |     -     | 52.1 |\n| OMG-LLaVA | 77.2 |  79.8   | 74.1  | 68.7 |   73.0   | 61.6  |   71.7   | 71.9 |  72.3   |     -     |  -   |\n|   POPEN   | 79.3 |  82.0   | 74.1  | 73.1 |   77.0   | 65.1  |   75.4   | 75.6 |  75.2   |   60.2    |  -   |\n|   F-LMM   | 76.1 |    -    |   -   | 65.2 |    -     |   -   |   68.5   |  -   |    -    |   46.7    | 46.2 |\n| SegAgent  | 79.7 |  81.4   | 76.6  | 72.5 |   75.8   | 66.9  |   75.1   | 75.2 |  75.4   |     -     |  -   |\n| HyperSeg  | 84.8 |  85.7   | 83.4  | 79.0 |   83.5   | 75.2  |   79.4   | 78.9 |  81.2   |   59.2    |  -   |\n|    UFO    | 81.0 |  82.6   | 78.6  | 77.1 |   80.4   | 72.6  |   76.7   | 77.3 |  78.3   |   63.5    | 67.0 |\n\n\n&nbsp;\n> [Q1] Ablation of supervision of image tokens.\n\nThanks for this insightful question. We explore the impact of applying supervision to image tokens on UFO-InternVL2.5-8B:\n\n\n| Supervision | REC      | RES      | ReasonSeg | MMBench  | HallusionBench |\n|:---:|:---:|:----------:|:-----:|:---------:|:---------:|\n|           | 91.6     | 79.2     | 58.8      | **84.4** | 50.3           |\n| ✓         | **91.8** | **80.0** | **60.0**  | 84.2     | **50.7**       |\n\n\n\nAs shown in the table, supervising image tokens can largely improve segmentation while maintaining original capabilities. We attribute this to two reasons:\n\n1. InternVL2.5-8B does not explicitly supervise image tokens during training; instead, it relies on sparse textual signals, which leads to insufficient preservation of fine details (e.g., object boundaries) in the visual features. Therefore, directly using these visual features can lead to degraded segmentation performance.\n2. The supervision in the segmentation provides fine-grained and direct signals that help preserve richer details in the visual features, thereby enhancing their discriminability.\n\n\n\n> [Q2] Can the embedding retrieval approach also be applied to detection tasks? For example, obtain box embeddings through interactions with image tokens.\n\nYes, there are two different implementations:\n\n1. Mask2Box: Convert the mask into a box by extracting its minimum enclosing rectangle. Its performance is slightly lower than predicting textual box. This is mainly because some low-confidence masks have outlier predictions,  leading to distorted boxes. The inference speed is also slower, as the mask sequences are generally longer than box.\n2. Special Token: Predicting box coordinates by interactions with image features.  Specifically, we still decompose the detection task into multiple sub-tasks, each corresponding to a grid point. Each point predicts the textual class and four special tokens (`<X1>`, `<Y1>`, `<X2>`, `<Y2>`) representing the box offsets relative to the point. Each offset (e.g., $\\Delta x_1$) is computed using the scaled dot product between the token embedding  $ e_{X1}$ and the grid feature $h_g$, followed by a sigmoid:\n\n$$\n\\begin{align*}    \\Delta x_1 &= \\sigma\\left(\\frac{e_{X1}^\\top h_g}{\\sqrt{d}}\\right)   \\end{align*}\n$$\n\n\nwhere $ \\sigma(\\cdot)$ is the sigmoid function and $d$  is the dimension.  As grid features are interpolated from image features, this can be seen as extracting box information from the image.\n\nThis approach is faster due to the shorter sequence, but its performance is worse than textual boxes. There may be two reasons: 1. Shorter sequences are less expressive. 2. This approach must store all offset information in a single grid feature, while the original method can predict textual boxes by attending to both image and grid features.\n\n| Method        | AP   | FPS  |\n| :------------ | ---- | ---- |\n| Textual Box   | 47.8 | 4.1  |\n| Mask2Box      | 45.6 | 3.6  |\n| Special Token | 43.7 | 4.5  |\n\n\n\n***\n\nThanks again for your professional, detailed, and valuable reviews! We have done our best to address each of your concerns and hope our response can resolve them. Please let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period. We are looking forward to hearing from you :-)!"}, {"review_id": "17ThHI48XK", "reviewer": "Reviewer_vFSU", "summary": "This paper introduces UFO, a novel framework that unifies various fine-grained visual perception tasks. The core motivation is to better align the representation of such tasks with an open-ended language interface, enabling seamless integration with multimodal large language models (MLLMs) without the need for auxiliary modules. To support segmentation, UFO reformulates the task as an embedding retrieval problem, generating segmentation masks based on the similarity between mask embeddings and visual features. Additionally, the paper proposes a multi-token prediction strategy that effectively refines the masks. Experimental results demonstrate that UFO achieves strong performance across multiple benchmarks, with significant improvements observed on COCO, ADE20K, and ReasonSeg.", "questions": "1.\tIn the segmentation task, the visual features are obtained from the outputs of the LLMs. Have the authors explored using the visual features directly from the Vision Transformer within the MLLM instead? I am interested in understanding how the visual features change before and after being processed by the LLM.\n2.\tIn open-ended decoding tasks like object detection, could the generation of synonyms for COCO categories potentially affect the evaluation metrics? Additionally, what is the method used to compute the detection score for each predicted bounding box?\n3.\tThe paper employs a method that directly predicts discrete text tokens for object detection. I am curious whether an alternative approach—inferring bounding boxes by computing the minimum enclosing rectangles from the predicted masks—would be a feasible and effective solution.", "limitations": "YES", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tA unified framework for visual tasks is essential for the development of multimodal foundation models. This work successfully integrates fine-grained visual perception with vision-language understanding, substantially reducing both architectural and training complexity while leveraging cross-task synergies.\n2.\tThe embedding retrieval approach to segmentation is innovative. It effectively leverages the previously underexploited visual representation capabilities of MLLMs. Additionally, the proposed mask up-sampling method is compelling, allowing for precise mask refinement using only a minimal number of tokens.\n3.\tThe method is thoroughly evaluated across various tasks, demonstrating competitive performance and clear superiority.\nWeaknesses:\nSome technical details and ablations are not sufficiently clear. See questions below.", "was_revised": true, "final_justification": "Thanks for the rebuttal, I am sticking to my original positive rating.", "rebuttal": "Dear Reviewer vFSU,\n\nMany thanks for your recognition of our work. We will address your questions one by one below.\n\n> [Q1] Have the authors explored using the visual features directly from the Vision Transformer within the MLLM instead?\n\nThanks for asking about this important design. We have conducted comprehensive experiments to compare the differences between using ViT features and using visual features from the LLM. As shown in the table below, relying on ViT features results in a significant drop in performance. We attribute this to the following reasons:\n\n1. Compared to ViT features, the visual features output by the LLM are better aligned with the feature space of the mask embeddings.\n2. The LLM-generated visual features are more discriminative. Because we **place the text before the image**,  the LLM can attend to the text prompt when processing the visual features. As a result, LLM can **dynamically emphasize image regions relevant to the prompt**, which helps the mask embeddings better localize the target. Reversing the input order (image before text, second row in the table) will also lead to a performance drop, highlighting the importance of contextualizing visual features with textual information.\n\n| ViT | LLM | Text First |  RES  | ReasonSeg |\n|:---:|:---:|:----------:|:-----:|:---------:|\n|  ✓  |     |            | 74.2  |   55.4    |\n|     |  ✓  |            | 77.9  |   56.7    |\n|     |  ✓  |     ✓      | 80.0  |   60.0    |\n\n\n\n> [Q2]  Could the generation of synonyms for COCO categories potentially affect the evaluation metrics? What is the method used to compute the detection score?\n\nIn the evaluation, we find that all categories predicted by the model are within the 80 classes provided by COCO, and there are no cases of predicting synonyms of the categories. This is mainly because we specify the category names in the text prompt, and the model is able to follow the prompt well.\n\nWe use the softmax logit of the first token predicted in each text sequence as the confidence score for the entire prediction. For example, for the sequence \"Duck, `<box>`...\", we use the score associated with the \"Duck\" token.\n\n> [Q3] Whether inferring bounding boxes by computing the minimum enclosing rectangles from the predicted masks is a feasible and effective solution.\n\nYes, predicting bounding boxes using the `mask2box` approach is feasible, and we have also included this ablation in Appendix F. The performance of UFO-ViT-B on the COCO Detection using this method is presented in the table below. The performance is slightly lower than predicting the textual box. This is mainly because some low-confidence masks have outlier predictions, leading to distorted boxes. The inference speed is also slower, as the mask sequences are generally longer than the box. Therefore, to achieve better accuracy and efficiency, we retain the method of predicting text coordinates for detection tasks.\n\n| Method   | AP | FPS  |\n| -------- | ----- | ---- |\n| Textual Box      | 47.8  | 4.1 |\n| Mask2Box | 45.6 | 3.6 |\n\n***\n\nThanks for your professional comments, we will add these experiments and details to our revision. Feel free to let us know if you have any further questions or concerns :-)."}]

Author rebuttal:

Meta-review: This paper initially received generally positive scores. The authors have submitted a rebuttal, and after considering it, all reviewers expressed satisfaction with the responses and updated their scores to acceptance (score 5). The reviewers also encourage the authors to incorporate the clarifications and experiments provided in the rebuttal into the final version. The AC concurs with the reviewers that this is a solid submission with advantages of strong performance, novel ideas, well-written and unified framework. Therefore, the AC would like to recommend acceptance to this paper.
Decision: Accept (spotlight)