ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension
Authors: Tianren Ma
Lingxi Xie
Yunjie Tian
Boyu Yang
Qixiang Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Learning
Visual Referring
Referring Expression Comprehension
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: ClawMachine, a new methodology that explicitly notates each entity using token collectives—groups of visual tokens that collaboratively represent higher-level semantics.

Abstract: Aligning vision and language concepts at a finer level remains an essential topic of multimodal large language models (MLLMs), particularly for tasks such as referring and grounding. Existing methods, such as *proxy encoding* and *geometry encoding* genres, incorporate additional syntax to encode spatial information, imposing extra burdens when communicating between language with vision modules. In this study, we propose ClawMachine, offering a new methodology that explicitly notates each entity using **token collectives**—groups of visual tokens that collaboratively represent higher-level semantics. A hybrid perception mechanism is also explored to perceive and understand scenes from both discrete and continuous spaces. Our method unifies the prompt and answer of visual referential tasks without using additional syntax. By leveraging a joint vision-language vocabulary, ClawMachine integrates referring and grounding in an auto-regressive manner, demonstrating great potential with scaled up pre-training data. Experiments show that ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency. It also exhibits the potential to integrate multi-source information for complex visual reasoning, which is beyond the capability of many MLLMs. Our code is available at https://github.com/martian422/ClawMachine.

Published as a conference paper at ICLR 2025
CLAWMACHINE: LEARNING TO FETCH VISUAL TOKENS FOR REFERENTIAL COMPREHENSION
Tianren Ma
Lingxi Xie
School EECE
Huawei Inc.
University of Chinese Academy of Sciences matianrent 8 @ma i iss. ucas ac. cn
8.8 88888xxc88800 8xc8gma com
Yunjie Tian
Boyu Yang
School EECE University of Chinese Academy of Sciences tianyunj iel 9@mail ill 8 ucas. ac on
Jiutian Team
China Mobile Research Institute
y yangbooy@chiiane namobi le com
Qixiang Ye | School of BECE University of Chinese Academy of Sciences qxye@ucas an
ABSTRACT
Aligning vision and language concepts at finer level remains an essential topic of multimodal large language models (MLLMs), particularly for tasks such as referring and grounding. Existing methods, such as proxy encoding and geom- etry encoding genres, incorporate additional syntax to encode spatial informa- tion, imposing extra burdens when communicating between language and vision modules. In this study, we propose ClawMachine, offering new methodol- ogy that explicitly notates each entity using token collectives Seteeivvee-gruuuss of visual tokens that collaboratively represent higher-level semantics. A hybrid percep- tion mechanism also explored to perceive and understand scenes from both discrete and continuous spaces. Our method unifies the prompt and answer of visual referential tasks without using additional syntax. By leveraging a joint vision-language vocabulary, ClawMachine integrates referring and grounding in auto-regressive manner, demonstrating great potential with scaled-up pre-training data. Experiments show that ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency. It also exhibits the potential integrate multi-source information for complex visual reasoning, which is beyond the capability of many MILMs. Our code is available at w/https://githai com/ mart 4222rtiiaan////aaaaachh
INTRODUCTION
Large language models (LLMs) (Devlin et al. 2018 Brown et al. 2020 OpenAl 2023
Gao
et al. 2023 Chiang et al. 2023) have opened
era of AI. To further un mmleash its potential. the
researchers proposed MLLMs (Alayrac et al.) 2022 Li et al. 2022 Liu et al. 2023b Li et al.) 2023a
for visual understanding and investigated the multimodal dialogue task to unify visual perception tasks. Recently, these tasks have been upgraded from image-level captioning or question answering
instance-level referring and grounding (Liu et al. 2023b Peng et al. 2023 You et al. 2023 Ma ct al., 2024), urging the MLLMs to align vision and language at finer (e.g.. region or instance) level. These referential tasks require the MLLMs understand users' intention describe referred visual entities and predict the position information of queried objects.
Work done during an internship at CMRL. Corresponding author.

---

Published as a conference paper at ICLR 2025
Proxy Encoding
Geometry Encoding
Token Collectives (ours)
Please describe this region =embee__1> details
Please describe this region [1ayil] (elp)) in detail..
Please provide detailed
description for this region:
Visual referring:
encode enrbeeddingg
location tokens lest
decodde bowes
Here mechine in the given image: Amom kid playing
Visual grounding:
have provided the claw machine'ss cembed 2>
the claw machine (oiva) [[ava] in the middle
and
A conceptual comparison between existing MLLMs and our ClawMachine in notating an object in the image. ClawMaching does not use extra syntax, but directly embeds visual tokens to the natural language, supporting fine-level visual understanding in native mechanism.
Figure I:
Existing methods for referential dialogues can be conceptually categorized into two genres: proxy encoding and geometry encoding. We summarize these works in Table for comparison. Please refer
to Figure for a conceptual demonstration. The proxy encoding methods introduce proxy tokens as
intermediates, which are processed by extra vision modules (e.g., RolAlign (Zhang al. 2023 Ma
2024), RegionCLIP (Chen et al. (023a). SAM (Lai et al., [023] Zhang et al.) [024 Rasheed ct al.
al.
2023). , and GroundingDINO (Tian et al.] (024g) identify objects. For these methods, MLLMs
visual grounding and referring abilities were trained separately, which brings difficulty to the joint
optimization, and imposes constraints on their generalization capacity. The geometry encoding methods adopt an end-to-end solution by bounding objects with coordinate-like attributives. Farly geometry-encoding MI.LMs (Peng et al. 2023 Wang et al.) 2023) report concerns about drawbacks in language ability, they introduced exotic location tokens align with, and the precision of grid-like data annotation was limited. Representative geometry-encodiee
MLLMs used plain text for location expression, but require large-scale grounded data and detailed visual information during alignment training (Chen et al., [0]3c]bb Bai et al., (002). Despite the referential understanding ability, we argue that both types of approach require additional syntax for fine-level vision-language alignment. These MILMS' reliance on extensive instruction-tuning results in complex and burdensome training procedures, and imposes significant constraints their generalization capacity. Please refer to Appendix A.5 for more discussion.
In this study, we propose ClawMachind MLLM with simple-yet-effective design achieve referential understanding without extra syntax. Claw Machine is equipped with a new referential method that notates visual entities explicitly using token collectives groups of visual tokens that collaboratively represent higher- level semantics. This avoids introducing proxy tokens or coordinates, which facilitates not only performance but also efficiency. Besides, ClawMachine introduces a hybrid perception mechanism, which simultaneously utilizes continuous and discrete visual information to facilitate the training procedure.
We conducted extensive experiments to evaluate ClawMachine's multimodal understanding ability. Through designed dual-training that fits our unified format, we feed the model to outperform current models that consume much more than us, while having less ballucination referred objects. When scaling up the pre-training data using GRIT-20M (Peng et al.] (023), the model gains larger improvement with minimal instruction tuning. Without additional codec. it demonstrates the ability to describe and ground multiple objects within one inference, and shows potential on more complex referential understanding tasks. This study reveals that, with proper design and adequate pre-training, pure auto-regressive models can outperform those with large modules and/or using heavy referential instruction-tuning It is even one of the fastest MLLMs performing top-tier referential dialogues (see Table 16 for details). We hope this study can equip MLLMs with native ability and shed light on unifying multiple modalities.
'The name 'ClawMachine' comes from analogy that our model fetches the visual tokens for interpretation,
just like claw machine that grabs prizes and returns them to the player.
2

---

Published as a conference paper at ICLR 2025
V-L mounting extraction
hybrid tokens
Continuous feature:
Discrete visual token: Language token:
Q: In the given image <
can find the person with > A: Here is the...
Visual tokenizer
Multimodal Large Language Model
match
<tokens>
A: Here the person with
CLIP Features
Next-Token prediction
&retrieve
MLP
CuanGuation
Embed
= dimy]
N dimi]
[B.N.dimv)
N. dimu]
:BB.M]
B..M, dim.]
Figure 2: Framework of ClawMachine. When an image (or a region) is referred to, the corresponding visual tokens are directly embedded the natural language. ClawMachine performs next-token prediction, and the output visual tokens are projected back to the image for grounding. represents batchsize, while dimy and dim, denotes the dimension of visual and language embeddings. Embed denotes LLM's embedding layer, and is demonstrated separately for intuitive explanation.
2
METHODOLOGY
2.1 MODEL ARCHITECTURE
Recent MLLMs mostly adopt architectures inspired by LLaVA (Liu et al. 2023a) to comprehend visual information. These models use a multi-layer perceptron to project image features encoded by CLIP directly onto the hidden states (embedding space) of LLM as visual prompts. This approach enables the model to understand fine-grained visual information encoded by CLIP after training, but experience difficulty to directly generate visual tokens. To address this limitation, recent studies (Tin) al. 2023 Team 2024) have expanded the model's vocabulary by incorporating quantized visual features as discrete tokens during training. Our approach takes advantage of both mechanisms, which we term hybrid perception, aiming to equip the model with more flexible visual generation capabilities while maintaining high perceptual accuracy.
To formulate the referential understanding tasks with unified notations, we utilize quantized patch features (visual tokens) alongside the text tokens in MILM's vocabulary and use visual token collectives to notate entities within the images. The overall architecture consists of four components, namely (1) a multimodal tokenizer with hybrid perception, (2) a vision-language mounting operation, (3) multimodal large language model for auto-regressive learning. and (4) region sampler to cluster the generated visual tokens.
Multimodal tokenizer with hybrid perception. For the language part, given sentence S, LLaMA-2's tokenizer (Tovrron et al. 2023) to convert it to a language token sequence L = {4, )=1- For the vision part, given an image
R HxW'C the EVA-CLIP (Sun et al. 2023b) encoder is employed extract N (W/P) x (H/P) features from non-overlapping patches with patch size P. We formulate two passes for the visual signal: the first one provides detailed visual information for scene understanding and object recognition, with the patch features projected into language model's 's embedding space using a 2-layer MLP, resulting in V. = {u;;) The second transforms the features into discrete index, where the patch features are quantized into discrete codes using LaVIT's Vector-Qunnizattin layer (Jin et al. 2023), and result in Vi2 { N Both V. and V.2 will be used as the visual prompt for the MLL.M. Please refer to Figure 2
Vision-language (V-L) mounting. The key element to visual referring and grounding is the repre- sentation referential regions. We propose to use a mounting operation to express with token collectives: We map the bounding box as rectangle to the image lattice and fetch the visual tokens in V. with their center coordinates located within the rectangle. These tokens are sorted according to their center coordinates with raster order form the token collectives, and then inserted to the dialogue correspondingly. The mounting operation can also support other types of referential regions, e.g., free-form mask.
3

---

Published as a conference paper at ICLR 2025
Table A detailed comparison with previous works. Grounded data refers the instruction-tuning data for referential tasks. [G] for geometry-encoding [P] for proxy-encoding. *; SAM used for segmenting scenes and mask decoding. DDETR used for object discovery.
Leene-Lovel Visual Perception
Method
Region-Level Representation
Grounded Data
Encoder
Image Tokens
GPT4Rol (Zhng et
2023
CLIP-H
256
[P] Rol Encoder
Florence-2 LAta0
DaVi--L
[G] Discrete bins
5B+
Shikra Chen al. 20236
CLIP-L/14
256
[G] Text Coordinates
IM+
CIIP-11/14+
[P] Resampler, [G]
hins
1M+
Qwen- VL Bir al 5025
OpenCLIP-G (1.9B)
1024-+256
[G] Text Coordinates
20M+
miniGPT-v2 (Chen al
2025b
EVA (L.0B)
1024-2256
[G] Text Coordinates
20M+
Groundhog [Zhing cl al.
CLIPH4 DINOv2
5764256
[P] Mask Extractor, SAM
GLaMM Rasheed et 81 2023
CLIP-H/14 (0.6B)
256
[P] Rol Encoder, SAM
6M+
Groma et 2024
DINOv2@448pt
1024-256
[P] Rol Encoder, DDETR
20M+
ClawMachine (ours)
EVA-G/14 (L.B)
256x2
Visual Token Collectives
700K
Large language model for auto-regressive generation. With the above mechanism for tokenization and mounting, the learning objectives of both visual referring and grounding unified in an auto- regressive form, so that the MLLM can learn more efficiently by predicting tokens using a simple classification loss. In practice, initialize the LLM from LaVIT-2-75 -B (Jin al. (023) (a model based on LLaMA-2-7B), We provide a detailed discussion in Section 3.4 about the initialization and comparison with common LLaMAs. L and Va get vectorized using LLM's embedding layer, while V. is projected into the same language embedding space. Therefore, both V. and Va function as the visual prompt for the LLM. This hybrid perception mechanism has been evaluated be effective for multimodal understanding with unified vocabulary design.
Region sampler for visual grounding. The model trained to interpret and generate visual tokens. During inference, simple Gaussian Mixture Model can suggest grounding boxes based on the generated token collectives. Since the default linguistic sampling strategy, which decodes logits into tokens, is not optimal for visual tokens, we propose efficient region sampler that incorporates object discovery priors. Details of this method can found in Appendix AA66] This decoding strategy uses the out-of-the-box region proposer, ensuring maximum utilization of the generated tokens without causing information leakage or dispersion during training. Section 3.I we will discuss the intermediate metrics and grounding performance in detail.
2.2 DATA PREPARATION
Data preparation plays crucial role in multimodal model research. Since the use of grounded data is often unrestricted in current MLLMs, evaluating data efficiency and the effectiveness of design can be challenging. During the training process, we made the following initial attempts.
The first model only utilize the object annotations from VG, and the RefCOCO series (including Ref- COCO, RefCOCO+, RefCOCOg: RefCOCO/+/g for short). We denote this model as ClawMachine--
The second model inherits exactly the same design but pre-trained on 10 times larger materials including GRIT-20M (Peng et al. 2023), a highly diverse and enriched text-image dataset with phrase- level annotations. We denote this model as Claw MachineX-7.. The detailed data composition summarized in Appendix. A.2
In the pre-training stage, scene-level captioning data like LLaVA-Pretrain and ShareGPT-4V, region- level captioning data from ReiCOCOg and VG are utilized. For ClawMachineXX we transform GRIT-20M to an image dataset with interleaved region-text captions. The format is described as follows:
Hybrid image tokens: <boi><feats_c>>ee <boi><<eats_ A transformed caption example: <ref> Mother <toi><cre_tokens>>eee and <ref> the little daughter wewt><rre_tokenneeee in <ref> hats bool><ref_tokens>>>>>eeee
and...
4

---

Published as a conference paper at ICLR 2025
In the instruction-tur tonning stage, mainly train ClawMachine to answer two kinds of referential questions, Le., visual referring and grounding, and show that it generalizes to more complex scenarios. The input and output of visual referring are curated into the following format,
User: In the given image <boi><feats_c><et <>><oi> <boi><feats_die _xsettd__ please provide detailed description for this <ref> region <boi><ref <boi><ree__tokees><eoil Assistant: <A detailed description of the region>.
When the input is loaded, >bo>><feats_c><ee <bei><feats_dks seeettssd>>>>>>> substituted with Va. The will be substituted with V. and will be substituted with token collectives extracted by V-L mounting. A trigger token <ref> is placed before the entity, notifying the MLLM that visual tokens will follow. Two special tokens, <boi> and <eoi>, used to wrap these visual tokens. Similarly, the input and output of visual grounding are curated into the following format:
User: In the given image <boi><feats bbi>xfeatssc>>eeii
can you find [object] Assistant: Here is <ref> [object] <bol><rref_token>>een)
where [object] can be substituted with any text description of the object. With the unified next- token prediction objective, we can combine the curated datasets for referring and grounding and train single model for both tasks. We call it the dual training data and will show in Section 3.4 that it benefits the model's performance. Refer to Appendix. A.2 for more data samples.
2.3
IMPLEMENTATION
The training process is partitioned into two stages, supervised by a next-token prediction loss. Please refer to Appendix A.3 for training configuration.
Stage 1: Alignment pre-training. We guide the model in performing basic captioning task this stage. The hybrid visual sequences are used. For ClawMachine-77B mixture of scene-level and region-level captioning data is used. At the end of this stage, we obtain an MILLM that can generate captions based on features of various scopes. For ClawMachine extra 15M non-instruction. tuning training data is used, and the MLLM can generate interlcaved image-text token sequences. The MLP and entire language model is trained while other components are frozen. Note that the MLP projector is priorly initialized. See Section 3.4
Stage 2: Instruction-tuning. We get the model accustomed to general VQA and referential dialogues in this stage. We collect and curate visual instruction tuning data from various sources for this stage. The model's referring and grounding ability is trained simultaneously with the dual curated data, which verified effective to improve the model's grounding performance. The MI. and entire language model is trained while other components frozen. All the experimental results are tested after this stage without task-specific fine-tuning.
EXPERIMENTS
3.1 REFERENTIAL COMPREHENSION
Visual Referring. We first evaluate Claw-aachine on the visual referring task to as- sess its ability of region-level understanding. The prompt for visual referring has the form of Please provide feef_tooens>, where <ref_tokens, is replaced by the visual tokens within the target re- gion explained in the V-L mounting part. Table 2 presents captioning benchmarks, RefCOCOg (Kazemzadeh et al.) 2014) and Visual Genome (Krishna et al. 2016). With scaled-up pre-training data, our model can better capture the token collective's semantics.
detailed description for this <ref> region
results
established region

---

Published as a conference paper at ICLR 2025
Table Results on the visual referring task. ClawMachine demonstrates state-of-the-ar perfor-
mance among recent MLLMs.
RefCOCOg
Visual Genome
Model
METEOR
CIDEr
METEOR
CIDEr
GRiT Wu ct al.
2022
15.2
71.6
17.1
142
Kosmos-2 [Peng
2023
14.1
62.3
GPT4Rol Zhang et
2025
17.6
146.8
Shikra-7B Chen al.
20230
15.2
GLaMM [Rasheed ct
2023
16.2
105.0
19.0
163.9
Osprey-7B (Yuan et 20.24
16.6
108.3
Groma-7B Ma al. 2024)
16.8
107.3
19.0
158.4
Claw.Machine-755 (ours)
17.1
113.4
19.3
168.9
ClawMachineX-778 (ours)
17.4
118.4
19.5
169.7
Visual Grounding. Next, study visual grounding, a.k.a. referring expression comprchension (REC), the counterpart task to visual referring that requires the model to identify the location of an object with language descriptions. By utilizing discrete visual tokens for image encoding, ClawMachine can understand visual content like reading paragraph. Consequently, cross-domain grounding is transformed into token retrieval task in the joint vision-language vocabulary. We use In the given image, can you find [ob as the question. The results are shown in T
Table 3: Results on the visual grounding (REC) task. We report accuracy with the IoU threshold
0.5. *; Groma interprets and selects pre-detected sequences instead of generating grounding tokens. :: OFA, UniTAB and Florence-2 are generalist foundation models.
RefCOCO
RefCOCO+
RefCOCOg
Model
Data
val
test-A
test B
val
test-A
test B
val
test
OFA-L
Wane et al
2022a
79.9
83.7
76.4
68.3
76.0
61.8
67.6
67.6
6M+
UniTAB
Yang et al
2022
88.6
91.1
83.8
81.0
85.4
71.6
84.6
84.7
1.5M+
Florence-2
Xiao
202.3
93.4
95.3
92.0
88.3
92.9
83.6
91.2
91.7
5B+
Shikra-7B hen et al_
87.0
90.6
80.2
81.6
87.4
72.1
82.3
82.2
1M+
MiniGPT-vZ Chen al.
b
88.7
85.3
80.0
85.1
74.5
84.4
84.7
20M+
Qwen- VL-7B (Bai et al. 20233
88.5
92.3
88.6
76.8
85.9
20M+
Ferret-7B You et al. 22233
87.5
91.3
82.5
80.8
87.4
73.1
83.9
84.8
1M+
Groma- 7B
(a et al.
2024
89.5
92.1
86.2
83.9
88.9
78.0
86.4
87.0
20M+
ClwwMachine-7B (ours)
88.6
92.1
84.3
83.2
88.2
76.9
85.7
86.3
700K
ClawMachineX-77 (ours)
89.7
92.5
86.9
84.4
88.9
78.0
86.7
87.1
16M
We the region sampler described in Section 2.1 to convert model's output visual tokens into bounding boxes required by visual grounding benchmarks. In this model, we have also used the dual data composition policy (i.e., by including the training data curated for visual referring) improve the performance. The comparison of Clawine against existing MLLMs for visual grounding is shown in Table 3 ClawMachine reports state-of-the performance among the MLLMs that utilize extensive training data or specifically designed architectures for visual grounding.
Additionally, with region-tex: interleaved GRIT-20M data during pre- pre-training, ClawnMachineX is capable of locating multiple objects and provide description within one inference, which most of current MILLMs cannot perform. Following Groma (Ma al., 22244, we conduct experiments with its benchmark, which focuses on on testing the model's ability to locate multiple, diverse, and variably-sized objects. See Appendix A.7 for details. The results are summarized in Table 4 ClawMMchhneX demonstrates superior ability with its native interleaved token generation ability, and surpasses existing models especially on small objects. The lack of small objects in popular datasets like RefCOCO/+/g is one important reason, besides, as discussed in Section 3.2 ClawMachine can recognize visual concepts in quantized tokens, which provides extra convenience for fetching small objects that occupy few patches in the image.

---

Published as a conference paper at ICLR 2025
Table 4: Results on LVIS-Ground benchmark. ClawMachineX demonstrates superior capability for grounding multiple objects at once.
Table 5: Evaluating intermediate results using the three metrics in equation on RefCOCOg-val.
Model
AR@s
@R@m
AR@1
AR
Training data
precision
recall
ToU
REC
Shikra-7B (Cben et al. 2023c
0.1
3.1
18.5
4,9
Visual Genore
29.1
39.5
36.8
74.7
MiniGPT-vo (2 nen et 2023b
0.3
8.0
41.1
11.4
Re|COCO/+1g
26.2
42.3
37.0
Ferret7BB You
1.6
16.7
51,1
16.8
Visual Genome
33.1
45.5
52.8
85.7
Groma-7B Madt 2024
8.7
35.6
64.3
28.8
GRIT
54.8
86.7
Clw Machine--78 (ours)
20.5
69.2
36.7
GRIT-20M
30.4
42.7
44.5
80.3
Quantitative evaluation of the token collectives. As the intermediate result of visual grounding. the quality of generated tokens heavily impacts the accuracy of grounding. Initially, the MILMM does not always generate complete visual tokens that cover the entire target. We considering several token sets, where Gimage. Gpred. and Gg: denote the set of whole-image tokens, predicted tokens, and ground-truth tokens, respectively. Note that the ground truth extracted from the bounding box, which may contain some background tokens. We then define four metries for precision, recall, and loU:
precision
|gread recall = Gpred O ggt [gtll
1 Gpeed n Ggt
IoU = 19pred O g (19pred Gimagel
[ggt
gred ggrl))
The results of different BlawMachinne variants are summarized in Table revealing several important insights. (1) The precision of retrieved tokens improves the amount and diversity of training data increases. (2) The recall of retrieved tokens is higher when the training data is from the same domain (RefCOCO/+/g), but data from other domains (such Visual Genome) helps fill gaps. This suggests that increasing the diversity of training data is an effective strategy. After scaling with GRIT-20M, the quality of token collectives shows significant improvement. The visualization results in Figure [ further support this observation. As GRIT-20M supports the model in contrastively learning object representations and their features (the associated image tokens in this study), it leads to improvements in IoU and precision. To further investigate the model's performance with controlled object size and IoU thresholds, we also made extended evaluations in Appendix A.4
3.2 FINDINGS
ClawMachine learns visual concepts in quantized tokens. Figure visualizes examples. One can see that the retrieved tokens are strongly correlated to the query. Delving into the output, we find that ClawMachine learns to connect visual words in the joint vocabulary with linguistic concepts, e.g., tennis s ba fall is correlated with the 13635-th visual word and person (in small scale) is correlated with the 1 14781-o 11thh visual word. Particularly, This has also been validated in -rornd evaluations on small objects in Table 4 The basic understanding of individual visual tokens can be extended token collectives, which collaboratively represent higher-level semantics. This makes it much easier for the MILM to perform visual grounding: only needs to fetch the tokens that are most related to [object] from the image.
visualizes visual grounding
Token collectives alleviate referential hallucination. Referential dialogues place higher demands on model's 's ability recognize and localize objects. However, current referential visual instructions can make MLLMS prone to hallucination, as many scenes and details are repeated across different auto-gun metto-generated datasets. To address this, we conducted additional experiments to evaluate the model's referential hallucination. Inspired by POPE (Li et al. 2023b), we curated Ref--Hl test based on GQA (Hudson & Manning, 2019), where the model is asked to verify answers based on referring objects. Details of the curation and testing process are provided in Appendix [A.8] The model can only give correct answers if truly understands the user's referential intent. The results, summarized in T show that the same non- non-tuning setting. Proxy-encoding models underperformed, despite some showing strong results in POPE, which attribute to their customized design for current tasks. model Qwen-V VL showed better overall performance, while Shikra lagged behind, likely due to its multiple passes over datasets during training.
exhibits significantly fewer hallucinations in object references under
7

---

Published as a conference paper at ICLR 2025
cal
person
CD player
cup
curtain
keyboard
book
right
left bird
front
TE ARRO
tennis ball:
ID=13635
person:
ID=14781
Figure 3: Clchine generates visual tokens, projects them to the image lattice (denoted by stars), and predicts the grounded box (denoted by rectangles). The top row shows the ability ground different objects within one image. The bottom row shows visual tokens with the same ID.
Table 6: POPE adversarial result for object hallucination. Tokens indicate the image tokens used for each image. Groundhog uses DINOv2 feature fusion, and traversed much more data.
Method
Tokens
Accuracy
Precision
Recall
Fl
Yes(%)
LLaVA-7B Liu et al.
2023b
256
50.77
50.39
99.87
66.98
99.10
Shikra-7B Chen et 2023c
256
83.10
85.6
79.60
82.49
46.50
Ferret-13B
You et al.
2023
576
82.36
83.60
80.53
82.00
48.18
Groundhog-Tun Zhang at
2024
5764256
86.33
85.93
86.63
86.28
49.60
Claw-Machine-BB (ours)
256x2
85.36
86.39
82.87
84.59
48.10
Table 7: Results on our designed hallucination test with referred objects (Ref-Hal).
Method
Referring Syntax
Accuracy
Precision
Recall
Fl
Yes(%)
Shikra-7B Chen et al.
2023e
Text Coordinates
47.3
48.3
76.0
59.1
78.7
Qwen-VL-7PB (Bai al
2023
Text Coordinates
61.5
58.3
81.0
67.8
Ferret-7B You al. 20255
Rol Features
56.8
54.9
76.0
63.8
GPT4Rol-711 (Zang cl
2023
Rol Features
44.0
45.7
53.1
Groma-7B MM al. 2024
Rol Features
53.9
52.8
ClawMachine-/P1 (ours)
Token Collectives
75.8
72.8
82.4
77.3
56.6
Complex visual comprehension. ClawMachine benefits from two-fold advantages: (i) it has native ability to perform visual referring and grounding so that a question and its answer can contain both image and text elements, and (ii) builds a clear relationship between visual and linguistic tokens so that the same concept can be delivered using either image or text. Integrating these advantages allows to solve complex visual reasoning tasks. Figure shows examples including: a. grounding multiple objects at once. b. image-ent embedded tokens, and c. multi-object referring segmentation, where multiple sets of retrieved tokens converted into instance segmentation results. it also supports d. and free-form and f. region-text interleaved VQA. These are beyond the capability of current methods within one model, and shows Cachnn's potential towards complex and flexible referential dialogues.
where the query to visual grounding contains
8

---

Published as a conference paper at ICLR 2025
In the image, can find all the
Where the this with image?
the image, where is the person and rocky shore standing
There are three petals
Here is the with mask on his face:
Here the and the rocky shore:
bes
on the top.
samp
In the image, can find way follow the instruction the sign?
What the objject <ref> this region
Call this logo brand
Call this logo
20 brand M.
in the image? Describe usage.
Call this logo
brand T.
Following the instruction, you shall turn right. Here way:
What the brand of the car the image? Answer with brand M
The object comic-sty/e saber, used the past by soldiers on harses.
Brand
Figure ClawMachine can solve complex visual reasoning tasks. See the for explanations.
3.3 SCENE-LEVEL PERFORMANCE
In addition to region-level tasks, we further evaluate ClawMachine on conversational VQA benchmark LLaVA Bench (Liu al. 2023b), and model's performance on object hallucination benchmark POPE (ii et al.) (023b)). The LA Bench contains questions about conversations, detailed descriptions and complex reasoning. As shown in Table 8 ClawMachine maintains competitive image understanding and visual chatting abilities. Without high-resolution visual encoders and task-specific fintuning, ClawMachine gains the highest average score comparing with other MILMs doing VQA tasks. We show the adversarial set's 's results POPE as complement to the Ref-Hal results. ClawMachine also demonstrates competitive results without data or feature augmentation.
3.4 DISCUSSION AND ABLATIVE STUDY
Initialization. In exploring recent MILMs with joint vision- -ssoun-language vocabulary Jin al. (2023);
Team (2024), we base our model on LaVIT for several reasons: unified vision-language generation model, LaVIT provides us with an MLLM with extended vision vocabulary and corresponding VQ model, utilizing large amount of data for text and image generation. Notably, LaVIT uses the MLP projector and standalone visual tokenizer weights for multimodal understanding, while has not been trained COCO captioning VQA datasets. So excluded all these weights to restore the MILM for fair initialization. Prior to pre-training, we retrained the projector to facilitate an equitable comparison with other models on scene-level understanding, We initialized the projector from scratch with learning rate of x 10 33 and used the CC-SBU-558K data introduced in LLaVA for training. During this process, disable quantization and only input continuous patch embeddings Ve, while keeping the tokenizer and MLLM frozen.
early endeavor towards
Hybrid Perception. A straightforward choice to use either continuous or discrete image features. However, our findings reveal some interesting insights: As can be seen in Table the quantization loss of discrete image tokens negatively impacts the model's visual perception. Despite this, quantization is pivotal for visual grounding, discrete tokens are easier to retrieve in our next-token prediction framework. We conducted experiments by toggling between continuous and discrete modes. The limitations of using separate modes are overcome by using both V. and Va. We attribute this improvement to the separate optimization the embedding states for the two signals- using MLP for V. and LLM's embedding layer for Va-which collectively enhances the adaptation of the original CLIP features. The experiments are conducted under the default setting with RefCOCO/+/g and VG data.

---

Published as a conference paper at ICLR 2025
Table 8: Results on LLa BLaVA-Bench (COCO).
Table 9: Ablation on image feature settings.
Method
Cony.
Desc:
Avg.
Stage-1
Stage-2
Referring
Grounding
VQAv2
LLaVA 7B
[ii 7B al.
20256
K5.4
68.3
92.1
81.9
cont.. conti.
16.8
73.1
78.4
Kosmos-2
Pong atar
2025
71.7
63.4
74.9
70.0
conti. discr.
16.5
81.1
77.9
Shikra-78
Chea et
20236
80.6
70.7
88.1
79.9
Ferret-BB
You e1
2025
84.4
79.4
96.3
86.7
discr. discr.
14.2
84.3
72.1
Groma-7B 8
MA 21.
2024
82.6
84.0
88.8
85.2
disce
14.4
75.8
72.3
Claw (ours)
concat(conti. discr.
17.1
85.7
78.9
Table 10: Ablation on Model design.
Table 11: Ablation on Encoder design.
Encoder
Syntax
Referring
Grounding
VQAv2
Encoder
Precision
IoU
REC
Continuous
text
14.9
81.0
78.4
EVA-CLIP
33.1
52.8
85.7
Continuous
tokens
15.8
79.1
78.4
EVA-L/14
30.3
51.4
83.9
Hybrid
text
14.9
81.1
78.6
CLIP-H11
31.4
51.9
84.4
Hybrid
tokens
17.1
88.6
78.9
CLIP-L/14
29.5
51.0
83.4
Model Design Ablation. We added more ablation studies in Table 10 under the strict same pre- training and instruction-tuning datasets, i.e., same used by ClawMachine-7B. We remove global discrete tokens in continuous, and did ablation experiments by replacing reference visual token collectives (v. tokens) with text coordinates. Referring results was tested on RefCOCOg using the METEOR score and grounding on RefCOCO-val.
Encoder Ablation. We compared model's performance with different encoders in Table [1 We pick EVA-CLIP as the image encoder for the following reasons: First, it is pre-trained with a mask image modeling procedure, which wre validated enjoying perfect semantic re-construction on visual token collectives. Even at the same size, the visual encoders with MIM pretraining scored better (compare EVA-L (EVAO2) and CLIPP-L). Second, the LaVIT's vanilla quantization layer (a 2-layer MLP with indexing in the codebook), which helps get the index of visual tokens, is tuned to tokenize EVA-CLIP features. We replace it with an ordinary ViT, and observe certain drop in grounding benchmarks. This can also be attributed insufficient training of the quantization layer. For comparison, Shikra, which uses CLIP-L as the encoder, scored 82.3 on the REC score.
CONCLUSION
We present ClawMachine, a multimodal large language model designed to unify various fine-level referential tasks. Central to ClawMachine is the approach of annotating visual entities with corre- sponding tokens, eliminating the need for additional syntax. This method demonstrates the model's ability to grasp high-level semantics through token collectives. Furthermore, ClawMachine excels in complex visual reasoning by seamlessly integrating visual and language tokens within the same sequence. Our research indicates that this unified design, combined with effective pre-training, enables pure auto-regressive models to outperform those with large modules and extensive referential instruction tuning.
Limitation and future work. Vision data is rich in information, but quantizing visual features often results in loss of fine- fine-grained details. We anticipate the development of more advanced encoding mechanisms that minimize information loss when embedding visual data into the joint vocabulary( SaanbbularyTTseeeee Tcchannen al. 2023). Additionally, the probabilistic nature of large language models can lead to unstable and incomplete outputs for visual (okens-for instance. failing to capture the entire grounded target. We envision a unified learning procedure that integrates the LLM with the visual encoder to address this challenge effectively.
ACKNOWLEDGMENTS
The authors thank Yuan Zhang for region sampler exploration, Yuzhong Zhao for technical discussion, and the GPU resources provided by the Jiutian Team of China Mobile Research Institute. We also thank the anonymous reviewers for their helpful comments. This work was supported by National Natural Science Foundation of China (NSFC) under Grant 62225208, 62450046 and CAS Project for Young Scientists in Basic Research under Grant No. YSBR-117.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Position- Enhanced Visual Instruction Tuning for Multimodal Large Language Models. arXiv preprint arXiv:2308. 13437, August 2023a.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large language model as unified interface for Dision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023b.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint arXiv:2306.15 15195, July 2023c.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. arXiv preprint arXiv:2311.12793, 2023d.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1it::1810000805. 2018.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng. Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
Zheng Ge, Songtao Liu, Feng Wang. Zeming Li, and Jian Sun. YOLOX: Exceeding YOLO Series in 2021. arXiv preprint arXiv:2107 0084000000....0.. 2021.
Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A Dataset for Large Vocabulary Instance Segmentation. arXiv preprint 1rXiv:1908.03195, 2019.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 558-567, 2019.
Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6693-6702. IEEE, 2019. doi: 10. 2011 00686.
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.0446 2023.
Sahar Kazemzadeh. Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
11

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv preprint arXiv:2304.02643, April 2023.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hormung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint MtXiv:2312.14125. 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. arXiv preprint arXiv:1602.07333. February 2016.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia LISA: Reasoning Segmentation via Large Language Model. arXiv preprint Ttl.::308.000999 August 2023.
Sofia Lemons, Carlos Linares López, Robert Holte, and Wheeler Ruml. Beam Search: Faster and Monotonic. arXiv preprint arXiv:2204.029) 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv preprint arXiv:2301.12551 May 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. arXiv preprint arX5s 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.0374 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual Instruction Tuning. arXiv preprint arXiv:2304. 08445, April 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In The International Conference on Learning Representations, 2019.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi, Unified-1O 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. arXiv preprint arXiv:2312.17172, 2023.
Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404.13013 2024.
OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 March 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World, arXiv preprint arXiv:2306.1455 14824, July 2023.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356 2023.
Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing Objects with Self-Supervised Transformers and no Labels. arXiv preprint 11.iv:2109.142799 2021.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv preprint DXXiv2446.006555. 2024.
12

---

Published as a conference paper at ICLR 2025
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint EXXi:23312.13225 2023a.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15386 2023b.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023c.
Chamelcon Team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint arXiv:2405.0965 00. 2024.
Yunjie Tian, Tianren Ma, Lingxi Xie, Jihao Qiu, Xi Tang. Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint arXiv:2401.13301 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1399 13971, 2023.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. GIVT: Generative Mnfnite-Vocabule Transformers. arXiv preprint arXiv:2312.02116 2023.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple peuuencc-to-Sqquene Learning Framework. arXiv preprint mtXiv:2202.03052 2022a.
Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang. Qingyun Li, Lewei Lu, Xizhou Zhu, Yu Qiao, and Jifeng Dai. The All-Seeing Project V2: Towards General Relation Comprehension of the Open World. arXiv preprint arXiv:2402. 19474, 2024.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiamnan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.1 11 11175, 2023.
Yangtao Wang. Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut. In 2022 EEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14523-14533. IEEE, June 2022b. ISBN 978- 1-66--6--566666 doi: 10. 10..V99992852888...... 202.00011...........
Jallian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. GRiT: A Generative Region-to-text Transformer for Object Understanding. arXiv preprint arXiv:2212.00280, 2022.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. arXiv 2023.
preprint arXiv:2311.0661
Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the Power of Referential Comprehension for Multi-modas LLMs. arXiv preprint arXiv:2310.00552. October 2023.
Zhengyuan Yang, 7he Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling. arXiv preprint arXiv:2111. 12085, 2022.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfci Yang. Ferret: Refer and Ground Anything Anywhere at Any Granularity. arXiv preprint arXiv:23 10.07704, October 2023.
Lijun Yu, José Lezama, Nitesh B Gundavarapu. Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint 23X0022310.05737, 2023.
13

---

Published as a conference paper at ICLR 2025
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10032. 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From Recognition to Cognition: Vi- sual Commonsense Reasoning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 67166673771111 6724. IEEE, 2019. doi: 10.1109/CVPR.2019.0000
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. GPT4Rol: Instruction Tuning Large Language Model on Region-of-Interst. arXiv preprint arXiv:2307003601, July 2023.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai GROUNDHOG: Grounding Large Language Models to Holistic Segmentation. arXiv preprint arXiv.2402.16846, 2024.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. arXiv preprint ar arXiv:2211.12266 2022.
A
APPENDIX
We provide the related work, some additional explanation and details in the Appendix section.
A.1 RELATED WORK
Multimodal large language model. The community is witnessing a trend towards unifying the vision and language modalities using multimodal large language models (MLLMs) (Alayrac et al.
2022 Li et al., 2023a; Liu et al. 2023b). Early endeavors like Flamingo adapted LLMs to visual tasks by internal cross-attention design (Alayrac al. 2022), while BLIP-2 (Li et al. 2023a) introduced Q-former as an external block for vision-language alignment. Later works like I.L.VA (Liu al.) 2023b a) applied effective projectors for alignment pre-training, providing ViT-MLP-LLM architecture for the community alongside the visual instruction-tuna tuning method. Recently, a stream of
works (Kondratyuk et al., [0233] Lu et al. 2023) unified more modalities like video and audio into the model's 's vocabulary with advanced autoencoder designs YYu et al. 2023), paving promising path for large multimodal models.
Referential notation. Referential notations are important for MILMs to align text and image modalities at a finer level. methods that utilized proxy encoding (Zhang et al.) 2023 Yuan et al.) 2024| Chen et al.) 2023a Rasheed et al.) (023), the model leverages the referential proxy signal for agents and incorporated a
module to encode region-features. Powerful foundation models like SAM (Kirillov et al. 2023) are also utilized to perform object discovery (Rasheed et al.) 2023 Zhang et al., 2024). When asked to ground visual objects, the model produces an intermediate embedding supervised by detection
segmentation decoders (Lai et al. 2023 Tian al.) 2024).
Another research line applies the geomelry encoding that utilizes discrete tokens to annotate the boundary of instance and train the MILM in end-to-end manner, providing a unified method for
the model to recognize and generate instance positions (Chen et al. 2023c Xuan et al. 2023 Peng
al. 2023 You al. 2023 Xiao al. 2023).
Tokenization of vision. Aligning visual tokens within the language space has gained significant attention in recent research. Using CLIP features of image patches, the Emu series (Sun al. 2023c a) combined visual generation tasks with multimodal comprehension, while LaVIT (Jin et al. 2023) introduced an extended visual vocabulary simplify training objectives. More recently. auto-regressive generation with directly quantized image patches (Team) 2024 Sun et al.) 2024) has sparked a new trend in the community. Although these early alignment methods are computationally intensive to replicate, they provide valuable foundation models for experimentation. In this work, aim to further strengthen the spatial correlation within the multimodal embedding space.
A.2 DATA COMPOSITION DETAILS
We provide data usage details in this section.
14

---

Published as a conference paper at ICLR 2025
During pre-training, the data is transformed into simple concatenation of visual signals and corresponding description with no instruction-tuning style templates. For scene-level captioning, the tokens of entire image are used, while only the cropped tokens from the V-L mounting operation are employed for region-level captioning. Some images of GRIT-20M are no longer available from the Internet, and we do simple filtration by removing images that focus celebrities and pure text images, so the final size of available data of GRIT is about 15M.
During fine-tuning, we mix the general VQA datasets with referential dialogues. We only reserve the COCO-VQA and GQA subset of LLaVA-mix665k, as the text-related subsets are of severe hallucination and do not help with general understanding tasks (model performs better with cleaned data on VQAv2 bench). Besides classical referential instructions, we collect some useful subsets from existing and widely used datasets. Osprey makes short annotations from original RefCOCO datasets more vivid and detailed. AS-V2 provides more flexible referring and grounding conversations, which helps to interleave vision and language tokens. Specifically, scene-graph like data are utilized for describing and grounding multiple objects at one inference with aligned output style. The claimed 700K grounded data refers to the sum-up of {ReCCOCO///g Visual Genome, Osprey: detailed, VS- scene graph, 12: conversation, and Chatterbox: CoQ). Please see Table 12 for more information.
Effectiveness of dual data. We study the impact of using different instruction tuning data in Stage 2, and evaluate the model's visual grounding performance. We compare the contribution of the plain data (the dialogue data without region-level questions, sourced from LLLVA-mix6655. (Liu et al. 2023a)) and dual data (the visual referring part), by only adding either of them apart from the grounding data, and find that the latter brings about 1.7% gain in REC average score (although the former is also useful in maintaining the model's VQA ability). This validates that Claw Machine can absorb the knowledge from various kinds of referential dialogues, which mainly owes to its formulation that unifies referring and grounding into the next-token prediction task.
Table 12: Data composition details.
Training Stage|
Data
Description and Usage
Image-Text Pairs
LLaVA-Pretrais (Liu et al.
2023b
Scene-Level Caption
558k
ShareGPT-4V CChen 20250
Scene-Level Caption
100k
Pre-training
RefCOCO/+g Kuzemzadeh ct 0
2014
Region-Level Caption
287k
Visual Genome (Kishma el al 2016
Region-Level Caption
247k
GRIT-20M (Peng al.
Interleaved Caption (for ClawMachine
15M (filterd)
LLaVA-mix665k (Liu el
2023h
General VQA
436k (filtered)
Re/COCO/+gg (Kzzmzzan
2014
Visual Referring& Groundiig
287k
Visual Genome Krishna el al.. 2
Visual Seferring&Grundiing
247k
Fine-tuning
Osprey: Detailed (Tuan 2021
Referring Details on RefCOCO
63k
AS-V2: Scene Graph (Wang al.
2024
Multiple Grounding on RefCOCO
42k
AS-V2: Conversation (Wang ci
2024
Refer&Ground convs. on RefCOCO
22k
Chatterbox: CoQ (Tian et al. 2024
Logical Chain of Grounding
40k
A.3 TRAINING DETAILS
We use the Adam W Loshchilov & Hutter 2019) optimizer with the cosine annealing scheduler (e et al. 2019) to adjust the learning rate. The initial learning rate is set to 2 x 10-5 and x 10 10-3 for the two stages respectively with a warm-up ratio of 0.03. The global batch size remains constant at 256. We freeze the tokenizer and train the projector and LLM in both stages. The training is conducted on 8 NVNIIA A A0 GPUs with 80GB memory. The FlashAttention-22 and DeepSpeed libraries with zero2 are employed for efficient training. The imput image size is set to 224 x 224 with a patch size P = 14, and the maximum sequence length in the MLLM is 2048. The codebook of the VQ process 16384, while the language tokenizer has 32000, resulting in the MI.LM with 48384 vocabulary size. The training datasets are combined into a single dataloader using the V-L. mounting operation. The image-text pairs are randomly selected during training and are only traversed for one epoch in each training stage. The training takes about 8 hours for each stage of Claw and the pre-training of Claw/Machine--777 for all the components that are tuned. The tokenization layer of visual tokens is frozen throughout the process, while the corresponding hidden state is trained within LLM's embedding layer. The P Projector alignment is stated in Section 3.4 for a fair comparison. Note that our model's training
takes about 3.5 days. We provide a configuration list
15

---

Published as a conference paper at ICLR 2025
Vedium
90
Acog90.8
Acogh0.77 7.70
(REFC)
60
40
RetCOCO+1g
+ Visual Genome
Visual Genome, GRIT-20M
Figure 5: Model's detailed grounding performance with controlled IoU thresholds and object sizes. Note: amall(3.66)): S E (0, 0.08), medium(49. %%): 5 (0.08, 0.25), large(27.3%)) S E (0.25, 1)
strategy, which is addressed as visual instruction tuning LLiu et al. 2023b is widely utilized by current state-of-the multimodal language models.
Table 13: A comparison of major dataset (> 1000 pairs) usage among popular models.
Method
Re1CO0O/+1g
VG
FlickršOK
Object365
GRIT-20M
Others
Shikra
Chen et
20236
VCR
Zellers al
2019
GLaMM (Rasheed et 2023
SA-18 18
Kinllow et ml
2023
Owen
Bai et ml
I house Data
Ferret You atall 2025
LV1S
Gupta ot 2019
Groundhog (Zhang
VCR, GQA
2024
ShareGPT-
ClawMachine (ours)
X
X
Pre-train
ShareGPT-VV
Chen
Table 14: Training Configuration.
Training Stage
MLP Projector
LLM
Data usage
Strategy
CC-SBU-558S Standard VA-VV-rrrraain
Ir 1x 10 epoch
Projector alignment
Training
Frozen
1.2M/16M caption data
x 10-6 epoch
Pre-training
Training
Training
Denoted Table 12
700k instruction-tune tuning
ir 10 eepoc
Fine-tuning
Training
Training
Denoted in Table [2
A.4 FURTHER EVALUATION ON GROUNDING METRICS.
An initial exploration of size-based evaluation in LVIS-Ground is presented in Table 4 conduct further evaluation here with controlled loU thresholds to highlight the model's performance dif- ferences under incremental data settings. We calculate the relative area S by multiplying object's relative w and (between and 1). To ensure the consistency with the REC metric, the x of Acc@x here is calculated using the IoU of final grounding results and ground truth. All the results evaluated on RefCOCOg-val
As shown in figure 5] pre traininnn GRIT-20M led to consistent improvements in performance across IoU scales, with particularly significant advantages at higher loU thresholds. The model also demonstrated enhanced detection capabilities for objects of varying sizes.
16

---

Published as a conference paper at ICLR 2025
A.5 AN INTUITIVE EXPLANATION ABOUT REFERENTIAL COMPREHENSION.
For existing methods, the L.M learns to understand the reference expressed by newly introduced token or embeddings. Specifically, for geometry-encoding methods, the grounding ability is represented by its learning of discrete location token numerical coordinates. This brings high demand on image's resolution and feature details for the alignment. As can been seen in the comparison Table while the grounded data demand is high for pre-training, most geometry-encodiin methods also employ nigher-resolution encoders to guarantee the perception details.
For proxy-encoding methods, the representation of referred objects is based vision priors like Rol embedding. However, while the embedding may be updated to align with the language space, its relationship with the original vision features not optimized. This makes the proxy embedding task-specific and limits MILM's generalization capacity. Moreover, the grounding embedding weaved by the hidden stated of last output tokens makes it technically hard to localize multiple objects during inference, let alone the decoder-specific design on rare foundation models like SAM and GroundingDINO. Groma tackles this issue by pre- pre-detect all objects an image and guiding the model to select the correct responses, although this increases the tuning complexity and latency.
Claw:Machine enhances referential tasks by integrating language and vision more deeply within an auto-regressive architecture. Using a joint vocabulary, it represents references through token collectives, facilitating stronger fusion of language and vision in the embedding space. This approach helps align finer concepts and allows the model to learn high-level semanties across multiple visual tokens, which has been roughly encoded by CLIP yet not utilized by existing MLLMs. Additionally, this hybrid perception design promotes joint optimization, improving the overall performance in referential comprehension tasks.
Geometry encoding
Proxy encoding
Token collectives (ours)
Vision
Vision
Language
Vision
Language
Proxy Embeds
Language
Location
Attr.
Figure 6: conceptual illustration about referential MLLMs. See text for explanation.
A.6 REGION SAMPLER DESIGN
We explain how our grounding method works and provide some details for decoding visual tokens into bounding boxes in this section.
Retrieving Tokens in Image. Let Gimage represent the group image tokens, Gpred the group of generated tokens. We get Gpredrimage Gimage (9pred as the primary material for further processing. For each element v, E G Gpredrimage: as its index in the visual tokenizer's codebook is determined, we can easily retrieve original place with Va = (N As the tokenizer has limited vocabulary and may not assign every patch of the image different code, we firstly retrieve visual tokens that appear only once in Gumage- Then for V that appears more than once in Gimage. we scan all of its possible origins and append the index that is nearest to the already picked tokens to minimize extra noise. After this procedure, we can get a retrieval map for model's 's output visual tokens, which just looks like the stars canvas illustrated in Figure
17

---

Published as a conference paper at ICLR 2025
Region Sampler with Detection Priors. As the default linguistic sampling strategy that decodes logits into tokens are not optimal for visual tokens, we propose an efficient region sampler with object dis- covery priors, which is similar to the modern beam search (Lemons) al. 2022) algorithm utilized by LLMs:
Where is the person?
First, a switchable region proposer is deployed for a primary object detection. For given image, the proposer provide prediction boxes with a score higher than 0.3 by default, and we use them as the region proposals. We do not give any clue like class or description. See Table 13 for for our experiments. Hit-Rate refers to the chance that the proposer can discover the object described by the dataset with IoU > 0.5. Some annotations are quite difficult even for human in RefCOCO series, and for the rest of the objects, the discovery results are close.
Figure 7: A figure that is diffi- cult for proposers. As no pro- posal is valid, ClawMachine will output the convex box for the decoded tokens.
Second, with the logits of generated token collectives, we conduct a fuzzy search after the the retrieval. For tokens that do not have explicit match in the image grid, their logits are decoded correspond- ingly if the top-3 index fall into the region proposals' lattice.
Only the proposals that already have decoded tokens will be considered in the fuzzy search. If no tokens have fell into the proposals' lattice with overlap > 0.1, we skip the nomination process and use a simple convex box for the decoded tokens as the result which is effective for small or hindered objects.
Finally, for each proposed box, we initialize a 2D Gaussian distribution based on it. Suppose the gaussian distribution's 's variance shall be inversely proportional box's size, while the covariance matrix of the distribution follows the shape of the rectangle. We use these pre-defined distributions to construct simple Gaussian Mixture Model (GMM), and predict the points' intention. This will result in prediction label for each point that decides which distribution it belongs. After using the prediction results and points' density to score each box, we nominate the box with highest scores as model's ' prediction results. For multiple object detection, as each generated token collectives is bounded with their object name in "<ref> object <boi> <ref_tokens <eoi>", this process is executed sequentially with no overlap.
The efficiency of MILMs has often been overlooked in recent referential studies. However, we believe it remains crucial indicator of model's capabilities. Geometry- encoding methods typically employ an end-to-end training procedure, where processing detailed visual
Primary Latency Test.
at higher resolutions accounts for most of the inference time. In contrast, proxy-encoding methods approach referential dialogues in an agent-like manner, utilizing LLMs to weave embeddings while relying on large foundation models peripheral modules for downstream decoding. This coupling introduces higher latency during both training and inference.
In our approach, the proposer used in region sampler also contributes to processing time during inference. So we conducted an average latency comparison with the released versions of these models (note that Groundhog not yet released), as shown in Table An esimation of FLOPstFo Point Operations) is also conducted to evaluate model's efficiency, as shown in Table 17 With the switchable proposer design, ClawMachine offers users flexibility in choosing between precision and efficiency. The latency was tested and averaged using the default evaluation scripts provided by the developers (where available), or we used similar questions to those posed to Claw
A.7 TESTING ON LVIS-GROUN
The LVIS-Ground benchmark introduced by Groma (Ma et al. 2024) focuses grounding multiple objects in the image. They randomly sample at most 5 images for each object category from the LVIS (Gupta et al. 2019) validation set to construct LVIS-Ground. The AS-MANY-Protocol is followed for evaluation: this protocol selects the top-k predicted boxes (where k is the number of ground-truth boxes) and measures recall over all ground-trutt -ruth boxes. For example, if there are 3 out of srouuddtruthh boxes hit by the top-5 predicted boxes, the recall is 60%. More details can be seen in the main manuscript paper.
18

---

Published as a conference paper at ICLR 2025
Table 15: Ablation on Region Sampler's is design choice. The REC score is tested on RefCOCO-val with ClawMachineeX. Hit-Rate refers to the chance that the proposer can discover the object described by the dataset with loU 0.5. NOTE: The previous best is 89.5 reported by Groma (ECCV'24), which adopts large DDETR-like detector pre-trained on SA-1B 1B to extract all the objects in scene, taking more than 500ms for each round detection.
Proposer
Hate (%)
Latency (ms)
REC score
Co-DETR(Swin-L) Zong et al. 2022
92.5
217
89.7
Co-DETR(R-500 (Zong et 2022)
92.1
165
89.1
YOLOX-X (Ge et al. 2021)
89.5
17
88.7
Table 16: A inference latency comparison with other models, The precision is set to bfloat16. Note that our model does incorporate any detection modules during training. The inference time of ClawMachine and its X version keeps the same. *: with Co-DETR or YOLOX-X as the proposer.
Model
Image tokens
Modules
Referring (ms)
Grounding (ms)
Ferret-7B
576
Spatial Resampler
378
431
GLaMM-7B
256
SAM DDETR
449
639
Groma-7B
576+256
733
757
ClawMachine--7B 256x2
Region Sampler
309
577 377*
Table 17: A FLOPs estimation of different methods. The precision is set to bfloat16. The computation cost of ClawMaching and its X version keeps the same. *; with Co-DETR YOLOX-X as the proposer.
Model
Vision Encoding
Language Decoding
additional modules
Thtal (THLLOPs)
Kosmos-2 (IB)
0.1
0.1 (-300 tokens)
-0,2
Shikra-7B
0.1
0.6 (-300 tokens)
-0,7
Ferret-7B
0.15
1.4 (-650 tokens)
0.03 (Resampler)
-1.6
GLaMM-7B
0.2
1.0 (-400 tokens)
1.0 (SAM)
-2.2
Groma- 7B
0.4
0.8 (-350 tokens)
0.8T (DDETR)
-2.0
Claw-Machine-7B
0.27
1.1 (-550 tokens)
0.4 / 0.1 (Region Sampler)
~1.8 1.5
image <image can you boy flying
the image tokens> find the cell phone?
RefCOCO/+/g
VVisual Genome
+RITT-20M
RefCOCO/+/g
VVisual Genome
+GRIT-20M
Figure 8: The diversity of training data improves the accuracy of retrieved visual tokens.
In each inference ClawMachine, it outputs a scene-graph like sentence describing all the instances in the image with grounding token collectives. For multiple objects with same category, it knows to use <boi> <eoi> to wrap up each objects' predicted tokens, just like Kosmos-2 (Peng et al. 2023) with GRIT uses the similar syntax to ground multiple objects. We first map all the output grounding tokens into boxes using the region sampler. the image, for each category in the ground-truie
caleulate whether Claw-achine mentions it, and the corresponding boxes will be used to calculate the recall. The average recall is calculated over 10 loU thresholds (ranging from 0.5 to 0.95) the primary metric on LVIS-Ground.
19

---

Published as a conference paper at ICLR 2025
A.8 REF-HAL: EVALUATING REFERENTIAL HALLUCINATION
Inspired by POPE Li et al., 2023b), we curate a simple Ref-Hal test dataset based on GQA (Hudson & Manning 2019), asking the model make verification based referring objects. Here is the details about Ref- Hal:
Table 18: A simplified demonstration of GQA composition. The data types that do not exist GQA is denoted with gray cells. Only the types with checkmarks v are sampled.
Compare
Logical
Verity
Query
Choose
Category
Attribute
x
v
x
X
Relation
V
X
X
Object
x
X
X
Global
Our primary intention for Ref-Hal is to evaluate model's hallucination on referred objects, while avoid overlapping with existing benchmarks on complex referential understanding abilities. The GQA serves as an ideal start point, as it focuses on forming strong bonds among objects in VG images. We make an summary in Table 18 As shown, the composition of GQA can be roughly categorized with two dimensions: semantic (vertical) and operation (horizontal). The logical, query, and choose operation is firstly excluded for the following reasons: logical relies on knowledge rather than recognition of objects, and overlaps with verify on question styles; query has no referred objects in the question; choose focus on the detailed attributes of the single object, which can be evaluated with general visual referring benchmarks. verify-object is excluded, too, as it asks the model whether something is in the image, much like the POPE for hallucination tests. The remaining three sets have the following format:
Compare-Attribun "Is the <obj_l> SAME <attr.> <obj_2>? " to verify the attributes of two referred objects. Examples: Do the mouse and the curtain have the same color? Is the bag made of the same material as the door frame?
Verify-Atrübute: "Is the <obj> <attr>> adj.?" ?" to verify certain attribute of the referred object. Example: Does the shirt has green color? Are the bleachers that are not long made wood?
Peiif-Reelation "Is the <obj_ 1> <relation> <obj 2>?" to verify the relationship of two referred objects. Example: Is the person wearing a hat? Is the car behind a motorcycle?
We extract 221, 334, and 445 Q&A pairs from GQA's testdev-balanced set correspondingly (same as the original ratio of these three types in the testdev-balanced set), and yes no = 1 1. We utilize GPT-4V replace one object in half of the questions with another object that appears in the same image, assuring that the answer is reversed (the replacement choice is random like POPE random set, but not replacing with something that not in the image, which makes any reference invalid). Then, we transform the object notations in the question with ob ject in the referential style. After this modification, the question "Is the car behind a motorcycles e is transformed into "Is the object- [R,] behind
object-2 [R2] ?". Where [R,] I is the referential notation. As GQA do not explicitly annotate the objects with coordinates, this substitution used the original ground-truth annotation of VG to ensure correspondence and precision.
With the curated Ref-Hal benchmark, we test model's hallucination on referential objects. The results are shown in Table 7 The model needs to recognize the referred object in the question before answering: e.g., "ob ject 11.113, D. 224, 0 355, 0. 0989 for geometry-encoding models, "ob IRoI embedding]] for proxy-encoding models, and "object [token collectives, " for Claw The model only needs to answer yes or no, and the eval- uation result is summarized in Table 7 For fair comparison, Claw is not trained any subsets of this benchmark.
A.9 MORE VISUALIZATION EXAMPLES
We provide more visualization examples of ClawiMachin'' 's output in this part. As the resulted points serve Jask-agnostic supervision intermediate for downstream tasks, also prompt
20

---

Published as a conference paper at ICLR 2025
person
vase
'19'
surfboard
tennis racket
knife
couch behind
pizza on the right
elephant the left
motorcycle behind
left car
horse on left
right suitcase
man on right with back to us
left woman in purple
top right cup
Figure 9: More visualization examples of Mlchine The bottom two rows show some segmenta- tion results we get from SAM using points' clustered center as the prompt.
for powerful pre-trained segmentation models (Kirillov al. 2023). The points' convex closure can be regarded as a semantic segmentation with rough granularity, which is finely post-processed by segmentation specialist models. We show some examples Figure 9 However, using multiple points as a weak supervision is of limited support from SAM and other segmentation models, related segmenting works like LOST (Siméoni et al. 2021) and TokenCut (Wang et al., (202bb) are still under development. We hope to address this problem with sampler with finer granularity and free-form support in the future.
21


Reviews:
[{"review_id": "ToRUQJQwtt", "reviewer": "Reviewer_8JFN", "summary": "The paper proposed a hybrid way to incorporate continuous features and discrete tokens in MLLM for referring and grounding. Different from using coordinate location to represent regions, this work introduced discrete image tokens in both input and output. A systemic workflow is proposed to accommodate the change. In the experimental result, the proposed model can outperform previous works in multiple tasks and shows good qualitative analysis.", "questions": "Please see the weaknesses. Will consider revising the review if the author(s) can solve my concern in rebuttal.\n\n\n===========================\n\nAfter reading the rebuttal of the author(s), I'd like to raise my rating as my concerns are alleviated.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of token collectives enables referring and grounding in MLLM in a hybrid way, unifying both continuous region features and discrete region tokens.\n\n2. Instead of generating coordinates as in previous works, this paper proposed to generate discrete tokens for corresponding regions, and then do a patch-based matching and retrieval to ground back to the region in the image. Also, a delicately designed resampler is introduced to ensure the accuracy of grounding boxes.\n\n3. The performance is impressive, outperforming many existing works in a wide range of tasks.", "weaknesses": "1. It's a bit unclear whether the good performance comes from data or model design. I.e., the proposed model design needs to be ablated and compared with other baselines in the same training and dataset setup as used in this paper. For example, there are a few necessary ablations I could think of: replacing the referring LaVIT token with discrete coordinates; replacing the grounding LaVIT token to be predicted with discrete coordinates, remove the LaVIT token for the global image. Please make sure the other settings are the same (using the same pre-training and instruction-tuning datasets as in the paper, the same training stages and steps, same backbone).\n\n2. Efficiency and Complexity: The proposed method requires two image encoders: LaVIT for discrete token generation and Eva-CLIP for continuous token. Also in decoding, the proposed method requires a detector to provide prior in the Region Sampler. Essentially, other existing works don't require that many additional components. A detailed comparison in terms of efficiency analysis (total model parameter, FLOPs, memory) with and without detector/LaVIT is needed.\n\n3. An advantage of hybrid representation is about free-form referring and grounding(segmentation). However, this paper doesn't benchmark that quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPZJT5Ee8r", "reviewer": "Reviewer_gC1c", "summary": "This paper proposes a novel approach for aligning vision and language concepts in multimodal large language models. Unlike other works in the literature that use additional syntax to encode spatial information, this approach introduces a new strategy for representing entities using token collectives—groups of visual tokens that represent entities at a semantic level", "questions": "- The paper uses an out-of-the-box object detector, as mentioned in line 205 and Table 12. Since this is also a pre-trained external model, what is the advantage of this in comparison with other detector modules in 'proxy encoding' and 'geometry encoding' methods?\n\n- As stated in line 821: '...We do not give it any clue like class or description...' Is it correct that the region sampler can propose an arbitrary number of boxes (as long as the confidence score is greater than 0.3), and that all tokens within these boxes are extracted to form the 'token collective' set?\n\n- In Table 1, the proposed method uses another image encoder, EVA-G/14 (1.0), compared to previous methods. Since the proposed strategy for extracting 'Token Collectives' does not depend on the choice of image encoder, how would the method's performance be affected if it used the same encoder as in previous works?\n\n- In Table 6, why does training on RefCOCO/+ and RefCOCOg lead to better recall and IoU in the Token Collectives generation step when compared with Visual Genome, yet result in a lower REC score? Additionally, what are the results when training only on GRIT-20M? (rather than on RefCOCO+GRIT-20M)\n\n- In line 188, the paper states that tokens are sorted with primary sorting from top to bottom and secondary sorting from left to right. Will the results be impacted if random shuffling or another sorting strategy is applied?\n\n- Is there any experiment conducted for \"free-form masks\" rather than bounding boxes, as mentioned in line 190?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea of representing entities using the introduced concept of a 'token collective' is novel and interesting, as it eliminates the need to train a separate object detector in 'proxy encoding' approaches, allowing for end-to-end joint optimization. Additionally, it removes the difficulty in aligning language with exotic location tokens in the other 'geometry encoding' methods.\n\n2. The proposed solution is simple yet effective. Extensive experiments across several datasets and settings are conducted to show the approach's effectiveness. The paper also provides adequate ablation studies to analyze the model's behavior. The qualitative results presented in Figures 3 and 7 provide insights and illustrate the proposed approach's ability to generate relevant visual tokens.", "weaknesses": "While the method itself is not very complex, the presentation in the paper is difficult to follow and understand the details. The two most important figures (Figures 1 and 2) are not referenced anywhere in the main text. The core section of the paper, Section 3.1, contains very limited information. Although additional explanations are provided in Appendix A.4, relying solely on textual explanations makes some aspects ambiguous.\n\nPlease also see further concerns in the question section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUa6FgSiAw", "reviewer": "Reviewer_E6or", "summary": "The paper proposes ClawMachine, a new method to train MLLMs for referring and grounding. The method allows a fine-grained alignment of vision and language in large transformers. It introduces “token collective”, which are groups of visual tokens representing regions of images. ClawMachine uses text-vision interleaved data and simple next token prediction to train MLLMs. ClawMachine can then perform several tasks, such as referring and grounding, together with other complex tasks necessitating a fine vision-language understanding.", "questions": "* What are the performances of the model after pre-training but without instruction tuning?\n\n* Table 6 shows that adding the GRIT-20M dataset as pre-training data leads to large improvements for the proposed precision / recall / IoU. However this does not lead to such a gap in REC performances. Have the authors interpretation on this?\n\n* In future works the authors mention not relying on quantized tokens. Would the type of approach described in GIVT [2] be an interesting way of doing so?\n\n[1] Xiao, Bin, et al. \"Florence-2: Advancing a unified representation for a variety of vision tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Tschannen, Michael, Cian Eastwood, and Fabian Mentzer. \"Givt: Generative infinite-vocabulary transformers.\" European Conference on Computer Vision. Springer, Cham, 2025.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* ClawMachine does not require additional syntax to embed the localisation. \n\n* ClawMachine can be trained with the simple next token prediction task and instruction tuning.\n\n* ClawMachine has strong experimental results in referring and grounding.\n\n* The paper claims that models and data will be made available.", "weaknesses": "* The section `Methodology` is very succinct and lacks some motivations:\n    * Using both continuous and discrete tokens (only motivated in experiments)\n    * Training details are tedious as ClawMachine leverages multiple existing elements. It would make it easier to follow to have a detailed table indicating the parts that are used from previous works / and which are fine-tuned or frozen.\n    * The visual token generation is not clear to me, is the supervision the patch index, or the vocabulary index?\n\n* ClawMachine is a supervised method. ClawMachine relies on datasets such as RefCOCOg etc. which are supervised datasets. The question remains of how scalable this method is. \n\n* The paper does not include the recent Florence-2 [1]; which can perform both referring and grounding. It also has much fewer parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBtiPZvKMc", "reviewer": "Reviewer_3BfC", "summary": "This paper proposes to use token collectives for representing visual concepts in MLLMs, which avoids learning proxy or geometric tokens. As a result, the visual referring and grounding tasks can be prompted and answered in a unified form. The extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of utilizing token collectives for object referring and grounding is innovative and may inspire future research on integrating multiple modalities.\n2. The experimental results show the effectiveness of the proposed method.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The input image is encoded to 16*16=256 patches, leading to a large number of tokens for referring or grounding an object. For grounding an object, geometry-encoding methods and proxy-encoding methods only one or a few tokens, while ClawMachine needs to output hundreds of tokens (for a large object), which largely increase the inference time cost. However, Table 13 does not appear to reflect this discrepancy in grounding time. Could you provide more details on how the inference time is measured and if there are any techniques used to maintain efficiency despite the potential for increased token output?\n2. The Region Sampler relies on a pre-trained detector to obtain grounding results, which may hinder overall model performance if the detector fails to identify the target object. This reliance on detection could limit scalability compared to one-stage methods such as Ferret and GLaMM.\n3. As stated in the Limitation section, the vector quantization of visual features results in a loss of fine-grained details, as well as a larger vocabulary usage than previous methods. This also poses challenges on aligning linguistic and visual tokens to some extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)