Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
Authors: Qihua Dong
Kuo Yang
Lin Ju
Handong Zhao
Yitian Zhang
Yizhou Wang
Huimin Zeng
Jianglin Lu
Yun Fu
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
Referring Expression Comprehensions
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Referring Expression Comprehension (REC) links language to region level visual
perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have
progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reason-
ing demand; (ii) images often contain few distractors, making the target easy to
find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine
text understanding and visual reasoning. We introduce Ref-Adv, a modern REC
benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The
dataset contains various expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and
descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning
beyond simple cues, and we evaluate a broad suite of contemporary multimodal
LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and
gaps in visual reasoning and grounding. We provide an in depth failure analysis
and aim for Ref-Adv to guide future work on visual reasoning and grounding in
MLLMs.
The dataset is available at \url{https://ref-adv.github.io/}.

Published as a conference paper at ICLR 2026
REF-ADV: EXPLORING MLLM VISUAL REASONING IN REFERRING EXPRESSION TASKS
Qihua Dong
Kuo Yang
Lin Ju
Handong Zhao
Yitian Zhang Yizhou Wang
Huimin Zeng
Jianglin Lu Yun Fu
Northeastern University
ABSTRACT
Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, ReiCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual rea- soning and grounding: (i) many expressions are very short, leaving little reason- ing demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shorteuts by pairing linguistically nontrivial expres- sions with only the information necessary to uniquely identify the target. The dataset contains referring expressions real images, curated with hard distrac- tors and annotated with reasoning facets including negation. We conduct compre- hensive ablations (word order perturbations and descriptor deletion sufficiency) show that solving Ref-Adv requires reasoning beyond simple cues, and we evalu- ate broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Ady, revealing reliance on shortcuts and gaps visual reasoning and ground- ing. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs. The dataset is available https://ref adv. github. 1o/
INTRODUCTION
Referring expression comprehension (REC) is the task of grounding natural language expression to specific region in an image (Mao et al. 2016 Kazemzadeh et al.) 2014 Yu et al. 2016). It has important applications in real world systems and downstream tasks, and has become key bench- mark for evaluating multimodal large language models (MLLMs) because it probes fine grained correspondence between language and vision. Recent MLLMs (Google [202a Bai et al. 2025 Zhu et al. 2025), both closed source and open source, have made substantial progress, achieving over 90% accuracy classic REC benchmarks, i.e., RefCOCO(+/g) (Kazemzadeh et al. 2014 Yu
ct al. 2016 Mao et al. 2016).
Despite this near saturated performance, we identify critical limitations of the classic REC bench- marks that motivate a modern benchmark capable of more challenging and comprehensive evalua- tion of MLLMs. We view modern REC for MLLMs components: (1) textual reasoning understanding the referring expression. identifying the target, and identifying its descriptors; and (2) visual reasoning searching for candidates and establishing correspondence between descriptors and image regions. The order of these steps can vary across models, but meaningful benchmark should require both textual and visual reasoning. From this perspective, highlight the following limitations of RefCOCO(+/g).
multistep reasoning task with two coupled
First, most of the referring expressions are extremely short, as shown in Figure I For RefCOCO and RefCOCO+, the average expression length is around 3 words. Such short expressions lead to two issues: (1) minimal linguistic effort is required, and (2) they typically entail less visual reasoning because fewer descriptors must be verified in the image. Second, there are few distractors in the images in RefCOCO(+/g). as shown in Figure 2 (b), with most cases of only distractors. Here we define distractor object of the same category as the target but different instance. When few

---

Published as a conference paper at ICLR 2026
"pizza"
Find "a DELL mouse connected to the laptop"
Find man in the background jumping to return the tennis ball"
User
Only one 'mouse", need to know the rest.
don't but see one jumping.
background;
(answer without reasoning)
MLLM
(a) Expression Too short?
(b) Few distractors in image? (c) Expression too descriptive?
Figure 1: Common limitations classic referring expression benchmarks that reduce the reasoning challenge. These include very short expressions, few visual distractors, and overspecified descriptors that enable shortcut matching without requiring genuine reasoning. The cyan box highlights the ground truth region.
distractors exist, the task requires far less textual and visual reasoning: models need only infer the target category and select from a small set of candidates, Figure (b) reveals negative correlation between the number of distractors and model performance.
It is worth noting that for reasoning assessment, task difficultyy does not monotonically increase with referring expression length due "grounding shortcuts". These shorteuts occur when long, descriptive expression is paired with few distractors, rendering many descriptors redundant. Consequently, model can localize the target by matching only a subset of descriptors, which can paradoxically lead to higher accuracy for longer expressions, as illustrated in Figure 2 (a). This highlights the need for modern REC benchmarks to mitigate such shortcuts by designing expressions that are concise and carefully balanced against the available distractors.
Meanwhile, prior work has acknowledged aspects of these limitations: Wei et al. (022)) Chen et al. 2024) point out the length limitations of ReiCOCO(+/g) and Chen et al. 2020) highlights the lack of distractors. However, the proposed datasets also raise new concerns. The former introduces REC data with average length 90 words, which may be unnatural and, more importantly, enable numerous shortcuts since the numbers of descriptors and distractors are heavily imbalanced. The latter proposes settings including referring from set of images, which shifts away from the classic REC setting, and the referring expressions are sampled from GQA (Hudson & Manning. 2019) scene graphs with fixed templates, reducing naturalness.
We therefore aim to build a REC benchmark that preserves the classic REC setting and natural expressions while substantially increasing the reasoning challenge aligned with the capabilities of modern LLMs. To this end, we introduce Ref-Ady, modern REC benchmark that avoids short rea- soning paths and imposes both reasoning and grounding challenges on contemporary MLLMs. To validate and ensure the quality of the benchmark, we conduct comprehensive in depth ablation stud- ies in section to explore what makes a rigorous modern REC benchmark and compare reasoning and grounding difficulty with RefCOCO(+/g) Lastly, in section 3 evaluate 13 contemporary MLLMs on Ref-Adv, both closed source and open source. We report changes in performance with and without Chain-of-Though (CoT) (Wei et al. (022) and provide in depth analyses. We believe these results demonstrate the value of Ref-Adv, offer new insights into the capabilities of currett MLLMs, and help guide future research visual reasoning and REC tasks.
THE REF-ADV DATASET
2.1 DATA SOURCE
We sample from the validation and test splits of COCO (Lin et al. 2014) and OpenImages v7 Kuznetsova al. 2020). We filter the images and only those with panoptic instance anno-
2

---

Published as a conference paper at ICLR 2026
1.00
Accuracy@00
Accuracye0
0.93
0.50
RefcaCD+
Refcacog
RefCOCO
0.90
RefCOCO+
0.75
02
RefCOCOG
0.70
Nurbber of Words in Expression
Number Distractors
(a)
(b)
Figure 2: Accuracy @0.5 (loU 2 0.5) of Qwen on the RefCOCO/+/g validation sets. Marker size proportional to the number of samples each bin. (a) is the Acc@0.5 on number of words in expressions, (b) is distractor count. We can see most cases have short expressions and few distractors.
Table 1: Basic statistics of the validation+test sets of RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv (Ours). The instance size is represented by its square root. Avg. length: average length of annotations. Vocab.: vocabulary size. Avg. distractors: average number of same category distractors per image. Negation ratio: percentage of expressions using explicit negation.
Benchmark
Images
Instances
Avg. Length
Avg. Distractors
Negation Ratio
Instance Size
Vocab.
RefCOCO 2014
3,000
7,596
3.6
3.99
0.99%
105-607
3,525
RefCOCO# 2016
3,000
7,578
3.6
3.96
3.36%
105-607
4,387
RefCOCOg 2016
3,900
7,596
1.41%
83-610
5,050
Ref-Adv (Ours)
2,833
5,000
11.5
4.01
21.25%
30-607
5,308
tations, since this is important for our later pipeline. For the bounding box annotations, we convert all to using the absolute coordinates in the format of [x1, y1, x2, y2]. The input for our data pipeline the image, the bounding box annotations and category name of each instance, and we will output the referring expression paired with the target instance.
2.2 COLLECTION GUIDELINES
As shown in Figure T we aim collect referring data that requires visual reasoning, avoids shortcut solutions, and challenges models. Based on these observations, we propose the following guidelines mitigate these limitations and yield cases requiring advanced reasoning.
Distractor Pressure Distractors instances of the same category the target but different in- stances. To avoid easy grounding based solely on the target category, we select images that have at least 3 candidate instances of the same category as the target, based on the instance annotations of each dataset.
Language Complexity RefCOCO(+/g) has an average expression length of around words, which limits language complexity and requires much less visual reasoning. Meanwhile, fixed tem- plates that extract referring information from scene graphs limit diversity in the referring expres- sions. Therefore, employ LLMS (e.g., GPT-4o) with carefully designed pipelines to generate more natural and diverse referring expressions while maintaining linguistic complexity.
Hard Distractors Simply increasing the number of distractors and the length the referring expression does not necessarily make the task more challenging because of the "grounding shortcut" illustrated in Figure (c). To reduce such shortcuts (i.e., reliance on redundant descriptors), we ensure the presence of "hard distractors" in the images, defined distractors that partially match, but do not exactly satisfy, the referring expression. Identifying such pairs and composing expressions around them central to data collection process.
3

---

Published as a conference paper at ICLR 2026
Find descrioters pair pair.
expression for disorimatoos using negation.
fsimiir group
"The person with short halk and weening sungless"
The expression is accurate. There exists hard distractors.
scal Sagoine
group disciminatoos "hair
"Nor waar ing neckioss the person in the light-oskores stint
individual minators: Faunglass
Ref Adv Data
Prepare. Image
Similarity Judgement
(c) Expression Generation
(d) Humon Verification
Figure 3: I.LM-authored data curation pipeline for Ref-Adv. (a) Prepare Image: filter images,
ensure 2 3 distractors, and add number tags to candidate instances. (b) Similarity Judgement: use GPT-4o to identify the most similar pair and elicit group-level and instance-level discriminators. (c) Expression Generation: compose minimally sufficient referring expressions using discriminators and optional negation. (d) Human Verification: verify expression accuracy and confirm the existence of hard distractors before inclusion.
Manual Check It is laborious and time-consuming to manually select images with hard distractors and generate the referring expressions, so we use LLMs to assist generation. However, LLMs can make mistakes hallucinate. To ensure accuracy, we perform a human verification pass to confirm the existence of hard distractors and the correctness and unambiguity of the referring expression.
2.3 REFERRING EXPRESSION GENERATION PROCESS
As shown Figure 3. the whole generation process is conducted in four stages. The prompts we provided in section
Input Preparation We first filter the images to only keep those with at least 3 candidate instances. We then put number tag each instance, similar Set-of-Marks (Yang et al. (202)), but since already have instance annotations, we only need to add the number tag to the candidate instances.
2.3.1 LLM-ARTH PIPELINE
Before detailing the pipeline, we note an important design choice. We first attempted single step prompting of GPT-4o to directly produce complete referring expressions from the image and can- didate instances. In practice, GPT-4o frequently produced overspecified descriptions with many redundant descriptors, which enabled shortcut grounding and weakened the need to understand the whole expression. To avoid this behavior, we adopt two stage procedure: we first elicit discrim- inative attributes (between group A and group B and within group A), and then compose the final expression from a minimal yet sufficient subset of those attributes.
Similarity Judgement If there is hard distractor and target instance, they will be similar in some ways. To encourage the LLMs to identify any such similar pair in the image, we define two groups, group A and group where group contains the hard distractor and the target instance, and group B contains the other distractors. We then prompt the LLMs to identify the two groups and describe (1) attributes that distinguish the groups and (2) attributes that distinguish the two instances within group A. We ask for multiple alternative descriptions for each distinction. This could help us generate multiple diverse referring expressions for one image and allow us to select the high quality ones.
Referring Expression Generation After the similarity judgement, we obtain list of paired de- scriptors that distinguish (1) group A from group B and (2) the two instances within group A. To ensure naturalness and diversity in phrasing, we prompt LLMs to compose referring expressions from combinations of these descriptors. Specifically, the target's 's descriptors and (2) use the negation the hard distractor's ' descriptors. This promotes more diverse and natural expressions. We also explicitly instruct the L.LMs to not include number
two alternative strategies: (1) employ
4

---

Published as a conference paper at ICLR 2026
Comperber Seteers
Figure 4: Dataset statistics across REC benchmarks. (a) Expression length comparison. (b) Distri- bution of distractor counts. (c) Instance size on log area scale.
tag related descriptions. Although the elicited descriptors alone are sufficient for generation, we find that including the image input at this stage yields more diverse and accurate expressions, so we include the image. After this stage, we obtain multiple candidate referring expressions for each target instance.
2.3.2 HUMAN-AUTHOREH PIPELINE
We also collect a subset of human-authored referring expressions. For each filtered image, annota- tors first confirm whether there is hard distractor pair and, if so, write a referring expression for it. Annotators are instructed to produce diverse and natural phrasing.
2.3.3 VERIFICATION PROTOCOL.
We verify each image-text pair. Three annotators answer two questions: (1) whether the expression correct and unambiguous and (2) whether hard distractors present in the image. Annota- tors first attempt grounding the original image (without number tags) using the LLM generated expression. We then show the ground truth box overlaid on the image for reference, allowing re- flection if their initial grounding was incorrect. Afterward, annotators record their final decisions on correctness/unambiguig and on the presence hard distractors. Pairs are presented in a random order per annotator, and a pair is kept only if all three annotators agree. The keep rate is 18.7% for LLM-authored expressions.
2.4 QUALITY ANALYSIS
Despite verification ensure correctness, there remain potential issues for REC benchmark that could affect fairness and the evaluation of reasoning skills. To further assess the quality of our data, conduct the following analyses.
Statistics As shown in Figure table Ref-Adv exhibits clear advantages in expression length, vocabulary size, distractor counts, and the negation ratio.
Model Bias Test Inspired by Cirik et al. (2018); Chen al. (2020). we conduct a bias test of modern MLLMs (Qwen2.5-VL-72B and Intern VL-3) ReiCOCO(+/g) and Ref-Adv. Here, bias refers to statistical regularities that may arise if training data comes from the same source as an evaluation benchmark, which can benefit performance. We design the as follows: expression with a fixed prompt ("the one") keep the same image, and prompt the model to output bounding box. This test reveals whether model bias helps localize the target. The results are shown in table They suggest that Ref-Adv is less affected by this bias than other benchmarks.
replace the referring
Textual Reasoning Necessity Test Prior work (Akula al. 2020) shows that shuffling word order in RefCOCOg often leaves performance largely intact, indicating weak necessity for textual reasoning in prior REC benchmarks. This lack of degradation could stem from two factors: (I) expressions that only mention the target (or its parts) without referencing distractors and (2) images with no or very few distractors. Both factors reduce the reasoning demand in REC. To validate that Ref- Adv requires reasoning, extend the test to RefCOCO(+/g) and Ref-Ady for comparison. Rather than shuffling while preserving meaning, we propose a simpler test: we convert the expression to bag of words and randomize its order in the prompt (e.g., "a red ball with yellow stripes" becomes
5

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy 0.5 after replacing the original referring expressions with the fixed "the one" prompt. A is Fixed@0.5 minus Ref-Adv Fixed @.5 (shown in blue). With fixed prompt, models achieve higher accuracy on RefCOco, RefCOCO+, and ReiCOCOg than Ref-Adv.
RefO000
ReiDoco:
RefCoCOg
Ref-Ady Fex
Model
Fixed00.55
vs Ref-Adv
Fixe0
va Ref-Adv
vs Ref-Ady
Qwen2.5-VL- IntemVL-3-148
35.1%
+13.7%
39.4% 38.0%
+18.0% +15.2%
38.3%
+16.9%
21.4%
+13.1%
Table 3: Bag-of-words ablation RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv. Acc@0.5
with original expressions vs bag-of-word. (word order removed). A denotes (BoW - Original).
RefOOCO
ReiCOCC
RefC0000
Ref-Ady
Model
Orige00.5
BoW190.
O
BoW
Orig
RoW
Org
BoW.0..
Qwen2.5. V-L--72
92.75
82.8%
-9.95
88 996
78.2% 81.0%
10.75%
89.9%
75.3%
-1466
58. 58.3%
41.5% 38.6%
-1688
IiteraVL-3-148
92.0%
84.7%
-7.3%
87.6%
46.5%
88.5%
74.9%
-33.6
52.3%
-13.7%
"with yellow red ball stripes a"). We evaluate Qwen2.5-VL-725 and InternV--3 under this setting. Results are shown in table indicating that Ref-Adv indeed requires texual understan and reasoning follow the referring expression exactly.
As illustrated in Figure I RefCOCO(+/g) admits "ground- ing shorteut," where a model can localize the target by checking small subset of descriptors, with- out reasoning over the entire expression. To validate that Ref-Adv avoids this shortcut, we con- duct descriptor-delttion sufficiency test. For given referring expression, we first use Qwen2.5- 72B (Yang et al.) 2024) to extract all descriptors, randomly delete one, and ask Qwen2.5-72B to rewrite the expression with that descriptor removed. We then evaluate MLLMs on the modified image-text pair. If deleting descriptor does not affect performance, the descriptor is unnecessary, suggesting a shortcut that succeeds without understanding the full expression. Such shorteuts are exacerbated in datasets with imbalanced numbers of descriptors and distractors. Results are shown in table indicating that Ref-Adv has far fewer grounding shortcuts than others.
Avoidance of "Grounding Shortcut"
3 EXPERIMENT
3.1 EVALUATION SETUP
Evaluated Models We evaluate contemporary state of the MLLMs, both closed source and
open source, on Ref-Adv. The suite includes Qwen2.5-Vi series (Bai et al. 2025). Intern
3 series (Zhu al. 2025). Gemini 2.5-Flash (Google 2025a), Gemini 2.5-Pro (Google 20256).
GroundnGruundin (Hong et al. 2024). GLM-4.5V (Team et al. 2025b), GPT-4o (OpenAl 2024
and 3la Sonnet (Anthropic) 2024).
Evaluation Methods Set-of-Marks (SoM) overlays numbered marks on candidate objects in the image and leverages a specialized segmenter to provide fine-grained localization, avoiding the need for the MLLM to perform grounding itself. Because GPT-4o and Claude-3.5 3.5 have limited grounding ability, we evaluate them using SoM (Yang et al., 2023) with Semantic-SAN (Li al. 2023). We use Semantic-SAM due to its strong performance on COCO images, one of the sources of Ref-Adv.
For each model (except CogVIMM M..M-Grunniiggg which does not support CoTT), we evaluate both with and without Chain-of-Thouu
(CoT), While CoT is uncommon in classic REC benchmark evalua- tion, Ref-Ady requires more reasoning, so we include CoT in our setup. Table 7 and table report results on Ref-Adv and RefCOCO(+/g) with and without COT.
Evaluation Prompts Models differ in prompt format and output conventions. For example, Wwen2.5-VL-72B uses absolute coordinates, while others normalized coordinates; CLgVLM- Grounding requires the question to strictly follow the form "Where is the 'referring expression'? '?" to output boxes. To ensure fairness, adopt best-practice prompts for each model.

---

Published as a conference paper at ICLR 2026
Table 4: One descriptor deletion ablation on RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv. Acc@0.5 with original expressions vs one descriptor deletion (removing a single descriptor in ex- pression). A denotes (1-Desc- Original).
RefOOCO Orig@00.5
ReCCOCO Orgg00
RefC0COg
Ref-Ady 1DG0.
Model
ID6005
ID805
Orig6r0.5
ID600.5
Org
Qwen2.5-vv V-L-772B 92.78
88.0%
4.7%
88 88.9%
83.6% 87.66
-5.3% -5.2%
89.9%
85.3% 83.8%
51.9% 52. 39.3%
58.
-6.4% -7.1%
InternVl-3-148
92.0%
87.1%
4.9%
82.4%
88.5%
4 4.7%
45.2%
Table 5: RefCOCO(+/g) and Ref-Adv Acc A0. with and without Chain-of-Thought (CoT), A de- notes (CoT Direct).
RefCoCO
RefCOCO+
RefCOCOs
ReF-Adv
Model
Diruct
Qwen2.5-VLL --
92.7%
89.3% 89.2%
-3.4% 2.8%
88.9% 87.4%
86.2% 85.8%
2.77% -1.0%
89.9% 88.6%
88.5% 87.4%
-145 -12%
54.1% 30.5%
58.3% 52.3%
+4.2% +1.8%
InternVL-3-14D 3-BB
92.0%
3.2 EVALUATION METRICS
Accuracy serves widely adopted metric for evaluating existing REC models. A referring expres- sion instance is deemed successfully grounded when the Intersection over Union (loU) between the predicted bounding box and the ground truth annotation surpasses 0.5. This conventional evaluation metric is designated as AccO.5, Here, we implement multiple evaluation protocols, i.e., Accuracy computed under different IoU thresholds such as AccO.5, Acc0.75, Acc0.9, and mean Accuracy (mAcc) across different loU criteria, to thoroughly evaluate the precision and robustness.
3.3 ANALYSIS
Effeet of Distractor Count In Ref-Adv, each expression is paired with at least 2 same-category dis- tractors, and images contain roughly 4 distractors on average. Compared with overall AccÔ.5, most models show modest change in the 4-6 group but larger drop in the 27 group (e.g.. DW 72B+CoT: -0.2 and 2.7). This trend indicates that handling larger numbers of similar distractors remains a key challenge for current MILMs.
Effect of CoT Table 7 and table 5 show that CoT generally improves performance on Ref-Adv, while it can reduce accuracy ReiCOCO(+/g). We attribute the improvement on Ref-Adv to its heavier reasoning demand; for RefCOCO(+/g), where grounding can often succeed without exten- sive reasoning, CoT may introduce unnecessary verbosity or error.
is worth noting that while Argus (Man et al. 2025) reports sizable CoT gains on RefCOCo, its CoT ablations conducted on VQA style benchmarks by augmenting training with additional CoT data, whereas our study uses off the shelf checkpoints and evaluates directly on RefCOCO(+/g) and Ref-Adv without extra training. RefCOCO(+/g) also contains many short expressions with few distractors, so CoT is often unnecessary and can even harm performance. Moreover, standard mul- timodal evaluation toolkits such as open compass and VL.EvalKit do not enable CoT for Ref- COCO(+/g), which is consistent with our finding that CoT brings limited benefit in this setting and more helpful on Ref-Adv, where reasoning demand is higher. This observation is in line with the recent study "To Think or Not To Think: A Study of Thinking in Rule- Based Visual Reinforcement Fine- Fine-Tuning" (Li al. 2025). which also reports limited CoT benefits on RefCOCO(+/g).
Main Results Table summarizes results on Ref-Adv. With SoM, GPT-4o attains the best per- formance on Ref-Adv under CoT, suggesting strong reasoning and visual perception capabilities. While other models perform well on RefCOCO(+/g). their accuracy drops markedly on Ref-Adv, revealing gaps in visual reasoning and perception.
Qualitative Analysis Figure 5 shows qualitative examples for Qwen2.5-VL-772 and Gemini 2.5- Flash, both with and without COT. With explicit reasoning, models often follow the intended chain, but in harder cases they fail partway due to incorrect visual perception the referring expression. Notably, models often select the hard distractor as the answer, which indi- cates that Rdv-Adv challenges models to both deeply understand referring expressions and perform accurate visual perception. This suggests that Ref-Adv stresses advanced reasoning and visual per- ception, and that current state of the art MILMS still exhibit clear gaps.
misunderstanding of
7

---

Published as a conference paper at ICLR 2026
LLM-Authored
Hluman-Authoree
Image
Expression
The cup that is empty positioned the center the pizzas.
The young elephant near the water not covered in more dust.
The car is brown or black with lighter on windows.
The person sitting far- to the one hokding goose feather
Image
Expression
The frishee holder who is not facing forward.
The occupied chair with the person's is arm not rest- ing the lap.
The sailboat, identified by being anchored, is the with
The frishee held by the person on the right hand side of the person in red
Table 6: Examples from Ref-Adv. Columns I to are LLM generated; column is human authored.
Table Main results on Ref-Adv. Rows list models; columns report accuracy at IoU thresholds 0.5, 0.75, and 0.9, and mean accuracy (mAcc). For distractor groups (4-6 and 27), we report Acc0.5 and the delta relative overall AccO.5.
Model
Setting
Acc0.5
Acc0.75
Acc0,9
mAcc
Distractors (Acc0.5)
CoT?
SoM?
4-6
A
27
GPT-40 2024
X
52.3
31,2
13.4
27.8
53.4
+1.1
51.7
-0.6
GPT-40
20ZA
7
63.7
38.4
19.7
34.1
62.9
-0.8
60.5
-3.2
Claude-3.5 Sonnet
2024
22.1
39.0
-1.8
37.4
Claude-3.5 Sonnet
2024
45.2
19.8
2.1
44.2
-1.0
42.3
-2.9
Gemini Flashl 2025a
X
X
50.6
23.7
6.9
19.2
49.5
-1.1
48.9
-1.7
Gemini 2.5-Flash
2022a
X
59.4
35.1
16.3
30.6
58.1
-1.3
55.6
-3.8
Gemini Pr5-Pro 2025b
X
X
51.9
28.4
11.7
23.7
50.3
-1.6
49.7
-2.2
Gemini 2.5 Pro 20256
X
59.1
32.6
14.2
28.3
58.0
-1.1
55.9
-3.2
Intern VL-377L 2025
49.5
39.2
21.4
33.1
49.2
-0.3
48.6
-0.9
Intern Vlee-VL-3-7B3 2025
48.7
37.9
20.1
31.8
47.5
-1.2
45.8
-2.9
VL- 2025
X
50.5
40.7
22.8
34,2
49.7
-0.8
50.3
-0.2
InternVL-3314 2025
$2.3
42.1
24.3
35.6
51.9
-0.4
49.1
-3,2
Intern VL-3388 2025
53.8
43.5
25.7
37.1
53.4
-0.4
$2.9
-0.9
ViemmVI-3-3355 2025
57.2
46.8
28.9
40.3
56.9
-0.3
54.1
-3.1
Intern V
2025
54.6
44.2
26.4
37.8
53.9
-0.7
53.4
-1.2
IntemmVL-3-78B
2025
58.4
47.9
29.6
41.2
57.2
-1.2
55.4
-3.0
Qwen2.5.VL-19
2025
39.0
21.2
-0.9
-1.2
Qwen2.5-VL-7B 2025
49.1
38.8
20.9
47.6
-1.5
46.0
Qwen2.5-VL-321g
2025
52.7
42.4
24.6
-0.2
52.0
Owen2.5-VL-3225
2025
28.7
-1.0
54.3
Dwen2.5-VL-722.
2025
$4.1
+0.0
53.6
Qwen2.5-VL-72B
2025
41.1
-0.2
55.6
GLM-4.5V 2025b
X
X
52.4
42.1
24.3
35.6
$1.9
-0.5
51.6
-0.8
GLM-4.5V
2025b
X
56.9
46.6
28.8
40.2
55.9
-1.0
54.6
-2.3
CogVL.M-Gnounddng 2024
x
X
51.5
41.2
23.4
35.0
52.4
+0.9
50.8
-0.7
4 LITERATURE REVIEW
Referring Expression Benchmarks. Segmentation based benchmarks constitute foundational category in computer vision, with numerous datasets spanning diverse domains and applica- tions (Kuznetsova al. 2020 Lin et al. 2014 Wang et al. 2022 Du et al., 2023 2025). The field's foundational benchmarks, including the ReferltGame (Kazemzadeh et al. (014)) and the de
facto standard RefCOCO suite (RefCOCO/+/g) (Yu al. 2016 Mao et al. 2016), have been in-
8

---

Published as a conference paper at ICLR 2026
With CoT
Withour CoT
With Cot
Without CoT
(o) Input
(b) Gemini
(c) Quen
Figure 5: Performance of representative multimodal LLMs on Ref-Adv. We include qualitative examples with and without CoT for Gemini 2.5- and Qwen2.5 --L--2... CoT answers are shown in gray box. Hard distractors in Ref-Adv challenge current MLLMs.
strumental in advancing research. However, subsequent analyses revealed that high scores on these datasets can overstate genuine grounding abilities. For example, performance on RefCOCOg often remains high even with shuffled word order, indicating reliance on superficial cues rather than robust compositional understanding (Akula al., 2020). To address these cracks in the founda- tion-namely simplistic expressions and lack of hard, same-category distractors new wave of benchmarks emerged. To directly target reasoning. Cops-Ref (Chen et al. 2020) and its successor FineCops-Ref (Liu et al. 2024) introduced more compositional expressions with explicit distractors and negative examples, while the synthetic CLEVR-Ref+ (Liu et al., 2019) offered fully con- trolled environment for diagnostic analysis. Concurrently, other efforts expanded the scope of the REC task itself. gRefCOCO (Liu et al.] 2023) introduced multi-target and no-target expressions, PhraseCut (Wu et al. 2020) scaled up phrase-level segmentation
categories, and recent more natural descriptions and corrected label noise in the original benchmarks.
works like HC-RefLoCo (Wei et al. 2024) and Ref-L4 (Chen et al. (024) have pushed for longer,
The need for such challenging benchmarks is further amplified by the rapid advancements in Multi- modal Large Language Models (MLLMs), which now dominate the field.
Multimodal Large Language Models. Recent progress in vision language Al has been driven by large multimodal language models (MLLMs) that combine powerful LLM backbones with vision encoders and aligmment tuning for instruction following. A growing body of work has explored visual understanding
L.IMs, with grounding ability emerging as an important focus (Bai et al. 2025 Hong et al. 2024 Team et al.) 2025b, Lu et al.) 2025b a 2026). Proprietary models like Ope- nAl's GPT-4 Vision and Google's s Gemini exemplify this trend, while open source counterparts such as Alibaba's Qwen-Vi VL and Shanghai AI Lab's InternVL offer similar capabilities (OpenA] 2024 Google 2025a Bai et al. 2025 Zhu et al. 2025). These systems, trained on massive image text cor- ceiling accuracy (often > >90%) on classic referring expression benchmarks Kazzemzadeh et al. 2014 Yu et al.) 2016; Mao et al. 2016). However, as the reasoning capabilities of MLLMs rapidly advance, it has become clear that these high scores insufficient measure
pora, achieve

---

Published as a conference paper at ICLR 2026
genuine multi-step reasoning, necessitating evolution in the REC task itself (Wei et al. 2024 Chen et al. 2024; Dong et al. 2025). This has spurred the development both more challeng- ing benchmarks and reasoning enhanced models. For example, Moonshot's Kimi-Vi LL (Thinking) applies chain of thought fine tuning and reinforcement learning to strengthen stepwise visual rea- soning (Team al. 2025a), and ZhipuAl's GLM--45V explicitly performs step by step grounding output precise object bounding boxes (Team et al.) 2025b). Similarly, new aligned vision language models like CogVLM and DeepSeek-VLa incorporate mixture of experts or reward optimization to improve visual grounding and coherence, and even commercial chatbots (e.g., Anthropic's Claude 3.5, xAI's Grok) are beginning integrate advanced multimodal reasoning. Our work builds on these efforts by evaluating broad suite of state of the art MLLMs both general purpose and reasoning centric-on on a novel REC benchmark designed stress test their visual grounding and reasoning abilities (Hong et al. 2024
Team et al. 2025b a
Wu et al. 2024 Anthropic 2024 xAI
2025).
CONCLUSION
In this work, we introduced Ref-Adv, modern REC benchmark designed to address the reliance on visual shortcuts in existing datasets by requiring genuine multi-step reasoning. We construct Ref-Ady through a two stage pipeline that uses an LLM to compose minimally sufficient referring expressions. Our comprehensive ablation studies (section[2 confirm that Ref-Adv effectively probes both complex textual and visual grounding capabilities. Strikingly, our evaluation of contemporary MILMM (section 3) revealed significant performance drop compared to their near-saturated scores on RefCOCO(+/g). exposing a critical overestimation of their visual reasoning abilities. These findings underscore the urgent need for benchmarks that reflect real world visual complexity and offer a clear path forward for developing more robust and capable MLLMs.
ETHICS STATEMENT
We use large language models draft candidate expressions and then apply a human verification step with three annotators to ensure correctness and remove ambiguous or unsafe content (Section (2). An- notators worked only with public images and could skip any example. Our benchmark intended for evaluating grounding and visual reasoning, not for surveillance biometric identification. We release only expressions, target regions, and dataset identifiers, and we provide usage guidance that discourages applications involving identity inference or sensitive attribute prediction. We are not aware of conflicts interest.
We follow the ICLR Code of Ethics (Ittss:/ic....coppubliici
REPRODUCIBILITY STATEMENT
Section 2 describes the complete data pipeline, including image sources, filtering with same-class distractors, descriptor elicitation, expression composition, and the three-annotator verification pro- tocol, with a step-by-step diagram Figure We will release the exact image identifiers, the final referring expressions, target regions, and the JSON schema of our annotations, together with scripts to load and evaluate the data. Evaluation protocols and metrics (Acc0.5/Acco..55) 75/Acc0.9 and mean Accuracy) are specified in Section 3 To facilitate exact replication, we will provide below artifacts upon publication: (i) the evaluation scripts that compute IoU and accuracy, (ii) the prompts and con- figuration files for each evaluated model. Together, these artifacts enable end-to-end reproduction of our tables and figures.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arjun R. Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chur Zhu, and Siva Reddy. Words aren't t
enough, their order matters: On the robustness of grounding visual referring expressions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp-
6555-6565, Online, 2020. Association for Computational Linguisties. doi: 116653111/2200. ael-main.586. URL http://////claatooooooooor////// ma.....0....mmm.............
Anthropic.
Claude 3.5 sonnet.
http:::/www..anthrr.. com/news/
glaude- 3- 5- sonnet 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint 2r0iv:2502. 11022.1392. 13923, 2025.
Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S.-H. Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the of large multimodal models. arXiv preprint arXiv:2406. 16866, 2024. URL https://arxxv
org/ hae//bb//220 2406. 16866
Zhenfang Chen, Peng Wang, Lin Ma. Kwan- Yee K. Wong, and Qi Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 10086-10092 June
2020. URL htttps:/ http://cpppccc...eepepppsp aerenaccesssthecv... www..f.ec.....ccm www.c///.//cccco/oooooooocc Cepntent__VRRR_et 10TFP__202000 2020/htm1/ Chen Cops-Ref A New Lewe__attaee____eeeeeeeeeaae__ee and Task on Compositioniallll.......o.oc...l Referring
Expressicn_cvPR wttpccca__eoppppeeeeeeee
Volkan Cirik, Louis-Phillipe Morency, and Taylor Kirk-arikaa Visual referring expression recognition: What do systems actually learn? arXiv preprint arXiv: /805. ///88 2018.
Qihua Dong, Luis Figueroa, Handong Zhao, Kushal Kafle, Jason Kuen, Zhihong Ding, Scott Cohen, and Yun Fu. Cot referring: Improving referring expression tasks with grounded reasoning. 2025.
URL https:://axxi...ott/
abs///bs//2000 2510. 06243
Hao Du, Qihua Dong. Yan Xu, and Jing Liao. Weakly-supervise 3d medical image segmentation using geometric prior and contrastive similarity. IEEE Transactions on Medical Imaging, 42(10): 2936-2947, 2023.
Hao Du, Qihua Dong, Yan Xu, and Jing Liao. Tdformer: top-down token generation for 3d medical image segmentation. IEEE Journal of Biomedical and Health Informatccs 2025.
Google. Gemini 2.5 flash.
flash/] 2025a.
Google. Gemini 2.5 pro. 2025b.
Wenyi Hong, Weihan Wang, Ming Ding. Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprini arXiv:2408. 16500, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Tost-, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. In- fernational Journal of Computer Vision, 128(7):156-1191 2020.

---

Published as a conference paper at ICLR 2026
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang. Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recogmize anything at any granularity, arXiv preprint ar arXiv:2307.04766 2023. httttts::
org/ abs 2200/337...0 04767
Ming Li, Jike Zhong, Shitian Zhao, Yuxiang Lai, Haoquan Zhang, Wang Bill Zhu, and Kaipeng Zhang. Think or not think: A study of explicit thinking in sule-based visual reinforcement fine- tuning. arXiv preprint arXiv:2503. 16188, 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23592-23601, June 2023. URL ltttps::/openaccessst theccfeee....cc com/
content http://///R2222mmmm///2 _aaiii_GG______ GRES
Bxpprerrrinn_Bxxessssooo..o Expression
Segmentation CVPR 2023 paper. .tmll
Junzhuo Liu, Xuzheng Yang. Weiwei Li, and Peng Wang. Finecops-ref: A new dataset and task for fine-grained compositional referring expression comprehension. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15440- 15457, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/
11/2024.emmlpmmiin.666 URL
emn le-eelmmmnee po-main....
8647
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan Yuille. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pat-
tern Recognition (CVPR), pp. 4185 4194, June 2019. URL
thecvf. wwt.coomcooooooo content CVPR 2009/htmmminee
Diagnosing Visual Reasoning With Referring Expressions CV________________ 2019 paper. html
-LEVR-ReE. -ww.aa--RRRe-Diaasinnnn.
Jianglin Lu, Hailing Wang, Yi Xu, Yizhou Wang, Kuo Yang, and Yun Fu. Representation potentials of foundation models for multimodal alignment: A survey. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 16669-16684, 2025a.
Jinnllin Lu, Hailing Wang, Kuo Yang. Yitian Zhang, Simon Jenni, and Yun Fu. The indra repre- sentation hypothesis. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Jianglin La, Simon Jenni, Kushal Kafle, Jing Shi, Handong Zhao, and Yun Fu. Seeing through words: Controlling visual retrieval quality with language models. In The Fourteenth International Conference on Learning Representations, 2026.
Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang- Yan Gui, Jan Kautz, Yu- Xiong Wang, and Zhiding Yu. Argus: Vision-centri reasoning with grounded chain-of-thouu In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14268-14280 2025.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL hhttps://openaccesss thecsf..eccc..cm wtt.coocccooooomm cont/nttttettttooe cerii__cvp____1011 201222206662000
Mao Generation and Comprehension cwtpesiiinnnnccnoeeeeeee 2016 paper.pdf
OpenAl. Gpt-4o. https://openal wttptaai..ooiiiide//eeee/ee 40/ 2024.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng. Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu,
12

---

Published as a conference paper at ICLR 2026
Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu. Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang. Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, and Zongyu Lin. KL technical report, 2025a. URL pestts://arxiie org/ abs /ttp//504.0700
Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang. Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang. Chenhui Zhang, Da Yin, Fan Yang. Guoqing Chen, Haochen Li, Jiale Zhu, Jiali Chen, Jiaxing Xu, Jiazheng Xu, Jing Chen, Jinghao Lin, Jinhao Chen, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang. Qinkai Zheng, Ruiliang Lyu, Shangqin Tu, Sheng Yang, Shengbiao Meng, Shi Zhong, Shiyu Huang. Shuyuan Zhao, Siyan Xue, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wei Jia, Wenkai Li, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyu Zhang, Xinyue Fan, Xuancheng Huang, Yadong Xue, Yanfeng Wang, Yanling Wang, Yanzi Wang. Yifan An, Yifan Du, Yiheng Huang. Yilin Niu, Yiming Shi, Yu Wang, Yuan Wang. Yuanchang Yue, Yuchen Li, Yusen Liu, Yutao Zhang, Yuting Wang. Yuxuan Zhang, Zhao Xue, Zhengxiao Du, Zhenyu Hou, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and glm-4. Iv- thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025b. URL https::p////mm //t:://rxxiii org/ abs/ /0t./2550..0 01006
Tao Wang, Jianglin Lu, Zhihui Lai, Jiajun Wen, and Heng Kong. Uncertainty-guidel pixel con- trastive learning for semi-supervised medical image segmentation. In Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI-22. pp. 1444-1450. Interna- tional Joint Conferences Artificial Intelligence Organization, 7 2022.
Fangyun Wei, Jinjing Zhao, Kun Yan, Hongyang Zhang, and Chang Xu. sarge-scale human- benchmark for referring expression comprehension in the Imm era. In NeurIPS Datasets and Benchmarks Track, 2024. URL http://sss/proceeeengg..ssseeee n neur cttpppeeccccppppppeeeeeeeeeeee
fi and Benchmarks Track.pds
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning in large language models. arXiv preprint 1.Ni:22001111933, 2022. URL httttttt////ttttt org abs 2201.
11903
Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10216-10225, June 2020. URL
thecvf. com/content....mmmmmmmmmmmo wwwponeett__eeereeeeeeeeee 20202h222221111 Sarage__eemmenttttioo
PhraseCut Language-Baser
Lin__he___________________ee _
2020 paper.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie. Yuxiang You, Kai Dong, Xingkai Yu. Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: Mfxture-of- experts vision language models for advanced multimodal understanding, 2024. URL https:
wwwxiv.org/abs//220 10302 10302
XAI. Grok-4 fast. https://x.x. 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan,
13

---

Published as a conference paper at ICLR 2026
Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11411, 2023. httptstt/txxxi...ttttt
wrg.i.org..a..oo.........o abs http://231..11...
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling con- text in referring expressions. In Computer Vision ECCV 2016, pp. 69 85. Springer, 2016. doi: 10. 10.1007997833-311444476666
URL https://link springer wtt. chapte/
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
Category Distribution Ratio Curves
rafcoco
refcorog
rehedy
Requesicy Rado (og 2cal))
10-4
10-3
Calagery per datavet)
Figure 6: Category distribution ratio curves for RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv. The frequency ratio is plotted on a logarithmic scale after sorting categories within each dataset.
A USE OF LLM IN WRITING.
We employed large language models (LLMs) to assist in polishing the text throughout this paper, including refining phrasing, improving clarity, and ensuring grammatical correctness.
B DATASET CATEGORY DISTRIBUTIONS
Figure visualizes the category level frequency ratios for RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv on a logarithmic scale after sorting categories within each dataset, and shows that Ref-Ady follows more long tailed distribution.
C PROMPT IN DATA COLLECTION
We include the core prompt templates used by our two-stage LLM--uuhored pipeline described in section 2 Query elicits group-level and intra-pair discriminators; Query 2 composes minimally sufficient referring expressions from those discriminators. Placeholders such as (num objects) and (target_class) are filled at runtime.
We use structured output in ISON format for the LLMs ensure the output the correct format.
D LLM API COST FOR DATA COLLECTION
The kept rate is 18.7% for LLM-authored expression, and each expression will cost about 2300 input tokens and 120 output tokens, with GPT-4o price of $2.5 1M input tokens and $10 per IM
output tokens, the cost for a L.LM-athorre expression (2300 x 2.5 120 10)/1, 000, 000
$0.00695, Given that we need to generate approximately 1 187 = 5.35 expressions to get one kept expression, the effective cost per kept expression 5.35 $0.00695 = $0.0372. For our dataset of 4,000 LLM-authored expressions (others are human-authoree the total cost is approximately 4000 x 0.0372 = $148.8.
15

---

Published as a conference paper at ICLR 2026
You are given an image with (num_objectss) (target_class) objects labe ed by integers (1..N).
**
1) Choose the most similar pair ((1,)))) and call that group **A**. Everything else is group *****,
2) Propose exactly ++2 group-level discriminatorss* to separate + vs B Each discriminator must have an A-side phrase and B-side phrase
3) For the two (target_class) objects inside A, propose exactly xx4 intra -pair discriminators- (2 "noticeable", 2 "unnoticeable") Each must provide phrase for object 'i' and phrase for object 'j', plus "unnoticeable".
noticeability" field with value "noticeable"
**Output JSON only**, matching this schema (no extra text):
{
"similar groupp: 11"ids": [int, "labe1":"""),,
"groups": [("A": lint,.....
"group discriminators"::
"f"id"="G1", "name""string" "A":string; "B":string]h: "("id":"G2", "name":string" "A":string striingi },
"in_pair_disee [ig_aaiirddserrmmintttsss""
{f"id";"p1" :string " "j":string, "noticeability":"
noticeable or unnoticeable")),
noticeable unnoticeable]]), "("ia":"33" "rrrrraeaaaaatttaaa..rrrrraaea"" " "ng.reiinnn"ttiieeaiiii.."""""""
noticeable or unnoti unnoticeable")),
[("id"""p4", "name":string" "i"stiing, :j"iitiing, "noticeability":: noticeable unnoticeable"))
1)
If the model is multimodal, attend to the image; otherwise roly on the provided description/ hot.e//aoonnnotaaooos.......
Listing Query 1: Similarity Judgement and Discriminator Elicitation
16

---

Published as a conference paper at ICLR 2026
System: You are a visual assistant that returns JSON only. Follow the user's s schema exactly. Do not include any extra text.
Image context template: This is an image with (num_objects) (target_class (s) overlaid with integers (1..N)
limage_contex.
You are given some observations and 'target_id'.
++bseervatiossw:
(qunryl_jscn]
**Target ID*+: (target_id) id]
**Target Class**; (target_class)
*+Task**; Write the referring expressions that refer to (target_class) target_id' based on the observations. Each sentence should use one group discriminator
and one intra-pair discriminator between the two A). Return total.
Return JSON only with this schema:
"expressiens":
(("id": "El", "target_id":int, "groppddddi" (" "pairddids""" ["p?",,"
http:p/www............aooaaaaaaa "text":strin
:string,"
H orap::/eeeaaaaaaaannooroorororrr
"P?"!,"
1h, hrdd:a/sseaaareeedddaaaaarraaaar
(("ia" "E3", hrgpgpaoottttannaaaaaaa..annanan "tex
("P?],,"
1
("ia": "target in": "group dids": (G'?' "pair_dids":
dids": ["P?"),
itppai__pssiiiiieeee positive phrase":string, "inpair negative phrase":string ptwaaaieeeepaaaaasssstsssssss
"only pwwpyyoosiiveee only regativelbothy "texx"sstrinn""
}}
Explanation example for 'inpair_phrase'; sitting" and simaarrnnegtivrehhraass is "standing", then only_positive" means "the one sitting"; Ponly_negative means "the not standing"; "both" means "the sitting rather than standing
is
Constraints: Use different combinat inns of group_dids ds and pair pair_dids. Vary phrasings and sentence structures. Do not mention numeric labels in the text.
Listing 2: Query 2: Referring Expression Composition
17


Reviews:
[{"review_id": "rgNDj6so7L", "reviewer": "Reviewer_iBZC", "summary": "This paper introduces a Referring Expression Comprehension (REC) benchmark dataset, Ref-Adv, which intentionally removes redundant descriptions and focuses on distractors, to make the dataset more challenging for VLMs. The samples are from the validation and test splits of COCO and OpenImages v7, and are labeled with GPT-4O before human reviews. This benchmark reveals the potential weaknesses of a few common VLMs, including GPT-4o, Gemini 2.5, InternVL-3, Qwen2.5-VL, and GLM-4.5v.", "questions": "1. GPT-4o has a low performance on Ref-Adv, why? It makes so many mistakes on the captions generated by itself?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The benchmark is generated based on a sound rationale. This seems effective at revealing weaknesses in current REC models, although these models tend to overfit the traditional benchmarks.\n2. The evaluation is comprehensive and overall convincing.", "weaknesses": "1. One thing to keep in mind is that the GPT-4O generated captions are a bit biased compared to the human captions, in that GPT-4o apparently generates more negations and humans usually unintentionally avoid negations. This is not necessarily bad, since in real applications, users may need to query with negations.\n2. The analysis of experimental results is minimal. In particular, I wish the authors could analyze the performance on captions containing negations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XvebRf7zoG", "reviewer": "Reviewer_txuD", "summary": "This paper targets the gap between classic REC benchmarks (RefCOCO series) and real visual–language reasoning by constructing a hard, shortcut-resistant benchmark of thousands of instances. \nReferring expressions are produced via a two-stage LLM pipeline that first extracts attributes and then composes a minimal-sufficient description, followed by tri-annotator human verification for the correctness.\nEvaluation uses Acc@IoU at multiple thresholds and tests a broad slate of MLLMs (open and closed, with/without CoT). \nResults show strong models on traditional datasets RefCOCO drop substantially on Ref-Adv. Anti-shortcut ablations demonstrate that Ref-Adv requires order-sensitive, compositional grounding rather than keyword matching, which is an potential issue in the traditional benchmarks.", "questions": "Could the author please add category/attribute/relation/occlusion histograms/stats for the benchmark as a comparison vs RefCOCO(/+/g). \nIt will be better if the author can clarify the CoT setup and explain the different conclusions with ARGUS on RefCOCO/+/g. Please consider address other evaluation pipeline & baselines.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It proposes anti-shortcut design including bag-of-words shuffling and descriptor-deletion both cause larger drops than on legacy benchmarks. This is helpful for the need for compositional, order-aware grounding.\nIts data construction is considerable with strong filter pipeline and covering negation. The final benchmark is processed with strict 3-human-annotator agreement. \nFrom experiments the benchmark exposes failure modes that legacy REC underestimates, creating a clear diagnostic “stress test” where many strong MLLMs collapse. Reporting across multiple IoU thresholds and conditional slices yields informative analysis.\nModel coverage spans major open/closed families and CoT settings.", "weaknesses": "Coverage analysis of this paper is limited. There is no thorough breakdown of the 2,833 images / 5,000 instances (categories/attributes/relations/occlusion, long tails) or side-by-side coverage vs RefCOCO/+/g traiditional widely used benchmarks.\n\nThe CoT conclusion on RefCOCO conflicts with prior work in top venue. ARGUS[1] reports grounded CoT improves MLLM performance on RefCOCO/+/g, but this paper finds CoT can hurt the performance on RefCOCO, making the conlcusion not convincing. \n\nThere are gaps and missing baselines in the evaluation. SoM + Semantic-SAM may propagate segmentation errors to IoU in evaluation with no sensitivity analysis(quantitative or qualitative) in this work. Also, this paper does not analyze the answer–grounding consistency, which is about the inconsistency between the text prediction and the actual grounding. Key baseline like LLaVA-OneVision does not been tested.\n\n[1] ARGUS: Vision-Centric Reasoning with Grounded Chain-of-Thought, CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wFZlKCxwPu", "reviewer": "Reviewer_2Uti", "summary": "This paper keenly identifies a core problem in the current field of Referring Expression Comprehension (REC): although Multimodal Large Language Models (MLLMs) have achieved near-saturation performance on classic benchmarks like RefCOCO(+/g), this high performance is due to models exploiting \"shortcuts\" (e.g., overly short expressions, lack of same-category distractors, redundant descriptors) rather than genuinely mastering visual reasoning.\n\nTo address this, the paper contributes Ref-Adv, a new, high-quality REC evaluation benchmark. The benchmark is constructed through a meticulously designed pipeline that combines LLM (GPT-4o) generation with rigorous human verification. This pipeline is specifically designed to generate scenarios containing \"hard distractors\" and create \"minimally sufficient\" referring expressions for them, while significantly increasing the proportion of complex logic like negation.", "questions": "No", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper not only points out the saturation of classic benchmarks but also deeply analyzes the three specific causes with data and examples.\n\nThe proposed four-stage data pipeline is outstanding. The two-stage LLM process, particularly \"Similarity Judgement\" and \"Minimally Sufficient Expression Generation,\" is cleverly designed. The extremely strict three-annotator verification process ensures the dataset's exceptionally high quality and trustworthiness.\n\nThe CoT experiment is an especially insightful finding, as it clearly reveals that Ref-Adv measures reasoning ability of MLLMs.", "weaknesses": "1. Limitation of an Evaluation-Only Benchmark: Ref-Adv (5k samples) is an evaluation benchmark, not a training set. It excels at diagnosing the flaws of current models but does not provide a pathway for models to learn how to solve these complex reasoning tasks. Given its high construction cost (low keep rate), how to scale this up into a training set is an open question.\n\n2. The paper repeatedly claims to test \"visual reasoning\" and \"multi-step reasoning.\" However, the scope of reasoning evaluated is quite narrow. It primarily focuses on combinatorial logic (e.g., attributes A and B, but not C) and spatial relationships. The benchmark is missing deeper forms of reasoning, such as:\n   - Functional Reasoning: (e.g., \"the object used for cutting,\" \"the item that can hold water\")\n   - Intent/Causal Reasoning: (e.g., \"the person about to jump,\" \"the dog that knocked over the vase\")\n\n    By focusing only on discriminative visual attributes, the paper overclaims the breadth of reasoning it actually tests.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5rZJ1DHaF", "reviewer": "Reviewer_EUYJ", "summary": "This paper introduces Ref-Adv, a new benchmark for Referring Expression Comprehension (REC) designed to address the limitations of existing datasets like RefCOCO, RefCOCO+, and RefCOCOg. The authors argue that these benchmarks are overly simplistic and allow models to exploit shortcuts rather than perform genuine visual reasoning. Ref-Adv is constructed using a semi-automated pipeline combining LLM-generated expressions and human verification, with a focus on hard distractors, minimal sufficiency, and negation. The paper includes comprehensive ablation studies and evaluations of modern MLLMs, showing a significant performance drop on Ref-Adv compared to traditional benchmarks.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses a well-known and important issue in the REC community — the overestimation of model capabilities due to dataset shortcuts. The motivation is clear and well-argued.\n2. The two-stage pipeline (LLM-authored + human verification) is well-designed. The inclusion of hard distractors, minimal sufficiency, and negation adds meaningful complexity.\n3. The evaluation covers a wide range of MLLMs (both open and closed-source), includes ablation studies, and uses multiple IoU thresholds, which strengthens the empirical analysis.", "weaknesses": "1. While the dataset is more challenging, the core task (REC) remains unchanged. The paper does not propose a new task formulation or evaluation protocol beyond traditional bounding box accuracy. The idea of “hard distractors” and “minimal sufficiency” is not entirely new — similar ideas have been explored in prior work (e.g., Cops-Ref, FineCops-Ref).\n2. The dataset is curated from COCO and OpenImages. It would be beneficial to explore how well Ref-Adv generalizes to more diverse or out-of-domain images\n3. The reliance on IoU-based accuracy is standard but limited. It does not capture partial correctness or reasoning steps. For example, the evaluation should reflect that the performance drop is only from the difficulty of object localization or distractors. The model locates the distractor or cannot localize the object correctly.\n4. As shown in Table 3 and 4, the word order removal and descriptor deletion are used to demonstrate that the proposed benchmark avoids the model from relying on the Grounding Shortcut. However, the performance gap is marginal compared with RefCOCO.\n5. The integration of COT also brings marginal performance gain, e.g., the performance of Qwen vl-3B decreases with COT. Therefore, we cannot conclude that the proposed benchmark relies on the reasoning.\n6. This paper only gives a conclusion and a benchmark. However, no solution is presented, which limits the contribution of this paper. In addition, only the evaluation data is provided. Even though we are aware of this conclusion, what can the community do to address such a problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)