Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Authors: Qing Jiang
Xingyu Chen
Zhaoyang Zeng
Junzhi Yu
Lei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Object Referring
Reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present Rex-Thinker, an object referring model with grounded reasoning.

Abstract: Object referring aims to detect all objects in an image that match a given natural language description. We argue that a robust object referring model should be grounded, meaning its predictions should be both explainable and faithful to the visual content. Specifically, it should satisfy two key properties: 1) Verifiable, by producing interpretable reasoning that justifies its predictions and clearly links them to visual evidence; and 2) Trustworthy, by learning to abstain when no object in the image satisfies the given expression. However, most methods treat referring as a direct bounding box prediction task, offering limited interpretability and struggling to reject expressions with no matching object. In this work, we propose Rex-Thinker, a model that formulates object referring as an explicit CoT reasoning task. Given a referring expression, we first identify all candidate object instances corresponding to the referred object category. Rex-Thinker then performs step-by-step reasoning over each candidate to assess whether it matches the given expression, before making a final prediction. To support this paradigm, we construct a large-scale CoT-style referring dataset named HumanRef-CoT by prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a structured planning, action, and summarization format, enabling the model to learn decomposed, interpretable reasoning over object candidates. We then train Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach the model how to perform structured reasoning, followed by GRPO-based RL learning to improve accuracy and generalization. Experiments show that our approach outperforms standard baselines in both precision and interpretability on in-domain evaluation, while also demonstrating improved ability to reject hallucinated outputs and strong generalization in out-of-domain settings. Code is available at https://github.com/IDEA-Research/Rex-Thinker

Published as a conference paper at ICLR 2026
REX-THINKER: GROUNDED OBJECT REFERRING VIA CHAIN-OF-THOUGH REASONING
Qing Jiang Xingyu Chen Zhaoyang Zeng², Junzhi
Junzii Yu², & Lei Zhang 1.2f
1 South China University of Technology International Digital Economy Academy (IDEA)
3 Peking University
Zhongguancun Adacemy
ABSTRACT
Object referring aims to detect all objects in an image that match given lan- guage description. robust model for this task must be grounded, providing explainable and visually faithful predictions. This demands two key properties: 1) Verifiability, through interpretable reasoning linked to visual evidence; and 2) Trustworthiness, by rejecting expressions without matching objects, thereby mit- igating hallucination. Current methods, often treating referring as direct bound- ing box prediction, largely fail to provide such interpretability stain. In this work, we propose Rex- Thinker, an MLLM framework that reformu- lates object referring as explicit Chain-of-Thought (Co'T) reasoning task. Rec- ognizing that standalone detectors lack sufficient language comprehension and MILMs struggle with precise localization, Rex-Thinke adopts a two stage ap- proach. It first leverages an open-vocabulary detector for candidate object pro- posals, then employs an MLLM to perform step-by-step reasoning over these candidates to assess their match with the expression. To enable this, we con- struct HumanRef-CoT, a large-scale (90,824 samples) CoT--ttyle referring dataset. Each reasoning trace follows a structured planning, action, and summarization for- mat, facilitating the learning of decomposed, grounded reasoning. Sex-Thinker is trained in two stages: supervised fine-tuning for structured reasoning, followed by GRPO-based reinforcement learning for enhanced accuracy and generaliza- tion. Extensive experiments demonstrate that Rex-Thinker not only achieves state- of-the-art performance in precision and interpretability but also drastically im- proves the ability reject hallucinated outputs on in-domain evaluation, while showing strong generalization in out-of-domain settings, Code is available at aatps::/qithuu: com IDEA- IDEA-RRsearcc/Rer. ThinkhieeTerr
effectively ab-
INTRODUCTION
Object Referring, also known as Referring Expression Comprehension (REC) Qiao et al. (202
Yu et al. (2016) 6) Mao et al. (2016) Kazemzadeh et al. (2014); Zhang al. (201 9) Yu et 2011 8)
Yang et al. (2019) : Liao et al. (2020) ). aims to predict the bounding boxes for objects in an image that match a given natural language description, which may refer visual attributes, spatial relations, or interactions. This task has broad applications. however, compared to standard open-vocabulary
object detection (Tiang et (2025a); Ren al. (2024a b Liu al. (2024); Yao al. (2022a) Jiang
et al. (2024a] 2023); Li al. (2024a) Ren et al. (2024a) Li et al. (2022): Cheng et al. (2024) Min-
derer et al. 2022); Zareian al. (2021): |): Wu al. (2023) ), REC significantly more challenging,
as it requires both fine-grained visual grounding and more complicated language understanding.
Benefiting from the strong language comprehension capabilities large language models (LLMs), multimodal large language models (MLLMs) have demonstrated impressive performance on this task. There are mainly two paradigms: one treats bounding box coordinates as text tokens and predicts them directly (Chen et al. al. (2023); You et al. (2023); Zhang et | (2024); Wang et al. (2023); Zhan et al. (2025) 2024): Bai et al. (2025): y Wu et al. (2024); Chen et al. (2024); McKinzie et al. while the other adopts retrieval-baat based strategy (Ma et al. (2024): Jiang al. (2024b
(2024)) )
Equal contributions, work done during internship or academic visit IDEA. Corresponding author.

---

Published as a conference paper at ICLR 2026
Question:
Plense detect the person sitting on the nurtle
Amswer:
<think> OK, the user needs us to find the passon sitting on the turtle. I need to break down into the following stege:
Porson
-Sep Ideniily all the turtles the image.
Step For each turtic found Step check if there is person sitting on
Planning
Step I: Identify all the turtles in the image.
locsted the right-hand card the top-ight soction the ovenall picture. grean turtle swimming pond
with lily puds.
Stap2: For each turtle found in Step 1. cleck if there is person sitting on I see five people this image, Lefs look their suroundinngs and put d if there is turtle and X if there is not.
Person
Person
Person 2: man and a woman sitting on a swing. Not sitting on a turile. X
sitting
swing. Not sitting
turtle X
Penson
Person 3:A. bearded figure wearing a red hat and red clothes. He is sitting on the green turtle. A
figure bost, fishing. Person St Figure near bechive. Not sitting on turtle. X
tartle. X
Action
Summarize and Re-Choek antwer Let's now rocheck our answer and put the target fish and X others.
Summarization
Penson 1,2.4.5 Not sitting on tirile X
Person Strtin the turtle. W <2think-+
"ansmer"" "jsonin[[""bou
[1021, 1307, 4201 "label": "the person sittng the turtle"y "11" <aanswr''
Figure 1: An example of Tex-Thinker for object referring with CoT reasoning of planning (task decomposition), action (evaluating each candidate), and summarization (final decision). Each step is grounded in specific hint box (as denoted in the left image), enabling interpretable predictions.
2025b)), where the model is given set of candidate boxes and predicts the box indices that match the expression, Although both approaches have shown promising results, they remain fundamentally implicit, lacking interpretable reasoning steps that reveal how the model arrives at its final prediction. Furthermore, these models are prone to hallucination (Jiang et al. (2025b)), often producing outputs for objects that do not exist in the image, thereby limiting their reliability in real-world applications.
We argue that a robust referring system should be grounded, i.e., its predictions must be both ex- plainable and tightly linked to visual evidence. This requires two essential properties: 1) Verifiable, by providing an explicit reasoning process that allows its decisions to be examined and traced to specific image regions; and 2) Trustworthy. by minimizing hallucinated outputs and learning to reject when no object in the image satisfies the given description. To meet these criteria, we draw inspiration from how humans naturally approach referring expressions. For example, when asked to locate "the person wearing a blue shirt", humans would typically first identify all people in the image, then examine each one to determine whether matches the described attribute. This step-by- step approach reflects a grounded reasoning process, i.e., first localizing relevant object candidates, and then carefully verifying each one against the expression.
Motivated by this observation, we propose Rex-Thinker, an MLLM framework that performs object referring through explicit Chain-of- Thouhht (CoT) reasoning. Specifically, given an image and a referring expression, we first use an open-vocabulary object detector (Liu et al. (2024) to extract all candidate object boxes corresponding to the referred category. These candidate boxes, along with the image and the expression, are then passed into the MLLM to form two stage system. Rex- Thinker follows a structured CoT framework consisting of three key stages as shown in Figure [ 1) Planning, where the model decomposes the referring expression into subgoals; 2) Action. where the model examines each candidate box to determine whether it satisfies its current subgoal; 3) Summarization, where it aggregates the intermediate decisions to produce the final prediction. Fol- lowing DeepSeek-RI (Guo et al. (2025a)). ). we instruct the model to place its reasoning steps within a <think>.... </thi..k../iik>> block and to output the final prediction inside a <answer>... answer>> block. This structured reasoning process not only improves interpretability, but also enables trans- parent and verifiable predictions, as each reasoning step is grounded in a specific candidate region.
To support this CoT framework, we construct a CoT-style referring dataset named HumanRef-CoT,, containing 90,824 samples generated by prompting GPT-4o (Hurst et al. (2024)) on the HumanRef (Jiang et al. (2025b)) dataset. Each example is annotated with a structured reasoning trace following the planning, action, and summarization paradigm, enabling explicit supervision for step-by-step reasoning. We train our model in two stages: a cold-start supervised fine-tuning phase to teach the model how to perform structured reasoning, followed by reinforcement learning (RL) based on Group Relative Policy Optimization (GRPO) (Shao et al. (2024)) to further improve accuracy and generalization. Experiments demonstrate that our CoT-based approach consistently outperforms direct coordinate prediction baselines. On the in-domain Human benchmark. our model achieves state-of-ther the- results with higher detection accuracy and significantly fewer hallucinated outputs, especially on rejection cases. In ofo-ain evaluations RefCOCOg (Mao et al. (2016)), the
2

---

Published as a conference paper at ICLR 2026
model trained only on HumanRef-Col shows strong zero-shot generalization. Further tne-ttuning with GRPO on RefCOCOg yields additional performance gains while preserving the model's 's ability to perform grounded CoT reasoning across arbitrary object categories. Our contributions lie in:
We formulate the grounded object referring task as a planning-action vuctioo-summmrrzatiinn prob- lem, leveraging Shain-of-Thouge reasoning to build a verifiable and trustworthy system.
We introduce HumanRef-CoT, the first large-scale dataset for grounded object referring with step-by-step reasoning annotations, enabling the supervised training of model interpretability. We propose Rex- Rex-ehinker, grounded object referring model trained via cold-start SFT and GRPO-based reinforcement learning. Ref benchmark and demonstrates strong generalization out-of-domall scenes and objects.
achieves SOTA performance on the Human-
RELATED WORK
MLLM Object Referring Methods.
Recent progress in multimodal large language mod- els (MLLMs) OpenAl 2023); : Bai et al. (2025); Wu et al. (2024); Chen et al. (2024): Deitke et al.
(2024); Agrawal et 2024); : Wang et al. (2024):
(003): Chen al. (2025a); Guo et al. (2025b): Yao et al. (2022b by: Yang al. (223b) has led to strong perfor-
; Li et al. (2024b 2025 ; Liu et al.
mance in referring expression comprehension. Existing approaches typically follow two paradigms.
One line of work treats bounding box coordinates as textual tokens Chen et al. (202T) and directly generates them during decoding Chen et al. (2023); You et al. (2023) Wang et al. (2023; ; Zhan et al. (2025); ;; Zhang et (2024) The other line formulates the task retrivaa Jiante al. (2024b);
Ma et al. (2024); Jiang et al. (2025b), where a detector proposes candidate regions and the model selects the best-matching box indices based the input expression. This decouples localization from semantic understanding and simplifies learning. While both paradigms achieve strong results on standard benchmarks such as Re/COCO/+/g Mao al. (2016); ; Yu al. (016)), they face key limitations: a lack of interpretability and an inability to abstain when no object in the image matches the expression Diang et al.) (2025b). To address this, we introduce Chain-of Thought reasoning framework that enables step- step-by-ster bt-step evaluation over candidate boxes. This improves interpretabil- ity, reduces hallucinations, and grounds the model's predictions in the input image.
Reasoning-based LLMs and MLLMs. Recent work in large language models (Jacch et al.
(2025)): Team al. (2025); Muennighoff et al. (202)); Xiang et al. (2025); ) Chu et al. (2025 y OpenAl et al. (2025)) has demonstrated that reasoning ability can be significantly enhanced through Chain-of-Thouud Thought (CoT) training or reinforcement learning- based post-training. OpenAl ol (Jacch et al. (2024) model demonstrates that inference-time scaling can greatly enhance performance on complex tasks like math and coding. DeepSeek-Ri (Guo et al. (222aa) introduces GRPO (Shao et al. (2024) as a post-training method to improve reasoning with- out requiring costly critic models. In the multimodal domain, efforts such as LLaVA-CoT! (Xu et al. (2024) and LlamaV-ol (Thawakar et al. (2025) aim to enhance reasoning by constructing CoT- style data or employing multi-step curriculum learning, without relying on reinforcement learning. For referring expression comprehension task, reinforcement learning has been used in early work to model the grounding process as a sequential reasoning problem, such as through iterative shrink- ing Sun et al. (202I) or dynamic reasoning steps Zhang et al.] (2023). More recently, inspired by DeepSeek-RI (uoo et al. (2025a)), a growing number of works adopt GRPO-based post-training to endow MLLMS with reasoning capabilities. GRPO has been successfully applied to enhance mul- timodal reasoning across wide range of domains, including mathematical problem solving (Yang) ct al. (2025); Peng et al. (2025); Zhang et al. (2025): Deng et al. (2025); Wei et al. (2025)), video understanding (Feng et al. (2025): Liao et al. (2025) and perception tasks (Liu et al. 2025aal b):
(2024): Guo et al.
Xiong et al. (2025)
Ma et al. (2025); Shen et al. (2025): Yu et al. (225)) such as object detection, segmentation, and referring expression comprehension. Following the DeepSeek-RI paradigm, first fine-tune Rex- Thinker on structured CoT data to teach the model how to perform grounded object reasoning. GRPO is then applied in second stage further improve accuracy and generalization.
3 CHAIN-OF-THOUGHH REASONING REFERRING DATA
High-quality supervision is critical for teaching the model reason explicitly. To this end, develop data engine that generates structured referring annotations aligned with our Cha Thought formulation. In this section, we introduce the design principles of CoT reasoning
3

---

Published as a conference paper at ICLR 2026
Planning:
OK. the you reeds us dince the person berweon the rwo aduits To accompiic todk Troed brok is dons isto the falloning aspo.
Step 1 ldeatis the two sdklts innge. Stpp Devoriee the perer located hetwoin the aealte
Acdio:
Imsage
people
Lafs look thiir relone strilurees da irtbey
X ir
Person L.A yourn wis long baiz,
sdul. X
Porer 21 mon with singhosn on hie host appoo
Poreee 21 Aoynngg bey with dson hure nor sdut
Information Question Relèmiig:
Person d. wounon with hes hax red beck, appeors to be adal:
[Ie parses between the oduls
Pewen Persen are two
CIPT-4
Sue right (penon solo Pessen
ponare
there people: Pensnn and Perwir
Summartzation:
Syutem Prompt (Partaal)
found the objocte) mutching descrigrion You shoul conduct aralydi of what you and hers ooch object ribrrs the provikad deserption and pun X & the ond...
analyra of
imnge and desciption oupar detailed
Lat's non reclask ou ssaner and pai Go ssreet possk and otaes Porern Locased betwoon too schils
Summronto and Reoh arower
Poree Une
perurn betwor
Persen Located
Exarple:
[Eureple 122
Figure 2: Overview of the proposed CoT reasoning referring data engine. We prompt GPT-4o to generate a shree-step CoT reasoning process, including planning, action, and summarization.
structure and present the data construction pipeline that transforms existing REC annotations into step-by-step reasoning traces suitable for supervised training.
3.1 COT FORMULATION
The core idea behind our CoT formulation for REC is to transform the task into structured, grounded reasoning process over a set of candidate objects. Rather than directly predicting the referred object, the model evaluates each candidate in sequence, guided by input box hints that localize specific regions in the image. We decompose this CoT process into three key stages:
The model analyzes the complexity of the referring expression and determines how many reasoming steps are needed. For simple expressions, it may plan single step directly match an attribute such as color or size. For more complex expressions, the model generates multi-step plan, where each step focuses on resolving a specific sub-aspect.
Planning:
Action: Based the reasoning plan, the model checks whether each candidate region, grounded via its input box hint, satisfies the current subgoal. This makes the reasoning clear and directly tied to specific regions in the image.
Summarization: Finally, the model reviews the reasoning results across all steps and determines which objects best match the overall expression and outputs the final prediction.
This structured CoT process improves both interpretability and verifiability. Each candidate is eval- uated corresponding to the input box hints, allowing every reasoning step to be explicitly grounded to a specific region of the image. This makes the model's decisions transparent and easy to trace. Additionally, breaking complex expressions into sub-tasks enables step-by-step reasoning, which enhances accuracy and reflects how humans typically process such tasks.
3.2 DATA ENGINE PIPELINE
Building on the structured CoT formulation, we develop a data engine that leverages GPT-4o (Hurst et al. (2044) to generate high-quality CoT annotations tailored to the referring task.
Data Acquisition We construct our CoT dataset based on HumanRef (Jing et al. (025bb recently proposed dataset specifically designed for REC in human-centric scenarios. Unlike prior RBC datasets such as RefCOCO/+/g (Mao et al. (2016); Yu et al. (2016)), HumanRef emphasizes multi-instance referring expressions, where single expression may refer to multiple target persons. It also categorizes expressions into six distinct subsets: attribute, position, interaction, reasoning, celebrity recognition, and rejection. Since the HumanRef dataset provides all person boxes in an image, it can be directly used in CoT annotation pipeline.
GPT-4o Annotation To generate high-quality CoT annotations, we employ in-context prompting with GPT-4o (Hurst et al. (2024)) as shown in Figure [ Given an image and the bounding boxes of all persons within it, apply the Set-of-Mark (Yang al. 2023al) strategy: each individual is
4

---

Published as a conference paper at ICLR 2026
Subset
GPT-4o Annotated Data
Removed Data
Removal Rate
Celebrity
6,775
294
4.3%
Interaction
5,875
72
1.2%
Position 15,950
533
3.3%
Reasoning
9,488
719
7.6%
Attribute
37,648
1,184
3.1%
Rejection
19,096
1,184
6.2%
Table I: Statistics of our two-stage automated filtering process on the generated HumanRef-CoT data. This process removes samples with logical inconsistencies or incorrect final predictions.
labeled with an indexed visual marker, where ground fround-truth truth targets are marked in green and others in red. This design grounds the answer and guides GPT-4o to reason along the correct path. The prompt includes three key components: 1) meta-information such as the referring question, the number of people, their left-to-right spatial order, and the correct answer; 2) a system prompt specifying the desired Danning-actions netiion-uummrizationn structure; and 3) several in-context examples written by humans to illustrate the expected reasoning format. In essence, we provide GPT-4o with both the referring expression and ground-tran answer, and prompt it to generate step-by-step reasoning in our CoT format.
Quality Control: ensure the quality and logical consistency these annotations, we imple- ment rigorous two-stage filtering process that goes beyond simply matching the final answer. First, we enforce internal logical coherence by verifying that the intermediate conclusions in the "Action" phase consistent with the final "Summarization"" phase. Second, ensure final accuracy by requiring the "Summarization"" outcome to perfectly match the ground-truth labels. This automated check is performed by parsing the correct and incorrect emojis generated by the model at each step. This filtering process crucial for data quality, and as shown in Table identifies and removes a significant number of initially generated samples due inconsistencies, especially in complex cases like the "Reasoning" subset (7.6% removal rate).
To further validate the quality of the filtered data, we conducted manual human evaluation study (full details in Appendix A.2.2). The study confirmed the high quality of the final dataset, finding zero logical or summarization errors in 600-sample review, with only minor factual error rate of 1.2% in intermediate reasoning steps. This confirms our dataset provides a reliable training signal.
In total, we construct total of 90,824 high-quality CoT annotations based on the HumanRef dataset, which we refer to as HumanRef-Coll CoT- This diverse and large-scale dataset serves as the foundation for both our initial cold-start SET and GRPO-based post-training.
METHOD
4.1 RETRIEVAL-BASEE OBJECT REFERRING
To leverage the CoT-style referring data, we present Rex- Thinker, a retrieval- based model that per- forms object referring through explicit Chain-of-Thoughh reasoning.
To support explicit Chain-of-Thoughe (CoT) reasoning, we reformulate referring expression com- prehension retrieval-based task. As shown Figure 3 rather than directly regressing bounding boxes, we first use an open-vocabulle detector (Liu et al. (2024)) to extract a set of candidate ob- ject boxes corresponding to the referred object category. These candidate boxes serve as box hints to guide both the reasoning path and final decision of the model. This retrieval- -asee formulation brings two key advantages. First, during the reasoning phase, the model evaluates each candidate region in the order they appear in the input box hints (e.g.. "Person I" corresponds to the first input box). This alignment ensures that each step in the CoT trace is explicitly grounded to a specific region in the image, making the reasoning process interpretable and visually verifiable. Second, during the prediction phase, the model can directly select from the input box hints when producing the final output, thereby easing the challenge of precise coordinate regression.
We build Rex-Theee Thinker on top of Qwen2.5- Diven.55VLL-7B VL-7B (Bai et al. (202))) preserving its original architec- ture and using ISON-formal bounding box coordinates the final output. The model input includes
5

---

Published as a conference paper at ICLR 2026
SFT Cold Start
RL Post Training
GT: ca
GRPO LSe
Per Taken CE Loce
Dovand
5-VV
the holinng
Frose sidegc brovi insert XX
Text Tokan
1. ferson holdngg
Duaitirs
Pieeppocce
E11 160, 2013 "umaa
18
I2 Frii del (11.. 2010 102, T1mme71.
Syster Morrs autorant viint thcarthe wariding process In yound mhee
Formonr 2015, 1AN 1933, Not 2017-
model
sareeeei Aop, sepporteety
Figure 3: Overview of the Rex-Thinker architecture and our two-stage training methods
<image>. A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer enclosed within <think> </think> and <answer> </answer> tags, respectively, ie.. <think> reasoning process here </think> <answer> answer here < </answer>. Hint: Object and coordinates in this image: Box Hint. User: Locate Referring. Assistant:
Table 2: Prompt Template for Rex-Thinker. Box Hint and Referring will be replaced with the input candidate boxes and the referring expression, respectively.
the image, the box hint, the referring expression, and a system prompt that guides the reasoning process. The input prompt format in shown in Table 2
4.2 TRAINING
Our training strategy is two-stage process, consisting of supervised fine-tuning (SFT) for cold start and GRPO-based reinforcement learning (RL) for post-training. We adopt this methodology, inspired by DeepSeck-RI (Guo et al. (2025a)) because it has proven highly effective for teaching models to follow complex, structured reasoning paths and for subsequently refining these behaviors through reward-gun -erudd optimization.
SFT Cold Start We begin by fine-tuning Rex-Thinker on the Humankef-CorT dataset to instill the ability to perform structured reasoning following our defined planning, action, and summarization format. We apply cross-entropy loss at the token level to both the reasoning trace and the final answer, providing strong supervision across the entire generation process. This stage teaches the model how to reason step-by-step in CoT manner and also how to utilize the provided box hints to guide its final predictions.
GRPO Post Training While SFT teaches the model to follow our grounded CoT format, its strict token-level supervision may constrain the model to explore alternative reasoning traces and gen- eralize beyond the training data. To enhance generalization beyond the limitations of supervised learning, we employ GRPO-based (Shao et al. (2024)) reinforcement learning for post-training. GRPO optimizes model performance by 1) sampling multiple candidate responses for each question and, 2) selectively reinforcing responses that achieve higher task-level rewards.
In our setting, given an image and referring expression (1, the model generates group of OG from the current model To- Each response contains a full
G complete responses 01, reasoning trace and final predicted bounding box set. For each o,, we compute scalar reward r; (detailed in below), and normalize these rewards to estimate group-relative advantages:
A; = (r:
Define the token-level advantage estimates A., Ai, and the importance ratio each decoding step as follows,
Pi. = To(oi, (I,2), 0i,ct)
(2)
To (oi,e (I, I), oi,es))

---

Published as a conference paper at ICLR 2026
where The is the model before the current update. Then, the GRPO objective is given as follows,
loal
(8) G
min
(pr,
1,,,, - BLDKL (To) [[ee]
(3)
ON
To(o
II,
log To(os,t 2,,
DKL (oo [Ted
(4)
Ter(or,
(1, R. 01,tt) Tre((o (1, I. 0s,ct)
where e hyperparameter controlling the clipping range, "ee the model fixed after SFT stage, and B is the KL penalty coefficient.
We argue that this formulation suited to policy exploration in our reasoning-driven task. Given that the model is already capable of producing structured reasoning traces after SFT, GRPO allows it to freely explore different reasoning paths. In each iteration, the model generates diverse reasoning strategies that may lead to different predicted object sets. The reward function then guides the model to reinforce reasoning paths that yield accurate predictions.
Accuracy Reward: We use the Fl score to jointly evaluate the precision and recall of the model's predictions. Given set of predicted boxes B and the ground-truth set B", since box hints provided as input, we define a match only when a predicted box exactly overlaps with a ground- truth box (i.e., loU 1), which encourages the model to select final outputs directly from the box hints. Let M B B* denote the set of matched box pairs under this criterion. We compute precision, recall, and the F1 reward as:
M B
JM |B*
2 Precisiio Recall
Precision
Recall
FI
(5)
Precision Recall
Format Reward: To encourage interpretable and well-structured output, we define format reward fint that equals if the output follows the required structure: the reasoning must be enclosed in <think>..</thinke and the final result <answer>...//an <..se//////w,, / and otherwise.
The total reward is weighted combination of the accuracy and format rewards, i.c., r; X r! + (1 A) rffk, where X = 0.9 to emphasize correct detection while still enforcing output structure.
EXPERIMENTS
In this section, we evaluate the effectiveness of our CoT-based reasoning approach for object refer- ring. We first introduce the experimental setup, then present in-domain results the HumanRef benchmark, followed by out-of-domain evaluation on the RefCOCOg benchmark. Lastly, we con- duct ablation studies to analyze key design choices.
5.1 EXPERIMENTAL SETUP
Model Setting. We use Owen2.5-VLL77 as our base model. Qwen2.5-VE outputs absolute bound- ing box coordinates rather than quantized tokens, which provides better localization accuracy for detection tasks. We adopt this native decoding format for final bounding box predictions.
SFT Training. We fine-tune the model on the full HumanRef-CoT dataset using supervised learn- ing. We use learning rate of 2e-5, weight decay of 0.01, and cosine decay scheduling. The max- imum generation length set to 2048 tokens. During SFT, the vision encoder and MLP projector are frozen, and we update only the LLM parameters. For each training instance, we use all person bounding boxes in the image as box hints.
GRPO Training. After SFT, we apply GRPO for reward-driven post-training. We continue train- ing on HumanRef-CoT, but randomly shuffle the box hint order in each training data to create novel input configurations. This leads the model to explore different reasoning paths than those seen dur- ing SFT. During this phase, train only the LLM. We use learning rate of 1e-6, 8 rollout samples per input, a batch size 8, and gradient accumulation steps of 2. The KL penalty coefficient / is set to 0.04, the sampling temperature to 1.0, and the output length remains 2048 tokens.

---

Published as a conference paper at ICLR 2026
Mahhel
Annitaro
DF1
Prittion
DF1
Intracton
F
DF1
Rewwning
F
DF1
Caleberty
F
DF
F
DFF
Rejezton
Scoce
DINOX RRp..tal
2026a
25.8
20.3
75.8
26.1
17.6
01.3
23.5
76-2
52.3
22.2
44.3
68.0
37.0
15.2
53.1
23.3
30,0
Intem
Sepee R0256
31.9
Pone-118
18
Challloe77 7M
Imppre Prosh
46.3
78.0
51.8
46.0
66.3
57.5
49.6
74.
56.3
96.6
68.1
43.8
71.3
76.3
14.3
90.4
73.3
55.
0.0
Qwen25 VL-TB
15g
48.1
71.3
54.4
50.2
61.7
52.8
45.2
66.3
53.2
34,5
61.2
40.3
98.3
81.9
50.1
52.5
68.5
56.2
7.1
DoopSock-V1 VI 2.
po04
52.3
76.0
53.3
58.4
66.1
58.1
55.4
75.3
60.3
36.6
61.7
90.3
85.9
76.3
70.7
99.3
71.2
59.5
1.1
Malmn-78-D
66.1
63.5
83.9
72.3
82.5
08.6
ReeSees
panly 20.00
55.2
54.6
54.7
51.5
56.5
85.9
55.8
54.1
Box Thnkrr-Stis
83,0
86.7
81.4
12.5
81.9
81.1
56.1
88.6
80.02
80.5
80.3
07.1
86.0
66.5
60.6
69.6
61
$1.
Res ThineerCCo
56.6
87.3
82.3
86.5
87.0
84.3
79.6
81.7
77.2
85.3
83.8
00.3
87.6
89.5
87.2
85.2
65.0
53.3
07.3
Box-Tinker-GiOO
S8.5
86.7
84.1
87.1
87.
84.8
81.5
83.5
79.1
83.3
K5.4
82.3
88.0
09.1
67.2
66.6
66.3
R33
68.2
Table 3: In-domain evaluation results the Hu benchmark. R, P, and DF1 represent Recall, Precision, and DensityF1. The blod and underline fonts indicate the best and second numbers.
Evaluation Protocol. For in-domain evaluation, we evaluate our model on the HumanRef bench- mark, which consists of six subsets: attribute, position, interaction, reasoning, celebrity recognition, and rejection. Following (Jiang et al. (2025b)) we report Recall (R), Precision (P), and DensityF1 (DFI) scores averaged over IoU thresholds from 0.5 to 0.95. For the rejection subset, report the rejection score, defined as the proportion of 1,000 images where the model correctly outputs no bounding box when the object described by the referring expression is not present in the image. For out-of-domain evaluation, we evaluate our model on the RefCOCOg dataset and report accuracy at an IoU threshold of 0.5. We compare three variants: 1) Rex-Thinker-PPe Phain, which is trained on HumanRef-CoTT using SFT only on the final detection outputs, without reasoning supervision; 2) Rex- Rox-Thinker-CoT, which is trained with SFT on both the reasoning process and the final answer; and 3) Tex-Thinker-GRPO which is initialized from Rex-Thineee Thinker- Coc and optimized with GRPO.
5.2 IN- IN-DOMAIN EVALUATION RESULTS
We begin by evaluating in-domain performance on the HumanRef benchmark to assess referring accuracy within the person domain. As shown in Table 3 Rex-Thinker-CoT. trained with struc- tured CoT supervision, consistently outperforms Mecx-Thinker-i Plain across most evaluation subsets. Specifically, it achieves average improvements of +2.6 Recall, +0.1 Precision, and +0.9 DensityF1, confirming that step-by-step reasoning leads to more accurate and well-grounded predictions. Most notably, the CoT-trained model shows remarkable 13.8 point improvement in terms of Rejection Score on the rejection subset. This metric directly quantifies the model's trustworthiness by as- sessing its ability correctly abstain when no valid target exists. This significant gain indicates substantially reduced hallucination rates, critical capability for real-world applications requiring high reliability.
Further performance gains are realized through GRPO-based reinforcement learning. Rex-Thinker- GRPO demonstrates consistent improvements over Rex-Thinker-Col. achieving gains of +1.4 Re- call, +0.9 Precision, and +1.2 DensityF1. These results demonstrate the effectiveness of reward- based optimization in improving both the model's reasoning process and prediction accuracy. While supervised CoT training provides strong guidance, GRPO enables dynamic exploration of alternative reasoning strategies that better optimize for task-level objectives, leading to improved performance.
5.3 OUT-OF-DOMAIN EVALUATION RESULTS
To evaluate the generalization of Rex-Thinker to unseen object categories, we conduct experiments on the out-of-domain RefCOCOg dataset. We first adopt a zero-shot setting: the model is trained only on HumanRef- CoT and directly evaluated on RefCOCOg. Given a referring expression and its original COCO category label, we use Grounding DINO (Liu et al. 22024) to detect all instances of the target object category and use the detected results as box hints to Rex-Thinker.
As shown in Table 4 the CoT-trained model already performs competitively without any task- specific tuning. Further gains are achieved by applying GRPO for post training, demonstrating that reward-driven training enhances the model's ability to generalize beyond the training domain. Inter- estingly, we find that Rex- Thinkeinker maintains its structured CoT behavior even on novel categories. As illustrated in Figure 4 Sek-Thinnke--rre ray (with "fish" " bounding boxes as hints) while adhering to its planning-actionn soning paradigm. Notably, the model demonstrates self-correction ability: when provided with an
successfully generalizes to detect the fish of manta
rea-
8

---

Published as a conference paper at ICLR 2026
Question:
Please detect the fish of manta ray.
Fish 1
Answer:
<think>OK, the needs detect the munta ray. order accomplish this task, need to break down into the following steps: Step Identify fish- creature their appearance determine they are munta ray. Let's analysis:
Fish
Stop 1:
Identify each fisb-like creature based their appearance and determine it ray.
see four creatures in this image. Let's look at them one by one and put a dl if mania and X if not.
Fish I: It an orange and white striped fish, which is clown fish. X
Fish
Fish a brown fish with spiky features, which puffer fish. X
Fish 3: It has lurge, wing like structure on its back and a flat body shage, which are churacteristic features of
fay.
Fish It is blue-colored fish with a long tail, which is humpback whale (not fish). X
Summarize Re-Check answer
Fish A
Let's now recheck our unswer und pat V for the target fish and X for others.
Fish 1: It clown fish, not menta ray.
Fish puffer fish, not manta my.
Fish 3: is manta ray.
Fish humpback whale, not fish. X </think>
<answer>"
"soonaaf (oniif("bbee bbox 2d": [83, 894, 501, 1240], "label": "fish of manta ray" </answer>
Figure 4: The out-of-domain result. We use Rex-Thinker-GPRO trained on HumanRef-CofT to infer an unseen category (i.e., fish), resulting in strong generalization. Boxes in the image denote hints.
RefC0C00
Question:
Plense desect person holding small ball
BesSeck-188 laen
Grounding Diso
$7.0
Answer:
QwenVL-2.5577
67.3
$7.2
<think>
ChatRes-7B (oaign
89.8
90.0
There one penson holding small ball the image. penson standing the left group.
Tis-Tinkr--OTT
81.3
50.
Boo Chiok
Rex Thinker GRPO
K3.2
$3.3
$88
<Nhhikk>
<onswer>
[100, 174, 369, 821](290, 538, (75)_6625 839,
Table 4: Out-of-domain eval- uation results on RefCOCOg. "Fine-tuned on RefCOCOg using GRPO.
761]
<oinsmrr
Figure 5: Predictions from model that trained with GRPO only, without CoT-based supervised fine-tuning as cold-start ini- tialization. Boxes in the image denote answers.
incorrect hint label (e.g., a whale was incorrectly labeled as a "fish" in hint boxes), Rex-Thinker rectifies the error through logical reasoning and explicitly rejects the misclassification.
To further explore the upper bound of the model, we fine-tune Rex- Rex-Thinker-CoT using GRPO di- rectly on RefCOCOg. This leads to additional performance improvements, achieving results com- parable to state-of-ther thethhe-attttt referring models. The experiment results highlight the adaptability of our reasoning paradigm across domains and the effectiveness of reward-baar based optimization extending CoT reasoning to unseen categories.
5.4 ABLATIONS
Necessity of the Two Stage Architecture. This study directly justifies our two stage detector- reasoner architecture by quantifying the inherent limitations of standalone components for REC. As shown in Table neither open-set detectors nor MILMs alone achieve robust REC performance. While powerful for opca-vocab detection, general-purpose detectors like Grounding DINO struggle with the nuanced language comprehension required for complex REC. Even after fine- tuning on HumanRef, its performance remains significantly low (Avg. Precision: 25.7), confirming that standalone detectors are insufficient for the detailed language understanding essential for REC. Conversely, powerful MLLMs like Qwen2.5-VL-7B possess advanced reasoning but struggle with lower recall rate, caused by pixel-imperfect object localization. For instance, fine-tuned Qwen2.5- VL-7B yields an Avg. Recall of 69.4, a substantial drop compared to that of Rex-Thinke Thinker. This demonstrates that while MLLMs are powerful reasoners, their native localization ability is a key bottleneck for REC. Thus, Rex-The Thinker rigorously combines the perceptual strength of detectors (for initial candidate generation) with the cognitive strength of MILMM (for complex reasoning and

---

Published as a conference paper at ICLR 2026
Typo
Method
Fine-tuned
With Box Hint
Arg. Precision
Avg. Recall
Arg. DFI
Open-set Detector
Grounding DINO
No
15.2
87.3
9.7
Yes
No
25.7
92.0
14.8
Qwen2.5
No.
52.5
68.5
56.2
Yes
No
74.1
69.4
69.6
Two Stage (ours)
Rex-Thinkar-Psin
Yes
Yes
85.7
82.6
81.4
Table 5: Limitations of standalone detectors and MLLMs for REC on the HumanRef benchmark, justifying the necessity of two stage architecture.
verification over these candidates), thereby addressing the individual limitations of the detector and MLLM for robust REC.
Impact CoT-hased Cold Start on GRPO. Rex-Thinker, we adopt two-stage training strat- egy where the model is first supervised using CoT-annotated data, followed by GRPO-based rein- forcement learning. To assess the importance of this CoT-based initialization, we compare GRPO training with and without the cold-start SFT stage. We find that the model with CoT-based SFT significantly outperforms the direct GRPO model. Specifically, for models trained with GRPO, those initialized with CoT SFT (Avg. Precision: 86.8, Avg. Recall: 86.8, Avg. DF1: 83.5, Rejec- tion: 68.2, please refer to the Appendix for details) achieve substantially higher final performance compared those trained without CoT SFT (Avg. Precision: 82.0, Avg. Recall: 81.2, Avg. DF1: 77.8, Rejection: 66.4). This indicates that the initial exposure to structured reasoning pat- terns provides a more effective starting point for reward-driven learning. Furthermore, as illustrated in Figure 5 models trained without CoT supervision tend to generate unstructured or incoherent reasoning traces, lacking the verifiable and trustworthy qualities we aim to promote. In contrast, Co'T-pretrained models produce well-formed thinking steps aligned with our planning, action, and summarization framework.
6 CONCLUSION
We have presented Rex- Gex-Thinker, a novel framework that has reformulated the object referring prob- lem as an explicit Chain-of- of-Thhtght reasoning process to achieve grounded and interpretable predic- tions. Unlike conventional approaches that have treated referring as direct bounding box prediction, our model has first detected candidate objects and then performed step-by-step verification against the referring expression through structured planning- caaaning-action--e this paradigm, we have constructed HumanRef-Col reee-accion-suue summarization reasoning. To support large-scale dataset with reasoning traces that have enabled learning decomposed and interpretable reasoning patterns. Through two-stage train- ing approach combining SFT and GRPO-based RL, Rex- Tex-Thinker has demonstrated superior perfor- mance over prior works in both referring accuracy and rejection.
ACKNOWLEDGMENT
This work was partly supported by the National Natural Science Foundation of China under Grant 62403012.
REFERENCES
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073. 2024.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL. technical report. arXiv preprint arXiv:2502. 2025.
Guo Chen. Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier lasion-language models. arXiv preprint arXiv:2504. /5271, 2025a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Im 's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023.
10

---

Published as a conference paper at ICLR 2026
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109. 2021.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
The Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo. Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance bound- aries of open-source multimodal models with model, data, and test-time scaling, 2025b. URL wtt./as/////oooooooooooooooooaer abs 2412 05271
httpss://att //rrvv/ www.rxivv...oooooooooooooo org/
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open vocabulayy object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16901-16911, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 2025.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jac Sung Park, Moham- madreza Salchi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17 17146, 2024.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision- vision-language reasoming via iterative self-improvement. arXiv preprint arXiv:2503. 2025.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milmss arXiv preprint arXiv:2503.21776. 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint 1xXXi:2501.1294: 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, al. Seed1. 5-v1 technical report. arXiv preprint Sidv:2505.070622 2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 4rXiv:2412.16720, 2024.
Qing Jiang, Feng Li, Tianhe Ren, Shilong Liu, Zhaoyang Zeng, Kent Yu, and Lei Zhang, T-rex: Counting by visual prompting. arXiv preprint arXiv:2311.13596 2023.
Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision, pp. 38-57. Springer, 2024a.
Qing Jiang, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, Lei Zhang, et al. Chatrex: Taming multimodal Ilm for joint perception and understanding. arXiv preprint arXiv:2411.18363 2024b.
11

---

Published as a conference paper at ICLR 2026
Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision, pp. 38-57. Springer, 2025a.
Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Qin Liu, and Lei Zhang. Referring to any person, 2025b.
htrggabb//5500 25000
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Feng Li. Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12861-12871, 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.077 00789,, 2024b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-iman pre-traue pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975, 2022.
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang. Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-languag models. arXiv preprint arXiv:2501. 2025.
Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross- modality correlation filtering method for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10880-10883 2020.
Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via rl-zero-like training. arXiv preprint arXiv:2504.00882 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for spen-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint wwXiv:2503.01788 2025b.
Chuofan Ma, Yi Jiang Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual
tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404.130133 2024.
Xinyu Ma. Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Derek Wong, Xiaoyi Feng, and Maosong Sun. Deepperception: Advancing rl-like cognitive visual perception in milms for knowledge-intensive visual grounding. arXiv preprint arXiv:2503. 12797, 2025.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Mur- phy. Generation and comprehension of unambiguous object descriptions. CVPR, pp. 11-20, 2016.
12

---

Published as a conference paper at ICLR 2026
Brandon McKinzie, Zhe Gan, Jean- Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du Futang Peng. Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang. Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: methods, analysis & insights from multimodal LLM pre-training. arXiv: 1, 2024.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple apee--oocbbllayy object detection. European conference on computer vision, pp. 728-755. Springer, 2022.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hanmaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsumori Hashimoto. s1: Simple test-time scaling, URL
OpenAl. Gpt-4v(ision) system card. ctttpss/cchh..penaa.ccom/// Card. pdf
2023.
OpenAl, :. Ahmed Ei-Kishky. Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive programming with large reasoning models, 2025. URL https:
//arxiv.
06807
Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, al. Skywork rlv: Pioneering multimodal reasoning with chain-of- thought. arXiv preprint arXiv:2504.05550 2025.
Yanyuan Qiao, Chaorui Deng. and Qi Wu. Referring expression comprehension: A survey of meth- ods and datasets. IEEE Transactions on Multimedia, 23:4426 4440, 2020.
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, et al. Dino-x: A unified vision model for open-world object detection and understanding. arXiv preprint arX14:2411.14347, 2024a.
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng. Wenlong Liu, Han Gao, Hongjie Huang. Zhengyu Ma. Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405. 10300, 2024b.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models, 2024.
03300
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint 2504:25504.07615, 2025.
Mingjie Sun, Jimin Xiao, and Eng Gee Lim. Iterative shrinking for referring expression grounding using deep reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14060-14069, 2021.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 12599, 2025.
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-ol: Rethink- ing step-by-step visual reasoning in llm. arXiv preprint arXiv:2501.066 16666, 2025.
13

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing rision-lanuue model's ' perception of the world at any resolution. arXiv preprint 2rXiv2209.122199 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:231 1. 1.03079, 2023.
Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, et al. Skywork rlv2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint arXiv:2504. 16656, 2025.
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- gpee-vocabylary object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15254-15264, 2023.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12: Mixture-ord Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412. 2024.
Violet Xiang. Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards system 2 reasoning in Ilms: Learning how to think with meta chain-of-thought, 2025. httpst/:: arxiv. org/ abs/ 2501. 04682
oww.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self- rewarding correction for mathematical reasoning, 2025, URL
org/abs/
2502. 19613
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023a.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023b. URL https://arxiv.
org/abs/2310. 11441
Sibei Yang. Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression com- prehension. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4644-4653, 2019.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. arXiv preprint arXiv:2503. 2025.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang. Dan Xu, Wei Zhang. Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125 9138, 2022a.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Col- orful prompt tuning for pre-trained vision-languee models, 2022b. URL https://arxiv:
/200. 11797
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint arXiv:2310.0077
2023.
En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-rl: Pioneering perception policy with reinforcement learning. arXiv preprint 2rX44:2504.07954 2025.
14

---

Published as a conference paper at ICLR 2026
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, volume 9906, pp. 69-85, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 1307-1315, 2018.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 14393-14402, 2021.
Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring. arXiv preprint ar arXiv:2433.93333, 2024.
Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. In European Conference on Computer Vision, pp. 405-422. Springer, 2025.
Chao Zhang, Weiming Li, Wanli Ouyang. Qiang Wang, Woo-Shik Kim, and Sunghoon Hong. Re- ferring expression comprehension with semantic visual relationship and word mapping. In Pro- ceedings of the 27th ACM International Conference on Multimedia, pp. 1258-1266, 2019.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 2024.
Jingyi Zhang, Jiaxing Huang. Huanjin Yao. Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vi: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 1237, 2025.
Zhipeng Zhang, Zhimin Wei, Zhongzhen Huang, Rui Niu, and Peng Wang. One for all: One-stage referring expression comprehension with dynamic reasoning. Neurocomputing, 518:523-533 -322, 2023.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USE LARGE LANGUAGE MODELS
In preparing this manuscript, we made limited use of a large language model (LLM) to assist with language editing. Specifically, the LLM was employed only to improve grammar, clarity, and word- ing of sentences. No part of the scientific content, analysis, or claims was generated by the LLM. The authors take full responsibility for aspects of the paper's content.
A.2
MORE DETAILS ON CONSTRUCTING HUMANREF-COT
A.2.1 PROMPT FOR GPT-40
To annotate HumanRef- dataset using GPT-4o, we designed a two-part prompting strategy that addresses the diverse reasoning requirements across different subsets. This strategy consists of unified system prompt and a set of subset-specific in-context examples.
The system prompt is shared across all subsets and instructs the model on how to interpret the input, which includes an image, referring expression, and candidate bounding boxes. It also defines the expected format of the response, including the use of structured reasoning and answer tags. In addition to the system prompt, each of the six subsets in HumanRef-CoT namely attribute, position, interaction, reasoning, celebrity recognition, and rejection, is paired with collection of in-context examples. These examples are carefully curated to reflect the specific annotation challenges and reasoning patterns required for each subset. They guide GPT-4o in producing chain-of- chain-of-thought (CoT) rationales that are consistent with human annotations in both style and logic.
In the following sections, first present the shared system prompt. Then, for each subset, provide the corresponding in-context examples and visualization results.
Unified System Prompt. The system prompt instructs the model to perform detailed visual rea- soning based on either positional or attribute- based referring expressions. It emphasizes step-by by-step analysis, beginning with predefined reasoning steps (first attributes, then orientation), and requires the model to explicitly evaluate each candidate object. Special symbols are also used to denote matching, non-matching, and reference entities during analysis.
Subset-Specific In-Context Examples. After the system prompt, provide in-context examples to guide the model toward producing outputs aligned with our CoT structure. These examples help reinforce consistent reasoning patterns. HumanRef-Coi ToT includes six subsets: attribute, position, interaction, reasoning, celebrity recognition, and rejection. Each subset uses its own set in-context examples tailored to its specific reasoning needs.
We show the in-context prompts used for each subset, along with representative outputs generated by GPT-4o.
Subset
attribute
position
interaction
reasoning
celebrity
rejection
Inner
outer
Inner
outer
inner positon
attribute
position
position
interaction
interaction
reasoning
reasoning
Prompt
Figure 54
Figure
36
Figure
18
Figure
40
Figure 42
Figure 44
Figure 46
Figure 4N
Figure 50
Example
Figure
35
Figure
37
Figure
Figure
IT
Figure
Figure
45
Figure
Figure
Figure
Table 6: Ablation study on the retrieval- based design of our model. We compare performance with and without box hints to assess their impact on referring accuracy.
A.2.2 HUMAN EVALUATION OF HUMANREF-COT DATASET QUALITY
To empirically validate the quality of our HumanRef-CoT dataset and assess the effectiveness of our automated filtering pipeline, conducted comprehensive manual evaluation study. This study was designed to verify the logical consistency and factual correctness of the generated Chain-of- Thought annotations that used for model training.
Methodology: I) Data Samlping: We randomly sampled 600 instances from the final, filtered HumanRef-Coo dataset. To ensure representative coverage, drew 100 samples from each of the
16

---

Published as a conference paper at ICLR 2026
Given the image and the description below, oulput a detailed analysis of how found the object(s) matching the description about position or attribute.
The position description something like "the third man the left of the boy wearing red shirt". Then you first need find the reference person, i.c. the boy wearing red shirt, determine for each person of what you see and whether fulfill this reference condition, and put d if he is the reference persen or object, and X he doesn't. If this question is about attribure, you should provide a description what you see and how each object relates the provided description, and put a X or the end. You need do tuple like (amalyiaa(onallysi),, and then output your answer in tuple like (answerjansweer, Note that your analysis,
your analysis you need to start by listing your action steps. The action steps must be about attribute first, then orientation. And your first step docsn't need to be to find all the people in the dingram, since I've already provided you with all of them. For example:
To find the woman in red dress, I will need excuse the following steps: Stepl: Find all woman
SStep::
From the person in stepl. need to find all the person wearing red dress
Each step you initially planned must be strictly enforeed, and you cannot omit a step or modify the execution of cach step.
Note that I will tell you which objects you can't refer to the answer gave your answer. pretend that you are thinking about yourself. will highlight all such objects as marks in the diagram. Each mark has a circle and a number. The number represents the serial number of the object, and the color of the cirele represents whether or not current object matches the description, with green circles representing objects that match the description, and red circles representing objects that do not match the description. Note that you can't mention this mark in your answer. this mark just for you to and speciffy the corresponding person with the corrssonning serial number, you can person but not mark 1.
ground truth that this description, you need
the answer give reference. But
Your final answer must be consistent with the analysis, e.g. you say in the analysis that Person and Person satisfy the condition, then you must also say in your final answer that it Person and Person 2. the rare case that I give reference answer that is incorrect, you need to trust your own judgment.
Additionally I'll calculate the xy coordinates of each mark and then them in order from left right give you a positional reference the question is about position. But you can't It mention this order, pretend that you figure the order all by yourself. Every useful information will be provided in METAINPO
INCONTEXT EXAMPLES
The system prompt used to instruct GPT-4o on visual reasoning for HumanRef-Cof! It specifies output format, reasoning steps, symbol conventions, and the expected alignment between intermediate analysis and final answers.
Figure 6:
six distinet subsets (Celebrity, Interaction, Position, Reasoning, Attribute, and Rejection) 2) Anno- tators: The annotation task was performed by five Ph.D. students with expertise in computer vision and natural language processing. All annotators were given detailed instructions and calibration examples to ensure consistent evaluation criteria. 3)Annotation Interface: developed custom HTML-based annotation interface to facilitate the review process. As shown in Figure 7 the in- terface presented annotators with the source image (including the Set-of- visual markers), the referring expression, and the complete, generated CoT trace (Planning, Action, and Summarization). This allowed for a holistic review of each data point in its full context.
Error Categories: Annotators were instructed to identify three specific types of errors, ordered
from most to least severe:
Wrong Summarization:
The final set of person indices identified in the <answer> block does not perfectly match the ground-truth labels.
SuttonSSummarizatioon Inconsistency: A logical contradiction exists between the step-by-step evaluation in the "Action" "phase and the final conclusion in the "Summarization" " phase. For example, the Action phase concludes person is non-match, but the Summarization phase incorrectly includes them.
Wrong Action Result: A specific reasoning step in the "Action" phase is factually incorrect with respect to the visual evidence. For example, the model incorrectly identifies the color of person's 's clothing or misjudges a spatial relationship.
Results and Analysis The results of our comprehensive human evaluation on the 600 samples are summarized in Table 7 First, our automated, two-stage filtering process is extremely effective at eliminating high- logical and summarization errors. The human evaluators found zero instances of Wrong Summarization or Suriai-Summaizztiion Inconsistency. This provides strong evidence that the dataset is reliable in its final conclusions and overall logical structure. Second, we findw
17

---

Published as a conference paper at ICLR 2026
34/100
4o Output
tanelysis)
OK, the usernedds to detect the person who squating the picture.
the
them
Person A woman wecaiing light blue shirt, standing upright, holding
Person man in red shirt hat, standing upright, carrying
camera around his neci. X
Person
standing
Referring Phrase
upright, holding a veter bott X
Person woman in pink top and cap, standing uprigt, holding
the person who squatting
binoculars, X
Peraon woman bright yellow top, standing upright, holding something smail her hand. X
Review
hand he seauttic position, Perron 7: person in purale top, standing upright, with one arm towards X
son an
Wrong Summarization
knee ground
Whong Action Reouk
Let's now fha
Yos
No
standing
stattddda upright
3. Action and Sumnarization Inconsistent Yeee
Penson 31 standinn upright a X
standing
No
standing upright
Penson 6: in equartine posttoo
2: (analyski
+X
Staefidiics
(answer)
summarizs, person one who squatting.
(sntawer)
Privicus
hest
Go to page:
94
Co
Figure 7: A screenshot of our custom human evaluation interface. The tool displays the image with visual markers, the referring expression, and the full CoT trace, enabling annotators to efficiently verify each reasoning step.
Error Type
Celebrity
Interaction
Position
Reasoning
Attribute
Rejection
Wrong Summarization
Action Summarization Inconsistent
Wrong Action Result
Table 7: Results of the human evaluation on 600 randomly sampled instances from the HumanRef- CoT dataset. No high-level logical or summarization errors were found.
that small number of low-level Wrong Action Result errors persist in the final dataset (7 out of 600 total samples, or 1.2%). Upon manual inspection of these cases, we found they typically occur when a referred person is very small, heavily occluded, or when the visual marker itself obscures a key attribute, foreing GPT-4o to make reasonable but incorrect guess. Despite these minor, infrequent imperfections in intermediate steps, the overall quality of the dataset is very high. The significant performance improvement of our model when trained with this data demonstrates that provides a valuable and effective training signal for learning grounded, step-by-ster reasoning.
A.2.3 EVALUATE GPT-40 ON HUMANREF
Since we use GPT-4o to annotate HumanRef-CoT,, a natural question is how well GPT-4o performs directly on the HumanRef benchmark when prompted a similar style. To investigate this, we adopt setup similar to the annotation phase, using the same SoM-style prompt and a set of visual marks (with all marks shown in red). However, we remove any hint indicating which objects are correct. We then evaluate GPT-4o the truth answers. As shown in Table GPT-4o achieves an average DFI score of 53.2 without any hint supervision. This result suggests that while GPT-40 can be used generate annotations when given
Bencheaf--BBcarkkk without prompting with ground-
18

---

Published as a conference paper at ICLR 2026
the correct answer reference, its standalone performance without answer supervision remains limited.
Method
Amribute
DF1
Position
DF1
Interctioon
Reasining
P
R
Celehnity
DE1
Averue
DE1
Rejection
Table 8: Evaluation of GPT-4o on the HumanRef- Benchhark test set using SoM-style prompts without answer hints, The model achieves 53.2 average DFI score, indicating limited standalone performance.
A.2.4 ETHICAL CONSIDERATIONS AND DATASET AVAILABILITY
This section details the ethical safeguards employed during the creation of the HumanRef-Cof dataset and outlines our commitment to its public release to the research community.
Ethical Considerations and Safeguards We acknowledge the critical importance of responsible and ethical data creation. Our data generation pipeline was designed with a multi-layered approach to mitigate the risk of generating biased problematic content.
The source images for HumanRef-Col are from the public HumanRef dataset. These images have undergone prior NSFW (Not Safe For Work) filtering, ensuring that the visual content is appropriate and does not contain sensitive material.
Filtered Image Source:
Use of Moderated Large Language Model: Our data generation process utilizes OpenAI's GPT-4o to produce the reasoning traces. As a state-of-the-art commercial model, GPT-4o is subject to rigorous safety protocols and content moderation filters developed by its provider. These built-in guardrails are designed to prevent the generation of offensive, biased, or otherwise harmful content and served as primary safeguard our pipeline.
Final Data Review: Throughout our automated and manual quality control stages (detailed in Appendix A.5), where we verified the logical and factual correctness of the reasoning traces, we also remained vigilant for any inappropriate content. We can confirm that we did not encounter any instances of ethically questionable language in the final, curated HumanRef- Con dataset.
Dataset Availability To ensure full reproducibility and to foster future research in grounded rea- soning and model interpretability, we are committed to making our dataset publicly available. The HumanRef-Cof dataset will be released upon the publication of this work.
A.3 EXPERIMENT DETAILS
A.3.1 COT SFT SETTINGS
Table 9 summarizes the full training hyperparameters and computational cost used during the CoT SFT stage. These settings were applied in the cold-start phase without prior instruction tuning.
batch size gradient accumulation
maximum gradient
precision
bfl6
learning rate scheduler
cosime
epochs
learning rate
2e-5 AdamW 0.03
max length
2048
times
10.1h
optimizer
deepspeed weight decay
zero3
GPU
8xA100 100
warm up ratio
0.01
trainable module
LLM
Table Training settings and cost statisties for CoT SFT.
A.3.2 GRPO SETTINGS
We provide the training configurations used during the GRPO stage in Table 15| full GRPO training on the entire HumanRef-Con CoT dataset. Instead, training was terminated when the reward signal plateaued, indicating convergence.
We did not run
A.3.3 GRPO TRAINING ANALYSIS
We analyze the training logs of the GRPO stage. As shown in Figure g we visualize the changes in both reward signals and completion length throughout training.
19

---

Published as a conference paper at ICLR 2026
batch size
num rollout
precision
bfl6
gradient accumulation
2
3 0.04
epochs
0.25
learning rate
le-6
temperature
1.0
times
112h
optimizer
Adam W
deepspeed
zero3
GPU
8xA 100
warm up ratio
0.03
weight decay
0.01
trainable module
LLM
Table 10: Hyperparameters used during the GRPO training stage.
370
Figure 8: GRPO training curves showing accuracy reward, format reward, and completion length over time.
Thanks to the cold-start CoT initialization, the model achieves a reasonably high accuracy reward the beginning of GRPO training. At the same time, the format reward is nearly saturated from the start, indicating that the model has already learned to follow the correct output structure after CoT supervision. Meanwhile, the completion length remains stable at around 560 tokens throughout training. We attribute this to the model having already acquired the basic reasoning skills required for the referring task during the CoT fine-tuning phase, resulting in consistent output lengths with minimal fluctuation.
A.3.4 DETAILED EVALUATION SETTINGS
This section provides detailed breakdown the experimental settings and evaluation contexts for all baseline models.
In-domain Evaluation Settings: For the in-domain evaluation on the HumanRef benchmark, as presented in Table of the main paper, the baseline models fall into two distinct categories:
The majority of the compared models (e.g.. DINOX, InternVL-2.5-- 8B, Ferret-7B, Groma-7B, ChatRex-7B, Qwen2.5-VL-7B, etc.) were evaluated a zero-shot setting. We used their publicly available, official checkpoints to generate predictions directly the HumanRef test set without any fine-tuning on our dataset. This setup is designed to measure challenging, human-centrie scenarios.
Zero-Shot Evaluation:
their out-of-the- -of-tt--hooox generalization capability
State-of-the-Art In-Domain Model: The RexSeek-7B model is the only baseline that was specifically trained on the HumanRef training set. Crucially, it was not trained using any form of Chain-of-Thougee (CoT) reasoning, as it is direcct-rediction model. We report the performance numbers directly from the original RexSeek paper, as this represents the current state-of-the- for non-CoT methods this benchmark.
For the out-of-domain evaluation the RefCOCOg dataset, as presented in of the main paper, the comparsion settings are:
Out-of-domain Evaluation Settings:
Baseline Models (Supervised)) The reported numbers for baseline models (RexSeck- -B, Grounding DINO, VwenVL-2.5-7B. ChatRex-7B) are taken directly from their original papers. As per their publications, all these models were trained on the RefCOCOg training set. Fur- thermore, none of them are based CoT' framework. They represent the state-of-the-art for direct-prediction or retrieval-based methods on this standard benchmark.
Our Model (Zero-Shot): is critical to note that our Thinker models are evaluated a strict zero-shot setting on this dataset. They were trained only on the HumanRef-Cof dataset and had no exposure to RefCOCOg data or its object categories during training. This setup specifically tests the generalization of learned reasoning framework.
20

---

Published as a conference paper at ICLR 2026
A.3.5 DETAILED METRICS FOR ABLATION STUDIES
As shown in Table [I] the without-hint model is naive Qwen2.5-VL-7B, while the with-hint model is Plain. It is seen that the box hint plays an important role in REC accuracy. As shown in Table 12 CoT-baseed SFT as cold start is an important pre-preeee pre-procedure for RL exploration, leading to significntt improvement in REC accuracy and rejection. This is also evidence that our planning- aattionnsummaatizationnn paradigm helpful for REC.
With
Atributt
Posttion
Interaciion
Reasoning
Celebrity
Average
Rejection Score
Box Hins
R
P
DFT
K
1
DFT
K
DF1
R
F
DFI
DF1
R
DF1
No
66.4
74.3
67.2
69.3
71.9
69.5
65.2
72.1
66.1
63.6
67.5
62.2
82.1
84.6
82.7
69.4
74.1
69.6
71.7
Yes
Table 11: Ablation study on the retrieval based design of our model. We compare performance with and without box hints assess their impact on referring accuracy.
With Cold
Attribute
Position
Interaction
Reasoning
Celebricy
Averugo
Rejection
DFT
No
81.4
85.8
78.1
80.2
80.2
77.5
79.6
82.6
78.0
77.6
75.0
70.6
87.3
86.5
84.8
81.2
82.0
77.8
66.4
Yes
88.5
N8.7
84.1
87.2
87.1
84.6
81.5
83.5
79.1
N7.7
85.4
82.3
88.0
89.3
N7.2
86.6
N6.8
83.5
68.2
Table 12: Ablation on the impact of CoT-based cold start on final performance after GRPO training.
A.3.6 COMPARISON WITH THE THINK-WITH-MMN PARADIGM
Amnibbune
Position
Interaction
Roasoning
Ceichrity
Rejection
Qwen2.5 VL
49.1
71.3
35.4
50.2
61.7
528
48.2
06.3
532
31.5
61.2
403
80.3
31.9
90.
32.5
68.5
562
X.1
DespEyes-78
36.7
60.1
41.6
34.6
43.7
36.1
40.2
56.9
44.6
28.4
40.7
30.8
40.2
40.9
40.2
36.0
48.4
38.7
27.3
Res. Thinker GRPU
83.5
88.8
84.1
87.3
87.8
85.1
82.1
33.6
79.4
87.0
84.3
81.3
88.6
90.4
58.1
86.7
87.0
K3.6
67.8
Table 13: Detailed performance comparison on the HumanRef benchmark. Our Rex- Tex-Thinker-GRPO significantly outperforms DeepEyes, a representative "Think-with-iman model.
We evaluated DeepEyes-7B., representative model from the "Think-with-imag"" paradigm, on the HumanRef benchmark. DeepEyes is designed for complex visual reasoning, where its Chain-of- Thought process can invoke tools interactively modify the image, such by zooming in crop- ping specific regions to gather more detailed information. The detailed results of our comparison presented in Table
The quantitative results reveal a significant performance gap. Our Rex-Thinker-GRPO outperforms DeepEyes by large margin across all categories, most notably in Average Recall (86.7 vs. 36.0) and Average DFI (83.6 vs. 38.7). This disparity points to a fundamental limitation in applying general visual reasoning paradigm to the specific problem of REC.
Our investigation into the performance gap reveals that stems from a core methodological dif- ference between our framework and the frhhinkwitt--maaeee approach. As illustrated in Figure g DeepEyes employ holistic, tool-augmented reasoning strategy. The model identifies a few salient regions of interest and may invoke tools (c.g., cropping, zooming) inspect these specific regions in greater detail before generating an answer. While this interactive approach is powerful for general visual reasoning tasks, is ill-suited for REC, especially for multi-instance referring. We observed that after investigating only one promising candidates, the model frequently concludes its reasoning process prematurely, leading to a high number of false negatives and thus very low recall.
A.3.7 FINE-GRAINED PERFORMANCE ANALYSIS
To provide a more nuanced understanding of our model's performance and the specific impact the Chain-of- Chain-of-Thought framework, conducted a fine- fine-grained analysis on the HumanRef benchmark. This analysis compares our full model, Rex- Thinker-GRPo against a baseline version trained with supervised fine-tune -uninu but without the structured CoT data TRex-Thinker-Paane
The primary finding of this analysis, detailed in Table 14 is that the CoT framework provides a sub- stantial and consistent boost in performance, particularly in Recall. This effect is most pronounced in the benchmark's 's most complex categories, such as Reasoning (+5.29 Recall) and Attribute (+4.48 Recall).
21

---

Published as a conference paper at ICLR 2026
DeepEves Answer: <hhnkk> To answer this question, I need to identify the penson with loop By examining image, appears
individuals are wearing belts with loops, However, the person on the fär right has a bell with a visible loop. which is more prominent than image zoom in_ tool
the will the details of the belt-<hhinke> inctool callsin ("name"; "image zoom in, tool", "arguments": ("bbox (718, 38, 945], "label": "the person with loop on belt"))
person the far right confirm
<think-Thee image las boen zoomed person the right, confirming this penson indeed has belt with loop. Therefore, the person with a loop on their belt the one on the far rights</linkk>
GT: the person with loop on their belt
DeepEyes Pred.:
"onowers" "joniin ("bbox 2d": [718, 993, 945), "label": "the person with a loop their bekt "al"""""""""" <lanswor
Rex-1 Dii-eerbker-RPPOO Answer: <think=OK, the user needs us to detect the person with a loop on their beil. In order to accomplish this task, need break down into the following sieps:
Step I: Examino each person's belt to check for a loop. Let's start our analysis:
Step Examine each person's belt to check for loop 1 see four people in this image, let's look at their belts and put A if there loop and X if there not
-Person 1: He has helt with visible loop. A
-Person
with visible loop. Jb.
-Person 3: He has a bell with visible loop.
Rex-Thinker-GRPO Pred.:
visible loop. X
Summarize and Re-Check answer
Let's now recheck our answer and par the targer penson and X for others
Person 1: He
loop on his belt.
Person 2: He has a loop on his belt.
Person 3: He
loop on his belt.
Person 4: He does not have loop on his helt. X-/think>--anwwwr 'json
[["bbox_ 2d": [429, 788, 950). "label": "the person with their belt"), ('bbox 2d":
[210, 33, 539, 916), "label": "the person with a loop on their beli"), ("bbov, 2d": [689, 36, 9666,
918]. "label": "the person with loop on their belt")]
Figure 9: Qualitative comparison of Chain-of-Thought Thought styles between DeepEyes and Rex- -ei-Thineer- GRPO. DeepEyes uses holistic, aug-enegeet approach, identifying only one of multiple targets after interactively inspecting it. Our model uses systematic, exhaustive framework, evaluating each candidate to correctly identify all matching targets.
Attribute
Position
Interaction
Reasoning
Celebrity
Method
P
P
P
Rex- Mex-TTinker-Plain
84.17
89.89
85.48
87.18
82,16
87.80
82.63
85.34
88.58
90.58
88.65
91.98
87.42
88.68
81.95
88.11
87.92
87.71
89.41
91.30
Rex- GRPP
(14.48)
(42.09)
(+1.94)
(+1.50)
(-0.21)
(40.31)
(45.29)
(42.37)
(+1.13)
(+0.72)
Table 14: Detailed performance comparison between our full model (Rex- Thex-TTinkkr-GRPOO and non-CoT baseline
Plain) across HumanRef subsets. The results highlight that the CoT framework significantly improves recall.
This improvement recall stems directly from our core design. The structured Panning-Action Summarization process forces the model to systematically evaluate each candidate box against the expression's n's sub-goals. Unlike methods that might holistically reason and identify only the most salient object, our approach's 's step-by-step verification minimizes the chance of overlooking po- tential target. This inherently reduces false negatives (missed detections) and leads to significantly higher recall, as illustrated in the qualitative examples in Figure TO Figure
This detailed analysis also allows us to transparently identify key limitation of our current model. The "Interaction" category is the only subset where recall experiences minor decrease (-0.21 %). Upon manual inspection of failure cases (e.g., Figure 12), we hypothesize this due to the inherent difficulty of grounding complex, multi-entity relationships in crowded scenes with significant oc- clusion. Interaction" expressions (e.g., "the person talking to the woman in red")) require the model to reason about relationship between least two individuals. In crowded scenes, these individuals may be physically close, overlapping, or occluding each other. This can lead to the initial candidate boxes being imprecise, with single box ambiguously containing parts of multiple people. Our
22

---

Published as a conference paper at ICLR 2026
Res-Thinker-Paain Answer: [("bbox 2d": [ 154, 988, 727J. "label" "the person werring glasses") []
Rex-Thinker--e Answer: <think=OOK,
person wearing
glasses. In order to accomplish this tisk, need to break down into the following step:
Determine what each person wearing their face and find the person wearing
glasses.
# Step 1: Determine what each person weating their face find person wearing glasses.
GT: the person wearing glasses
suc len people not. glasses and
image,
they wearing
if
Person He wearing glasses, A.
Rex-Thinker- -hinier-Plain Pred.:
Person
wearing glasses,
Person He is not wearing glasses.
weasing glasses.
PPrson Hc not wearing glasses.
Person Not visible enough determine if wearing glasses. X
Person Not visible enough to determine if wearing glasses.
Person Not visible enough deterninx il wearing glasses. visible enough determine wearing plasses. Person 10: Not visible enough determine if wearing glasses.
So Person and Person are the people wearing glasses.
GRPP Pred.:
A Sammarize and Re-Check answer
Let's now recheck our answer and put for the target person and X for others
wearing glasses.
Person 2: She is wearing glesses.
Person He wearing
Person He not wesring glasses.
Person He not wearing glasses
enough determine wearing glasses. Person Not visible enough determine if wearing glasses.
visible enough deternine wearing glasses.
Person 9: Not visible enough to determine if wearing glasses.
Person 10: Not visible enough determine if wearing glasses.
[["bbox 2d": [551, 158, 804, 727). "label": "the person wearing glasses"), ("bbox_2d")
[786, 154, 988, 727), "label": "the wearing ginsses")
"
Figure 10: Comparsion betwee Rex-Thinker- Te--hhhh-nrPPPP and Rex- Bex--hinkerr-lan Chain-of ofoughttttttt im- proves attribute- based referring.
current CoT framework evaluates one candidate region a time. When presented with heavily oc- cluded or merged bounding box, the model struggles to definitively confirm the "interaction" based on the features within that single region. This can lead the model conservatively (and sometimes incorrectly) reject the candidate, thus slightly lowering the recall rate for this specific, challenging scenario.
A.4 LIMITATIONS AND BROADER IMPACTS
A.4.1 INFERENCE SPEED
While the CoT-based design improves both interpretability and performance, it also introduces addi- tional computational overhead at inference time. To quantify this, we randomly selected 100 images from the HumanRef Beeeee--BBcmaarkkk test set and compared the average inference time per image between RexThinker- Plain and Rex Thinker-GRPO.
All experiments were conducted using the vLLM framework single NVIDIA A100 GPU. As shown in Table 3, RexThinker TrPO exhibits slower inference due to its longer CoTT-style outputs. This observation aligns with the general principle of test-time computation, where improved inter- pretability and accuracy often come at the cost of slower response time.
model
Rex-Thinie Thinker-Plain 1.13s
Rex- Thinker-GRPO
average inference time
6.68s
Table 15: Comparison of average inference time between Rex Thinker--PPPP -lain and Rex Thinker-GRPO.
The CoT design in GRPO leads slower inference.
23

---

Published as a conference paper at ICLR 2026
Rex-Thinker- Piaker--Plann Answer[[ ("bbox 24" (340, 272, 449, 504), "label": "the person in the second from the bottom"), ("bbox_ 2d": (256, 272, 475], "label": "the
the second row from the botiom" ). ('bbox 2d": [476, 263, 561, 495). "label": "the penson in the second row from the bottom"" "
("bbox 20": [773, 262, 895, 496). "label"; "the person in the secund from the botion" "),
Rex-Thinker-CRPP thaner-think--O., the user needs us to detect the person in the second row from the bottom. la order to accomplish this task, I need break down into the following steps:
-Sep Idemiify the ruws of people image.
GT: the person in the second
-Stup 2: Deternine which row the second-row from the buttom.
Stup 3: Last
people the second from the bottom.
row from the bottom
Step I:
lddentify the rows of people in the image
children sitting holding papers, and the top row consists of children standing behind them also holding papers.
There rows people the imagee The
Rex-Thinker-Plan Pred.:
Determing which row is second-row from bettomn The socond-row from the bettomn the now of children standing behind the sittigg children.
Step 2:
Step 3:
List all the people in the secand row from bottom
Lets check each pesson and sce if they in the second-row from the bottom:
He
(silting),
second-row
-Person 2:
Shee is in the bottom now (sitting), not in the second-row from the botteen
-Person 3: She in secoddrrww from the botlom-
row from bottom
-Person 5: Shhe is in the second-row from the bottom
Rex-Thinker-GRPO Pred.:
-Person Shhe in second-row from bottom
botion row (sitting), not the second-roo row from the botion -Person He the bottom row (silting),
-Persun 7: He
the sccond-row from the botton secord-row the -Pesson 10: He the bottom row (sittin)), not the second-row from the bottom
-Person He the
(sitting),
Person 11: She is in second from buttom
Person
bottom (sittin),,
second-row
bottom
Person 13: He the bostam row (sitting), not in the second row from the bottom second-row from botton E
14:
Summarize and Re-Check
now rechock answer pur
target penson X others:
..robbikk--answer 'json
[["bbox 2d": [340. 449, 504). "label": "the
second from bottom")
["bbox 2d": (256, 272. 344, 475),
"label": "the person the second row from the botiom" ).
("bbox 2d": [476, 263, 561, 1951,
"label"; "the person the second row from (he botion"
496],
person the
("bhox, 2d": [566, 265, 659, 549).
"lahel";
"the person the second row from the bottom" I.
("bhox 2d": 256, 790, 513). "label": "the person the second row from the bottam")
Figure 11: Comparsion betwee Rex-Thinker-GRPO and Rex-Thinker-Plaie Chain-of Tfoo-hugh-tttttttt im- proves position- based referring.
A.4.2 INCONSISTENT REASONING
We observe occasional inconsistencies between the model's reasoning process and its final output. For example, as shown in Figure [3] the model identifies nine candidate objects the reasoning phase but only includes eight in the final predicted coordinates.
We attribute such inconsistencies to the lack explicit supervision enforcing alignment between the reasoning chain and the final answer during GRPO training. While most predictions remain consistent, these rare cases highlight potential gap in our current framework. One possible solution is introduce consistency reward, which evaluates whether the number of objects summarized in the reasoning matches the number of bounding boxes in the final answer. This direction remains open for future exploration.
A.4.3 BROADER IMPACTS
Referring models enable more flexible and natural interaction with visual scenes compared to tradi- tional object detection, as they can understand complex, context-dependen language. This makes them valuable in applications such as surveillance, smart cities, and smart homes.
However, these models also raise concerns. particular, they may expose privacy risks when deployed in sensitive environments and can inherit biases from training data user input. Although CoT reasoning improves interpretability, ensuring its consistency and robustness remains an open challenge. Future work should address these risks ensure safe and responsible deployment.
24

---

Published as a conference paper at ICLR 2026
Rex-Thinker-PHar Answer: [[ [('box (270, 809). "label": people holding hands"), ", ("bbox 2d": (627, 294, 892, 784). "label"; "twe people holding hands"
Rex-Thinker- Answer: OKhnk-
the parents holding girl.
In order to accomplish this task, I need to break down into the following steps:
Step Identify
girls among people.
-Step 2: For each girl udentified in Step I, check if there is parcat-like figure holding her.
GT: the parents holding the girl
Stop I: Ideniify all the girls among the people
with white lace details. A. -Person 2: A bay wearing red and white checkered shirt.
1: young girl wearing light-blue
Rex-Thinker- -Pakk Pred.:
Person 3: A wearing blue sports
with "Fly Emiirates" written it. X
Person 4: A buy wearing yellow and blue striped shirt. X
Person St A
wearing gray and blue patterned shirt. X
Person 6: A young girl wearing a white top with cartoon design.
Person 7: A woman wearing light-blue
Person man wearing striped shirt.
So Penson Penson 6, and Person are the girls among the people.
Step 2:
For each girl identified in Step chock there parent-lias like figure bolding her She sitting and not being held by anyone. X
Penson I:
Person 6: She standing close to an adult (Person 7) who appears to be a parent- figure and has her hand near Person A..
Person 7:
standing next
scems parent-figure
Bex-Tinker-GRPO Pred.:
prosimily and bas prokective stance, A
Summarize and Re-Check pes
recheck
target person X
Person I: Not heing held by parent.
Person A boy, relevant
description
Person 3: A boy, not relevant to the description.
boy,
the description.
Person S: A boy. not relevant to the description.
Being by a paront-like figure (Person
Person 7: A parent-like figure holding Parsen X-sthink><casswer> "json description.
man, relevant
[["bbos 2d": (523, 121, 701, 540), "label": "the parcats bolding the girl", ("bbox, 2d":: (665, 75,
517], "label": parents holding gist")]
"*s/anwwwr>>
Figure 12: A failure case of Sox-Thinker-GRPPO where CoT leads to wrong answer.
A.5 VISUALIZATION RESULTS
In this section, we present qualitative results of the RexThinker model across different scenarios, as shown in Figures 33 All visualizations generated by the RexThinker GRPP model, which trained solely on the HumanRef-CoT dataset.
25

---

Published as a conference paper at ICLR 2026
Question: Please detect persons backgrouad.
Box bint
Anserr
<think=OX, needs detect all persons bockground. In oeder accompllssh uskk, I need to bresk down into the following sopp:
Step Determine the position each person relative the foreground background distinction. our analysis:
Step I. Determine the position of each person relative to the foreground-back grddd-ekkkenndddd distanction and put Ed it
see 18 people this Let's analyze their positioss the background
they are
-Prson She the foreground, sitting and holding X
Penson 10:
background, purtially visible behind
-Pesson 11: He is the background. partilly visible behind Penon
-Penoo 12: He is in the background, partially visible behind Penive
background, partially visible behind Person
-erson 14: He is the background, purtially visible bchind Person
Penso 15: lle the background, partially visible behind Person
-erson 16:
background, partially visible behind Persoa
Penson 17:
background, partially visible behind Person
Penson 18: He is in the background, partially visible behind Person
Summmarize
Check answer
Let's now recheck our answer and put for the target person and X for others Person 1-9: in the foreground
-erson 10-18 In backgrond
<hhink>
"answer"" "jsod] ( ("bbox
(916, 200, 952, 420). "label": "all persons background")
("bbox, 20t [750, 296, 780, 405), "label"; "all persons in the bockgrund""; ; ('bhox 2d"; 1769, 307, 790, 4013, "label "all persons the background"), ( ('bbox (0": [939, 300, 963, 418],
311) "label": the background" 1, /'bbox, 2d" (678, 366, 704, 103). "label" "all pensoes in the background"), ("bbox 2d": [571. 361, 290, 400). "label": "all pensons in the bockground").
"label": pensons
backgroun")) 1. ('bbot 28"; [125,
persons
('bbox 32" (147, 308, 159, 338). "labe!" persens bockgrund") []]
Figure 13: Example of reasoning tween reasoning and the final output.
-answer answer mismatch. The number of predicted objects differs be-
Box bint
Question: the youngest one.
Answer:
OKh the user needs us to detect the youngest one. To accompllish this task, I need to break down
into the followang steps:
Step Detemmin age-related sttributes each pemson
Step Compare the ago related atributes find the youngest one.
Determine the ago ago-related attrihutes of eachc person
Person A young girl light-ooloead hair, wearing a white outfit She sppears
child A.
-Pesoon 2: A woman with long brown hair, wearing light-coloral top and while pants She is an adult. X
Person A with short hair, wearing beige soeater and brown pants. He adult. X
Person A young boy light-colored hai, mearing pattened shirt He appears to child.
Answer
Step Compare the age related attributes to find the youngest person
Among the four people, Person and Persoe are children, while Pesson 2 and Person are adults. Person they
Comparing
be similar young-aps, hased visuall like height and overall appeorance, Persun appears to be slightly younger.
Summarize and Re-Check
rechook answer and put for target person X others Person I: She is yoing gid but not the youngest. X
Person, 2: She adull. X
an X
Person 4: He is the youngest one. -hinkk
Sanswers"" "json[["" ('bbox 20" (298, 94, 597, 410), "label": "the youngest one";] ""</anwwr>>
Figure 14: Attribute referring example.
26

---

Published as a conference paper at ICLR 2026
Question:
Please desect the parents.
Box bint
Answer:
<think=OK, OK. the user needs detect the parent. in order to accocuplish this task, I need break it
down into following stope:
Step I: Detenmine the gender age-related attributes of each person idantiyy adults.
Step 2: Among the sdults, desermine which ones are likely to be pirents bosed on their interaction with children. Let's start anaysis:
Step I: Deternine the garder and age-clated attrihutes of each pensen to idensity adults
Persion E: younng girl with long hair, wearing white clothes She appears to be a child X long hair, wearing light colored top and white pants. She an adult-like
Person 2: woman
appearance.
Person 3:A man with short hair, wearing beige swester and brown llle has adult-like
appearance.
ABSWET
Person A young buy with short hair, w wearing checkered shirt. Ilee appears to be a child,
Stup 2: Among the adults, determine which ones are likell to be parems based on their interaction with children
nurtating like posture with asound girl (Porson 1). Person is also in close prosimity to the children and has father-like role as lhe interacting with the boy (Ponon 4).
Person sitting
Summarize and Re-Check answer
Let's new recheck our answer and put E for the target person and X for others parent. X
Shee is child,
Person She an adult and inceracting with children in purental way.
Person He is an adult and interacting with children in a parental way. &
He child,
parest. X-thhin=-
<answers"" "json] "joon[""bxx ("bbox, 28"; (545, 1093, 740). "label": "pareats"), ("bhox 2d": [62, 95, 757).
"lebel" "parents"]] 11" "chmmmer>>
Figure 15: Attribute referring example.
Box
Question:
Please detect apple with discase, and tell me what idlnd discase may be..
Auswer:
<thinlo-KK, the user needs detect the apple with disease and tell what kind of discase it may be. In
order accocaplish this task. need break any visible signs of disciie. -Sepp Examine each apple determine
the following steps:
Step 2: If apple found to have disease, identiyy the type of the discase. Let's start our analysiis:
any visible signs disease see five upples this image. Let's look them une-by me--by-ome and put there are signs of dliscuse
Step Examine apple determine
and X there are none.
Apple I: appears to be bealthy green apple with no visible signs disease. X
Apple It
bealthy-Jooking green apple with obvious signs discase.
Apple 3: This apple looks like a heality green apple with no clear signa of diserse. X
Apple 4. Asodlee healthy green apple with visible signs of diácnso. X spot surfice, which indicates
Answer
Apple This apple his
has discrsse,
Step 2:
Identify the type of the disease
brownish spor Apple suggests that may have fingall hactorial infection, which common ty pee diseuses for apples.
Summarize and Re Check ansver
Let's now recheck
answer and put for target apple and X for others:
Apple 1: It is healthy apple with no signs of disease.
Apple 2: is a healthy apple with signs of disease.
Apple 3.
healthy apple with signs discase.
Apple 4: It is healthy apple with no signs of diseser. X
Apple 5: It has brownish spot, indicating possible fingaal or bacterial infection. Ehiiiiik
sanwerr ( "boox 2d": [429, 261, 570, 395], "label:: "apple with disense" 1]" <lansmer>
Figure 16: Attribute referring example.
27

---

Published as a conference paper at ICLR 2026
Question:
Pleuse desect ripe tomate.
Answer:
<think>OK, the user noeds detect the ripo tomatoo In order accoroplish this mokk need to break
down into the following
-Step Detennine npenees tamato
Let's start analysis:
Seep I: Determine the ripeness of each tumano based on its color tieir colors and put A. ripe and X it
image,
not
Tomato It has deep red color, indicating it is ripe tomato.
Tomato It also has deep rod color, indicating is ripe tomato, A. fully cipe X
orange
typical
Tomato It green color, so it not ripe tomato.
Tomato S: It green color, it not tomato.
Answer
Tomato
color, not ripe tomato.
tomato.
Tomato green
Summanize and Re- Re-Check
necheck Tomato lt is rge tomatio
put for tirget
Tomato 2: It
soatato.
Tomato
Tomato It not ripe tomato
Tomato S: It not ripe tomntoo
Tomato not tomato
Tomato
<answerl ( "bbbxx, 20th [255, 35x, 565, 657), "labet": "riee somnato"), ". ("bbox_24" 2d": (681, 455, 744, 726).
"label": "ripe tomiato") <janswer>
Figure 17: Attribute referring example.
Question:
Please diseet damaged container.
Answer:
<think>OK,, the user needs detcet the damaged container. In order accomplisti this task, need braak
following steps:
Step Examine each of 31 comtainers determine they have any visible signs damage.
Step Step I: Examine euch
containers desermine have any visible of
damage:
Conisiner I: appears to be good condition with no visible signs damago
Container 2: There no visible sign of daraage on this container. X
Contsiner This
has large hole side, indicating
damagee d.
signs dimage. X
Container 31: No visible of damage. X
Summarize and Re-Chrck unswer Let's recheck snower and For the container and X for others
signs damage
Container 2: No visible signs of dumage
Comsiner 3: Has a linge hole, is dhmaged
Contziner ft No visible
of damage X
Contiiner 30: No visible signs of damage Continer 31; No visible signs of damage
<answerf "Woox, 24" [288, 183, 609, 459), "Inbel": "damaged containe:"]] <lanswer>
Figure 18: Attribute referring example.
28

---

Published as a conference paper at ICLR 2026
Box bint
Question:
an even number number plates.
Answer:
the user to detect athicies even-mmnteere mumber plates, accomplish
this task, need to brenk down into the following steps:
Step Identify the number plate each uthlete.
number
it Step I: Idencify the mumber plate tach athlese
I see athletes this imuge. Let's look the number plate each them:
number plate "530".
number plate
AAtdete His mamber plate reads "660".
Aihlete Iis number plate reads "267".
AAhlene 3: Hiis mumber plate reads '485".
Athlete There is no visible number plate this athlete.
Answer
Steo Determine number plate
AAhlete The number plane "510" even
AAbleee The number plase "660" is even
-Atilete mumber plate "435"
X
AAbleto mumber plate
even
Athlete The number plate "26"" is odd.
AAhleete Since there visible number plate, cunnot determine if odd, but based
requirement of
numbered
consider meeting condition. X
Summarize and Re-Chre mewwer
Tet's rechock our anower and
2 the tanos athletos and X others:
-Atlleic
even-munberod AAblete His an even comn-mmmened number plane
place Athlete llas numbered number ?
plate.
Athlete 3: His an edd numbered number plase. X Athlete 6 No visible mimber plate,
<ithink>
<answer [ ('bbox 20":
616),
even number
planee" "...."
"slanserr>
Figure 19: Attribute referring example.
Question:
Please detect two people holding hands.
Answer:
<think~OK, the user needs us to detect two people hokting hands. To accomplish this task, need
break down into the following stepe:
SSeep Idautify pairs of people engaged hand-holding
A Step Identify of people engaged hand holding internction.
peopli this image.
intoractions:
Person 3: laking photo, so hand- holding interaction
Parson Standing near others, no hand-holding interaction...
X
Holding
Person & Holding hands with Person
Parson 9t. Sitting with hand-holding interaction.
Standing others, land-holding
Summarize Re-Check
recheck answer put
torget people X odfien:
Person 3: Not involved in hund-hull hulding inseraction
Person Not involved hand holding interaction
Person involved lund-holding interaction
Parson 4: Involved in hand-holding interaction with Penson
Person 6 Involved a hand holding inseraction with Penson
invalved
hand-hhllding interaction
Parson 9: Not invalved in hand-holding interaction
Person Not involved hand hooddng interaction
Canswers" "joonf ["soo[[""boxx ("boox 20" [169, 161, 591], "labe": "Twe people holding hamds") "
("bbux (259, 127, 400, 450). "label": "Two people holding hands")]" <lanswer>
Figure 20: Interaction referring example.
29

---

Published as a conference paper at ICLR 2026
Box hint
Answer
Question:
Pleuse detect car crash
Auswer:
<think~OK, user noeds to detect the cars that are in crash. In order to accomplish this lask, need to break down into the follow illlwwing
stops:
Step Examinc each 22 cars determine they are involved in crash. Let's start our analysis:
Stop I: Examine each of cars to determine they are involved crash
CCar 1: This is clearly damaged and surrounded emergency vehicles and people, indicaring it cruch, A.
energency-reelag activity, Car 3: There are no visible signs of damage or being involved in an accident like situation. X
damaged
proximity
damaged
Appeans undamaged not part any crash-related activity. X
Car S: Is damaged and is among other damaged cars, so it crash. A. signs being in crash. X
6: There are
visible signs being in crush. X
Summarize
answer
Let's now recheck our answer and put for the target cars and X for others
Car 2: Is in crash
crash Car 4: Is not in crash
7
crash X-<think-
[["bbox_ 2d": [401, 208, 303). "label": Ycar in crash"). ). ("bbox_2d") [252, 364, 362), "label": "car crash"). ("bbox__at 2d": 2177 309), "label"; crash";]
""<lanswer>
Figure 21: Interaction referring example.
Box hint
Question: Please detect people with something their hands,
people somethhing their accomplish
this task. need to break down into the following stops:
SSepp Check each penson see if tbey have something their hands.
Slep each person see they have acmething
hands
sse 18 people this image. let's check what is ench person holding and put d. if they have
something in their hands and X they don'. X
nothing
Penson
A. Person She has nothing in hes hands, X
bolding
-Pesso B: He is holding camera. A.
-ersonn S: Hi has nothing his hands, X
Ponson He is holding ice-eneam. A.
nothing
X
holding lollipop.
-erson 9: Shte holding car.
Answer
Summmarize Re-Check
Let's now recheck our answer and put for the torget people and X others
Penson I: He has nothing in his hands. X
Person She bolding book. VV
Penson holding
She nothing
X
nothing
Penova He is holding an ice-cream.
Penson 7: lle has nothing in his hands. X
Person He is holding a Iollipop. V
Person She
K <ithlak>
"jum]
(999, 271, 811], "label:: "all peuple
hands"). ). ('bbox, 20t: (0, 231, 178, 783), "label": "all people with something in their hands" "
("hbox. 2d" (259, 127, 400, 450), "labet; "all people with something in their hands").
people something their lands") ("bbox 24t: [175, 411, 168, 811), "label": "all people with something in their
[768, 485, 956, 811],
hands"]] <laaserr>
Figure 22: Interaction referring example.
30

---

Published as a conference paper at ICLR 2026
Box hint
Question: person holding footballs
desect person bolding two
accomplish
task, I need break i down into the Sallowing steps:
Step Determine what each person holding.
Find
holding footballs.
anclysis:
A Step Determine what ench penson helding
1 see people in this image, let's look what they are holding and pur d if is two footballis and X
if Person
holding anything
Person not holling snythine
Parson 2: He holding anything.
Person 12: holding foothall. X
Answer
Person 3: llle not holding anything.
holling footballls
holding seything.
Person the person holding two footbaiis.
Summariin
answer
Let's now rochook our snswer and put for the target person and for others
Person I: He not holding any unything.
liolding anytlnng.
Person not holding anything.
Person f:: lile holding two footballs.
<answer[["
('box ("box <lanswer>
2d" (652, 49, 791, 412J, "label". "person holding two foothalls") [
Figure 23; Interaction referring example.
bint
Question:
Please detect the leader.
Amswer:
<think-OK, the user needs detect the leader. In order accomplish this task, need to break down the following
SSep
positions Step Determine which car in the lead position among themm.
allll 18 There une 18 cars visible un the iruck the image. They are spread out across diffirent
Stop
track
positions on the circait, some closer to the inner-lme and some near the outer-liee
Step 2: Determine which car the lead position among them
layout the positioned ahead the other cars, occupying most forward--aa most sput on the track compared to the others.
the
the
Summarzz and Let's now recheck our answer and put for the target car and for others
answer
xxx
not lead position
lead position
Car [3: He not the lead position
the lead
position
Car 16: He not the lend position
Cur 17: not the lead position
position
<answers" [[[
"thox 2d"": (1177, 579, 1380, 624), "labe":; leade" "f" "lansswerr>
Figure 24: Position referring example.
31

---

Published as a conference paper at ICLR 2026
hint
Question:
Please detect all person the right of person wearing a yellon the
Answer:
-think- OK, the user needs us to detcet all persou to the right of the person wearingg yellow tic. order
accomplish this task, need to break down into the following steps:
weasing yellow Step 2: Identify all the people the right person found Step
Find
# Stop Find the person wenring yellow
Pemon people this
cleck
and pat d. yellow and X
wearing redrosloeed uniform visible X
Person 2: Ille wearing red colored uniseem and so visible tie. X
Answer
Person 3: He wearing dark sui with red tic.
wearing
Person 5: Il wearing dark suil with a yellow
Person & He wearing dark suit with green X
Person 7:
red colored unifomm visible tic.
So Penson S the person wearing yellow tic.
people
The arder from left right (Person Penson 5, Penso Person 4, Pensca Person 2, Person LJ.
People to the right of Penson 5 are Person 3, Person 4, Person 6, Person 2, Penson
answer Let's now recheok our answer and put the target people and for others Person I: Sat to the right of the person wearing yellow tu.
Summarize
Person
persan wearing: a yellow (seference person).
Person & He the right of the person wearing a yellow K wearing yollow tic. X
Parson 7: She the left of the
<Wthink>
<answer "joon[" "bbbx 24": [338, 323, 1028, 783]. "labe!". person the right the penson
weating yellow tie" I..] <lanswer>>
Figure 25: Position referring example.
Box bint
Question: dogs on grill
Answer:
<think OK, nceds detect dogs the grill, accompllish teok, mood break down into the following stepe: Determine the
each
the
N Step Determine position euch hot dog in relation to the grill
Let's analyze the position of each hot
Hot It placed
of
table, not the grill.
dag 2: also piece paper sable, not grill.
Hot dog 3: It placed on the metal grill.
Hot dog 4 is placed on the metal grill.
# Summarize and Re Check answer
Let's now recheok our nnswer and put the target hot-dogs and X for others:
Hot
the grill. X
2:
Hot dog 3. On the grill.
Hot dag 4: On the galll
"answer" "ison] "ebux, 2d": (782, 118, 1070, "label" "Hot doys on the grill"), ("bbox, 20f% 1901, 281, 1284, 550), "aabe"" "Hoi dogs on the grill!!" """"""""""onn <lunswers
Figure 26: Position referring example.
32

---

Published as a conference paper at ICLR 2026
Box bist
Question:
Please detect Geuffry Hintoon say what he doing.
Answer:
<think=OK, the user needs us to detect Geofirey llinton and say what he doing. In onder break dowa following steps:
accomplist: this
Stop Identity Geaffry Hinson amsong people.
Step 2: Detemine what Goodfey Hinton is doing.
Step Identify Geoflrey Llinion among
people
ace 68 people in the image. Based on general knowledge of what Geottrey Hinton looks like (an
older man with gray-uhite hair), I nead check each person ono-hy-one.
Person
older
gna-whitt
dressed fornial slaking hands
another person.
general appearzzee Giodfryy
PRIZE
Person 2: He also an older man with gray white hair, dressed in formal wear, shaking hands with
Penso Dut he is Geoline llintom. X
Anewer
remaining people, by observing their appearance and position nolutive the
interatioon, they do not mutch the expected appeamace of Gloffleey Hinton. X
Step 2: Decermine what Geoffrey Hinton is doing
moother individual. He appears be slakinn hands exchanging something Penson This action scems part fornal event, likely rolated
ougaged
the "NODEL PRIZE" as indicated by the test on the floor.
answer
Let's now rechock our answer and put & the tnrger pesson for others
Person 1: He Goufhhy Himeon and shiking hands, participating in fermal event.
Gcoffrey Hinton. X
For the remaining people: They are not Geoffrey liiooon <ithink
PRIZE
<amswers"" [("buxx 2d': [391 43, 652, 7063. "label": 'Geofiny llinton and say what he is
doing"))
Figure 27: Celebrity referring example.
Question:
Please detect all non-real persons.
Answer:
<think=-XK, the user noeds to detect all non-renl persoos. In onder accompplish this task, break
following steps:
Determine which people the image are real or not based on their appearance and context.
Step i:
Step 1: Determine which people the image are real or not bused on their appearance und context. see d8 people this image. Let's analyyee each penson one one:
formal attire life participam: the eveni.
He
with another penson. appears
PRIZE
Person 3: He man standing the audisnee area, He seems to be real-lid' attendes. X the background. Person 13: part the podium-like structure, not real person.
Answer
di.
Summarize and Re-Check answer
Let's now recbeck our unswer and put Y for the target person and X for others
He real person.
Person real person.
Person 4: It status, not person.
13:
the stuueture. real person.
<ithinl>>
<answers" "joonf ('bhox 3/": (962, (562, 149, 619, 619,433) 433], "Inbel" "all non-rcal parsons"), ), l'bboxx 2d": [850, 382, 908, 162), "label": nom pensons") "chammer>
ISU PRIZE
Figure 28: Reasoning referring example.
33

---

Published as a conference paper at ICLR 2026
Box lilt
Question:
one
Answer:
<thinloOK, the user needs us to detect the one with rich Vitamin C. In order to accomsplist this task, need break it down into the following steps:
Identity fruits vegusables items the image agsinst
knowa be Vitamin C.
Step Chack each our analysis:
and vegetables. Let's
Step Identify
and vegetables that
rich Vinain
Fruits and vegetables rich Vitamin C includo citrus fruits like oranges and lemonn, limes, bell pepper,
broccoli, and some leaty greens.
Stop 2: Check ench of the 46 items in the image against the ideatilled fruits and vegetables
Avocado,
Vitamin C X
-iem Time, rich Vitamin C A.
Beatioot,
Vitamin X
Lime,
Vitain C.A.
Answer
Item II: Kale, rich Vitamin C A.
Item 13: Broccoll, rich in Vitamin Item 16: Lime, rich in Vitamin C A.
Item 1S: Carlic, not rich Vitamin C:
Oragge, Vitamin A..
Orange,
Summarize and Re-Cheek
at's rocheck anower
target and X
Item Lime, rich in Vitamin
-tem Lime, dich Vitamia C
Them II: Kale, rich Vitamin
Reacooli, Vitamin
16:
rich Vitamin
Lemson,
Item 21: Onnge, rich Vitamin
Foom 24: Grapetruit, rich in Visami C
Item 29: Orange, rick Vitamin C
Others: Not rich Vitamin C X <hhhnle>
<answer>" [[ ["bbox, 2d": [599,
743). "label": one
C*) ...] <'answrr>>
Figure 29: Reasoning referring example.
hint
Question:
Plewe detect the dog slrop on the hed with pot under its body
Auswer:
<think OK, the user needs us detect the dog sleep on the bed with pot under rls body. To accomplish this need break down following stepe:
Step the dogs that sleeping
- Step 2: From the dogs found step I, check there is pot under thear bodies. they sleeping
sleeping bed Find the image. look pocitions the bed and X not.
Dog I:A brown dog lying on the had, appeass to be sllcpping A.
dark- cark-colered
appears be sleeping
Answer
Cat lying on hed,
dog
Dog
dogs sleeping bod.
V Step From dogs found in step check if there is pot under their budies
Dog I: There no under its body.
Dog There
under its body.
However, if we consider the overall description and the fiet that the reforence for the existing condition mentinn only
ro-evalinte. Since the referencs only includes Dag we foeas on Dog
pot under body. there dog that the deseription as stated, but we consider the reference related aspect, only Dog I relevant among the sleeping dogs on the bed.
dog sleeping
answer
Let's now recheok our asswer and put the target person and for others
Dog A dog sleeping on the bed but no pot under its body -
Dog sleeping the bed but pot under budy
dog
<Whink>>
Sanswers/hhee such thing in the imagge-/answer>
Figure 30: Rejection referring example.
34

---

Published as a conference paper at ICLR 2026
hint
Question:
Please detect person wearing cloth that has letter
Answer:
<think> OK, the user noeds us detcett the person wearing eloth that has letter A. To accocoplish this task, I need break it down into the following steps:
image. Step Check the clothes each person to see there the letter on them.
Find the people
TEIR
It I: Fini all the people
DONATE
people in image, from lefi-tu-right penson pemion person person person 2, persoon 7, person I, penson
Check clothes
the them..
Answer
4:
Wearing pink
ward
no
Pensn 6: Weering an orange dress with the word "VOL" on letter A
Penon 5: Weaning blue shirt with the word "L" no letter A
Penso 2: Weating pink dress with
"L"
letter
3: Wonring yellow shirt
the
letter
-Penon 7: Weering a bnown shirt with the werd "TT on it, no letter
ER
Penson 1: WWeuiin blue shirt with the word "ER" on letter A
blue insãde
penson visible
wearing clothes
a letter A, and among the people visible, none have letter A on their clothes.
We also
person
group
the additional information,
they wtll not
their elothes.
# Summarize and Re-Check answer
rocheck
put targel person X others
Person 6 no letter on clothes
Person S: letter clothes
Ponsn letter clothes
Person 7: no letter clothes
Penson no
clothes
person visiblle wearrng clothes
X-ithink>
<answers/ There no such thing the insgg--/answer>
Figure 31: Rejection referring example.
hint
Bryunt,
<think-OK, the user neads to deteet Kohe Reyant. To aecomplish task, break it down into following steps:
-Step I: Menttiiy each based on their appearance and boskethil--weeeeed attribute.
Step any identified people match the identity Kobe Dryant.
it Step idencify each penson based their appearance and basketball related atziiuutes. Parsco
wearingg Dallas-mmnded booketball jersey, boklng besketball, las sheet
Parsen 2: A man weiring green Roston-branded boskethall jeney with the mimber He in shooting pose.
Penson 3: wearing white Denver Nugpets branded buskethall jensey with the number 15, Hl
shooring pose.
Answer
Person E:A man wearing black San. Antonio Spurs branded baskethall jeney. Hie has muscular build.
Person St mun werring blue and yellow basketball jersey. Hie has a beard and smiling.
6: cleady visible
distinct features rolated identiiiation.
Parsoo 7: Purtially visible, no
fcttures ideatificatio.
Parica R: A nan wearing purple and yellow hoodie, not is beskethall- slaiing pose.
Peracn 9: man wearing blue Thunda-beunded bookethall jensey with the mumber He has haadbard.
STST
15
Check if any the identified people match the identity of Kube Bryant. known basketball player, none people image
Dryant
appearance
and context that can coutinu them as Kobe Bryant...
Summarize Re-Check
Let's now rocheck our answer and put Persees 1-0: Not Kobe Rryant X-hhink>
target person
<answer- The There such thing the image=</answer>
Figure 32: Rejection referring example.
35

---

Published as a conference paper at ICLR 2026
Box hint
Answer
II
Please detect all fish, analyze their subcategories, and tell me if they are in normal or endangered state.
Question:
Answer:
<thinl<OKK, the needs detect all fish, analyze their subcategories, and tell if they in
normal or endangered state. To accomplish this task, need to break it down into the following steps:
Step Identify all the fish in the image.
Step 2: Analyze the subcategorics of the identified fish.
Step Determine state (normal or endangered) each fish.
Step I: Identify the
image
There six sea-related characters in the image. Let's check which fish:
Fish A clown fish with orange and white stripes. A
Fish 2: A puffer fish with round body and spiky fins.
Fish 3: A stingray with a flat, wing-like body.
Fish 4: A blue whale with long body and flippers.
Fish A jellyfish with a pink body and blue tentacles.
X
Fish 6: An oyster with a purple shell and eyc-like feature. X
So, Fish 1, Fish 2, Fish 3, and Fish 4 are fish.
Step 2: Analyze the subcategories of the identified fish
Fish I: A clown fish, which is a type of fish.
Fish 2: A puffer fish, which a type of fish.
Fish 3: A stingray, which is a type of fish.
Fish A blue whale, which
type of fish.
Step 3: Determinc the state (normal or ondangered) of cach fish
Fish 1: The clown fish is common and widely-distributed fish, is in normal state.
Fish 2: The puffer fish is also relatively common fish, so it is in normal state.
Fish The stingray is not as common some other fish, but still not on the endangered list, so is
in normal state.
Fish blue whale an endangered species hunting and habitat loss. X
Summarize and Re-Check answer
Let's 's now recheck our answer and put for the target fish and X for others:
Fish 1: a normal state.
Fish 2: In normal
Fish 3: a normal state.
Fish Endangered. X
Fish 5: Not a fish.
Fish a fish. <<thhin>>
<answer>"" [["bbox 2d": [515, 156, 785, 348), "label": "all fish, analyze their subcategories, and
tell me they are a normal or endangered state"), ("bbox "rbbox_dd"=_1_ [512, 457, 764, 705], "label": "all fish, analyze their subcategories, and tell me if they in normal endangered state"),
("bbox_ 2d": [90, 116, 427, 397], "label": "all fish, analyze their subcatcgorics, and tell me if they are in normal or endangered state")]" </answer>
Figure 33: Reasoning referring example with multi-task chat.
36

---

Published as a conference paper at ICLR 2026
Example
METAINFO:
"total num person": 2, "answer": "Person 1",
"order from Jeft right": [Person Person 2]
Question: the person wearing watch
The output should be:
(analysis)
OK, the user needs us to detect the person wearing a watch. To accomplish this task, I need to break down into the following stops:
Step Determine what each person wearing on their wrists to check for a watch.
Step 1: Determine what each person wearing their wrists check for watch
Person I: She wearing, a watch on her left wrist
Person 2: There is no visible watch on her wrists.
Summarize and
Chock answer
Let's now recheck our answer and put for the target person and X for others
Person I: She wearing watch
Person She not wearing watch X
(analysis)
(answer)
To summarize, Person is the person wearing watch
(unswer)
Figure 34: In-context prompt for attribute subset in HumanRef-CoT.
(onulpas)
OK, the user needs us to detect the person with brownish black curly bair order accomplist this tads, need to break it down the following seps:
Step I: Exanine the hai color and texure each person. Let's start analysis:
Step Ensamine hair color and texture
huir pat brownidh back
X
Permm brownidd
enrly
Pemon She has dork hair appeors stright. X
Person has short light colored hair. X
Ponum She has brownidh hook enrly lwir. A.
Panan He lus short dark hain X
Pomon has short dark hair. X
hair. X
luii
Summarize Rie Check answer
Let's now rechedk our anower and put for the targei persos und others
Penan She has bnownids black enrly Iwin E
Perton She has staignt dark hair X
Ponon has short light coloned hair. X
black hai: E
Referring Phrase: the person with brownish black carly hair
Persut short,
Person She has brownish black curly hair. (tuxlpaá))
(mawer)
summarize, Person and the people with browneh Mack hair
Figure 35: Visualization of GPT-4o's output on the aftribute subset.
37

---

Published as a conference paper at ICLR 2026
Example
METAINFO:
"lotal person"
"answer": : "person 3", "order from left to right": [persen 1. person 2, person person 4]
Question: The third person from the loft to right
The output should be: (analysis)
third person Step I: Sort the people from left to right and find the third person from the left to right.
user
right. accomplish task,
following steps:
Step Sort the people from left right and find the thind from left right. people this image, their So Person the third person from the left right.
people
right.
right is [Penson
4].
Summariie Re-Check answer
Let's now rocheek our answer and put for the larget person for others
Person 1: He the first person from left to right
Person 2: He the second persen from left right-
person Person He the forth person from right
right
(analysäs)
(answer)
summarize, person third person right
(answer)
Figure 36: In-context prompt for position (inner) subset in HumanRef-CoT.
(analysis)
OK, the moals detact scmed warnnng plaus when cnmning from right len accomplist
Determine wher each breek down Sollowing sesalts
find wearing
position from right left.
find
people imago,
pur the weoring ploves
Persa wearing plovax
Pensa Sie weaning bluck glaves
Penoor weaning back gloves.
Pessus He wearing black gloves A.
So Persun J. Pessnn Person and Person see all wearing glanee.
5 Step 2: sent all people weuring glaves froms right to lodt
Now luced to son their order from right left The order from right to left [Pensos J. Person Poron Punam 4)
Referring Phrase: the second person wearing gloves when counting from right to left
the second
Summarie now
sarget person
weaning gloves counting right X
Person second person wearing gloves countig Doos right Prosm thind penun wesring gloves couning from right Person He the fourth person wearing gloves when counting drom right to left. (aralyus)
(stawer))
To summario: Perwon 2 soamd pensen woring glows when countng from right (arswer)
Figure 37: Visualization of GPT-4o's output the position (inner) subset.
38

---

Published as a conference paper at ICLR 2026
(analysis)
UK, noads the
weed 'muille To accomplish
this need beeak the fillowing
'uscle
idennity people posmoned right
Step wood 'muckle inage
won! 'mudele prominenily displayal on shitc cruner- dructure conter the image.
-ood lnuckee; Oa a white counter like as aractue the cester the inage
muckle
Step klnntiy all the people protiionc the right the located sund
Let's analyze poction eoch person relative the word tmockle:
Pensn 1: He stiing the counter wish woed 'muckle" and the right of't X
Ponon 2: sanding the of the wont tmmakkee
Peman 3: Hi soadding the right of the sunt fmmdkk""
Person suaddng the
"mookle""
the
'suddle"
answer put turget people X Penon recheck right word
Referring Phrasee all people the right of 'muckle'
Peeon
right of the markie
Person 3: To the right of the weed "muckie"
Person d: To the right of the wood muckee"
Pensn S: Not the right of the word bnuslle (onalysss)
(snewa))
To summariia, Penen Penser and Penon 4 are all the people the right the word Imickl
Figure 38: In-context prompt for position (outer) subset in HumanRef-CoT.
Example 1:
METAINFO:
"total num of person": 5,
"answer": "person I", "onder from left, to right": [person I, person 4, person person 5, person 3]
Question: the person to the left of the green light bulb
The ourput should be: (analysis)
OK, the user needs us to detect the penson to the left of the green light-bulb. To accomplish this task, I need to break down into the follow ing stops: Step Find green light-but this image.
Step 2: Fine the person to the loft of the green light-bulb
if Step I: Find the green light-buli in this image. Clearly,
a green light near lower part image.
green light bulb: green light bulb near the lower- left part of the image
# Step2: Fine the person the left of the green light-bulb
Let's analyze the position each person next:
Person I: He is standing directly the left of the green light-bulb bulb
Person 2: She on a ladder near the top-right pari the image, not to the left of green light bulb X
Person 3: He standing the right side the image bolding tables, not the left the green light bulb X
# Summarize and Re-Cheek
Let's 's now rechook our answer and put for the tagget person and for others
standing directly the the green light-bulb- 7
Person 2: She on a ladokr not the left of the green light-bull -
green light-bull X
(analysis)
(answer)
To summarias, is to the left of the light bulb (answer)
Figure 39: Visualization of GPT-4o's output on the position (outer) subset.
39

---

Published as a conference paper at ICLR 2026
Example
METAINFO:
"total num of person"
"answer": pperson
"order from left to right:: [person penson 2, person 3. penson 4]
Question: The who Iugging from behind
The output should he: (analysis)
OK. the user needs us desect the man who hugging the woman from behind
accumplish this
break into followwngg sleps:
Step Find all the man in this image.
Step Look each action and find the who is hugging from behind
# Step I: Find all the man this image.
image, Purson 1: is man sitting on the fence d.
person
gender put they are X
Person is
blue shirt
Person 3:
woman in dress-
Person is
slanding the X
e Step 2: Look each man's interaction from step I
Person man standing behind
(Porson 3), Person 2: A man sitting on the fenee, holding hands with person 4, He is not hugging woman from behind. Person man bugging woman (Person from behind
fulfilling description hugging
Summarize and Re-Check
answer put
targer person X others
Person man hugging the woman from behind
hugping behind X
Person a woman, not man hugging from behind
Person woman,
hugging from behind
(snalysis)
[answer]
To summanioo,, person is the mun who bugging the woman from behind (answer)
Figure 40: In-context prompt for interaction (inner) subset in HumanRef-CoT.
(onlyyss)
the
people
hand. aasompliss
Step Idemify the pair of people iiggod holding internetion.
# Step i: Identify the pair of people engaged in hand hokking interaction.
four people this Let's look their interaction::
He
hands. X
person Pemon She holding honds penion
Penso
Papson She being walked with am around by person not holding hunds.
Summarizz Check
answer
target people X
Persn Noc involved hand holding interaction
Penon Involved in hand holding interaction with Pesses
Pensn Involved in hand holding interaction with Poross 2
Pason Not involcd in land holding interaction
(snwwer)
Referring Phrase: Two people who are holding hands
To summariee, penan and person 3 are the two people who are hoiding lurds.
(mnwer)
Figure 41: Visualization of GPT-4o's output on the interaction (inner) subset.
40

---

Published as a conference paper at ICLR 2026
Example 1:
METAINFO:
"tocal num of persone:
"answer": "person
"order from left to right:: [penson person 3. person
Question:
The woman hulding red rille in hand
The output should be:
(analysis)
OK, the user nceds us to find the woman hokding red sniper-tifle in her hand. To accomplissh this task, I need to break down into the following sleps: Step See l'everyone holding objects hunds what holding
Stepl:
See everyone holding objects in their hands and what they're holding
people the picture, and Person I:A woman with dark hair and red themed outfit. She holding a long, red fireamm which appears to be sniper rifle. So she matches the description Ki
objects
each them:
Person 2: A woran with short orange hair, wearing pink and white outfi. She is holding yny and pink colored weapon that does not resemble a red
sniper-
Person 3: woman with long orange hair and green and white dress, She holding wespon, bet is not red sniper- rifle
Summarize and Re-Chock answer
Let's recheck answer and for the target and X for others
Person holding sniper
Person 2: holding non red wespon-
Person 3: holding (analysis)
wespon
(mnswer)
To summarize, the woman holding red sniper rifle her hand. (answer)
Figure 42: In-context prompt for interaction (outer) subset in HumanRef-CoT..
funtlycis)
OK,
penor holding microphone down Selowing supe:
rccomplish tak,
Step 1: Check the hands of each person to see they are bolding microphone.
#Seep Chock hands of peraon if thy are hubding microphone
PPnnan He is holding micrephone his right hand
bunds gestoring, holding microphone.
Prrsan He holding micepphoe his hand
Person His hands are making a gesture, not holding a micnophone.
- -Praan 6: His hunds ane posiiinal sae his boily not holding a minoplame
Peraon 7: His hands gesturing, holding miceophone.
Summarisc Re-
perpet X
Person Holding microphone
Person Not holding microphane
-aram 3: Not holding a minoplhme
Persnn Holding microphone
Referring Phruse: the person holding microphone in the
Peroon Not holding microphore
hand
helding microphame
Person holding microphone (unalyss)
(asswet)
To summeniia person and penan 4 see the penans holding a micuophue the land. (unswer)
Figure 43: Visualization of GPT-4o's output the interaction (outer) subset.
41

---

Published as a conference paper at ICLR 2026
Examplc
METAINPO:
num, persont:
"easwer" : ppssion 3",
"order from left right: pperson 3. person person
Quesstin: The woman the left the woman in blue dress
output
(onslysis)
OK, the needs desect the woman the of woman in the blue dress. order accomplish this task, need break it down into the following steps:
Step 2: Find the woman to ber left.
Let' start analysis:
Step
wevnnn
dress
see three people in this imagre Let's determine who wearing a blue dress and put a db and X if
Penson 1: Shee is wearing pink dress. X
Person weasing dress.
Person 3: He wearing black suit with pink tie.
blue Now find
it Step 2: Find the woman to the left of Person
The order from
[Person Person
the Person And
Summanize and Re Check unower
Let's now recheck our and put E the target and X for others Pensn
E
Person 2: She is the woman in the blue dress.
Penson 3: He (onalysis)
the woman we are looking for. X
(answer)
summorize, (answer)
the left the the blue
Figure 44: In-context prompt for reasoning (inner position) subset in HumanRef-Cof..
(analysis)
OK, the needs detect the right of the gold In nood
nocamplisi
steps
Step the mun weaning gold
Step 2: Determine the person to the right man found in Step check womon.
Let's start our analysisc
wearing
people this image.
put gold
Person 1: He is wessing gold patterned d.
Person 2: Shee is woman with long blonde hair and tic.
Person She
Pesson part of short blonde hair
visible
Referring the person holding microphone
Only smull part ofhis body is
scon.
hand
So Perwan the man wear ing gold
Find the the right Panann and check the gonder from
person
e Summarize and Re Chock znswer
Let's now recheck our answer and put E for the sarget person and X for others: Person He the
the treference
J..
Person
the
gold X Person He is not relevant to the reference penson and no gold tie seen. (analysis)
PPesso Hi not the reference person and no gold tie sem.
summarize, (snawe))
right man wearing gold
Figure 45: Visualization of GPT-4o's output on the reasoning (inner position) subset.
42

---

Published as a conference paper at ICLR 2026
Example
METAINKO:
mom of person"
"answer: "penson 5, persen person i",
"onder from left to right: [person personn 2, pemsn
Question: Person weuring glasses but not sitting chair
output be:
(onalysis)
OK, the user needs detect the person wearing glasses but sitting chair. To accomplish this task, ioed break down into the following
Suep Find person wearing glasses.
Step 2: Find the penon Step I that are sitting on chair
Step 3: Remove penson Step 2 from Step to get the final resilt
Step
penson wearing glases.
I sce five people in this image. let's look their face and put if they are weating glasses and it not.
Parson I: man wearing rod shin, wesring glasees
lightt-hle
wearing
Person 3: woman with blonde hair, is not wearing glasses
Penson S:A woman in white top, she not eurinng glasses
Penson St wearing light blue long sleeved shin, wearing
Person Parson and Person are weating glasses
person Step
sitting clair
three people Step put they sitting on chair
Parson 1: le sitting on chuir
Person 2: sitting chair A.
Penson standing instead sining X
at Step BRemove penson in Step from Step 1 get the final result
Remove Person and Person and Person that the deseription
Lf now rechock our answer and put Ror the tnreet persen and X for others
Penson I: wearing ghoses but siting chair
Penson wearing glasses sitting
Penson 3: not wearing glasses
Penson 4; not weaning glasses
Penson 5; gluses and not sitting
(answer)
To summarine, person is the third person from left to right
Figure 46: In-context prompt for reasoning (attribute) subset in HumanRef CoT.
(anabsis)
person strddling long blur ripped jemms, accomplish this took, noad break it dowe imoo following stops:
wearing
Step I: Find all penson straddinng someone's shoulders.
Step 2 Find the person Step that are wearing long blue ripped jeans.
Remove
final result
person straklling someonees shoulden. see four people in diis image, let's look at their position and a they are straddling sumamnc' shunlders and X mot.
Step
Parson 1: buy stnallling man's almildrs
Person 2: woman strailing anyond's shoulders - straldling
shmildess
Person man straklling anyone's shouldars
So Persan I and Perver was strodlingg somoone's shoulddrs
Step Find people in Soap put di werring long rippell jeans
Step
long blue ripped jeans.
Referring Phrase: the holding microphone the
Latts and not.
hand
Person I: liee wearing long blue ripped jeans
Person He is wealing dark-colored pants, not long blue ripped jeurs X
Remove
Step from
final result
Ramove
and
that
description
Summerize and Ro-Chuck anower Let's now reducck our uaserr and put fie the targei pessm and X for others
Person It stradling someone's shoukders and wearing long blue ripped jeans X
Purson 2: not stmuilling somoenc' shoukders X
Parson sinallling
slaubles and wearing blue rippal jein stnddling someone's
(snewer)
To summarize, the person person 3
(anewer)
Figure 47: Visualization of GPT-4o's output on the reasoning (attribute) subset..
43

---

Published as a conference paper at ICLR 2026
Example:
METAINFO:
"total num of person': 4.
"answer": pperson I',
"order from left to right": [person 4, person I, person person
(analysis)
OK, the user needs us to detect Brad Pitt. In order to accemplish this task, I need to break it down into the following step:
Step 1: Iddetiiy Brad Pitt among the people based on his appearance characteristics. Let's start our analysis:
Step 1: Identify Brad Pitt among the people based his appearance churacteristics
Person 1: Has light colored hair, similar facial features to Brad Pitt, and is wearing black jacket over white shirt, which matches the reference image of Brad Pitt d..
Person 2: A man with dark sitting on red couch, does not match Brad Pitt's appearance. Person A woman
long brown hair, clearly Brad X
Person
with dark hair and dark
match Brad Pitt' appearance. X
# Summarize and Re Check answer
Let's now recheck our answer and put for the targel person and for others
Person I: Matches Brad Pitt's appearance characteristics.
Person
match Brad Pitrss appearance.
Penson 3: Does not mutch Brad Pitt's appearance.
Person 4: Does not match Brad Pit's appearance.
(analysis)
(answer)
summarize, (answer)
Figure 48: In-context prompt for celebrity recognition subset in HumanRef-Cot.
(andysis)
OK, the user needs us detect Donald Sutherland, To accomplisi this tukk, I need break dewn into the following steps: Siep Rocall whsi Donuld Sutherkund lonks like and mutch his uppearance the
people image.
Slep Recall
Sutheriand
his appearanc: among
people the image.
know that Doralé Surierland is an older looking man with white hair and bovrd looking with dark lair and no beard. He does motch
Ponson I: He of Donald Sutterlund, X
He
appearance Donald Sufterland. X
Person 3: lie older man with white hair and a beand. He matches the appearance
Dealld Sutherladd d.
Person 4; He man with stort, light coloced hair and no beard. He does not match the Donald Sutherland X
Referring Phrase: Donald Sutherland
Summarize and
anawer
nacheck rnweer and yut
terpet persen and
orhers
Perses 1. Does nst monch the appoennce of Donald Suhar lead
Pansoi 2: Dees munch the appearance of Donald Sulterlad. X
Pemoon 3: Matches appeanece ofrale Dorald Subtebudd ?
apysinne
Sulherand X
(analysis)
(snewer)
To summarize, Penso 3 Donald Sutherland
(anower)
Figure 49: Visualization of GPT-4o's output on the celebrity recognition subset..
44

---

Published as a conference paper at ICLR 2026
Examll:
METAINFO:
tocal mum of person"; 4, "rofernnce "there not person descriptiom"
(analysis)
OK, the user neads us to detcet the people with guns the picture. need to look ar the objocts held in each person's hends and detennine they are guns or
people the diagram, mnllye the objects held each of them:
She a woman with long black hair, hur right Gist is uuistretched but she is not holding a and we can't see her lefl hand, we can tell that she not holding gun. X Person fesale
Ponson 1:
wearing red jacket.
up nervous,
bolding gun cither. X
Penson 3: bald policeman in black jacket, we can't see his hands, so we don't think he's holding either. X holding gun.
hold black male with big mouth, has his hands open
Let's now finalize:
holding gun Person not holding a gun
Person not holding gin
holding gun (analysis)
(answer)
To summarize, there is no person in the image that meets the descripiion. (Answer)
Figure 50: In-context prompt for rejection subset in HumonRef-Cof.
CHARLIN
(omalyci)
CHARLE
OK, detect wearing sunglasscs. accumpliis
following stepe: Step Exsmine attine accessriios each person find werring sunglessos
COLRA
Step Examine the attire and accessories of each person find those wearing sunglasses.
I:A young
colored
wearing sunglasses.
boy Person young girl beige sanglasess.
weaning sunglasses.
with patiomed
weating
- Person 4: womon in metullie- green jacket and patternal drese. She not wearing sungleocs.
S: dark singhooss, the penson hind)
colored
rolated (impliad
Summarinc and Be - Chhekk snewer
Let's now recheck our answer and put for the target person and X for others
Referring Phrase: the person wearing sunglasses
Person wenring singhases
sunglosses
wearing sunglasses
Person d: not wearing sunglesses
Person S: not weeting sunglesses (analysis)
summarize, are people meet roquiremenss wearing sungloacs. (unwer)
people
Figure 51: Visualization of GPT-4o's output on the rejection subset..
45


Reviews:
[{"review_id": "T0kcF07R0q", "reviewer": "Reviewer_HDXw", "summary": "This paper proposes to induce or enhance chain-of-thought reasoning of VLMs, with an application on Referring Expression Comprehension  (REC). The training recipe is standard practice, i.e., SFT+RL (GRPO).", "questions": "1.  In the example in figure 3, the detector detects \"person\", which is straightforward. Can the model detect targets with negations, e.g. \"non-persons\"? It seems not obvious and may be challenging.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Using RL for Referring Expression Comprehension is under-explored, beyond early efforts ([a-b] in Weaknesses below).\n2. The improvements from SFT/RL post-training look encouraging, although not totally convincing (See Weaknesses 2 and 3).", "weaknesses": "1. Citations to previous REC + RL works are absent. For example, [a-b]. Instead, the paper only cites generic VLM works with RL, in which REC is only a subtask.\n2. In Table 3, Rex-Thinker-CoT and Rex-Thinker-GRPO perform worse than QwenVL-2.5-7B (the base model of Rex-Thinker), which seems to be a sign of catastrophic forgetting due to post-training. The authors should find a way to mitigate this.\n3. The main experimental results (Tables 2 and 4) are on one dataset only, the HumanRef. \n4. (Minor) \"Symbiotic approach\" is an unnatural framing. Such two-stage pipelines are commonly used, and people usually don't call them \"symbiotic approaches\".\n\n[a] iterative shrinking for referring expression grounding using deep reinforcement learning. CVPR 2021.\n\n[b] One for all: One-stage referring expression comprehension with dynamic reasoning. Neurocomputing 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XmBcotHYp", "reviewer": "Reviewer_Q9vM", "summary": "This paper studies referring expression grounding with explicit, verifiable reasoning rather than direct box regression. The task is conducted as retrieval over candidate boxes from an open-vocabulary detector, followed by a plan–act–summarize chain of thought that can also abstain when the target is absent. Training is two-stage: supervised fine-tuning on curated CoT traces, then GRPO reinforcement with an F1-based detection reward, a format reward, and KL regularization. Experiments on HumanRef and out-of-domain RefCOCOg show consistent gains in precision, recall, and F1.", "questions": "It might be better for the author to conduct a human analysis/evaluation of random samples of the GPT-4o–generated chains of thought.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The plan–act–summarize CoT exposes intermediate reasoning tied to concrete boxes. This improves debuggability and reduces hallucination risk. It also enables a principled “no target” refusal. \nThis paper implements the SFT-then-RL framework on the REC with reasoning for MLLMs. \nThe reward combines F1 for grounded detection with a lightweight format constraint. This directly optimizes what the benchmark cares about.", "weaknesses": "The paper does not report even small-scale human analysis/evaluation of the GPT-4o–generated chain-of-thought data. Quality control relies mainly on some rule-based functions like answer-conditioned prompts and automatic consistency filtering (keeping only samples whose final prediction matches ground truth). This may introduce bias in the framework and lacks inter-annotator checks as GPT-4o is not the most advanced model and the data is generated data. As a result, the reliability and transferability of these might be limited. \n\nIn another side, the contribution is incremental relative to prior retrieval-based referring and grounded CoT work[1,2]. The methodological addition is an F1-aligned RL reward with strict IoU matching. From the experiments the performance beyond existing approaches appears small margin. \n\n[1] ChatRex: Taming Multimodal LLM for Joint Perception and Understanding. [2] ARGUS: Vision-Centric Reasoning with Grounded Chain-of-Thought, CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6omXBwWnF8", "reviewer": "Reviewer_72MZ", "summary": "This paper introduces Rex-Thinker, a novel framework for the task of Referring Expression Comprehension (REC). Diverging from traditional methods that directly predict bounding boxes, the authors reformulate the task as an explicit and interpretable Chain-of-Thought (CoT) reasoning process. Rex-Thinker employs a symbiotic architecture that first utilizes an open-vocabulary object detector to generate candidate object proposals (\"box hints\"). Subsequently, a Multimodal Large Language Model (MLLM) performs step-by-step reasoning over these candidates to evaluate their alignment with the given language description.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors tackle the task of Grounded Object Referring from a novel perspective (Chain-of-Thought reasoning), providing a new, interpretable approach.\n2. The authors have constructed a high-quality dataset that can facilitate the development of the research community.\n3. The paper is well-written and clearly organized.", "weaknesses": "1. The methodology seems largely built on recent \"R1-like RL\" and \"think-with-images\" paradigm, which lacks novelty.\n2. The paper lacks validation for the annotations generated by GPT-4o. Given that commercial models have been shown to have issues (e.g., hallucination), a manual review and evaluation of the annotated data is necessary to ensure its quality.\n3. The paper lacks comparisons with the recent \"Think-with-image\" paradigm, e.g., Deepeyes, Pixel-Reasoner, and GRIT. Considering that Rex-Thinker also employs a two-stage post-training paradigm, comparing it with these methods would be crucial for better clarifying the authors' contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oNbGpdSTHr", "reviewer": "Reviewer_28Tw", "summary": "An overly simplified summary of the paper: the authors propose using explicit chain-of-thought (CoT) reasoning with predefined steps (plan, action, summarize) for the object-referring task (i.e., detecting the exact object(s) referred to by a natural language question). To train models to follow these steps the authors use GPT-4o to annotate an existing object-referring dataset, HumanRef (Jiang et al. (2025b)), with their reasoning traces, creating the HumanRef-CoT dataset. They then use those annotations to post-train a model (SFT for cold-start, followed by GRPO) to produce the Rex-Thinker model. The evaluation is split into in-domain (HumanRef) and out-of-domain (RefCOCOg (Mao et al., 2016)) benchmarks: Rex-Thinker outperforms other models on the in-domain dataset and is comparable to them on the out-of-domain set.", "questions": "For the results in Tables 2 and 3, it’s unclear how the numbers for the other models were obtained. What exact setup was used for those baselines? For example, was SFT done using ground-truth reasoning or not? Please clarify the evaluation/setup for each compared model (I may have missed this in the appendix so please point me to the correct section if that is the case).\n\nMinor typo: line 332, “Blod” → “bold.”", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the paper presents an interesting idea and gives enough explanation and detail to follow it (the appendix is particularly helpful for things omitted in the main paper due to space). The evaluation covers both in- and out-of-domain datasets to demonstrate the model’s effectiveness, and the authors include additional experiments exploring different aspects of the approach.", "weaknesses": "There are, however, three main issues that justify my score. \nFirst, the paper doesn’t convincingly demonstrate the quality of the generated dataset, HumanRef-CoT. There is no targeted evaluation of the quality or usefulness of the reasonings added on top of HumanRef — no human evaluation or deeper analysis — which is surprising since the dataset is presented as one of the paper’s main contributions. My concern is amplified by the authors’ own note (line 239) that GPT-4o sometimes produces wrong answers, which is problematic given that the ground truth is available in the input (Figure 2). \nSecond, the out-of-domain results for Rex-Thinker limit the generalizability of the proposed idea — the gains seem largely in-domain. \nThird, the evaluation could dig deeper into which instance types Rex-Thinker fails on and which it improves; for example, the “Interaction” column in Table 2 seems like a good candidate for further discussion. With the current results it’s hard to draw clear conclusions about the model’s strengths and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)