Re-Aligning Language to Visual Objects with an Agentic Workflow
Authors: Yuming Chen
Jiangyan Feng
Haodong Zhang
Lijun GONG
Feng Zhu
Rui Zhao
Qibin Hou
Ming-Ming Cheng
Yibing Song
Venue: ICLR 2025 Poster (2025)
Keywords: agentic workflow
language-based object detection
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: An agentic workflow including planning, tool use, and reflection steps to improve the alignment quality of language expressions and visual objects for LOD model.

Abstract: Language-based object detection (LOD) aims to align visual objects with language expressions. A large amount of paired data is utilized to improve LOD model generalizations. During the training process, recent studies leverage vision-language models (VLMs) to automatically generate human-like expressions for visual objects, facilitating training data scaling up. In this process, we observe that VLM hallucinations bring inaccurate object descriptions (e.g., object name, color, and shape) to deteriorate VL alignment quality. To reduce VLM hallucinations, we propose an agentic workflow controlled by an LLM to re-align language to visual objects via adaptively adjusting image and text prompts. We name this workflow Real-LOD, which includes planning, tool use, and reflection steps. Given an image with detected objects and VLM raw language expressions, Real-LOD reasons its state automatically and arranges action based on our neural symbolic designs (i.e., planning). The action will adaptively adjust the image and text prompts and send them to VLMs for object re-description (i.e., tool use). Then, we use another LLM to analyze these refined expressions for feedback (i.e., reflection). These steps are conducted in a cyclic form to gradually improve language descriptions for re-aligning to visual objects. We construct a dataset that contains a tiny amount of 0.18M images with re-aligned language expression and train a prevalent LOD model to surpass existing LOD methods by around 50% on the standard benchmarks. Our Real-LOD workflow, with automatic VL refinement, reveals a potential to preserve data quality along with scaling up data quantity, which further improves LOD performance from a data-alignment perspective.

Published as a conference paper at ICLR 2025
RE-ALIGNING LANGUAGE TO VISUAL OBJECTS WITH AN AGENTIC WORKFLOW
Yuming Chen'
Jiangyan Feng²
Haodong Zhang Ming-Ming Cheng'
Lijun Gong² Yibing Song'
Feng Zhu?
Rui Zhao?
Qihin Hou!
VCIP, Nankai University
Sense' Research
chenyuningénail. nankal.edu.co edu..on
houdpenankai.edu edu.oo
yibingsong sibingsong.cvégaal..co ovēgmail.com
ABSTRACT
Language-based object detection (LODD aims to align visual objects with language expressions. A large amount of paired data is utilized to improve LOO model gen- eralizations. During the training process, recent studies leverage vision-langune language models (VLMs) to automatically generate human-like expressions for visual ob- jects, facilitating training data scaling up. In this process, we observe that VLM hallucinations bring inaccurate object descriptions (e.g., object name, color, and shape) to deteriorate VL alignment quality. To reduce VLM hallucinations, pose an agentic workflow controlled by a large language model (LLM) to re-align language to visual objects via adaptively adjusting image and text prompts. We name this workflow Real-LOD. which includes planning, tool use, and reflection steps. Given an image with detected objects and VLM raw language expressions, Real-LOD reasons its state automatically and arranges action based on our neural symbolic designs (i.e., planning). The action will adaptively adjust the image and text prompts, and send them to VLMs for object re-description (i.e., tool use). Then, we use another LLM to analyze these refined expressions for feedback (i.e., reflection). These steps are conducted in cyclic form to gradually improve lan- guage descriptions for re- aligning to visual objects. We construct a dataset that contains tiny amount of 0.18M images with re-aligned language expression and train prevalent LOD model to surpass existing LOO methods by around 50% on the standard benchmarks. With automatic VI refinement, our Real-LOO workflow reveals a potential to preserve data quality along with scaling up data quantity, further improving LOD performance from a data-alignment perspective.
pro-
INTRODUCTION
Aligning language expressions with visual objects has been continuously evolving. Initially, single noun word is used as a category label (Redmon et al., 2016; Ren et al., 2016; Carion et al., 2020) to connect visual object. Then, phrases are introduced (Akbari et al., 2019; Li et al., 2022: Gao et al., 2023) to describe objects. Further, referring expressions (Su et al., 2020; Zhang et al., 2022) and complete descriptions (Schulter et al., 2023; Yao et al., 2024) are developed for object detection. Although language evolves from coarse labels to mne-grained expressions, the essence of object detection is align the language data to visual objects. This alignment is challenging as language expressions become diverse to represent various human intentions. As for the same visual object, different people usually describe it in various forms, as they focus on different aspects of object properties (e.g., color, shape, texture, and relationship with surroundings). This diversity makes vision language (VL.) alignment cumbersome, where comprehensive set of language expressions should be collected for model training. Fortunately, emerging VLMs (Zhang et al., 2021; Liu et al., 2023a; Ye et al., 2023; Sun et al., 2024; Yuan et al., 2024; You et al., 2024: Zhang et al., 2024) have recently been leveraged to produce human-like expressions. The auto auto-generation of language expressions for visual objects eases the difficulty of collecting training data pairs. By training LOD models with more VL data, studies (Pi et al., 2024; Dang et al., 2024; Kong et al., 2024) improve detection performance accordingly, especially when the language query is diverse to describe the target object.
*Q. Hou and Y. Song corresponding authors. J. Feng and H. Zhang equal contribution. The code is available https: //gi thub.

---

Published as a conference paper at ICLR 2025
<Prompt>: What is the objact in the red box?
<Prompt>: W What is the object in the red box?
<Prompt>: Desoribe the person the red bux.
<Prompt>: What the color person's clotbes.
<Reesooss>:: The object a blue and red hast worn by gou.g.....
<Il eteeponee:: The object is a flag with a desiggn that resembles the Union Tack
<Itesponsee:: The person woman with long heir wearing a black top
<Ieepoose>: The person weering a blackand while stripelshirt, durk panks Hackshoes
(a) VLM for a small object
(b) VLM with extended crop
(c) VLM with general prompt (d) VLM with specified prompt
Figure 1: Examples of adaptive image and prompt modifications refine language expressions. For small object in (a), VLM produces erroneous content marked in red. In (b), we crop the local region of (a) and obtain refined content marked in green. Another example is in (c), where general prompt leads to erroneous content while specific prompt in (d) does not.
The language expressions generated via VLMs, although aligned with human preference, may not accurately describe the target object due to model hallucinations. Fig. I shows two examples. A small object shown in Fig. (() leads VLMs to generate erroneous expressions. Moreover, general text prompt without specifying the target object shown in Fig. 1(c) makes VLMs incorrectly describe visual content. For model hallucinations on small objects, we analyze that VLMs are trained via extensive image-caption pair data (Radford et al., 2021; Schuhmann et al., 2022), where the caption mainly depicts global image content rather than local objects. The ignorance of local object context in training data makes VLMs hallucinate small objects. On the other hand, text prompts without specifying the target object (e.g., 'in a red box") ) lead to incorrect detail descriptions from VLMs. A lack of object identity in the prompt makes VLMs insensitive to object details and expresses them erroneously. When adding inaccurate language expressions, the alignment of object and language becomes fragile and impedes LOD performance improvement along with VL data scaling up.
In this work, we propose to re-align lan- guage expressions to visual objects auto- matically to refine VL data quality from the
Raw expressions
Planning
alignment perspective. Our re-alignment is conducted via a workflow controlled by an LLM -pwwered agent (i.e., Real-Agent). Fig. 2 shows glimpse of where there are three steps (i.e., planning, tool use, and re- flection) to form a cycle. Given input image with detected objects, we first con- vert this image into captions, which are sent into Real-LOD together with object loca- tion, category, and raw language expres- sions from the VLMs initially used. Then, our agent automatically reasons the current state and arranges further actions. The state/action represents our neural symbolic design in the workflow, where we have predefined five states indicating how language aligns with the visual objects. Each state is followed by an arranged action. After the planning step. our agent takes action to construct adaptive VI. prompts for the tool models (i.e., VL.M/LLMM. Customized prompts enable tool models to collect more visual observations or refine current expressions. After the tool use step, the refined expression is sent to an LLM-based reflector for feedback. The feedback is then provided to our agent for planning in the next cycle. Fig. 5 shows an example in which the raw expression is gradually refined to align with the target object.
An image with detected objects
Tool
Real-LOD
Re-aligned expressions
Reflection
Glimpse of our Real-LOD. takes image captions with detected objects and raw expressions as inputs. It gradually re-aligns expressions to match ob- jects well. By using better-aligned training data pairs, we improve the performance of LOD.
Figure 2:
For presentation clarity, we refer to Real-L.Ol -LO agentic workflow, Real-Agent as the ....-ee agent, Real-Data as our constructed dataset, and Real-Model trained LOD model.
2

---

Published as a conference paper at ICLR 2025
Our Real-LOD refines language-object data pairs via re-alignment for LOD model training. Our Real-Model is a prevalent model structure with a Swin-B backbone (Liu et al., 2021). We train this model using our constructed dataset Real-Data, where there are 0.18M images that contain 1.4M language-object paired data. In the standard benchmarks (Mao et al., 2016; Schulter et al., 2023; Xie et al., 2023), we surpass existing methods by around 50%. This indicates that data quantity and quality are important for LOD training. In addition to the amount of image data that scales up, our Real-LOD can preserve the quality of the data pair. This potential directs new trend that expanding high-quality paired data further improves LOD performance from data- alignment perspective.
2 RELATED WORK
Language-ban based object detection. LOD requires models to locate the associated instances according to diverse expressions. Benefiting from visual-language detector development, the accuracy of LOD tasks is improved rapidly. MDETR (Kamath et al., 202.1) first proposes an end-to-end modulated detector that detects objects by a given query. GLIP (Li et al., 2022) presents a language-iman pre-train model for understanding object-level, category-aware visual representations. GDINO (Liu et al., 2024b) introduces an open-set object detector within an effective fusion module that allows the detection of objects with textual inputs such as category names or referring expressions. FIBER (Dou et al., 2022) designs new visual- risual-language model architecture that can handle different tasks such as visual question answering (VQA), image caption, object detection, and so on. APE (Shen et al., 2024) introduces a universal visual perception model to align visual and language representation on broad data at once so that it can conduct different language-visure tasks without task-specific fine-tuning. OWL-V2 (Minderer et al., 2023) proposes an architecture without any fusion modules. They use 1B language-object pair data to align image and textual features directly. The above methods utilizes lanjuage-object pair data to train their detectors, including COCO (Lin et al., 2014), Objects365 (Shao et al., 2019), OpenImage (Kuznetsova et al., 2020), SBU (Ordonez et al., 2011), I), GoldG (Li et al., 2022). CC (Sharma et al., 2018; Changpinyo al., 2021; Xu et al., 2023), LVIS (Gupta et al., 2019), Flickr30K (Plummer et al., 2017), GRIT (Gupta et al., 2022), and V3Det (Wang et al., 2023a).
Agentic workflows. Intelligent agents empowered by LLMs are able to solve wide range of complex tasks by following user's s instructions (Askell et al., 2021; Liu et al., 2025; Significant Gravitas, 2023; Yohei Nakajima, 2023; Reworkd, 2023). Due to the strong understanding and reasoning abilities of LLM (Wei et al., 2022; Wang et al., 2023b), these agents are capable of making plans to achieve specified goals, mastering tools execute tasks (Yao et al., 2023; Liu al., 2023b; Tang al., 2024; Yang et al., 2023a; Guo et al., 2024; Shen et al., 2023; Cai et al., 2024), generating reflection to refine outputs (Madaan et al., 2023; Shinn et al., 2023; Yu et al., 2024; An et al., 2023; Gou et al., 2024), and even collaborating with other agents (Chen et al., 2025; Xu et al., 2024; Holt et al., 2024). HuggingGPT (Shen et al., 2023) is presented as a powerful agent that leverages L.M to connect various AI models for solving different tasks. This agent is designed to understand and dismantle given Al tasks, as well as plan and select appropriate Al models to execute each subtask automatically. Similarly, LLa (Liu al., 2023b) maintains skill repository that contains a wide range of vision-language tools to fulfil many real-world multi-modal tasks. Other examples include Gorilla (Patil et al., 2023), GPT4tools (Yang et al., 2023a), and ToolAlpaca (Guo al., 2024), which are fine-tuned LLMS with the ability to utilize available APIs Additionally, recent studies have also shed light on improving agent performance through train-free approaches. One of the main ideas is reflection, where agents provide feedback to themselves and use it to refine their outputs. Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) are the typical examples to reinforce agents with linguistic feedback, while CRITIC (Gou et al., 2024) introduces external tools into the reflection process with a human ounan-predefined execution logic which relatively fixed. Different from previous works, Real-L.OD pioneerly designs an entire agentic workflow containing the above three steps to advance the alignment quality of VL data for LOD tasks.
RE-ALIGNING LANGUAGE TO VISUAL OBJECTS
In this section, we first revisit the LOD framework, showing how paired VL-inputs predict target objects and previous methods to generate language expressions in Sec. 3.1. Then, we illustrate the key steps of our Real-LOD (i.e., planning, tool use, reflection) in Sec. 3.2. An example is provided in Fig. 5 to intuitively demonstrate how language expression is refined via our re-alignment scheme.
3

---

Published as a conference paper at ICLR 2025
Reasoning:
Right
Raw expressions
expression is ... can reason .. and
Actionl: Stop
VLM. LLM
action...
Action2: Rewrite
Wrong
Image
Planning
Uncertain (calegory attribbute
Action3: VLM with object crop
Global captions
Uncertain
accessory)
Objoct locations
Action4: VLM with extended object
State 5: Uncertain (location / behavior)
ActionS: VLM with object highlight
Reflection
Tool use
Re-aligned expressions
Analysis:
Given the output of tools.
LLM: Vicuma-v1.5-13B
Feedback:
The expression mentions....
VLM: LLaVA-v1.6-448
Real-LOD
but the tools....
Figure 4: Overview of our agentic workflow. The inputs are images with captions, detected objects, and raw expressions. Our Real-Agent reasons the state and arranges the action (i.e., planning). During action execution, our Real-Agent uses VLM and LLM to re-perceive visual content and refine expressions (i.e., tool use). Then, the output results analyzed by an LLM (i.e., reflection). The feedback is provided to Real-Agent for planning in the next cycle.
In Sec. 3.3, we also analyze the refined expressions, which constitute training data pairs to improve
LOD performance.
3.1 LOD FRAMEWORK AND LANGUAGE EXPRESSION GENERATIONS
The language-based object detection (LOD) frame- work typically consists of two encoders, few in- teraction modules, and decoder. Fig. shows an overview. The inputs of LOD are one image and language expressions formulated by words, phrases, or sentences. LOD uses image and text encoders to obtain their embeddings independently. Then, the expressions interact with visual objects to formulate joint cross-modal feature space. These interactions are usually conducted via cross-attention operations. Afterwards, LOD introduces decoder module localize the corresponding object based each ex- pression. The training losses (e.g., LI loss, GIOU loss (Rezatofighi et al., 2019), contrastive loss (Li ct al., 2022)) are typically from DETR-basee methods (Carion et al., 2020; Kamath et al., 2021; Zhang al., 2023).
Enbediia
lagneient rodpe
Tot
Crew Modblit Estencion
Dender
Irsage Erecike
Image Enbeddiep
Overview of a general LOO frame- work. The paired VL data are independently encoded and then interacted to decode results.
Figure 3:
The LOD framework establishes the connection between language and objects. The training data contains images, object bounding boxes (bbxs), and language expressions. Previous datasets (Mao et al., 2016; Plummer et al., 2017; Krishna et al., 201 7)) tend to collect expressions from human partic- ipants, which constructs a limited amount of paired data and bottlenecks the detection performance. Recently, studies (Dang et al., 2024; Pi et al., 2024) have leveraged VLMs to generate human-like expressions for visual objects. The training data amount extensively scaled up. and the learned LOD model captures diversified object descriptions. Following their spirit, we use a VLM model, ALaVA-v1.66334 (Liu et al., 2024a), to generate 673%: language expressions for 188k images with 336.5k objects. Also, we use an LLM model, Vicuna-v1.5 1.5B (Zheng et al., 2023), to expand the number of expressions from 673k to 1.346. by generating synonyms. The details raw expression generation presented in Sec. A of the Appendix. After obtaining language-objest paired data, we use SigLIP (Zhai et al., 2023) to calculate the VL matching score. For the paired data whose score is lower than 0.5, we Real-LOD to re-align raw language expressions as illustrated in Sec. 3.2. This because we leverage SigLIP to exclude about 75% of training data from our workflow, leaving only nearly 25% to be processed.
4

---

Published as a conference paper at ICLR 2025
Object Category
horse
Bbox Coordinates
[225,00 267.79, 101,47,99 99.399
Image Capdio
image depicts group people on hossebatic forested area, pensn riding
the specars kealing group.
worting blue
pantis
brown The are weating bacipodk,, suspesting they night guidkl The fonat lack green tron, da thruuhh ramopy. indirating it's samyy day. owaral atmoophere the image uuiiuer advenitaee cxplaration.
Cau you specids if the caw exgressoos that Urree horse waakinn an sand) is refening to the uageet oijest arac,, please modicy to corret
expreecion.
Request
Reasoning: The oxgression descrbes brown home that king on sand, the maag cxption moniion the homer walking on din poth, and there to riention of suad it Bie caption. Based on this infomutioo, reason the state & where au uncertais object relaton with sunoundings. So plaz fox Action where can cmp local object region this mage for VLM turterr penceive The descriptius of YLM supllament the current information mee conduct more thanough expresiion oxasination.
Plamning
Action: Tool Jamee" 'VLM; Tool, paraneten:: i: (Phoorp:: "Questous the hoose walling on sand dia pata., "Image oditigg escended object cmpp:131
Respoase: I will consult with VLM ucquire supplement information that will esable me to evaluate the cureent expezssion more theeouglly.
Tool
VLM
Reflection
exgression
wrong sand)
Based on feedocck, I Lieason the state 2 where an ceruin the expression (boosnt botse motlinng an vend) wonny. So plan for Autur where LLM rowelic the cogressius, and Bhem chock lee comecncce the new expresdon.
Reasoning:
Real-L-on
Planning
Actions [[Tool name: 1. M. Teol, permmntenn ("hoopp: Imag Caprion, Resporer,,
Resuosse The exprossion (drmnn hone walking oo vond! contire flowe, now will fine lovenge the I.M cteaic altmmativ espeossion for yus.
Tool use
LLM sutput: New Descciption: ldreen horze sulking an suler awtk niker in Mne shitt het Joaaltor the
Reflection
Feedback: The exgrescon (érees Acrse wuitioe dir eath with rider Mur shior and Aandin the war) descrites the object canguory 'ueese' und provides similer contest the cxption.
Reasoning: Basod the feadback. reason the vane where um cartin the expression (érosu hone malking dir park with rider Alse skir and hat, fending the war) comest So plas fir Action stop. Actions.1
Planning
Requuser: The expeosioo thrsn barn valling vand)
onuld be (drome
barne walking path with rider Maa shirr and bat lrading shrr
The expeession (branes Burze walking cand is wenng one, and comeet exprasioo could by (brace hure wafklny on dlr earh witk
ddder in Shic.hhizaan bot, foatles sxel.
expression the given image. Based on the input image, caption, and detected objects, Real-LOD performs planning, tool use, and reflection in eyclic workflow for state reasoning, action execution, and result feedback. The image and prompt are adaptively adjusted for tool models to supplement customized object descriptions, which benefit expression re-alignment.
Figure 5: An example of how Real-LOD re re-aligns
3.2 AGENTIC WORKFLOWS FOR LANGUAGE EXPRESSION RE-ALIGNMENT
The generated language expressions may not match visual objects. As illustrated in Sec. 1, either local context ignorance or unspecific text prompts lead model hallucination. To solve this problem, we design cyclic workflow to enable VLM to adaptively focus on local regions and specify text prompts according to the target object. Based on the finding that an LLM reasons more accurately in pure language form than in VL form, we choose fine-tuned ChatGLM-6B (Zeng et al., 2023) with text-only input as our Real-Agent to control this workflow. Fig. shows an overview. It consists of planning, tool use, and reflection steps to gradually refine expressions. To facilitate VL re-alie algiigmmeen, we have performed neural symbolic design the planning and tool use steps where we predefined states and actions, which are illustrated as follows.

---

Published as a conference paper at ICLR 2025
Planning. In this step, we have predefined 5 states indicating how expressions are aligned to the target object from the view of VLM. Each state corresponds to one action to be executed. Formulating these states/actions is motivated via our data analysis in Sec. 3.3, where observe how VL misalignment occurs in practice. Given the text-only input containing the expression, image caption, object category, and reflector output from the last cycle (Empty if at first eycle), our I.LM-powered Real-Agent reasons the current state and arranges action accordingly. The five predefined states/actions are as follows:
Stop. Real-Agent is certain that the current language expression matches the target object. Real-Agent will stop the workflow and output the current expression.
State 1:
Right. Action 1:
State 2: Wrong. Action 2: Rewrite. Real-Agent is certain that the current expression does not match the target object. Hence, Real-Agent will use an LLM to regenerate the expression. The in-context prompt for rewriting will be generated following the template in Tab, 9 of the appendix for rewriting.
State 3: Uncertain (category/attribute) Action VLM with object crop. Real- Agent is uncertain whether the current expression matches the target object. The uncertainty resides in the object category or attribute. So Real-Agent plans to crop the object region and use a VLM for further re-perception. The description from VLM will be kept in the text prompt for the next step.
Uncertain irelation/accessory)) Action 4: VLM th extended object crop. Similar to State 3, Real-Agent is uncertain of object relation (with surroundings) or accessory. It plans to crop a larger region covering the target object and uses VLM for re-perception. The description from VLM will be kept in the text prompt for the next step.
State 4:
Action 5: VLM with object highlight. Similar to State 3, Real-Agent is uncertain of object location (in image) or behavior. It plans to highlight the object region using a red rectangle (Shtedritski et al., 2023) and uses a VLM for re-perception. The description from VLM will be kept in the text prompt.
State 5:
Uncertain [location/behavionn
When executing actions, we only refine language expressions in Action 2, while in Actions 3,4.5, we use VLM to supplement descriptions in-context prompts. These prompts will be utilized in the next cycle to facilitate state reasoning and action executions.
Tool use. In the planning step, Real-Agent has scheduled use several tools (i.e., VLM and LLM) when executing actions. We prepare a toolset in advance where there is one I.LVV--V 16-34B model for VLM usage and one Vicuna-v1.5-132 model for LLM usage. Based on its reasoning about the state of the current expression, Real-Agent adaptively modulates visual content and text prompts by setting up "Prompt" and "Image editing" parameters for scheduled tools. Then, the tool can be used effectively to get desired responses from VLM to improve the expression refinement. For example, when executing VLM for visual content re-perception, Real-Agen real-Agent will edit the image via cropping or highlighting planned according to the object bbxs. In addition, the customized text prompts designed by Real-Agent are more specifically related to the target object. this way, Real-LOOD can effectively reduce model hallucinations, improving language and object connections by reliiigning expressions. The visual and language prompts for VLM are shown in Tab. 12 of the appendix.
Reflection. After using tools, Real-Agent has finished action executions. We use LLM (i.e., Vicuna-v1.5-133 -15--13B as a reflector to analyze the results by incorporating the image caption. It verifies whether the current expression matches the target object. For State 3-5, where Real-Agent is uncertain, the reflector helps Real-Agent be confident in judging whether the expression is correct or wrong. For State 2 where Real-Agent has planned to rewrite the expression, the reflector examines the correctness of the new expression. The analysis of the reflector will be formulated as feedback to Real-Agent to facilitate its planning in the next cycle.
3.3 DATA ANALYSIS OF LANGUAGE AND VISUAL OBIECTS
Training data for Real-Agent. We prepare training data in the text form to fine-tune Real-Agent from ChaiGLM-6B. First, we randomly collect images with detected objects from Objects365 (Shao et al., 2019) datasets. Then, similar to Sec. 3.1, we use VLM to generate raw expressions and collect the data pairs that are filtered out by SigLIP, i.e., the matching score is lower than 0.5. In total, prepare 15k input data to train Real-Agent. Each input data contains the object category, raw expression. and reasoning from the LL-M-based reflector defined in Sec. 3.2 examine whether the raw expression

---

Published as a conference paper at ICLR 2025
Category
4. 4.4%
60.6%
5.6%
2,2%
Behavior
Attributc
2.2%
75,0%
88.2%
Category
Attribute
18.9%
Accessory
66. 7%
Relation
Location
100%
200%
Behavior
Location
Accessory
60.0%
Relation
Figure 6: Percentage of 6 aspects for mismatch expressions where category and attribute con- sume the majority.
Figure Success rate of expression re- alignment via Real-LOD in aspects.
matches the target object. Then, we manually set the state for each input data by ourselves and collect responses (including "reasoning" and "actions") via an LLM (i.e. (ccuna-v1.5-131 with text prompts including several hand-crafted in-context examples in Tab. 11 following the spirit of LLaVA (Liu et al., 2023a). Finally, manual check is conducted to ensure no error in the fine-tuning data. The training process conducted in parameter-efficient form, i.e., LoRA (Hu al., 2021), that does not affect the reasoning ability of ChatGLM-6B.
Analysis of language and visual objects. Our Real-LOD corrects raw expressions filtered out via SigL.P. As we design ac- tions for expression correction in advance, we analyze how these expressions misalign to the target object. We randomly select three hundred filtered expressions and man- ually check each for detailed observation. Overall, we summarize the misalignment reasons in 6 aspects based on the observed expressions: 1) Category: the expression describes another object rather than the tar- one; 2) Attribute: the expression pro- vides wrong attributes such as color, shape, and texture of the target object; 3) Acces- sory: incorrect accessory descriptions of the target object; 4) Location: wrong rel- ative location of the target object in the image; 5) Relation: incorrect object rela- tionship with surroundings; 6) Behavior:
Catspay
Rebation
Bar blcck and ailers sann
Haw
Roncro hnated extrmmnt
ponen sctiiltc theh side ar
racher
Aftelatte
Row prayy jacket playing
Actavt warag Mi
Relbol pereee viiti
Figure 8: Example summary of misaligned raw expres- sions in aspects, followed by our re-alignment.
incorrect object/human behaviors. Fig. shows representative examples of how expressions listed in each aspect will be corrected. These aspects motivate our neural symbolic action designs in Sec. 3.2 where image editing operations (i.e., 'object crop', 'extended object crop', and 'object hig highlight right) are utilized for VLM to perceive object related contents further. Fig. 6 shows the percentage of these 6 aspects in our observed expressions where most of them reside in the category and attribute aspects. After refinement, we compute the success rate of expressions from each aspect as shown in Fig. 7, where major expressions from category and attribute aspects can be effectively refined.
Analytical experiment for Real-L.OD. Besides summarizing 6 aspects of mismatched raw expres- sions, we analyze how effective our Real-LOOD is for re-alignment. Although we have designed corresponding actions to enable VLM for a re-perception, the accurate state reasoning and action planning will determine the refinement quality. For the inputs listed in Fig. 4, Real-Agent shall reason accurately to identify which state they belong to and execute the action accordingly. To analyze the reasoning ability of Real-Agent, by replacing the planning step with a step where one of the states/actions is selected randomly for further expression refinement, The reflector is used in both workflows identify whether the final
sample 11k samples and introduce a scheme for comparison
7

---

Published as a conference paper at ICLR 2025
expression refinement is successful, and set the maximum round to 3. After refining expressions using Real-LOD and the random selection, we find that the success rate 2 is 74.7% v.s. 35.6%. This comparison shows that accurate reasoning from Real-Agent significantly improves expression correctness. Furthermore, we examine the matching score between image and expressions refined by these refinement schemes via IIgL.P. Our Real-Agent improves the average matching score by 66.27% (i.e., from 0.0673 to 0. 0.1119), while random selection improves by 32.65 69% (i.e., from 0.0673 to 0.0893). This indicates Real-Agent improves the SigLIP matching score more than the random selection scheme (i.e. 66.27% v.s. 32.69%). From the comparisons of re-alignment success rate and SigLIP score improvement, our Real-Agent demonstrates effectiveness in reasoning input state, planning action correctly, and successfully refining raw expressions. We also provide more analytical experiments in Sec. K.
EXPERIMENTS ON LANGUAGE-BASER OBJECT DETECTION
Real-Model is a prevalent LOD model structure illustrated in See. 3.1. We use the re-aligned data Real-Data to train this model. The training details are provided in the Sec. 1.2 of the Appendix. In this section, we focus on evaluating our model in the LOD scenario. We illustrate benchmark datasets, ablation studies, evaluations of existing methods, and computational cost analysis.
Standard benchmarks. The benchmarks we use for evaluation are OmniLabel (Schulter et al., 2023), DOD (Xie et al., 2023), RefCOCO/g/+ (i.e., RefCOCo, ReiCOCOg, RefCOCO+) (Yu al., 2016; Mao et al., 2016) and OVDEval (Yao et al., 2024). OmniLabel is collected from three object detection datasets, i.e., Objects365 (Shao al., 2019), Openlmage (Kuznetsova et al., 2020), and COCO (Lin et al., 2014). is divided into these three subsets for evaluation. There are 12 images. 20.4k object bbxs, and 15.8k expressions. The evaluation metrics are AP, AP-des-pos, and AP-des-S/M/L, which measure the average precision of object descriptions from overall, only positive, and various length perspectives. DOD contains 1k images with 18k bbxs and 422 descriptions. It uses 'Presence' and 'Absence" to evaluate detection performance upon positive and negative queries. The ReiCOCO/g/+ are from the COCO datasets with 9.9k images, 22.9k bbxs, and 46.5k descriptions. The details of OVDEval are shown in Sec. E. For all the benchmarks, follow standard protocols to ensure fair comparison.
Real-Data. Our Real-LOD naturally constructs dataset via re-aligned language expressions. We randomly select images from Objects365, OpenImage, and LVIS datasets with all categories covered. There are 188k images with 1, 3.346.1k object-query pairs in total. Among them, 473.8k pairs filtered out by SigLIP, with 30 being re-aligned. The final pairs for our Real-Model training are 179.4k. We name our dataset Real-Data.
4.1 COMPARISONS WITH STATE-OF OFEE--F-TTEE TEE-THE-ART LOD METHODS
We evaluate our Real-Model with existing LOD methods on the standard benchmarks, including OmniLabel, DOD, and ReSCOCO/g/+ in Table 1-3, In each table, we list the vision backbones leveraged by LOD methods, source of training images (i... 'Source'), and the number of images used for training (i.e., '#lmg'.. We use VG, OI, 0365, RefC/g/+, and CC to denote Visual Genome (Krishna al., 2017), OpenImage (Kuznetsova et al., 2020), Objects365 (Shao al., 2019), RefCOCO/g/+ (Yu et al., 2016), and Conceptual Captions (Sharma et al., 2018; Changpinyo et al., 2021; Xu et al., 2023), respectively. Besides, the detailed training image sources for each method can be found in Tab. 15.
In the OmniLabel benchmark, our Real-Model significantly outperforms existing LOD methods on all test sets. Especially on the OI set, under the AP-des metric, Real-Model surpasses the second-best mm- nun-GDINO by a large margin (i.e., 40.5% v.s. 23. 23.2%). Mcanwhile, under the AP-des-pos metric, Real-Model surpasses the same second-best mm-GDINO significantly (i.e., 51.4% v.s. 34.5%). The superior performance of Real-Model is due to the high-quality Language-object paired data provided by Real-Data. On the other hand, we observe that the training data size used for GLIP is larger than Real-Model, but the accuracy is around 50% of ours. This indicates that data quality is as important as quantity to achieve superior results.
SSuppose N and N, represent the number of total expressions and correctly refined expressions, respectively. The success rate can he formulated as Nz.
8

---

Published as a conference paper at ICLR 2025
Table State-of-the-art comparisons on the OmniLabel benchmark.
Subset
LOD method
Backbone
Source
Fing
AP-dex
AP-des pus AP-des-S
AP-des M AP-dey- L
MDETR (Kamath ot al., 2021)
ENB3
COCO, VG, Flickr30K
0.3M
13.2
31.6
15.4
13.5
12.4
GLIP (Li et al, 2022)
Swin.L
0365, OI, Re/C/g/+, etc
17.5M
13.9
36.8
28.9
12.9
11.5
COCO
mm GDINO (Zhao et al., 2024)
Swin-
GoldG, 0365, COCO, etc
12M
15.2
47.0
29.3
14.9
15.1
FIBER (Dou al. 2022)
Swin-B
COCO, CC3M. SBU, etc
4M
14.3
38.8
31.3
12.7
16.1
Real-Model
Swin-R
Real-Data
0.18M
26.2
59.7
39.4
25.4
24.3
(Kamath al., 2021)
ENB3
COCO, Vú. Hickr30K
0.3M
GLIP at, 2022)
Swin-L
0365, RefClp/+, etc
17.5M
24.0
35.2
44.5
20.5
0365
imm-DDINO (Zhao al., 2024)
Swin-B
GoldG, O365, COCO,
12M
19.6
31.0
32.3
FIBER (Dou al. 2022)
Swin
COcO, CC3M, SBU, elc
4M
25.9
38.2
44.7
22.5
14.1
Real-Model
Swin B
Real Data
0.18M
36.0
52.1
55.7
32.3
23.7
MDETR (Kamath et al,, 2021)
ENR3
COCO, VG, Flickr30K
0.3M
6.1
10.6
9.6
5.7
4.1
GLIP (Li et at., 2020)
Swin-L
0365, OL RetC/g/+,
17.5M
20.1
31.2
33.3
18.7
10.3
O1
nun-DDINO (2hao al., 2024)
Swin-B
GoldG, 0365. COCO,
12M
23.2
34.5
32.3
23.8
16.9
FIBER (Dou 2022)
Swin-B
COCO, CC3M, SBU,
4M
20.1
30.9
34.1
18.5
10.5
Real-Model
Swin-B
Real-Dota
0.18M
40.5
51.4
37.8
30.6
MDETR (Kamath et al., 2021)
ENB3
COCO, VG, Flickrö0K
0.3M
4.7
4.6
GLIP (1i al., 2022)
Swin
0365, OI, RefCig/+, etc
17.5M
21.2
33.2
37.7
18.9
10.8
ALL
mm GDINO (Zhan et all, 2024)
Swin-B
GoldG, 0365, COCO, etc
12M
20.8
33.1
31.9
19.8
14.1
FIBER (Dou et al., 2022)
Swi-BB
COCO, CC3M, SBU, etc
4M
22.3
34.8
38.6
19.5
12.4
Real-Model
Swin-B
Real-Data
0.18M
36.5
52.1
54.4
33.2
25.5
Table 2: Evaluation results on the DOD benchmark.
LOD method
Backbone
Source
#lmg
Full
Presence
Absenee
OWL-V2 (Minderer et al., 2023)
ViT-L
WebLI
10B
9.6
10.7
6.4
UNINEXT (Yan al., 2023)
VIT-H
0365. RetC/g/+
0.7M
20.0
20.6
18.1
GDINO iL all, 2024h)
Swin-R
CCAM, 0365, RefC/p/s,
5.8M
20.1
20.7
22.5
mm-GDINO (Zhao al,, 2024)
Swin-B
GoldG, 0365, COCO,
12M
24.2
23.9
25.9
2023)
RNI01
CC12M, SBU, VG,
APE-B (Shon et al,, 2024)
VIT-L
LVIS, 0365, RefC/g/+, etc
2.6M
30.0
Real-Model
Swin-B
Real-Data
0.18M
34.1
34.4
33.2
Tables 2-3 shows the evaluation results on DOD and RefCOCO/g/+ benchmarks, respectively. The results are similar to those the OmniLabel benchmark. Using a small amount of training data, our Real-Model achieves favourable results under various metrics, which surpasses existing LOD methods. This performance gain due to our Real-Data data pairs, where diversified language expressions improve the generalizations of language and object alignment. As result, Real-Data datasets, with the same images and objects but diversified language descriptions, benefit Real-Model in achieving state-of-the-ar performance. In addition, the evaluation results OVDEval and the application of our method to other LOD models are presented in Sec.
4.2 ABLATION STUDY
We train our Real-Model by using three training data pair configurations (i.e., A, B, and C forms) and evaluate the corresponding L.OD) performance on the Omnil Label benchmark. We randomly select 94k images from 0365 and Ol datasets covering all categories, which is a subset of Real-Data. These images, together with target objects and raw expressions, constitute our original training data pairs with an amount of 933k (i.e., A form). Moreover, we use SigLIP IP to filter out some data pairs where expressions do not match the target object. The remaining pairs are 695k (i.e., B form). Furthermore, we use Real-LOD to re-align mismatched pairs to add them back to B, which increases the number of pairs to 863k (i.e., C form). We use data pairs in A, B, and forms to train Real-Model separately and evaluate the corresponding performance. This helps analyze how our Real-L.OD improves LOD from a data-alignment perspective.
Tab.4 shows the LOD results via three data configurations (i.e., A, B, and C. forms). It demonstrates that on the COCO test set, Real-Model achieves 21.2% AP by using all the training data pairs (i.e., A form). After removing data pairs with mismatched expressions, Real-Model increases to 22. 22.2% (i.e., B form). This improvement indicates that data quality essentially benefits LOD performance. Then, our Real-LOD refines filtered data pairs for supplement, which improves Real-Model to 24.2% (i.e., C form). It shows that Real-LOD increases data quantity with high quality, leading to further LOD improvement. The results the other test sets (Lee, 0365 and Ol) indicate

---

Published as a conference paper at ICLR 2025
Table Evaluation results on the RefCOCO/g/+ benchmark. ** indicates that the model employs RefCOCO/g/+ for training.
RefCOCO
ReiOoCO+
RefCOCO0g
LOD method
Backbone
Source
fimg
val
testA
testB
val
testA
testB
val-u
test-u
MDETR (Kamath et al, 2021)
ENB3
COCO, VG, Flickr30K
0.3M
73.4
58.8
57.1
APB-A (Shen et al., 2024)
VIT-L
COCO, LVIS, O365, etc
2.0M
34.2
34,8
36.1
33.5
32.3
36.0
38,9
40.5
Real-Model
Swin
Real Data
0.18M
74.0
79.6
66.0
76.4
83.1
68.5
80.8
81.2
GLIP" et al., 2022)
0365, Of, RefCly/t, etc
17.5M
59.4
59.4
60.7
GDINO" (1.iu al,, 2024b)
Swin-R
CC4M, 0365, Reillg/t,,
73.6
64.1
78.3
APE-B" (Shen et al., 2024)
VIT-L
LVIS, 0365, RetCig/t, etc
2.6M
84.6
89.2
80.9
76.4
82.4
66.5
80.0
80.1
Real-Model"
Swin-B
RetCig/+, Real-Data
0.24M
91.3
93.1
88.0
85.4
90.3
78.6
88.4
89.0
Table 4: Ablation study on OmniLabel benchmark. Our training data pairs consist of images, target objects, and expressions (expr). We adjust training data pairs by processing raw expr differently (i.e., SigLIP filter and Real-LOD) and evaluate the corresponding performance. Note that we use a subset of Real-Data.
Test subset
Training data type
#lmg
AP-des
AP-des-pos
AP-des-S
AP-des-M
AP-des-L
raw expr (A)
933k
21.2
59.4
31.3
21.1
18.6
COCO
raw expe wiitlte (B)
695k
22.2
59.4
32.4
21.9
19.4
raw expr wifittr Real-LLOD
8634
24.2
59.6
35.2
24.2
21.1
expr
933N
27.6
43.1
39.8
25.5
17.9
0365
expr wiftlter (B)
28.5
43.7
40.9
26.2
raw expe wifilter Leal-LOOD (C)
raw cxpt (A)
9333
30.5
43.0
37.2
30.3
23.2
raw expr w.filter (B)
695k
31.4
43.5
38.1
31.2
24.0
r&W expe w.lllter + Real-LOD (C)
8633
33.5
44.9
42.2
32,9
24.8
similar phenomena. When training Real-Model, data quality also has an influential impact LOD performance, especially when the data quantity is increasing. Our Real-LOD re-aligns mismatched object and language pairs to increase data quantity while preserving data quality. To this end, Real-Modes learned with re-aligned data in C form performs best on the OmniLabel benchmark.
4.3 ANALYSIS ON COMPUTATIONAL COST
In our Real-LOD, we also leverage two strategies to further mitigate workflow time costs: 1) We leverage SigLIP to exclude 75% of training data from our workflow, leaving only 25% to be processed. 2) We set the max cycle number of our workflow as 4 to trade off time cost and performance. We elaborate on our computation cost in Tab. 13. For refining one expression, we report the average number of calls for each step and the time cost during each call. The time cost reported based on workflow takes 1.579 seconds in total, with an average cycle number 3.08. We also provide the distribution of iteration numbers in Fig. 18. Note that the max iteration number here is 10 for investigation. In addition, our workflow is completely offline without bringing additional computational burden to LOD model inference.
48 V100 32G GPUs for workflow execution. For refining one expression,
CONCLUSIONS
Re-aligning language visual objects has been developed from manual descriptions automatic VLM generations. The data pairs are scaling up to advance the connection performance of LOD. The generated descriptions may not match the objects due model hallucinations. We thus propose Real-LOD to refine the language expressions gradually via agentic workflows. The data quality is preserved along with the increased data quantity. We train prevalent LOD model using our data to largely surpass existing LOD methods. Our automatic workflow contains the expanding potential to re-align language descriptions of any objects. With an open vocabulary detector to locate objects with short category labels and VLMs to expand expression, our Real-LOD will continuously produce high-quality training pairs to scale up LOD performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was funded by NSFC (No. 62225604, 62176130), the Science and Technology Support Program of Tianjin, China (No. 23JC/DJC01050), The Supercomputing Center of Nankai University partially supported computation.
ETHICAL STATEMENT
We declare that our research does not present any potential ethical issues. The study does not involve human subjects, sensitive data, methodologies that could result in harmful outcomes or biases. All data used this work publicly available, and no privacy or security concerns are implicated.
REPRODUCIBILITY STATEMENT
Transparency and reliability crucial research. In this statement, summarize the measures taken to facilitate the reproducibility of our work and provide references to the relevant contents in the main paper and appendix.
Source code. We intend to make our source code, model weights, and datasets available to the public following the paper's acceptance. will allow the following researchers to access and utilize our code to reproduce our experiments and results. The detailed installation and execution instructions will be listed in 'README.md'.
Experimental setup. We provide the basic implementation information of our Real-LOD in Sec. 3.1 and Sec. 3.2. Besides, we provide the experimental setup and evaluation settings in Sec. 4 and Sec. I.3. The details of Real-Data are listed in the Sec. 3.3 and Sec. 4. Moreover, the training and architectural details of Real-Model can be found in the Sec. 1.2 and Sec. 1.4 of the Appendix.
We provide the above resources and references to ensure the reproducibility of our work. It enables fellow researchers to verify our method. We also welcome any inquiries or requests for further clarification on our methods.
REFERENCES
Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi-level multimodal common semantic space for image-phrase grounding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes Ilm better reasoner, 2023. arXiv preprint arXiv:2310.22
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment, 2021. arxiv preprint arXiv:2
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. In International Conference on Learning Representations, 2024.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li. Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark, 2019. aix preprint aiviv 190600715.
11

---

Published as a conference paper at ICLR 2025
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. In International Conference Learning Representations, 2025.
Yixin Chen, Sai Kumar Dwivedi, Michael J Black, and Dimitrios Tzionas. Detecting duman-object contact in images. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Ronghao Dang, Jiangyan Feng. Haodong Zhang, Chongjian Ge, Lin Song, Lijun Gong, Chengju Liu, Qijun Chen, Feng Zhu, Rui Zhao, et al. Instructdet: Diversifying referring object detection with generalized instructions. In International Conference Learning Representations, 2024.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, al. Coarse-to-fine vision-language pre-training with fusion in the backbone. In Advances in Neural Information Processing Systems, 2022.
Shangqian Gao, Burak Uzkent, Yilin Shen, Heng Huang, and Hongxia Jin. Learning to jointly share and prune weights for grounding based vision and language models. In International Conference on Learning Representations, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujuu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations, 2024.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolhench: Towards stable large-scale benchmarking on tool learning of large language models. In International Conference on Learning Representations, 2024.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. Grit: General robust image task benehmark, 2022. arXiv preprint arXiv:2204. 1355333
Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. L2MAC: Large language model automatic computer for extensive code generation. In International Conference on Learning Representations, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representutions, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetrmodulated detection for end end-to-end multi-modal understanding. In JEEE/CVF International Conference on Computer Vision, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Fanjie Kong, Yanbei Chen, Jiarui Cai, and Davide Modolo. Hyperbolic learning with synthetic captions for open- open-world detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification. object detection, and visual relationship detection at scale. International Journal of Computer Vision 2020.
12

---

Published as a conference paper at ICLR 2025
Liunian Li, Pengehuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, and Jenq-Neng Hwang. Grounded language-iman image pre-traine nee-training. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a. URL https://llava-vi. gi sithhub.r wttpii/////////// 20000////22-00000 -101-00000 -00
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents, 2023b. arXiv preprint arXiv:23 1.00437..
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, 2024b.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Agenthench: Evaluating ilms as agents. International Conference on Learning Representations, 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision, 2021.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2016.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In Advances Neural Information Processing Systems, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: captioned photographs. In Advances in Neural Information Processing Systems, 2011.
Describing images using million
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. arXiv preprint 1rXiv:1912.01703
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. arXiv preprint arXiv: 2533.1
Renjie Pi, Lewei Yao, Jianhua Han, Xiaodan Liang. Wei Zhang, and Hang Xu. Ins-detclip: Aligning detection model to follow human-languse instruction. In International Conference on Learning Representations, 2024.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phene thrasee correspondences for richer image-to-sentence models. International Journal of Computer Vision, 2017.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2016.
Reworkd. AgentGPT, 2023. URL https: thub....ttttuuuu.. com//rewrrd//ggenttr.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, 2022.
Samuel Schulter, Yumin Suh, Konstantinos M Dafnis, Zhixing Zhang, Shiyu Zhao, Dimitris Metaxas, et al. Omnilabel: A challenging benchmark for language-basee object detection. In IEEE/CVF International Conference on Computer Vision, 2023.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In IEEE/CVF International Conference on Computer Vision, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018.
Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 2023.
Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Alcksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vims. IEEF/CVF International Conference on Computer Vision, 2023.
Significant Gravitas. AutoGPT, 2023. URL AutoGPT.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual- linguisic representations. In International Conference on Learning Representations, 2020.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
14

---

Published as a conference paper at ICLR 2025
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang. Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases, 2024. arXiv preprint 23Xi:2330600301...
Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In IEEE/CVF International Conference on Computer Vision, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. -hain-of-thoughtt prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating object detection with flexible expressions. In Advances in Neural Information Processing Systems, 2023.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
Peng Xu, Haoran Wang, Chuang Wang, and Xu Liu. Caca agent: Capability collaboration based ai agent, 2024, arXiv preprint 115
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching Ilm to use tools via self-instruction In Advances in Neural Information Processing Systems, 2023a.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023b. arXiv preprint arXiv:2303.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? benchmark for comprchensive open- open-vocabulary detection. In AAAI Conference on Artificial Intelligence, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality, 2023. arXiv preprint arXiv:2304. 14178.
Yohei Nakajima. BabyAGI, 2023. URL https: / thub. httpub..com//ohiinaaaaaaaaa ima wwwp/naaaaaaaaaaaaaaaaaaaaaaaaaa
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In International Conference on Learning Representations, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, 2016.
Zishun Yu. Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. Samathcal B S-coder: Value-based deep reinforcement learning for program synthesis. In International Conference on Learning Representations, 2024.
15

---

Published as a conference paper at ICLR 2025
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In International Conference on Learning Representations, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVE International Conference on Computer Vision, 2023.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2023.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, lenq-Neng Hwang, and Jianfeng Gao. Glipv2: unifying localization and vl understanding. In Advances in Neural Information Processing Systems, 2022.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Change Loy Chen, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In Advances in Neural Information Processing Systems, 2024.
Xuying Zhang, Xiaoshuai Sun, Yumpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection, 2024. arXiv preprint arXiv:2401.0236:
Lianmin Zheng, Win-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023.
16

---

Published as a conference paper at ICLR 2025
APPENDIX OVERVIEW
We provide an overview to present clear understanding this section.
In Sec. A, we provide an overview of the pipeline for language data generation.
In Sec. B, show more examples of raw expressions corrected by our Real-LOD.
In Sec. C, present visual comparisons of existing LOD methods under various queries.
In Sec. D, we illustrate several examples of how Real-LOD refines raw expressions.
In Sec. E, we provide additional evaluation results on the LOD benchmark.
In Sec. F, we present pseedo-code proposed Real-LOD workflow.
In Sec. G, we show prompts for LLM and VLM to execute different tasks.
In Sec. H, provide the statistical results of the computation cost.
In Sec. I, we outline the specifics of the training, evaluation, datasets, and model structure.
In Sec. J, we provide additional discussion of our paper.
In Sec. K, we provide more analytical experiment for Real-LOD.
A EXPRESSION GENERATION PIPELINE
VLM Prompt
Original Expressions
Enriched Expressions
Real-Data
Sannoo info Canepoyy <img>
LLM
Filler
cxpr
espo
exP-
etp-3
LLAVA
expp-2
VICUNA
SigLIP
Filtered
exp-3
Figure 9: overview of our language generation pipeline for Real- Rata- In this pipeline, we first use LLaVA-v1 v1.6-34B (Liu et al., 2024a) to generate descriptions. For each object, we randomly select two prepared prompts presented in Tab. with an image and corresponding category for LLaVA to generate expressions. Second, Vicuna-vl 1.5-13 1-1 (Zheng et al., 2023) is introduced to generate synonyms to expand the number of expressions using the prompt in Tab. We repeat the process two times for each expression. Then, use LigLIP to filter expression -imaio pairs with low scores. Finally, we maintain correct data pairs and refine filtered expressions via our Real-LOD to build the final dataset Real-Data.
17

---

Published as a conference paper at ICLR 2025
B RE-ALIGNMENT EXAMPLES OF REAL-LOD
man wearing black haf correctly referiing the image? not, please respond with correct one.
Is the expression yellow flower in vune' correctly describing flower image? not, please provide with contoct onc.
No, the provided expression has errors in it. correct can green helmet for motorcycle rider'
The expression has flaws it. Here correct wall.
paper suufloowe on
wearing jackef correctly describing the person in image' If not, please respond
Decide the expression le man in a black shirt jeuns' correctly describing the person in image. noi, please provide with
No, the provided expression is wrong one. correct one can person stading near blue sout wearing green shirt and white punts'
Yes. The provided expression is comect.
Please specify the expression is white plastic chair correctly describing the chair image. If is wrong, please respond with carrect
8
expression brrow wooden chair with purple and white clathing correctly describing the chair in image? please provide with conrect
The expression white plastic chair' Here light brown finish'.
Yes. This expression correct.
'wooden chair
Figure 10: We show examples of the re-alignment by Real-LOD, Real-LOD can correct wrong expressions and remain correct ones.
18

---

Published as a conference paper at ICLR 2025
VISUAL COMPARISON RESULTS OF LOD MODELS C
Query: "This item is used keep warm in colder weather:"
Query: "Woman in wedding dress in suit."
Query: "Pillow placed at the head of the bed."
Query: "These two people each have pink surfboard."
Query: "The fire extinguisher on the left."
Query: "Cows that are laid down."
(a) GLIP-L
(b) APE:B
(c) (c)mm-Gii IDM-GDINO
(d) Real-Model
Figure 11: Visual comparison with existing language and vision detectors. The backbone of GLIP and APE-B is VIT-L, and the backbone mm-GDINO is Swin-B, We use 0.3 as the score threshold for the fair comparison. For convenience, use bbxs with different colors to distinguish each model. The color we used for Real-Model red.
19

---

Published as a conference paper at ICLR 2025
D EXAMPLES OF RE-ALIGNMENT BY REAL-LOD
In Fig. 12- 17, we show several examples of how Real-LOD works.
Onject
[61.00 04, 527.28, 292.15)
Caption
bounding large
sunding on moky and visitle bockgroand herr appoars liok ing to the The bear's for is prolimineally whitc, with sume derker arca around the eyes and ears. The pround beneats the bear is mix of and red rocks, and there are putches of gromn vegetation. The hoors position relative to the bounding box centrll The bear's far looks suggesting
tlack, funy
with some vesctution
been water recensiyy.
you specity expresioo of i. polar bey usliiing en teraale teznale: reforiig to my tagget object?
If tull please sespona cerrett cxpression.
Request
Reasening:
The
deseribes
negg calegery "poler
about walking
Plunning
radky lumain.
Actions:
Agent
Responor The expression (d Lerpe polor bror wailingg on nochy ternate nocumnely describing the features of the sugee object.
The exprossun (dierrg galer bear suiliing on toraiii) is sorrect onc..
Figure 12: An example of how Real-LOD handles correct expressions.
cooker
Coordinates
[1144,,
Captiso
The beuasing box rics it prodeninattly incleting plonnd
display
Thhimee cocker appoon he moderr aleett movtl designe canting nin. Is not cummtty ose, excrcand by tue shoraoe of sictte oven or hest. "The mee cackin pepettioned diseronnd oftte tmags with obgaoe tockgrennd indicatin
photogroph.
Cinyae egecity the experscon
stdo vouve e1o5) icnetortng my onge object? If nun plaas rupeer with cornat exprosice..
Request
sooker phose
caaprey
cotagery ahat
angi is voury plas for Actis wlere te LLM sowile expression click ourreraco Do noe expeesite.
Planning
Actionc If Tool mane!
parmetend ('hrrge: Captin,.... Respenss.... .111 avenl
Peponee exposioo oxpession
loverege
LLM New
adite oler BV deeereeer
Reflection
Feelback. ig
doe los atayyy
Agent
anverancce about coloraae the the
intormatos
Remening Road en the Sodfick mevoo the din is Tar cerals the exprssios saddi contore connor 1 Andion to sop
Planning
Respree exprostion
anou) wrrmggggg
The copression iaubhii AAAI VaRRE one, and corret oqnemaass coold auut MoA.c
Figure 13: An example of how Real-LOD corrects a wrong expression. It consists of the 'Rewrite' action.
20

---

Published as a conference paper at ICLR 2025
Calegery
chinggaaee
Caprios
Thi ang hase las moloe
twe si viive in The silte leutted in the enaie of
socts eour cusced erunaas Thk rours husa sumr aurting
and for dasng sociiiitas and pas siced avars sciling wilh
rover the wolk.
objuat
Resseiig The dour odl of diding
nan reme agane, He thn astemat trouee the care sippet teffti
ser'l VLM io socuut
cooerppaato VLl supp tenano de cerre internation
nedias (Tiled, nawe uut. Thad anntten" 12 sProoet Desstios rucomaly cttiing Vajeet crerls)
Phanaing
Regpoee aill prasuna get oest siforaaiio abpoct
meniica
Tealine
VE. cuspet: 1. Th ande hewei The ratic mack wodd.
Refbecticn
Frelbauk The pieae the oele tewwn dine the oleer of rakee wilch macche
Reossing foother reesetteee cate wher
Antinn
Ragesas oppcaiion Na Anams
exprossun (thrmhier Aovent
Figure 14: An example how Real-LOD handles an uncertain expression. It consists of the 'VLM with object crop' action.
Objert Catrgary
Cordiztto
J4Z 34545, MUSE 18415)
Capekon
"ie "orrtteening hex contamo wes with york and whtte stngos, The lervet hraghts reel appaanto to beardo of ouf ahe rulacia Thely intudied for nipling hunds
dolkg poddoced
of ooms
appllann with glass
abric a derc whec macowinn The lachaa oumre wirch makk navel matblec Imngrvend reneas siemis name
Casyse qpacls oe if LAke antend urfering my sugan objert trass plane supard with cecrict
Repeot
Reassdag The expesaiio desc the get crtepas laghll
color of vettor the livd not the cmatc whe encic Bavid the
where encerin
be
Auttor whbre
pencaiie
Aetioe (Todl oe wor 9I swetii Iamage ditaug Vieec oner)
wiee (Pr "oersiow What
saase to furber assos dee pensoniio ot canca oup oxprsaiess flease nattiola
Reflerke
prn Cow bacc which and wiili.
Phaniig
cnd's Anoweed wndl nevert Artion
Figure 15: An example of how Real-LOD handles an uncertain expression. It consists of the 'VLM with object crop' action.
21

---

Published as a conference paper at ICLR 2025
Object Category
his
Bbox Courdinates
94,
71.92)
Caption
The bus in the red bounding box is modem, red and while vehicle. The bus appears to be statioary, possibly at bus sop traffic light. It made metal and has glass windsheid. The bis likety used for public cransporation, providing
resdenss and visitors it not intoracting eirectly part uhba
with other ousidee the bounding inchalos
Can specify the expression of aa long rod nain stcel mach) reforring target object? not, plaase respond with conect
Request
Reasoning: indoctation,
dossibes object exteueyy, expecssion fong rof
cnlegery
where
taac wamp. plan Action usc rewnit expressoon
Planning
corfectters new expeession
Actions: ("Tool, ramet: 'LM. 'lool parametens (Promph: Image Caption, Responss, ...]
Respoase: The expression la roale on sted (nac) contains flaws, therefore. Iwill levenaee LLM crease altemaaivve expression for you.
Tool
LLM outpor: New Description lnd and riite bus an the riebs ride the toarg:
Reflection
Feedback:: The phrase lned and white Ans the right side of the inazei describes the catesory 'buu and provides information coloes. However,
doeurt mention location of bus.
fandbask, maan am uncertan obsect locations is the image So plan for Action 5 where can crop the whole image fic VLM to fortier to percetve. The descripcion of VLM supplements the current infomationn for
Reasoning:
cunduct more through expresiio camination.
Planning
Action: [[ Tool nume: 'VLM,, "lool_ pommetenn (Promp::
Agent
Queatione:
image?, 'haae cliiing!
highlight)))
Response:
sequie supplementayy
evaluste cument exprossion more thoroughts: This might take while, thank you for your patience.
informetion
Tool
VLM Yos, the bas the right side the
Reflection
Feodlack The plaase (red and whir bus on dhe right ride the image doscribes the trgget enegury of buo ind providas information about in colie and location
Planning
fesdlbeck,
expecesum and
ride (osscc)
plan stop.
Response: The expeession la fulaa tain stoel trakk is wrong one, and correct one could tuad and ahhii foz on the right skde of Insacc.
The expecosion long and maan
strel mrck) is and cornect could be lzed and vebite Aus the right vide the
Figure 16: An example of how Real-LOD refines a raw expression. This workflow consists two actions ('Rewrite' and 'VLM with object highlight')
22

---

Published as a conference paper at ICLR 2025
Object Category
bench
Bbox Coordinates
[43.59, 345.52, 184.37, 88.21]
Caption
The bench the red bounding woodem bourdwallk
white wooden bench situated posiiuued the left the image and facing towands the right side of the image The bench is emppy and appears to be stationary: located
lamp and railing, suggesting dosigned public space, possibly park beach
scating and is likely used by visitoes to the area rest and enjoy the view. The bench's white coler contrasts with darker of the surounnding weodon
you specify the expression of (a whbit boach with the whining referring
object?? not, please
respond concet expression.
Request
Reasoning:
The plrase deseribes the target category of banch and provides. information color
sunliggt..
However, the caption does not memioon anything about the sun shining on the benchi, Rased this mation,, reason the state where am object
for re-pereaive. deseription of VLM supplements the current information for me conduct more thorough expression examination.
where crop
object
image
Planning
pammeters' (PPrompt: 'Questiona11 Is the bunch the image white?2. the bunch in the image located in place where the sun can shine it?, Tmmago editing" 'abjert crop']]
Action: [ ('Tool same? "VLM,
extra information image to firther asses the correctnees current expression. Please wait of moment.
Response: will peompt
object
Toul use
VLM outputt: 1. Yes, the bench the image white. 2. Yes, the hench located place where the shime
Feedback:
The phrase is whbite bencc wilh the shining on i! describes the eategory of provades information directly shining on the bench.
Reflection
and lighting
But
Reasoning: Based on the feedback. I reason the state is where I for Action where cun image
Agent
uncertain the suroundings
to perceive. description of VLM supplements the curent information for me conduct more thorough expression examination.
object region
('Tool name': 'VLM', parameters': /Prompt:: 'Questions:.. the sun dinecily shining on the white bench", Image editing". 'extumned object crup)))]
Planning
Response: will prompt VLM
information about the object expression.
image assess correctness moment.
Tool use
VLM output: 1. Yes, the sun dircetly shining the white bends.
Reflection
The phrase inuhhite banch weitt the suu shining on ari describes target category of 'benelt' and provides information about the banchtss color and the saa shining on it.
Peedback:
Reasoning:
Rased on the foulback, resson the state is where am certain the exprosiion (a whiite with ohiming
Planning
So
stop.
Actions: []
Response:
The expression la whdie bench with the use shining on àt correct.
The expression (a sobitt brach with the sun whiming on is connect
Figure 17: An example of how Real-LOD handles uncertain expression. It consists of VVLM with object crop' and 'VLM with extended object crop' actions.
23

---

Published as a conference paper at ICLR 2025
E ADDITIONAL EVALUATION RESULTS
E. APPLICATION TO OTHER LOD MODELS
In this subsection, we conduct experiments to demonstrate the generalization ability of our method. We apply our Real-Data UNINEXT (Yan et al., 2023) and tiny version of mm- nm-GDINO (Zhao et al., 2024). We report the results of OmniLabel and DOD benchmarks in Tab. 5 and Tab. 6, respectively. For UNINEXT, the AP-des of OmniLabel and of DOD are significantly increased to 23.4% and 24.2% without adjusting any training parameters. Notably, we only train UNINEXT for five epochs based relatively small backbone (i.e., ResNet-50). For mm-GDINO with Swin-T backbone, the AP-des of OmniLabel and of DOD are significantly improved to 29.9% and 30.8%. The results demonstrate the holistic nature of method.
Table 5: Application to other LOD models on OmniLabel benchmark.
Test subset
LOD method
BackBone
Real-Data
AP-des
AP-des-pos
AP-des-S
AP-des-M
AP-des-L
10.1
31.2
19.7
10.4
MM GDINO (Zhao et al., 2024)
Swin
30.9
COCO
3.6
UNINEXT (Yan et al,, 2023)
ResNet-50
14.6
41.4
24.8
14.8
16.1
24.7
32.7
12.7
7.9
MM-GDINO (7ho al., 2024)
Swin-T
29.6
44.4
49.0
26.2
19.6
0365
7.6
13.9
10.8
8.1
4.6
UNINEXT (Yan al., 2023)
24.3
38.2
37.7
21.3
17.3
20.6
30.4
37.2
12.9
MM GDINO (Zhao et al., 2024)
Swin
33.7
45.1
48.5
30.8
25.1
01
9.2
UNINEXT (Yan et al., 2023)
2023)
ReaNet-50
25.3
36.3
36,9
19.1
17.0
26.7
33.3
13.9
9.3
MM-GDINO (Zhao al.,
Swin-T
29.9
44.3
47.5
26.9
20.7
ALL
6.0
10.4
9.6
5.8
3.5
UNINEXT al., 2023)
23.4
37,3
36.3
20.7
17.3
Table 6: Application other LOD models on DOD benchmark.
LOD method
BackBone
Real-Data
Full
Presence
Absence
23.0
21.9
26.0
MM-GDINO (Zhao al., 2024)
Swin-T
30.8
30.3
32.7
10.7
10.6
10.9
UNINEXT (Yan al., 2023)
ReaNet-50
23.2
26.9
E.2 COMPARISONS WITH STATE-OF-THE-ARE LOD MODELS ON OVDEVAL BENCHMARK
In this subsection, we show evaluation results in OVDEval benchmark (Yao et al., 2024). The benchmark contains 15. 1/ images with 28.1K bbxs and 10.11 10.1k language expressions. The dataset is divided into several sub-datasets according aspects such 'color' and 'relationship'. We select language-based sub-dataset to compare our Real-Model model with other detectors. Tab. shows the results. The OmDet method performs best in the 'Relationship' sub-dataset. The reason that the testing data is collected from HOI (Chen et al.. 2023) dataset. which is used to train the OmDet model. Our Real-Model outperforms existing LOD models on average in this benchmark.
Table 7: State-of-ther comparison on OVDEval benchmark. We report evaluation results AP (%) of each sub-dataset. 'Source' refers to the source training images. '#Img' refers the number of images.
LOD method
Backbone
Source
#img
color
material
Position
Relationship
Negation
Avg
GLIP (Li ei al., 2022)
Swin-L
0365, OI, RefCig/+,
17.5M
6.7
15.8
48.1
33.2
51.8
31.1
OmDet (Yao al., 2024)
ComvNext B
0365, GoldG, HOLLA, A,
1.1M
24.5
22.5
47.7
51.8
55.8
40.4
FIBER (Dou al,, 2022)
Swin.B
coco, CC3M, SBU,
4M
9.4
17.7
48.1
33.2
58.1
33.3
Real-Model
Real-Data
24

---

Published as a conference paper at ICLR 2025
F ALGORITHM OF AGENTIC WORKELOW
Algorithm Pseudo code of Real-LOD. We show the detailed code of our workflow in flexibly leveraging tools to re-align raw expressions to given objects.
Input: image object locations O, caption C, raw expression Er
Output: re-aligned expression ER
I: agent -Real-L.ODDiitt assistant, init vim_tool, init Ilm_tool) // Agent initialization
2: info po_ <- --"mmage: ... "expressions": [E-]] //Initialized input
ileration
4: stop + False
5: solued + False
6: while not stop do
//Stage 1. planning based current info_pool
reasoning, actions, values +- agent.assistant
9:
/ //Stage 2. tool use and update in
10:
if (actions is not empty) and (iteration mar mar_iter) then for action in actions do
11:
tool_ mol_name, tool tool_ppootteeoooo params case tool_name is "VLM", update in pool with agent. vlm params) case tool_name is "LI."", update info_pool with agent. ggent.llm_a ham__toollie (tool_prrams)) end for
action
13:
14:
15:
16:
//Stage reflection tool outputs update Seedback with agent.lm tool ileration <- ileration + else if actions is empty then
18:
19:
20:
solved +- True reach correct expression
21: end if
22: stop + solved or (iteration == max_iter
23:
end while
24: ER <- iinfo_pool "rpoos""rprrssionns"]]
25: return ER
G PROMPTS FOR LLM AND VLM
G.1 PROMPTS FOR RAW EXPRESSION GENERATION
Prompts for VLM to generate expression
1. For the given image <image>, please provide unique description for the <object> in the <boxes>.
What the content depicted the <object> located in the area <hoxes> of the image <image>?
3. Please describe the <object> in the area <boxes> of this image <image>.
4. I would like know the description of the <object> in the <boxes> of the picture <image>.
Kindly describe <object> the area <boxes> of picture <image>.
Give me detailed descriptions the <ohject> area <boxes> color, material, attribute, etc.
image <image>, including
7. What's 's the difference between <object> in area <hoxes> and other <object> in this image <image>?
What the relative position of the <object> in <boxes> of the picture <image>?
What's relationship <object> area <boxes> with its' surrounding objects?
Prompt for LLM to diversify raw expression
synonymous expression provider. I will give you text of phrase short sentence, which is an expression that describes a object while mentioning some other objects. And you will reply to me with new expression that the same semantic meaning and describes the same main object as the provided expression. The new expressions should also he phrases or short sentences longer than 25 words. Do not write explanations on replies. Do not repeat.
I want you
Table 8: Prompts for raw expression generation.
25

---

Published as a conference paper at ICLR 2025
G.2 PROMPTS FOR REWRITE TASK
Task Description
You are an excellent text analyst and generator. want a short text description that correctly describes chosen object in an image. Now aaready have one description, but there might be mistakes in it, and I want you help with this. I will provide you with the following background knowledge: 1. Image Caption: caption describing the content in image.
Chosen Object: one or more objects chosen in this image describe their corresponding coordinates. The top-left corner of the image has coordinates [0, 0] and the bottom-right corner has coordinates [1, 1J. Each object is represented as ("id": unique object identification, "category name": object category. 'box" [[tp-left top-left y. box width, box height]).
Other Object: other objects image and their corresponding coordinates, which are provided you the same format as the 'Chosen Ohject'.
4. Object Description: short text for you to analyze (this description could he correct, partially correct wrong).
As an assistant, analyze the 'Object Description' and generate 'New Description" based it, which correctly describes the chosen object:
1. Your new description should be centered on the chosen object, and describe the correct object category
Your new description should be consistent with information provided by the given caption and with general knowledge
3. Your new description should be short phrase and has reasonable meaning
One In-context Example
Image Caption: The image showeases and wooden cabinets. The kitchen features white refrigerator, a sink with a window above it, and a stove with an oven below it. Additionally, there is dishwasher and a washing machine in the kitchen. neatly arranged, and the overall appearance of the kitchen is tidy and inviting. Chosen Object: [('id':O, "category name": Home appliance' 'box': :00668, 0.52, 0.15, 0.051), ('id':1, "category name" Hooee appliance', "box':[0.0 :0000, 0.26, 0.24, 0.741), { ('id' :2, "category name", Hoome appliance', 'box': [b055, 0.56, 0414, 0.28]). { (", 'category name' : Homme appliance', 'box': [0.68, 0.22, 0.15, 0.06]]. ( :4, 'category name": Hooee appliance', 'box' (00667, 0.57, 0.16, 0.21])]
Prompt:
and well-organized kitchen wooden floors
The appliances
Other Object: Object Description: 'appliance for cleaning dishes with water and detergent'
Response: Reasoning: The description is centered on home appliances, but it only refers to the washing machine in the kitchen, which is only one of the chosen objects. The object category is 'home appliance', which also includes the refrigerator, the sink and the stove mentioned in the given caption. So the description does match all the objects that helong 'home appliance' in the image. can rewrite description into a more specific one. New Description: 'home appliances of refrigerator, sink, stove, dishwasher and washing machine'
Python code together with above text prompts directly sent to LLM--fr-Rewriting
messages = [("role": system, "content": task_descriptionn]
for example in in_contextt_xxappess
messages, append r(" user "content" "eampeeeeeammmeepppnptt"]]
messages. append (("role":as) assistant, "content" "expneenttexxmmleeeeeesrsse
"}]) caption "\n"
content=f"Image caption:
ff Object: " chosen_ object "\n" content+ Other Object: " other_objeed "An"
content==f Object Description:
object_description "An"
messages ((" user, "content": content))
Table 9; Task description and an in-context example for LLM-for-Rewritee tool. There are five in-context examples used in the text prompt, we show one example of them here for simplicity.
26

---

Published as a conference paper at ICLR 2025
G.3 PROMPTS FOR REFLECTION
Task Description
You are an excellent text analyst. want to get correct descriptions of target object an image. Now I already have phrase describing this target object, but the texts might contain mistakes and I want you help with this.
I will provide you with this phrase to be checked, along with 'Object Category' and 'Caption' reference information:
1. Object Category: the exact category name of my target object.
2. Caption: a long text describing content in the image, information provided in this caption is correct. Note that some details in the image might be missing in this caption. Given reference information, your task verify phrase correctly describes my target object.
## Process Instruction
1. Correct phrase: If the phrase describes the target category and provides similar information that can be found the caption, this phrase is correct.
Uncertain phrase: If the phrase describes the target category provides information that missing in the caption. this phrase uncertain. Please tell me what information in this phrase is not mentioned in the given caption. Extra information could be object color, object material, object location in the image, object action, object relation with other objects in the image, etc. Wrong phrase: If the phrase describes different object category, or the phrase provides information that conflicts with the caption, this phrase wrong.
One In-context Example
Prompt: Object Category: Laptop
Caption: image, there are two women sitting at table, both focused on their laptops. One the women is holding up her middle finger. possibly as a gesture of defiance or humor. On the table, there is bottle. a cup, and a laptop being used by one of the women. The woman with the laptop is wearing scarf, and the other woman positioned the other side of the table. Drink: A clear plastic bottle water laptop is black and silver Dell computer. The woman using it while sitting at the table. Table: The table is dining table where the woman is sitting with her laptop and a bottle of vater.Bottle The bottle is a clear plastic bottle of water. It is placed on the table in front of the woman with the laptop. Phrase: 'black and silver Dell computer'
the table. It placed front of the woman with the laptop. Laptop: The
Response: Feedback: The phrase describes computer and provides its color and brand, 'Computer' similar with 'Laptop' and the caption claims the laptop in the image to be a black and silver Dell computer, this phrase is correctly describing the target object.
Python code together with above text prompts directly to L.MM-Reflecto
messages = [["role": system, "content". task_descriptionn) for example in
messages append r("role" user, "content" example["prompp messages. append (( ("role" assistant, "content" "xample["respones
"}))
content=f"Objeet Category: content=="" content== phrase messages r"pole""" rrolee: user, "content"; content])
category "\n"
caption "\n"
"An"
Table 10: Task description and in-context example for agent reflection. There are five in-context examples used in the text prompt. We show one example of them here for simplicity.
27

---

Published as a conference paper at ICLR 2025
G.4 PROMPTS FOR RESPONSE GENERATION FINE-TUNING DATA
Generate question uncertain state
Task Description
You excellent text analyst. I have phrase that describes object in image. have already analyzed this phrase based a caption. And have reached the conclusion that this phrase uncertain. want you to help me generate questions asking about the missing information in the caption.
I will provide you with statement of the reason why the phrase is uncertain. As an assis- tant, please analyze my statement and respond with your generated questions. Ask about the object in image only. not involve the caption in your questions. Ask as few questions as possible.
One In-context Example
Prompt: Phrase: "a book with the word 'Fix' it"
Reasoning: The phrase "a book with the word 'Fix' on it" describes the target category of 'book' and provides extra information that word "Fix" on the book. However, caption does not mention any specific word on the book, so extra information is needed to determine if the phrase is correct. Conclusion: The phrase is uncertain. The caption does not mention any specific word on the book, so it is unclear if the phrase is correct not.
Response:
Questions:
Is there any specific word the book? If so, what the word the book?
2. Generate target object for action during uncertain state
Task Description
You are an excellent text analyst. I have a phrase that describes an object an image. have already analyzed this phrase based on a caption. And have reached the conclusion that this phrase is uncertain.
First, want you help generate questions asking about the missing information in the caption. I will provide you with my statement the reason why the phrase is uncertain. an assistant, please analyze my statement and respond with your generated questions. Ask about the object the image only. Do not involve the caption in your questions. Ask as few questions as possible.
Secondly, based on the questions you have generated, I want you to give me target area where the missing information may most likely be found. Choose one receptive field from the following 3 options: 1) the object itself, 2)the object and surrounding areas, 3)the whole image. Here are some hints for you. If the missing information is about some attribute of the object, such as colors, materials, the target area is likely the object itself 1). If the missing information is something interacting with the object, the target area is likely the object and is
the surrounding area 2). the missing information is about another object, the target likely the whole image 3). Give me number the chnices, and only chnose one target area per task.
One In-context Example
Prompt: Phrase: 'bread loaf tan bowl'
Reasoning: The phrase deseribes large tan colored bowl with a bread loaf sitting However, the caption docs not mention the color or the presence food the bowl. The size of the bowl is also not mentioned.
Conclusion: The phrase is uncertain. The color of the bowl, the presence of food and the size of the bowl not mentioned in the caption. Questions: What the color of the bowl? 2. Is there any food in the bowl? If so, what type of food in the bowl? 3. What is the size of the bowl? Is small or large? Response: Target area: 1)ehe object itself.
Task description and an in-context example for response generation of fine-tuning data. There five in-context examples used in each text prompt. We show one example here for simplicity. We omit the Python code together with text prompts, which is similar to Tab. 9 and Tab. 10.
Table 11:
28

---

Published as a conference paper at ICLR 2025
G.5 VISUAL AND LANGUAGE PROMPTS FOR VLM
Language prompt
Here prior knowledge about this image. The object in the image is (object cate- gory). There is a possible description of this image, it may not be precise enough: { current expression). Answer the following questions. (Questions generated by Real-LOD).
Visual prompt
Visual prompts of three actions for VLM:
object itself
object and surrounding areas
whole image
Object Crop
Extended Object Crop
Object Highlight
Table 12: Visual and language prompts for VI.M tools. We show examples of the three image editing actions for VLM. Visual and language prompts are generated case by case via our Real-LOD.
H ANALYSIS ON COMPUTATION COST
In this section, we present statistical results analyze the computation cost of our Real-LOD, which are referred to by Sec 4.3. Tab. 13 is the average number of calls for each step and the time cost during each call for one expression, and Tab. 18 presents the distribution of iteration number Real-LODS Note that the max iteration number here is set to 10 for investigation.
Table 13:
Average number of calls for each step and time cost during each call for one expression.
Iteration Num
2
3
4
5
6
10
0.5
Step
Avg num of calls
Time cost of each call
Planning 3.09
0.265s
LLM-tool
0.65
0.131s
VLM-tool
0.43
0.159s
0.0
Reflection
2.08
0.291s
Figure 18: Distribution of iteration number Real-LOD. Note that max iteration number here is 10 for investigation.
29

---

Published as a conference paper at ICLR 2025
1 TECHNICAL DETAILS
1.1 LICENSES OF DATASETS, CODES AND MODELS
In Tab. 14, we present the Licenses and URLs of datasets, codes and models used in our paper.
Table 14: The License and URL of datasets, codes and models utilized in this paper.
Assert
Type
License
0365 (Shao 2019)
Dataset
Creative Commons Attribution 4,0 License.
OpenIinage (Kuznetsova al., 2020)
LVIS (Gupta 2019)
Attribution
OmaiLabel (Schulter et 2023)
Datasct
MIT License.
DOD (Xic al., 2023)
Dataset
Creative Commons Attribution 4,0 License.
OVDEval (Yio et al,, 2024)
Dataset
Apache-2.0 2.0 Ticense.
Refencolg/+ (Mao et al,, 2016)
Dataset
Apache-2.0 Ticense.
MMDetection (Chen et al., 2019)
Code
Apache-2.0 2.0 license.
CharGLM (Zong et al., 2023)
Code
Apache-2.0 Tioense.
LLaVA (Liu et al., 2023a)
Model
Apacho-2.0 lieensc.
Vicuna (Zheng et al., 2023)
Model
Llama Community License Agreement.
I.2 TRAINING DETAILS OF REAL-MODEL
The implementation of Real-Model is based on the MMDetection (Chen al., 2019) framework and PyTorch (Paszke et al., 2019). The input size of all the experiments is 1333 x 800, and the batch size is 4 per GPU. In the ablation study, there is only single machine with 8 NVIDIA V100 GPUs for training to guarantee impartiality. For the final result, we train on 16 NVIDIA V100 GPUs for better performance. During training, we employ the AdamW optimizer (Kingma & Ba, 2015) with momentum of 0.9 and weight decay of 0.05. The learning rate setting includes a 1000-iteration warm-up with start factor of 0.1 and multi-step schedule with an initial value of 4 x 10 for 10 epochs. To be specific, the weights used for model initialization are referenced from the office repository of mm-GDINO (Zhao al., 2024).
1.3 EVALUATION DETAILS
In Tab. 15, we provide the detailed training data information of other LOD methods, which compare within the OmniLabel and DOD benchmark.
Table 15: A detailed list of training data information for other LOD methods.
method
Source
Hmng
MDEIR (Kamathh et al., 2021)
ENB3
COCO, RefCig/+. YG, GQA, Flickr3Uk
0.3M
GLIP (1. et al., 2022)
Swin-L
0365, COCO, OI, YG, ImageNet, GoldGi CCSM, CC12M, SBU
17.5M
FIBER (Dou et al., 2022)
Swin-B
cOCO, CC3M, SBU, VG
4M
OWL-VZ (Minderer el al., 2023)
VIT-L
WebLI
10B
UniNext (Yan al., 2023)
VIT-H
0365, RefCig/+
0.7M
UniNext (Yan et al., 2023)
ResNet-50
0365, RefCig/+
0.7M
GDINO (T.in al., 2024h)
Swin-B
Mossoo..GaldG.CCAM..o RefC/g/+
5.8M
OFA-DOD (Xie al, 2023)
ResNet-101
CC12M, CC3M, SBU, COCO, VG, RefC/g/
16M
(Shen et al., 2024)
COCO. LV1S, 01,
200
(Shen el al., 2024)
VIT-L
COCO, LVIS, 0365, 01, VG, RefC/g/+
2.6M
mm-GDINO (Zhao et al., 2024)
Swin-T
0365, GoldG, GRIT, V3Dst
2.8M
mm- GDINO (Zhao et al., 2024)
Swin
GoldG, 0365, COCO, OI, RefGigh, V3Dei, LVIS, GRIT
12M
1.4 MODEL DETAILS
In Fig. 19, we present more architectural details of the Real-Model, which is based on the mm- GDINO (Zhao et al., 2024). As shown in Fig. 19, the text encoder and image encoder first extract the text and image features, respectively. The bidirectional feature enhancement module is then used to integrate the text and image features through cross-modality cross-attention. After integration,
30

---

Published as a conference paper at ICLR 2025
cross modsl-ty queries extracted from the image features with the language -uidedguuddddddd query selection module and then subsequently input into the decoder with a further cross- mrosa-modality fusion. The final output queries then utilized for contractive loss and localization loss. More details can be found in (Yang et al., 2023b; Zhao et al., 2024).
Cross- Modality Decoder
Lunguage guided Query Selection
Text Foatues
Nodalit
Bidirectoonal Festure Entarcemeent
Fand
Pased
Irage Fustuena
Cooss Nodaliy Dacsdar
Cross
In-izaee
Image Uection
Image Tist ttention
Localtation Loss
Self-Atantion
Dedemrazle
Sef-Aitaaiicn Jon
Text Encoder
Image Encoder
LontoatheeLoo
"Talarna people cack
Figure 19: More architectural details of the Real-Model. The Real-Model is built upon the mm- GDINO (Zhao al., 2024) and trained with Real-Data re-aligned with Real-LOD. The right part is the overall framework before the cross-modality decoder. The left part is the cross-modality decoder and loss calculation.
DISCUSSION
J.1 LESSONS OF AGENTIC WORKFLOW DESIGNING
States/actions of agentic workflow. The design of agentic workflow states/actions should follow the neural-symbolic spirit, guided by task-specific requirements analysis. In Real-LOD, we establish core states/actions through systematic analysis of hallucinations brought by VLMMs produce expres- sions. These carefully designed actions, including re- perceive ROI areas and customized questions, are controlled by our agent for VLM adaptively deal with different types of hallucinations.
Agent selection suggestion. During the implementation of our Real-LOD, we empirically found that LLM reasons are more accurate in pure language form, especially when facing long text prompts and cases requiring strong logic. In contrast, the when perceiving language and visual data simultaneously. despite their inherent strengths in visual content comprehension. A showcase of this ineffectiveness is the model hallucination that produces incorrect language expressions, as discussed the second paragraph of Sec. 3.3. This ineffectiveness due to the naturally inadequate alignment between image and language data used for VLM training (e.g., it is difficult to fully represent what an image conveys in 1-2 sentences). Therefore, we use VLM convert visual content deseriptive text and choose LLM with pure language form the agent to leverage its strong logical reasoning ability for accurate and robust planning within our workflow.
VLM can not effectively reason
Training data to fine-tune agent. Regarding the training data to fine-tune the agent, it does not necessarily include all the recurrent steps. This is because the planning process of the agent in each
31

---

Published as a conference paper at ICLR 2025
recurrent step is the same, and only need to focus on the behavior of the agent in recurrent step.
J.2 BROADER IMPACT
Agent and baase object detection have shown significant applications in various real- world scenarios, particularly intelligent roboties and autonomous driving. Our proposed method exhibits potential for these two research areas, offering valuable insights to the community. Our method focuses on correcting language expressions for the LOD dataset without specific application goal. Hence, it does not directly involve societal issues.
J.3 LIMITATION
Real-LOP employs the VLM to perceive the content of given target in various scenes, providing external information to help the linguistic descriptions correcting process for reducing the model hallucination. Although workflows and visual tools, there are still some unmanageable hard cases limited by the original performance of VLMs. As shown in Fig 20, there are two main kinds of data refinement error caused by the eтror perception of VLM:
method strongly stimulates the potential of VLMs by introducing agentic
Typical visual hard cases. Object-detectio datasets include low-light or low-quality scenes and extremely small or teffoonlttt-rrcogmiie objects. It could be difficult for VLM to generate appropriate expressions for these targets.
2. Expression describes a foreground object instead of the target background object. VLM may ignore the target object the background when an occlusion exists. In order to ensure high quality at the bbx level, the reflection module regards these expressions wrong.
When conducting large-scale data refinement, we set the maximum iteration to With several extra iterations, some failure cases are likely to be solved. Since we already have large amount of high-quality data for downstream training with a task-solved rate of 75%, we choose not to increase this parameter for the sake of efficiency. This indicates the requirement of developing more powerful and robust VLMs to handle complicated situations more efficiently.
blurry object
Object category: desk
Latest expression: in red jacket holding coffee
Case 2. Object low-light area
Object category:
red cabbage
expression:
purple on banana
Case 3, Expression describes foreground object
Object category: car
expression: white glass on building
Figure 20: Visualization of some failure cases.
32

---

Published as a conference paper at ICLR 2025
K MORE ANALYTICAL EXPERIMENT
In this section, we provide additional analytical experiments about our Real-LOD to further demon- strate its effectiveness. Tab. 16 reports the accuracy choosing the corresponding statelaction on the validation set of fine-tuning data (Ik samples). It indicates that Real-Agent can accurately reason the state/action, especially for the "Wrong" and "Uncertain" states. In addition, the relatively lower accuracy of the "Correct" state indicates that our agent is strict with the quality of expression to prevent the hallucinations as much as possible. According to Tab. 17, we provide more detailed ablation study of Real-L.OD. The experimental setup can be found in Sec. 3.3. The "w/o Planning" is the same as the random selection schema in Sec. 3.3. "w/o Cyclic Workflow" indicates the workflow with only one cycle. The results intuitively illustrate the importance of each component to agentic workflow.
Table 17: More detailed ablation study of our agentic workflow.
Table 16: Accuracy in choosing the correspond- ing state/action on the validation set of fine- tuning data.
State/Action
Success Rate
Real-LOD
74.7%
Statc/Action
Accuarcy
w/o Planning
35.6%
Correct/Stop
93.1%
w/o Action 2
18.0%
Wrong/LLMM
99.4%
w/o Action 3
53.0%
Uncertain/Object Crop
99.6%
w/o Action 4
51.8%
Uncertain/Extended Object Crop
95.8%
w/o Action 5
57.4%
Uncertain/Highlight
90.0%
w/o Cyclic Workflow
60.7%
33


Reviews:
[{"review_id": "jecCf24eF9", "reviewer": "Reviewer_J3CB", "summary": "The paper presents an agentic approach to improving text-image data for language-based object detection. The agent has access to tools (LLMs and VLMs) and performs planning and reflection steps. The proposed agentic pipeline is used to re-align text-image data and train a model that significantly outperforms prior works.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed data filtering and processing mechanism improves the data, and leads to better models\n2. The method is evaluated on several datasets and the improvements are consistent\n3. The method relies on open-source LLMs and VLMs, and the prompts used have been provided, so the work could be reproduced", "weaknesses": "1. The agent has access to a variety of actions to choose from (from the states it can be) and there is some intuition that the 6 mis-alignment reasons (category, attribute, etc) will each be beneficial for some of these failure modes. It would be good to see if the used actions and tools actually reflects that intuition with some analysis on which actions were chosen when different mistakes were made\n2. Further to the previous point, the agentic pipeline as is now has quite a lot of components. It might be very beneficial if any of them can be removed without significant loss in performance.\n3. The paper several times mentions the model trained is a \"prevalent\" model -- can this be backed up with other works that use the same model? Comparing to these more explicitly would make the contribution of this work stronger\n4. The training of the agent is not properly explained. Some things that should be there are 1) how many samples were used to train the agent? It says that these were manually created by the authors using the outputs of the various tools. Did this include all the recurrent steps?\n\nMinor: Section 4.1 introduces forms A,B,C which are then not references in the table and is a bit confusing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdMacTs9wV", "reviewer": "Reviewer_LXzJ", "summary": "The authors of this paper propose a method for improving the quality of text descriptions used for training language-based object detection (LOD) models. As done in prior work, in the first step, they use a vision-language model with language generation capabilities (LLava in their case) to generate text descriptions for objects.\n\nTheir main contribution is a multi-step iterative method for improving the generated descriptions for each object. The authors do a manual analysis and define six different problems that can occur for each description.  They group these problems into three categories (referred to as states) and pre-define an action for each problem (e.g., if the attributes in the description are not correct, crop the image and generate a new description).\n\nIn each iteration of their method, a fine-tuned unimodal LLM (i.e., their agent) inspects the description and decides if the description is good enough or needs further modification. If further modification is needed, the agent selects one of the three predefined states (i.e., types of problems) and suggests taking the corresponding action to refine the description or generate a new one.\nOnce the action is complete, the authors use another LLM to reflect on the output of the executed action. Although it is not entirely clear in the paper how it is accomplished, the results of the reflection process are integrated into the next iteration to further improve the newly generated or updated object description. The authors repeat the iterations for each description for a certain number of steps or terminate the loop if the agent decides the description is good enough and no further modification is needed.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The idea that several models work together to collectively improve the quality of the training data is very interesting and could be potentially useful for other applications as well.\n* The authors follow a reasonable design process to develop their method. For example, they define the set of states and corresponding actions based on the analysis of the existing data.\n* The authors ask the right questions when evaluating their method. For example, they attempt to study the importance of correctly choosing the state/actions for each description. They also try to isolate the impact of higher-quality data from the original data that they are using (paragraph 2 of Section 4.1).", "weaknesses": "I have two major concerns about this work. Although the idea is interesting, and the authors have come up with the correct questions, the quality of the writing and experimental setup can be improved significanlty.\n\n**Writing:**\n\nThe paper is not well written and is hard to understand. It takes a couple of passes to understand the arguments. There are also a lot of ambiguities that need further clarification. Here are some examples:\n* Second paragraph of section 3.1: Since the authors are generating the initial training data for LOD (that they will later re-align with Real-LOD) and not using the data that is out there from previous work, the setup and how the data generation is accomplished should be clearly explained. The authors just mentioned the name of the model they use.\n  * For the initial expression generation with LLava, the authors should clearly state the prompts used, formatting structure, number of expressions generated, etc.\n  * The authors also mention, “vicuna is used to diversify the generated descriptions”. What does “diversify” mean in this context? How exactly is it done, given the generated data?\n* Since the authors are fine-tuning “ChatGLM-6B” as their agent (which is responsible for choosing the state and action to be executed), it seems like the agent does not have access to the image, and it reasons only based on the textual prompts. However, the figures and texts give the impression that the image is also used as input to the agent to choose the state/action. This should be clearly stated in the text that the agent does not have access to the image.\n  * Also related, it seems like the agent does not use the bounding box location either. This should also be clarified.\n* Line 260: Vague statement: “When using these models, Real-LOD agent adaptively modulates visual content and text prompts based on its reasoning thoughts”. It is not clear which aspect of the agent this statement is referring to.\n* How is the output of the reflector incorporated in the next iteration of Real-LOD? If, in the next iteration, the agent decides that no further modification is needed, is the reflector output not used?\n* The authors should discuss the expected behavior of the planner in more detail in the main text. For example, does it format the questions (i.e., prompts) that are used with other tools (i.e., LLM and VLM) in free form, or is it expected to follow specific guidelines/templates for generating the prompts?\n* Section 3.3 needs more organization. The authors should clearly separate the analysis process, the statistics of the created data for training real-LOD-agent, and the analytical experiment to test the quality of the generated data.\n* Line 340: SigLip filtering here seems to be different from the one explained in Line 197. It is not clear what is being filtered here and how the filtering is accomplished.\n* Line 341: “Each input data contains the object category label, raw expression, and reasoning from LLM.” what reasoning from LLM? How is it generated? My impression is that this is the input to train the agent. So, we do not have the agent yet to get the reasoning as explained in other parts of the paper. So, this step should be further clarified.\n* Line 341-342: “Then, we manually set the state/action and collect responses as the outputs.” Is the manual annotation done for all 15k input samples? If yes, it should be reported who has done the labeling (i.e., authors themselves or mechanical Turk, etc.), and the details of the instructions given to labelers should be provided.\n* Line 350: “We randomly select hundreds of filtered expressions and manually check each for a detailed observation”. This is an important detail of the analysis. “Hundreds” should be quantified with exact numbers.\n* Line 379: “we find that the success rate is 74.7% v.s. 35.6%.” these numbers and the experimental setup used to achieve these are not clear.\n* it is mentioned in the abstract that the method results in \"50% improvement\". Does it mean that the proposed method improved the performance by 50 percentage points? that does not seem to be the case in the experiments.\n* It is also more informative if the authors report the average improvement rather the maximum.\n\n**Minor suggestions:**\n\n* It might be easier to understand if section 3.3 (DATA ANALYSIS OF LANGUAGE AND VISUAL OBJECTS) comes before section 3.2 (AGENTIC WORKFLOWS FOR LANGUAGE EXPRESSION RE-ALIGNMENT)\n* Similarly, it might be helpful to have section 4.2 ( COMPARISONS WITH STATE-OF-THE-ART LOD METHODS) before section 4.1 (ABLATION STUDY)\n\n**Experiments**\n\n* The authors claim significant boosts in performance but the experiments are very limited (only three datasets). More extensive experiments are needed to show the merits of the proposed method.\n* For section 3.3, although authors have done the analysis that shows choosing states and actions by their agent achieves better performance than random selection of states/actions. It is important to also report the accuracy of the agent in choosing the corresponding state/action for each input sample. \n* Correct me if I am wrong, but it seems like the agent is only trained on data built from Objects365. It would be interesting to evaluate the model on Objects365 dataset exclusively to analyze how transferable the agent is to other datasets. Although object365 is included in Omnilabel but exclusive numbers for Objects365 help better understand the impact of distribution shift from training to inference.\n\nI also have some questions about the design choices: \n* Why use an LLM as the agent when the input contains both image and text? Isn’t a language-generative VLM a better choice?\n* Similarly, in state2/action2, why use an LLM to write a new object description based on the existing caption rather than using a VLM to write the description based on both the image and the caption?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1pQg1wEp1V", "reviewer": "Reviewer_e1U7", "summary": "The paper aims at the language-based object detection task that detect visual objects based on language queries. Since previous vision-language models (VLMs) may face the hallucination challenge, authors present an agentic workflow based on an LLM to re-align the language query and visual objects. The workflow contains three main steps: planning, tool use, and reflection. Some evaluation results on some benchmarks show the performance comparison.", "questions": "I hope that authors can address our questions and concerns in Weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Figure and tables are clear. Many figures and tables are used to present the language expressions and evaluation results.\n\nA state-of-the-art method mm-GDINO (Zhao et al., 2024) is used for performance comparison.\n\nIn Figure 5, the detail of how the presented Real-LOD method re-aligns one raw language query to given image is clear. Figure 5 can help reader understand the model implementation.\n\nAblation study on the Omnilabel dataset is conducted for model anaysis.", "weaknesses": "Authors use an agentic workflow controlled by an LLM to reduce VLM hallucinations. However, there is still hallucinations in the LLM. In fact, hallucinations are not effectively reduced.\n\nThe general LOD framework in Figure 3 is very common. Thus, I have concern about the technological novelty of the presented method.\n\nIn Section 3.2, there are \"Right\" state and \"Wrong\". How to judge whether he is truly right or wrong, rather than the hallucinations from the LLM?\n\nThe works seems to directly use the LLM and VLM. There is no real innovation for both LLM and VLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vUF1PKBHsA", "reviewer": "Reviewer_M1hG", "summary": "This paper begins by explaining that directly using vision-language models (VLMs) for re-captioning target regions can lead to hallucinations, highlighting that VLMs alone may not be suitable for training data optimization. To address this, the authors propose an Agent Workflow (Real-LOD) to improve training sets for open-set object detection, which consists of Planning, Tool Usage, and Reflection stages. This workflow combines the reasoning ability of LLM (ChatGLM-6B) and the perception ability of VLM (LLaVAv1.6-34B) to effectively correct inaccurate object descriptions within the training dataset.  The authors then re-train an open-set object detector based on the Swin-B backbone using the re-aligned dataset. This detector performs well across multiple open-set detection benchmarks, including the new challenging OmniLabel benchmark. Extensive ablation studies on these benchmarks further validate the effectiveness of each step in the data refinement process.", "questions": "Above all this is a good idea. Refining training data to gain more high-quality data for training the model is highly practical. However, I have a question about whether the region-controllable captioner is better than a complex agent workflow. The former is more cheap and easy to use.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Innovative Data Optimization Method**: The proposed Agent Workflow offers an innovative approach to training data refinement by employing multi-step processes—namely, Planning, Tool Usage, and Reflection—to mitigate hallucinations generated by Vision-Language Models (VLMs) and enhance data quality.\n\n2. **Demonstrated Effectiveness in Experiments**: The retrained open-set object detector achieves strong performance across challenging benchmarks, including OminLabel, underscoring the practical efficacy of this approach.\n\n3. **Relatively Comprehensive Ablation Studies**: The paper includes extensive ablation studies that validate the necessity and contribution of each step within the Agent Workflow, providing robust empirical support and reinforcing the reliability of the study.\n\n4. **Clear and Informative Visualizations**: The figures are well-designed, effectively illustrating the issues with VLMs and detailing the steps within the Agent Workflow, enhancing reader understanding of the methodology.", "weaknesses": "1. **Choice of VLM**: The authors use the commonly utilized large-parameter version of VLM (LLaVAv1.6-34B). However, several newer Controllable VLMs / MLLM / Captioner allow interaction through a variety of visual prompts (such as Points, Boxes, and Masks) to obtain more focused descriptions for target regions. Directly using these controllable VLMs may have fewer hallucinations on region caption task and may be better than LLaVAv1.6-34B with region cropping. Examples Region Controllable VLMs include: AlphaCLIP (CVPR 2024) [1], Osprey (CVPR 2024) [2], Ferret (ICLR 2024) [3], OMG-LLaVA (NeurIPS 2024) [4] and so on.\n2. **Lacking ablation experiments on the influence of the fine-tuning of ChatGLM**. What is the role of this step? Can fine-tuning enhance the effect of all three stages of the Agent. \n3. **Error analysis about the workflow of agent**. Lack of analysis of Agent workflow failure cases. To which part of the Agent should the cause of the data refinement error be attributed: inference? Tool call? Error perception of the tool itself? Error of reflection?\n4. **Computation cost is two large**, 48 V100 GPUs, Normal LABS can't afford it. A lite version should be studied in the future.\n5. **Minors**:\n    1. Some expressions are more obscure, such as **Re-Aligning Language to Visual Objects**, so it is better to directly use the refine object description in open-v object detection data, which is more clearly and easy to understand.\n    2. The use of symbols and naming are not easy to distinguish. For example, the base name $\\text{Real LOD}$ used three times, $\\text{Real LOD}^{data}, \\text{Real LOD}^{model}, \\text{Real LOD}^{agent}$. This discourages the reader from quickly distinguishing between these symbols and leads to confusion.\n\n[1] Sun Z, Fang Y, Wu T, et al. Alpha-clip: A clip model focusing on wherever you want[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13019-13029.\n\n[2] Yuan Y, Li W, Liu J, et al. Osprey: Pixel understanding with visual instruction tuning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28202-28211.\n\n[3] You H, Zhang H, Gan Z, et al. Ferret: Refer and ground anything anywhere at any granularity[J]. arXiv preprint arXiv:2310.07704, 2023. \n\n[4] Zhang T, Li X, Fei H, et al. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding[J]. arXiv preprint arXiv:2406.19389, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)