LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
Authors: Amaia Cardiel
Eloi Zablocki
Elias Ramzi
Oriane Siméoni
Matthieu Cord
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Vision-Language Models
Black-Box Adaptation
Referring Expression Comprehension
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Fine-tuning an LLM to reason on VLM outputs offers a way to adapt said VLM for the Referring Expression Comprehension task in a black-box, model agnostic and efficient manner.

Abstract: Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires ‘white-box’ access to the model’s architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for ‘black-box’ adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box to match the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile and works with any VLM, transfers to new VLMs, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. The code will be open-sourced.

Published as a conference paper at ICLR 2025
LLM-WRAPPER: BLACK-BOX SEMANTIC-AWARE ADAP- TATION OF VISION-LANGAAGE MODELS FOR REFERRING EXPRESSION COMPREHENSION
Amaia Cardiel Éloi Zablocki', Elias Ramzi', Oriane Siméoni',
Matthieu Cord 2.3
Valeo.ai APTIKAL, LIG, Université Grenoble Alpes
Sorbonne Université
(amaia. cardiel, eloi. zablocki, elias elias. ramzi, matthieu. wtw.:/eww...o///ee.....oom
ABSTRACT
Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine- tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires 'white-box' access to the model's architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose ALM-wrappers for the REC task using Large Language Models (LLMs). LIM-wrappez capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it versatile as it works with any VLM, it transfers new VLMs and datasets, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrappes on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM- wrapper is not meant to directly compete with standard white-box fine-tuning, it offers practical and effective alternative for black-box VLM adaptation. The code and the checkpoints are available at https: / /gi www.tcoooce.oe.ttcooooooooooetot con/val 1 eoaa i
method for 'black-box' adaptation of VLMs
INTRODUCTION
Vision Language Models (VLMs), class of foundation models (Bommasani et al., 2021), trained on large- scale and diverse tasks and datasets, have shown remarkable abilities to solve various open-vocabulary tasks, as image captioning (Xiao et al., 2024; Li et al., 2023), visual question answering (Alayrac al., 2022; Liu al., 2023; Li et al., 2023), text-image retrieval (Radford et al., 2021; Zhai al., 2023; Li et al., 2023), object detection (Liu et al., 2024c; Xiao et al., 2024; Cheng et al., 2024), or semantic segmentation (Ding et al., 2023; Xiao et al., 2024). Recent VLMs show promising zeoo generalization abilities new tasks and data domains (Wei et al., 2022; Alayrac et al., 2022). However, there is still a significant performance gap between zero-shot VLMs and those that have been specifically trained adapted for particular task and data domain. This work focuses on the challenging voca-lacy detection task of Referring Expression Comprehension (REC) (Mao et al., 2016), which involves localizing object in an image based on complex textual query, requiring both spatial and semantic reasoning, While zero-shot VLMs can typically detect most objects mentioned in the query with reasonably accurate bounding boxes and labels, they struggle to identify only the described object. Moreover, VLMs often have difficulty understanding complex descriptions that involve relations between objects, attributes, or negations (Xie al., 2023: Yao al., 2024).
To improve performance, VLMs are typically fine-tuned on the specific REC task and corresponding datasets. This fine-tuning is usually done in a 'white-hox' manner, with full access to the model's architecture and weights for back-propagation. However, this process requires expertise to design fine-tuning objectives and optimize hyper-parameters, specific to each VLM and downstream task. Moreover, white-box fine-tuning is not always feasible. Some models are closed-source, either because they are proprietary and released behind APIs (Ren et al., 2024a), or because they are trained on private data, making their weights and gradients inaccessible. While companies may provide APIs for adapting proprietary models, e.g., (OpenAI, 2024). these solutions are limited to predefined scopes and require sharing data with externa private companies, raising legal and privacy concerns.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison between white-box fine-tuning and LiM-wrapper on the REC task.
Classic white-box fine-tuning
LIM-wrapper
Model access
Needs complete access (loss, archi- tecture, weights, gradients)
Only needs forward call of frozen black-box VLM
Fine-tuning knowledge
Specific for each VLM
Agnostic to the choice of VL.M and
LLM, and parameler-efficient
Specificity adaptation Fine-tuning generalization
None
Leverages LLM semantic reasoning
Limited: no ensembling, transfer to new VLMs or datasets
Flexible: supports ensembling and transfers across VLMs and datasets
Expected performances
Best results
Good results
To address these challenges, we explore 'black-box' adaptation of VLMs, where only forward calls to the model are possible. We propose LIM-wrapper, a new method for the black-box adaptation of VLMs for the REC task, by using an LLM to reason on the VLM's outputs. This approach builds on the recent development of Large Language Models (ILMs)) (Touvron et al., 2023; Jiang et al., 2024; Mesnard et al., 2024), which have shown interesting reasoning capabillites. The underlying idea is that a zero-shot black-box VLM can generate high-quality labeled bounding boxes, and that L.M-wrrpper can then leverage the semantic and spatial reasoning abilities of the LLM (Lian et al., 2024a;b) to 'reason' on such outputs, further enhanced with a light fine-tuning, As illustrated in Figure I, our method involves translating VLM's outputs into natural language prompt and feeding it to the LLM. The LLM is then tasked with identifying the box that best matches the referring expression from the given candidates.
LLM hiM-wrapper offers several advantages, summarized in Table 1. First, since the adaptation is done in a black-box manner, it removes the need for back-propagation through the VLM and allows for the adaptation of closed-source models, without requiring access to the model's architecture, weights, or gradients. This makes black-box fine-tuning versatile and easy to use, as it can be applied to any model without requiring specific model knowledge assumptions about the model's architecture. Additionally, as the adaptation is delegated to the LLM via text, wrapprr retains the pre-trained knowledge of the original VLM. We find that LIM-wrapper transfers well to other VLMs, generalizes model updates, and can adapt effectively to new datasets without additional fine-tuning. Finally, LLM-wrappes enables the adaptation of an ensemble of VLMs, leveraging the flexibility of text-based adaptation to handle varying numbers of bounding boxes, thereby combining strengths from multiple models.
We experiment with LLM-wrapper on three classic REC datasets - RefCOCO, RefCOCO+ (Kazemzadeh et al., 2014), RefCOCOg (Mao et al., 2016) - and on Talk2Car (Deruyttere et al., 2019), us- ing two notably different VLMs - Grounding Additionaly, we evaluate LIM-wrapper on the recent and challenging HC-RefLoCo (Wei et al., 2024) benchmark. We also experiment with two LLMs for the adaptation: Mixtral 8x7B (Jiang et al., 2024) and Llama 3 8B (Dubey et al., 2024). While LLM-wrapper is not meant to outperform standard white-box fine-tuning, we show that I.I.-wrappes significantly enhances the VLM's performances, across all combinations of VLMss, LLMs and datasets, thus demonstrating the versatility of our method. Notably, on RefCOCOg and Talk2Car, our most challenging benchmarks with respect to semantic understanding, HLM-wrapper improves the results of zero-shot VLMs with gains ranging from +8.7 up +18.3 P@ I.
(Liu et al., 2024c) and Florence-2 (Xiao et al., 2024).
2 RELATED WORK
Referring Expression Comprehension (REC) and Vision Language Models (VLMs). Referring Expression Comprehension (REC) (Kazemzadeh et al., 2014: Mao et al., 2016; Qiao et al., 2021) the task of identifying objects in an image based on referring expressions. Typically, REC involves selecting the best region from a set of region proposals extracted from the image, guided by referring expression. This task is challenging because the referring expression can range from a short phrase (Kazemzadeh et al., 2014) to a long sentence that may require multi-step reasoning (Mao et al., 2016). REC is distinct from similar tasks such as visual grounding (Rohrbach et al., 2016), where multiple object regions described by multiple noun phrases must be localized in an image. also differs from object detection (Girshick, 2015), which uses predefined categories instead of natural language expressions. The recent surge of interest in VLMs, starting from CLIP (Radford et al., 2021) to the recent Florence-2 (Xiao et al., 2024) has significuntly improved performance on tasks requiring both vision and language. For REC, several approaches have
2

---

with Published as a conference paper at ICLR 2025
Prompt
"You nebifll assistant, capable understanding spetal Information
black-box VLM
plate
In an image, there are bowes:
In box 'Bowerr with asardinelts 603, 224].
bowl
In "plate with flower syny coardinalle 15, 212, 502, 5871 In box 'yollow bowl with ocordinates [1182, 211, 1882.
plate
plate with flower details
bowl
Which box is best matching The plate with flowor dedails next the yellow Ansswer just best box. No explanation. Arsoweer
Query: plate with details next to the yellow bowl'
bowl
LORA
Illustration of LIM wiaper. Our method adapts black-box VLM for the REC task. The VLM output is translated into natural language to prompt an LLM. The latter is tasked with identifying the box that best matches the query among the given candidates. The LLM must learn to identify the query's subject and to disambiguate the correct object from distractors (c.g., several plates with 'flower details").
Figure I:
used VLMs. For instance, Soounding-DINO (Liu et al., 2024c) relies on multiple stages of modality fusion to align visual and textual features, while Florence-2 (Xiao et al., 2024) is sequence-to-sequenec model trained on a huge collection of data. However, despite being trained on extensive amounts of image-text data (e.g.. 126 million images with 500 million to 3.6 billion annotations for Florence-2), these models' zero-shot performances on REC are sub-optimal compared to their performances when including REC data in their training set or when fine-tuned (see Section 4.2). Our approach, LIM-wrapper, addresses this clear limitation in black-box manner, without the need to re-train the VLM with task-specific data.
VLM adaptation. While retraining the entire VLM on a new task or data domain is computationally expensive (Liu et al., 2023; Wang et al., 2023; Chen al., 2023; Xiao et al., 2024), fine-tuning offers more efficient alternative. However, even fine-tuning can be costly. To address this, parameter-efficient fine-tuning, e.g., LoRa (Hu et al., 2022), DoRa (Liu al., 2024a), or VeRa (Kopiczko et al., 2024), and soft prompt (Li & Liang, 202.1) learning have been proposed. These methods avoid updating the model's pre-trained weights but require access to the model's architecture and gradients.
A few recent methods propose strategies to adapt VLMs when sible (Ouali et al., 2023; Yu et al., 2023; Liu et al., 2024b; Oh et al., 2023; Wan et al., 2024). Both Ouali et al. (2023) and Yu et al. (2023) focus CLIP-based methods and adapt the VLM by learning either feature projection or soft prompts. They however require access to the inner representations, which is generally not possible with APIs. Liu et al. (2024b) optimize the input prompt template, but also focuses CI.IP-based models. Instead, we target all VL.Ms that perform open-vocabulary detection, e.g., Grounding-DINNO and Florence-2. Oh et al. (2023) adapt a VLM in black-box manner by modifying the image imput. However, this method is designed to address visual domain gaps and is not directly applicable to adapt VLM to a new task. Moreover, it requires multiple API calls for each training image. Finally, Wan al. (2024) introduce an adaptation method for frozen VLMs, applicable to REC. For each candidate bounding box, they perform a VLM inference to compute the probability of the input text based on modified image where the box has been blacked out. The candidate that achieves the highest probability contrast with respect to the unmodified input image chosen. Though training-free free, this approach requires the access to the model's output probability distribution and performs an additional VLM inference for each candidate bounding box. In this work, we propose a strategy to adapt VLMs, and more specifically open-voue necnn-oabbulary detectors, to a new task, in complete black-box fashion. Our method does not require access to the model's intermediate representations, output distribution, or gradients, runs a single inference/API call for each (text-image) input pair, and enables adaptation to a new task (REC in our case) that the VLM was not originally trained for.
through the VLM is not fea-
3 LIM-WRAPPER BLACK-BOL -LX SEMANTIC-AWARD ADAPTATION OF VLMS.
We present LLM-wr wraper, a novel LLLM-base approach to adapt VLMs for the REC task. We detail the general idea in Section 3.1, the prompt construction in Section 3.2 and the L.LM fine-tuning in Section 3.3.
3

---

Published as a conference paper at ICLR 2025
3.1 GENERAL IDEA
Our method wraps the open- vocabulary detections of frozen black-box VLM with an L.MM that reasons over these outputs. An overview is presented in Figure 1. Given complex textual query, LLM-wIz wrapper leverages the fact that detection-oriented VLMs can typically localize well most nouns of the query, even if they struggle with the reasoning step required to precisely select the object of interest among several distractors. Therefore, LLM-wrapper delegates the reasoning task to an LLM, which has interesting abilities to handle difficult text queries, including attributes, negation, and relational or spatial descriptions of objects. As the gradients of the black-box VLM are not accessible, we propose to adapt its oulputs with the I.M.M It gives access the L.M gradients and we thus specialize the LLM for the task, with a simple and light fine-tuning, performed on the outputs of the VLM to learn to select the right box among them. Overall, LLM-wrase wr prapper only requires black-box access the VLM, whereas standard fine-tuning strategies need white-box access to perform paoo-propagation. This makes our approach more flexible and applicable in scenarios where the internal workings of the VLM are not accessible.
3.2 PROMPT CONSTRUCTION
The key idea of our method is to convert the VLM's outputs into natural language. To achieve this, we list all predicted outputs in the LLMM prompt, including their hox coordinates, labels, and, when applicable, prediction scores (displayed below in light gray). This allows the 'blind' LLMM, which only reads text, to understand the scene and reason about the image. The prompt then reminds the query and asks the LLM to select the best matching box. For instance, given the query (in green in Figure 1), and the associated outputs (e.g.. 'flower', 'plate with flower details', 'yellow bowl', etc.), we ask the LLM for the best matching box index, follows:
You are helpful AI assistant, capable of understanding spatial information. In an image, there are boxes:
" In box O: 'flower' with xyxy coordinates '[402, 181, 603, 224]' with score 0.92,
In box I: 'plate with flower details' with xyxy coordinates '[5, 212, 502, 587J with score 0.88,
* IIn box 2: 'yellow bowl' with xyxy coordinates '[1182, 211, 1582, 452]' with score 0.86, [...]
Which box is best matching 'The plate with flower details next to the yellow bowl', ?
Answer with just the index of the best box. No explanation.
Answer:
3.3 FINE-TUNING THE LLM
While zero-shot LLM can already reason the new task to extent, find that fine-tuning the LLM significantly improves performances on the task. Therefore, we fine-tune the LLM for prompt completion with cross-entropy loss, using the prompt described above. Specifically, the expected answer for the LLM is the index of the best box proposal, corresponding to the closest match to the known ground truth box. To build the training dataset, we use the REC training data, which consists of (image, query) pairs and ground truth boxes. The detection outputs (boxes, labels, scores), used to create the training prompts, are inferred using the VLM being adapted. We only keep the samples where at least one of the VLM box proposals has an Intersection over Union (IoU) with the ground truth box higher than 0.5, ensuring no noisy samples. To make LLM-wrapper robust to any shortcut learning based on the box order, we randomly permute the order of the box proposals in the prompt during training.
We leverage the extensive literature on L.M fine-tuning to specialize the LLM of L.M--aapper. Specifically, we use LoRA (Hu et al., 2022), which introduces additive updates to the model's ss activations, parameterized by low-rank modules. Doing so reduces the number of new parameters to learn while preserving the LLM's general knowledge. We also use flash attention (Dao et al., 2022) and 4-bit quantization (Dettmers et al., 2024), making ALM-wrapper trainable on a single 40GB-A100 GPU in less than 7 hours. This approach makes the training efficient in terms of compute and very simple to implement in practice. Overall, we find that LIM-wrapper is not very sensitive to the choice the few hyper-parameters introduced (see Section 4.4).
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
In this section we present the experimental validation of LIM-wrapper. We first state our protocol in Section 4.1. In Section 4.2, we show how LiM-waapper improves VLMs. We then conduct further analysis of ALM-wrapper benefits in Section 4.3. We finally conduct ablation studies showcasing wrapper''''s robustness in Section 4.4.
4.1 EXPERIMENTAL SETUP AND TECHNICAL DETAILS
4.1.1 REFERRING EXPRESSION COMPREHENSION (REC) TASK
Datasets. We evaluate LLM-wrapper
the REC task. In REC, given input pair (image, quezy), a model is expected to predict a single bounding box around the object described the query, as illustrated in Figure 1. We use three standard datasets for REC: RefCOCO, RefCOCO+ (Kazemzadeh al., 2014), and RefCOCOg (Mao et al., 2016) which is more challenging as it contains longer descriptions (8.3 words in average). We also REC dataset. Additionally, we evaluate Liapwr on HC-ReiLoCo (Wei et al., 2024) for zero-shot dataset transfer experiments. All of these datasets contain multiple distractor objects for the given text query. Dataset statistics are given in Table 2.
the driving, human-centric, "aak22ar (Deruyttere et al., 2019)
Table 2: Dataset statistics.
Metric. We measure performance with the standard precision@) @ 1 (P@) 1) metric, de- scribed in (Qiao et al., 2021). A true posi- tive is defined when the predicted box has an Intersection- Snpeection-vve---e Unir-Union (IoU) greater than 0.5 with the ground truth box. The metric is averaged over the evaluation set.
Split
Size
words
Dataset
used
train
val
test
query
RefCOCO
unc 120,624
10,834
10,752
3.5
RefCOCO+ RefCOCOg
unc
120,191
10,758
10,615
3.5
umd
80,512
4,896
9,602
8.3
Talk2Car
8,348
1,163
2,447
11.0
HC-RefLoCo
13,360
31,378
84.6
4.1.2 THE VLMS
We evaluate the impact of LIM-wrapper on two different VLMs. In all cases, we use the official model checkpoints and unless specified otherwise, we use the model versions that are not fine-tuned for the REC task, meaning that the models have not been exposed to any of the RefCOCO/+/g and Talk2Car datasets. Using the model setup described below, we feed from 45 boxes to L.IM-wrapper per prompt.
Brounding-DINO GD) (Liu et al., 2024c) aligns visual queries with text through stages of modality fusion and contrastive learning. Initially designed for apen-uocabulary grounding, the model produces 900 bounding boxes, achieving high recall but under-performing precision the REC task. For REC, we use GD by selecting the bounding box with the highest score relative to any token in the query. However, this method sometimes selects boxes based query parts unrelated to the main object, such as other nouns in the sentence. To address this, we introduce Grounding Crounding-DINNO-RR DINO-REC (GDrec), a more
targeted approach for REC. GDrec identifies the query's 's subject, defined the first noun group detected by SpaCy's dependency parser (Honnibal et al., 2020), and selects the bounding box that scores hest against it. GDrec outperforms on Re/COCO/+/g datasets, and particularly on RefCOCOg, where the noun identification is more challenging, with e.g. +9.1 P@ (test) for zero-shot models as shown in Table 3. To ensure a rather short prompt, we limit the number of box proposals by setting the box confidence score threshold to 0.15 for GDrec and to 0.2 for GD. GD/ / GDrec, not fine-tuned, use a Swin'T (T) backbone while the fine-tuned versions are based on SwinB (B). as they are the only publicly available models. We also run experiments a recent detector behind API (online at: Ground ng-DDND-
subset of RefCOCOg using Grounding- 1.5 (Ren et al., 2024a) (GD-1.5 .5).
APTT)., which provides 300 free API calls. GD-1.5 extends GD with larger backbone, ViT-L (Fang et al., 2024), and training dataset.
Florence-2 (Flo2) (Xiao al., 2024) is a sequence-to-seqqeeee multi-task model. We use the Florence-2 Large version from Hugging Face. It is composed of a DaViT vision encoder (Ding et al., 2022) and multi-modal decoder. Flo2 can be prompted to deal with different grounding tasks. To evaluate Flc2 on the REC task without adaptation, we use the box predicted for the 'open vocabulary detection' task. For LLM-wrapper's 's adaptation of Flo2, we keep and concatenate the boxes predicted for both the 'open vocabulary detection' and the "phrase grounding" task modes.
5

---

Published as a conference paper at ICLR 2025
Table 3: Main results of wrapper on the REC task, in P@ 11, on Re/COCO/+/g and 2al
datasets. 1
(T)' and (B)' / stand for the 'wwnnI' and
Swwin backbones respectiell..
Model
ReCOCCO
ReCOCO
RefOOCO+
Talk2Car
Adaplation
uccess
VLM
LLM
ual-umd
lest und
val-ume
lest unc
val--me
unc
val
lest
(zeroo shoi)
60.09
59.52
50.69
50.94
51,65
51.79
55.37
58.44
Fine- tunnng
Whitebor
60 (R)
78.51
77.99
83.86
84.12
73.46
73.46
NIA
N/A
L:26 wxapper
Black-box
Mistral
77.57 117.5
77.05 117.7
74.61/239
73.46|22.5
60,3218.7
60,08/8.3
64.75
19.4
67.14
18.7
1229-weappez
Black box
CD (2)
Llama
78.12 118.0
77.36
118.0
74.78/24.1
73.98123.0
64.18:12.5
63.82112.0
65.95
100
68.61
110.2
(2cro-shot)
67.61
68.37
51.82
52.12
53,28
53.16
47.64
51.04
Fine furing
Whit-box (Brac())
80.19
79,85
83.84
84.21
73.68
73.58
NIA
N/A
1i2g-waappes
Black-box
Mistral
78.47 110.9
77.92
19.6
72.612008
7148819.4
63.79/10.5
63,69110.5
63.89
116.3
66.78
11577
LUtt-waapper
Black-bor bux
3a2 112
Liann
78.25 (10.6
78.01
19.6
73.97/22.2
73.07/21.0
64.13;10.9
64.08/10.9
63.97
11..3
66.45 1154
(zcro-shot)
Flo2
67.91
66,16
55,94
57.21
53,31
54.26
46.78
47.53
Fine-tuning
White box
F1c2
90.32
91.02
93.07
93:42
88.19
88.49
NIA
N/A
LUM Meageee
Black-bon box
Flo2
Mixtral
78 1111.1
77.69
111.5
1889
68.21-110
57,5814.3
58 26/40
61.65
114
65.14 14 117.6
11M wir apper
Black-box
o2
Llama3
110.9
78.03
11.99
11.7411588
71.91/1477
62.63/9.3
62.73/8.5
61.74
1550
65.84
118.3
4.1.3 THE LLMS AND THEIR FINE-TUNING
Our main experiments are conducted using two different LLMs: Mixtral 8x7B Instruct (Jiang et al., 2024) (v0.1) and Llama 3 8B Instruct (AI@ Meta, 2024) with Hugging Face's 's implementation. We use Hugging Face's supervised fine-tuning pipeline (SFT) (HuggingFace, 2024), that allows to implement the training choices discussed in Section 3. Specifically, for a same LoRA's rank r, we train 352M parameters for Llama 3 8B (which is 4.20% of the model original size) and 1 114M parameters for Mixtral 8x7B (which is 0.24% of its original size). To further study the impact of the LLM scale, we also experiment with two additional families of models, Gemma 2 (Gemma, 2024) and GPT-Nco (Gao et al., 2021). The latter a class of LLMs, based on a replication of the GPT-3 architecture, trained on the large curated Pile dataset (Gao et al., 2021). and which was not 'instructed' (Ouyang et al., 2022). We train LIM-wrapper with Adam (Kingma, 2014), with batch-size of four, until convergence. We discuss in Section 4.4 the sample efficiency of LLM wrapper. Unless stated otherwise, we use learning rate of 10-6 and rank of r = 128 for LORA. These hyper-parameters work consistently across four datasets, two LLMs and two VLMs, demonstrating the robustness of LLM wrappr. We study the performance robustness of -Lapeeraperr for different papeeeprraeeters in Section 4.4. We provide ablations quantifying the impact of trainable parameters count in Appendix C and statistics on invalid LLM outputs in Appendix D.
4.2 LLM-WRAFPER WRAFPER CAN ADAPT VLMS TO THE REC TASK
In Table 3, we report the performances of VLMS on the REC task in three settings: zero-shot off-the-shelff VLM, after classic white-box fine-tuning, and after black-box adaptation with L.M.-wapper.. The results are obtained by adapting the VLM using exclusively the training data specific to each benchmark.
Our first observation confirms that while VLMs demonstrate remarkable zero-shot performance on new tasks and datasets, their performance is still significantly lower than models specifically adapted to the given task (white-box fine-tuning). As shown in Table 3, for GD, GDrec and Flo2, there is notable gap in performance, ranging from -11. 11.5 to -37.1 P@ I, between the zero-shot and fine-tuned versions.
Interestingly, observe that for all combinations of VLMs and LLMs, LLM-wrapper to the new REC task, despite the black-box setting. For instance, on RefCOCOg, L.M-wrapper brings improvements over the zero-shot models ranging from +9.6 to +18.0 @@ I. Notably, while our proposed variant GDrec outperforms GD by large margins in a zero-shot setting (e.g.. +9.1 P@ I-test) due to better subject identification, both models perform similarly when adapted with L.MM-rrppe that MLM--rrapper particularly improves VLMs that lack abilities useful for the task, such as subject identification for GD. On RefCOCO, LIM-wrapper improves again by significant margins zero-shot VLMs. e.g., +23.0 P@ I (test) for GD with Llama 3 8B. Given that textual queries in RefCOCO are very short (3.5 words on average), subject identification relatively easy. Therefore, these gains indicate that LLM-wrapper greatly helps to disambiguate the object of interest from the distractors. On RefCOCO+, the improvement brought by LLM- wrapper is lower but still significant (between +4.0 and +12.5 P@ I). This is expected as RefCOCO+ is designed to exclude location words, and most referring expressions are thus purely appearance-baas based descriptions (Qiao et al., 2021) where LIM-wrapper brings less value for adaptation. On the driving Talk2Car dataset, LIM-wrappez's results remain consistent with those observed on the classic human-centric RefCOCO/+/g benchmarks, with score boosts provided for all VLMILLM combinations, ranging from +8.7 up + +18.3 P@
adapt VLMs
This shows

---

Published as conference paper at ICLR 2025
LLM-wwapper
Flo2
Ground truth
KRIST
PPerson on skateboard"
(b) "The tie at the second from the left"
(c) "Green plant behind table visible behind lady's 's head"
(d) "She said something about a big sign on a fence. Maybe that is her? ! Pull over here by this person and will find out"
(e) "Try to get in front of the car that passed us on the left. He is driving like madman"
(f) "My friend said she would be standing the corner waiting for me, I think that might be her, will you stop there ?"
Figure Qualitative results of Flo2 on RefCOCOg (first row) and Talk2Car (second row), before and after adaptation with LIM-wrapper, provided with queries as captions. Adapting Flo2 (in orange) with wrapwrr (using Llama 8B, in blue) leads to improved reasoning and box selection.
We present some qualitative results in Figure 2, where we show predictions from Flo2 before and after adaptation with wraperr using Llama 3 8B. We observe that adaptation with MMMM-waaper enables better subject identification (Fig. 2a, Fig. 2d), spatial understanding (Fig. 2b, Fig. 2e), relational reasoning (Fig. 2a, Fig. 2c) and avoids selecting a more visible object when the ground truth is comparatively small (Fig. 2d, Fig. 2f). We display additional qualitative results, including failure cases, Appendix E.
LLM-wrapper r is not designed to outperform classic white-box fine-tuning but demonstrates competitive performance some settings. For example, with GD and GDrec on RefCOCOg, LIM-wrapper achieves comparable results despite using smaller vision backbone. Additionally, I.M.-wrapper can complement white-box fine-tuning by adapting VLMs already optimized for the REC task. As shown in Appendix A, applying LIM wrapper on top of fine-tuned models does not degrade performances and, in some cases, provides slight boost. This highlights LLM- H.MM-wapper's compatibility with state- of-the-art methods and its ability enhance existing adaptations. Finally, in Appendix B, extend evaluation to the related task of Referring Expression Segmentation, leading to consistent score boosts.
4.3 BENEFITS OF WRAPPING VLMS' OUTPUTS WITH LLM- WRAPPER
Ensembling VLMs. One advantage of LLM-wrapper is its free-text input format, which allows adapting to various inputs. Indeed, we show that LIM-wrapper can learn to ensemble outputs from different VLMs, shown in Table 4. Specifically, described in Section 3. The results show that when ensembling GDrec and Flo2, scores are boosted by
concatenate the predictions of the two VLMs in the prompt
Table Results of VLMs ensembling using LIM-wrapper with Llama 8B, in P@ 1 on Ref- COCOg using 'umd' splits. (Comparable findings for Mixtral).
Adaptation
VLM
P@1 val
P@1 test
(zero-shot VLM)
GDrec
67.61
68.37
Ø (zero-shot VLM)
Flo2
67.91
66.16
wraper
GDrec
78.25
78.01
LLM LLM-wrapeer
Flo2
78.76
78.03
I.M--waapper
F1o2 + GDrec
81.25
80.13
7

---

Published as a conference paper at ICLR 2025
Query: "A bottle of wine between the vegetablex"
Vegerabres
bottle
Végetabies
bottle wine
Nine
wine
Hlle les
bouie
boptre Kina
Ground Irufh
F122
GDrac
(a) Ground truth
(b) All candidates Flo2
(c) All candidates of GDrec
bottle
bottle wine
LLM--r
R122
GDrcc
(d) Final pred. of LLM- araper
(c) Final pred. of r1
(f) Final pred. of GDrec
Figure 3: Visualizations of the candidates and predictions for the query "A bottle of wine between the vegetables". We visualize the ground truth (a) and the set of box candidates generated by Flo2 (b) and GDrec (c). In the second row, we visualize the final predictions of F1o2 (e) and GDrec (f) and in (d) the prediction of LLM wrapper applied on the ensemble of both VLMs' outputs, using Llama 3 8B. We observe that HLM-wrappez discards the distractor bottle and selects the correct object.
+2.5 P@1 (val-umd) and +2.1 P@ (1 (test-umd) when compared to those obtained with the pest-performin VLM adapted with LIM-wrapper, namely Flo2. This demonstrates that LIM-wrapper r is capable of reasoning on multiple sources and leveraging the strengths of different models.
Indeed, we observe that Flo2 has high precision, while GD and GDrec have lower precision but high recall. We show in Figure 3 qualitative example where L.M--rraped leverages the complementarity of the two models. The figure displays predictions from Flo2 (Fig, 3e), GDrec (Fig. 3f), and LLM- wrapper ensembling predictions from Flo2 and GDrec (Fig.. 3d). We observe that while Flo2 fails to detect the target object as possible candidate (Fig. 3b), the additional proposals from GDrec (Fig. 3c) enable LLM-wrapper to find the correct object (Fig. 3a) that each independent model missed.
Transferring a trained LrMpprrpper to a new VLM. Another advantage of using text as the input format that L.M-waapper does not rely on model-specific activation values, making transferable from one VLM to another. For instance, L.M-wrapper can be fine-tuned on F Flo2 and transferred to GD or GDrec. This is illustrated in Table 5, where, for instance, when fine-tuned on GDrec's or Flo2's 's outputs, transfeming it at inference time to the other model's outputs gives an increase from +5.1 to +6.9 P@1 over zero-shot VLMs. This shows that during fine-tuning. and semantic notions that generalize to other models. This capacity of LLM-wrapper is particularly useful for private models, such as GD-1. 5 (Ren et al., 2024a), where creating the training set can be expensive for instance, getting predictions for RefCOCOg train would cost x $1,600 ($20 per 1,000 API
learns spatial
Table 5: Results of LIM-wrapper when using different VLMs' outputs during fine-tuning and inference, in P@ If on RefCOCOg using 'umd' splits. Results obtained with Llama 8B. Comparable findings for Mixtral. tScores obtained on subset of 300 samples from RefCOCOg val-umd.
VLM (fine-tningg
VLM (inference)
P@1 val f
P@ - val f P@1 test f (full)
Adaptation
(subset 300)
(full)
0 (zero-shot VLM)
0
GDrec
66.00
67,61
68.37
LLM-wrapper
Flo2
GDrec
74.00
18.0
73.90
16.3
73.45 55.1
0 (zero-shot VLM)
0
Flo2
71.671
67.91
66.16
LLM-wrapper
GDrec
Flc2
75.331
13.7
73.86
16.0
73.03 *6.9
0 (zero-shot VLM)
0
GD-1.5
47.67t
LLM-wrapper
GDrec
GD-1.5
76.67t
129.0
8

---

Published as a conference paper at ICLR 2025
Table 6: LIM-wrapper performance on zero-shot dataset transfer. Results obtained with Llama 3 8B. "FT" stands for "fine-tuning'. With 1, Flo2 is fine-tuned using "a collection of public supervised data on wide range of downstream tasks', including but not limited to Re/COCO/+/g (Xiao al., 2024).
Adaptation
VLM Fine-tuning Data Inference Data
P@1 - val 1 P@1 - test
0 (zero-shot VLM)
Flc2
HC-RefLoCo
48.04
47.39
White-box FT
Flo2
RefCOCO/+/g
HC-RefLoCo
56.75 18.7
55.62 18.2
LLM Lrapper
Flo2 RefCOCOg
HC-RefLoCo
66.93 188.9
66.45 $19
0 (zero-shot VLM)
Flo2
RefCOco
55.94
57.21
LIM-wrapper
Flc2 RefCOCO
RefCOCo
71.74 115.8
71.91 114.7
LiM-wrapper
Flo2
RefCOCOg
RefCOCO
69.00 113.1
68.88 111.7
0 (zero-shot VLMM
Flc2
RefCOCO+
53.31
54.26
LIM- wrapper
Flc2 ReiCOCO+
RefCOCO+
62.63 19.3
62.73 18.5
LIM wrapper
Flo2 RefCOCOg
RefCOCO+
61.00 17.7
61.07 16.8
calls). To illustrate this use case, we use a RefCOCOg val subset corresponding to 300 free API calls to GD-1.5. When LLM-wEe ara-wrrpper is fine-tuned on GDrec's outputs and applied on GD-1.5 5's outputs for inference, results are boosted by a significant +29.0 P@ 1. To ensure that the val-subset reflects the full val set's difficulty, we evaluate GDrec and F Flo2 on both sets. The results are consistent, confirming that the val-subset is a good proxy for the full set and that the performance gains from -iM-wrr on GD-1.5 are significant This showcases how LLM- wwapperpprrr can be used on private models or on continuously updated versions of models without the need to re-train it with each model update.
Transferring a trained LIM-waapper to datasets. L.M-wrapper demonstrates strong gen- eralization across datasets, reducing the need for training data and resources on new target domains. To demonstrate this property, we adapt Flo2 with AMM-wrapper on RefCOCOg and evaluate on RefCOCO and RefCOCO+. Results in Table 6 show substantial gains over zero-shot F Flo2, with improve- ments up to +13.1 P@1 1 on RefCOCO and +7.7 P@ on RefCOCO+. Although slightly below results from direct fine-tuning target datasets, these improvements demonstrate LLM-wrapped knowledge effectively. To test generalization to more complex scenarios, we evaluate Flo2 adapted with LLM-wrapper using RefCOCOg on HC-RefLoCo, benchmark with no training split and longer, more complex, referring expressions, with an average length of 84 words. LLM-wr wraper achieves +18.9 and + P@ improvements Flc2. Interestingly, when white-box fine-tuned Flo2 is transferred to HC-RefloCo, its performance boost zero-shot Flo2 is less than half of that achieved by L.M-wrapper: These results highlight LLM- wrapperss ability to handle in zero-shot setting complex referring expressions despite being fine-tuned on ten times shorter expressions. More details on the impact of input complexity on performance are given in Appendix C.2. Qualitative examples on HC-RefLoCo are shown in Appendix E.I.
ability to transfer
the validation and test sets, respectively, compared to zero- shot
4.4 ABLATION STUDIES
Unless specified otherwise, we run ablations on RefCOCOg val-umd, using GD's outputs and Llama 3 8B.
Impact of LLM scale. In Figure 4a, we report the P@1 1 REC performance (y-axis) for various L.M sizes (I--axis). We observe that L.M-wrapper effective across all model families, and that the gain in performance positively correlates with the LLM size. Indeed, though the sub-billion model, GPT-Nco 139M, slightly boosts the P@ performance of GD zero-shot (+2.2 P@ I), this increase is only incremental when compared Llama 3 8B's gains (+18.0 P@ 1). Further analysis shows Pearson correlation of 0.88 between the LLM's original performance on reasoning benchmarks, measured by the HellaSwag, score (Zellers al., 2019), and REC performance (details in Appendix C) Overall, these findings demonstrate that larger, more capable L.MMs are substantially more effective, while smaller models offer only limited improvements.
Robustness to hyper-parameters, In Figure 4b, we show that L.M-wrapper is not overly sensitive to the value of the LoRA rank, which controls the number of fine-tuned parameters. In the explored range, E (12,64,128,10 192), corresponding to fine-tuning (0.41%, 2.15% 4.20%, 6.18%) ) of total parameters, the performance varies by only =0.4 P@ We provide a more in-depth analysis of the impact of the number of fine-tuned parameters on performance in Appendix C.I. Similarly, we observe in Figure 4e that HLM-wrapper is loosely sensitive to the learning rate choice for values in {10-b 60- 7,5 10 ,_10 15]. With learning rate values higher that 10-5 observe unstable training behaviors.

---

Published as conference paper at ICLR 2025
77.5
75.0
72.5
78
70.0
GPT-Neo
76
77
77
Gemmar2
65.0
Liama
75
75
62.5
Mixtral
74
74
zero-shor shot
73
60.0
0.11.
2.8
7.0
9.7
12.9
72
fx10-
5x10-4
(billions).
LoRA rank (r).
Learning rate.
(a) LLM size.
(b) LoRA parametrization.
(c) Learning rate.
Figure Study LIM-wrapper's sensitivity hyper-parameters. Impact on wrapprr's
performances (with GD RefCOCOg) of the (a) LLM scale; (b) LoRA rank r; and (c) learning rate.
Ablation on the number of training samples. Figure 5 displays the P REC performance on ReiCOCOg val with respect the number training samples, when fine-tuning Llama 3 8B on RefCOCOg train. shows sharp increase in P@ for all VLMs (most impressive for GD) during the first 30k samples, which takes 2h training in our setting. Thus, even with a restricted amount of samples, can boost performances. In the extreme case of no training samples for the LLM, i.e., zero-shot LLM, we observe on Table that most settings the VLM and the VLM zero-shot LLM have very similar results. This follows the observation that, while LLMs have extensive general knowledge, they may lack off- the-shelf reasoning (Kazemi et al., 2023; Fu et al., 2024).
TLM-wrapper
P@ 1-valf
P@ P@I-testi
74
VLM
VLM
+LM
VLM
I.I.M zero-shot
only
zero-shot
only
P@1
GD zero-shot LLM
GD
60.09
58.05
59.32
58.47
LLM-wrapper
GDrec
67.61
67.48
68.37
68.17
66
GDrec + zero-shat
GD-1.5
47.671
59.001
62
GDrec + LLM-wrapper
F1o2
67.91
67.69
66.16
66.44
Fla2 + zero-shot LLM
Flo2 + LLM-wrapper
58
Table Results wrapping the VLM's outputs with a zero-shot LLM. "VLM only' are the scores of the VLM without any adap- tation, and '+ LLM zero-shot' corresponds to LTM-wrapper without any fine-tuning of the LLM (Llama 8B here). see Table 5.
20K
40K
60K
100K
120K
# of fine-tuning samples
Figure 5: Performance (P@1) 1) of LIM-wrapper on RefCOCOg (val) with respect to the number of training samples. We fine-tune Llama 3 8B on ReICOCOg (train).
5 CONCLUSION
This work introduces LLM wrapwrr simple approach for the black-box adaptation of VLMs the REC task, that leverages an LLM to reason on VLMs' outputs, translated into natural language. We demonstrate that LLM-wrapper significantly boosts the performance of VLMs, for several combinations of L.LSs and VI.Ms, and, in some settings, even bridges the gap to classic fine-tuning. We also show how hiM-wrrpper can ensemble predictions from different VL.Ms leverage their respective strengths and how it can transfer across VLMS and to out-of-domain datasets. Thanks to efficient and well-studied methods for LLM fine-tuning, LLM-Wr SKK-wrappr is simple to use in practice, requiring limited hyper- parameter tuning, and computationally efficient. Future works include relying fewer examples to fine-tune LIM-wrapper, and applying LLM-wrapper to different tasks, such as text-video retrieval (Fang al., 2021).
Limitations. LLM--rapper comes with some limitations. First, an additional inference cost is intro- duced by integrating L.LM reasoning VLM outputs. Second, the effectiveness of LIM-wrapper relies on the quality the underlying VL.M since diverse and accurate bounding boxes are crucial for success. Third, bounding box information alone (box coordinates and labels) may not be sufficient for LLM- wrapper to 'understand" the scene in some cases. Examples are shown in Appendix E.2. To address this issue, promising direction is to enhance LLM-Wa wrapper by using the LLM identify missing visual information and suggest augmented queries for the VLM. This would help disambiguate cases where additional visual cues are needed.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Pr. Eric Gaussier for his support and insightful feedbacks on this project. This work was performed using HPC resources from GENCI-DRRS (Grant 2024-AD01 100-AD0111114440 1014446) and with the support of the ANR MultiTrans (AN3-2I-CE23-0033).
REFERENCES
Al@Meta. Llama 3 model card. 2024, 6
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebustian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Conference on Neural Information Processing Systems, NeurlPS, 2022. 1
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bemsttein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint, 2021.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint, 2023. 3
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Conference on Neural Information Processing Systems, NeurlPS, 2022. 4
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. Talk2car: Taking control of your driving car. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, EMNLP. IJCNLP, 2019. 2,5
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Conference on Neural Information Processing Systems, NeurIPS, 2024. 4
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang. and Lu Yuan. Davit: Dual attention vision transformers. In European Conference on Computer Vision, ECCV, 2022.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. In International Conference on Machine Learning, ICML, 2023. 1
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. The llama herd of models. arXiv preprint, 2024. 2
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint, 2021. 10
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024.
Deqing Fu, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. arXiv preprint, 2024. 10
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, 2021. 6
11

---

Published as a conference paper at ICLR 2025
Team Gemma. Gemma 2: Improving open language models a practical size. arXiv preprint, 2024. 6
Ross B. Girshick. Fast R-CNN. In International Conference Computer Vision, ICCV, 2015. 2
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations, ICLR, 2021. 16
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength
Natural Language Processing Python. 2020. 5,
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference
Learning Representations, ICLR, 2022. 3, 4
HuggingFace. Supervised fine-tuning trainer, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint, 2024. 2,6
Mchran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint, 2023. 10
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Conference Empirical Methods in Natural Language Processing. EMNLP, 2014. 5,
Dicderik P Kingma. Adam: method for stochastic optimization. arXiv preprint, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yes Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In International Conference on Computer Vision, ICCV, 2023. 15, 16
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation. In International Conference Learning Representations, ICLR, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmentation via large language model. In Conference on Computer Vision and Puttern Recognition, CVPR, 2024. 15, 16
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning, Optimizing continuous prompts for generation. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing, ACIAJCNLL,, 3021.
Long Lian, Boyi Li, Adam Yala, and Trevor Darell. Lim-grounder diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Trans. Mach. Learn Res., 2024a. 2
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Lim- grounded video diffusion models. In International Conference on Learning Representations, ICLR, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In Conference on
Neural Information Processing Systems, NeurlPS, 2023. 1,3
Shih- Yang Liu, Chien Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight Weight-decomposed low-rank adaptation. International Conference Machine Learning, ICMI, 2024a. 3
12

---

Published as a conference paper at ICLR 2025
Shihong Liu, Zhiqiu Lin, Samuel Yu, Ryan Lee, Tiffany Ling, Deepak Pathak, and Deva Ramanan. Language models as black-box optimizers for vision-language models. In Conference on Computer Vision and Pattern Recognition, CVPR, 2024b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision, ECCV, 2024c. 1, 2, 3,5
Junhua Mao, Jonathan Huang. Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Conference on Computer Vision and Pattemn Recognition, CVPR, 2016. 2, 15
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Mor- gane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christo- pher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Cristian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. arXiv preprint, 2024. 2
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung. Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Conference on Computer Vision and Pullern Recognition, CVPR, 2023. 3
OpenAL. Fine-tuning api, 2024. 1
Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. In International Conference on Computer Vision, ICCV, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Conference Neural Information Processing Systems, NeurlPS, 2022. 6
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint, 2023. 15
Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring expression comprehension: A survey of methods and datasets. IEEE Trans. Multim., 2021. 2, 5,
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, ICML, 2021. 1,2
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing. Ming-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In Conference on Computer Vision and Pattern Recognition, CVPR, 2024. 16
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding DINO 1.5: Advance the "edge" of open-set object detection. arXiv preprini, 2024a. 5,
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiay Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded SAM: assembling open-world models for diverse visual tasks. arXiv preprint, 2024b. 15
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision, ECCV, 2016. 2
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint, 2023. 2
David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding vision-language models without training. European Conference on Computer Vision, ECCV, 2024. 3, 15
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. arXiv preprint, 2023. 3
Fangyun Wei, Jinjing Zhao, Kun Yan, Hongyang Zhang, and Chang Xu. A large-scale human-centric benchmark for referring expression comprehension in the Imm era. In Conference on Neural Information Processing Systems, NeurlPS, Datasets and Benchmarks Truck, 2024. 2,
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities large language models. Truns. Mach. Learn. Res., 2022. 1
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Conference on Computer Vision and Pultern Recognition, CVPR, 2024. 3,5,,
Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang, Described object detection: Liberating object detection with flexible expressions. In Conference on Neural Information Processing Systems, NeurlPS, 2023.
Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang, Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? A benchmark for comprehensive open-vocabulary detection. In Conference on Artificial Intelligence, AAAI, 2024.
Lang Yu, Qin Chen, Jiaju Lin, and Liang He. Black-box prompt tuning for vision-language model as a service, In International Joint Conference on Artificial Intelligence, IJCAI, 2023. 3
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Conference of the Association for Computational Linguistics, ACI, 2019. 9,
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In International Conference on Computer Vision, ICCV, 2023.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS: LIM-WRAPPES ON ALREADY REC-TUNED VLMS
We explore the use of LIM-wrapper on VLMs that are already optimized to deal with the REC task. This supplementary experiment aims to confirm whether waap-eee compatible with pre-existing REC-specific adaptations. Our analysis includes VLMs from our main experiments after white-box fine- tuning on REC data i.e., GD SwinB, GDrec SwinB and Flo2 o2 fine-tuned. We also include an additional VLM, Kosmos-2 (Peng et al., 2023), that is directly designed to ground referring expressions. For each candidate VLM, we compare P@ I scores, with and without additional SLM--raapes adaptation using RefCOCOg-trail (umd) data. The evaluation is made on RefCOCOg val-umd and test-umd and results shown in Table 8.
Table 8: Results of LIM-wrapper on the REC task, when applied to already REC-adapted VLMs, in P@I1; 1t. 'FT' stands for 'fine-tuning' and ' (B) for the 'SwinB' backbone. marks results directly taken from Contrastive Region Guidance (CRG) paper (Wan et al., 2024).
RefCOCOg
Adaptation
VLM
LLM
val-umd
test-umd
White-box FT
GD (B)
N/A
66.30*
White-box FT + CRG (Wan et al., 2024)
GD (B)
N/AT
69.60t 13.30
White-box FT
CD (B)
78.51
77.99
White-box FT+ LIM-wrapper
(B)
Mixtral
82.31 13.80
82.15 14.16
White-box F LLM MMM-wraper:
GD (B)
Llama3
82.76 14.25
82.61 14.62
White-box FT
CDrec
80.19
79.85
White-box FT+ LIM-wrapper
GDrec (B)
Mixtral
82.58 12.39
81.95 12.10
White-box FT+ T.M.wrrapper
GDrec (B)
Llam3
81.66 11447
81.47 1166
White-box FT
Flo2 Mixtral
90.32
91.02
White-box FT LIM-waapper
Flo2 FT
90.40 10.08
90.92 [0.10
White-box FT + LIM-wrapper
F1o2 FT
Llama3 90.50 10.18
91.03 10.01
Designed for REC
Kosmes 2
60.60
61.41
Designed for REC + LLM wrapper
Kosmos 2 Mixtral 62.03 1,.43
62.59 1,18
Designed for REC+ + LLM-wrapper Kosmos-2 Llama3 62.09 11.49
62.39
10.98
While the performance gains are modest an expected outcome the VLMs are already adapted and optimized for the REC task -, it is important to note that LIM-wrapped avoids degrading performances by any significant value and, in some cases, achieves slight improvement of up to +4.6 P@1. These results highlight the compatibility of LLM-wrapper with any state-of-the-art VLM, whether previously fine-tuned on REC data or not.
Evaluating LLM-wrapper, when applied to GD SwinB, also allows for direct comparison with related work Contrastive Region Guidance (CRG) (Wan al., 2024), mentioned Section 2. As shown the first 5 rows of Table 8, LLM-wrapper displays both higher final scores and bigger score boosts than CRG on RefCOCOg test set, while remaining fully black-box and requiring a single VLM inference per input (text query, image) pair.
B EVALUATING LIM-WRAPPER ON REFERRING EXPRESSION SEGMENTATION
We explore the use of LIM-wrapper on Referring Expression Segmentation (RES). This task, closely related to REC, consists in outputting a segmentation mask instead of a bounding box - based on (text query. image) input pair. Following Ren et al. (2024b), we use Segment Anything (SAM) (Kirillov et al., 2023) to convert predicted bounding boxes into segmentation masks. As for metrics, we follow Kazemzadeh et al. (2014), Mao et al. (2016); Lai et al. (2024) and compute cloU (cumulative intersection over cumulative union on all referring expressions) and gloU (average of referring expression-be based IoUs, which is less biased in favor of large objects). This experiment aims to extend LLM-wrapper new visual tasks, beyond REC, Moreover, the use of Jos-b metrics (instead of P@ in REC) further tests the robustness of our predicted outputs.
15

---

Published as a conference paper at ICLR 2025
Table 9: Results of LIM-waapper on the Referring Expression Segmentation (RES) task, in cloU/gloUf, alongside SOTA RES-specific models. *FT' stands for 'fine-tuning', * (T) and (B) r for the 'wwin'' and 'Swin backbones. t indicates results directly taken from cited papers. Finally, 'SAM' indicates the use of Segment Anything (Kirillov et al., 2023) to convert bounding boxes into segmentation masks.
RefCOCOg (und)
cloU
gloU
Method
VLM
LLM
val
test
val
test
LISA-7B FT (Lai et al., 2024)
67.9
70.6
N/A
N/A
GLaMM (Rasheed et al., 2024)
74.2
74.91
N/A
N/A
Zero-shot VLM + SAM
GD (T)
40.90
41.37
50.56
50.46
shro VLM+ + LIM-waapper SAM
(D)
Mixtral
57.80 116
58.00
16.6
64.2
"4.6
Zero-shot VLM+ + LIM-waapper SAM
(T)
Lluma3
57.86 117.00
58.59
11722
65.12
65.42/15.0
White-box + SAM
(B)
57.93
58.53
65.62
65,96
White-hox FT + L.M.-wrapeer + SAM
(B)
Mixtral
62.42 14.55
63,39
14.9
68.54 12.9
69.04/3.1
Whitc-box FT + WIM-waapper + SAM
(B)
Lluma3
62.44 [4.5
63.84
15.3
68.81
(3.5
Zero-shot VLM SAM
F1o2
48.12
47.04
57.24
56.19
Zero-shot VLM LIM- wxapper SAM
F1o2
Mixtral
59.38 111.
58,49
111.5
66.1718.9
65.5979.4
Zero-shot VLM + LIM-wzapper SAM
F1o2
Lluma3 59.53 111.4
59.58
122.5
66.07
66.00-9.8
White-box FT SAM
F1o2 FT
70.73
72.61
74.81
75.93
White-box FT + LEM-wrapper SAM
Flo2
FT
Mixtral
70.81 10.1
72.50 J0.1
74.
75,8910.04
White-box FT
I.M-wrapper SAM
Flo2
Llama3 70.92 10.2
72.61
0.2
75.94"0.01
In Table 9, we show results of LLM wrapper and of SOTA RES-specific methods - pixel grounding Large Multimodal Models (LMMs). The latter approach integrates image encoding information into LMMs, using vision-language projections and vocabulary augmentation with specialized tokens related to segmentation masks (Lai et al., 2024) image region encodings (Rasheed et al., 2024). They are designed and trained, in a white-box end-to-end manner, to output fine-grained segmentation masks. Contrasting with this approach, I.M.-waapprr is a model-agnostic black-box adaptation framework that emphasizes improves cloU/gIoU scores in almost all cases, in accordance with our REC results in Table 3 and Table 8. Note that LLM--waaper's RES scores depend on the VLM being adapted and on SAM's box-to-mask conversion performance.
object-level reasoning. Table 9 shows that
C ADDITIONAL ABLATIONS ON POSSIBLE VARIABLES IMPACTING PERFORMANCES
C.I IMPACT OF THE NUMBER OF FINE-TUNED PARAMETERS ON PERFORMANCE
Our ablation study on LoRA's rank r, presented in Figure 4b, shows the P@| variations for different r values. As the number of trainable parameters scales linearly with LoRA's rank r, this analysis highlights how trainable parameters count impacts performance. To ablate more precisely this aspect, we analyze the impact of the number of fine-tuned parameters on the P@ performance in Table 10. In this table, we also report the original performance of the respective LLMs the standard MMLU benchmark (Hendrycks et al., 2021) and HellaSwag (Zellers et al., 2019), a benchmark for LLM commonsense reasoning evaluation. As part of our analysis, also compute the Pearson correlation between the P@) scores and other variables from Table and summarize results in Table 11.
Table 11 shows that the absolute number of fine-tuned parameters has only loose correlation with performance (pearson=0..7)) This supports findings from Table 10 showing the limited impact of the number of fine-tuned parameters, such as:
Variations in LoRA's rank r do not have large effect on the P@] 1 score of Llama3 8B on RefCOCOg (cf Table 10, rows to 4).
Llama3 8B (r 64, 176M trainable parameters) and Mixtral 8x7B (r = 128, 114M trainable parameters) yield very similar P@ I scores (77.72 vs. 77.57), despite Llama3 8B fine-tuning much higher percentage (2.15% vs. 0.24%) of parameters. The similarity of results, despite
16

---

Published as a conference paper at ICLR 2025
Table 10: Impact of the number of trainable parameters on the performances of adapted by LIM-wrapper and evaluated on RefCOCOg.
Trainable params
Total params
Trainable params %
P@1
LLM MMLU
LLM HellaSwag
LLM
Rank (r)
(val splits)
Llama3 8B
12
33M
8.1B
0.41
77.84
66.6
82
Llama3 8B
64
176M
8.2B
2.15%
77.72
66.6
82
Llama3 8B
128
352M
8.4B
4.20
78.12
66.6
82
Llama3 8B
192
529M
8.6B
6.18%
77.70
66.6
82
Mixtral 8x7B
128
114M
46.8B
0.24
77.57
70.6
84.4
Gemma-2 2 9B
128
465M
9.7B
4.79
74.12
71.3
81.9
Gemma-2 2B
128
199M
2.8B
7.08 %
70.51
52.2
72.9
Table 11: Pearson Correlation with LIM-wrapper's P@1 for various variables.
Variable
Pearson Correlation with P@1
L.MM Total Params (count)
0.33
LLM Trainable Params (count)
-0.07
LLM Trainable Params (%)
-0.64
LLM MMLU score
0.72
LLM HellaSwag score
0.88
different fine-tuning settings, could then due architectural differences, with Mixtral being Mixture of Experts model.
Gemma-2 9B (r= 128, 465M trainable params) underperforms compared to Llama3 8B (r 64, 176M trainable params), even though its setting includes fine-tuning more than double the number of parameters.
These findings suggest that the architecture of the LLM and training specifics may have more influence on performances. Furthermore, our analysis finds strong correlation between I.M.-waaper's P@ scores and the LLM's performances on reasoning tasks, with Pearson correlation of 0.88 with HellaSwag. In conclusion, the number of fine-tuned parameters has a small impact, but the LLM's architecture and original performance on reasoning tasks play more significant role on LLM HLM-wrapper's REC performance.
C.2 IMPACT OF INPUT COMPLEXITY
PERFORMANCE
80
80
80
Subsutt Pgp
60
Subsett PgP
60
60
40
80
20
Wowooopp 3
20
115 400 Limmz
LLM Uame
Merrppp W7 Miciol
115 201 Masre
LLM wkasper Matza!
11
groups Quety
candiaad prompt
Mao of booes with the wete label
(a) Query complexity.
(b) Number of candidate boxes.
Box label redundancy.
Figure Study of LIM-wrapper's sensitivity input complexity. LIM-wrappez's performance in P@ on subsets of the data with varying levels of complexity, coming from the query (number of noun groups the query) or box listing the prompt (number of candidate boxes and box label redundancy, defined as the highest number of boxes with the same label detected for a query). I.M-wrapper is used with F Flo2 and subsets are aggregated from ReiCOCO/+/g and Talk2Car val sets.
In Figure 6, we aggregate results from RefCOCO/+/g and Talk2Car val sets to show LLM-wrappa P@ I scores different data subsets. These subsets split based varying levels of input complexity, with respect to the number of noun groups in the query (Fig. 6a), the number candidate boxes the prompt (Fig. 6b) and the redundancy of box labels (Fig. 6c). We first study in Figure 6a, I.M--aaaper's
17

---

Published as a conference paper at ICLR 2025
P@ I performances when the the number of noun groups' increases. For both Llama 3 8B and Mixtral, we see that LIM-wrapper is robust to, and even benefits from, increasing textual complexity, with stable scores around 67 P@ I for 0 to 9 noun groups, and 100 P@] for 10 to 11 noun groups. We then display in Figure 6b how performance evolves with respect to the number of boxes listed in the prompt. L.M.-aapper is robust to increasing number of candidates, with P@1 around 66 for 2 to 15 boxes, and only a few performance variations for more than 8 boxes. Finally, we test finer-grained notion of box label redundancy, defined the highest number of boxes with the same label detected for query. Figure 6c shows how performance changes based on this redundancy. For both Llama 3 8B and Mixtral, a clear decreasing tendency observed on the performance as more boxes share a same label, in particular when that is the case for more than 6 boxes. This study indicates that LIM-wrapper is robust to increasing levels of input query and box listing complexity, as long as the detected boxes display diversified labels.
D FAILURE CASE ANALYSIS
failure case arises when the LLM does not succeed in producing the index of a candidate bounding box output. This happens if the LLM outputs non-integer value integer that is out of range with respect to the candidate boxes' list However, these issues are very rare. In our experiments, a fine-tuned LLM-waapper shows 0% of non-integer output instances, and only the issue of out of range integer generation remains. In Table 12, we report the percentage of instances where the output integer is out of range, when Flo2 is adapted with LiM-wrapper, using Mixtral or Llama 3 8B. We observe that such issues occur in less than 0.3% of instances. In these rare cases, we use a simple fallback strategy: the best-ranked box from the zero-shot VLM is used as prediction. For qualitative examples of failure cases due to reasoning issues, rather than generation issues, intuitions and visualizations are given in Appendix E.2.
Table 12: Percentages of out of range integer generation when using LIM-wrapper on Flo2. indicates that results are obtained while using transferred LIM-wrapper, trained on RefCOCOg.
RefCOCOg
RefCOCO
RefCOCO+ Tal HC-RefLoCo
Method
LLM
val
test
val
test
val
test
val
test
val
test
LLM-wrapper Mixtral
0.04%
0.03%
0.01%
0%
0.02%
0.03%
0%
0%
LLM waapprr
Llama3
0%
0%
0.30%
0.22%
0.06%
0.09%
0%
0%
0%
0%1
E
ADDITIONAL QUALITATIVE EXAMPLES
In this section, we use LIM-wrapper with Flo2 and a Ilama3 8B fine-tuned on RefCOCOg data.
E QUALITATIVE SUCCESS CASES ON COMPLEX QUERIES
We show qualitative examples of LIM-wrapper's successes against white-box fine-tuned Flo2 on HC-ReiLoCo Figure 7 and Figure 8. In these examples, LLM-Wal wrapper is able to properly process more than 10 candidate boxes, aligned with long and complex queries, to identify the correct box, while woite-box fine-tuned Flo2 is predicting distractor objects.
E.2 QUALITATIVE FAILURE CASES
We display qualitative examples of LIM-wrapper's failures against zero-shot Flo2 on RefCOCOg. In particular, as mentioned in Section the main failure case of LLM wrapper occurs when bounding box coordinates and labels alone are insufficient to ground certain expressions that require additional visual cues. We give various examples showing this type of failure in Figure 9. For instance, L.M-wrapper: missing visual information on the zebra's 's head direction in Fig. 9a, on the relative position of the curtains and chairs in Fig. 9b and on the position of each person with respect the camera in Fig. 9c.
Noun groups in queries identified using Spacy's 's noun chunks method (Honnibal et al., 2020).
18

---

Published as a conference paper at ICLR 2025
Query: "This individual appears to be woman with long, struight blonde huir that drapes over her right shoulder. She is wearing light-colored, possibly cream or pale yellow blazer. The woman is engaged
activity where she bringing clear glass, which she holds in her right hand, towards her mouth as if to take sip of bevenage. Her left hand is not visible the image. She is positioned on the far left in the group of people captured in the photograph."
woman
hair
the woman
the woman
ENIS individual
ETIS individuaa
mouth
ner Hand
Ground truth
Flo2
cleler glass
(a) Ground truth
(b) All 13 candidates from F1o2
LLM-wrapper
Flo2 FT
(c) Final pred. of LIM-wrapper
(d) Final pred. of fine-tuned Flo2
Figure 7: First qualitative result of LIM-wrapper tuned RefCOCOg, evaluated on HC-RefLoCo. LLM-wrapper takes multiple candidates from zero-shot Flo2 as inputs (in orange in Fig. 7b) to identify the best box (in blue, in Fig. 7c), while white-bot fine-tuned Flo2 fails (in red Fig. 7d).
Query: "The individual middle aged man with short, dark hair, appearing startled comically alarmed. He wearing pale dress shint and is positioned as if emerging from a mirror, with his left side showing."
mirror. with his left side showing
dark ha
Inoo
aged man
middle aged man
dress shie the individual
dress SHIFL
Ground truth
Flo2
Ground truth
(b) candidates from Flo2
LLM-wrapper
Flo2 FT
(c) Final pred. of Lra-wrrpper
(d) Final pred. of fine-tuned F1o2
Figure 8: Second result of LIM-wrapper evaluated on HC-RefLoCo, Same legend as Figure 7.
19

---

Published as a conference paper at ICLR 2025
RESTAURANTE CERVE
BOCK Power
Fluz
Gorund
(a) Query; "Zebra whose head is not (b) Query: "Chair without the curtain facing down."
(c) Query: "A man with his back turned toward the camera, enjoying conversation with friend."
on it"
Figure 9: Visualizations on RefCOCOg of LIM-wrapper's main failure case. L.M-wrapper's predictions are in blue vs. zero-shot Flo2's predictions orange. these three examples, LLM- wrapper fails to identify the best candidate box coordinates and labels are not providing enough visual cues for the LLM to choose correctly. For instance, LIM-wrapper is missing visual information on the zebra's 's head direction (Fig. 9a), the relative position of the curtains and chairs (Fig. 9b) and the position of each person with respect to the camera (Fig. 9c).
Going beyond w.M-waapper's main failure scenario, we add visualizations of corner failure cases in Figure 10. They illustrate possible issues hindering the L.LM's reasoning. i.e., when proper candidate box is identified by the zero-shot VLM in the first place (Fig. 10a), when the VLM fails to detect important contextual objects, necessary to 'perceive" the scene and localize the object of interest (Fig. 10b), when rich candidate boxes exist but without adequate labels (Fig. 10c).
doc
chickon
Bog
The TEOO
SOBS
Floz
Flo2
wrapper
wropper
B
LLM
Gonnndluu
(a) Query: 'Black and white dog with pointy ears!
(b) Query: Dark chicken clasest the fence.'
(c) Query: 'The head and shoulders and one leg of goat closest the dog.'
Figure 10: Visualizations of three corner failure cases of LIM-wrapper RefCOCOg. As we LLM-wrapper to adapt a zero-shot F Flo2, we show on the first row in orange, for each sample, respective Flo2's candidate boxes that LLM-wrapper takes inputs. In the second row, visualize the prediction of FLMM-rrappe in blue, chosen among Flc2's candidates, as well as the ground truth box. We observe that the LLM reasoning be hindered if candidate boxes missing the object of interest (the correct dog in Fig. 10a), necessary contextual objects (the fence in Fig. 10b) or proper labels (goats detected but not properly labeled in Fig. 10c, bringing confusion with respect the scene description).
20


Reviews:
[{"review_id": "kTGOMSozU3", "reviewer": "Reviewer_q6Km", "summary": "The paper presents LLM-wrapper, a method for black-box adaptation of VLMs for the Referring Expression Comprehension (REC) task. It leverages the reasoning capabilities of LLMs to process and select the best-matching object proposals generated by VLMs like Grounding-DINO (GD) by converting VLM outputs (bounding boxes, labels, scores) into natural language prompts.", "questions": "- Can you explain in more detail how GroundingDino (GD) is used as zero-shot method for REC task, given that GD outputs bounding boxes that cover various objects in an image but cannot directly perform the reasoning needed to decide which box matches a complex query?\n\n- Is the number of fine-tuned parameters an important aspect in achieving better performance? How does Llama with 0.42% parameters tuned performs in comparison with the Llama version in the paper (with 4.2% parameters fine-tuned)?\n\n- Does the LLM always adhere to the generated bbox index?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel method, LLM-wrapper, for adapting VLMs in a black-box manner, which is an interesting way to sidestep the need for model-specific fine-tuning which involves access to model weights or architectures.\n- The paper compares LLM-wrapper with baseline zero-shot and fine-tuned VLMs across 3 different REC dataset variants.\n- LLM-wrapper can ensemble and transfer across different VLMs, which is an interesting future area of research.\n-  LLM-wrapper requires modest resources (e.g., a single 40GB-A100 GPU)\n- The paper includes thorough ablation studies on the number of training samples used for fine-tuning, comparing zero-shot LLM-wrapper with fine-tuned LLM-wrapper, showing the transferability of LLM-wrapper trained on one VLM's outputs to another VLM or ensembling VLMs, etc.\n- The paper shows improvements over zero-shot, but marginally closes some gap from finetuning. Still, there is significant room for improvement which is a good direction for future work.", "weaknesses": "- The method requires constructing new training data to fine-tune the LLM, which partially defeats the purpose of a black-box approach that aims to minimize resource and data preparation efforts. \n- The paper lacks comparisons with existing black-box adaptation methods or prompting strategies from the literature, e.g., prompting methods that were discussed in the related work section. \n- There is noticeable performance variability depending on the choice of LLM or more likely the number of parameters finetuned (e.g., Llama 3 4.2% vs. Mixtral 0.24%). A deeper analysis of why these differences occur is needed, e.g., by fine-tuning fewer parameters in Llama and comparing Llama 4.2% and Llama 0.42% fine-tuned parameters. \n- There is limited discussion on how well the LLM adheres to producing only the best box as the output and whether there are cases where the LLM generates unexpected or incorrect responses. More generally, the paper does not provide enough detail on failure cases where the LLM-wrapper may not perform well.\n- How far are results from SOTA performance on RefCOCO REC?\n\nMinor comment:\n- The paper is too verbose in many places, e.g., section 3.4 seems unnecessary as the benefits have been explained well in the previous sections. Instead, it would be nice to provide more analysis with SoTA baselines, more diverse qualitative examples, limitations, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6TLDDXEke", "reviewer": "Reviewer_f3st", "summary": "Fine-tuning VLMs requires users to access the architecture, weights, and gradients of models, which can be difficult due to model privacy. This paper focuses on exploring the \"black-box\" adaptation of VLMs on referring expression comprehension (REC), which means only calling VLMs through forward functions. They propose LLM-wrapper, which uses the reasoning ability of LLMs to select the best matching box generated by VLMs. \nIn LLM-wrapper, the VLMs first generate bounding box candidates and these boxes are transferred into text prompts including the box coordinates, labels and/or prediction scores. Next, these prompts are provided to fine-tune the LLMs using LoRA. Finally, these LLMs are capable of reasoning about the image using these prompts and selecting the best matching box from the prompts. In the experimental section, they conduct experiments on three REC benchmarks. For VLMs, they choose Grounding-Dino and Florence-2, and for LLMs they choose Mixtral and Llama3. With LLM-wrapper, they show the LLMs can outperform the zero-shot VLMs by a large margin. Moreover, they find that REC performances can be further boosted by ensembling outputs from different VLMs. They also show that LLM-wrapper can be fine-tuned on one VLM and transferred to another VLM.", "questions": "Are the results reported on different benchmarks coming from LLMs fine-tuned on corresponding training data? For example, in table 1, when the LLM-wrapper gets 78.12 on RefCOCOg val-umd, is the Llama3 fine-tuned using only the bounding boxes and images from RefCOCOg or is the Llama3 fine-tuned using samples coming from all three benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This approach does not need to access the original parameters of VLMs, but can still get benefits from them. In the case that the VLMs are closed-source, LLM-wrapper can be useful.\n2. Finetuning LLMs using LoRA reduces the number of training parameters. \n3. LLM-wrapper trained using one specific VLM can learn general spatial and semantic knowledge, and therefore can be used during inference with another VLM. For some expensive VLMs, with such knowledge transferring, it is not necessary to re-train another LLM-wrapper.", "weaknesses": "1. In table 3, for Grounding-Dino, white-box experiments are conducted using SwinB, but both zero-shot baselines and LLM-wrapper experiments are conducted using SwimT. It would be better if the authors could provide White-box results for SwinT as well.\n2. In table 5, the last two rows show the results on 300 samples, but for the first four rows, the results are reported on the whole val/test sets. I would like to see the improvements when transferring from GDrec to Flo2 and from Flo2 to GDrec on the same 300 samples. \n3. LLM-wrapper still uses the bounding box annotations. For REC, models with a similar number of parameters fine-tuned using the same amount of bounding boxes can even get better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8aq9p8u4ZV", "reviewer": "Reviewer_DAad", "summary": "This paper introduces LLM-wrapper, a novel method for enhancing the Referring Expression Comprehension (REC) capabilities of black-box VLMs without directly fine-tuning them. LLM-wrapper leverages a fine-tuned LLM to refine the bounding box proposals generated by a frozen, zero-shot VLM. Specifically, LLM-wrapper takes in a natural language description of each proposal along with spatial coordinates and the original referring expression, then outputs the proposal most relevant to the given expression as the final REC prediction. This black-box adaptation approach allows closed-source VLMs to be adapted without access to their internal structure or gradients, making it adaptable to any VLM and capable of integrating proposals from multiple VLMs to improve performance through ensembling.", "questions": "Please refer to weaknesses 4 and 5.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, making the proposed method easy to understand with clear motivation and steps.\n\n2. LLM-wrapper only needs to fine-tune the LLM once, allowing it to work with newer, more powerful VLMs without retraining, making it adaptable as VLM technology advances.\n\n3. By combining outputs from multiple VLMs, LLM-wrapper reaches performance levels that compete with some fully fine-tuned VLMs, showing its robustness.\n\n4. The method requires only a single GPU for fine-tuning, keeping training costs low and making it accessible.\n\n5. Black-box approach is ideal for closed-source VLMs, adding value in scenarios where model internals aren’t accessible.", "weaknesses": "1. The LLM-wrapper approach doubles the inference cost compared to existing VLMs.\n\n2. The effectiveness of the LLM-wrapper is highly dependent on the performance of the chosen VLM.\n\n3. Several recent VLMs, such as Ferret [1], KOSMOS-2 [2], and Qwen-VL [3], support both referring expression comprehension (REC) and grounding. The paper could be strengthened by including more baseline experiments that compare the original REC performance of these VLMs with their LLM-wrapped versions.\n\n4. In the LLM-wrapper paradigm, the number of entities in the referring expression likely influences the final performance. If the VLM can accurately predict all entities in a referring expression, further analysis is warranted on whether an increased number of entities aids or hinders the LLM's ability to ground the correct one. An ablation study examining the effect of entity count on performance would be valuable.\n\n5. Extending point 4, the experiments are conducted on the RefCOCO series, which contains relatively simple referring expressions (with a length of 3.6–8.4 words, as reported in the paper). This limitation suggests a confined number of entities per sentence. It would be insightful to evaluate the model’s performance on longer expressions with more entities, such as HC-RefLoCo benchmark, to determine how the LLM performs when faced with more complex captions and a greater number of entities.\n\n[1] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang. Ferret: Refer\nand ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.\n\n[2] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal\nlarge language models to the world. arXiv preprint arXiv:2306.14824, 2023.\n\n[3] Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qL1fyJbJaG", "reviewer": "Reviewer_o1yY", "summary": "This paper proposes a method called LLM-wrapper, aiming at ‘black-box’ adaptation of VLMs for the REC task using Large Language Models.", "questions": "please refer to above-mentioned.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is impressive.\n2. The paper is a well-structured format.", "weaknesses": "1. The performances of this method is limited. The performances of many REC methods such as [1], [2], [3] don’t rely on the LLM are better than the performances of this method.\n2. The experimental evaluation of this method is constrained. Additional dataset comparisons, such as Flickr30K entities and RefClef, should be taken into account. \n\n[1] Qiu, Heqian and Li, Hongliang and Zhao, Taijin and Wang, Lanxiao and Wu, Qingbo and Meng, Fanman, RefCrowd: Grounding the Target in Crowd with Referring Expressions, MM2022.\n\n[2] Zhao, Peizhi and Zheng, Shiyi and Zhao, Wenye and Xu, Dongsheng and Li, Pijian and Cai, Yi and Huang, Qingbao, Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point, AAAI2024\n\n[3] Jing, Chenchen and Wu, Yuwei and Pei, Mingtao and Hu, Yao and Jia, Yunde and Wu, Qi, Visual-Semantic Graph Matching for Visual Grounding, MM2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)