LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

## Abstract

Abstract Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires ‘white-box’ access to the model’s architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper , a method for ‘black-box’ adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile as it works with any VLM, it transfers to new VLMs, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. The code will be open-sourced.

## 1 Introduction

Vision Language Models (VLMs), a class of foundation models (Bommasani et al., ), trained on large-scale and diverse tasks and datasets, have shown remarkable abilities to solve various open-vocabulary tasks, as image captioning (Xiao et al., ; Li et al., ), visual question answering (Alayrac et al., ; Liu et al., ; Li et al., ), text-image retrieval (Radford et al., ; Zhai et al., ; Li et al., ), object detection (Liu et al., ; Xiao et al., ; Cheng et al., ), or semantic segmentation (Ding et al., ; Xiao et al., ).
Recent VLMs show promising zero-shot generalization abilities to new tasks and data domains (Wei et al., ; Alayrac et al., ). However, there is still a significant performance gap between zero-shot VLMs and those that have been specifically trained or adapted for a particular task and data domain.
This work focuses on the challenging open-vocabulary detection task of Referring Expression Comprehension (REC) (Mao et al., ), which involves localizing an object in an image based on a complex textual query, requiring both spatial and semantic reasoning.
While zero-shot VLMs can typically detect most objects mentioned in the query with reasonably accurate bounding boxes and labels, they struggle to identify *only* the described object. Moreover, VLMs often have difficulty understanding complex descriptions that involve relations between objects, attributes, or negations (Xie et al., ; Yao et al., ).

To improve performance, VLMs are typically fine-tuned on the specific REC task and corresponding datasets. This fine-tuning is usually done in a *‘white-box’* manner, with full access to the model’s architecture and weights for back-propagation.
However, this process requires expertise to design fine-tuning objectives and optimize hyper-parameters, specific to each VLM and downstream task.
Moreover, white-box fine-tuning is not always feasible. Some models are closed-source, either because they are proprietary and released behind APIs (Ren et al., ), or because they are trained on private data, making their weights and gradients inaccessible.
While companies may provide APIs for adapting proprietary models, e.g., (OpenAI, ), these solutions are limited to predefined scopes and require sharing data with external private companies, raising legal and privacy concerns.

To address these challenges, we explore *‘black-box’* adaptation of VLMs, where only forward calls to the model are possible.
We propose LLM-wrapper, a new method for the black-box adaptation of VLMs for the REC task, by using an LLM to reason on the VLM’s outputs.
This approach builds on the recent development of Large Language Models (LLMs) (Touvron et al., ; Jiang et al., ; Mesnard et al., ), which have shown interesting reasoning capabilities.
The underlying idea is that a zero-shot black-box VLM can generate high-quality labeled bounding boxes, and that LLM-wrapper can then leverage the semantic and spatial reasoning abilities of the LLM (Lian et al., ; ) to ‘reason’ on such outputs, further enhanced with a light fine-tuning.
As illustrated in Figure 1, our method involves translating VLM’s outputs into a natural language prompt and feeding it to the LLM. The LLM is then tasked with identifying the box that best matches the referring expression from the given candidates.

LLM-wrapper offers several advantages, summarized in Table 1.
First, since the adaptation is done in a black-box manner, it allows for the adaptation of closed-source models, without needing access to the model’s architecture, weights, and gradients.
This makes black-box fine-tuning versatile and easy to use, as it can be applied to any model without requiring specific model knowledge or assumptions about the model’s architecture.
Additionally, as the adaptation is delegated to the LLM, which learns the specifics of the task, the adaptation retains the pre-trained knowledge of the original VLM, and we find that LLM-wrapper transfers well to other VLMs and generalizes to model updates.
Finally, LLM-wrapper enables the adaptation of an ensemble of VLMs, thereby allowing to combine strengths from multiple models.

We evaluate LLM-wrapper on three datasets – RefCOCO (Yu et al., ), RefCOCO+ (Yu et al., ), RefCOCOg (Mao et al., ) – using two notably different VLMs – Grounding-DINO (Liu et al., ) and Florence-2 (Xiao et al., ). We also experiment with two LLMs for the adaptation: the recent Mixtral 8x7B (Jiang et al., ) and Llama 3 8B (Dubey et al., ).
While LLM-wrapper is not meant to outperform standard white-box fine-tuning, we show that LLM-wrapper significantly enhances the VLM’s performances, across all combinations of VLMs, LLMs and datasets, thus demonstrating the versatility of our method. Notably, on RefCOCOg, our most challenging dataset with respect to semantic understanding, LLM-wrapper improves the results of zero-shot VLMs with gains ranging from +9.5 up to +18.0 P@1.

**Table 1: Comparison between white-box fine-tuning and LLM-wrapper on the REC task.**
|  | Classic white-box fine-tuning | LLM-wrapper |
| --- | --- | --- |
| Model access | Needs complete access (loss, architecture, weights, gradients) | Only needs a forward call of a frozen black-box VLM |
| Fine-tuning knowledge | Specific for each VLM | Agnostic to the choice of VLM and LLM, and parameter-efficient |
| Specificity of adaptation | None | Leverages LLM semantic reasoning |
| Fine-tuning generalization | Cannot directly transfer to nor ensemble other models’ outputs | Transfers well from one VLM to another and can ensemble outputs |
| Expected performances | Best results | Good results |

## 2 Related work

##### Referring Expression Comprehension (REC) and Vision Language Models (VLMs).

Referring Expression Comprehension (REC) (Kazemzadeh et al., ; Mao et al., ; Qiao et al., ) is the task of identifying objects in an image based on referring expressions. Typically, REC involves selecting the best region from a set of region proposals extracted from the image, guided by a referring expression. This task is challenging because the referring expression can range from a short phrase (Kazemzadeh et al., ; Yu et al., ) to a long sentence that may require multi-step reasoning (Mao et al., ).
REC is distinct from similar tasks such as visual grounding (Rohrbach et al., ), where multiple object regions described by multiple noun phrases must be localized in an image. It also differs from object detection (Girshick, ), which uses predefined categories instead of natural language expressions.
The recent surge of interest in VLMs, starting from CLIP (Radford et al., ) to the recent Florence-2 (Xiao et al., ) has significantly improved performance on tasks requiring both vision and language. For REC, several approaches have used VLMs. For instance, Grounding-DINO (Liu et al., ) relies on multiple stages of modality fusion to align visual and textual features. The more recent Florence-2 (Xiao et al., ) is a sequence-to-sequence model trained on a huge collection of data.
However, despite being trained on extensive amounts of image-text data (e.g., 126 million images with 500 million to 3.6 billion annotations for Florence-2), these models’ zero-shot performances on REC are sub-optimal compared to their performances when including REC data in their training set or when fine-tuned (see Section 4.2).
Our approach, LLM-wrapper, addresses this clear limitation in a black-box manner, without the need to re-train the VLM with task-specific data.

##### VLM adaptation.

While retraining the entire VLM on a new task or data domain is computationally expensive (Liu et al., ; Wang et al., ; Chen et al., ; Xiao et al., ), fine-tuning offers a more efficient alternative. However, even fine-tuning can be costly.
To address this, parameter-efficient fine-tuning, e.g., LoRa (Hu et al., ), DoRa (Liu et al., ), or VeRa (Kopiczko et al., ), and soft prompt (Li & Liang, ) learning have been proposed. These methods avoid updating the model’s pre-trained weights but require access to the model’s architecture and gradients.

A few recent methods propose strategies to adapt VLMs when back-propagation through the VLM is not feasible (Ouali et al., ; Yu et al., ; Liu et al., ; Oh et al., ). Both Ouali et al. () and Yu et al. () focus on CLIP-based methods and adapt the VLM by learning either a feature projection or soft prompts. They however require access to the inner representations, which is generally not possible with APIs. Liu et al. () optimize the input prompt template, but also focuses on CLIP-based models. Instead, we target VLMs that perform open-vocabulary detection, e.g., Grounding-DINO and Florence-2. Finally, (Oh et al., ) adapt a VLM in a black-box manner by modifying the image input. However, this method is designed to address visual domain gaps and is not directly applicable to adapt a VLM to a new *task*. Moreover, it requires multiple API calls for each training image.
In this work, we propose a strategy to adapt VLMs, and more specifically open-vocabulary detectors, to a new task, in a complete black-box fashion.
Our method does not require access to the model’s intermediate representations or gradients, runs a single API call for each (text-image) input pair, and enables adaptation to a new task (REC in our case) that the VLM was not originally trained for.

Figure: Figure 1: Illustration of LLM-wrapper. Our method adapts a black-box VLM for the REC task. The output of the VLM is translated into a natural language text that is used to prompt an LLM. The LLM is then tasked with identifying the box that best matches the referring expression among the given candidates. The LLM must learn to identify the subject of the query and to disambiguate the correct object from other distractor objects (e.g., several plates with ‘flower details’).
Refer to caption: x1.png

## 3 LLM-wrapper : black-box semantic-aware adaptation of VLMs.

In this section, we present LLM-wrapper, a novel LLM-based approach to adapt an open-ended VLM for the REC task. We present the general idea in Section 3.1, the construction of the prompt for the LLM in Section 3.2, the fine-tuning details in Section 3.3. Lastly, we present the benefits of having a language-based adaptation of VLMs’ outputs in Section 3.4.

### 3.1 General idea

Our method *wraps* the open-vocabulary detections of a frozen black-box VLM with an LLM that reasons over these outputs.
An overview is presented in Figure 1.
Given a complex textual query, LLM-wrapper leverages the fact that detection-oriented VLMs can typically localize well most nouns of the query, even if they struggle with the reasoning step required to precisely select the object of interest among several distractors.
Therefore, LLM-wrapper delegates the reasoning task to an LLM, which has interesting abilities to handle difficult text queries, including attributes, negation, and relational or spatial descriptions of objects.
As the gradients of the black-box VLM are not accessible, we propose to adapt its *outputs* with the LLM. It gives us access to the LLM gradients and we can thus specialize the LLM for the task, with a simple and light fine-tuning, performed on the outputs of the VLM to learn to select the right box among them.
Overall, LLM-wrapper only requires black-box access to the VLM, whereas standard fine-tuning strategies need white-box access to perform back-propagation. This makes our approach more flexible and applicable in scenarios where the internal workings of the VLM are not accessible.

### 3.2 Prompt construction

The key idea of our method is to convert the VLM’s outputs into natural language. To achieve this, we list all predicted outputs in the LLM prompt, including their box coordinates, labels, and, when applicable, prediction scores (displayed below in light gray). This allows the ‘blind’ LLM, which only reads text, to understand the scene and reason about the image. The prompt then reminds the query and asks the LLM to select the best matching box.

For instance, given the query (in green in Figure 1), and the associated outputs (e.g., ‘flower’, ‘plate with flower details’, ‘yellow bowl’, etc.), we ask the LLM for the best matching box index, as follows:

### 3.3 Fine-tuning the LLM

While a zero-shot LLM can already reason on the new task to some extent, we find that fine-tuning the LLM significantly improves performances on the task.
Therefore, we fine-tune the LLM for prompt completion with a cross-entropy loss, using the prompt described above. Specifically, the expected answer for the LLM is the index of the best box proposal, corresponding to the closest match to the known ground truth box.
To build the training dataset, we use the REC training data, which consists of (image, query) pairs and ground truth boxes. The detection outputs (boxes, labels, scores), used to create the training prompts, are inferred using the VLM being adapted. We only keep the samples where at least one of the VLM box proposals has an Intersection over Union (IoU) with the ground truth box higher than 0.5, ensuring no noisy samples.
To make LLM-wrapper robust to any shortcut learning based on the box order, we randomly permute the order of the box proposals in the prompt during training.

We leverage the extensive literature on LLM fine-tuning to specialize the LLM of LLM-wrapper.
Specifically, we use LoRA (Hu et al., ), which introduces additive updates to the model’s activations, parameterized by low-rank modules.
Doing so reduces the number of new parameters to learn while preserving the LLM’s general knowledge.
We also use flash attention (Dao et al., ) and 4-bit quantization (Dettmers et al., ), making LLM-wrapper trainable on a single 40GB-A100 GPU in less than 7 hours. This approach makes the training efficient in terms of compute and very simple to implement in practice. Overall, we find that LLM-wrapper is not very sensitive to the choice of the few hyper-parameters introduced (see Section 4.4).

### 3.4 Benefits of wrapping VLMs’ outputs

Our approach aims at adapting VLMs in a black-box manner.
We convert their outputs into natural language and wrap them with a language model, which has several benefits.
First, by working on the outputs of the VLM rather than its inputs, we can back-propagate into the LLM.
Second, during fine-tuning, LLM-wrapper learns general knowledge about the task at hand. This allows us to transfer LLM-wrapper from one VLM to another. For example, we can leverage the updated versions of models available via API, such as moving from Grounding-DINO to Grounding-DINO-1.5, without retraining the language model (see Section 4.3).
Finally, since the input format of LLM-wrapper is free text, the method can easily adapt to a varying number of input bounding boxes. For instance, our method can naturally extend to an ensemble of multiple VLMs (see Section 4.3). We observe that this ‘ensembling’ approach is particularly useful to combine the VLMs’ strengths and mitigate their different biases. Overall, the input format of our method makes it flexible for new use cases.

## 4 Experiments

In this section we present the experimental validation of LLM-wrapper. We first state our protocol in Section 4.1. In Section 4.2, we show how LLM-wrapper improves VLMs. We then conduct further analysis of LLM-wrapper’s benefits in Section 4.3. We finally conduct ablation studies showcasing LLM-wrapper’s robustness in Section 4.4.

### 4.1 Experimental setup and technical details

#### 4.1.1 Referring Expression Comprehension (REC) task

Datasets. We evaluate LLM-wrapper on the REC task. In REC, given an input pair (image, query), a model is expected to predict a single bounding box around the object described in the query, as illustrated in Figure 1. We use three standard datasets for REC, RefCOCO (Yu et al., ), RefCOCO+ (Yu et al., ), and RefCOCOg (Mao et al., ) which is the most challenging as it contains longer and more complex descriptions (8.4 words per query in average). All of these datasets contain multiple distractor objects in each image for the given text query. Dataset’s statistics are reported in Table 2.

**Table 2: REC datasets statistics.**
|  | Split | Size | # words |  |  |
| --- | --- | --- | --- | --- | --- |
| Dataset | used | train | val | test | / query |
| RefCOCO | unc | 120,624 | 10,834 | 10,752 | 3.6 |
| RefCOCO+ | unc | 120,191 | 10,758 | 10,615 | 3.5 |
| RefCOCOg | umd | 80,512 | 4,896 | 9,602 | 8.4 |

Metric. We measure performance with the standard precision@1 (P@1) metric, described in (Qiao et al., ). A true positive is defined when the predicted box has an Intersection-Over-Union (IoU) greater than $0.5$ with the ground truth box. The metric is averaged over the evaluation set.

#### 4.1.2 The VLMs

We evaluate the impact of LLM-wrapper on two different VLMs.
In all cases, we use the official model checkpoints and unless specified otherwise, we use the model versions that are not fine-tuned for the REC task, meaning that the models have not been exposed to any of the RefCOCO/+/g datasets.
Using the model setup described below, we feed from 2 to 20 boxes to LLM-wrapper per prompt.

Grounding-DINO (​GD)
(Liu et al., ) aligns visual queries with text through stages of modality fusion and contrastive learning. Initially designed for open-vocabulary detection tasks, the model produces 900 bounding boxes, achieving a high recall but an under-performing precision on the REC task.
We propose a straightforward improvement for REC, Grounding-DINO-REC (​GDrec), in which instead of assigning to each box the maximum similarity score with respect to any query word as confidence score, we assign the similarity score corresponding to the query’s subject(^1^11The subject is defined as the first noun group identified by Spacy’s dependency parser (Honnibal et al., ).).
​GDrec significantly outperforms ​GD on all datasets, and particularly on RefCOCOg, where the noun identification is more challenging, with e.g., +7 P@1 for zero-shot models as shown in Table 3.
To ensure a rather short prompt, we limit the number of box proposals by setting the box confidence score threshold to $0.15$ for ​GDrec and to $0.2$ for ​GD.
​GD / ​GDrec, not fine-tuned, use a SwinT(T) backbone while the fine-tuned versions are based on SwinB(B), as they are the only publicly available models.
We also run experiments on a subset of RefCOCOg using Grounding-DINO 1.5 (Ren et al., ) (GD-1.5), a recent detector behind API (online at: [Grounding-DINO-1.5-API](https://github.com/IDEA-Research/Grounding-DINO-1.5-API)), which provides 300 free API calls. GD-1.5 extends ​GD with a larger backbone, ViT-L (Fang et al., ), and training dataset.

Florence-2 (Flo2) (Xiao et al., ) is a sequence-to-sequence multi-task model. We use the Florence-2 Large version from the Hugging Face Hub. It is composed of a DaViT vision encoder (Ding et al., ) and a multi-modal encoder-decoder. Flo2 can be prompted for several tasks, and we keep and concatenate the boxes from the ‘open vocabulary detection’ and the ‘phrase grounding’ task modes.

#### 4.1.3 The LLMs and their fine-tuning

Our main experiments are conducted using two different LLMs: Mixtral 8x7B Instruct (Jiang et al., ) (v0.1) and Llama 3 8B Instruct (AI@Meta, ) with Hugging Face’s implementation. We use Hugging Face’s supervised fine-tuning pipeline (SFT) (HuggingFace, ), that allows to implement the training choices discussed in Section 3.
Specifically, for Llama 3 8B, we train 352M parameters, which is 4.20% of the original model size. For Mixtral 8x7B, we train 114M parameters, which is 0.24% of the original size.
To further study the impact of the LLM scale, we also experiment with two additional families of models, Gemma 2 (Gemma, ) and GPT-Neo (Gao et al., ).
The latter is a class of LLMs, based on a replication of the GPT-3 architecture, trained on the large curated Pile dataset (Gao et al., ), and which was not ‘instructed’ (Ouyang et al., ).
We train LLM-wrapper with Adam (Kingma, ), with a batch-size of four, until convergence. We discuss in Section 4.4 the sample efficiency of LLM-wrapper.
Unless stated otherwise, we use a learning rate of $10^{-5}$ and a rank of $r=128$ for LoRA. These hyper-parameters work consistently across three datasets, two LLMs and two VLMs, demonstrating the robustness of LLM-wrapper. We study the performance robustness of LLM-wrapper for different hyper-parameters in Section 4.4.

### 4.2 LLM-wrapper can adapt VLMs to the REC task

**Table 3: Main results of LLM-wrapper on the REC task, in P@1$\uparrow$, on RefCOCO/+/g datasets. ‘(T)’ and ‘(B)’ stand for the ‘SwinT’ and ‘SwinB’ backbones respectively.**
|  | Model |  |  | RefCOCOg | RefCOCO | RefCOCO+ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Adaptation | access | VLM | LLM | val-umd | test-umd | val-unc | test-unc | val-unc | test-unc |
| $\emptyset$ (zero-shot) |  | ​GD(T) |  | 60.09 | 59.32 | 50.69 | 50.94 | 51.65 | 51.79 |
| *Fine-tuning* | White-box | *​GD(B)* |  | 78.51 | 77.99 | *83.86* | *84.12* | *73.46* | *73.46* |
| LLM-wrapper | Black-box | ​GD(T) | Mixtral | 77.57 $\uparrow$17.5 | 77.05 $\uparrow$17.7 | 74.61$\uparrow$23.9 | 73.46$\uparrow$22.5 | 60.32$\uparrow$8.7 | 60.08$\uparrow$8.3 |
| LLM-wrapper | Black-box | ​GD(T) | Llama3 | 78.12 $\uparrow$18.0 | 77.36 $\uparrow$18.0 | 74.78$\uparrow$24.1 | 73.98$\uparrow$23.0 | 64.18$\uparrow$12.5 | 63.82$\uparrow$12.0 |
| $\emptyset$ (zero-shot) |  | ​GDrec(T) |  | 67.61 | 68.37 | 51.82 | 52.12 | 53.28 | 53.16 |
| *Fine-tuning* | *White-box* | *​GDrec(B)* |  | 80.19 | 79.85 | *83.84* | *84.21* | *73.68* | *73.58* |
| LLM-wrapper | Black-box | ​GDrec(T) | Mixtral | 78.47 $\uparrow$10.9 | 77.92 $\uparrow$9.6 | 72.61$\uparrow$20.8 | 71.48$\uparrow$19.4 | 63.79$\uparrow$10.5 | 63.69$\uparrow$10.5 |
| LLM-wrapper | Black-box | ​GDrec(T) | Llama3 | 78.25 $\uparrow$10.6 | 78.01 $\uparrow$9.6 | 73.97$\uparrow$22.2 | 73.07$\uparrow$21.0 | 64.13$\uparrow$10.9 | 64.08$\uparrow$10.9 |
| $\emptyset$ (zero-shot) |  | Flo2 |  | 68.28 | 66.90 | 56.32 | 57.01 | 53.71 | 54.43 |
| *Fine-tuning* | *White-box* | *Flo2* |  | 90.32 | 91.06 | 93.08 | 93.42 | 88.20 | 88.49 |
| LLM-wrapper | Black-box | Flo2 | Mixtral | 77.74 $\uparrow$9.5 | 76.98 $\uparrow$10.1 | 67.29$\uparrow$11.0 | 66.62$\uparrow$9.6 | 57.38$\uparrow$3.7 | 57.48$\uparrow$3.1 |
| LLM-wrapper | Black-box | Flo2 | Llama3 | 77.94 $\uparrow$9.7 | 77.15 $\uparrow$10.3 | 70.19$\uparrow$13.9 | 70.00$\uparrow$13.0 | 61.95$\uparrow$8.2 | 61.87$\uparrow$7.4 |

In Table 3, we report the performances of VLMs on the REC task in three settings: zero-shot off-the-shelf VLM, after classic white-box fine-tuning, and after black-box adaptation with LLM-wrapper.

Our first observation confirms that while VLMs demonstrate remarkable zero-shot performance on new tasks and datasets, their performance is still significantly lower than models specifically adapted to the given task (white-box fine-tuning). As shown in Table 3, for both ​GD (and its variant ​GDrec) and Flo2, there is a notable gap in REC performance, ranging from -11.5 to -37 P@1, between the zero-shot and fine-tuned versions.

Interestingly, we observe that for all combinations of VLMs and LLMs, LLM-wrapper can adapt VLMs to the new REC task, despite the black-box setting.
For instance, on RefCOCOg, LLM-wrapper brings improvements over the zero-shot models ranging from +9.5 to +18 P@1. Notably, while our proposed variant ​GDrec outperforms ​GD by large margins in a zero-shot setting (e.g., +9.05 P@1-test) due to better subject identification, both models perform similarly when adapted with LLM-wrapper. This shows that LLM-wrapper particularly improves VLMs that lack abilities useful for the task, such as subject identification in the case of REC for ​GD.
On RefCOCO, LLM-wrapper improves again by significant margins zero-shot VLMs, e.g., +23.0 P@1 (test) for ​GD with Llama 3 8B.
Given that textual queries in RefCOCO are very short (3.6 words on average), subject identification is relatively easy for these queries. Therefore, these gains indicate that LLM-wrapper greatly helps to disambiguate the object of interest from the distractors.
On RefCOCO+, the improvement brought by LLM-wrapper is lower but still significant (between +3.1 and +12.5 P@1). This is expected as RefCOCO+ is designed to exclude location words, and most referring expressions are thus purely appearance-based descriptions (Qiao et al., ) where LLM-wrapper brings less value for adaptation.

We present some qualitative results in Figure 2, where we show predictions from Flo2 before and after being adapted with LLM-wrapper, with Llama 3 8B.
We observe that the adaptation with LLM-wrapper enables better spatial understanding (Fig. 2(a), Fig. 2(d)), subject identification (Fig. 2(b),
Fig. 2(e)) and relational reasoning (Fig. 2(b), Fig. 2(c),
Fig. 2(e)).

Finally, while LLM-wrapper is not intended to outperform classic white-box fine-tuning, it can approach its performance in some settings (e.g., with ​GD and ​GDrec on RefCOCOg, despite a smaller vision backbone for LLM-wrapper).
In other setups, a gap remains, as fine-tuned models specifically train their vision encoder for the task.
Overall, we demonstrate that LLM-wrapper significantly improves VLMs’ zero-shot performance in a black-box manner, consistently across all different VLMs and LLMs.

Figure: (a) “The front most cow to the right of other cows”
Refer to caption: x2.png

### 4.3 Benefits of wrapping VLMs’ outputs with LLM-wrapper

**Table 4: Results of VLMs ensembling using LLM-wrapper with Llama 3 8B, in P@1$\uparrow$ on RefCOCOg using ‘umd’ splits. (Comparable findings for Mixtral).**
| Adaptation | VLM | P@1 - val $\uparrow$ | P@1 - test $\uparrow$ |
| --- | --- | --- | --- |
| $\emptyset$ (zero-shot VLM) | ​GDrec | 67.61 | 68.37 |
| $\emptyset$ (zero-shot VLM) | Flo2 | 68.28 | 66.90 |
| LLM-wrapper | Flo2 | 77.94 | 77.15 |
| LLM-wrapper | ​GDrec | 78.25 | 78.01 |
| LLM-wrapper | Flo2 + ​GDrec | 81.25 | 80.13 |

Ensembling VLMs.
One advantage of LLM-wrapper is its free-text input format, which allows to adapt to various inputs. Indeed, we show that LLM-wrapper can learn to *ensemble* outputs from different VLMs as shown in Table 4.
Specifically, we concatenate the predictions of the two VLMs in the prompt described in Section 3. The results show that when ensembling ​GDrec and Flo2, scores are boosted by +3.0 P@1 (val-umd) and +2.1 P@1 (test-umd) when compared to those obtained with the best-performing VLM adapted with LLM-wrapper, namely ​GDrec. This demonstrates that LLM-wrapper is capable of reasoning on multiple sources and leveraging the strengths of different models.

Indeed, we observe that Flo2 has a high precision, while ​GD and ​GDrec have lower precision but high recall.
We show qualitatively in Figure 3 an example where LLM-wrapper leverages the complementarity of the two models.
The figure displays predictions from Flo2 (e), ​GDrec (f), and LLM-wrapper
ensembling predictions from both Flo2 and ​GDrec (d).
We observe that while Flo2 fails to detect the target object as a possible candidate (b), the additional proposals from ​GDrec (c) enable LLM-wrapper to find the correct object (d) that each independent models missed.

Figure: (a) Ground truth
Refer to caption: x3.png

**Table 5: Results of LLM-wrapper when using different VLMs’ outputs during fine-tuning and inference, in P@1$\uparrow$ on RefCOCOg using ‘umd’ splits. Results obtained with Llama 3 8B. Comparable findings for Mixtral. ^†Scores obtained on a subset of $300$ samples from RefCOCOg val-umd.**
| Adaptation | VLM (fine-tuning) | VLM (inference) | P@1 - val $\uparrow$ | P@1 - test $\uparrow$ |
| --- | --- | --- | --- | --- |
| $\emptyset$ (zero-shot VLM) | $\emptyset$ | ​GDrec | 67.61 | 68.37 |
| LLM-wrapper | Flo2 | ​GDrec | 73.90  $\uparrow$6.3 | 73.45 $\uparrow$5.1 |
| $\emptyset$ (zero-shot VLM) | $\emptyset$ | Flo2 | 68.28 | 66.90 |
| LLM-wrapper | ​GDrec | Flo2 | 73.86  $\uparrow$5.6 | 73.03 $\uparrow$6.1 |
| $\emptyset$ (zero-shot VLM) | $\emptyset$ | GD-1.5 | 47.67^† | — |
| LLM-wrapper | ​GDrec | GD-1.5 | 76.67^† $\uparrow$29.0 | — |

Transferring a trained LLM-wrapper to a new VLM.
Another advantage of using text as the input format is that LLM-wrapper does not rely on model-specific activation values, making it transferable from one VLM to another. For instance, LLM-wrapper can be fine-tuned on Flo2 and transferred to ​GD or ​GDrec. This is illustrated in Table 5, where, for instance, when fine-tuned on ​GDrec’s or Flo2’s outputs, transferring it at inference time to the other model’s outputs gives an increase from +5.6 to +6.3 P@1 over zero-shot VLMs.
This shows that during fine-tuning, LLM-wrapper learns spatial and semantic notions that generalize to other models. This capacity of LLM-wrapper is particularly useful for private models, such as GD-1.5 (Ren et al., ), where creating the training set can be expensive – for instance, getting predictions for RefCOCOg train would cost $\approx$ $1,600 ($20 per 1,000 API calls).
To illustrate this use case, we use a RefCOCOg val subset corresponding to 300 free API calls to GD-1.5.
When LLM-wrapper is fine-tuned on ​GDrec’s outputs and applied on GD-1.5’s outputs for inference, results are boosted by a significant +29.0 P@1. This showcases how LLM-wrapper can be used on continuously updated versions of models without the need to re-train it with each model update.

### 4.4 Ablation studies

Unless specified otherwise, we run ablations on RefCOCOg val-umd, using ​GD’s outputs and Llama 3 8B.

Impact of LLM scale.
In 4(a), we report the P@1 REC performance ($y$-axis) for various LLM sizes ($x$-axis).
We observe that LLM-wrapper is effective across all model families and that the gain in performance positively correlates with the LLM size.
Indeed, though the sub-billion model, GPT-Neo 139M, slightly boosts the P@1 performance of ​GD zero-shot (+2.2 P@1), this increase is only incremental when compared to Llama 3 8B’s gains (+18 P@1).
This ablation study demonstrates that using a highly-capable LLM is beneficial, and that smaller or less semantically rich models do not perform as well.

Robustness to hyper-parameters. In 4(b), we show that LLM-wrapper is not overly sensitive to the value of the LoRA rank. In the explored range ($r\in\{64,128,192\}$), the performance varies by only $\pm$0.4 P@1.
Similarly, we observe in 4(c) that LLM-wrapper is loosely sensitive to the learning rate choice within the range $\{10^{-6},10^{-5}\}$. With learning rate values higher that $10^{-5}$, we observe unstable training behaviors.

Figure: (a) LLM size.
Refer to caption: x9.png

Ablation on the number of training samples.
Figure 5 displays the P@1 REC performance on RefCOCOg val with respect to the number of training samples, when fine-tuning Llama 3 8B on RefCOCOg train.
It shows a sharp increase in P@1 for all VLMs (most impressive for ​GD) during the first $30$k samples, which takes $\sim$2h of training in our setting.
Thus, even with a restricted amount of samples, LLM-wrapper can boost performances.
In the extreme case of no training samples for the LLM, i.e., a zero-shot LLM, we observe on Table 6 that in most settings the VLM and the VLM + zero-shot LLM have very similar results. This follows the observation that, while LLMs have an extensive general knowledge, they may lack off-the-shelf reasoning (Kazemi et al., ; Fu et al., ).

Figure: Figure 5: Performance (P@1) of LLM-wrapper on RefCOCOg (val) with respect to the number of training samples. We fine-tune Llama 3 8B on RefCOCOg (train).

|  | P@1-val$\uparrow$ | P@1-test$\uparrow$ |  |  |
| --- | --- | --- | --- | --- |
| VLM | VLM<br>only | + LLM<br>zero-shot | VLM<br>only | + LLM<br>zero-shot |
| GD | 60.09 | 58.05 | 59.32 | 58.47 |
| GDrec | 67.61 | 67.48 | 68.37 | 68.17 |
| GD-1.5 | 47.67^† | 59.00^† | — | — |
| Flo2 | 68.28 | 67.69 | 66.90 | 66.44 |

Table: Table 6: Results of wrapping the VLM’s outputs with a zero-shot LLM. ‘VLM only’ are the scores of the VLM without any adaptation, and ‘+ LLM zero-shot’ corresponds to LLM-wrapper without any fine-tuning of the LLM (Llama 3 8B here). ^† see Table 5.

## 5 Conclusion

This work introduces LLM-wrapper, a simple approach for the black-box adaptation of VLMs to the REC task, that leverages an LLM to reason on VLMs’ outputs, translated into natural language.
We demonstrate that LLM-wrapper significantly boosts the performance of VLMs, for several combinations of LLMs and VLMs, and, in some settings, even bridges the gap to classic fine-tuning.
We also show how LLM-wrapper can ensemble predictions from different VLMs to leverage their respective strengths and how it can transfer across VLMs.
Thanks to efficient and well-studied methods for LLM fine-tuning, LLM-wrapper is simple to use in practice, requiring limited hyper-parameter tuning, and computationally efficient.

Future works include relying on fewer examples to fine-tune LLM-wrapper, and applying LLM-wrapper to different tasks, such as text-video retrieval (Fang et al., ).
We also aim to address a limitation of our method where the information on bounding boxes is not sufficient to identify referring expressions that require additional visual cues.
This is illustrated in Fig. 2(f), where localizing correctly the query requires knowing the direction of the planes to distinguish the ‘White plane in the front’.
Accordingly, a promising direction is to use the LLM to provide feedback regarding the visual cues that are missing to identify the referring expression and to suggest augmented queries for the VLM to disambiguate such cases.

#### Acknowledgments

We thank Pr. Eric Gaussier for his support and insightful feedbacks on this project. This work was performed using HPC resources from GENCI–IDRIS (Grant 2024-AD011014446) and with the support of the ANR MultiTrans (ANR-21-CE23-0032).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can you explain in more detail how GroundingDino (GD) is used as zero-shot method for REC task, given that GD outputs bounding boxes that cover various objects in an image but cannot directly perform the reasoning needed to decide which box matches a complex query?

- Is the number of fine-tuned parameters an important aspect in achieving better performance? How does Llama with 0.42% parameters tuned performs in comparison with the Llama version in the paper (with 4.2% parameters fine-tuned)?

- Does the LLM always adhere to the generated bbox index?
soundness: 3
strengths: - The paper introduces a novel method, LLM-wrapper, for adapting VLMs in a black-box manner, which is an interesting way to sidestep the need for model-specific fine-tuning which involves access to model weights or architectures.
- The paper compares LLM-wrapper with baseline zero-shot and fine-tuned VLMs across 3 different REC dataset variants.
- LLM-wrapper can ensemble and transfer across different VLMs, which is an interesting future area of research.
-  LLM-wrapper requires modest resources (e.g., a single 40GB-A100 GPU)
- The paper includes thorough ablation studies on the number of training samples used for fine-tuning, comparing zero-shot LLM-wrapper with fine-tuned LLM-wrapper, showing the transferability of LLM-wrapper trained on one VLM's outputs to another VLM or ensembling VLMs, etc.
- The paper shows improvements over zero-shot, but marginally closes some gap from finetuning. Still, there is significant room for improvement which is a good direction for future work.
summary: The paper presents LLM-wrapper, a method for black-box adaptation of VLMs for the Referring Expression Comprehension (REC) task. It leverages the reasoning capabilities of LLMs to process and select the best-matching object proposals generated by VLMs like Grounding-DINO (GD) by converting VLM outputs (bounding boxes, labels, scores) into natural language prompts.
weaknesses: - The method requires constructing new training data to fine-tune the LLM, which partially defeats the purpose of a black-box approach that aims to minimize resource and data preparation efforts. 
- The paper lacks comparisons with existing black-box adaptation methods or prompting strategies from the literature, e.g., prompting methods that were discussed in the related work section. 
- There is noticeable performance variability depending on the choice of LLM or more likely the number of parameters finetuned (e.g., Llama 3 4.2% vs. Mixtral 0.24%). A deeper analysis of why these differences occur is needed, e.g., by fine-tuning fewer parameters in Llama and comparing Llama 4.2% and Llama 0.42% fine-tuned parameters. 
- There is limited discussion on how well the LLM adheres to producing only the best box as the output and whether there are cases where the LLM generates unexpected or incorrect responses. More generally, the paper does not provide enough detail on failure cases where the LLM-wrapper may not perform well.
- How far are results from SOTA performance on RefCOCO REC?

Minor comment:
- The paper is too verbose in many places, e.g., section 3.4 seems unnecessary as the benefits have been explained well in the previous sections. Instead, it would be nice to provide more analysis with SoTA baselines, more diverse qualitative examples, limitations, etc.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Are the results reported on different benchmarks coming from LLMs fine-tuned on corresponding training data? For example, in table 1, when the LLM-wrapper gets 78.12 on RefCOCOg val-umd, is the Llama3 fine-tuned using only the bounding boxes and images from RefCOCOg or is the Llama3 fine-tuned using samples coming from all three benchmarks?
soundness: 2
strengths: 1. This approach does not need to access the original parameters of VLMs, but can still get benefits from them. In the case that the VLMs are closed-source, LLM-wrapper can be useful.
2. Finetuning LLMs using LoRA reduces the number of training parameters. 
3. LLM-wrapper trained using one specific VLM can learn general spatial and semantic knowledge, and therefore can be used during inference with another VLM. For some expensive VLMs, with such knowledge transferring, it is not necessary to re-train another LLM-wrapper.
summary: Fine-tuning VLMs requires users to access the architecture, weights, and gradients of models, which can be difficult due to model privacy. This paper focuses on exploring the "black-box" adaptation of VLMs on referring expression comprehension (REC), which means only calling VLMs through forward functions. They propose LLM-wrapper, which uses the reasoning ability of LLMs to select the best matching box generated by VLMs. 
In LLM-wrapper, the VLMs first generate bounding box candidates and these boxes are transferred into text prompts including the box coordinates, labels and/or prediction scores. Next, these prompts are provided to fine-tune the LLMs using LoRA. Finally, these LLMs are capable of reasoning about the image using these prompts and selecting the best matching box from the prompts. In the experimental section, they conduct experiments on three REC benchmarks. For VLMs, they choose Grounding-Dino and Florence-2, and for LLMs they choose Mixtral and Llama3. With LLM-wrapper, they show the LLMs can outperform the zero-shot VLMs by a large margin. Moreover, they find that REC performances can be further boosted by ensembling outputs from different VLMs. They also show that LLM-wrapper can be fine-tuned on one VLM and transferred to another VLM.
weaknesses: 1. In table 3, for Grounding-Dino, white-box experiments are conducted using SwinB, but both zero-shot baselines and LLM-wrapper experiments are conducted using SwimT. It would be better if the authors could provide White-box results for SwinT as well.
2. In table 5, the last two rows show the results on 300 samples, but for the first four rows, the results are reported on the whole val/test sets. I would like to see the improvements when transferring from GDrec to Flo2 and from Flo2 to GDrec on the same 300 samples. 
3. LLM-wrapper still uses the bounding box annotations. For REC, models with a similar number of parameters fine-tuned using the same amount of bounding boxes can even get better results.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to weaknesses 4 and 5.
soundness: 3
strengths: 1. The paper is well-written, making the proposed method easy to understand with clear motivation and steps.

2. LLM-wrapper only needs to fine-tune the LLM once, allowing it to work with newer, more powerful VLMs without retraining, making it adaptable as VLM technology advances.

3. By combining outputs from multiple VLMs, LLM-wrapper reaches performance levels that compete with some fully fine-tuned VLMs, showing its robustness.

4. The method requires only a single GPU for fine-tuning, keeping training costs low and making it accessible.

5. Black-box approach is ideal for closed-source VLMs, adding value in scenarios where model internals aren’t accessible.
summary: This paper introduces LLM-wrapper, a novel method for enhancing the Referring Expression Comprehension (REC) capabilities of black-box VLMs without directly fine-tuning them. LLM-wrapper leverages a fine-tuned LLM to refine the bounding box proposals generated by a frozen, zero-shot VLM. Specifically, LLM-wrapper takes in a natural language description of each proposal along with spatial coordinates and the original referring expression, then outputs the proposal most relevant to the given expression as the final REC prediction. This black-box adaptation approach allows closed-source VLMs to be adapted without access to their internal structure or gradients, making it adaptable to any VLM and capable of integrating proposals from multiple VLMs to improve performance through ensembling.
weaknesses: 1. The LLM-wrapper approach doubles the inference cost compared to existing VLMs.

2. The effectiveness of the LLM-wrapper is highly dependent on the performance of the chosen VLM.

3. Several recent VLMs, such as Ferret [1], KOSMOS-2 [2], and Qwen-VL [3], support both referring expression comprehension (REC) and grounding. The paper could be strengthened by including more baseline experiments that compare the original REC performance of these VLMs with their LLM-wrapped versions.

4. In the LLM-wrapper paradigm, the number of entities in the referring expression likely influences the final performance. If the VLM can accurately predict all entities in a referring expression, further analysis is warranted on whether an increased number of entities aids or hinders the LLM's ability to ground the correct one. An ablation study examining the effect of entity count on performance would be valuable.

5. Extending point 4, the experiments are conducted on the RefCOCO series, which contains relatively simple referring expressions (with a length of 3.6–8.4 words, as reported in the paper). This limitation suggests a confined number of entities per sentence. It would be insightful to evaluate the model’s performance on longer expressions with more entities, such as HC-RefLoCo benchmark, to determine how the LLM performs when faced with more complex captions and a greater number of entities.

[1] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang. Ferret: Refer
and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.

[2] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal
large language models to the world. arXiv preprint arXiv:2306.14824, 2023.

[3] Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: please refer to above-mentioned.
soundness: 3
strengths: 1. The idea is impressive.
2. The paper is a well-structured format.
summary: This paper proposes a method called LLM-wrapper, aiming at ‘black-box’ adaptation of VLMs for the REC task using Large Language Models.
weaknesses: 1. The performances of this method is limited. The performances of many REC methods such as [1], [2], [3] don’t rely on the LLM are better than the performances of this method.
2. The experimental evaluation of this method is constrained. Additional dataset comparisons, such as Flickr30K entities and RefClef, should be taken into account. 

[1] Qiu, Heqian and Li, Hongliang and Zhao, Taijin and Wang, Lanxiao and Wu, Qingbo and Meng, Fanman, RefCrowd: Grounding the Target in Crowd with Referring Expressions, MM2022.

[2] Zhao, Peizhi and Zheng, Shiyi and Zhao, Wenye and Xu, Dongsheng and Li, Pijian and Cai, Yi and Huang, Qingbao, Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point, AAAI2024

[3] Jing, Chenchen and Wu, Yuwei and Pei, Mingtao and Hu, Yao and Jia, Yunde and Wu, Qi, Visual-Semantic Graph Matching for Visual Grounding, MM2020.

Meta-review: This paper proposed a simple yet effective approach for tackling the referring expression task. Concretely, the paper proposed to attach a LLM to the output of the VLM model. The VLM model takes the input image and the text, and output the bounding box and the bounding box description. Based on the input text and the bounding box description, the LLM generates the output. The LLM can be further LoRA finetuned to improve its performance.

Strength:
1. The idea is simple yet effective.
2. This idea provides a way to further improve the close source model's output.
3. The proposed approach achieves good performance.

All the reviewers vote support for this paper, I would recommend accept.

Decision: Accept (Poster)