LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

Authors: Amaia Cardiel, Eloi Zablocki, Elias Ramzi, Oriane Siméoni, Matthieu Cord
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Language Models, Vision-Language Models, Black-Box Adaptation, Referring Expression Comprehension

Abstract:
Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires ‘white-box’ access to the model’s architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for ‘black-box’ adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box to match the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile and works with any VLM, transfers to new VLMs, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. The code will be open-sourced.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730013986144

thread_type:
official_review

<thread_object>
creator: reviewer_o1yy

<content>
summary:
This paper proposes a method called LLM-wrapper, aiming at ‘black-box’ adaptation of VLMs for the REC task using Large Language Models.

strengths:
1. The idea is impressive.
2. The paper is a well-structured format.

weaknesses:
1. The performances of this method is limited. The performances of many REC methods such as [1], [2], [3] don’t rely on the LLM are better than the performances of this method.
2. The experimental evaluation of this method is constrained. Additional dataset comparisons, such as Flickr30K entities and RefClef, should be taken into account. 

[1] Qiu, Heqian and Li, Hongliang and Zhao, Taijin and Wang, Lanxiao and Wu, Qingbo and Meng, Fanman, RefCrowd: Grounding the Target in Crowd with Referring Expressions, MM2022.

[2] Zhao, Peizhi and Zheng, Shiyi and Zhao, Wenye and Xu, Dongsheng and Li, Pijian and Cai, Yi and Huang, Qingbao, Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point, AAAI2024

[3] Jing, Chenchen and Wu, Yuwei and Pei, Mingtao and Hu, Yao and Jia, Yunde and Wu, Qi, Visual-Semantic Graph Matching for Visual Grounding, MM2020.

questions:
please refer to above-mentioned.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730290465981

thread_type:
official_review

<thread_object>
creator: reviewer_daad

<content>
summary:
This paper introduces LLM-wrapper, a novel method for enhancing the Referring Expression Comprehension (REC) capabilities of black-box VLMs without directly fine-tuning them. LLM-wrapper leverages a fine-tuned LLM to refine the bounding box proposals generated by a frozen, zero-shot VLM. Specifically, LLM-wrapper takes in a natural language description of each proposal along with spatial coordinates and the original referring expression, then outputs the proposal most relevant to the given expression as the final REC prediction. This black-box adaptation approach allows closed-source VLMs to be adapted without access to their internal structure or gradients, making it adaptable to any VLM and capable of integrating proposals from multiple VLMs to improve performance through ensembling.

strengths:
1. The paper is well-written, making the proposed method easy to understand with clear motivation and steps.

2. LLM-wrapper only needs to fine-tune the LLM once, allowing it to work with newer, more powerful VLMs without retraining, making it adaptable as VLM technology advances.

3. By combining outputs from multiple VLMs, LLM-wrapper reaches performance levels that compete with some fully fine-tuned VLMs, showing its robustness.

4. The method requires only a single GPU for fine-tuning, keeping training costs low and making it accessible.

5. Black-box approach is ideal for closed-source VLMs, adding value in scenarios where model internals aren’t accessible.

weaknesses:
1. The LLM-wrapper approach doubles the inference cost compared to existing VLMs.

2. The effectiveness of the LLM-wrapper is highly dependent on the performance of the chosen VLM.

3. Several recent VLMs, such as Ferret [1], KOSMOS-2 [2], and Qwen-VL [3], support both referring expression comprehension (REC) and grounding. The paper could be strengthened by including more baseline experiments that compare the original REC performance of these VLMs with their LLM-wrapped versions.

4. In the LLM-wrapper paradigm, the number of entities in the referring expression likely influences the final performance. If the VLM can accurately predict all entities in a referring expression, further analysis is warranted on whether an increased number of entities aids or hinders the LLM's ability to ground the correct one. An ablation study examining the effect of entity count on performance would be valuable.

5. Extending point 4, the experiments are conducted on the RefCOCO series, which contains relatively simple referring expressions (with a length of 3.6–8.4 words, as reported in the paper). This limitation suggests a confined number of entities per sentence. It would be insightful to evaluate the model’s performance on longer expressions with more entities, such as HC-RefLoCo benchmark, to determine how the LLM performs when faced with more complex captions and a greater number of entities.

[1] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang. Ferret: Refer
and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.

[2] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal
large language models to the world. arXiv preprint arXiv:2306.14824, 2023.

[3] Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.

questions:
Please refer to weaknesses 4 and 5.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681451929

thread_type:
official_review

<thread_object>
creator: reviewer_f3st

<content>
summary:
Fine-tuning VLMs requires users to access the architecture, weights, and gradients of models, which can be difficult due to model privacy. This paper focuses on exploring the "black-box" adaptation of VLMs on referring expression comprehension (REC), which means only calling VLMs through forward functions. They propose LLM-wrapper, which uses the reasoning ability of LLMs to select the best matching box generated by VLMs. 
In LLM-wrapper, the VLMs first generate bounding box candidates and these boxes are transferred into text prompts including the box coordinates, labels and/or prediction scores. Next, these prompts are provided to fine-tune the LLMs using LoRA. Finally, these LLMs are capable of reasoning about the image using these prompts and selecting the best matching box from the prompts. In the experimental section, they conduct experiments on three REC benchmarks. For VLMs, they choose Grounding-Dino and Florence-2, and for LLMs they choose Mixtral and Llama3. With LLM-wrapper, they show the LLMs can outperform the zero-shot VLMs by a large margin. Moreover, they find that REC performances can be further boosted by ensembling outputs from different VLMs. They also show that LLM-wrapper can be fine-tuned on one VLM and transferred to another VLM.

strengths:
1. This approach does not need to access the original parameters of VLMs, but can still get benefits from them. In the case that the VLMs are closed-source, LLM-wrapper can be useful.
2. Finetuning LLMs using LoRA reduces the number of training parameters. 
3. LLM-wrapper trained using one specific VLM can learn general spatial and semantic knowledge, and therefore can be used during inference with another VLM. For some expensive VLMs, with such knowledge transferring, it is not necessary to re-train another LLM-wrapper.

weaknesses:
1. In table 3, for Grounding-Dino, white-box experiments are conducted using SwinB, but both zero-shot baselines and LLM-wrapper experiments are conducted using SwimT. It would be better if the authors could provide White-box results for SwinT as well.
2. In table 5, the last two rows show the results on 300 samples, but for the first four rows, the results are reported on the whole val/test sets. I would like to see the improvements when transferring from GDrec to Flo2 and from Flo2 to GDrec on the same 300 samples. 
3. LLM-wrapper still uses the bounding box annotations. For REC, models with a similar number of parameters fine-tuned using the same amount of bounding boxes can even get better results.

questions:
Are the results reported on different benchmarks coming from LLMs fine-tuned on corresponding training data? For example, in table 1, when the LLM-wrapper gets 78.12 on RefCOCOg val-umd, is the Llama3 fine-tuned using only the bounding boxes and images from RefCOCOg or is the Llama3 fine-tuned using samples coming from all three benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730867739379

thread_type:
official_review

<thread_object>
creator: reviewer_q6km

<content>
summary:
The paper presents LLM-wrapper, a method for black-box adaptation of VLMs for the Referring Expression Comprehension (REC) task. It leverages the reasoning capabilities of LLMs to process and select the best-matching object proposals generated by VLMs like Grounding-DINO (GD) by converting VLM outputs (bounding boxes, labels, scores) into natural language prompts.

strengths:
- The paper introduces a novel method, LLM-wrapper, for adapting VLMs in a black-box manner, which is an interesting way to sidestep the need for model-specific fine-tuning which involves access to model weights or architectures.
- The paper compares LLM-wrapper with baseline zero-shot and fine-tuned VLMs across 3 different REC dataset variants.
- LLM-wrapper can ensemble and transfer across different VLMs, which is an interesting future area of research.
-  LLM-wrapper requires modest resources (e.g., a single 40GB-A100 GPU)
- The paper includes thorough ablation studies on the number of training samples used for fine-tuning, comparing zero-shot LLM-wrapper with fine-tuned LLM-wrapper, showing the transferability of LLM-wrapper trained on one VLM's outputs to another VLM or ensembling VLMs, etc.
- The paper shows improvements over zero-shot, but marginally closes some gap from finetuning. Still, there is significant room for improvement which is a good direction for future work.

weaknesses:
- The method requires constructing new training data to fine-tune the LLM, which partially defeats the purpose of a black-box approach that aims to minimize resource and data preparation efforts. 
- The paper lacks comparisons with existing black-box adaptation methods or prompting strategies from the literature, e.g., prompting methods that were discussed in the related work section. 
- There is noticeable performance variability depending on the choice of LLM or more likely the number of parameters finetuned (e.g., Llama 3 4.2% vs. Mixtral 0.24%). A deeper analysis of why these differences occur is needed, e.g., by fine-tuning fewer parameters in Llama and comparing Llama 4.2% and Llama 0.42% fine-tuned parameters. 
- There is limited discussion on how well the LLM adheres to producing only the best box as the output and whether there are cases where the LLM generates unexpected or incorrect responses. More generally, the paper does not provide enough detail on failure cases where the LLM-wrapper may not perform well.
- How far are results from SOTA performance on RefCOCO REC?

Minor comment:
- The paper is too verbose in many places, e.g., section 3.4 seems unnecessary as the benefits have been explained well in the previous sections. Instead, it would be nice to provide more analysis with SoTA baselines, more diverse qualitative examples, limitations, etc.

questions:
- Can you explain in more detail how GroundingDino (GD) is used as zero-shot method for REC task, given that GD outputs bounding boxes that cover various objects in an image but cannot directly perform the reasoning needed to decide which box matches a complex query?

- Is the number of fine-tuned parameters an important aspect in achieving better performance? How does Llama with 0.42% parameters tuned performs in comparison with the Llama version in the paper (with 4.2% parameters fine-tuned)?

- Does the LLM always adhere to the generated bbox index?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732147032955

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General answer to all reviewers

comment:
We thank all reviewers for their thoughtful feedback, valuable insights and for underlining several strengths and contributions of our work. 

Reviewers appreciated that LLM-wrapper is **"novel"** (q6km) and "**impressive**" (o1yY). They highlighted the value of our **black-box** setting, which avoids the need for model-specific fine-tuning expertise and direct access to VLM parameters (q6Km, f3st, DAad). The **flexibility** of LLM-wrapper was also noted, particularly its ability to *transfer* spatial and semantic knowledge from one VLM to others without retraining (q6Km, f3st, DAad), as well as its capability to *ensemble* outputs from multiple VLMs (q6Km). We are pleased that reviewers found the **experiments and ablation** studies thorough (q6Km) and acknowledged the efficiency of our method, which requires only modest resources (q6Km, f3st, DAad). The paper’s **clarity** was also appreciated, making the method and its motivations easy to understand (DAad, o1yY).

We address reviewers’ concerns and questions with direct answers to each reviewer, and we will incorporate the suggested improvements in the final version of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732613393447

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message to all reviewers

comment:
Dear reviewers, 

Thank you again for the time and effort you have invested in reviewing our work.
We addressed your concerns and questions and incorporated suggested improvements in the updated version of the paper, where changes are marked in blue.
As the author-reviewer discussion period is ongoing, we would greatly value your feedback on whether our responses have sufficiently addressed your concerns.
If you have any further comments, please feel free to share them with us.

Best regards,
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a simple yet effective approach for tackling the referring expression task. Concretely, the paper proposed to attach a LLM to the output of the VLM model. The VLM model takes the input image and the text, and output the bounding box and the bounding box description. Based on the input text and the bounding box description, the LLM generates the output. The LLM can be further LoRA finetuned to improve its performance.

Strength:
1. The idea is simple yet effective.
2. This idea provides a way to further improve the close source model's output.
3. The proposed approach achieves good performance.

All the reviewers vote support for this paper, I would recommend accept.