From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

Authors: Yang bai, Yang Zhou, Jun Zhou, Rick Siow Mong Goh, Daniel Shu Wei Ting, Yong Liu
Primary area: foundation or frontier models, including LLMs
Keywords: Vison Language Models, Task Specific Models, Visual Instruction Tuning

Abstract:
Large vision language models (VLMs) combine large language models with vision encoders, demonstrating promise across various tasks. However, they often underperform in task-specific applications due to domain gaps between pre-training and fine-tuning. We introduce VITask, a novel framework that enhances task-specific adaptability of VLMs by integrating task-specific models (TSMs). VITask employs three key strategies: exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT) to improve the task-specific performance of VLMs by adjusting their response distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs, thereby reducing the risk of generating undesired responses. Experiments on 12 medical diagnosis datasets across 9 imaging modalities show that VITask outperforms both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to integrate complementary features from both models effectively. Additionally, VITask offers practical advantages such as flexible TSM integration and robustness to incomplete instructions, making it a versatile and efficient solution for task-specific VLM tuning.

Decision: reject
Reviewer scores: 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730207310138

thread_type:
official_review

<thread_object>
creator: reviewer_z7hl

<content>
summary:
This paper first investigate why fine-tuned (VLMs) often lag behind TSMs in performance (use image classification as a case study) and then propose a new framework that can enhance the performance of VLMs on task-specific applications. They conduct it through integrating task-specific models (TSM) into VLMs in fine-tuning with 3 strategies, exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT). EP allow TSM features to guide VLMs while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs by maximizing likelihood of correct image-response pairs while minimizing that of  incorrect pairs. They conduct two-stages training. The first stage is fine-tuning VLMs using vanilla visual instruction tuning in conjunction with EP and RDA. The second stage is fine-tuning VLMs using all strategies. The experiments focus on classification tasks on medical domain.

strengths:
- The idea and strategies used to combine TSMs to VLMs to boost performance of VLMs (without using TSMs to VLMs on inference stage based on RDA mechanism) on task-specific applications are interesting. It helps to remove the limitation of VLMs when fine-tuning on task-specific applications.
- The performance of experiments on medical image classification are good.
- The paper is well-written and easy to follow.

weaknesses:
- The current experiment results are good, but the number of experiments and baseline models are insufficient, only 4 baseline models (including task-specific model). Can you compare your method with more other baselines, including some works you have mentioned in related works section which attempt to integrate VLMs with task-specific models and more VLMs such as InstructBLIP, BLIP-2, and Qwen-VL-Chat? 
- Because you did classification experiments on medical domain and there are currently many VLMs pre-trained on medical datasets, why you chose to test on the particular VLMs used in the study. In my opinion, it is important to test this approach on more VLMs which were pre-trained on medical datasets such as LLaVA-Med, Biomed GPT, etc. to make sure applying this method on these VLMs give significant improvement results.
- I am concerned about the time used to fine-tune a VLM using your method, because you need to fine-tune a task-specific model on the dataset first before combining it with VLMs and then fine-tune the VLMs. Can you give me more detail about the resource you use to fine-tune VLMs and the total time for the training process to finish using your method and vanilla methods (comparisons between your method and baseline approaches) ?

questions:
As you have mentioned in the paper, testing this approach only on medical domain is not enough to make sure your method is good. I hope that you can give more experiment results on other non-medical domains such as natural images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611383116

thread_type:
official_review

<thread_object>
creator: reviewer_yd9v

<content>
summary:
Although VLMs have excelled in understanding generic images, they fall apart for out-of-distribution task-specific applications. This paper claims that this could be due to the lack of comprehensive image representation and indirect tuning objective VLMs used to optimize. This paper proposes a framework that leverages representation from task-specific models as complementary information and optimizes the VLMs. Additionally, it explores how the inference can be done without task-specific models using two proposed methods RDA and CRT.
Considering the high costs of fine-tuning and inference with LLMs, along with the minimal performance gains, why should a user prefer using VITask over enhancing TSMs?

The paper is well-written but could be improved for better readability.

strengths:
The motivation of the paper is clear and the authors provided adequate evidence to show why the proposed method can be useful in task-specific applications.

The task-specific instruction tuning section is quite interesting and the papers validate how each proposed component improves the performance of VLMs.

The paper has a detailed section providing intuition and empirical evidence to support the claims.

weaknesses:
As mentioned in the limitations section, exploration of the method is highly limited to classification tasks. Additional tasks would be more interesting using an LLM, which has rich information embeddings.

Given the high computation budget and a marginal improvement in performance compared to TSMs, I would try to improve the TSMs which are cost-effective for training and inference. ( I consider the results as marginal improvement, the considered dataset is comparably simpler than the actual applications.) For example, optimize the model architecture, focused hyperparameter selection, Select and Fuse futures, and an ensemble of smaller models.

The paper is comprised of a lot of detailed text, which also seems to be a weakness of the paper. Authors could provide more visualization to interpret the results and behaviour of each module. Overall, the interpretation of results could be improved for better readability. You may provide more visualization to explain the benefits of the proposed modules or interpretation analysis how the proposed module benefits  compared to TSMs or vanilla VLMs.

The paper presents a few concerns :

1. The figure related to the RDA analysis referenced in Lines 264-266 is missing. As it stands, the current figure appears to be a bar chart of the EP results. Could please provide the figure?

2. The authors should consider using different formatting styles for equations (3-6). When the terms in the equations are presented in the same style as the surrounding text, it can be confusing (Line 280). Differentiating the formatting will enhance readability.

3. There are spelling errors in Figure 1 (d).

4. The word "often" is repeated in Line 041.

questions:
1. Authors should explain why using Vision-Language Models (VLMs) with a high computational budget is more beneficial than improving Task-Specific Models (TSMs). I would recommend an empirical analysis of the cost and performance benefits of VITask versus TSMs to.

2. TSMs provide better task-specific representations than general vision encoders. Could you clarify why the results decline significantly when embeddings are completely replaced by task-specific representations? Did you utilize the CLS token from Vision Transformers (ViT) or the entire patch embeddings?

3. Why do we need to optimize Stage 2 with the same loss functions used in Stage 1 without scaling them? After optimizing the Stage 1 model to minimize the Stage 1 loss, using it in Stage 2 could potentially harm the fine-tuning process. Have you considered applying a scale to the Stage 1 loss functions?

4. In line 423, it is mentioned that a novel vision-language connector is introduced in the paper, but no details about this connector are provided in the text. Could you please include more information about it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685761551

thread_type:
official_review

<thread_object>
creator: reviewer_rqyz

<content>
summary:
This paper proposes a combination of strategies, exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT), to adapt general-purpose generative VLMs for specialized medical classification tasks. Through experiments, the authors show that their method VITask , leveraging these three algorithmic components, can outperform task-specific models (TSMs), which are essentially fine-tuned ViTs, on a variety of medical classification tasks. In ablation experiments, the authors demonstrate that EP contributes the most performance boost while RDA and CRT are also complementary.

strengths:
The paper states the problem it aims to solve clearly. The proposed method is validated on a wide range of medical datasets.

weaknesses:
The biggest issue of the paper is the lack of depth. While it ablates the impact of each of the algorithmic components, they authors spent little effort trying to understand why each of them work and to compare them against existing methods.

1. It’s not clear what makes EP successful. 
    - I strongly suspect the performance gain is mostly due to the fine-tuning of the connector module. The critical experiment of simply having both the connector and the LLM (LoRA params) trainable is missing.
    - Additionally, an experiment comparing EP with prefixing tuning [1] will tell whether it’s necessary to condition the prefix (additional tokens to the LLM’s embedding space) on the image at all to get good performance. Essentially, I need to see experiments showing me EP > fine-tuning the original VLM’s connector + prefix tuning to be convinced it’s novel. 
    - I also don’t buy the claim that fine-tuning the Vision model in VLM will distort vision language alignment at all. If fine-tuning the Vision model is harmful, wouldn’t the trained LoRA weights be more harmful as well? A controlled experiment where the vision encode is also trained is needed. I am confident this will make EP perform even better. 
    - Finally, other works with the same core methodology should be discussed. For example, Graph Neural Prompting [2] builds a knowledge graph based on the prompt and multiple choice candidates and generates a conditional prefix to prompt the LLM. I think the idea is extremely similar to EP. 
2. Regarding RDA: this is essentially a fancy way of saying knowledge distillation but no relevant papers are cited. Regarding implementation, the author mentions gradient detachment. If I understood it correctly, this just means the TSM, or the “teacher”, is not trained while the goal is to train the student. Shouldn’t this be the default setting anyway? 
3. Contrastive Response Tuning: as part of the core methodology, the paper should compare its effectiveness against existing methods, such as contrastive decoding [3][4]. 

Issues mentioned above should be addressed. Otherwise this work should aim for a more application-oriented venue. 

The notations issues. 

- In equations (1), (2), (3), (5), (6), why is there a min() operator on the left hand side? The author seems to mix it up with the argmin notation. I think the author should remove the min() and avoid argmin() like notation since not all parameters are trained. 

Minor grammar issues
- For example, Takeaway #1: TSM features can prompts (prompt) VLMs to generate desired responses.

References:

[1] Li, Xiang Lisa, and Percy Liang. "Prefix-Tuning: Optimizing Continuous Prompts for Generation." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
[2] Tian, Yijun, et al. "Graph neural prompting with large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.
[3] Leng, Sicong, et al. "Mitigating object hallucinations in large vision-language models through visual contrastive decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[4] Favero, Alessandro, et al. "Multi-modal hallucination control by visual information grounding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
1. For TSM, does it use the same encoder as the one used by the target VLM? How about the connector?
2. What parameters are tuned in RDA exactly? I also don’t get how is it really different from tuning from the ground truth labels, which are used anyway. In other words, isn’t p_theta(response | exemplar) just an approximation of the ground truth label? In fact, this pseudo-label is worse since it can be wrong? Also for classification tasks, if the model only decodes a single label, does it really matter to learn the whole distribution instead of an one-hot label? If we look at table 2, RDA indeed improves upon vanilla, but what does vanilla mean? Is it just the base VLM that’s not fine-tuned at all for those medical tasks? I need to see an experiment where you just fine-tune with the ground truth labels in the training set to really understand the role of RDA. Now I tend to think you can totally get rid of it. 
3. For both RDA and CRT, what did you do to deal with the situation when the label consists of more than 1 token due to tokenization?
4. I want to know the implementation detail of how you extract the answer from the generated output from the VLM. Apparently since it’s free-form generation the format might be slightly off. Are there any instances where the model outputs a semantically equivalent label that’s not an exact match? What’s the percentage of such instances before and after your fine-tuning?
5. I wonder if the authors can explain more on the motivation of adapting the VLM for classification tasks. I think this task is only meaningful if either 1) the original generative capabilities can be preserved, or 2) the fine-tuned VLM is actually better than SoTA vision models, otherwise we might as well use a predictive classifier that’s smaller and easier to fine-tune. Thus, I think the authors should either 1) present evaluations on non-medical tasks to show there’s very little degradation, or 2) compare against the SoTA vision model, which is probably not a ViT pretrained in 2020.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710318753

thread_type:
official_review

<thread_object>
creator: reviewer_fsnv

<content>
summary:
This paper proposes a VITask framework that enhances the adaptability of Vision Language Models (VLMs) for specific tasks by integrating Task-Specific Models (TSMs). It employs strategies like guiding VLMs with TSM features, enabling adaptation without TSMs during inference, and optimizing the ranking of correct image-response pairs. Experiments on medical diagnosis datasets demonstrate that VITask outperforms standard VLMs and TSMs, providing flexible integration and robustness to incomplete instructions.

strengths:
1.	The overall writing is relatively smooth and easy to understand. 
2.	The paper has made a detailed design in the method of constructing the dataset. 
3.	The paper has conducted detailed experiments on existing medical datasets.

weaknesses:
1.	The overall motivation is to finetune existing VLM for downstream tasks. After fine-tuning, the model can only be used for a specific task, which may hinder the original generalizability of VLM. The authors seem do not consider this issue.
2.	From the experiment, it can be observed EP is most useful, while other modules bring marginal improvements. Whether the other two modules are necessary requires to be further explored.
3.	Lisa [1] also adopts similar prompt tuning techniques like EP, and what’s the difference between these methods, and what is the technical advantage of EP?
4.	The comparison lacks comprehensiveness, as numerous medical VLMs exist in the field.
5.	The effectiveness of the proposed method is only demonstrated in the medical field, therefore the title is kind of over claimed. If the method is general, more experiments on other datasets are required.

[1] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579–9589, 2024.

questions:
See the comments
</content>
</thread_object>

</thread>
<thread>
creation date:
1732641604433

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision (1/2)

comment:
We sincerely appreciate the insightful comments and constructive feedback from all reviewers, which have significantly enhanced the quality of our paper. In particular, we have made the following improvements:

**1.** Following the suggestions of Reviewers **fSNV** and **rqyz**, we verified that **the VLM fine-tuned with our method not only adapts to new tasks but also retains the original capabilities** of the pre-trained VLM. Specifically, we evaluated different fine-tuning approaches on eight benchmark datasets originally used to assess the pre-trained VLM (InternVL2-2B in our case). We compared the VITask-tuned model with the original VLM and included two variants where either the vision encoder or both the vision encoder and the connector is made learnable during fine-tuning. 

**Table 1. Performance comparison of InternVL2-2B before and after fine-tuning.**
| Method                                | MMBench_en[1] | CCBench[1] | POPE[2] | MMMU[3] | MMVP[4] | MMVet[5] | GQA[6] | AI2D[7] |
| ------------------------------------- | ------------- | ---------- | ------- | ------- | ------- | -------- | ------ | ------- |
| Original | 73.2          | 74.7       | 87.3    | 0.342   | 0.353   | 44.6     | 61.03  | 0.7409  |
| Tuning LLM + Vision Encoder           | 27.6          | 27.7       | 0.1     | 0.274   | 0.112   | 15.5     | 30.18  | 0.0213  |
| Tuning LLM + Encoder + Connector | 17.15         | 8          | 0.7     | 0.291   | 0.073   | 13       | 27.8   | 0.005   |
| Ours                                | 72.2          | 72.5       | 87.7    | 0.341   | 0.34    | 36.2     | 59.6   | 0.702   |

---

**2.** Based on the suggestions of Reviewers **fSNV** and **Z7hL**, we have **enhanced the comprehensiveness of our experiments**. To better demonstrate the effectiveness of our method, we compared it with additional baseline models and applied it to medical VLMs.

**Table 2. Performance (Acc/F1) of the fine-tuned VLMs on medical image diagnosis tasks.**
| Model                     | Path       | Chest      | Derma     | OCT        | Pneumonia   | Retina    | Breast    | Blood     | Tissue    | OrganA    | OrganC    | OrganS    | Mean      |
| ----------------- | ------------------- | --------------- | ------------------- | ------------------- | ------------------- | --------------- | --------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |
| LLaVAv1.5-7B      | 0.891/0.840         | 0.528/0.020     | 0.726/0.369         | 0.744/0.723         | 0.934/0.929         | 0.482/0.207     | 0.596/0.553     | 0.890/0.767         | 0.514/0.346         | 0.735/0.692         | 0.633/0.588         | 0.521/0.468         | 0.683/0.542         |
| LLaVAv1.5-13B     | 0.935/0.905         | **0.535**/0.073 | 0.731/0.355         | 0.788/0.786         | 0.881/0.864         | 0.557/0.279     | 0.750/0.671     | 0.951/0.832         | 0.613/0.497         | 0.878/0.855         | 0.796/0.750         | 0.689/0.621         | 0.759/0.624         |
| Qwen2-VL-7B       | 0.823/0.754         | 0.510/0.051     | 0.716/0.384         | 0.738/0.729         | 0.438/0.383         | 0.280/0.166     | 0.494/0.510     | 0.286/0.166         | 0.575/0.411         | 0.807/0.777         | 0.724/0.681         | 0.672/0.618         | 0.589/0.469         |
| LLava-Med         | 0.939/0.915         | 0.513/0.088     | 0.800/0.556         | 0.868/0.868         | 0.910/0.900         | 0.542/0.280     | 0.212/0.382     | 0.975/0.856         | 0.642/0.540         | 0.916/0.908         | 0.865/0.843         | 0.738/0.687         | 0.743/0.652         |
| LLaVA-Med+Ours    | **0.964**/**0.949** | 0.518/**0.118** | 0.856/0.723         | 0.942/0.942         | **0.952**/**0.948** | 0.650/**0.544** | 0.859/**0.833** | 0.987/0.867         | 0.755/0.685         | 0.953/0.947         | **0.922**/**0.909** | 0.799/0.750         | 0.846/0.768         |
| InternVL2-8B      | 0.938/0.911         | 0.520/0.045     | 0.755/0.487         | 0.798/0.788         | 0.886/0.872         | 0.575/0.394     | 0.788/0.699     | 0.960/0.841         | 0.594/0.465         | 0.865/0.838         | 0.817/0.783         | 0.698/0.650         | 0.766/0.648         |
| InternVL2-2B      | 0.926/0.896         | 0.523/0.024     | 0.770/0.499         | 0.726/0.704         | 0.886/0.873         | 0.590/0.370     | 0.744/0.524     | 0.931/0.818         | 0.569/0.419         | 0.828/0.801         | 0.778/0.742         | 0.635/0.578         | 0.742/0.604         |
| InternVL2-2B+Ours | 0.953/0.937         | 0.517/0.129     | **0.877**/**0.772** | **0.952**/**0.952** | 0.931/0.923         | **0.632**/0.522 | **0.865**/0.828 | **0.991**/**0.870** | **0.761**/**0.690** | **0.955**/**0.950** | 0.920/0.908         | **0.809**/**0.765** | **0.847**/**0.771** |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces VITask, a framework for adapting vision-language models (VLMs) to task-specific applications using exemplar prompting, response distribution alignment, and contrastive response tuning. While reviewers appreciated the clarity of the methodology and its strong performance on specific tasks, they highlighted several significant concerns. The title's claim of generality is hyperbolic, as the original experiments were limited to an extensively curated set of medical datasets that primarily involve an array of relatively straightforward classification tasks, which do not fully validate the model's broader applicability. The additional tasks presented in the rebuttal, such as medical phrase grounding and natural image classification, lacked comprehensive baselines and did not adequately support the broader claims of the paper. Moreover, since the substantial changes made during the rebuttal were not incorporated into the main paper, the revision may turn into something beyond a simple revision. Therefore, it is difficult to recommend the paper to be accepted in this state.