From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

## Abstract

Abstract Large vision language models (VLMs) combine large language models with vision encoders, demonstrating promise across various tasks. However, they often underperform in task-specific applications due to domain gaps between pre-training and fine-tuning.
We introduce VITask, a novel framework that enhances task-specific adaptability of VLMs by integrating task-specific models (TSMs). VITask employs three key strategies: exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT) to improve the task-specific performance of VLMs by adjusting their response distributions.
EP allows TSM features to guide VLMs, while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs, thereby reducing the risk of generating undesired responses.
Experiments on 12 medical diagnosis datasets across 9 imaging modalities show that VITask outperforms both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to integrate complementary features from both models effectively.
Additionally, VITask offers practical advantages such as flexible TSM integration and robustness to incomplete instructions, making it a versatile and efficient solution for task-specific VLM tuning. Our code are available at https://github.com/baiyang4/VITask .

## 1 Introduction

Large Vision Language Models (VLMs) combine the capabilities of large language models (LLMs) with pre-trained vision encoders, enabling them to process and understand both text and images Liu et al. (; ); Driess et al. (); Dai et al. (); Chen et al. (; ); Alayrac et al. (); Bai et al. ().
This integration allows VLMs to perceive visual inputs, comprehend complex queries, and perform sophisticated reasoning across a wide array of tasks and domains.
The success of VLMs drives the growing trend of adapting VLMs for a wide range of task-specific applications such as medical diagnosis, autonomous driving, and content creation He et al. (); Moor et al. (); Li et al. (); Wu et al. (); Zhou et al. (); Xu et al. ().

Despite the wide applicability of VLMs, recent studies have noted that their performance often often falls short compared to task-specific models (TSMs) when fine-tuned for specific tasks or domains Singhal et al. (); Yang et al. ().
The performance gap between VLMs and TSMs represents a critical limitation, particularly in real-world scenarios that demand high accuracy and reliable service quality.
Although substantial progress has been made in enhancing the performance and versatility of VLMs Wu et al. (); Liu et al. (); Lai et al. (); Wang et al. (), most of these approaches do not focus on effectively adapting pre-trained VLMs to specific tasks or datasets.
This leads to a fundamental question: can we adapt VLMs to perform as well as, or even surpass, task-specific models?

In this study, we use image classification as a case study to investigate why fine-tuned (VLMs) often lag behind TSMs in performance. We identify two main factors contributing to this decline:
1) Unspecialized Image Representations: Image features learned during pre-training for vision-language tasks are not effective for specific classification tasks. They often miss important details needed for these tasks, making it hard for the vision encoder to extract useful information.
2) Indirect Tuning Objective: Fine-tuning VLMs typically emphasizes enhancing text generation, such as predicting the next word, rather than directly addressing image classification. This approach can hinder the models from learning the essential features required for effective image classification, resulting in subpar performance.

To address these challenges, we propose VITask, a novel framework that combines the strengths of TSMs and VLMs to improve task-specific performance without sacrificing the versatility and instruction-following capabilities of VLMs.
Our main idea leverages small, easily obtainable TSMs and a task-specific tuning objective to improve the learning of desired response distributions.
To maintain the vision-language alignment in pre-trained VLMs, we avoid directly updating the vision encoder for new tasks.
Instead, we propose exemplar prompting, using TSM features as exemplars to enhance VLM adaptability without altering pre-trained image features, while incorporating specialized task representations.
Additionally, we introduce response distribution alignment to align the response distributions between VLMs with and without exemplar prompting. This allows the VLM to implicitly learn from the TSM by utilizing its own responses during fine-tuning.
Finally, we propose contrastive response tuning, which maximizes the likelihood of correct image-response pairs (e.g., $p(\texttt{cat}|\texttt{<cat image>})$) while minimizing the likelihood of incorrect pairs (e.g., $p(\texttt{cat}|\texttt{<dog image>})$). This approach promotes more discriminative and accurate response rankings for visual instructions, thereby enhancing task-specific performance.

We evaluate VITask on 12 medical image diagnosis datasets and show that it consistently outperforms both TSMs and vanilla instruction-tuned VLMs. Furthermore, VITask demonstrates robustness to incomplete instructions, providing flexibility for real-world applications where task descriptions may not be comprehensive. Our results highlight the potential of VITask to generalize beyond medical tasks, making it a versatile framework for task-specific VLM tuning.

Figure: Figure 1: Overview of the proposed VITask framework. (a) Traditional visual instruction tuning. (b) Exemplar Prompting (EP) enhances VLM’s image representations using TSM features without modifying pre-trained features. (c) Response Distribution Alignment (RDA) aligns EP and non-EP responses to capture task-specific information. (d) Contrastive Response Tuning (CRT) leverages negative samples to improve the VLM’s response ranking capability by maximizing the margin between correct and incorrect image-response pairs.
Refer to caption: x1.png

## 2 Related Work

#### Large Vision Language Models.

Vision Language Models (VLMs) are multimodal models designed to process and understand both visual and textual information.
Inspired by the success of large language models (LLMs), such as GPT-4 Achiam et al. (), LLaMA-2 Touvron et al. (), and PaLM-2 Anil et al. (), the development of VLMs has evolved from simply aligning image-text pairs, as seen in models like CLIP Radford et al. (), BLIP Li et al. (), to integrating vision encoders into LLMs, enabling them to process and interpret visual information.
Examples of such models include GPT-4V [1], InstructBLIP Dai et al. (), PaLM-E Driess et al. (), MiniGPT-4 Zhu et al. (), LLaVA series Liu et al. (; ; ), InternVL Chen et al. (; ), the Gemini series Team et al. (); Reid et al. (), Claude-3 Anthropic (), and Qwen-VL-Max Bai et al. ().
Recent advancements in VLMs focus on improving model architectures Liu et al. (); Chen et al. (; ), training strategies Liu et al. (; ); He et al. (), and datasets Yu et al. (); Li et al. (); Liu et al. (); Li et al. (), resulting in enhanced capabilities and broader applications.

#### Visual Instruction Tuning.

Current VLM training pipelines usually follows a two-stage protocol. First, the vision language alignment stage align the image features from the vision encoder with the word embeddings encoded in LLMs.
Second, the visual instruction tuning stage adapts VLMs to follow instructions that involve both visual and textual inputs, making VLMs able to respond to natural language commands or questions based on the content of an image Liu et al. (); Dai et al. ().
Visual instruction tuning is a crucial step for making VLMs more interactive, versatile, and context-aware, allowing them to follow instructions related to specific tasks, enhancing its accuracy and adaptability to real-world applications where users provide visual and textual inputs.
There are many existing works in the field of visual instruction tuning. Typical research topics focus on gaining specialized visual understanding ability Yue et al. (); Nisar et al. (); Chen et al. (); Lai et al. (), reducing computational costs Hu et al. (); Luo et al. (); Lee et al. (), mitigating hallucination Leng et al. (); Zhou et al. (); Hu et al. (), creating or augmenting instruction data Yu et al. (); Li et al. (); Liu et al. (); Li et al. ().

#### Integrating VLMs and TSMs.

Several approaches have been proposed to integrate VLMs with task-specific models in an attempt to leverage the strengths of both Liu et al. (); Lai et al. (); Li et al. ().
However, these works primarily focus on utilizing TSMs as task-specific heads or tools for constructing a new VLM, without addressing the challenges of fine-tuning pre-trained VLMs for specific tasks or datasets.
Our work focuses on improving the visual instruction tuning paradigm to achieve better task-specific performance, especially when the model faces domain gaps with downstream task data.

## 3 Instruction-Tuned VLMs vs. Task-Specific Models

In this section, we compare instruction-tuned VLMs with TSMs to evaluate their performance on domain-specific tasks.
While instruction-tuned VLMs are designed to handle both image and text inputs in a generalized manner, TSMs are optimized for a particular task or dataset, often leading to superior performance for specific applications.
Despite the wide range of potential downstream tasks, image classification serves as a fundamental task for benchmarking.
We thus conduct a head-to-head comparison between the VLMs and TSMs on a single classification task, as a case study for our analysis.

Figure: (a) Accuracy
Refer to caption: x2.png

#### Setting.

We consider fine-tuning a pre-trained VLM and a naïve task-specific model on a given classification dataset, which may have domain gaps with the data used for pre-training. Specifically, we use InternVL2-2B Chen et al. () as the pre-trained VLM and a ViT-Base model Dosovitskiy et al. () pre-trained on ImageNet-21k Deng et al. (), with a randomly initialized linear classification head, as the task-specific classifier.
Both models are fine-tuned for multi-class image classification on the HAM10000 dataset Tschandl et al. (), which contains 10,015 dermatoscopic images across 7 classes for diagnosing pigmented skin lesions.
We follow the same setting in Yang et al. () to set the training, validation, and test set as $70\%$, $10\%$, $20\%$, respectively.
In what follows, we conduct our analysis within this setting for simplicity and validate our findings through formal experiments on 12 medical datasets across 6 domains, as detailed in Section 5.

#### Instruction Formatting.

Since the classification dataset is not originally designed for instruction tuning, we convert the training data into an instruction-following format as follows He et al. ():

Here, the tags 
 and 
 are used to indicate instruction-following for ease of reading and do not affect the experimental results. The <image> tag represents the image features extracted from the vision encoder of the pre-trained VLM. Using this format, an instruction for the HAM10000 dataset Tschandl et al. () could be: “Analyze the given dermatoscope image for diagnosis. The possible diagnoses are: {possible disease names}.”. The corresponding response for an image with vascular lesions would be vascular lesions.

#### Model Training.

For VLMs, we follow the common practice of instruction-tuning the LLM component while keeping the vision encoder and vision-language connector frozen, utilizing LoRA Hu et al. () to improve training efficiency.
For TSMs, we fully fine-tune the ViT classifier using class labels, updating both the ViT model and the classification head during training.
More implementation details are provided in Section 5.

#### Observations.

As shown in Figure 2, the ViT classifier (TSM) achieves an F1 score of 0.790, significantly outperforming the instruction-tuned VLM (VLM for short subsequently), which only reaches an F1 score of 0.531.
This highlights the difficulty of fine-tuning VLMs for specific tasks.
The large performance gap likely stems from the fact that pre-trained image features may not encompass all the essential representations required for new tasks.
When the VLM’s vision encoder is made trainable (denoted by VLM^∗), the model’s performance improves to an F1 score of 0.658, which, while better than VLM, still lags behind TSM.
It is worth noting that although making the vision encoder trainable enhances performance, this approach may be undesirable, as it risks distorting the valuable vision-language alignment and conversational abilities that VLMs rely on.
These findings suggest that vanilla visual instruction tuning may struggle when adapted to specific downstream tasks, facing unique challenges in achieving task-specific performance on par with TSMs.
This is particularly notable given that TSMs are generally much smaller and easier to train for specialized tasks.
Can we adapt a VLM to achieve comparable or superior task-specific performance while preserving its pre-trained vision-language alignment and conversational abilities?

## 4 Task-Specific Visual Instruction Tuning

In this section, we investigate why fine-tuned VLMs may underperform in classification tasks and highlight two key issues in the current visual instruction tuning paradigm:
1. Unspecialized Image Representations: The pre-trained vision encoder learns representations optimized for vision-language alignment, which are often sub-optimal for downstream classification tasks.
2. Indiect Tuning Objective: The tuning objective focuses on next token prediction, which is more suited to text generation than to classification tasks that require fine-grained discrimination.
To overcome these challenges, we proposed VITask, a novel framework (Figure 1) that bridges TSMs and VLMs to enhance task-specific adaptability and performance.

#### Exemplar Prompting.

We first introduce Exemplar Prompting (EP).
A VLM takes a visual image $\mathbf{v}$ and a textual instruction $\mathbf{x}$ as inputs, aiming to generate relevant and helpful response $\mathbf{y}$.
Visual instruction tuning can be framed as conditional probability estimation $p_{\theta}(\mathbf{y}\mid\mathbf{v},\mathbf{x})$, where $\theta$ represents the learnable parameters of the VLM.
Given a visual instruction dataset $D=\{\text{image}_{i},\text{instruction}_{i},\text{response}_{i}\}_{i=1}^{N}$ containing $N$ image-instruction-response triples, visual instruction tuning adapts the VLM by minimizing the following objective:

$$ $\displaystyle\min_{\theta}\ \ \mathcal{L}^{\text{Van}}=\frac{1}{N}\sum_{i=1}^{ N}-\log p_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{instruction}_ {i}).$ (1) $$

For image classification, we can train a TSM, such as the ViT classifier mentioned in Section 3, on the same dataset $D$ without instruction formatting and extract the latent feature for each $\text{image}_{i}$. We define this latent feature as $\text{exemplar}_{i}$ for $\text{image}_{i}$.
Exemplar prompting utilizes the TSM features to prompt VLMs during fine-tuning by augmenting the VLM’s image features $\text{image}_{i}$ with $\text{exemplar}_{i}$. This is achieved by modifying the tuning objective (1) as follows:

$$ $\displaystyle\min_{\theta}\ \ \mathcal{L}^{\text{EP}}=\frac{1}{N}\sum_{i=1}^{N }-\log p_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{exemplar}_{i}, \text{instruction}_{i}).$ (2) $$

The rationale behind exemplar prompting is that since the TSM is optimized to learn specialized features for downstream tasks, it can offer task-specific latent features that guide the VLM in learning a better mapping between the visual instruction and the desired response. This enhances the VLM’s adaptability without directly altering its pre-trained image features, thereby preserving the vision-language alignment while incorporating relevant task-specific knowledge.

#### Implementation and Analysis.

As shown in Figure 1, we implement exemplar prompting by introducing a learnable vision-language connector to align TSM features with the LLM of the VLM. This connector is updated along with the LLM, while the vision encoders of both VLM and TSM remain frozen during fine-tuning.
For a ViT classifier as the TSM, exemplars can be derived from all patch embeddings (EP${}_{\text{all}}$), the CLS token (EP${}_{\text{cls}}$), or by replacing all VLM image features with TSM features (EP${}_{\text{rep}}$).
From Figure 2, we observe that replacing all VLM image features with TSM features results in poor performance, showing that TSM features alone cannot maintain VLMs’ instruction-following ability for new tasks.
However, exemplar prompting with all patch embeddings or the CLS token significantly boosts classification performance compared to standard instruction tuning.
Notably, VLM+EP${}_{\text{cls}}$ matches or even exceeds the performance of both TSM and VLM with a trainable vision encoder, demonstrating that incorporating just one TSM feature (CLS token) enhances task-specific instruction-response mappings.
Conversely, using all patch tokens (EP${}_{\text{all}}$) is less effective, suggesting that irrelevant features may degrade performance.
Therefore, if not specified otherwise, we use the CLS token for EP, considering it is the most effective and efficient.

#### Response Distribution Alignment.

One key intuition behind exemplar prompting is that it creates a shortcut between exemplars and desired responses, making instruction-following easier.
While effective, using exemplars requires combining TSM and VLM during both fine-tuning and inference. This increases the size of the model, which may be impractical when dealing with multiple tasks and corresponding TSMs.
A natural question arises: can task-specific adaptability be improved without relying on TSMs and exemplars during inference? The answer is yes.
Instead of explicitly learning the exemplar-response mapping, we propose Response Distribution Alignment (RDA) to implicitly learn the distribution of desired responses.
The idea is for the VLM with exemplar prompting to ”teach” the VLM without exemplar prompting during fine-tuning.
Specifically, we minimize the Kullback-Leibler (KL) divergence between the response distributions of VLM and VLM+EP:

$$ $\displaystyle\min_{\theta}\ \ \mathcal{L}^{\text{RDA}}=\frac{1}{N}\sum_{i=1}^{ N}D_{\mathrm{KL}}(p_{\theta}(\text{response}_{i})\|p_{\theta}(\text{response}_ {i}\mid\text{exemplar}_{i})),$ (3) $$

where we omit the common conditions on $\text{image}_{i}$ and $\text{instruction}_{i}$ in the response distributions for simplicity.
This approach allows the VLM to learn specialized task information from TSM by mimicking the behavior of VLM+EP, all without requiring exemplars during inference.

#### Implementation and Analysis.

The proposed RDA strategy optimizes (3) alongside the basic objectives in (1) and (2). Since our aim is to learn from the exemplar-prompted VLM rather than the other way around, we detach the gradient of the exemplar-prompted distribution $p_{\theta}(\text{response}_{i}\mid\text{exemplar}_{i})$ when computing (3).
Figure 2 demonstrates the impact of RDA on classification performance. We also test a variant, RDA^∗, which is identical to RDA but without gradient detachment.
The results show that VLM+RDA improves the F1 score by $6\%$, demonstrating that TSM can effectively guide VLM to learn a better response distribution even without using exemplar prompting during inference.
In contrast, VLM+RDA^∗ shows no significant improvement over the baseline VLM, verifying that RDA’s gains are due to the task-specific information transferred from VLM+EP.

#### Contrastive Response Tuning.

The success of response distribution alignment suggests that we do not need to teach VLM explicit mappings from instructions to responses; instead, these mappings can be implicitly learned by refining the distribution of desired responses.
Motivated by Hewitt et al. (), we propose the concept of visual response ranking capability, referring to a VLM’s ability to assign a higher likelihood to correct image-response pairs than to incorrect ones for a given instruction.
For two independent image-instruction-response triples $(\text{image}_{i},\text{instruction}_{i},\text{response}_{i})$ and $(\text{image}_{j},\text{instruction}_{j},\text{response}_{j})$, with $\text{instruction}_{i}=\text{instruction}_{j}$ and $\text{response}_{i}\neq\text{response}_{j}$, the visual response ranking capability holds for a VLM $p_{\theta}$ if

$$ $\displaystyle p_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{ instruction}_{i})>p_{\theta}(\text{response}_{i}\mid\text{image}_{j},\text{ instruction}_{i}),$ (4) $$

where we assume the instruction $\text{instruction}_{i}$ is the same for both triples for clarity.
Intuitively, a VLM with this capability will more likely generate correct responses for visual instructions.
The degree to which a VLM possesses this ranking capability reflects how well it can differentiate between correct and incorrect image-response pairs for a given instruction.
We argue that vanilla visual instruction tuning often fails to establish this ranking capability because it focuses solely on learning instruction-response mappings and does not explicitly account for the critical relationship between images and responses.
As a result, an instruction-tuned VLM might rank incorrect image-response pairs higher than the correct ones, leading to suboptimal performance on specific tasks.
To address this issue, we propose Contrastive Response Tuning (CRT) to maximize the margin between correct and incorrect image-response pairs. This is done by minimizing the following objective:

$$ $\displaystyle\min_{\theta}\ \ \mathcal{L}^{\text{CRT}}=\frac{1}{N}\sum_{i=1}^{ N}-\log q_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{image}_{j}, \text{instruction}_{i}),$ (5) $$

where the margin distribution is defined as:

$$ $\displaystyle q_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{image}_ {j},\text{instruction}_{i})=\text{Softmax}(\mathbf{y}_{i}^{\text{pos}}-\mathbf {y}_{i}^{\text{neg}}).$ (6) $$

Here, $\mathbf{y}_{i}^{\text{pos}}$ represents the logits for the positive response distribution $p_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{instruction}_{i})$, and $\mathbf{y}_{i}^{\text{neg}}$ represents the logits for the negative response distribution $p_{\theta}(\text{response}_{i}\mid\text{image}_{j},\text{instruction}_{i})$.
CRT encourages the model to maximize the likelihood of the correct image-response pair (positive) while minimizing the likelihood of incorrect pairs (negative), thus promoting more discriminative and accurate response rankings.
This approach enhances the VLM’s visual response ranking capability, improving task-specific adaptability and accuracy in scenarios like image classification.

#### Implementation and Analysis.

For each triple $(\text{image}_{i},\text{instruction}_{i},\text{response}_{i})\sim D$, we randomly select a negative $\text{image}_{j}$ from another triple $(\text{image}_{j},\text{instruction}_{j},\text{response}_{j})\sim D$, ensuring that $\text{instruction}_{i}=\text{instruction}_{j}$ and $\text{response}_{i}\neq\text{response}_{j}$. Then, CRT (5) can be applied to each token of $\text{response}_{i}$ given $\text{image}_{i}$, $\text{image}_{j}$, and $\text{instruction}_{i}$ autoregressively.
To gain a deeper understanding of how CRT improves the visual response ranking capability, we evaluate its effect on the HAM1000 test set.
We compute the average probability of each token in $\text{response}_{i}$ for both positive and negative image pairs based on three different VLMs: a pre-trained VLM without fine-tuning, a VLM tuned with vanilla visual instruction tuning, and a VLM tuned with our CRT strategy.
Figure 3 illustrates the normalized density of response probabilities for positive and negative image pairs across these VLMs.
Figure 3(a) shows that the pre-trained VLM, without any fine-tuning, does not possess the visual response ranking capability, as the probability distributions for positive and negative image pairs are nearly identical.
This confirms that the pre-trained VLM lacks task-specific instruction-following ability.
Figure 3(b) indicates that while vanilla instruction tuning enables the VLM to some extent to differentiate between positive and negative image pairs, there remains a significant overlap.
Many incorrect image-response pairs still receive high probabilities, posing a risk of undesired responses.
Figure 3(c) demonstrates that CRT effectively sharpens the distinction between correct and incorrect image-response pairs by maximizing the margin distribution $q_{\theta}(\text{response}_{i}\mid\text{image}_{i},\text{image}_{j},\text{
instruction}_{i})$.
The CRT-tuned VLM shows a clear increase in the probability for correct image-response pairs and a corresponding decrease for incorrect ones, signifying that CRT substantially enhances the model’s ability to generate desirable and accurate responses compared to vanilla instruction-tuned VLMs.

#### VITask Framework.

To bring together all the proposed strategies, we introduce the VITask framework, a two-stage pipeline designed for task-specific visual instruction tuning, analogous to the way VLMs are trained.
Stage 1: we make the task-specific connector learnable and fine-tune the VLM using vanilla visual instruction tuning in conjunction with EP and RDA. The objective for this stage is: $\mathcal{L}^{\text{Stage1}}=\mathcal{L}^{\text{Van}}+\mathcal{L}^{\text{EP}}+
\alpha\mathcal{L}^{\text{RDA}}$.
The primary goal of Stage 1 is to establish the basic visual instruction-following ability and learn an effective task-specific connector that aligns TSM features with the LLM.
Stage 2: After the task-specific connector is trained, we freeze it and then fine-tune the VLM with all the proposed loss functions. The objective becomes:

$$ $\displaystyle\mathcal{L}^{\text{Stage2}}=\mathcal{L}^{\text{Van}}+\mathcal{L}^ {\text{EP}}+\alpha\mathcal{L}^{\text{RDA}}+\beta\mathcal{L}^{\text{CRT}},$ (7) $$

where $\alpha$ and $\beta$ adjust the weight of $\mathcal{L}^{\text{RDA}}$ and $\mathcal{L}^{\text{CRT}}$, respectively.
In this stage, the model fine-tunes its visual response ranking capability through CRT while maintaining the learned visual-instruction mapping from Stage 1.
Although so far our framework and analysis focus on a single task and dataset, VITask can be generalized to multi-task or multi-dataset settings by expanding the label space and training a joint TSM.
This flexibility allows the framework to build more robust, domain-specific VLMs, capable of handling a variety of downstream tasks.

#### Advantages.

VITask offers several advantages beyond improving task-specific performance.
One major benefit is its ability to decouple image representation learning from visual instruction tuning by incorporating TSMs into VLMs.
This flexibility allows for the use of any TSM architecture, giving practitioners the freedom to choose the best model for their specific task.
Furthermore, once fine-tuned, the VLM can perform inference without needing the TSM, maintaining task-specific adaptability while reducing model complexity.

Another key advantage of VITask is its plug-and-play collaboration between VLMs and TSMs.
When a new task is introduced, a new TSM can be separately trained and directly connected to the VLM without requiring further instruction tuning.
Since TSMs are generally smaller and easier to train than VLMs, VITask provides an efficient way to adapt VLMs to new tasks, making the framework highly scalable and adaptable to multiple domains.

Additionally, VITask demonstrates robustness against the content of instructions.
Instruction-tuned VLMs often rely on carefully crafted instructions for optimal performance.
For instance, in experiments with the HAM10000 dataset, detailed class information is typically included in the instruction to enhance accuracy.
However, in real-world applications, users may not always know such detailed information in advance.
VITask mitigates this limitation by adapting the response distribution based on task-specific information from TSMs rather than solely relying on the instruction itself, enabling strong performance even with more generalized or incomplete instructions.

Figure: (a) No Tuning
Refer to caption: extracted/5910653/imgs/vis_raw_2b.png

## 5 Experiments

In this section, we evaluate the proposed VITask framework in fine-tuning a VLM for medical diagnosis.
Our experimental setup is designed to test the following key aspects: 1) the ability of VITask to improve task-specific classification performance; 2) the flexibility of VITask in adapting to various tasks without retraining the entire model; 3) the robustness of VITask against incomplete instructions.

#### Datasets and Metrics.

We utilize the MedMNIST 2D Dataset collection Yang et al. () for fine-tuning and testing our VLM.
This comprehensive collection encompasses 9 distinct biomedical imaging modalities, such as X-ray, OCT, ultrasound, CT, and electron microscopy, and supports various types of analysis, such as binary/multi-class classification, ordinal regression, and multi-label categorization, covering a total of 70 unique classification categories.
The dataset comprises a total of 518,175 training samples, 70,467 validation samples, and 119,320 testing samples, covering a broad spectrum of diseases and classification types.
For external validation, we employ the IDRiD Porwal et al. (), MESSIDOR Decencière et al. (), and APTOS Decencière et al. () datasets. More dataset details are provided in Appendix.
We report results using standard metrics such as accuracy and F1 score.

#### Implementation Details.

In this work, we primarily evaluate our proposed method based on the 2B version of InternVL2 Chen et al. () due to its effectiveness and efficiency, which demonstrates comparable or superior performance to other VLMs with larger parameter sizes in our experiments. InternVL2-2B consists of a ViT-Large vision encoder (InternViT‑300M Chen et al. ()) and a 1.8B-parameter language model (InternLM2-1.8B Cai et al. ()). During fine-tuning, we freeze the vision encoder and apply LoRA Hu et al. () for efficient adaptation of the LLM component. Additionally, we introduce a novel vision-language connector specifically for the TSM model while keeping the TSM parameters fixed.
For our VITask framework, we train stage 1 for 1 epoch, followed by stage 2 for an additional epoch.

**Table 1: Performance of VLMs on medical image diagnosis tasks. Bold indicates the best result, underline indicates the second-best, and * denotes results from the original paper He et al. ().**
| Dataset | Metric | TSM | MedDr* | LLaVA<br>13B | InternVL<br>2B | VITask |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| w/o EP | w/ EP |  |  |  |  |  |  |
| PathMNIST | Accuracy$\uparrow$ | 0.933 | - | 0.935 | 0.926 | 0.939 | 0.953 |
| Macro-F1$\uparrow$ | 0.926 | - | 0.905 | 0.896 | 0.911 | 0.937 |  |
| ChestMNIST | Accuracy$\uparrow$ | 0.533 | 0.519 | 0.535 | 0.523 | 0.513 | 0.517 |
| Macro-F1$\uparrow$ | 0.095 | 0.134 | 0.073 | 0.024 | 0.102 | 0.129 |  |
| DermaMNIST | Accuracy$\uparrow$ | 0.846 | 0.690 | 0.731 | 0.770 | 0.810 | 0.877 |
| Macro-F1$\uparrow$ | 0.792 | 0.395 | 0.355 | 0.499 | 0.633 | 0.772 |  |
| OCTMNIST | Accuracy$\uparrow$ | 0.934 | 0.692 | 0.788 | 0.726 | 0.853 | 0.952 |
| Macro-F1$\uparrow$ | 0.941 | 0.661 | 0.786 | 0.704 | 0.846 | 0.952 |  |
| Pneumonia-<br>MNIST | Accuracy$\uparrow$ | 0.968 | 0.929 | 0.881 | 0.886 | 0.888 | 0.931 |
| Macro-F1$\uparrow$ | 0.965 | 0.926 | 0.864 | 0.873 | 0.872 | 0.923 |  |
| RetinaMNIST | Accuracy$\uparrow$ | 0.472 | - | 0.557 | 0.590 | 0.625 | 0.632 |
| Macro-F1$\uparrow$ | 0.424 | - | 0.279 | 0.370 | 0.457 | 0.522 |  |
| BreastMNIST | Accuracy$\uparrow$ | 0.897 | 0.878 | 0.750 | 0.744 | 0.846 | 0.865 |
| Macro-F1$\uparrow$ | 0.866 | 0.842 | 0.671 | 0.524 | 0.798 | 0.828 |  |
| BloodMNIST | Accuracy$\uparrow$ | 0.987 | 0.955 | 0.951 | 0.931 | 0.983 | 0.991 |
| Macro-F1$\uparrow$ | 0.990 | 0.954 | 0.832 | 0.818 | 0.864 | 0.870 |  |
| TissueMNIST | Accuracy$\uparrow$ | 0.697 | - | 0.613 | 0.569 | 0.643 | 0.761 |
| Macro-F1$\uparrow$ | 0.681 | - | 0.497 | 0.419 | 0.538 | 0.690 |  |
| OrganAMNIST | Accuracy$\uparrow$ | 0.934 | 0.846 | 0.878 | 0.828 | 0.924 | 0.955 |
| Macro-F1$\uparrow$ | 0.950 | 0.822 | 0.855 | 0.801 | 0.917 | 0.950 |  |
| OrganCMNIST | Accuracy$\uparrow$ | 0.869 | - | 0.796 | 0.778 | 0.889 | 0.920 |
| Macro-F1$\uparrow$ | 0.898 | - | 0.750 | 0.742 | 0.871 | 0.908 |  |
| OrganSMNIST | Accuracy$\uparrow$ | 0.726 | - | 0.689 | 0.635 | 0.758 | 0.809 |
| Macro-F1$\uparrow$ | 0.737 | - | 0.621 | 0.578 | 0.710 | 0.765 |  |
| Average | Accuracy$\uparrow$ | 0.816 | N.A. | 0.759 | 0.742 | 0.806 | 0.847 |
| Macro-F1$\uparrow$ | 0.772 | N.A. | 0.624 | 0.604 | 0.710 | 0.771 |  |

#### Compared Methods.

We compare our VITask-tuned VLM (VITask for short) against both a task-specific ViT classifier (TSM) and vanilla visual instruction-tuned VLMs on the MedMNIST dataset to analyze its task-specific performance, flexibility, and robustness.
In particular, we test LLaVA1.5-13B Liu et al. () and InternVL2-2B Chen et al. () with vanilla visual instruction tuning. For comprehensiveness, we also compare a recent medical VLM, MedDr He et al. (), which is also trained on the MedMNIST dataset.

#### Main Results.

Table 1 presents the medical image diagnosis performance across different models.
Comparison with TSM:
Most instruction-tuned VLMs, except VITask, show a significant performance gap compared to TSM, highlighting the challenges of fine-tuning VLMs for specialized tasks and domains.
In contrast, VITask with Exemplar Prompting (EP) consistently delivers the best results, achieving the highest accuracy and F1 scores on 8 out of 12 datasets.
This demonstrates that features derived from TSM are highly effective in providing VLMs with task-specific features, enabling VLMs to achieve TSM-level performance.
Moreover, the superior performance of VITask relative to TSM suggests that it not only learns a good exemplar-response mapping but also leverages complementary information from both the pre-trained VLM and the TSM, offering enriched representations for maintaining basic conversation while excelling at specific tasks.

Comparison with instruction-tuned VLMs:
Although MedDr performs well in some cases, this is likely due to its large size (26B parameters) and training on more medical datasets.
Nonetheless, VITask with and without EP, despite having only 2B parameters, significantly outperforms MedDr on datasets like DermaMNIST, OCTMNIST, and OrganAMNIST.
This further underscores the effectiveness of VITask in boosting task-specific performance.
When comparing VITask to other VLMs tuned using vanilla visual instruction methods, its advantages become even more pronounced.
VITask with and without EP outperforms LLaVA-13B, the second-best instruction-tuned VLM, by an average of $8.6\%$ and $14.7\%$ in F1 score, respectively.
Furthermore, compared to InternVL-2B, which shares the same pre-trained VLM as VITask, our approach shows improvements in both accuracy and F1 score.
This reinforces that VITask’s enhancements are derived from its unique framework and strategies for task adaptation.

**Table 2: Ablation study of the proposed components. RDA represents Response Distribution Alignment, CRT denotes Contrastive Response Tuning, and EP stands for Exemplar Prompting.**
|  | Method | Chest | Derma | OCT | Retina | Tissue |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 |  |
| w/o EP | Vanilla | 0.523 | 0.024 | 0.770 | 0.499 | 0.726 | 0.704 | 0.590 | 0.370 | 0.569 | 0.419 |
| +RDA | 0.517 | 0.078 | 0.799 | 0.585 | 0.844 | 0.837 | 0.615 | 0.401 | 0.632 | 0.523 |  |
| +CRT | 0.513 | 0.088 | 0.786 | 0.593 | 0.817 | 0.810 | 0.593 | 0.413 | 0.622 | 0.505 |  |
| +Both | 0.513 | 0.102 | 0.810 | 0.633 | 0.853 | 0.846 | 0.625 | 0.457 | 0.643 | 0.538 |  |
| w/ EP | Vanilla | 0.514 | 0.118 | 0.863 | 0.725 | 0.951 | 0.950 | 0.608 | 0.489 | 0.760 | 0.689 |
| +RDA | 0.514 | 0.123 | 0.873 | 0.760 | 0.949 | 0.950 | 0.627 | 0.471 | 0.761 | 0.691 |  |
| +CRT | 0.513 | 0.122 | 0.878 | 0.774 | 0.949 | 0.950 | 0.623 | 0.509 | 0.762 | 0.691 |  |
| +Both | 0.517 | 0.129 | 0.877 | 0.772 | 0.952 | 0.952 | 0.632 | 0.522 | 0.761 | 0.690 |  |

#### Ablation Study.

In this section, we analyze the effectiveness of the three core components, exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT), through ablation studies to understand their individual contributions to the overall performance.
As shown in Table 2, when EP is disabled during inference, applying RDA improves the base model, InternVL-2B, by an average of $8.16\%$ in F1 score.
Similarly, CRT alone improves the base model by $7.86\%$ in F1 on average.
These results highlight that both RDA and CRT can independently boost task-specific performance.
When RDA and CRT are combined, we observe additional improvements in both accuracy and F1 score, indicating that these two strategies complement each other to achieve optimal performance.
When EP is used during inference, RDA does not yield notable gains.
This is expected, as RDA is primarily designed to enhance performance in the absence of exemplars during inference.
CRT, on the other hand, can still provide an improvement even with EP, but the margin of improvement is smaller.
This is likely because the exemplar-prompted features have already adjusted the response distribution, reducing the necessity for further fine-tuning via CRT.

#### Validation on External Datasets.

We further validate the external performance of instruction-tuned VLMs on the APTOS, IDRiD, and MESSIDOR datasets for diabetic retinopathy grading.
These datasets use the same instruction formatting as RetinaMNIST but were not included during instruction tuning.
We evaluated the TSM, vanilla instruction-tuned VLM, and VITask w/ EP models, all of which were trained on RetinaMNIST.
Additionally, we tested a variant of VITask, VITask${}^{\text{plug}}$, which uses a newly trained TSM on the external datasets, replacing the original TSM for VITask without further fine-tuning.
The results, as shown in Table 3, indicate that performance drops significantly for all models when tested on external datasets, highlighting the challenge of out-of-distribution generalization.
As expected, the TSM, optimized for the specific task, achieves the best external performance.
VITask is the second-best method, showing some generalization to external datasets.

**Table 3: Validation on external datasets.**
|  | ATPOS | IDIRD | Messidor |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| TSM | 0.593 | 0.377 | 0.398 | 0.316 | 0.584 | 0.263 |
| Vanilla | 0.523 | 0.291 | 0.417 | 0.223 | 0.587 | 0.212 |
| VITask | 0.456 | 0.336 | 0.379 | 0.262 | 0.521 | 0.321 |
| VITask${}^{\text{plug}}$ | 0.668 | 0.407 | 0.544 | 0.359 | 0.652 | 0.438 |

The vanilla VLM baseline achieved higher accuracy but lower F1 scores than VITask, likely due to the external datasets being biased with many normal cases, inflating accuracy.
VITask${}^{\text{plug}}$ outperformed other VLM-based methods, demonstrating VITask’s flexibility in adapting to different tasks without the need for retraining the entire model.

#### Robustness to Incomplete Instructions.

We also tested the robustness of instruction-tuned VLMs to incomplete instructions on the DermaMNIST dataset.
We modified the dataset by removing references to possible disease names from the original instructions, eliminating necessary context information and making the instruction-following task more challenging.
We then fine-tuned both the vanilla instruction-tuned VLM and VITask (with EP disabled for fairness) on this modified dataset.
As illustrated in Figure 4, the vanilla visual instruction-tuned model’s F1 score dropped dramatically from 0.531 to 0.423 when trained with incomplete instructions, showing that it heavily relies on detailed instructions for generating accurate responses.
In contrast, VITask showed only a slight decrease in performance, demonstrating much better robustness against incomplete instructions.
This resilience can be attributed to VITask’s ability to implicitly align the VLM’s response distribution with that of the TSM, providing a well-defined latent space that effectively characterizes desirable responses, even in the absence of detailed instructions.

Figure: Figure 4: Robustness to incomplete instructions.
Refer to caption: x4.png

#### Limitations and Discussions.

Our work has several limitations. Firstly, we primarily focus on image classification tasks, where training a single TSM for all tasks is straightforward.
However, for other instruction-following tasks, such as image captioning and VQA, training such a TSM may not be as feasible or effective.
Extending the VITask framework to these types of tasks remains a challenge and could be an avenue for future research.
Secondly, our experiments are limited to medical datasets. While the results demonstrate the effectiveness of VITask in the medical domain, testing across a broader range of domains would be necessary to fully validate its generalizability.
Exploring VITask’s applicability to datasets beyond the medical field is an important next step.
Lastly, we focus on task-specific training during the fine-tuning stage.
However, we believe that our method has the potential to enhance both the pre-training and fine-tuning phases of VLMs to achieve task-specific model-level performance.
Exploring VITask’s application to pre-training could lead to further improvements in adaptability and performance across diverse tasks.

## 6 Conclusion

In this paper, we proposed VITask, a novel framework that bridges task-specific models (TSM) and visual language models (VLM) to enhance task-specific adaptability and performance.
Through exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT), VITask leverages specialized task features from TSMs and aligns them with the instruction-following capabilities of VLMs.
Our experiments demonstrate that VITask outperforms both conventional instruction-tuned VLMs and TSMs across a variety of datasets, showcasing its ability to integrate complementary features from both models effectively.
VITask not only improves task-specific performance but also introduces practical advantages, such as flexibility in incorporating any TSM architecture in a plug-and-play manner, and robustness to incomplete instructions.
By decoupling image representation learning from instruction tuning, VITask offers an efficient and adaptable solution for new and unseen tasks without the need for extensive retraining.

## Appendix A Appendix

Figure: (a) Accuracy
Refer to caption: x5.png

### A.1 Dataset and Instruction Prompt.

We utilize the MedMNIST dataset collection Yang et al. () as our primary training and testing dataset for VLM, which comprises 12 distinct 2D datasets. Detailed descriptions of each dataset are provided below:

- •
PathMNIST Kather et al. (): Derived from the NCT-CRC-HE-100K dataset based on colorectal cancer histology slides, this dataset includes $100,000$ training image patches and $7,180$ test patches from a different clinical center, classified into $9$ tissue types for multi-class classification.
- •
ChestMNIST Kermany et al. (): Based on the NIH-ChestXray14 dataset, it comprises $112,120$ frontal-view chest X-ray images of $30,805$ unique patients, labeled with $14$ disease categories for multi-label classification.
- •
DermaMNIST Tschandl et al. (); Codella et al. (): Sourced from the HAM10000 dataset, a large collection of multi-source dermatoscopic images, it contains $10,015$ images categorized into $7$ different skin conditions for multi-class classification.
- •
OCTMNIST Kermany et al. (): Derived from a prior dataset on retinal optical coherence tomography (OCT) images, it comprises $109,309$ samples categorized into $4$ diagnostic classes for multi-class retinal disease classification.
- •
PneumoniaMNIST Kermany et al. (): Based on a collection of pediatric chest X-ray images, it includes $5,856$ samples for binary classification of pneumonia against normal cases.
- •
RetinaMNIST Liu et al. (): Developed from the DeepDRiD challenge dataset, this collection includes $1,600$ retina fundus images labeled for $5$-level diabetic retinopathy severity and formulated as an ordinal regression task.
- •
BreastMNIST Al-Dhabyani et al. (): Sourced from a dataset of $780$ breast ultrasound images, it is categorized into $3$ classes—normal, benign, and malignant—and simplified into binary classification for the current study.
- •
BloodMNIST Acevedo et al. (): This dataset features $17,092$ images of individual normal blood cells, categorized into $8$ classes based on cell type, for multi-class classification.
- •
TissueMNIST Ljosa et al. (): Developed from the BBBC051 dataset, it contains $236,386$ human kidney cortex cell images segmented into $8$ tissue types for classification.
- •
Organ{A,C,S}MNIST Bilic et al. (); Xu et al. (): Sourced from the Liver Tumor Segmentation Benchmark (LiTS) dataset, it contains 2D images obtained from 3D CT scans of $11$ body organs. The dataset is split into three separate views (axial, coronal, and sagittal), each forming a multi-class organ classification task.

The diverse array of datasets provides a solid foundation for evaluating our method across multiple biomedical imaging domains, supporting both binary and multi-class classification tasks.

We construct the instruction-response pairs for medical image classification following the approach outlined in previous work He et al. (). Specifically, to prepare the image classification data for ViTask training and testing, each dataset is converted into an instruction-tuning format by rephrasing the classification task as a question about the disease observed in the image, along with a set of possible disease options. The response corresponds to the correct disease name. The data construction template is shown below.

User:
Analyze the given {Modality} image.
The possible diagnoses are:{Label Set}.
VITask:
{Label}.

### A.2 Implementation details.

Training of Task-Specific Model for Exemplar Prompting.
For the task-specific model used in Exemplar Prompting, we employ a ViT Dosovitskiy et al. () base model pre-trained on ImageNet-21K Deng et al. (), and fine-tune it on the MedMNIST dataset for training, testing, and validation. We combine all 2D datasets in MedMNIST and jointly train the ViT model across all 70 classification tasks (i.e., using a shared classification head with 70 classes). During training, the loss is computed only over the class subset corresponding to the current sample’s dataset. We train the ViT model for 30 epochs and select the best model based on validation set performance.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See the comments
soundness: 2
strengths: 1.	The overall writing is relatively smooth and easy to understand. 
2.	The paper has made a detailed design in the method of constructing the dataset. 
3.	The paper has conducted detailed experiments on existing medical datasets.
summary: This paper proposes a VITask framework that enhances the adaptability of Vision Language Models (VLMs) for specific tasks by integrating Task-Specific Models (TSMs). It employs strategies like guiding VLMs with TSM features, enabling adaptation without TSMs during inference, and optimizing the ranking of correct image-response pairs. Experiments on medical diagnosis datasets demonstrate that VITask outperforms standard VLMs and TSMs, providing flexible integration and robustness to incomplete instructions.
weaknesses: 1.	The overall motivation is to finetune existing VLM for downstream tasks. After fine-tuning, the model can only be used for a specific task, which may hinder the original generalizability of VLM. The authors seem do not consider this issue.
2.	From the experiment, it can be observed EP is most useful, while other modules bring marginal improvements. Whether the other two modules are necessary requires to be further explored.
3.	Lisa [1] also adopts similar prompt tuning techniques like EP, and what’s the difference between these methods, and what is the technical advantage of EP?
4.	The comparison lacks comprehensiveness, as numerous medical VLMs exist in the field.
5.	The effectiveness of the proposed method is only demonstrated in the medical field, therefore the title is kind of over claimed. If the method is general, more experiments on other datasets are required.

[1] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579–9589, 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. For TSM, does it use the same encoder as the one used by the target VLM? How about the connector?
2. What parameters are tuned in RDA exactly? I also don’t get how is it really different from tuning from the ground truth labels, which are used anyway. In other words, isn’t p_theta(response | exemplar) just an approximation of the ground truth label? In fact, this pseudo-label is worse since it can be wrong? Also for classification tasks, if the model only decodes a single label, does it really matter to learn the whole distribution instead of an one-hot label? If we look at table 2, RDA indeed improves upon vanilla, but what does vanilla mean? Is it just the base VLM that’s not fine-tuned at all for those medical tasks? I need to see an experiment where you just fine-tune with the ground truth labels in the training set to really understand the role of RDA. Now I tend to think you can totally get rid of it. 
3. For both RDA and CRT, what did you do to deal with the situation when the label consists of more than 1 token due to tokenization?
4. I want to know the implementation detail of how you extract the answer from the generated output from the VLM. Apparently since it’s free-form generation the format might be slightly off. Are there any instances where the model outputs a semantically equivalent label that’s not an exact match? What’s the percentage of such instances before and after your fine-tuning?
5. I wonder if the authors can explain more on the motivation of adapting the VLM for classification tasks. I think this task is only meaningful if either 1) the original generative capabilities can be preserved, or 2) the fine-tuned VLM is actually better than SoTA vision models, otherwise we might as well use a predictive classifier that’s smaller and easier to fine-tune. Thus, I think the authors should either 1) present evaluations on non-medical tasks to show there’s very little degradation, or 2) compare against the SoTA vision model, which is probably not a ViT pretrained in 2020.
soundness: 3
strengths: The paper states the problem it aims to solve clearly. The proposed method is validated on a wide range of medical datasets.
summary: This paper proposes a combination of strategies, exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT), to adapt general-purpose generative VLMs for specialized medical classification tasks. Through experiments, the authors show that their method VITask , leveraging these three algorithmic components, can outperform task-specific models (TSMs), which are essentially fine-tuned ViTs, on a variety of medical classification tasks. In ablation experiments, the authors demonstrate that EP contributes the most performance boost while RDA and CRT are also complementary.
weaknesses: The biggest issue of the paper is the lack of depth. While it ablates the impact of each of the algorithmic components, they authors spent little effort trying to understand why each of them work and to compare them against existing methods.

1. It’s not clear what makes EP successful. 
    - I strongly suspect the performance gain is mostly due to the fine-tuning of the connector module. The critical experiment of simply having both the connector and the LLM (LoRA params) trainable is missing.
    - Additionally, an experiment comparing EP with prefixing tuning [1] will tell whether it’s necessary to condition the prefix (additional tokens to the LLM’s embedding space) on the image at all to get good performance. Essentially, I need to see experiments showing me EP > fine-tuning the original VLM’s connector + prefix tuning to be convinced it’s novel. 
    - I also don’t buy the claim that fine-tuning the Vision model in VLM will distort vision language alignment at all. If fine-tuning the Vision model is harmful, wouldn’t the trained LoRA weights be more harmful as well? A controlled experiment where the vision encode is also trained is needed. I am confident this will make EP perform even better. 
    - Finally, other works with the same core methodology should be discussed. For example, Graph Neural Prompting [2] builds a knowledge graph based on the prompt and multiple choice candidates and generates a conditional prefix to prompt the LLM. I think the idea is extremely similar to EP. 
2. Regarding RDA: this is essentially a fancy way of saying knowledge distillation but no relevant papers are cited. Regarding implementation, the author mentions gradient detachment. If I understood it correctly, this just means the TSM, or the “teacher”, is not trained while the goal is to train the student. Shouldn’t this be the default setting anyway? 
3. Contrastive Response Tuning: as part of the core methodology, the paper should compare its effectiveness against existing methods, such as contrastive decoding [3][4]. 

Issues mentioned above should be addressed. Otherwise this work should aim for a more application-oriented venue. 

The notations issues. 

- In equations (1), (2), (3), (5), (6), why is there a min() operator on the left hand side? The author seems to mix it up with the argmin notation. I think the author should remove the min() and avoid argmin() like notation since not all parameters are trained. 

Minor grammar issues
- For example, Takeaway #1: TSM features can prompts (prompt) VLMs to generate desired responses.

References:

[1] Li, Xiang Lisa, and Percy Liang. "Prefix-Tuning: Optimizing Continuous Prompts for Generation." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
[2] Tian, Yijun, et al. "Graph neural prompting with large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.
[3] Leng, Sicong, et al. "Mitigating object hallucinations in large vision-language models through visual contrastive decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[4] Favero, Alessandro, et al. "Multi-modal hallucination control by visual information grounding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Authors should explain why using Vision-Language Models (VLMs) with a high computational budget is more beneficial than improving Task-Specific Models (TSMs). I would recommend an empirical analysis of the cost and performance benefits of VITask versus TSMs to.

2. TSMs provide better task-specific representations than general vision encoders. Could you clarify why the results decline significantly when embeddings are completely replaced by task-specific representations? Did you utilize the CLS token from Vision Transformers (ViT) or the entire patch embeddings?

3. Why do we need to optimize Stage 2 with the same loss functions used in Stage 1 without scaling them? After optimizing the Stage 1 model to minimize the Stage 1 loss, using it in Stage 2 could potentially harm the fine-tuning process. Have you considered applying a scale to the Stage 1 loss functions?

4. In line 423, it is mentioned that a novel vision-language connector is introduced in the paper, but no details about this connector are provided in the text. Could you please include more information about it?
soundness: 3
strengths: The motivation of the paper is clear and the authors provided adequate evidence to show why the proposed method can be useful in task-specific applications.

The task-specific instruction tuning section is quite interesting and the papers validate how each proposed component improves the performance of VLMs.

The paper has a detailed section providing intuition and empirical evidence to support the claims.
summary: Although VLMs have excelled in understanding generic images, they fall apart for out-of-distribution task-specific applications. This paper claims that this could be due to the lack of comprehensive image representation and indirect tuning objective VLMs used to optimize. This paper proposes a framework that leverages representation from task-specific models as complementary information and optimizes the VLMs. Additionally, it explores how the inference can be done without task-specific models using two proposed methods RDA and CRT.
Considering the high costs of fine-tuning and inference with LLMs, along with the minimal performance gains, why should a user prefer using VITask over enhancing TSMs?

The paper is well-written but could be improved for better readability.
weaknesses: As mentioned in the limitations section, exploration of the method is highly limited to classification tasks. Additional tasks would be more interesting using an LLM, which has rich information embeddings.

Given the high computation budget and a marginal improvement in performance compared to TSMs, I would try to improve the TSMs which are cost-effective for training and inference. ( I consider the results as marginal improvement, the considered dataset is comparably simpler than the actual applications.) For example, optimize the model architecture, focused hyperparameter selection, Select and Fuse futures, and an ensemble of smaller models.

The paper is comprised of a lot of detailed text, which also seems to be a weakness of the paper. Authors could provide more visualization to interpret the results and behaviour of each module. Overall, the interpretation of results could be improved for better readability. You may provide more visualization to explain the benefits of the proposed modules or interpretation analysis how the proposed module benefits  compared to TSMs or vanilla VLMs.

The paper presents a few concerns :

1. The figure related to the RDA analysis referenced in Lines 264-266 is missing. As it stands, the current figure appears to be a bar chart of the EP results. Could please provide the figure?

2. The authors should consider using different formatting styles for equations (3-6). When the terms in the equations are presented in the same style as the surrounding text, it can be confusing (Line 280). Differentiating the formatting will enhance readability.

3. There are spelling errors in Figure 1 (d).

4. The word "often" is repeated in Line 041.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: As you have mentioned in the paper, testing this approach only on medical domain is not enough to make sure your method is good. I hope that you can give more experiment results on other non-medical domains such as natural images.
soundness: 3
strengths: - The idea and strategies used to combine TSMs to VLMs to boost performance of VLMs (without using TSMs to VLMs on inference stage based on RDA mechanism) on task-specific applications are interesting. It helps to remove the limitation of VLMs when fine-tuning on task-specific applications.
- The performance of experiments on medical image classification are good.
- The paper is well-written and easy to follow.
summary: This paper first investigate why fine-tuned (VLMs) often lag behind TSMs in performance (use image classification as a case study) and then propose a new framework that can enhance the performance of VLMs on task-specific applications. They conduct it through integrating task-specific models (TSM) into VLMs in fine-tuning with 3 strategies, exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT). EP allow TSM features to guide VLMs while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs by maximizing likelihood of correct image-response pairs while minimizing that of  incorrect pairs. They conduct two-stages training. The first stage is fine-tuning VLMs using vanilla visual instruction tuning in conjunction with EP and RDA. The second stage is fine-tuning VLMs using all strategies. The experiments focus on classification tasks on medical domain.
weaknesses: - The current experiment results are good, but the number of experiments and baseline models are insufficient, only 4 baseline models (including task-specific model). Can you compare your method with more other baselines, including some works you have mentioned in related works section which attempt to integrate VLMs with task-specific models and more VLMs such as InstructBLIP, BLIP-2, and Qwen-VL-Chat? 
- Because you did classification experiments on medical domain and there are currently many VLMs pre-trained on medical datasets, why you chose to test on the particular VLMs used in the study. In my opinion, it is important to test this approach on more VLMs which were pre-trained on medical datasets such as LLaVA-Med, Biomed GPT, etc. to make sure applying this method on these VLMs give significant improvement results.
- I am concerned about the time used to fine-tune a VLM using your method, because you need to fine-tune a task-specific model on the dataset first before combining it with VLMs and then fine-tune the VLMs. Can you give me more detail about the resource you use to fine-tune VLMs and the total time for the training process to finish using your method and vanilla methods (comparisons between your method and baseline approaches) ?

Meta-review: The paper introduces VITask, a framework for adapting vision-language models (VLMs) to task-specific applications using exemplar prompting, response distribution alignment, and contrastive response tuning. While reviewers appreciated the clarity of the methodology and its strong performance on specific tasks, they highlighted several significant concerns. The title's claim of generality is hyperbolic, as the original experiments were limited to an extensively curated set of medical datasets that primarily involve an array of relatively straightforward classification tasks, which do not fully validate the model's broader applicability. The additional tasks presented in the rebuttal, such as medical phrase grounding and natural image classification, lacked comprehensive baselines and did not adequately support the broader claims of the paper. Moreover, since the substantial changes made during the rebuttal were not incorporated into the main paper, the revision may turn into something beyond a simple revision. Therefore, it is difficult to recommend the paper to be accepted in this state.

Decision: Reject