FREE: Fast and Robust Vision Language Models with Early Exits
Authors: Divya Jyoti Bajpai, Manjesh Kumar Hanawal
Accepted at: acl25

FREE: Fast and Robust Vision Language Models with Early Exits

Divya Jyoti Bajpai and Manjesh Kumar Hanawal
Department of IEOR, IIT Bombay
{divyajyoti.bajpai, mhanawal}@iitb.ac.in

Abstract
In recent years, Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-Language tasks. However, their large size poses challenges for real-world applications where inference latency is a concern. To tackle this issue, we propose employing Early Exit (EE) strategies in VLMs. However, training exit classifiers in VLMs is challenging, particularly with limited labeled training data. To address this, we introduce FREE, an adversarial training approach within a GAN-based framework. Here, each exit consists of a transformer layer and a classifier. The transformer layer is adversarially trained to produce feature representations similar to the final layer, while a feature classifier serves as the discriminator. Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance. Experimental results demonstrate the effectiveness of our approach in enhancing accuracy and model robustness by mitigating overthinking and the phenomenon of mid-crisis that we highlight. We experimentally validate that our method speeds up the inference process by more than \(1.51 \times\) while retaining comparable performance. The source code is available at https://github.com/Divy290/BLIPEE.

1 Introduction
Vision-language pre-training (VLP) has evolved significantly with the emergence of sophisticated pre-trained Vision Language Models (VLMs). These models have consistently pushed the performance boundaries across various vision-language tasks. However, their demanding computational requirements and inference latency pose challenges for real-time applications. Several models, such as BLIP-2 (Li et al., 2023), MiniGPT (Zhu et al., 2023) etc., leverage off-the-shelf large-scale pre-trained models as building components with their parameters frozen. This reduces VLMs training parameters but makes them extremely slow during inference, as highlighted in (Bajpai and Hanawal, 2024a) leading to higher inference time.

The use of the Language Model (LM) component with frozen parameters not only makes VLM susceptible to overthinking but also to another phenomenon that we term mid-crisis (Elhoussi et al., 2024). This phenomenon occurs when intermediate layers suffer performance drops due to the search for irrelevant features. While initial layers capture shallow representations and syntactic information, and deep layers learn semantic-fusion relations (Fei et al., 2022), intermediary layers tend to capture dataset patterns that degrade their performance, even losing the information learned by initial layers, and the model has to regain the lost information again in deeper layers.

We illustrate this phenomenon in the left figure of Fig. 3 and Sec. 3.1, showcasing accuracy on the VQAv2 (visual question answering) (Das et al., 2017) dataset across different exits when trained with dedicated classifiers as in vanilla early exit setup for the intermediate layers. This raises the question: how can we mitigate mid-crisis and overthinking to enhance the accuracy and efficiency of VLMs?

We address this using Early Exit (EE) techniques (Xin et al., 2020; Zhou et al., 2020; Zhu, 2021), an input-adaptive method that reduces computational costs by bypassing certain layers while preserving the knowledge encoded in large-scale models. Since real-world datasets contain both “easy” and “hard” samples, EE ensures adaptive computation per sample.

However, applying EE to VLMs presents challenges: (1) Exit classifiers introduce significant overhead—e.g., a single exit for OPT_{2.7B} (Zhang et al., 2022a) decoder adds approximately 130M parameters; (2) Training these exits requires sub-
Figure 1: This figure provides some example outputs of FREE ViT-g OPT_{2.7B}. The different colours show the difficulty levels of tokens in captions. Red: Easy to predict and predicted at initial (1-12) layers. Green: Mediocre hard, predicted at intermediary (13-24) layers. Black: Hard to predict, predicted at deeper (25-32) layers.

stantial labeled data, limiting EE adoption in zero-shot VLMs that require minimal fine-tuning.

We propose FREE Fast and Robust Vision-Language Models with Early Exits. This efficient EE training framework minimizes training costs while maintaining accuracy. Our method employs a Generative Adversarial Network (GAN)-based (Creswell et al., 2018) framework, leveraging the pre-trained VLM outputs to align feature representations between intermediate exits and the final layer. Unlike cosine similarity, which hampers generalization, the adversarial setup improves feature consistency, ensuring robust exit predictions.

Our method attaches exits and Feature Classifiers (FCs) to intermediary layers of the decoder of the VLM. Each exit consists of a single exit transformer (ET) and an exit classifier (EC). The exit transformer is a replica of the layers in the decoder LM of the VLM. They are used as generators and feature classifiers as discriminators in a GAN-based setup, as shown in Fig. 2. The task of the feature classifier is to correctly classify if the input is from the exit or final layer, and the task of the exit transformer is to generate representations similar to the final layer to fool the feature classifier of that exit.

As the exit transformer is trained to generate representations similar to that of the final layer, we can use the final layer classifier at all exits as an EC with frozen parameters, which are used to map the outputs of the exit transformer to class probabilities. As the size of ETs is significantly smaller than that of ECs, it substantially reduces the number of parameters. In this way, a single LM layer attached to the exits helps produce similar feature representations and reduces the training parameters by utilizing a frozen final layer classifier. By attaching EEs, our method reduces the chances of mid-crisis or overthinking and makes the inference process faster. Fig. 1 shows how our method can speed up inference while maintaining comparable performance by using the EE methods.

Adversarial training methods are prone to catastrophic forgetting (Kirkpatrick et al., 2017; Ryu et al., 2022) and mode collapse, which can hinder exit training. To address these challenges, we propose both supervised and unsupervised strategies. In the supervised setting, when a small
Figure 2: This figure illustrates our model’s architecture. The Q-Former output and previous tokens pass through the backbone. The final layer and ET outputs go to feature classifiers, refining ET to match final representations. During inference, the final classifier is used, and confident exits generate the caption.

labeled dataset is available, we use hard labels to stabilize training, preventing catastrophic forgetting and avoiding local optima. In the unsupervised case, when labeled data is unavailable, we utilize soft labels from vanilla VLM or generate high-quality synthetic labels using CapFilt (Li et al., 2022b), as employed in BLIP and BLIP-2.

Our method remains effective across different scenarios. When a labeled dataset is available, it is used to mitigate catastrophic forgetting and mode collapse. If no labeled data is present, we employ knowledge distillation to solve this issue. When both labeled data and annotations are unavailable but computational resources are accessible, we generate synthetic labels using CapFilt. By providing solutions for all data constraints, our approach ensures stable and effective exit training while enhancing efficiency and generalization. In summary, our contributions are as follows:

• We introduce an EE strategy named FREE for VLMs to effectively mitigate inference latency by reducing unnecessary computations inherent in their large-scale architecture.

• We propose an efficient training strategy to improve performance at EE classifiers. FREE emulates the behaviour of the final layer at the exits through adversarial learning. This reduces the need for labeled training datasets.

• Our model reduces the number of trainable parameters of the exits by reutilizing the frozen final layer classifier at the exits.

• We experiment both qualitatively (see Fig. 1 (recommended)) and quantitatively on various tasks such as image captioning, visual question-answering and visual dialogue dataset. Our method provides inference speed \( > 1.51 \times \) with comparable accuracy than vanilla VLM inference. We show the robustness of our method in Appendix A.4.

2 Related works

We discuss the VLPs with LM components and EE strategies related to our work below.

Vision-language Pre-training: Different model architectures have been proposed for specific downstream tasks in VLPs, including dual-encoder architectures (Radford et al., 2021; Jia et al., 2021), encoder-decoder architectures (Cho et al., 2021; Wang et al., 2021; Chen et al., 2022b). Various pre-training objectives have also been introduced, such as image-text contrastive learning (Radford et al., 2021; Yao et al., 2021; Li et al., 2021, 2022b), the image-text matching (Ju et al., 2021; Li et al., 2022b; Bao et al., 2021), and masked language modeling (Li et al., 2021, 2022b; Yu et al., 2022; Wang et al., 2022b). However, these end-to-end models are inflexible to leverage readily available pre-trained models, such as LLMs (Brown et al., 2020; Zhang et al., 2022a; Chung et al., 2024).

Recent approaches in vision-language pre-training have adopted the strategy of utilizing off-the-shelf pre-trained models and keeping them frozen during training. Some methods freeze only the image encoder (Chen et al., 2020; Li et al.,
2020; Zhang et al., 2021), and recent LiT (Zhai et al., 2022) which uses a frozen pre-trained encoder for CLIP (Radford et al., 2021) pre-training, while others freeze the language model to leverage knowledge from language-only pre-trained models for vision-to-language generation tasks (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Chen et al., 2022a; Mañas et al., 2022; Tiong et al., 2022; Guo et al., 2022). The primary challenge lies in aligning visual features with text space. To address this challenge, techniques like Frozen (Tsimpoukelli et al., 2021) finetune image encoders or insert new cross-attention layers into language models to incorporate visual features. BLIP-2 (Li et al., 2023) employs both frozen image encoders and language models for vision-language tasks, achieving strong performance.

Early Exits: To minimize inference latency in deep neural networks, BranchyNet (Teerapittayanon et al., 2016) introduced attaching exits classifiers at intermediary layers. This concept was extended by Shallow-deep (Kaya et al., 2019), which effectively determines when to exit based on confidence distribution at each exit classifier. Approaches like (Huang et al., 2017; Yang et al., 2020; Han et al., 2023) improve EE for image tasks by dynamically choosing different depths for different regions of the image. Approaches like (Phuong and Lampert, 2019) have employed knowledge distillation for image classification.

For NLP tasks, several early exit frameworks have emerged (Xin et al., 2020; Liu et al., 2021; Zhou et al., 2020; Liu et al., 2020; Wang et al., 2019, 2020; Zhu, 2021; Ji et al., 2023; Zhang et al., 2022b; Bajpai and Hanawal, 2024b; Bajpai et al., 2023, 2024), primarily built on the BERT backbone. DeeCap (Fei et al., 2022) introduces EE for image captioning tasks, employing an imitation network to replicate outputs from transformer layers. MuE (Tang et al., 2023) applies EE to OFA (Wang et al., 2022a), a VLM tailored for multi-modal applications. CapEEN (Bajpai and Hanawal, 2024a) makes the EEs robust to noise by adapting to the exiting threshold.

The key differences in our work are: 1) We employ adversarial training for efficient learning of EE models. 2) Our method can work under both supervised and unsupervised setups by utilizing the zero-shot capabilities of the VLMs, while previous methods require a good amount of high-quality labeled training data.

3 Methodology

We begin with a pre-trained Vision-Language Model (VLM) comprising three key components: a visual encoder, a projection layer, and a language model (LM). The visual encoder processes the input image, which may be a standalone image encoder (Wang et al., 2022a) or include an image-grounded text encoder (e.g., Q-Former in BLIP-2). The projection layer then transforms the visual features to align with the LM’s input space which is passed as an input to the decoder. As autoregressive decoding requires multiple forward passes of LM, our focus will primarily be on optimizing the LM component for efficient inference.

We assume that LM consists of \( N \) layers, and we attach exits to the \( K \) chosen layers. Each exit consists of one Exit Transformer (ET) layer with the same configuration as one LM layer and an Exit Classifier (EC). The exit layer is such the parameters of the ET are trainable, and those of EC are frozen. Before indulging into architectural details, we motivate our method by highlighting the major concerns on the well-known BLIP-2 model.

3.1 Motivation

In Fig. 3, we show the VQA accuracy for various layers on the VQAv2 dataset on the BLIP-2 model. In the left side of the figure, we show the performance of vanilla EE methods where just an EC is attached to the intermediate layer output. As seen, performance dips at the middle layers after the initial improvement. This is due to the LM component, which is of large size and kept frozen during training. During pre-training, BLIP-2 aligns the features of text and images using the Q-former, the querying transformer. However, the Q-former provides the image-grounded text embeddings to the LM component in such a way that it produces high-quality predictions only at the final layer and not the intermediary layer.

To address this, we enhance exits by incorporating a transformer layer instead of relying solely on an EC. This layer mimics the final layer’s behavior, enabling intermediate exits to access deep-level representations. By doing this, we are not directly passing the information of intermediate layers to the classifiers instead we first enhance the information through ETs and then pass the hidden representations to classifiers which improves the performance of exits. The right side of Fig. 3 demonstrates how our approach reduces mid-crisis
Figure 3: Left: VQA accuracy on the VQAv2 dataset with BLIP-2-ViT-g-FlanT5XL showcasing mid-crisis and overthinking. Right: Our training process improves the mid-crisis. Overthinking can then be solved using EE.

by leveraging deeper layer information. Overthinking is further alleviated by performing inference through EEs using thresholds.

Previous EE methods that attach a trainable classifier at exits introduce substantial parameter overhead, given the large vocabulary size. For instance, a single classifier added to BLIP-2 with OPT_{2.7B} contributes 130\( M \) trainable parameters, and 7 exits would scale this up to 900\( M \) parameters. In contrast, our method only trains the LM layer at each exit, adding 63\( M \) parameters per exit and 588\( M \) for 7 exits—reducing the trainable parameters by approximately 52% (see Appendix D.1). This efficiency is achieved by using the final layer classifier and training only the LM layer. These issues also translate to other VLMs such as MiniGPT (Zhu et al., 2023) and InstructBLIP (Dai et al., 2024) due to similar architectures.
We next discuss our method which consists of two parts: 1) backbone finetuning and 2) exit training.

3.2 Backbone finetuning

The backbone is fine-tuned using the cross-entropy loss between the predicted token and the ground truth token. The loss function for fine-tuning could be formulated as:

\[
\mathcal{L}(I) = \sum_{t=1}^{T} \log P_N(y_t^*|y_{1:t-1}^*, I)
\]

where \( I \) denotes the input image, \( y_{1:T}^* \) is the true caption and \( T \) is the caption length. \( P_N \) denote the probability output from the final layer. In this step, the backbone learns to produce high-quality features at the final layer and a classifier \( C_N \) to map the feature representations at the final layer to class probabilities. Note that \( C_N \) is part of the backbone. Once the fine-tuning is complete, we freeze the parameters of the backbone. This is done to maintain the optimality of the backbone.

3.3 Exits training

After fine-tuning the backbone, we attach \( K \) exits to the LM component of the VLM. We denote the set of exit indices by \( [K] \). At each exit, we use a feature classifier \( D_i \) that discriminates the feature representations of the transformer layer of the \( i \)th exit from that of the final layer. Specifically, it provides a score to an input feature representation, whether it is from the final layer. We have a separate feature classifier for every exit as feature representations at different exits can differ.

In our setup, the feature classifier acts as a discriminator, and the exit transformer layer as a generator; the goal of the transformer layer is to generate feature representations similar to that of the final layer. We train them alternately as in the original GAN framework. This training problem can be set up as an unconstrained optimization problem. Let \( E_i \) denote the transformer layer in the \( i \)th exit. The feature classifier loss for an exit \( i \in [K] \) and an input image \( I \) could be formulated as:

\[
\mathcal{L}_i^{fc}(h_t^i, h_t^N | y_{1:t-1}^*, I) = -\log D_i(h_t^i | y_{1:t-1}^*, I) \\
-\log(1 - D_i(E_i(h_t^i | y_{1:t}^*, I)))
\]

where \( h_t^i \) is the feature (hidden) representation at \( i \)th layer and \( h_t^N \) is the feature representation at the final layer of the LM for the \( t \)th token in the sentence. The overall loss across all exits could be written as \( \mathcal{L}^{fc} = \sum_{i \in [K]} \mathcal{L}_i^{fc} \). It simultaneously updates the feature classifiers across all the exits.

The generator loss for the transformer layer in \( i \)th exit could be formulated as:

\[
\mathcal{L}_i^{gen}(h_t^i|y_{1:t-1}^*, I) = -\log D_i(E_i(h_t^i|y_{1:t-1}^*, I)))
\]

However, because the weights of the transformer layer of the exits are untied from the original backbone, they can face the issue of catastrophic for-
getting or mode collapse. To circumvent these issues, we utilize the small labeled dataset to fine-tune the backbone. It guides the model to correct the learning trajectory and not let it get stuck to the local minima. The labeled data not only removes the issue of catastrophic forgetting but also helps in reducing overthinking as exits mimic the final layer and learn from the hard labels. The loss could be written as:

\[
\mathcal{L}_i^{CE}(I, y_{1:t-1}^*) = \log P_i(y_t^*|y_{1:t-1}, I)
\]

where \( P_i \) denotes the probability score at the \( i \)th exit. The loss at exit \( i \) will be \( \mathcal{L}_i = \mathcal{L}_i^{CE} + \mathcal{L}_i^{gen} \). The overall loss for exits training will be \( \mathcal{L} = \sum_{i \in [K]} \mathcal{L}_i \). After this step, the backbone is learned with attached exits.

3.4 Unsupervised setup

Recall from the previous section that labeled data was utilized to reduce the issue of catastrophic forgetting and mode collapse. As the VLMs has good zero-shot performance, we can utilize it to either distill the knowledge at the final layer or create a small set of pseudo labels to fulfill the requirements of the labeled dataset. We provide two methods for unsupervised learning.

Using Knowledge Distillation: In this case, we can directly utilize the soft labels from the final layer to distill the knowledge to the exits. The knowledge distillation loss for the \( i \)th layer could be formulated as:

\[
\mathcal{L}_i^{KL} = KL(p_i^t, p_i^N)
\]

where \( p_i^t = C_N(E_i(h_i^t|y_{1:t-1}^*, I)) \), \( p_i^N = C_N(h_i^N|y_{1:t-1}^*, I) \) and \( KL \) is the KL-divergence loss defined as \( KL(p_i^t, p_i^N) = \sum_{v \in \mathcal{V}} p_i^t \log \frac{p_i^t}{p_i^N} \) where \( \mathcal{V} \) is the vocabulary. We can train the exits by replacing the \( \mathcal{L}_i^{CE} \) by \( \mathcal{L}_i^{KL} \). Then the overall loss for exit \( i \) is \( \mathcal{L}_i = \mathcal{L}_i^{KL} + \mathcal{L}_i^{gen} \).

This method also utilizes the zero-shot capabilities of the VLMs model. However, this method has a slightly lower performance than the CapFilt method proposed next, still, it comes with lower computational cost and has comparable performance to vanilla VLM inference.

Using CapFilt: CapFilt (Li et al., 2022b, 2023) is a method that is used in both the original BLIP and BLIP-2 models to generate high-quality synthetic captions. We use similar ideas to generate the labeled dataset. In this step, a sample is passed through the BLIP-2 model, which then provides us with 10 possible captions for the given samples. We then use the CLIP ViT-L/14 model to rank the synthetic captions based on the image-text similarity produced by the CLIP model. We then keep the top-2 captions and keep them as synthetic captions that can be later utilized for training the exits by treating the synthetic captions as true captions. Creating synthetic captions using the CapFilt is more accurate but computationally heavy (Li et al., 2022b). Hence it is recommended when the computational resources are available.

3.5 Inference

We perform captioning in an autoregressive manner. This entails making a token-by-token prediction for a given image, where the layer at which the token is predicted is determined by the confidence score \( S_i = \max_{v \in \mathcal{V}} C_N(E_i(h_i^t|y_{1:t-1}, I))(v) \) where \( \mathcal{V} \) is the vocabulary. The input to the decoder is processed sequentially through the decoder layers until the confidence score \( S_i \) is greater than a predefined threshold value \( \alpha \). The inference starts with the begin of the sentence token and the next token is predicted either at the exits or at the final layer. The inference process stops when the end of the sentence token is predicted either at intermediary layers or at the final layer. Note that if the prediction confidence is below \( \alpha \) for all the exits, then the sample is predicted at the final layer, irrespective of the confidence in the prediction.

4 Experiments

Dataset and Metric: We evaluate the performance of our method using the COCO (Lin et al., 2014) and NoCaps dataset (Agrawal et al., 2019) for image captioning. For Visual Question-answering tasks, we utilize the VQAv2 (Goyal et al., 2017), OK-VQA (Marino et al., 2019) and GQA (Hudson and Manning, 2019) datasets. For visual dialogue, we use the VisDial (Das et al., 2017) dataset. We report key metrics including BLEU-4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al., 2016) scores for captioning. For VQA tasks, we report the VQA accuracy and for the Visual Dialog, we use the Mean Reciprocal Rank (MRR) (Dai et al., 2024). To effectively consider the speedup, we define the speedup as inverse of the fraction of parameters used for inference on average. The
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">#Train Params</th>
    <th colspan="6">NoCaps Zero-shot</th>
  </tr>
  <tr>
    <th>in-domain</th>
    <th></th>
    <th>near-domain</th>
    <th>out-domain</th>
    <th>full-dataset</th>
    <th>Spd</th>
  </tr>
  <tr>
    <td>Vin VL</td>
    <td>345M</td>
    <td>102.9</td>
    <td>14</td>
    <td>94.8</td>
    <td>13.7</td>
    <td>88.1</td>
    <td>11.9</td>
    <td>95.1</td>
    <td>13.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP</td>
    <td>446M</td>
    <td>114.9</td>
    <td>15.2</td>
    <td>110.6</td>
    <td>14.6</td>
    <td>114.8</td>
    <td>14.3</td>
    <td>112.8</td>
    <td>14.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SimViLM</td>
    <td>1.4B</td>
    <td>113.7</td>
    <td>14.9</td>
    <td>110.6</td>
    <td>14.2</td>
    <td>114.6</td>
    <td>14.4</td>
    <td>112.1</td>
    <td>14.3</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP-2 ViT O<sub>2.7B</sub></td>
    <td>1.1B</td>
    <td>123.0</td>
    <td>15.8</td>
    <td>117.8</td>
    <td>15.4</td>
    <td>123.2</td>
    <td>15.0</td>
    <td>119.6</td>
    <td>15.4</td>
    <td>1.07×</td>
  </tr>
  <tr>
    <td>BLIP-2 ViT FT5<sub>XL</sub></td>
    <td>1.1B</td>
    <td>123.7</td>
    <td>16.3</td>
    <td><b>120.2</b></td>
    <td><b>15.9</b></td>
    <td>124.8</td>
    <td>15.1</td>
    <td>121.6</td>
    <td>15.8</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <th colspan="11">Early Exit models</th>
  </tr>
  <tr>
    <td>DecBLIP</td>
    <td>1.8B</td>
    <td>115.2</td>
    <td>15.3</td>
    <td>111.5</td>
    <td>14.7</td>
    <td>115.4</td>
    <td>14.5</td>
    <td>112.4</td>
    <td>14.5</td>
    <td>1.41×</td>
  </tr>
  <tr>
    <td>PABEE-BLIP</td>
    <td>1.8B</td>
    <td>117.7</td>
    <td>15.4</td>
    <td>114.2</td>
    <td>14.8</td>
    <td>117.6</td>
    <td>14.7</td>
    <td>112.9</td>
    <td>14.6</td>
    <td>1.29×</td>
  </tr>
  <tr>
    <td>LeeBLIP</td>
    <td>1.8B</td>
    <td>119.4</td>
    <td>15.5</td>
    <td>115.8</td>
    <td>14.8</td>
    <td>120.1</td>
    <td>14.9</td>
    <td>116.3</td>
    <td>15.1</td>
    <td>1.38×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>1.8B</td>
    <td>118.1</td>
    <td>15.4</td>
    <td>115.3</td>
    <td>14.8</td>
    <td>118.7</td>
    <td>14.8</td>
    <td>114.8</td>
    <td>14.9</td>
    <td>1.44×</td>
  </tr>
  <tr>
    <td>FREE ViT O<sub>2.7B</sub></td>
    <td>1.5B</td>
    <td>122.7</td>
    <td>15.7</td>
    <td>118.1</td>
    <td>15.5</td>
    <td>123.9</td>
    <td>15.1</td>
    <td>119.9</td>
    <td>15.6</td>
    <td>1.63×</td>
  </tr>
  <tr>
    <td>FREE ViT FT5<sub>XL</sub></td>
    <td>1.4B</td>
    <td><b>124.3</b></td>
    <td><b>16.5</b></td>
    <td>120.0</td>
    <td><b>15.9</b></td>
    <td><b>125.5</b></td>
    <td><b>15.4</b></td>
    <td><b>122.7</b></td>
    <td><b>16.1</b></td>
    <td>1.51×</td>
  </tr>
</table>

Table 1: Results on the Nocaps dataset during zero-shot transfer when the model is trained on the COCO dataset across various domains. Spd is the speedup achieved by the model. O<sub>2.7B</sub> is OPT<sub>2.7B</sub> and FT5<sub>XL</sub> is FlanT5<sub>XL</sub>.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">#Train Params</th>
    <th colspan="2">VQAv2</th>
    <th colspan="2">VQAv2</th>
    <th>Spd</th>
  </tr>
  <tr>
    <th>train</th>
    <th>test</th>
    <th>train</th>
    <th>test</th>
    <th></th>
  </tr>
  <tr>
    <th colspan="7">Without Exits</th>
  </tr>
  <tr>
    <td>ALBEF</td>
    <td>314M</td>
    <td>72.5</td>
    <td>71.5</td>
    <td></td>
    <td></td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP</td>
    <td>385M</td>
    <td>73.9</td>
    <td>72.1</td>
    <td></td>
    <td></td>
    <td>-</td>
  </tr>
  <tr>
    <td>OFA</td>
    <td>930M</td>
    <td>75.7</td>
    <td>75.6</td>
    <td></td>
    <td></td>
    <td>-</td>
  </tr>
  <tr>
    <td>Flamingo80B</td>
    <td>10.6B</td>
    <td>77.9</td>
    <td>78.1</td>
    <td></td>
    <td></td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP-2 V-O</td>
    <td>1.2B</td>
    <td>78.3</td>
    <td>78.5</td>
    <td>1.07×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>BLIP-2 V-F</td>
    <td>1.2B</td>
    <td>78.8</td>
    <td>78.7</td>
    <td>1.00×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th colspan="7">Early Exit models (on BLIP-2)</th>
  </tr>
  <tr>
    <td>DecBLIP</td>
    <td>1.9B</td>
    <td>75.4</td>
    <td>75.9</td>
    <td>1.52×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PABEE-BLIP</td>
    <td>1.9B</td>
    <td>77.4</td>
    <td>77.1</td>
    <td>1.39×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LeeBLIP</td>
    <td>1.9B</td>
    <td>78.1</td>
    <td>77.8</td>
    <td>1.65×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>FREE-V-O</td>
    <td>1.6B</td>
    <td>78.7</td>
    <td>79.0</td>
    <td>1.77×</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>FREE-V-F</td>
    <td>1.5B</td>
    <td><b>78.9</b></td>
    <td><b>79.1</b></td>
    <td><b>1.71×</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Results of semi-supervised application of our model to Visual-Question Answering tasks.

speedup is formulated as:

\[
\text{Speedup} = \frac{\text{Total parameters}}{\text{Average number of parameters used}}
\]

where the average number of parameters could be written as \( \frac{1}{M} \sum_{l=1}^{M} \sum_{i=1}^{N_l} w_i \times (i + k) \times p \) where \( p \) denotes the number of parameters in one layer of the LM component, \( N_l \) denotes the number of predicted words in the caption for the image \( I \), \( M \) denotes the total number of input images and \( k = 1 \) is the number of LM layers in the exit. Total parameters denote the total number of parameters in the backbone. The baseline for comparing speedup for BLIP-2 models is BLIP-2 ViT-g FlanT5<sub>XL</sub>. We only compare the speedup of early exiting methods and the BLIP-2 variants. In the Appendix B and Table 5 and 8, we show the results of the MiniGPT (Zhu et al., 2023) model and in Appendix C and Table 7, we show the results on the InstructBLIP (Liu et al., 2024) model.

**Training:** In our setup for BLIP-2, we utilize two variations of the BLIP-2 model with the same image encoder (ViT-g/14 (Dosovitskiy et al., 2020)) and frozen LLMs that are OPT-2.7B (Zhang et al., 2022a) and FlanT5-XL (Chung et al., 2024). We use the LAVIS (Li et al., 2022a) library for implementation, training and evaluation. For training, we use the validation split of the datasets. We use 80% of validation split for training and the remaining 20% for development.

We use labels of the validation dataset when the task is semi-supervised, else we just use the samples without labels. First, the backbone is fine-tuned for 10 epochs with a starting learning rate of 1e-5, which decays by 0.5 every 3 epochs. The backbone weights are then frozen post-fine-tuning and exits are attached to the backbone. We train exit weights for a further 3 epochs. We employ the Adam (Kingma and Ba, 2014) optimizer and a batch size of 16. Similar to Bajpai and Hanawal (2024c), we use a feature classifier, with one hidden linear layer with a hidden state of size 3072 and a LeakyReLU activation function.

For the unsupervised tasks, we train the model for 5 epochs on the validation dataset (without labels) with knowledge distillation from the final layer. Optimizers and learning rates are kept the same as given above. Note that in CapFilt we apply the CapFilt method on the validation dataset (without labels) and generate synthetic labels. After this, we perform a similar procedure by treating the synthetic labels as the true labels as done for the semi-supervised tasks. We do not finetune the backbone in an unsupervised setup.

**Inference:** Inference is conducted with a batch size of 1. We provide the results on the test dataset. For image captioning, we use a prompt as ‘a photo of’. The threshold is cho-
sen as the best-performing threshold from the set \(\{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}\) on the held-out split of the validation dataset. More details on the hyper-parameter setting can be found in Table 10 and in the Appendix A.2 with the values of hyperparameters. All experiments were performed with a combination consisting of two NVIDIA RTX A6000 and four NVIDIA GeForce RTX 3080 Ti GPUs.

**Baselines:** We establish baseline models for performance evaluation, including vanilla BLIP-2 inference. Additionally, we compare with multi-modal models VinVL (Zhang et al., 2021), ALBEEF (Li et al., 2021), SimVLM (Wang et al., 2021), OFA (Wang et al., 2022a), and Flamingo (Alayrac et al., 2022). We also assess state-of-the-art early exit methods originally proposed for the BERT backbone, such as DeeBERT (Xin et al., 2020), PABEE (Zhou et al., 2020), and LeeBERT (Zhu, 2021), adapted to the BLIP-2 backbone as DeeBLIP, PABEE-BLIP, and LeeBLIP, respectively. DeeBLIP uses confidence-based exiting, PABEE-BLIP employs patience-based exiting, and LeeBLIP combines knowledge distillation from the final layer with hard label learning. Furthermore, we apply the MuE (Tang et al., 2023) early exiting method to the BLIP-2 backbone, using exits from the better-performing BLIP-2 variant for our baselines.

5 Results

**Visual Question Answering:** We provide results on unsupervised (see table 3) as well as semi-supervised setups (see table 2). We observe that our method outperforms all early exit methods in terms of accuracy and speedup even with less number of trainable parameters. We even outperform the vanilla BLIP-2 inference due to overthinking in the BLIP-2 backbone which is mitigated by our input-adaptive inference. We also provide results on an unsupervised visual dialogue dataset where the task is similar to VQA but there is an additional context before the question i.e. a dialogue history between the user and the model.

**Image Captioning:** We provide results of semi-supervised and unsupervised setup in table 1 and 4 respectively. We clearly outperform the existing models in terms of both accuracy as well as speedup. For the NoCaps dataset, the model is fine-tuned on the COCO dataset. The speedup for NoCaps dataset is lower as there is a domain change from the training which lowers the confidence in prediction taking more samples to deeper exits for inference.

We observe performance improvement over previous baselines as we attached exits to the BLIP-2 model and by performing input-adaptive inference, we perform better than the BLIP-2 model, and as BLIP-2 outperforms other models, FREE also outperforms others. For the early exiting baselines on BLIP-2, we outperform them as we have an additional component in the exits rather than just a linear classifier which helps in better performance of exits in terms of both performance and speedup. Note that there is a decrease in accuracy when we are in an unsupervised setup, as our model mimics the final layer hence some amount of overthinking still remains. Still, we are comparable to the BLIP-2 inference. On the other hand, the labeled dataset in semi-supervised tasks helps the model learn the hardness of the incoming sample. This helps the model to overcome the overthinking issue.

We have utilized two versions of the BLIP-2 model that have decoder as FlanT5XL and OPT2.7B. We observe that the speedup in BLIP-2 with OPT2.7B was higher as there are more layers in this hence they are more susceptible to overthinking issues. The speedup for VQA tasks was higher as these tasks are simpler than image captioning tasks. We have not reported the speedup of the models other than the variants of BLIP-2 as they have different types of architectures. In terms of speedup, our objective is to make BLIP-2 faster.

In table 9, we have shown the result of using the CapFilt method to generate synthetic captions in the absence of the labeled dataset. We have reported the CIDEr score over the NoCaps dataset. We can observe that the model has improved upon the performance using CapFilt and the speedup has significantly increased. This effect is due to the good quality captions that help the exits learn better, hence it outputs more samples early increasing the speedup as compared to knowledge distillation.

6 Conclusion

In this study, we introduced a novel inference technique FREE, which leverages adversarial training of exits alongside the zero-shot capabilities of the VLMs. By employing FREE, we effectively reduce the dependency on a vast amount of labeled training data typically required for exit training.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#Total params</th>
    <th rowspan="2">VisDial test</th>
    <th colspan="2">VQAv2 train/test</th>
    <th rowspan="2">OK-VQA test</th>
    <th rowspan="2">GQA test</th>
    <th rowspan="2">VizWiz test</th>
    <th rowspan="2">Speed</th>
  </tr>
  <tr>
    <th>train</th>
    <th>test</th>
  </tr>
  <tr>
    <td colspan="9"><i>Without exits</i></td>
  </tr>
  <tr>
    <td>Flamingo3B</td>
    <td>3.2B</td>
    <td>-</td>
    <td>53.2</td>
    <td>49.4</td>
    <td>41.5</td>
    <td>-</td>
    <td>28.9</td>
    <td>1.28×</td>
  </tr>
  <tr>
    <td>Flamingo9B</td>
    <td>9.3B</td>
    <td>-</td>
    <td>55.7</td>
    <td>51.8</td>
    <td>44.7</td>
    <td>-</td>
    <td>28.8</td>
    <td>0.44×</td>
  </tr>
  <tr>
    <td>Flamingo80B</td>
    <td>80B</td>
    <td>-</td>
    <td>59.1</td>
    <td>56.2</td>
    <td><b>50.4</b></td>
    <td>-</td>
    <td><b>31.5</b></td>
    <td>0.05×</td>
  </tr>
  <tr>
    <td>BLIP-2 ViT-g OPT<sub>2.7B</sub></td>
    <td>3.8B</td>
    <td>34.1</td>
    <td>54.6</td>
    <td>52.0</td>
    <td>31.2</td>
    <td>34.2</td>
    <td>27.0</td>
    <td>1.07×</td>
  </tr>
  <tr>
    <td>BLIP-2 ViT-g OPT<sub>6.7B</sub></td>
    <td>7.8B</td>
    <td>37.5</td>
    <td>55.9</td>
    <td>53.7</td>
    <td>36.1</td>
    <td>36.4</td>
    <td>27.2</td>
    <td>0.52×</td>
  </tr>
  <tr>
    <td>BLIP-2 ViT-g FlanT5XL</td>
    <td>4.1B</td>
    <td><b>45.9</b></td>
    <td><b>64.9</b></td>
    <td><b>62.5</b></td>
    <td><b>40.6</b></td>
    <td><b>44.5</b></td>
    <td><b>29.8</b></td>
    <td><b>1.00×</b></td>
  </tr>
  <tr>
    <td colspan="9"><i>Early Exit models (on BLIP-2 ViT-g FlanT5XL)</i></td>
  </tr>
  <tr>
    <td>DeeBLIP</td>
    <td>4.7B</td>
    <td>33.4</td>
    <td>41.3</td>
    <td>42.8</td>
    <td>23.4</td>
    <td>27.8</td>
    <td>20.1</td>
    <td>1.39×</td>
  </tr>
  <tr>
    <td>PABEE-BLIP</td>
    <td>4.7B</td>
    <td>35.7</td>
    <td>49.6</td>
    <td>51.3</td>
    <td>31.2</td>
    <td>34.3</td>
    <td>23.6</td>
    <td>1.31×</td>
  </tr>
  <tr>
    <td>LeeBLIP</td>
    <td>4.7B</td>
    <td>39.1</td>
    <td>57.7</td>
    <td>57.1</td>
    <td>37.1</td>
    <td>39.7</td>
    <td>26.4</td>
    <td>1.29×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>4.7B</td>
    <td>36.6</td>
    <td>55.4</td>
    <td>53.6</td>
    <td>33.7</td>
    <td>37.1</td>
    <td>24.7</td>
    <td>1.36×</td>
  </tr>
  <tr>
    <td>FREE ViT-g OPT<sub>2.7B</sub></td>
    <td>4.3B</td>
    <td>32.3</td>
    <td>55.5</td>
    <td>53.4</td>
    <td>35.6</td>
    <td>44.7</td>
    <td>26.8</td>
    <td>1.51×</td>
  </tr>
  <tr>
    <td>FREE ViT-g FlanT5XL</td>
    <td>4.5B</td>
    <td><b>45.5</b></td>
    <td><b>64.5</b></td>
    <td><b>62.1</b></td>
    <td><b>40.3</b></td>
    <td><b>44.0</b></td>
    <td><b>29.5</b></td>
    <td><b>1.45×</b></td>
  </tr>
</table>

Table 3: Results of the unsupervised Visual-Question Answering and VisDial dataset. For VQA tasks, we report the VQA accuracy and for the visual dialogue, we report the Mean Reciprocal Rank(MRR).

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">COCO Karpathy test</th>
  </tr>
  <tr>
    <th>B@4</th>
    <th>C</th>
    <th>S</th>
    <th>M</th>
    <th>Spd</th>
  </tr>
  <tr>
    <td>OFA</td>
    <td>37.5</td>
    <td>130.3</td>
    <td>25.2</td>
    <td>31.1</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Flamingo</td>
    <td>38.5</td>
    <td>134.1</td>
    <td>24.1</td>
    <td>27.8</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SimVLM</td>
    <td>38.6</td>
    <td>138.3</td>
    <td>24.8</td>
    <td>29.8</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BLIP-2-V-O</td>
    <td>41.7</td>
    <td>139.8</td>
    <td>25.5</td>
    <td>30.5</td>
    <td>1.07×</td>
  </tr>
  <tr>
    <td>BLIP-2-V-F</td>
    <td>40.4</td>
    <td>141.5</td>
    <td>25.2</td>
    <td>29.1</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td colspan="6"><i>Early Exit models</i></td>
  </tr>
  <tr>
    <td>DeeBLIP</td>
    <td>32.8</td>
    <td>115.1</td>
    <td>20.9</td>
    <td>25.3</td>
    <td>1.65×</td>
  </tr>
  <tr>
    <td>PABEE-BLIP</td>
    <td>34.2</td>
    <td>119.8</td>
    <td>21.4</td>
    <td>26.2</td>
    <td>1.45×</td>
  </tr>
  <tr>
    <td>LeeBLIP</td>
    <td>37.4</td>
    <td>132.0</td>
    <td>22.8</td>
    <td>27.6</td>
    <td>1.59×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>37.9</td>
    <td>137.5</td>
    <td>23.6</td>
    <td>28.5</td>
    <td>1.41×</td>
  </tr>
  <tr>
    <td>FREE-V-O</td>
    <td><b>41.9</b></td>
    <td><b>142.5</b></td>
    <td><b>25.2</b></td>
    <td><b>30.8</b></td>
    <td><b>1.75×</b></td>
  </tr>
</table>

Table 4: Results of semi-supervised training on the Karpathy test split of the COCO dataset.

Our approach involves adversarially training exits to generate representations similar to those of the final layer, thereby minimizing the need for extensive labeled data. Moreover, our exit design reduces the number of trainable parameters, resulting in lower computational costs. Experimental results demonstrate that our method significantly enhances inference speed while yielding high-quality outputs.

7 Limitations

For attaching exits to a large model such as BLIP-2, the crucial part is to decide where to attach exits within a given budget, i.e., what could be the best places for an exit in the LM component of the backbone without exceeding a certain amount of parameters. We answered that question by explaining the mid-crisis. However, the placements of exits with given budget criteria still remain unexplored, which can make these models even faster within computational boundaries.

Acknowledgements

Divya Jyoti Bajpai is supported by the Prime Minister’s Research Fellowship (PMRF), Govt. of India. Manjesh K. Hanawal thanks funding support from SERB, Govt. of India, through the Core Research Grant (CRG/2022/008807) and MATRICS grant (MTR/2021/000645), and DST-Inria Targeted Programme. We also thank funding support from Amazon IIT-Bombay AI-ML Initiative (AIAIMLI).

References

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer.

Divya J Bajpai, Vivek K Trivedi, Sohan L Yadav, and Manjesh K Hanawal. 2023. Splice: Early exit in deep neural networks with split computing. arXiv preprint arXiv:2309.09195.

Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024a. Capeen: Image captioning with early
exits and knowledge distillation. arXiv preprint arXiv:2410.04433.

Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024b. Ceebert: Cross-domain inference in early exit bert. In To appear in proceedings of the 62nd conference of the Association for computational linguistics: Findings Volume.

Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024c. Dadee: Unsupervised domain adaptation in early exit plms. arXiv preprint arXiv:2410.04424.

Divya Jyoti Bajpai, Aastha Jaiswal, and Manjesh Kumar Hanawal. 2024. I-splitee: Image classification in split computing dnns with early exits. arXiv preprint arXiv:2401.10541.

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, and Furu Wei. 2021. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022a. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grynec, Basil Mustafa, Lucas Beyer, et al. 2022b. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer.

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pages 1931–1942. PMLR.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. 2018. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53–65.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. Instructclip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36.

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshray Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326–335.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710.

Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. 2022. Deecap: Dynamic early exiting for efficient image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12216–12226.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. 2022. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846.

Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng, Junlan Feng, Shiji Song, and Gao Huang. 2023. Dynamic perceiver for efficient visual recognition. arXiv preprint arXiv:2306.11248.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Yixin Ji, Jikai Wang, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang. 2023. Early exit with disentangled representation and equiangular tight frame. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14128–14142.

Chao Jia, Yinfai Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR.

Weiyu Ju, Wei Bao, Dong Yuan, Liming Ge, and Bing Bing Zhou. 2021. Learning early exit for deep neural network inference on mobile devices through multi-armed bandits. In 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 11–20. IEEE.

Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow-deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pages 3301–3310. PMLR.

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526.

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. 2022a. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 121–137. Springer.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. 2020. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178.

Xiangyang Liu, Tianxiang Sun, Junliang He, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021. Towards efficient nlp: A standard evaluation and a strong baseline. arXiv preprint arXiv:2110.07038.

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. 2022. Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. arXiv preprint arXiv:2210.07179.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Mary Phuong and Christoph H Lampert. 2019. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1355–1364.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish
Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Minho Ryu, Geonseok Lee, and Kichun Lee. 2022. Knowledge distillation for bert unsupervised domain adaptation. Knowledge and Information Systems, 64(11):3113–3128.

Shengkun Tang, Yaqing Wang, Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding, Yanzhi Wang, Yi Liang, and Dongkuan Xu. 2023. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10781–10791.

Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE.

Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. 2022. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.

Meiqi Wang, Jianqiao Mo, Jun Lin, Zhongfeng Wang, and Li Du. 2019. Dynexit: A dynamic early-exit strategy for deep residual networks. In 2019 IEEE International Workshop on Signal Processing Systems (SiPS), pages 178–183. IEEE.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pages 23318–23340. PMLR.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhajit Som, et al. 2022b. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442.

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904.

Zizhao Wang, Wei Bao, Dong Yuan, Liming Ge, Nguyen H Tran, and Albert Zomaya. 2020. Accelerating on-device dnn inference during service outage through scheduling early exit. Computer Communications, 162:69–82.

Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.12993.

Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. 2020. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2369–2378.

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133.

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Zhen Zhang, Wei Zhu, Jinfan Zhang, Peng Wang, Rize Jin, and Tae-Sun Chung. 2022b. Pcee-bert: accelerating bert inference via patient and confident early exiting. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 327–338.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.

Wei Zhu. 2021. Leebert: Learned early exit for bert with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2968–2980.

A Appendix

A.1 Ablation Study

Qualitative analysis: In Figure 1, we provide some examples of the output provided by the FREE model. The figure shows how the early exit models increase the speedup by predicting easier tokens earlier. For instance, the image in the last row and last column of the figure is an example of an easy sample where the tokens are predicted at initial layers. Similarly, for the image with a zebra, it can easily predict the easier token such as ‘A zebra standing in a snowy field’ at the initial layers while the part of the image that is not easy to predict ‘with a wall behind’ is predicted at deeper layers and predicting a high-quality caption overall while speeding up inference using the easiness of sample as well as token.

Also note that the common tokens are mostly predicted from the initial layers while the rare tokens are predicted from the deeper layers. This observation suggests that the tokens that occur more number of times are considered as easy by the model while the token that have rare appearances are the ones treated as hard.

A.2 Accuracy vs speedup

In figure 4b, we show the accuracy vs speedup curve which could be obtained by changing the threshold parameter \( \alpha \). As we decrease the threshold parameter, samples exit from the initial layers even with less confidence, in this way all the samples are more prone to be incorrect decreasing the accuracy but as the threshold is decreased more samples exit from the initial layers and increase the speedup. One key observation is as we start decreasing the threshold, we observe that sometimes the performance even increases, this is the effect of overthinking, where some samples are correctly predicted at initial layers and might become wrong as they reach the final layer. We have also plotted the curves for other exiting methods and observed that our method has better stability as compared to other early exiting methods.

A.3 Importance of different components

In figure 4a, we show the importance of different components of our method. We observe that there is a huge performance drop if we remove the knowledge distillation or cross-entropy loss from the overall loss function. This occurs due to catastrophic forgetting or mode collapse where
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">COCO Karpathy test split</th>
    <th>Speed</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU4</th>
    <th>CIDEr</th>
    <th>S</th>
    <th>M</th>
  </tr>
  <tr>
    <td>MiniGPT</td>
    <td>38.6</td>
    <td>133.9</td>
    <td>24.2</td>
    <td>30.8</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>DeeMini</td>
    <td>31.4</td>
    <td>107.0</td>
    <td>20.1</td>
    <td>25.9</td>
    <td>1.42×</td>
  </tr>
  <tr>
    <td>PABEE-Mini</td>
    <td>33.5</td>
    <td>117.3</td>
    <td>20.9</td>
    <td>27.3</td>
    <td>1.40×</td>
  </tr>
  <tr>
    <td>LeeMini</td>
    <td>37.3</td>
    <td>126.8</td>
    <td>22.8</td>
    <td>29.1</td>
    <td>1.49×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>37.8</td>
    <td>129.5</td>
    <td>23.2</td>
    <td>29.7</td>
    <td>1.52×</td>
  </tr>
  <tr>
    <td>FREE</td>
    <td><b>38.3</b></td>
    <td><b>132.2</b></td>
    <td><b>23.9</b></td>
    <td><b>30.5</b></td>
    <td><b>1.67×</b></td>
  </tr>
</table>

Table 5: Results of MiniGPT-4 model on the COCO Karpathy test split.

![Line graph showing BLEU-1 score for COCO with different loss functions](page_184_370_370_246.png)
(a) Different combinations of loss function

![Line graph showing speedup vs accuracy curve](page_563_370_370_246.png)
(b) Speedup vs accuracy curve.

Figure 4: Left: BLEU-1 score for COCO with model BLIP-2-ViT-L-OPT_{2.7B} with different components. Right: Speedup vs BLEU-4 curve for COCO dataset with ViT-L-OPT_{2.7B}.

<table>
  <tr>
    <th>Noise</th>
    <th>BLEU-4</th>
    <th>METEOR</th>
    <th>CIDEr</th>
    <th>Speed</th>
  </tr>
  <tr>
    <td rowspan="2">σ = 0.0</td>
    <td>BLIP-2</td>
    <td>40.4</td>
    <td>29.1</td>
    <td>141.5</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>BLIPEE</td>
    <td>40.5</td>
    <td>29.1</td>
    <td>142</td>
    <td>1.77×</td>
  </tr>
  <tr>
    <td rowspan="2">σ = 0.5</td>
    <td>BLIP-2</td>
    <td>38.6</td>
    <td>28</td>
    <td>137.9</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>BLIPEE</td>
    <td>39.8</td>
    <td>28.8</td>
    <td>139.5</td>
    <td>1.64×</td>
  </tr>
  <tr>
    <td rowspan="2">σ = 1.0</td>
    <td>BLIP-2</td>
    <td>35.3</td>
    <td>26.4</td>
    <td>130.8</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>BLIPEE</td>
    <td>36.7</td>
    <td>27.9</td>
    <td>134.2</td>
    <td>1.59×</td>
  </tr>
  <tr>
    <td rowspan="2">σ = 2.0</td>
    <td>BLIP-2</td>
    <td>28.6</td>
    <td>20.5</td>
    <td>112.6</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>BLIPEE</td>
    <td>31.1</td>
    <td>22.7</td>
    <td>120.8</td>
    <td>1.45×</td>
  </tr>
</table>

Table 6: Results on BLIP-2 and FREE on COCO dataset when some level of noise \( \sigma \) is added into the images during inference phase.

the model gets stuck into local minima. On the other hand, if we remove the adversarial training part, there is again a performance drop, as we only train the classifier but we are not mapping the feature representations of the final layer and the exits hence exits only have low-level features which is insufficient to make correct predictions, hence resulting in a performance drop.

A.4 Robustness of FREE

In table 6, we provide results to prove the robustness of FREE. To obtain these results, we perform inference on images with Gaussian noise \( \sigma \) added to it. The higher the value of \( \sigma \), the more noise is present in the image. Observe from the table that when the level of noise is increased in the image the performance of the BLIP-2 model is affected. However, the impact of the noise is smaller for FREE.

The reason for the smaller impact of noise on FREE is that FREE uses outputs from multiple exits and performs inference only when a classifier is confident enough. This makes it robust to noise present during the inference phase.

B Results on MiniGPT

In Table 5 and 8, we show the results of FREE on the MiniGPT backbone. First note that the performance of MiniGPT is not as good as the BLIP-2 model. The reason is stated in the Appendix of the MiniGPT-4 paper as the number of training parameters of the model is significantly lower than the full BLIP-2 model. However, it has a better performance over long sequence generation tasks. We use the Llama (Touvron et al., 2023) model in its decoder.

From the results, we observe that our method can be well generalized over MiniGPT as the results are similar to the BLIP-2 backbone. Note that during fine-tuning MiniGPT, we have used similar hyperparameters as used to train BLIP-2 as the overall architecture of MiniGPT is similar to BLIP-2 except for the number of training param-
<table>
  <tr>
    <th>Model</th>
    <th colspan="4">COCO Karpathy test split</th>
    <th>Speed</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU4</th>
    <th>CIDEr</th>
    <th>S</th>
    <th>M</th>
    <th></th>
  </tr>
  <tr>
    <td>InstructBLIP</td>
    <td>42.6</td>
    <td>140.5</td>
    <td>24.3</td>
    <td>31.2</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>DeeIB</td>
    <td>35.2</td>
    <td>123.7</td>
    <td>20.8</td>
    <td>26.8</td>
    <td>1.39×</td>
  </tr>
  <tr>
    <td>PABEE-IB</td>
    <td>36.5</td>
    <td>126.3</td>
    <td>21.7</td>
    <td>28.0</td>
    <td>1.35×</td>
  </tr>
  <tr>
    <td>LeeIB</td>
    <td>38.0</td>
    <td>132.8</td>
    <td>22.9</td>
    <td>29.9</td>
    <td>1.47×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>39.4</td>
    <td>136.2</td>
    <td>23.4</td>
    <td>30.6</td>
    <td>1.42×</td>
  </tr>
  <tr>
    <td>FREE</td>
    <td><b>42.1</b></td>
    <td><b>138.9</b></td>
    <td><b>23.8</b></td>
    <td><b>31.0</b></td>
    <td><b>1.58×</b></td>
  </tr>
</table>

Table 7: Results of InstructBLIP model on the COCO Karpathy test split.

<table>
  <tr>
    <th>Model</th>
    <th>VQA</th>
    <th>OKVQA</th>
    <th>GQA</th>
    <th>Speed</th>
  </tr>
  <tr>
    <td>MiniGPT</td>
    <td>68.6</td>
    <td>65.2</td>
    <td>41.9</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>DeeMini</td>
    <td>59.8</td>
    <td>59.1</td>
    <td>35.7</td>
    <td>1.53×</td>
  </tr>
  <tr>
    <td>PABEE-Mini</td>
    <td>61.4</td>
    <td>60.8</td>
    <td>36.9</td>
    <td>1.49×</td>
  </tr>
  <tr>
    <td>LeeMini</td>
    <td>65.7</td>
    <td>63.2</td>
    <td>39.0</td>
    <td>1.61×</td>
  </tr>
  <tr>
    <td>MuE</td>
    <td>67.8</td>
    <td>64.6</td>
    <td>39.9</td>
    <td>1.59×</td>
  </tr>
  <tr>
    <td>FREE</td>
    <td><b>68.2</b></td>
    <td><b>65.1</b></td>
    <td><b>41.6</b></td>
    <td><b>1.63×</b></td>
  </tr>
</table>

Table 8: Results on MiniGPT for Visual Question Answering tasks.

eters. Note that the baselines are similar to those given in the main body with DeeBERT replaced as DeeMini, PABEE as PABEE-Mini, and LeeBERT as LeeMini. All the hyperparameters were kept the same for all the baselines with their approach applied.

C Results on InstructBLIP

InstructBLIP is an upgraded version of the BLIP-2 model where the underlying architecture is same but the difference is with the instruction given to generate the output. InstructBLIP can perform better than BLIP-2 over long sequence generation tasks however takes a slight hit on the performance of the image captioning tasks. The hyperparameters for fine-tuning InstructBLIP were the same as for BLIP-2.

However, the behavior of our method applied to this model is the same as it shows minimal performance drop as compared to other EE methods while keeping the speedup better than them. This is the result of the access to deeper level knowledge as our method tries to mimic the behavior of the final layers at the exits. Also, we have used only the validation split to train the exits which makes our method less resource-heavy. Note that the baselines are similar to those given in the main body with DeeBERT replaced as DeeIB, PABEE as PABEE-IB, and LeeBERT as LeeIB. All the hyperparameters were kept the same for all the baselines with their approach applied.

Note that both MiniGPT and InstructBLIP share same architectural details as BLIP-2 model with different training strategies, hence we do not explicitly explain the architectural details. Also the training and inference procedure with hyperparameters is similar to the BLIP-2 model.

D Analysis on computational cost

Training stage: In our method, the fine-tuning stage is similar to vanilla fine-tuning of the backbone. After this step, we have additional parameters that are used at the exits consisting of one transformer layer similar to the decoder of the model. While other methods use a classifier at all the exits, note that due to huge size of the vocabulary the classifier size is very large even larger than one transformer layer. Due to this our method has lesser number of training parameters. Now as we train in a GAN-based setup, we alternatively train the exit transformer layer, where it is trained to generate similar representations as the final layer. The generator is the exit transformer layer and a separate feature classifier is also added with a job to discriminate if the features are from the final layer or exit layer. The full training procedure along with fine-tuning takes approximately 26 hours to complete on a setup of 6 GPUs with 4 NVIDIA RTX 3090 and 2 NVIDIA RTX A6000 GPUs while on a similar setup training of vanilla BLIP-2 training requires 18 hours of training.

Inference: While we have a heavy training setup, post deployment the inference process is very simple and fast. Once the image has been passed through the image encoder as well as the query generator. After this step the query embedding are passed through decoder after the projection layer. Then the decoder generates the text autoregressively. The next token is generated at one of the attached exits where the input to the decoder is processed sequentially at the exits. If an exit is confident on its prediction, it stops the inference process by assigning the token generated by it as the next predicted token by the model. This im-
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">No Caps</th>
    <th>Zero-shot</th>
    <th>full-dataset</th>
    <th>Spd</th>
  </tr>
  <tr>
    <th></th>
    <th>in-domain</th>
    <th>near-domain</th>
    <th>out-domain</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>w/o CapFilt</td>
    <td>122.3</td>
    <td>118.9</td>
    <td>123.1</td>
    <td>120.7</td>
    <td>1.45×</td>
  </tr>
  <tr>
    <td>Capfilt</td>
    <td>123.5</td>
    <td>120.4</td>
    <td>124.7</td>
    <td>122.0</td>
    <td>1.77×</td>
  </tr>
</table>

Table 9: Difference between CapFilt and Knowledge distillation method for an unsupervised setup.

<table>
  <tr>
    <th>LLM</th>
    <th>OPT</th>
    <th>FlanT5</th>
  </tr>
  <tr>
    <td>Exit Config</td>
    <td>[3, 6, 9, 12, 24, 27, 30]</td>
    <td>[3, 5, 7, 9, 12, 20, 22]</td>
  </tr>
  <tr>
    <td>AdamW beta</td>
    <td>[0.9, 0.999]</td>
    <td>[0.9, 0.999]</td>
  </tr>
  <tr>
    <td>Threshold</td>
    <td>0.8</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>Inference beam size</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>500</td>
    <td>500</td>
  </tr>
</table>

Table 10: More hyperparameter details of FREE on different LM component in the BLIP-2 model. Note that the thresholds are chosen from the set {0.5, 0.6, 0.7, 0.8, 0.9, 1.0}

<table>
  <tr>
    <th>Dataset</th>
    <th>Layer 3</th>
    <th>Layer 6</th>
    <th>Layer 9</th>
    <th>Layer 12</th>
    <th>Layer 24</th>
    <th>Layer 27</th>
    <th>Layer 30</th>
  </tr>
  <tr>
    <td>VQA-v2</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.13</td>
    <td>0.09</td>
    <td>0.15</td>
    <td>0.18</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Ok-VQA</td>
    <td>0.03</td>
    <td>0.09</td>
    <td>0.11</td>
    <td>0.09</td>
    <td>0.13</td>
    <td>0.15</td>
    <td>0.18</td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>0.01</td>
    <td>0.06</td>
    <td>0.09</td>
    <td>0.08</td>
    <td>0.11</td>
    <td>0.17</td>
    <td>0.22</td>
  </tr>
</table>

Table 11: Fraction of samples exiting from different layers of the backbone across various datasets.

<table>
  <tr>
    <th>Component</th>
    <th>Parameters per Layer</th>
    <th>Total Parameters(32-layers)</th>
  </tr>
  <tr>
    <td>Transformer Layer</td>
    <td>~62.8M</td>
    <td>~2.01B</td>
  </tr>
  <tr>
    <td>Layer Norms & Embeddings</td>
    <td>-</td>
    <td>~560M</td>
  </tr>
  <tr>
    <td>LM Head</td>
    <td>-</td>
    <td>~128.7M</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>-</th>
    <th><b>2.7B</b></th>
  </tr>
</table>

Table 12: OPT-2.7B Parameter Breakdown
proves the speed benefits of the full model where not all tokens are processed till the final layer. We observed that more common tokens usually exit at earlier exits while rare tokens are processed deeper into the backbone. As formation of sentence require a large number of common tokens such as 'the', 'and', etc., the early exits makes the generation process significantly faster. The fraction of token exited from different layers is given in Table 11. **The inference time over the Karpathy test split of the COCO dataset was 7 minutes of wall clock time approximately while for vanilla BLIP-2, it was approximately 11 minutes. This also proves the importance of exits.**

In summary, during training, we have additional exits as compared to vanilla BLIP-2 model as well as feature classifiers at the exits. The feature classifiers are dropped during inference and are not further required. While the attached exits help the model to make adaptive predictions by performing inference using some fraction of layers from the full model based on the input sample and token complexity.

**D.1 OPT-2.7B Architecture**

OPT-2.7B is a decoder-only transformer model with 2.7 billion parameters. Each transformer layer consists of multi-head self-attention, a feed-forward network (FFN), and layer normalization. The detailed architecture is as follows:

• **Hidden Size (\( d_{\text{model}} \)):** 2560

• **Number of Layers:** 32

• **Attention Heads:** 32

• **FFN Inner Size:** \( 4 \times d_{\text{model}} = 10240 \)

• **LM Head Size:** \( 2560 \times 50272 \approx 128.7M \) parameters

The total parameter breakdown is summarized in Table 12. Now if we use classical exit setup, that uses independent exit classifiers for attaching exits it adds 128.7\( M \) parameters at every exit. While in our setup, we attach exit transformers and exit classifiers at intermediate layers where the exit classifier have shared weights as final layer classifier and is frozen. Since the exit transformer is a replica of transformer layer of the model, its adds only 62.8\( M \) parameters. This reduces the training parameter size by 52%.

Reviewer paper summaries: This paper introduces FREE, a fast and robust early exit (EE) strategy for vision-language models (VLMs), primarily tested on BLIP-2, to improve inference efficiency without compromising accuracy. Using GAN-based adversarial training, FREE aligns intermediate layer representations with the final layer, mitigating overthinking and mid-crisis phenomena while enabling adaptive computation per input.

This paper introduces FREE, an early exit framework for vision-language models that reduces inference latency while maintaining high accuracy. It addresses a newly identified issue termed “mid-crisis,” where performance drops in intermediate layers, by using adversarial training to align mid-layer representations with the final layer. FREE is compatible with both supervised and unsupervised training and is evaluated across multiple benchmarks including VQAv2, GQA, and NoCaps. The results show competitive performance and speedups over baseline EE methods like PABEE and MuE.
Summary of strengths: * The paper introduces an early exit (EE) strategy that significantly reduces inference time (1.51× speedup) while maintaining high accuracy.

1. The paper identifies and formalizes the “mid-crisis” phenomenon—performance degradation in intermediate layers of vision-language models (VLMs)—which is both novel and insightful.
2. The proposed FREE framework introduces an adversarial early exit (EE) strategy that is flexible and applicable in both supervised and unsupervised settings, reducing inference cost while maintaining performance.
Summary of weaknesses: *  The method requires training exit classifiers using adversarial learning, which introduces instability issues such as mode collapse and catastrophic forgetting. This could be further explained and analyzed in this paper.
* The paper chooses fixed exit layers for early exits,how about adaptive exit layer selection, which may further improve efficiency.

1. The discussion of how EE specifically mitigates the mid-crisis issue is limited.
2. From the results, much of the performance gain seems to stem from the backbone (e.g., BLIP-2 FlanT5XL) rather than the choice of early exit strategy itself, raising questions about the novelty of the EE contribution relative to existing models.
Comments, suggestions and typos: N/A

Q1. Page 1, Lines 057–059: You claim that “intermediary layers tend to capture dataset patterns that degrade their performance, even losing the information learned by initial layers.” Can you provide empirical evidence or theoretical rationale to support this? 

Q2. Have you observed the mid-crisis phenomenon in VLMs beyond BLIP-2, such as MiniGPT, or LLaMA-based VLMs?

Q3. Figure 2 illustrates mitigation of mid-crisis, how much final task performance improves as a result?

Q4. Table 3. Were the baseline models (e.g., MuE, LeeBLIP) trained with the same BLIP-2 backbones (ViT-g with OPT2.7B or FlanT5XL) as FREE?
Overall assessment scores: 3

3
Metareview score: 3
Metareview: This paper presents FREE, a novel adversarial training framework for early-exit strategies in vision-language models. The reviewers generally acknowledge the paper's methodology, experimental results, and motivation, recognizing its strong performance across multiple benchmarks. Although some areas for improvement exist (e.g., generalizability, training stability, and computational trade-off analysis), these issues do not undermine the overall quality of the work. In my view, with appropriate revisions, this paper is suitable for acceptance as a **Findings** paper.
Summary of reasons to publish: 1. **Methodology**  
   - **Novel Early Exit Framework (FREE)**:  
     - Addresses critical challenges: "mid-crisis" (intermediate-layer performance drop) and "overthinking" (final-layer degradation).  
     - Uses GAN-based adversarial training to align intermediate/final-layer representations.  
   - **Efficiency**: Reuses final-layer classifiers + lightweight transformers for 1.51× faster inference.  
   - **Flexibility**: Works with supervised, unsupervised, and synthetic data (leverages VLM zero-shot capabilities).  

2. **Experiments**  
   - **Broad Validation**: Tested across benchmarks (VQAv2, GQA, NoCaps, MMVet) and architectures (BLIP-2, MiniGPT-4).  
   - **Proven Effectiveness**: Mitigates mid-crisis, maintains robustness, and accelerates inference.  

3. **Clarity**  
   - Clearly defines limitations of existing early-exit methods for VLMs.  
   - Solution aligns logically with identified problems.