Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings
Year: 2023

Abstract:
Adaptive inference is a simple method for reducing inference costs. The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty. In this work, we compare the two main approaches for adaptive inference, Early-Exit and MultiModel, when training data is limited. First, we observe that for models with the same architecture and size, individual Multi-Model classifiers outperform their Early-Exit counterparts by an average of 2.3%. We show that this gap is caused by Early-Exit classifiers sharing model parameters during training, resulting in conflicting gradient updates of model weights. We find that despite this gap, Early-Exit still provides a better speed-accuracy trade-off due to the overhead of the Multi-Model approach. To address these issues, we propose SWEET,1 an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. We compare SWEET’s speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find that it outperforms both methods at fast speeds while maintaining comparable scores to EarlyExit at slow speeds. Moreover, SWEET individual classifiers outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of both methods, paving the way for further reduction of inference costs in NLP. We publicly release our code.2

heading: 1 Introduction
text: Pre-trained Transformer-based language models such as BERT (Devlin et al., 2019), DeBERTa (He et al., 2020), and GPT3 (Brown et al., 2020) have become the go-to tool in NLP . Although powerful, the growing size of these models has been a major drawback (Thompson et al., 2020; Schwartz et al.,
1Separating Weights in Early Exit Transformers. 2https://github.com/schwartz-lab-NLP/SWEET
2020a), making them costly to run. Various attempts to reduce inference cost have been proposed, including distillation (Hinton et al., 2015), pruning (LeCun et al., 1989) and quantization (Courbariaux et al., 2014). This work focuses on adaptive inference (Graves, 2016; Liu et al., 2020), a recent approach in which the variability of sample difficulty is leveraged toward a smarter allocation of computational resources. An appealing property of adaptive inference is that it enables dynamic control of the speed-accuracy trade-off.
There are two main approaches to adaptive inference, both using a set of classifiers of different sizes. In Early-Exit (Schwartz et al., 2020b; Xin et al., 2020), multiple classification heads are added to the same model at different layers, allowing for early exit during inference (Fig. 1, left). Another approach (henceforth Multi-Model) is to apply multiple independent classifiers of varying capacities serially until a prediction is made (Varshney and
14836
Baral, 2022a; Li et al., 2020, Fig. 1, right). These approaches have complementary benefits; MultiModel allows for easier batching at inference time, and potentially larger savings due to using very efficient models (Mamou et al., 2023). Early-Exit on the other hand is more memory efficient, faster to train, and enables re-use of early computation if an early exit was not taken.
In this work, we compare the speed-accuracy behavior of the two approaches when training data is limited.3 We first observe that Early-Exit model weights are updated by multiple conflicting gradient signals throughout the training process (Fig. 2, left). We show that this leads to a decrease in performance of individual Early-Exit classifiers compared to Multi-Model ones (2.3% gap on average). We find that this gap is higher for the earliest classifiers (5.2% on average) than for later ones (1.4%).
We also find that while each Multi-Model classifier outperforms its Early-Exit counterpart, it does not translate to an overall better speed-accuracy trade-off. Instead, we find that each method dominates performance in a different region: MultiModel outperforms Early-Exit at fast inference speeds, while Early-Exit is better at slow speeds. Multi-Model downgraded scores at slow speeds are likely caused by the overhead of running models sequentially to predict hard samples.
Inspired by our findings, we present SWEET,4
an Early-Exit method for bridging the performance gap between standard Early-Exit and Multi-Model. In SWEET, each Early-Exit classifier only updates the parameters of layers preceding it up to the previous classifier. This way, each model parameter is updated by a single classifier, thus avoiding conflicting gradients during the training of Early-Exit models (Fig. 2, right).
We experiment with two established pretrained models: BERT (Devlin et al., 2019) and DeBERTa (He et al., 2020). We fine-tune EarlyExit models using SWEET on seven text classification tasks from GLUE (Wang et al., 2018) and compare them to Early-Exit and Multi-Model baselines. The speed-accuracy curve of SWEET dominates both baselines at fast speeds for 21 out of 28 experiments conducted. As for individual clas-
3Reducing inference costs is particularly helpful when computational resources are limited. Such conditions are often paired with restricted access to labeled data or a low budget for data annotation. To evaluate the effectiveness of adaptive inference methods in such scenarios, we limit training data size to a few thousand examples in all our experiments.
4Separating Weights for Early-Exit Transformers.
sifiers, SWEET performs 1.1% better on average than Early-Exit, mostly improving earlier classifiers, where conflicting gradients are more dominant.
We summarise our main contributions: (1) We propose a way of measuring conflicting gradients, and show that they exist in Early-Exit training process; (2) We empirically compare Early-Exit and Multi-Model classifiers and show that conflicting gradients lead to individual Early-Exit classifiers being less accurate; (3) We propose a novel finetuning method, SWEET, which alleviates the conflicting gradients problem in Early-Exit, and leads to improved results at fast inference speeds; (4) We publicly release our code.5

heading: 2 Background: Adaptive Inference
text: Adaptive inference aims to reduce inference costs of deep neural nets by matching model and sample complexities. Sample difficulty usually varies in real-world data, meaning not all instances require processing by the most powerful classifier. Therefore, we can allocate fewer resources to easier instances, and reduce the average inference cost, potentially at the cost of performance degradation.
5https://github.com/schwartz-lab-NLP/SWEET
Several exit strategies have been developed to control the speed-accuracy trade-off of a model by deciding when to make an early prediction and halt computation (Xin et al., 2020; Zhou et al., 2020; Xin et al., 2021; Schuster et al., 2021; Zhang et al., 2022). In this work, we mainly experiment with confidence-based exiting (Schwartz et al., 2020b), in which computation halts if the softmax probability assigned to a given label exceeds a predetermined threshold. Dynamic control of model’s inference speed is gained through setting different threshold values. There are two main adaptive inference approaches, Early-Exit and Multi-Model, which we describe below.
Early-Exit Early-Exit models are deep neural nets with multiple output points, following intermediate layers. In this work, we focus on EarlyExit implementation as presented in Schwartz et al. (2020b). During fine-tuning,6 instances are passed through the model, and a loss is calculated based on predictions made by all classifiers. This leads to some model weights being updated by gradient signals from multiple classifiers (Fig. 2, left).
At inference time, an instance is passed through the model and its label is predicted sequentially until a decision to exit early is taken, and computation is halted. An appealing property of Early-Exit is that it allows for efficient re-use of previous computation from lower layers in higher classifiers. However, this means that some model parameters are updated by multiple classifiers, which may lead to sub-optimal performance of individual classifiers due to conflicting gradients. In this work, we study the effect of this property on Early-Exit models.
Multi-Model In Multi-Model adaptive inference, a set of independent models of increasing capacity are fine-tuned separately for the same task. At inference time, the models are used sequentially from smallest to largest until a prediction meets some predetermined criterion or until the largest model has been used. This method is more robust than Early-Exit, being easier to extend and enabling the use of different architectures. Additionally, MultiModel potentially allows for further computational savings by using models smaller than the smallest Early-Exit model (a single layer of the backbone
6Some works have pre-trained Early-Exit Models from scratch (Liu et al., 2021b) but due to budgetary constraints, it is common practice to fine-tune pre-trained models with the added classifiers on the downstream task (Liu et al., 2020; Xin et al., 2020; Schwartz et al., 2020b).
model, Mamou et al., 2023). However, it may add overhead to the prediction time of hard instances, as those pass through multiple models with early computations being discarded, bringing the total runtime to exceed that of using the largest model.

heading: 3 Early-Exit vs. Multi-Model

heading: 3.1 Conflicting Gradients in Early-Exit
text: Unlike Multi-Model, when fine-tuning Early-Exit models, model weights are updated by multiple gradient signals, originating in different classification layers (Fig. 2, left). We hypothesize that this leads to sub-optimal performance for all classifiers involved, as gradient signals might conflict with one another and derail the classifiers from their goal. To test this hypothesis, we compare the gradient similarity of different Early-Exit classifiers.
We fine-tune a BERTBASE model with four exit points (following layers [1, 4, 6, 12]) for 400 training steps on the MNLI dataset (Williams et al., 2018), using a batch size of 16 with a learning rate of 2e-5. We pass a new training batch through the model,7 and inspect the gradients with respect to the last feed-forward matrix in each layer preceding a classifier.8 To measure the degree of alignment between gradient updates of different classifiers, we average the cosine similarity between the rows of gradient matrices for every pair of classifiers. High similarity indicates that the classifiers are updating the weights in a similar direction, while low similarity (in absolute values) suggests that the updates are close to orthogonal and potentially detrimental to both classifiers.
This section studies the strengths and weaknesses of both adaptive inference approaches. We start by describing a limitation of Early-Exit approaches: lower model layers are updated by conflicting gradient signals during training. We show that this leads to inferior performance of individual Early-Exit classifiers compared to corresponding Multi-Model classifiers. We then compare the effects of these performance drops on the speedaccuracy curve of Early-Exit models compared Multi-Model ones.
Fig. 3 shows that the gradients of future classifiers are generally orthogonal to those of the current classifier for each examined Transformer block. This indicates that model weights indeed suffer
7We repeat this experiment with an additional batch. Results (Appendix B) show a very similar trend.
8Except for the last layer, updated by a single classifier.
C-1 C-2 C-3 C-4
C-1
C-2
C-3
C-4
1.0 -0.01 -0.02 -0.01
-0.01 1.0 0.47 0.36
-0.02 0.47 1.0 0.53
-0.01 0.36 0.53 1.0
Layer 1 C-2 C-3 C-4
C-2
C-3
C-4
1.0 0.03 0.02
0.03 1.0 0.33
0.02 0.33 1.0
Layer 4
C-3 C-4
C-3
C-4
1.0 0.03
0.03 1.0
Layer 6
0 0.25 0.5 0.75 1 Avg. cosine similarity
Figure 3: Average cosine similarity between classifiers’ gradient updates of model layers. C-i stands for Classifier i. Layer 1 (preceding C-1) is updated by 3 classifiers, while layers 4 (preceding C-2), and 6 (C3) are updated by 3 & 2 classifiers respectively. For each layer, the gradient update of the following classifier is roughly orthogonal to those of future classifiers, whereas gradient updates of higher classifiers tend to better align with one another.
from conflicting gradient updates during the training process, which might affect the performance of each individual classifier. Interestingly, when multiple future classifiers are present, they tend to align with each other, as indicated by the relatively high similarity between layer 1’s gradient updates originating in classifiers 2, 3, and 4, and between layer 4’s gradient updates from classifiers 3 and 4.

heading: 3.2 Effects of Conflicting Gradients
text: To evaluate the effect of conflicting gradients on individual Early-Exit classifiers, we compare the different classifiers of an Early-Exit model and a Multi-Model model. For a clean comparison, we use the same backbone model, exit points, hyperparameters,9 and random seeds. As an example, for a given 12-layer Transformer model, an EarlyExit model with exit points following layers [4, 12] would be compared to a Multi-Model one consisting of two classifiers: the first four layers fit with a classification head, and a full (12 layer) model. The models differ only in the fact that for the EarlyExit model, model weights are updated by multiple future classifiers during the fine-tuning process, while each Multi-Model classifier is an independent model. To isolate the effect of conflicting gradients
9Except for the learning rate, which was tuned for each task and model individually.
Size Method Exit Layer
1 4 6 12
BASE MM 60.90.6 71.40.1 74.71.2 79.90.9 EE 57.30.3 70.30.3 74.40.7 78.70.5
SWEET 60.90.5 71.60.5 74.00.4 77.40.6
1 6 12 24
LARGE MM 60.10.1 66.90.4 74.40.9 81.60.3 EE 56.60.3 65.60.9 74.00.6 79.91.9
SWEET 59.80.3 66.50.7 74.50.3 81.31.0
Table 1: Results of individual classification layers averaged across all tasks using BERT as a backbone model. Multi-Model (MM) classifiers outperform their Early-Exit (EE) counterparts, with the gap being the largest for early classifiers. SWEET closes much of this gap, especially for early classifiers. Standard deviation (across random seeds) is reported in subscript. Results for DeBERTa are presented in Table 3.
on the classifiers, we evaluate each one separately on the entire validation set. The performance gap between each individual Multi-Model classifier and its corresponding Early-Exit classifier, allows us to directly measure the latter’s downgrade in performance caused by conflicting gradients.
We experiment with BERT and DeBERTa {BASE (∼110M parameters), and LARGE (∼350M parameters)}. For BASE versions, we install classifiers following layers [1, 4, 6, 12]. For LARGE versions, we use layers [1, 6, 12, 24]. We fine-tune an Early-Exit model and a corresponding Multi-Model model on seven NLP tasks from the GLUE benchmark (Wang et al., 2019): SST2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), RTE (Dagan et al., 2006; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), CoLA (Warstadt et al., 2019), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and QQP (Iyer et al., 2016). We report accuracy for all tasks except for CoLA (Matthews corr.). We finetune the models for two epochs on all tasks. As our goal is to test Adaptive inference in a low resource setting, we limit the training set size for each task to 6K.10 We report the mean validation scores and standard deviation across three random seeds. Other hyper-parameters are listed in Appendix A.
10For datasets smaller than 6K, we use the entire dataset. The training sets are randomly sampled for each random seed, and are shared across methods with the same seed.
Table 1 shows results for BERT classifiers,11 averaged across all tasks. Our results show that multiple classifiers updating the same weights during fine-tuning diminishes the performance of Early Exit classifiers by 2.3% on average (1.7% for BERT, 3.0% for DeBERTa), with earliest classifiers affected the most (3.5% for BERT, 7.0% for DeBERTa). The increased effect on early classifiers supports our hypothesis regarding conflicting gradients: parameters used by early classifiers receive updates from the largest number of classifiers, thus increasing the chance for conflicting gradients and leading to a larger decrease in performance.

heading: 3.3 Speed-Accuracy Trad-eoff
text: So far we observed that Multi-Model classifiers outperform Early-Exit ones. On the other hand, we also note that there is considerable overhead when using a Multi-Model model. For Multi-Model, an instance that makes an exit on classifier Ci, runs all classifiers up to (including) Ci, each being an independent model, while Early-Exit early layer computations are reused by later classifiers. We turn to evaluate the trade-off between these two factors and compare the overall speed-accuracy curve of each adaptive inference method.
We evaluate model scores across different inference speeds using 11 threshold values, evenly spaced in the range ( 1 # of labels , 1 ) . We note that t = 1# of labels corresponds to the earliest classifier predicting all labels, while for t = 1, the final classifier is used on the entire validation set.
We compute the speedup ratio for each instance as the number of layers it passes through divided by the number of layers of the full backbone model (12 for BASE models, 24 for LARGE models), and average across all instances. As an example, for classifiers following layers [1, 4, 6, 12], an instance that exits on the third classifier (i.e., after layer 6), will have a speedup ratio of 612 for Early-Exit and 11 12 for Multi-Model ( 6 12 + 4 12 + 1 12 ). 12
We evaluate models on the seven tasks listed in Section 3.2, and report the average scores across all tasks. BERTBASE results are presented in Fig. 4, while the results for all other models can be found in Fig. 5. The speed-accuracy curves reveal two important observations. First, as expected, MultiModel achieves better scores at fast speeds (when most instances are predicted by early classifiers).
11See Appendix C, Table 3 for DeBERTa results. 12Further experimental details can be found in Appendix A.
Second, although each Multi-Model classifier outperforms its corresponding Early-Exit one, the Multi-Model overhead leads to this approach being outperformed by Early-Exit at slow speeds (when more instances are predicted by later classifiers).

heading: 4 SWEET: Separating Weights for Early-Exit Transformers
text: Based on our findings, we aim to design an EarlyExit method that takes advantage of the benefits of both Multi-Model and Early-Exit approaches: making the lower Early-Exit classifiers as accurate as the Multi-Model ones, without the additional overhead of the Multi-Model approach.

heading: 4.1 SWEET
text: We present SWEET—a method for fine-tuning Early-Exit models that avoids the harmful impact of conflicting gradients. SWEET grants each classifier exclusive control over part of the model’s parameters, such that each model parameter is only updated by a single classifier. This is done by truncating the loss signal from classifier Ci when reaching the Transformer layer corresponding to classifier Ci−1 (Fig. 2, right).
In SWEET, each classification head receives complete control over a portion of the Model’s parameters and can alter them in a way that optimizes its own goals. Truncating future loss signals makes the very first classifier an independent model, as
opposed to a model that shares its parameters with future classifiers. The following classifiers update only a subset of the model’s parameters but are affected by early model parameters, allowing them to still make use of earlier computations when processing an instance. We clarify that all model weights are updated simultaneously, as opposed to training the classifiers (and their corresponding layers) sequentially, which would have led to significantly longer fine-tuning, matching that of a Multi-Model model.

heading: 4.2 A Better Speeds-Accuracy Curve
text: We turn to evaluate the speed-accuracy curve of SWEET. We use the same experimental setup as in Section 3.2; we fine-tune two pre-trained LMs (BERT and DeBERTa) in two sizes (BASE, LARGE) over the same seven text classification tasks. We use the same exit points and evaluate over the entire validation set using confidence-based exiting. We compare SWEET to two baselines: a standard Early-Exit model and a Multi-Model model. We compute the speedup ratio using the same method as described in Section 3.3. For further implementation details, see Appendix A.
Fig. 5 presents the speed-accuracy curves of all models, averaged across all tasks. The figure shows that at the fastest speeds, SWEET outperforms Early-Exit for all models and is comparable to, or outperforms, Multi-Model. However, for 3 out of the 4 models (all but BERTLARGE), Early-Exit surpasses the performance of SWEET at slow speeds. SWEET’s reduced scores at slow inference speeds are likely due to the lower capacity of the later classifiers, stemming from their restricted influence on early model parameters during fine-tuning.
Fig. 6 presents results on individual tasks for
BERTBASE.13 On five out of seven tasks, SWEET outperforms both baselines at fast speeds (up to a speedup ratio of 1/2), suggesting that SWEET improves the performance of lower Early-Exit classifiers by avoiding conflicting gradients during training. For two tasks (MRPC and CoLA), SWEET suffers a considerable decrease in performance at slow inference speeds. For the other five tasks SWEET maintains comparable results to Early-Exit.
It is also interesting to examine how SWEET affects individual Early-Exit classifiers. Table 1 shows the results of BERT’s individual classifiers trained with SWEET compared to Early-Exit and Multi-Model.14 SWEET classifiers close much of the gap between Early-Exit and Multi-Model: they outperform those of Early-Exit by 1.1% on average (1.2% for BERT, 1.0% for DeBERTa), with the margin being larger for the earliest classifier (3.4% for BERT, 6.2% for DeBERTa). The final two classifiers trained with SWEET achieve lower scores than those of Early-Exit (0.9% on average), probably due to the restricted influence those classifiers have on early model parameters. Our results hint that SWEET is able to effectively bridge the gap between Early-Exit and Multi-Model early classifiers, leading to a speed-accuracy curve favoring fast inference speeds.
We note that during our experiments with EarlyExit and Multi-Model models, some models did not converge. In order to ensure the validity of our results, we repeated these experiments with different random seeds, which led to convergence. We emphasize that this did not occur during the training of models using SWEET.
13Fig. 10 in Appendix D shows individual results for the other models.
14See Table 3 in Appendix C for DeBERTa results.

heading: 5 Further Analysis
text: We turn to evaluate the robustness of our approach. We start with evaluating SWEET using a different exit strategy. We then test how our results generalize when using varying amounts of training data.
A different exit strategy: Learning to Exit Our experiments so far have used a confidence-based exit strategy (Section 2). Here we consider a different strategy—learning to exit (Xin et al., 2021), in which a separate module is trained for each classifier, learning whether the classifier’s prediction should be trusted, and an exit should be made. This method also allows for a natural extension to regression problems, unlike confidence-based methods that are limited to classification tasks.
We use BERT {BASE, LARGE} as a backbone
model and fine-tune in the same procedure described in Section 3.3. Our results (Fig. 7) reveal that the methods behave similarly to the confidencebased setup; SWEET outperforms both baselines at the early stage of the curve (fast speeds), while performing on par with Early-Exit at slow speeds. We also measure the performance of individual exit layers, as in Section 3.2, for models fine-tuned using learning-to-exit. Our results (Table 4 in Appendix E) reveal a similar behavior to that of models fine-tuned using confidence based early exiting: Multi-Model classifiers surpass the performance of Early-Exit classifiers. Moreover, SWEET’s early classifiers outperform those of Early-Exit, while later ones show a slight decrease in performance.
Varying data sizes Our experiments so far have focused on low-resource setups (a training set of several thousand examples). We now evaluate how SWEET performs as we vary the amount of training data. We fine-tune a BERTBASE model on different portions of the MNLI dataset using the same experimental setup as in Section 3.2. Our results, shown in Fig. 8, indicate that SWEET is mostly beneficial when training data is limited to a few thousand examples, but still somewhat useful even with as many as 60K training instances. Nonetheless, the effects of conflicting gradients on the earliest EarlyExit classifier tend to disappear when training on the full dataset (400K instances), making it as good as the smallest Multi-Model classifier. Moreover, in that setup, the use of SWEET seems to substan-
tially decrease the performance of later classifiers, suggesting that the harm caused by limiting the influence of classifiers on model parameters may grow with the amount of training data.

heading: 6 Related Work
text: Several approaches have been proposed for reducing the inference time of large models (Treviso et al., 2022). These include knowledge distillation (Ba and Caruana, 2014; Hinton et al., 2015), in which the knowledge of a large teacher model is transferred to a smaller student model; pruning (LeCun et al., 1989; Frankle and Carbin, 2018), which removes unnecessary model parameters; and weight quantization (Jacob et al., 2018; Zafrir et al., 2019), which reduces the floating point precision of model weights. See (Treviso et al., 2023, Section 6), for a recent survey. Adaptive inference methods studied in this work are an orthogonal approach, which can be used in combination with these methods (Schwartz et al., 2020b).
The use of Adaptive inference in deep neural networks has been extensively studied, with successful implementations in various types of networks including recurrent neural networks (Graves, 2016) and convolutional neural networks (Teerapittayanon et al., 2016; Huang and Chen, 2018). In the context of this work, it has also been applied to existing backbone pre-trained language models: Xin et al. (2020) implemented early exit strategies on top of classification tasks. Zhou et al. (2020) Introduced a patience-based exiting, requiring sequential classifiers to agree on a label for a prediction to be made. Xin et al. (2021) extended the use of Early-Exit in Transformers to regression tasks, as well as addressed the issue of reduced performance of the final classifier through the use of an alter-
nating training algorithm. Schuster et al. (2021) proposed a confidence-based early exit model with guarantees on the agreement between early exit predictions and the final classifier.
Liu et al. (2022) presented a strong baseline for efficient NLP by adding multiple classifiers to a BERT model during pre-training. Recently, Schuster et al. (2022) adjusted the Early-Exit method to language modeling for text generation, making dynamic computation at the single token level.
Multi-Model approaches to adaptive inference have been proposed for vision tasks (Enomoro and Eda, 2021) as well as for NLP (Li et al., 2020; Varshney and Baral, 2022b). Mamou et al. (2023) introduced a two-tier design with a highly efficient model serving as the first model and a powerful model serving as the second, enabling the possibility of achieving extremely fast inference speed.
Finally, the concept of conflicting gradients has been mainly studied in the context of multi-task learning, where a single model is faced with solving different tasks (Yu et al., 2020; Liu et al., 2021a). To the best of our knowledge, no previous work examined this in the context of Early-Exit.

heading: 7 Conclusion
text: In this work, we analyzed the performance of two common adaptive inference methods–Early-Exit and Multi-Model. We found evidence that model weights are updated by conflicting gradients in the training process of Early-Exit models, causing classifiers to perform at a sub-optimal level. Despite this, we showed that regarding the entire speed-accuracy curve, Early-Exit is still favorable to Multi-Model due to the overhead of using independent model runs in a Multi-Model setup.
To address these findings, we proposed SWEET,
a novel Early-Exit method, which avoids conflicting gradients by allocating each Early-Exit classifier a subset of model weights which are updated solely by it. We found that for Early-Exit models trained with SWEET, early classifiers perform better than those of standard Early-Exit, but later classifiers of SWEET are not as good. These measures lead to SWEET outperforming both Early-Exit and Multi-Model in fast speeds, with slightly worse results than Early-Exit at slow speeds.
Overall, our results demonstrate that Early-Exit models can benefit from fine-tuning algorithms that are tailored to their architecture, and that SWEET is a promising approach for improving the speedaccuracy trade-off of Early-Exit models in the context of adaptive inference.

heading: 8 Limitations
text: This work focuses on the effects of adaptive inference in a low-resource setting, specifically when training data is limited. Our experiments (Section 5) suggest that the negative impact of conflicting gradients may be less prominent when larger amount of training data is available.
Our experiments were conducted using relatively small pre-trained language models (≤ 350M parameters) due to computational constraints, and we defer the replication of our findings with larger, more powerful models to future work. Nonetheless, our results have important implications for the growing trend of increasingly large language models. We hope this work inspires further research on methods to reduce the computational cost of NLP.
This work concentrates on evaluating the speedaccuracy trade-off of Multi-Model and Early-Exit at inference time. We recognize that there are additional factors, such as memory usage, batch processing, and training duration, that could be considered when comparing these methods.
Finally, we experimented with seven text classification tasks in English. We recognize that results may vary for other tasks and languages.

heading: Acknowledgements
text: We acknowledge Yarden Tal for her early contributions to this work, and Gabriel Stanovsky for his advice and meaningful feedback. This work was supported in part by the Israel Science Foundation (grant no. 2045/21), NSF-BSF grant 2020793, and by a grant from Intel Labs.

heading: B Conflicting Gradients
text: We replicate the experiment done in Section 3.1 with another batch of size 16 to examine if our findings generalize. results presented in Fig. 9 show
15For LARGE models we evaluate at 1/24 as the fastest speedup ratio.
16For Multi-Model we also use (13/8, 13/4) as overhead causes the model to perform at such "speedup" ratios.
a similar trend to Fig. 3: Gradient updates of current classifiers are roughly orthogonal to those of future classifiers, whereas future classifier updates are more aligned.
C-1 C-2 C-3 C-4
C-1
C-2
C-3
C-4
1.0 0.0 -0.0 0.01
0.0 1.0 0.54 0.26
-0.0 0.54 1.0 0.62
0.01 0.26 0.62 1.0
Layer 1 C-2 C-3 C-4
C-2
C-3
C-4
1.0 0.03 -0.0
0.03 1.0 0.4
-0.0 0.4 1.0
Layer 4
C-3 C-4
C-3
C-4
1.0 0.04
0.04 1.0
Layer 6
0 0.25 0.5 0.75 1 Avg. cosine similarity
Figure 9: Average cosine similarity between classifiers’ gradient updates of model layers. C-i stands for Classifier i. Layer 1 (preceding C-1) is updated by 3 classifiers, while layers 4 (preceding C-2), and 6 (C3) are updated by 3 & 2 classifiers respectively. For each layer, the gradient update of the following classifier is roughly orthogonal to those of future classifiers, whereas gradient updates of higher classifiers tend to better align with one another.

heading: C DeBERTa Individual Layer Comparison
text: Table 3 Shows individual classifier results for DeBERTa models. As with the BERT results, Multi-Model classifiers outperform corresponding Early-Exit classifiers. SWEET early classifiers are better than Early-Exit ones, while later classifiers tend to downgrade in performance.
D Individual Task Results
Fig. 10 shows results on individual tasks for BERTLARGE and DeBERTa (BASE & LARGE). For BERTLARGE, SWEET outperforms both baselines throughout the entire speed-accuracy curve over all tasks examined. For DeBERTa models, results are similar to those of BERTBASE, where SWEET performs better at high speeds (small speedup ratio) and is dominated at low speeds (where later classifiers do most of the heavy lifting).
Size Method Exit Layer
1 4 6 12
BASE MM 60.10.4 74.60.6 78.31.1 82.20.7 EE 52.30.4 70.90.6 77.90.8 81.70.1
SWEET 59.11.5 72.40.2 75.11.1 79.50.5
1 6 12 24
LARGE MM 61.90.1 76.20.4 80.20.9 86.80.2 EE 55.80.9 74.71.1 79.20.8 83.61.9

heading: E Learning-to-exit Individual Layer Comparison
text: Table 4 Shows individual classifier results for BERT models fine tuned with the learning to exit strategy. As with the confidence-based results, Multi-Model classifiers outperform corresponding Early-Exit classifiers. SWEET early classifiers are better than Early-Exit ones, while later classifiers tend to downgrade in performance.
Size Method Exit Layer
1 4 6 12
BASE MM 53.70.5 66.10.4 71.10.3 75.42.7 EE 49.51.1 63.00.2 70.00.6 75.30.4
SWEET 53.50.4 64.90.9 68.51.4 74.10.4
1 6 12 24
LARGE MM 51.60.3 59.90.4 67.62.6 77.90.8 EE 50.50.8 59.30.9 68.81.3 76.61.6
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8
7 A2. Did you discuss any potential risks of your work? Our work does not present any risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: We used ChatGPT for assistance, purely with the language of the paper
B 3 Did you use or create scientific artifacts? Section 3 + 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 + 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use publicly available data sets which are commonly used during research
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Such terms were not specified.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use publicly available data sets which are commonly used during research
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We use publicly available data sets which are commonly used during research
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 + 4, Appendix A
C 3 Did you run computational experiments? Section 3 + 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 1 + 3, Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 + 4, Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 + 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: 3: Gradient updates of current classifiers are roughly orthogonal to those of future classifiers, whereas future classifier updates are more aligned. C-1 C-2 C-3 C-4
C-1
C-2
C-3
C-4
1. 0 0. 0 -0. 0 0. 01
0. 0 1. 0 0. 54 0. 26
-0. 0 0. 54 1. 0 0. 62
0. 01 0. 26 0. 62 1. 0
Layer 1 C-2 C-3 C-4
C-2
C-3
C-4
1. 0 0. 03 -0. 0
0. 03 1. 0 0. 4
-0. 0 0. 4 1. 0
Layer 4
C-3 C-4
C-3
C-4
1. 0 0. 04
0. 04 1. 0
Layer 6
0 0. 25 0. 5 0. 75 1 Avg. cosine similarity
Figure 9: Average cosine similarity between classifiers’ gradient updates of model layers. C-i stands for Classifier i. Layer 1 (preceding C-1) is updated by 3 classifiers, while layers 4 (preceding C-2), and 6 (C3) are updated by 3 & 2 classifiers respectively. For each layer, the gradient update of the following classifier is roughly orthogonal to those of future classifiers, whereas gradient updates of higher classifiers tend to better align with one another