How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Authors: Piotr Kubaty
Bartosz Wójcik
Bartłomiej Tomasz Krzepkowski
Monika Michaluk
Tomasz Trzcinski
Jary Pomponi
Kamil Adamczewski
Venue: ICML 2025 poster (2025)
Keywords: early exiting
multi-exit models
conditional computation
dynamic neural networks
efficient inference
Primary area: deep_learning->everything_else
TL;DR: We identify and address gaps in early-exiting literature by analyzing the weaknesses of commonly used multi-exit model training strategies.

Abstract: Early exits enable the network's forward pass to terminate early by attaching trainable internal classifiers to the backbone network. Existing early-exit methods typically adopt either a joint training approach, where the backbone and exit heads are trained simultaneously, or a disjoint approach, where the heads are trained separately. However, the implications of this choice are often overlooked, with studies typically adopting one approach without adequate justification. This choice influences training dynamics and its impact remains largely unexplored. In this paper, we introduce a set of metrics to analyze early-exit training dynamics and guide the choice of training strategy. We demonstrate that conventionally used joint and disjoint regimes yield suboptimal performance. To address these limitations, we propose a mixed training strategy: the backbone is trained first, followed by the training of the entire multi-exit network. Through comprehensive evaluations of training strategies across various architectures, datasets, and early-exit methods we present strengths and weaknesses of the early exit training strategies. In particular, we show consistent improvements in performance and efficiency using the proposed mixed strategy.

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Piotr Kubaty" 1
Bartosz Wojeik *1
Bartlomiej Krzepkowski Monika Michaluk Tomasz Trzcinski 245
Jary Pomponi
Kamil Adamczewski 75
Abstract
0.70
Early exits enable the network's forward pass to terminate early by attaching trainable inter- nal classifiers to the backbone network. Existing early-exit methods typically adopt either a joint training approach, where the backbone and exit heads are trained simultaneously, or a disjoint ap- proach, where the heads are trained separately. However, the implications of this choice of- ten overlooked, with studies typically adopting one approach without adequate justification. This choice influences training dynamics and its im- pact remains largely unexplored. In this paper. we introduce set of metrics to analyze early-exit training dynamics and guide the choice of training strategy. We demonstrate that conventionally used joint and disjoint regimes yield suboptimal perfor- mance. To address these limitations, we propose a mixed training strategy: the backbone is trained first, followed by the training of the entire multi- exit network. Through comprehensive evaluations of training strategies across various architectures, datasets, and early-exit methods, strengths and weaknesses of the early exit training strategies. In particular, we show consistent im- provements in performance and efficiency using the proposed mixed strategy.
0.65
Accuracy
0.55
0.50
VITT
Joint
0.45
Mixed
Disjoint
1.1
Inference FLOPs
Figure I. Performance-cost trade-off of the multi-exit VIT-T model trained on the ImageNet- Ik dataset by using different training regimes. The choice of the training regime significantly impacts the performance across all computational budgets.
of these networks significantly contributes to their enhanced performance, the necessity of using large models for all inputs, especially in resource-constraneed environments like mobile and edge computing devices, is questionable.
Early exit methods for deep neural networks have gained importance due their potential significantly improve computational efficiency. By exiting earlier layers, these methods can decrease the number of operations needed for computation of the forward pass, leading to faster infer- ence times. In doing so they allow the network to adapt its computational cost to the difficulty of the input sample. Simpler inputs can be processed with fewer layers, while more complex inputs can utilize the full capacity of the network.
present the
1. Introduction
Deep neural networks have achieved remarkable results across a variety of machine learning tasks. While the depth
Early exit methods are implemented through augmentation of the original architecture with internal classifiers (ICs) attached to selected intermediate layers (Kaya et al., 2019). These ICs are designed to perform classification tasks based on the representations available at their respective positions in the network. A common approach for training early- exit models involves training the entire multi-exit network, including the added classifiers, from scratch (Huang et al., 2018; Yang et al., 2020; Meronen et al., 2024) ("joint" regime). Alternatively, some methods train the backbone
"Equal contribution. t Equal supervision. 'Jagiellonian Uni- versity 'Warsaw University of Technology Uniiversity of Warsaw "Tooploox 'IDEAS Research Institute DDpparmeent of Informa- tion Engineering, Electronics, and Telecommunications (DIET) Sapienza, University of Rome, Italy 'Wroclaw University of Science and Technology. Correspondence to: Bartosz Wójeik <bartwojo@ Saattojo@@gmai..omm gmail.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
network first, then freeze its weights and train the parameters of the newly added ICs in the second, separate phase of training (Teerapitayynon et al., 2016; Liao et al., 2021; Zhou et al., 2020) ("disjoint" regime). To the best of our knowledge, no study compares or explores the relationship between these training regimes.
low frequency elements. This regularity is disrupted in the case of early exit architectures the backbone network is given additional classifiers that are placed in earlier parts of the network. These changes in architecture require a different approach for training and more nuanced analysis how the training should proceed.
In this study, we perform an extensive assessment of early- exit regimes and notice the choice of training strategy has significant impact the final model's 's performance. We identify the relationship between computational budget and the choice of the regime. Using the disjoint regime results with a network that is significantly impaired when smaller computational budget is assumed. While the joint regime might initially seem as the appropriate way of training multi- exit networks, we demonstrate that due to its training dy- namies it biases the network and produces model with subpar performance on higher computational budgets.
In the early-exit setting, parameters can be divided into backbone parameters and internal classifier (IC) parameters. Each of these two groups can be trained separately or jointly. In this paper, we frame the training process of any early-exit method as consisting of three following phases:
Train the backbone network parameters 4 by minimizing the loss at the final output layer (could be the last IC or an added final classifier).
Phase 1:
[C(K) o((
0g arg min DE (a On
(I)
In order to address the weaknesses of multi-exit networks, we propose a novel "mixed" regime: train the backbone net- work until convergence, then train the entire model jointly, including the internal classifiers, until convergence. This ap- proach ensures that the backbone architecture adequately trained before optimizing it alongside the internal classifiers for improved performance.
During this phase, Ire are either not trained or not present at all.
Phase 2: Train both the backbone network and the ICs simultaneously.
0- = arg min ar B. (0, ofa)
(2)
To gain a deeper understanding of learning and optimization in multi-exit architectures, we propose a set of metrics to describe the training dynamics of early-exit models trained under various regimes. Through gradient dominance metric reveal the set of ICs that have the largest impact the backbone during training and explain the performance gains of popular technique of loss and gradient scaling depend on the training strategy choices. Furthermore, mode con- nectivity allows to analyze solution similarities of different regimes. Moreover, numerical rank, and mutual information metrics aid to explain the performance of regimes under different computational budgets.
Phase 3: Freeze og and train only the IC parameters Inc.
Σ on (e (1 [
Occ arg min Oi:
(3)
Correspondingly, we generalize the early exit training regimes into three types based on which of the phases are used:
Disjoint training (Phases 1+3). The model parameters undergo training during the first and third phases. That is, the backbone architecture is trained first, and then the ICs are trained separately while the backbone parameters are frozen.
Finally, we provide a thorough empirical evaluation of early- exit regimes across different network architectures, data modalities, datasets and carly-exit methods. Our results show that proposed alternative strategy enables significant improvements in performance in medium and high budgets over the commonly used joint training.
Joint training (Phase 2). The training consists only of the second phase in which the entire model - including the ICs is trained. It is currently the most common way of training early-exit models (Matsubara et al., 2022).
2. Training Regimes
Mixed training (Phases 1+2). The training consists of two phases. The backbone is trained in isolation first, and then the entire network, including the ICs, is trained jointly. The regime emphasizes the importance of backbone pre- training as a better way to initialize the architecture for further training. This our proposed way to improve multi- exit model training.
Early exit methods fundamentally alter the organization of neural networks. It is widely believed that neural networks develop a hicrarchical representation of features, where car- lier layers learn basic shapes and patterns, while later layers progressively capture more complex abstractions (Zeiler & Fergus, 2014). In other words, the earlier layers are charac- terized by higher frequency features while later layers learn
2

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
3. A Framework for Analyzing Multi-Exit Models
( = (1- + A
(4)
This section proposes a framework for analysing and com- paring multiple early exit models. This framework com- prises multiple evaluations, each focusing on a different aspect of a model.
The core idea is based on the fact that, if two models trained with different strategy can be interpolated such that the resulting model has low loss over the whole interpolation path, then the regime solutions lie in the similar loss basin.
3.1. Gradient Dominance
3.3. Numerical Rank
The use of internal classifiers during training in early-exit training regime fundamentally alters the training dynamics, as these classifiers contribute to the overall loss. The gradi- ent update now comes from multiple classifiers instead of just the final one, as in a standard neural network.
So far, the proposed tools analyse the gradients of the model or their final predictions. Here, we propose to evaluate the expressiveness of a model by analysing its intermediate activations, using the numerical ranks of activation maps (Masarczyk et al., 2024). Mathematically, for given layer i, the rank is evaluated as:
To study this phenomenon we calculate which gradient is more prominent during training. We define gradient dom- inance (GD) as the cosine similarity between the gradient from an individual internal classifier, gi, and the overall gradient of the model gooal:
r3 = Rank(A)), A; E RRxm
(5)
where As is the activation matrix of dimensions n (number of samples) and (number of features).
The rank of the internal representations associated with dif- ferent layers can provide insight into the "expressiveness" or capacity of the network. A higher rank (closer to the maximum possible for a given layer's 's matrix dimensions) indicates that the layer can capture more complex patterns or features in the data, as it implies a greater degree of lin- ear independence among the feature detectors in that layer. High-rank activations matrices in a network suggest that the network is utilizing its capacity to learn diverse, high- frequency features, whereas low rank might indicate that the network is not fully exploiting its potential.
GD; << gi Siotal
where < <.,-> >..>>a is the cosine similarity. Gradient Domi- nance measures the consistency of the gradient directions produced by the early-exit classifiers and evaluates how well gradients from separate classifiers align with the overall gra- dient across the entire model. If the cosine similarity is close to I, the auxiliary classifier's gradient is highly aligned with the total gradient, indicating that it potentially dominates other ICs in its impact the total gradient.
3.2. Mode Connectivity
3.4. Mutual Information
Mode connectivity theory suggests that independently trained models often exhibit similar characteristics. No- tably, after training two independent models, it is possible to find a continuous path in the parameter space where the loss remains low, enabling the models to be connected without encountering high-loss regions (Garipov et al., 2018).
Adding intermediate early exits to a model could alter the information flow within the model itself. To study how much such exits affect this aspect, we use the concept of mutual information. In the context of neural networks, the mutual information between X and Z represents how much information the input X provides about the internal repre- sentation Z after passing through neural network. For random variables X and 7, the mutual information is de- fined as: 1(X; = SnEx Sre p(z, 2) log p(z)p(z)
Building on the observation that independently trained neu- ral networks can be linearly connected in weight space after accounting for permutation symmetries, as described in (Ainsworth et al.), we extend this idea to carly-exit architec- tures trained under different regimes.
p(x.s) da dz where p(ar,)) is the joint probability distribution of X and Z, and p(æ) and p(2) are the marginal distributions of X and Z, respectively. In practical terms, for neural networks, we use Monte Carlo sampling to estimate I [(X; Z) due to the high dimensionality of feature spaces, and as such the re- sults are not always monotonically decreasing (Shwartz-Ziv & Tishby, 2017).
Instead of focusing solely on independently trained net- works, we investigate early-exit architectures trained in distinet regimes. To align models for mode connectivity, we define an optimal permutation II" by solving II" arg minn ||O 1T where || ||r is the Frobenius norm, and A and θg are the parameters of two early- exit models trained under different strategies as described in Sec. 2. Given the optimal permutation, the permuted interpolation path is:
In their work, (Kawaguchi al., 2023) utilize the concept of mutual information between X and Z to study the infor- mation bottleneck (IB) principle. It aims to find a balance
3

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
64
0.
8
0.
-02-
0.0
0.7
0.1
0.5
0.3
1.0
inining progress
Taining proguesr
(a) Joint regime
(b) Mixed regime
(c) Avg. difference between joint and mixed
Figure 2. Gradient dominance for different regimes. Each line indicates how well gradients from different ICs align with the total gradient over the course of the training. The last IC dominates the most in the mixed regime, which explains its excellent performance on higher computational budgets (Re Tinyimagenet).
between the information carried by Z in predicting the tar- get and its complexity in terms its mutual information with the input X. Specifically, minimizing [X;; Z) reduces the complexity and overfitting by ensuring Z retains only the essential information from X, and maximizing I(Y; Z) ensures that the representation Z is informative enough to predict the target variable Y effectively. To visually show how mutual information is affected by the training regime, we calculate the mutual information for each layer averaged over multiple inputs.
the Cosine Annealing scheduler with warm restarts. To ensure fair convergence across different regimes, we incor- porate an early stopping mechanism. Training is terminated only when, over consecutive epochs, none of the exits achieve an improved performance compared to their best scores recorded thus far. These scores accuracy for classi- fication tasks and loss for regression tasks are evaluated on a dedicated early-stopping validation set. All results in this section are averaged over three runs, each one having a different initial seed.
Evaluation protocol. The core of our analysis is the frame- work proposed to understand how adding early exits affects the model's internal state and hence the predicted values. To this end, in Section 4.2, we use the proposed framework to compare all the training regimes. Additionally, following the core idea that a model having early exits must be used to spare computational power, we evaluate all the resulting models under the lens of FLOPs saved and the obtained accuracy. These results presented in Section 4.3.
4. Empirical Evaluation of Training Regimes
4.1. Experimental Setup
In this section, we outline the setup for our empirical experiments. We release the source code of our ex- periments at: https:://// /gthhbb..uuu./gg wtt./////aaaaaaaeaaaa../aaaaaa o flly-exxit-becchmee
A more detailed description can be found in Appendix E.
Architectures and datasets. We conduct experiments across various datasets spanning computer vision (CV) and natural language processing (NLP). For CV, we uti- lize CIFAR-100 (Krizhevsky, 2009), ImageNet-Ik (Rus-
4.2. Framework Evaluation
In this section, we evaluate the training regimes through the framework proposed in Section 3.
sakovsky et al., 2015). TinylmageNet (Le & Yang, 2015), and Imagenette (Howard, 2019). For NLP, we evaluate on 20-Newsgroups (Lang, 1995) and STS-B (Wang et al., 2019) datasets. As far vision architectures are concerned, we evaluate ResNet (He et al., 2016) and Vision Transformers (ViT) (Dosovitskiy et al.). Additionally, we explore MS- DNet (Huang et al., 2018), an architecture dedicated for multi-exit models. For NLP tasks, we utilize BERT (Devlin et al., 2019). Unless otherwise specified, the Shallow-Deep Network (SDN) (Kaya et al., 2019) early exit model is im- plemented on top of chosen backbone.
Gradient Dominance. In Figure 2, we present the gra- dient dominance results for the joint and mixed regimes, highlighting their distinct training dynamics. In the joint regime, optimization focuses on subnetworks in the mid- dle of the architecture, where gradients from intermediate classifiers exert the strongest influence. Conversely, in the mixed regime, gradients are dominated by deeper classifiers, causing the early layers to primarily support the learning objectives of later classifiers, potentially at the expense of earlier ones.
The dominance of the final intermediate classifier (IC) in the mixed regime suggests its suitability for scenarios with
Training set-up. We train our models, employing the Adam optimizer (Loshchilov & Hutter, 2019) alongside
4

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
2.00
Resivetse Ei dedd
SON (imowed) Moad
Maxed
2500
1.50
1.25
1500
0.75
0.50
Disjoint
JoinE
0.25
Figure 4. Numerical ranks as measured in the backbone network of the multi-exit model which was trained with different regimes. The rank increases when the ICs are allowed to affect the backbone network.
Figure 3. Mode connectivity between models trained with different training regimes. Colors represent the values of loss function, with yellow representing high loss (2 2.0). Disjoint and mixed regimes produce similar models, while the model trained in joint regime lies in a different basin (ResNet-20, CIFAR-10).
the layers, becoming comparable to the numerical rank of multi-exit network trained entirely from scratch. This sug- gests that well-functioning multi-exit models require layers with greater expressiveness, which underpins the effective operation all subsequent internal classífiers. Such an in- crease in expressiveness is impeded when employing the disjoint training strategy.
higher computational budgets, which is supported by our empirical results. This behavior mirrors the training of backbone, where the model is primarily guided by the loss from the final classifier. Consequently, the mixed regime tends to produce models that closely resemble standard neu- ral networks, a fact we confirm in the next section through mode connectivity analysis.
Early-exit architectures attach in- termediate classifiers (ICs) to internal layers, altering the distribution of information flow as seen in Figure 5. The effect is two-fold and differs between earlier and later lay- ers. Farlier layers: The mutual information between X and larger compared a network trained without addi- tional classifiers. Deeper layers: The mutual information for early-exit architecture is lower in the final layers.
Mutual Information.
Mode Connectivity, After accounting for permutation sym- metries, the loss remains low during linear interpolation of the weights of models trained in the mixed and disjoint regimes, as shown the mode connectivity plot in Figure This indicates that the resulting weights lie the same basin, and the regimes produce similar solutions. However, the disjoint regime is constrained because it trains only the internal classifiers during the second phase. In contrast, the mixed regime updates the backbone to accommodate the added ICs, resulting in lower overall loss.
The above effect is seen in both regimes but is more pro- nounced in the joint regime. The information flow in the joint regime more skewed and different from backbone- only training. Backbone training in the mixed regime makes the information flow fall between backbone-only and joint training. This is due the fact that the representation of easy samples is not complex (that is, it is processed with just few layers before exiting through an early IC). As
Joint training, on the other hand, leads to a model a differ- basin of the loss landscape. This suggests that training the entire model once, without "pre-trained" backbones, produces solutions that differ significantly in structure and performance.
2.00
1.70
Numerical Rank. In our framework, we analyze the nu- merical rank of the backhone model under different early- exit regimes. A regular neural network typically exhibits higher rank in earlier layers and a lower rank in deeper layers, illustrated in Figure 4. Note that training only the backbone corresponds to the model obtained in the dis- joint regime, this training strategy does not modify the backbone.
$25
2.75
830
100
We observe a distinct change in network expressiveness once intermediate classifiers are attached and permitted to influence the backbone. The numerical rank rises across
Figure 5. Mutual information ((x; 2) between the input X and the internal representation Z of the backbone at different depths for the three multi-exit model training regimes.
5

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Table 1. Testtst accuracy of multi-exit models trained with dif- ferent regimes on the CV datasets. Each column represents the maximum averaged computational budget assumed for the model, indicated a percentage of the computational cost of the back- bone. The mixed regime achieves significant improvements over the commonly used joint and disjoint regimes.
Table 2. Testset accuracy (classification) and MSF (regression) of multi-exit BERT-B models trained using different regimes on the NLP datasets. The results align with those observed for classifica- tion lasks.
Dataset
Regime
25%
50%
75%
100%
Disjoint Joint
68.53
83.75
85.75
85.69
Setting
Regime
25%
50%
75%
100%
Newsgr. (class.)
84.24
84.41
84.41
84.41
Disjoint
52.57
67.56
73.49
73.79
Mixed
84.99
85.25
85.25
85.25
ResNet-34 C-100
Joint
62.84
72.80
74.32
74.17
Disjoint Joint 1.70
2.37
1.55
0.54
0.51
Mixed
62.24
73.81
75.92
75.88
STS-B (regr.)
0.61
0.53
0.52
Disjoint
56.74
63.96
68.59
70.36
Mixed
2.43
0.59
0.53
0.51
MSDNet C-100
Joint
65.93
72.02
74.73
75.86
Mixed
65.94
72.01
75.46
76.51
ure demonstrate similar trend, but also highlighting that the joint regime experiences performance gap at higher computational budgets. The mixed regime per- forms consistently well across all scenarios. Due to space constraints in the main paper we present the result averaged over three runs, but in Appendix C we present these results with standard deviations. In Appendix A we also examine four alternative training regimes.
Disjoint
28.33
47.32
61.50
63.99
ViT-T C-100
Joint
40.87
60.48
66.22
66.49
Mixed
41.65
64.07
70.09
70.25
Disjoint
38.49
49.25
60.50
65.71
ResNet-50 Tiny-IN
Joint
52.98
62.16
65.14
65.01
Mixed
52.89
63.28
67.20
67.24
Disjoint 10.22
35.15
61.23
71.61
VIT-T IN-1k
Joint 36.39
61.89
68.08
68.39
Generalization to Natural Language Processing & Re- gression Tasks. To explore the generalizability of these results, we first conduct similar experiments on natural lan- guage processing data and on a regression task, with the results presented in Table 2. These results are consistent with those observed before, further highlighting the weak- nesses of both the joint and disjoint regimes.
Mixed
35.91
62.96
70.79
71.20
Disjoint
10.23
33.91
68.02
78.38
VIT-S
Joint
50.10
73.99
76.38
76.44
IN-1k
Mixed
49.50
75.17
78.28
78.33
the sample is easy, it is clearly and distinctly located within the boundaries of a single class. To describe it in terms of mutual information, the network does not need to reduce the complexity of X fit the internal representation Z. as X has little irrelevant details. Consequently, the input X is not compressed and the internal representation Z has similar complexity to the representation of X, hence [(X; 2) is higher.
Generalization to Other Early-Exit Methods. To show that our conclusions extend to other carly-exit approaches, we conduct three additional experiments for different early- exit methods: GPF (Liao et al., 2021), PBEE (Zhou et al., 2020), and one that uses entropy instead of max-softmax probability as a proxy for confidence (Tecrapittayanon et al., 2016). The findings from these experiments are consistent with our main results, again showing the effectiveness of the mixed regime.
Following this observation, we note that with higher T(X; in earlier layers, the joint strategy more suit- able for easy datasets where more samples exit earlier layers. Similarly, the mixed regime learns more uniform representation of the T(X; Z) across the network (one may observe an analogy to the numerical rank results in Sec- tion 3.3) and may be preferred for more difficult datasets that exit at later internal classifiers.
Generalization to the Pre-trained Setup. Nowadays, most applications begin with model pre-trained on a large dataset, which is then fine-tuned for specific tar- get task. We emphasize that during the fine-tuning step, any of the multi-exit training regimes can be applied. To assess whether our findings extend to this scenario, we use the weights of pre-trained ViT-B model from the torchvi- sion library and fine-tune it on CIFAR- 100 using the three training regimes. The results, presented in Table 4, demon- strate patterns similar to those observed in the from-scratch training setup. Specifically: (1) the disjoint regime contin- ues to show significantly lower performance under lower computational budgets, and (2) the joint regime still exhibits
4.3. Performance-Cose Evaluation
Table presents the performance of the training regimes on various dataset and architectures. The disjoint regime shows weaker performance under lower computational budgets, as freezing the backbone limits the ability learn effective early-layer representations for classification. The results for ImageNet- 1k. which are also reported in Fig-
6

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Table CIFAR-100 testset accuracy of multi-exit models trained with different regimes on the following early-exit methods: GPF (Liao al., 10 PBEE (Zhou et al., 2020), and entropy- exit criterion (Teerapittayanon et al., 2016). The results are consis- tent with the main experiments.
Table 5. Accuracy improvement the Tinyimagenet dataset when using various IC loss and gradient scaling methods. While gradient equilibrium enhances performance in the joint regime, it does not benefit the mixed regime, highlighting the effectiveness of the mixed regime for training multi-exit models.
EE Method
Regime
25%
50%
75%
100%
Regime
Scaling
25%
50%
75%
100%
Disjoint 19.28
44.11
57.85
63.91
Joint Mixed
Inc.
- 1.7
+0.4
+0.98
+1.06
PBEE
Joint Mixed
28.66
56.49
64.31
66.52
Inc.
-1088
-0.27
+0.22
+0.25
28.87
59.42
68.33
70.33
Joint
Dec.
+0.6
-0.29
-0.08
+0.0
Disjoint
33.16
53.19
62.54
64.01
Mixed
Dec.
+0.69
+0.32
-0.44
-0 -0.46
GPF
Joint 46.21
62.23
66.84
67.19
Joint
SDN
-3.75
0..3
+0.89
+0.94
Mixed
47.21
64.51
68.92
69.10
Mixed
SDN
-3.39
-.1
+0.07
+0.23
Disjoint
27.14
46.09
60.30
63.99
Joint Mixed
GE
+0.31
+0.4
+0.73
+0.71
Entropy
Joint
41.16
58.60
65.69
66.49
GE
-1.16
00.11
+0.02
-0.02
Mixed
41.54
62.20
69.63
70.25
et al., 2019). For the joint regime, gradient scaling improves performance at higher computational budgets. However, for the mixed regime, there is no additional benefit, further confirming that they achieve similar effect. The mixed regime achieve superior results and it obviates the need for application of gradient equilibrium.
Table 4. Test accuracy of ViT-B model pre-trained on ImageNet- Ik and fine-tuned on CIFAR-100. The disjoint regime struggles low budgets, while the joint regime shows slight but notable performance gap at higher budgets. The mixed regime performs consistently well, aligning with the patterns observed in the main experiments.
Regime
35%
50%
75%
100%
4.5. Impact of IC Size
Disjoint
28.50
55.64
87.49
89.95
The size of an internal classifier (IC) in early exit archi- tectures refers to the number of layers and neurons within the internal classifier. Smaller ICs are computationally effi- cient and enable faster early exits with minimal overhead. Conversely, larger ICs offer greater capacity, potentially en- hancing accuracy, but may offset the computational benefits of multi-exit models.
Joint 73.18
85.41
87.82
87.85
Mixed
73.38
85.70
88.06
88.23
a performance gap higher computational budgets.
4.4. Impact of Gradient and Loss Scaling
The gradient dominance results demonstrated how the mixed regime prioritizes deeper intermediate classifiers. A similar effect also be achieved by adjusting the loss coefficients for deeper ICs, as proposed in prior work (Kaya et al., 2019; Han et al., 2022), or by scaling the gradients of each IC, as proposed by Li et al. (Li et al., 2019). We revisit these tech- niques within both the joint and mixed regimes to evaluate their impact, with the results presented Table
A recent study by (Wójcik et al., 2023) investigated the impact of IC size and found that larger heads significantly improve performance. However, this analysis focused solely on the disjoint training regime. In our study, we examine the effect of varying IC sizes using VIT-T model trained the Tinylmagenet dataset. Each IC consists of either one two fully connected layers with hidden dimensions of 1024 or 2048, followed by softmax layer. As summarized in Table 6, under the disjoint regime larger ICs indeed yield clear gains: moving from single-layer IC with 1024 units to a two-layer IC with 2048 units can increase top-1 accuracy by up 7.6 percentage points, and this effect is especially visible for lower budgets. However, in the mixed regime this trend reverses, and the change can result in an actual performance degradation. These results demonstrate that optimal IC architecture depends critically on the chosen training strategy, and that conclusions drawn under a disjoint setup may not transfer when the backbone is also trained.
We first evaluate the loss scaling method proposed by Kaya et (Kaya et al., 2019) (SDN) by training multi- exit ResNet-50 models on the TinylmageNet dataset. We also test (Han et al., 2022) constant loss weighting scheme, which adjusts the loss coefficients by either increasing (Inc.) or decreasing (Dec.) them along the model's depth. These schemes allow the model to prioritize different computa- tional budgets. However, the improvements observed in the mixed regime are notably smaller compared to those seen in the joint regime.
Finally, we evaluate the gradient equilibrium method (Li
7

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Table The effect of varying head size the ViT-T model. Larger heads translate to inferior performance for the mixed regime. In contrast, the disjoint model clearly benefits from larger ICs when small budgets are considered.
Table 7. The effect of varying head placement schemes the SDN
early-exit architecture with Re as backbone, trained on the Tinylmagenet dataset.
Scheme
Regime
25%
50%
75%
100%
Regime
IC arch.
25%
50%
75%
100%
Disjoint 38.92
49.25
60.10
65.76
IL 26.35
41.69
53.06
56.72
Every-1
Joint
52.20
62.49
65.52
65.59
Disjoint
2024
33.85
47.21
54.97
56.72
Mixed
52.22
63.03
67.21
67.35
21.-2048
33.96
45.98
54.74
56.72
Disjoint
37.34
48.03
60.34
65.65
IL 2024
42.47
53.24
56.02
56.03
Every-2
Joint
51.81
62.60
65.55
65.38
Joint
45.59
55.11
57.73
57.66
Mixed
52.41
63.19
67.14
67.33
2L-2048
43.60
53.89
56.94
57.00
Disjoint
47.91
60.95
65.77
IL
44.03
57.91
60.42
60.28
Every-3
Joint
63.33
67.22
67.21
Mixed
202-1024
44.92
57.11
59.32
59.22
Mixed
62.52
66.72
66.71
2L-2048
44.61
56.94
60.18
60.15
Disjoint
41.32
57.78
65.78
Every-4
Joint
62.54
66.30
66.27
4.6. Impact of IC Placement Scheme
Mixed
62.07
67.08
67.14
The scheme of placing the internal classifiers in early exit architectures refers to where these classifiers are inserted at different layers within the model. This can range from being placed at every layer to being placed at strategically selected layers. To explore the impact of the placement scheme on the final performance, we train a ResNet-50 model with different regimes for every placement scheme. An IC can be placed at any block from index 0 to 14 for this architecture; however, we do not place it at index 0, as this would entirely bypass the backbone. In the Every-n placement scheme, an IC is inserted at every n-th block. In the Pense-Sparse configuration, ICs are placed at blocks: 2,, 3, 4, 5, 6, 7, 11], while in the Sparse-Dent Dense scheme, they are placed at blocks: [1, 4, 8, 9, 10, 11, 12, 13, 14).
Disjoint
39.85
56.20
65.72
Every-5
Joint
61.10
65.61
65.79
Mixed
61.95
67.33
67.40
Disjoint 38.47
50.64
62.04
65.74
Den.-Spa.
Joint Mixed
53.14
62.23
64.76
64.93
53.48
63.17
66.24
66.27
Disjoint 37.12
47.03
59.79
65.68
Spa.-Den.
Joint
50.47
61.19
65.36
65.42
Mixed
51.19
62.27
67.26
67.47
was the fastest and took 523 (=156, averaged over multiple seeds) epochs on average. Joint training converged after 1610 (+335), while mixed training required 1166 (+13 epochs. These results indicate that while disjoint training converges most rapidly, the mixed regime offers substan- tial speed-up over joint training by alleviating the interfer- ence effect between intermediate classifiers.
The placement scheme plays significant role in shaping network performance across different training regimes, as shown in Table 7. While densely placed ICs generally en- sure strong performance across a range of computational budgets, the model can be tailored to prioritize a particular budget by using the Dense-Sparse or Sparse Dense schemes. In most cases, models trained with the mixed regime outper- form those trained under other regimes.
5. Related Work
Early exiting is notable application of the conditional com- putation paradigm (Bengio al., 2013). While conceptually similar to earlier classifier cascades (Xu et al., 2014; Wang et al., 2018), it differs in that all classifiers are integrated within a single model, enabling end-to-end training. The first multi-exit model was introduced by (Teerapitaayann et al., 2016), and the field has expanded considerably since its inception.
4.7. Training time
The ultimate aim of majority of multi-exit models is achieving the Pareto frontier in model performance and computational cost of the model during inference (Scarda- pane et al., 2020; Han et al., 2021). As such, to prevent under-training, we apply an early-stopping criterion in every experiment in this work. However, this raises an interest- ing question: how long does it take for the training in each regime converge?
Joint training is the most widely used and well-established strategy for early-exit models (Matsubara et al., 2022). This approach has been successfully applied to dynamic infer- ence under various constraints, such as energy or time limita- tions (Wang et al., 2020), and extended to diverse early-exit
We explore the training time of the ResNet-50 model trained on the TinyImagene dataset (see Table 1). Disjoint training
8

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
applications, including low-resolution classification (Xing al., 2020), quality enhancement (Yang al., 2020), and Question-Answering systems (Soldaini & Moschitti, 2020). While joint training has proven effective, several studies have demonstrated significant improvements through modi- fications of the training process. For instance, knowledge distillation from the final classifier to earlier internal classi- fiers has been shown to enhance their performance (Phuong & Lampert, 2019; Li et al., 2019; Liu et al., 2020). Similarly, ensembling multiple intermediate classifiers can improve the prediction accuracy (Qendro et al., 2021; Sarti et al., 2023). The Global Past-Future (GPF) method (Liao et al., 2021) goes a step further and incorporates information from both earlier predictions and surrogate later predictions to improve inference. Additionally, recent works (Han al., 2022; Yu et al., 2023; Chataoui et al., 2023) identify a train- test mismatch in conventional multi-exit approaches and propose strategies that address this issue, further enhancing the robustness of early-exit models.
vide a detailed analysis and evaluation of various training approaches for early-exit architectures in deep neural net- works. Our findings reveal that the manner in which the backbone and internal classifiers are trained has a significant impact on the performance and efficiency of these models. Furthermore, we show that when using the mixed training regime, gradient scaling is unnecessary. Below we sum- marize some practical takeaways for the selection of the training regime.
Mixed. Mixed regime demonstrates substantial robustness across various factors, including different data modalities and early-exit approaches with varying exit criteria. There- fore, the mixed regime is generally preferred, combining the benefits of both disjoint and joint training. The mixed regime ensures that the backbone network well-optimized before integrating internal classifiers, leading to improved computational efficiency and accuracy. It is particularly recommended for cases where the performance on medium and high computational budgets is the main requirement.
SDN (Kaya et al., 2019) was one of the first to explore the training of early-exit models through the pre-training of the architecture's 's backbone followed by separate training of the classifiers. Multiple subsequent works have focused on optimizing carly-exit models based only on this setup (Wójcik et al., 2023; Lahiany & Aperstein, 2022; Liu et al., 2020), potentially limiting the general applicability of their findings. For instance, (Wolezyk et al., 2021) employ an ensembling technique that combines predictions from ear- lier internal classifiers, weights of which are trained in a separate, third training phase. (Lahiany& & Aperstein, 2022) propose PTEENet, which augments pre-trained networks with confidence heads that dynamically adjust based on available resources and unlabeled data.
Joint. The joint regime is simple to implement and can perform well for small computational budgets. However, it generally underperforms, especially for medium to higher budgets, which are more relevant in practical applications.
Disjoint. This regime is generally inferior compared to the others in most setups, but performs well for the highest budgets. It may be preferred when the backbone is shared, or the lack of resources prevents us from training the backbone network.
A Acknowlegdements
The research was funded by the program Excellence Initia- tive - Research University at the Jagiellonian University in Kraków.
(Kaya et al., 2019) were the first to explore both joint and disjoint training approaches for early-exit models. These approaches are also briefly reviewed in surveys such as (Scardapane et al., 2020; Matsubara et al., 2022). Further- more techniques like weighting the losses at each exit head (Zhou et al., 2020; Kaya et al., 2019; Han et al., 2022) or scaling the individual gradients (Li et al., 2019) can be re- garded as variations of the joint training paradigm. In the
This research has been supported by a grant from the Faculty of Mathematics and Computer Science under the Strategic Programme Excellence Initiative at Jagiellonian University.
This research was partially funded by National Science Centre, Poland, grant no 2022/45/B/ST6/028117
context of LLMs, a concurrent work of Bae et al. (2024) in- cludes an empirical comparison of joint and disjoint training strategies, with similar conclusions about the weaknesses of both approaches. To the best of our knowledge, our work is the first to present a broad and systematic analysis of the carly-exit training strategies.
We gratefully acknowledge Polish nigh-performance com- puting infrastructure Pl.Grid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grants no. PLG/2023/016270. PI.G/2024/017188 85, and Pl.G//024/017385, and the LUMI consortium through PLL/2024/070017641
The contribution of Bartosz Wójcik to this research was conducted at the Faculty of Mathematics and Computer Science, and the Doctoral School of Exact and Natural Sciences of the Jagiellonian University.
6. Conclusion
This study emphasizes the critical importance of training regimes for early-exit models, an aspect that has been largely neglected in prior research. Addressing this gap, we pro-
9

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Impact Statement
and fast ensembling of dnns. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8803- 8812, 2018. URL
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
neurips /c http/paper/2118//eee be3087e 1000077744111004cc4c22eeeddeeeed Abstract.. html. 3
Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Repre- sentations. 3
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., and Wang. Y. Dynamic neural networks: A survey. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44 (11):743--7456, 2021.
Bac, S., Fisch, A., Harutyunyan, H., Ji, Z., Kim, S., and Schuster, T. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora. ArXiv preprint, abs/2410.20672, 2024. URL httpt::tt///ae. /ttp:v////xxvvvv////e org/ 9
Han, Y., Pu, Y., Lai, Z.., Wang, C., Song. S., Cao, J., Huang, W., Deng, C., and Huang. G. Learning to weight samples for dynamic early-exiting networks. In European Confer- ence on Computer Vision, pp. 362-378. Springer, 2022. 7,9
Bengio, Y., Léonard, N., and Courville, A. Estimat- ing or propagating gradients through stochastic neu- rons for conditional computation. ArXiv preprint, abs/1308.3432, 2013. URL https: abs/1308. 3433 8
He, K., Zhang. X., Ren, S., and Sun, J. Deep resid- ual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 1..100//CVR.20166622 URL attps:://ol org/ 10. 2t.1009/CVPR.201 6. 90. 4
Chataoui, J., Coates, M., al. Jointly-learned exit and inference for a dynamic neural network. In The Twelfth International Conference on Learning Representations, 2023. 9
Devlin, J.. Chang. M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, Volume (Long and Short Pupers), pp. 4171-4188 86, Minncapolis, Minnesota, 2019. Association for Compu- tational Linguistics. doi: 10.18665//1/N1199140 URL ht https: http//aaaanannnaaaaaaaaaaaaaaaaa NtggNN19-1423. 4
Howard, J. Imagenette: A smaller subset of 10 easily classified classes from imagenet, 2019. URL https: //github. com/ ccom/fastaa/immgenetteee Accessed: 2025-01-28. 4
Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., and Weinberger, K. Q. Multi-scale dense networks for resource efficient image classification. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: /oppnrvveeeeeeeeeeee net forum?
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,, Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. 4
Ld-Hk2a H.-HHaIIxxbb... 1,
Kawaguchi, K., Deng, Z., Ji, X., and Huang, J. How does information bottleneck help deep learning? In Inter- national Conference on Machine Learning, pp. 16049 16096. PMLR, 2023. 3
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layerskip: Enabling early exit inference and self-speeulative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12622-12642, 2024. 13
Kaya, Y., Hong, S., and Dumitras, T. Shallow-deep net- works: Understanding and mitigating network overthink- ing. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3301 -3100. PMLR,
Garipov, T., Izmailov. P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity,
10

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
2019. URL http:: ngs. wwigg.mll....... ww..lr.pp.. press// v97/kayal 9a. htm1. httmm.... 7,4,7,9,1 17
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:// openrev ew net forum? d=BkggRi 4
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, 2009. 4
Lahiany, A. and Aperstein, Y. Pteenet: Post-trained early- exit neural networks augmentation for inference cost op- timization. IEEE Access, 10:69680-6960 2022.
maintainers, T. and contributors. Torchvision: Pytorch's computer vision library. https:
grtorch/visiin 2016. 20
Lang. K. 20 newsgroups dataset, 1995. URL http:// gwone. com/ batoom/jaso////oe jas.ja 2tww:///ossggsrss/s///// Accessed: 2025-01-28. 4
Masarczyk, W., Ostaszewski, M., Imani, E., Pascanu, R., Miloš, P., and Trzcinski, T. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024. 3
Le. Y. and Yang. X. S. Tiny imagenet visual recog- nition challenge. 2015. URL https://api. semanticschon wtw.:/aar..rrrroooopiii6666.
Matsubara, Y., Levorato, M., and Restuccia, F. Split com- puting and early exiting for deep learning applications: Survey and research challenges. ACM Computing Sur- veys, 55(5):1-30, 2022. 2, 8,
4
Li, H., Xu, Z. Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Desa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 639 609663996400 1-6411 2018. URL wwwps:t/p//cm wtt.tttttt huarripps.cc/appee/////////// http:///pccccccesps...oo
Meronen, L., Trapp, M.. Pilzer, A., Yang. L., and Solin, A. Fixing overconfidence in dynamic neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2680-2690, 2024.
Phuong, M. and Lampert, C. Distillation- training for multi-exit architectures. In 2019 IEEE/CVF Inter- national Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October -7-oembmbr 2019, pp. 1355-1364. IEEE, 2019. doi: 10. 10.110//CCC22229000044444 URL https: //to: org 10 1109/ 00144. 9
html. 13
Li, H., Zhang, H., Qi, X., Yang. R., and Huang, G. Improved techniques for training adaptive deep networks. In 2019 IEEE/CVF International Conference on Computer Vision, 2019, Seoul, Korea (South), October 27 November 2019, pp. 1891-1900. IEEE, 2019. doi: 10, 10.1109/ 2019.00198. URL https: /tss:///oo..i http://iooo////.//////ooo/oo wro.oorgg//./.. 1109/ ICCV.2019. 00198. 7,
Qendro, L.., Campbell, A., Lio, P., and Mascolo, C. Early exit ensembles for uncertainty quantification. In Machine Learning for Health, pp. 181-195. PMLR, 2021. 9
Rahmath P. H., Srivastava, V., Chaurasia, K., Pacheco, R. G., and Couto, R. S. Early-exit deep neural network-a com- prehensive survey. ACM Computing Surveys, 57(3) 2024. 13
Liao, K., Zhang, Y., Ren, X. Su, Q. Sun, X., and He, B. A global past-future early exit method for accelerating inference of pre-trained language mod- els. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 2013-2023, Online, 2021. Association for Compu- tational Linguistics. doi: 10. (0.18653/1122021.mmmc// 162. URL htwps:ttt//.......p. / ac: wwt.anooooooorrgoooooo...o.. naacl mmain. 1 62. 2, 6, 9,9
Russakovsky, O., Deng, J., Su, H.,, Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-255 2015. doi: 10. 10.1007/s11263-11 01000-00---11---- 4
Sarti, S., Lomurno, E.,, and Matteucci, M. Anticipate, ensem- ble and prune: Improving convolutional neural networks via aggregated early exits. Procedia Computer Science, 222:519-55 528, 2023. 9
Liu, W., Zhou, P., Wang, Z., Zhao, Z., Deng, H., and Ju, Q. FastBERT: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp. 6035-6044, Online, 2020. Association for Computational Linguistics. URL https://
Scardapane, S., Scarpiniti, M., Baccarelli, E., and Uncini, A. Why should we add early exits to neural networks? Cognitive Computation, 12(5):954 966, 2020. 8, 9, 13
doi: 10. 100186655 ac gelannhology. wrw.yyyrrg2200000 wal.220.cl-ae acl- 537.9

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint 0rXiv:1703.008111 2017. 3
Linguisticx: Main Volume, pp. 91-104, Online, 2021. As- sociation for Computational Linguistics. URL https: / //aclanthology httpr/hhoooooooo./22111111 org/2021.
mamen.mman... 8. 13
Soldaini, L. and Moschitti, A. The cascade transformer: an application for efficient answer sentence selection. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 5697-5708, Online, 2020. Association for Computational Linguistics. URL https:// 504.9 9
Xing. Q. Xu, M., Li, T., and Guan, Z. Early exit or not: Resource-efficient blind quality enhancement for com- pressed images. In European Conference on Computer Vision, pp. 275-292. Springer, 2020. 9
Xu, 7.,, Kusner, M. J., Weinberger, K. Q., Chen, M., and Chapelle, O. Classifier cascades and trees for minimizing feature evaluation cost. The Journal of Machine Leurning Research, 15(1):2113-211
doi: 10. 1020 ac aclanthen sclannhology wtw.e/ooo..oo.oggoee.e wwwp///g.........///. http:/2220.cccclmmmm
Tecrapittayanon, S.. McDanel, B.. and Kung, H.-T. Branchynet: Fast inference via early exiting from deep neural networks. In Proceedings of the International Con- ference on Pattern Recognition, ICPR, pp. 2464-2469, 2016. 2, 7,8
2014. 8
Yang, L.,, Han, Y., Chen, X., Song, S., Dai, J., and Huang, G. Resolution adaptive networks for efficient inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020. Seattle, WA. USA, June 13-19, 2020, pp. 2366-2375. IEEE, 2020. doi: 10. 10.1109/ 2020000000000200004244444 URL ht httpp:://///..mmm / 20.1109/CVPR440
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In
. 2020. 00244. 1,9
7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019, URL https: / cpenreviee. d=r34kk2255777 4
Yu, H., Li, H.,, Hua, G., Huang, G., and Shi, H. Boosted dy- namic neural networks. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pp. 10989- 10997, 2023. 9
Wang, X., Luo, Y., Crankshaw, D., Tumanov, A., Yu, F., and Gonzalez, J. F. IDK cascades: Fast deep learning by learning not to overthink. In Globerson, A. and Silva, R. (eds.), Proceedings of the Thirty-Fouthh Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp- 580-590. AUAI Press, 2018. URL http: ///////uaa../ wwwooorgiiii /uap/218// proceedi ngs / hwwppaapppppaaa /212. pdf. 8
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., and Choe, J. Cutmix: Regularization strategy to train strong clas- sifiers with localizable features. In 2019 IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2019, Seoul, Korea (South). October 27 - -Novmber 2. 2019, pp. 6022-6031. IEEE, 2019. doi: 10.1 10.100CCC.22000900000
URL https: / htt.///.iioo/oo///ooo org/ 10. www.11119......... 00612. 19
Wang, Y., Shen, J., Hu, T.-K., Xu, P., Nguyen, T., Bara- niuk, R., Wang, Z., and Lin, Y. Dual dynamic inference: Enabling more efficient, adaptive, and controllable deep inference. IEEE Journal of Selected Topics in Signal Processing, 14(4):623-633. 2020. 8
Zeiler, M. D. and Fergus, R. Visualizing and understand- ing convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818- 833. Springer, 2014. 2
Wojcik, B., Przewiezlikowski, M., Szatkowski, F., Wotezyk, M.. Balazy, K., Krzepkowski, B., Podolak, I., Tabor, J., Smieja, M., and Trzciáski, T. Zero time waste in pre- trained early exit neural networks. Neural Networks, 168: 580-601, 2023. 7,9
Zhang, H., Cissé, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations,
ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings, OpenReview.net, 2018. URL https:/ http//pppeeeweweeeee/ee/ee net.ewww.eeeeeee htt.rviriii.ee/ooun Ad=r1Ddp1-Rb 19
Wolczyk, M., Wójeik, B., Batazy, K., Podolak, T., Ta- bor, J., Smieja, M., and Trzcinski, T. Zero time waste: Recycling predictions in early exit neural networks. Ad- vances in Neural Information Processing Systems, 34: 2516-2523 2021. 9
Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., and Wei, F. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 1188883318880 2020. 2, 7,7
Xin, J., Tang, R., Yu, Y., and Lin, J. BERxiT: Early ex- iting for BERT with better fine-tuning and extension to regression. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational
12

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
A. Alternative Regimes
In addition to the training regimes discussed the main paper, early-exit surveys describe alternative training strategies (Scardapane al., 2020; Rahmath al., 2024). Moreover, (Xin al., 2021) have proposed the "Alternating" approach, which also attempts to combine the advantages of joint and disjoint regimes. Additionally, we propose a refinement of the mixed regime, termed the mixed-graduse training regime.
We describe these alternative regimes below:
Branch-wise training. In this regime, ICs are trained sequentially, with only the part of the backbone corresponding to the current IC being unfrozen.
Separate training. Similar to branch-wise training, ICs are optimized sequentially. However, at each step, the preceding ICs are jointly trained along with the current one. The entire model is unfrozen during the training process.
Alternating training. Proposed in (Xin al., 2021), this regime alternates between training the backbone independently (Equation (1)) and jointly training the backhone with all ICs (Equation (2)) in each training step.
Mixed-gradual training. Building on the strengths of the mixed regime, we propose a more gradual optimization approach. Training proceeds in m phases, where m is the number of ICs. In the i-th phase, the last ICs are jointly optimized, and no weights are frozen at any stage. This ensures flexibility throughout the training process while allowing the model to adapt more effectively to the optimization landscape. This approach is similar to gradual early exit curriculum proposed by the concurrent work of Elhoushi et al. (2024), with the difference being that we utilize early-stopping, while they enable the next IC at constant intervals of training steps.
In Appendix C, we present the results for these alternative training regimes. Both the branch-wise and separate training approaches demonstrate subpar performance when evaluated under higher computational budgets. Although the alternating training regime resembles the mixed regime in its motivation, its performance consistently falls short of both the mixed and joint regimes. Interestingly, the mixed-gradual regime outperforms the mixed regime, providing strong evidence in favor of our approach to enhancing early-exit optimization.
B. Loss Landscape
The concept of loss landscape in the context of neural networks is crucial for understanding the training dynamics and generalization properties of models. The loss landscape provides visual and analytical representation of how the loss function changes with respect to the model's parameters. By visualizing the loss landscapes of different neural network architectures, we can understand how design choices affect the shape of the loss function.
For a trained model with parameters 8", one can evaluate the loss function for the numbers Z,
f(z y) L(8 + x8 + yn)
(6)
such that y random directions sampled from probability distribution, usually Gaussian distribution, fitternnrmmaized (Li et al., 2018), obtaining 3D plot. In contrast to a typical neural network architecture, in early-exit set-up, both the final and internal classifiers are considered. We consider total training loss and separate losses for each IC.
When evaluating head losses, we use common random directions(6, 17) for each IC. The 6, directions contain both backbone and head parameters.
As shown in Figure there is a significant difference in loss landscapes between the Disjoint regime and the Joint one. The Joint and Mixed regimes are similar in this regard.
C. Full Results
Due to space constraints in the main paper we have presented only the averaged results. In this section, we present the main results with the standard deviation reported for each experiment. Furthermore, we include the results for additional training regimes, which are described in Appendix A.
13

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
10
2.00
150
1.00
2
0.75
0.5
-1.0
0.0
-1.0
0.0
-1.0
0.0
-0.5
y
-0.5
-0.5
-0.5
0.0
-0.5
X
Figure 6. Training loss landscapes: comparison for Joint, Mixed, and Disjoint regimes (left to right), head 1. Landscapes for SDN architecture with Resnet20 as backbone trained on the CIFAR-10 dataset.
Table ResNet-3 CIFAR-100
Regime
25%
50%
75%
100%
Unlimited
Disjoint
52.57 #0.07
67.56
=0.22
73.49
=0.93
73.79
#1.00
73.79
#1.00
Joint
62.84 +0.67
72.80
-0.34
74.32
00.88
74.17
+0110
74.19
+0009
Mixed
62.24 +0.78
73.81 =0.51
75.92 =0.4
75.88 +0.40
75.88 +0.40
Branch-wise
64.19 +0.47
67.03
-0.76
66.88
=0.60
Separate
66.25 =0.54
=0.38
72.87
=0.41
12.82:0.44
72.82 #0.46
Alternating
60.35 +1.15
71.99
+0.45
74.57
00117
74.37+0.00
74.36 +0.09
Mixed-gradual
60.11 #0.68
73.16 =0.70
76.08 0.54
75.94 #0.75
75.94 #0.75
Table MSDNet CIFAR-100
Regime
25%
50%
75%
100%
Unlimited
Disjoint
56.74 =0.41
63.96 00.74
68.59
0.72
70.36
+0.84
70.54
+0.93
Joint
65.93 =0.20
72.02
=0.27
74.73
=0.18
75.93 #0.21
Mixed
65.94 +0.55
72.01
=0.47
00155 75.46
+0.08 76.51
76.71 +0.17
Branch-wise
58.24 +0.45
59.73 =0.41
60.16 =0.47
59.97 10.38
59.73 10.29
Separate
64.90 10.56
60.84
69.02
69.21
#0.23
69.15 +0.23
Alternating
64.69 10.45
71.67 =0.27
74.93
=0.16
76.06 #0.25
76.34
#0.31
Mixed-gradune
64.73 +:066
72.10 =0.47
75.66 0.51
77.21 +0.61
77.49 +0.64
Table 10. VIT-T CIFAR-100
Regime
25%
50%
75%
100%
Unlimited
Disjoint
28.33 +0.47
47.32 =0.81
61.50
=1.82
63.99
1166
63.99
1166
Joint
40.87 #0.84
60.48
=0.94
66.22
=1.07
66.49
+1.0
66.49
+1.08
Mixed
41.65 +0.19
64.07 10.38
70.09 0064
70.25
+0.56
70.25
+0.56
Branch-wise
33.46 -0.76
34.33 =0.49
34.51
+0.45
34.51
+0.45
Separate
42.67 #0.89
51.48 =3.58
54.29 =4.68
54.42 +4.83
54.42 +4.83
Alternating
37.85 40.40
59.11 =0.20
67.81
=0.69
68.39
20.78
68.39
20.78
Mixed-gradual
40.61 +0.08
64.83 +0.46
71.42 0.34
71.60 +0.38
71.60 +0.38
14

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Table 11. ResNet-50 Tinyimagenet
Regime
35%
50%
75%
100%
Unlimited
Disjoint
38.49 #0.46
49.25
11.33
60.50 +1.27
65.71 #0.90
65.80 #0.90
Joint
52.98 +0.77
62.16
+0.61
65.14 +0.57
65.01 #0.75
65.00 #0.75
Mixed
52.89 +0.23
63.28
+0.59
67.20 +0.65
67.24 +0.71
67.22 +0.71
Table 12. VIT-T ImageNet-Ik
Regime
25%
50%
75%
100%
Unlimited
Disjoint
10.22 +0.15
35.15 +0.97
61.23 +1.37
71.61
+0.68
71.61 +0.68
Joint
36.39 +0.12
61.89 +0.71
68.08 +0.84
68.39
+0.87
68.39 #0.87
Mixed
35.91 #0.19
62.96 +0.12
70.79 #0.36
71.20 +0.34
71.20 #0.34
Table 133. BERT-B 20-Newsgroups
Regime
25%
50%
75%
100%
Unlimited
Disjoint
68.53 +0.97
83.75 00.34
85.75
00.6
85.69
+0.28
85.54
+0.34
Joint
84.24 +0.48
84.41 =0.41
84.41 =0.41
84.41
#0.41
84.51 #0.60
Mixed
84.99 +0084
85.25 -0664
85.25 0064
+0.64 85.25
85.46 +0.44
Branch-wise
80.20 10.19
80.05 =0.12
80.05 =0.12
10.05:0.11
79.83 10.06
Separate
79.87 #0.36
79.87 -0336
79.87 0.36
19.87.0.33
79.78 10.30
Altemaatin
83.51 #0.33
83.92 =0.06
83.92 00.66
83.92 #0.06
84.18 20.34
Mixed-gradual
81.35 +1.24
81.35 1.22
81.35 1.24
81.35 +1.24
81.24 +0.93
Table 14. BERT-B STS-B
Regime
25%
50%
75%
100%
Unlimited
Disjoint
2.37 10.01
1.55 #0.11
0.54
#0.00
0.51 10.02
0.51 #0.02
Joint
1.70 =0.23
0.61 #0.02
0.53 +0.01
0.52 #0.02
0.52 #0.01
Mixed
2.43 +0.03
0.59 +0.01
0.53
0.51 +0.00
0.51 +0.00
Branch-wise
2.14 +0.24
0.82 +0.09
40.10
0.83 10.10
Separate
2.69 #0.14
1.41 #0.09
1.41 10.09
1.40 #0.08
1.39 =0.0
Alternating
1.76 +0.33
0.60 #0.00
0.51 #0.01
0.50 #0.01
0.50 #0.01
Mixed-gradus
2.37 +0.03
0.51 +0.00
+0.00
0.46 +0.01
0.46 +0.01
Table 15. VIT-T PBEE CIFAR-100
Regime
25%
50%
75%
100%
Unlimited
Disjoint
19.28 +0.32
44.11 +1.73
57.85 +1.61
63.91 11.68
63.99 +1.68
Joint
28.66 +0.50
56.49 +1.48
64.31
+0.87
66.52 +1.08
66.49 #1.08
Mixed
28.87 +0.44
59.42 +0.50
68.33
+0.60
70.33 +0.68
70.25 +0.56
15

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Table 16. VTTT GPF CIFAR-100
Regime
25%
50%
75%
100%
Unlimited
Disjoint
33.16 +:064
53.19
+:0.66
62.54 +1.59
64.01 11.67
63.99 #1.68
Joint
46.21 +0.30
62.23
+0.81
66.84 #0.68
67.19 =0.78
67.19 =0.80
Mixed
47.21 +1.02
64.51 +0.84
68.92 +0.30
69.10 +0.38
69.10 +0.38
Table ViT-T Entropy CIFAR-I00
Regime
25%
50%
75%
100%
Unlimited
Disjoint
27.14 #0.36
46.09 +0.59
60.30 +1.11
63.99
#1.68
63.99
#1.68
Joint
41.16 +
58.60 +0.89
65.69 +1.23
66.49
+1.0
66.49 +1.08
Mixed
41.54 +0.52
62.20 +0.35
69.63 +0.47
70.25 +0.56
70.25 +0.56
Table 18. ViT-B pretrained on ImageNet- Ik to CIFAR-100
Regime
35%
50%
75%
100%
Unlimited
Disjoint
28.50 +-2.02
55.64 +33.1
87.49 +-0.40
89.95
+-018
89.95 +0.18
Joint
73.18 11.3
85.41 +0.23
87.82 =0.22
87.85 +0.29
87.84 #0.29
Mixed
73.38 +0.35
85.70 +0.63
88.06 +0.31
88.23 +0.29
88.23 +0.29
Table 19. ResNet-50 Tinyimagenet loss and gradient scaling experiment
Regime
25%
50%
75%
100%
Unlimited
Joint
52.98 +0.77
62.16
#0.61
65.14
0.57
65.01 +0.75
65.00 =0.75
Joint GE
53.29 10.90
62.56
10.15
65.87
0.36
65.72 +0.56
65.69
+0.57
Joint Inc.
51.28 +0.19
62.56
#0.12
66.12 =0.39
66.07
#0.42
66.06 =0.41
Joint Dec.
53.58 +0.44
61.87 +0.45
65.06 00.35
+0.29 65.01
65.00 +0.30
Joint SDN
49.23 10.64
01.83:0.3
66.03 00.18
10.25
65.93 +0.24
Mixed
52.89 +0.23
63.28 10.59
67.20 0.65
67.22 +0.71
Mixed GE
51.73 +0.12
63.17
67.22 =0.39
10.10
67.21 #0.12
Mixed Inc.
51.81 +0.24
63.01
=0.49
67.42 0.39
67.49
+0.32
67.47 +0.34
Mixed Dec.
53.58
10.69
63.60
10.25
66.76
0.50
66.78
+0.47
66.75 +0.47
Mixed SDN
49.50 +0.74
62.18 00.76
67.27 0066
+0.55
67.47 00.56
Table 20. VIT-T Imagenette head size experiment
Regime, Head
25%
50%
75%
100%
Unlimited
Disjoint IL
75.61 10.10
78.72 1.24
78.32 1.62
78.29
1..6
78.28
1.66
Disjoint 2L-1024
77.41 +0.39
79.04 +1.26
78.36 +1.63
78.29
11.63
78.28 +1.65
Disjoint 2L-2048
77.10 +0.40
79.02 +1.6
78.39
+1.69
78.30 +1.6
78.28 11.65
Joint IL.
$1.4441.5
82.51 +1.40
82.46 +1.41
82.46 11.41
82.45 #1.40
Joint 2L-1024
80.00 +0.69
81.06 +0.57
80.99 10.69
80.99 +0.69
81.00 +0.68
Joint 2L-2048
79.68 +0.28
80.67 40.08
40.14
80.64 10.15
80.61 #0.13
Mixed II.
82.21 +0.08
83.20 +0.39
40.49
83.19
83.18 +0.45
Mixed 20-1024
80.25 +0.59
81.28 +0.44
81.13 #0.40
81.13 #0.40
81.11 20.37
Mixed 21-2008
79.63 +0.67
80.77 +0.58
80.70 +0.51
80.70 +0.51
80.67 +0.52
16

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
D. Results as Performance-Cost plots
Following Kaya et al. (2019), we presented most of the results in this paper in the form of tables. In this section, we provide the same results as model performance vs. computational costs plots.
0.70
Accuracy
Accuracy
6.63
0.60
0.51
ResNet: Disjoint Mixed Joint
MSDNet
Disjoint
0.0
Joint
Inference FLÖPs
1e0
Inference FLOPs
Figure 7. Performance-com cost trade-off of the multi-exit ResNet-34 on the CIFAR-100 100 dataset
Figure 8. Performance-coe trade-off of the MSDNet architecture on the CIFAR-100 dataset
06
0.6
Accuracy
Accuracy
0.4
Joint
0.2
Joint
Mixed
Disjaint
Disjoint
0.2
0.4
0.6 Inference FLOPs
0.8
1.0
1.2
Inference FLOPs
1eB
lef
Figure Performance-cost trade-off of the multi-exit ViT-T
Figure 10. Performance cost trade-off of the VIT-T architecture on the ImageNet- dataset
the CIFAR-100 dataset
2.50
BERT-6
Joint
2.30
Disjoint
Accuracy
1.75
Loss
1.50
3.25
Disjeint
Mixed
0.75
0.
Joint
0.50
inference FLOPs
1w10
Inference FLOPs
60
0.6
Figure 11. Performance-cox trade-off of the multi-exit BERT-B on the 20Newsgroups dataset
Figure 12, Performance cost trade-off cost ture the STS-B dataset
BERT-B architec-
17

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Accuracy
Accuracy
VIFT
VTT
Joint Mixed Disjoint
Jeint Mixed
02
Disjoint
Inference FLOPs
leB
Inference FLOPs
le8
Figure 13. Performance cost trfde-off of the multi-exit PBEE with
cost cost trade-off of the multi-exit GPF with VIT-T as the backbone on the CIFAR- 100 dataset
Figure 14. Performance
VIT-T as the backbone on the CIFAR- 100 dataset
0.7
6.
Accuracy
0.
Accuracy
0.4
VIFT
VTT-B
Dijoint
Mixed Disjoint
Mixed
Joint
1.00
1:25
2,00
Inference FLOPs
Inference FLOPs
1e10
Figure 15. Performance-com trade-off of the multi-exit ViT-T on the CIFAR-100 dataset, using normalized entropy as the crite-
Figure 16. Porformance-com tradc-off of the multi-exit ViT-B on the CIFAR-100 dataset, pre-trained on ImageNet- fine-tuned CIFAR-100
rion
64
Accuracy
ViLS
Diejaint Maed
Joint
Inference FLOPs
1e9
Figure 17. Performance trade-off of the multi-exit ViT-S on the ImageNet-Il 1k dataset
E. Training Details
Here provide more information training setup, in addition what is described in Section 4.1
E.1. ResNet-34, CIFAR-100
Model set-up. Exit heads are placed at positions (2,4.... 14).
18

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Training set-up. We train each model with batch size of 128. We a learning rate of 5e-4 and weight decay. We set the early stopping patience to 50 epochs. CutMix and Mixup used augmentations.
E.2. MSDNet, CIFAR-100
Model set-up. We CIFAR variant of MSDNet with blocks. Exit heads are placed at positions {0, 1, 1...., 173.
Training set-up We train each model with batch size of 512. We use a learning rate of 1e-3 and no weight decay, We set the early stopping patience to 50 epochs. CutMix and Mixup are used as augmentations.
E.3. VIT-T, CIFAR-100
Model set-up. ViT-T hyperparameters are as follows: a patch size of 4, an embedding size of 256, an MLP dimension of 256, layers and attention heads. Exit heads are placed at positions (0, 1, 1,....6,, .6. For GPF we set head embedding size to 512.
Training set-up We train each model with batch size of 256. We use learning rate of 5 and weight decay. We set the early stopping patience to 30 epochs. Following augmentations are used: random resizing, cropping, rotation, contrast adjustment, random erasing, CutMix and Mixup.
E.4. ResNet-50, Tinyimagenet
Model set-up. Exit heads are placed at block positions (1..14).
Training set-up. We train each model with batch size of 256. We use a learning rate of 1e-3 and weight decay. We set the early stopping patience to 50 epochs. CutMix and Mixup used augmentations.
E.5. VIT-T, ImageNet-1k
Model set-up. ViT-T hyperparameters are as follows: patch size of 16. an embedding size of 192, an MLP dimension of 768, 12 layers, and attention heads. Exit heads placed positions 21,2,...., 10).
Training set-up. We train each model using A-100 GPUs with an effective batch size of 2048. We use a learning rate of 5e-4 and no weight decay. We set the early stopping patience to 25 epochs. Following augmentations are used: random resizing, cropping, rotation, contrast adjustment, random erasing, CutMix (Yun al., 2019) and Mixup (Zhang et al., 2018).
E.6. 20 Newsgroups, Bert-B
Model set-up
We set maximum sequence length to 512. Exit heads are placed at positions 31.3...., 211
Training set-up We train each model with batch size of 32. We use learning rate of 5e-5 and weight decay of 1e-2.
We set the early stopping patience to 3 epochs.
E.7. STS-B, Bert-B
Model set-up
We set maximum sequence length to 128. Exit heads are placed at positions 31, ,,21
Training set-up We train each model with batch size of 16. We use a learning rate of 1e-5 and a weight decay of le -4.
We set the early stopping patience epochs.
E.S. VIT-B, CIFAR-100
Model set-up. ViT-B hyperparameters are as follows: patch size of 16, an embedding size of 768, an MLP dimension of 3072, 12 layers and 12 attention heads. Exit heads are placed at positions {3, 5, ,... 9)..
19

---

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Training set-up We use pretrained weights from torchvision (maintainers & contributors, 2016), trained ImageNet- 1k. To fine-tune, learning rate of 2e-5 and weight decay of 3.-2. We set the early stopping patience to 30 epochs. We use Mixup as augmentation.
20


Reviews:
[{"review_id": "cfCHpNXhKt", "reviewer": "Reviewer_kw5N", "summary": "This submission analyses different training strategies for early-exit models, namely disjoint (frozen backbone), joint (end-to-end) and the proposed mixed (backbone pretraining + joint) approach. Several metrics are proposed, including gradient dominance, mode connectivity, numeric rank and mutual information each capturing different angles of the training dynamics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for assessing our work and recognizing the importance of this previously overlooked aspect. We hope that the answers below adequately answered the reviewer's questions and concerns. If that is the case, we kindly ask for a reconsideration of the score.\n\n\n> number and placement of early exits, which have not be adequately considered in the submission\n> …\n> but not adequate variations in the configuration of the early-exit models\n> …\n> consider conduction an ablation on the number and placement of ICs\n\nIn Appendix B **we provide this kind of an analysis** for our ViT-T model trained on the ImageNette dataset. These results show that our findings generalize to different placement densities. However, to make our case even stronger, **we rerun those experiments on Tinyimagenet/ResNet-50** and **include additional placement schemes** (Dense-Sparse places ICs at blocks: [1, 2, 3, 4, 5, 6, 7, 11], Sparse-Dense places ICs at blocks: [1, 4, 8, 9, 10, 11, 12, 13, 14]). We present the results below:\n\n|Placement|Regime|25%|50%|75%|100%|\n|-|-|-|-|-|-|\n||disjoint|38.92|49.25|60.10|65.76|\n|Every-1|joint|52.20|62.49|65.52|65.59|\n||mixed|52.22|63.03|67.21|67.35|\n||disjoint|37.34|48.03|60.34|65.65|\n|Every-2|joint|51.81|62.60|65.55|65.38|\n||mixed|52.41|63.19|67.14|67.33|\n||disjoint|-|47.91|60.95|65.77|\n|Every-3|joint|-|63.33|67.22|67.21|\n||mixed|-|62.52|66.72|66.71|\n||disjoint|-|41.32|57.78|65.78|\n|Every-4|joint|-|62.54|66.30|66.27|\n||mixed|-|62.07|67.08|67.14|\n||disjoint|-|39.85|56.20|65.72|\n|Every-5|joint|-|61.10|65.61|65.79|\n||mixed|-|61.95|67.33|67.40|\n||disjoint|38.47|50.64|62.04|65.74|\n|Dense-Sparse|joint|53.14|62.23|64.76|64.93|\n||mixed|53.48|63.17|66.24|66.27|\n||disjoint|37.12|47.03|59.79|65.68|\n|Sparse-Dense|joint|50.47|61.19|65.36|65.42|\n||mixed|51.19|62.27|67.26|67.47|\n\nThe results are consistent with the main findings of the paper, that is:\n1. The mixed regime still presents generally better performance over the joint regime.\n2. The disjoint regime is still inadequate for low budgets.\n\n> The proposed \"mixed\" approach lacks novelty, … starting with an ImageNet-pretrained backbone can be considered common practice…\n> …essentially comprises a backbone pre-training followed by joint training, and cannot be considered novel as it is common practice in ML deployment…\n\nWe emphasize that the proposed “mixed” approach is **not** in common use. The common practice of using pre-trained models for transfer learning is independent of the choice of the training regime. Starting with a model pre-trained on dataset A does not preclude us from fine-tuning on dataset B with different training regimes.\n\nIn particular, let’s assume that a ML practitioner starts with a backbone model (e.g. ViT-B) pretrained on dataset A (e.g. ImageNet-1k). His aim is a multi-exit model that performs well on dataset B (e.g. CIFAR-100). If he takes the backbone, attaches classification heads, and trains (finetunes) everything jointly on dataset B, then **that is still joint training** (finetuning) according to our terminology.\n\nTo perform mixed regime training in such a setting, we first finetune (on dataset B) the backbone only, and only then proceed to finetune everything together. In the paper we compare all regimes in the transfer learning setting and present the results in Table 4. In this setting the behaviour of each regime and the findings are the same as in other experiments - both joint and disjoint training display a significant performance gap on some budgets.\n\nIf A == B, then indeed taking a pre-trained model, attaching ICs and training jointly indeed would be equivalent to our “mixed” approach. However, we argue that such cases are extremely uncommon.\n\nAgain, we apologize for the misunderstanding, which resulted from our insufficiently thorough description of the transfer learning experiment. In our current version of the manuscript we modify this section to present the pre-trained setup more clearly.\n\n> …alternative presentation…\n\nWe thank the reviewer for this valuable suggestion regarding the positioning of our contribution. Although it may not have been emphasized enough, our goal in this work is to provide a comprehensive comparative analysis of different early-exit training regimes, and investigate their training dynamics and implications. In the paper, we point out scenarios where the joint or even disjoint training regimes can be more suitable or advantageous. For instance, we observe that the joint regime may be preferable at very low computational budgets (where early classifiers dominate), and the disjoint regime can show good performance when the backbone model is already well-trained or fixed and further training resources are limited. Nevertheless, our empirical results consistently suggest that, in most practical scenarios and computational budgets, the mixed regime tends to outperform the others, hence our emphasis on highlighting its benefits.", "claims_and_evidence": "In my opinion, there are limited evidence for the claims made in the submission, in some cases. This is because the joint training scheme can be severely affected by factors such as the number and placement of early exits, which have not be adequately considered in the submission.", "methods_and_evaluation_criteria": "In my opinion, the proposed methods and evaluation criteria are meaningful for the examined problem.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "In my opinion, the experimental analysis considers numerous backbone models and dataset; but not adequate variations in the configuration of the early-exit models.", "relation_to_broader_scientific_literature": "In my opinion, the analysis conducted in this manuscript offers numerous insights for the training of early-exit models for CV and NLP. The proposed \"mixed\" approach lacks novelty, not because of its simplicity, but mostly because starting with an ImageNet-pretrained backbone can be considered common practice for ML practitioners deploying multi-exit models.", "essential_references_not_discussed": "In my opinion, relevant literature has been adequately cited.", "other_strengths_and_weaknesses": "Strengths:\n- Overall the paper is well written and easy to follow; and studies an interesting problem.\n- The conducted empirical analysis and proposed metrics are insightful and offer numerous findings that can be adopted to guide future research in the field. \n\nComments:\n1. The findings of the this analysis, however, may not generalise across different configurations of early-exit models as the number and placement of ICs affect the training dynamics of end-to-end (joint) approaches, as does the architecture of ICs which has been discussed in the manuscript.\n2. The proposed mixed training approach, essentially comprises a backbone pre-training followed by joint training, and cannot be considered novel as it is common practice in ML deployment. Nonetheless, the methods discussed in the appendix and their comparison to traditional techniques comprise a more interesting discussion.", "other_comments_or_suggestions": "I would suggest an alternative presentation, where \"mixed\" training is not posed as a novel contribution of this work. Instead the comparative analysis of different early-exit model training regimes could potentially be the main contribution of this work.\n\nPost-rebuttal edit: Provisionally increasing my score from WR to WA, having considered the authors' rebuttal and other reviewers' comments.", "questions_for_authors": "Please consider conduction an ablation on the number and placement of ICs, and how the affect the findings of the discussed analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l9XH0JWPdE", "reviewer": "Reviewer_DJjQ", "summary": "The paper presents an enhanced early-exit training approach that combines two phases: initial backbone training followed by full multi-exit network training. This mixed strategy addresses the shortcomings found in both joint and disjoint training methods. While the paper presents its methodology clearly and provides thorough empirical validation, several limitations exist. The authors do not do experiment on SOTA settings, and, the proposed method, though well-explained, lacks technical innovation - essentially combining two existing approaches.\n\n## update after rebuttal\nIn the rebuttal period, the authors have add the FLOPs-Accuracy curve in the rebuttal, which is a valuable evaluation approach in early-exit models. The authors also clarify the novelty and the relationship between method and analysis, which well address my concerns. As a result, I raise my rating from Weak Reject to Weak Accept. I hope the authors can also add the FLOPs-Accuracy curve (budgeted batch classification, proposed in MSDNet[1], and widely used in following works[2~5]) in the final revision, because it is a very clean way to show the performance of early-exiting networks.\n\n[1] Huang, Gao, et al. \"Multi-Scale Dense Networks for Resource Efficient Image Classification.\" International Conference on Learning Representations. 2018.\n\n[2] Li, Hao, et al. \"Improved techniques for training adaptive deep networks.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019. \n\n[3] Yang, Le, et al. \"Resolution adaptive networks for efficient inference.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[4] Han, Yizeng, et al. \"Learning to weight samples for dynamic early-exiting networks.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[5] Han, Yizeng, et al. \"Dynamic perceiver for efficient visual recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the effort spent on reviewing our paper and the valuable insights.  If we further addressed the remaining concerns, we would kindly ask for possible score reconsideration.\n\n> The authors do not do experiment on SOTA settings. \n> The results presented in Table 1~6 are problematic. They provide the x-axis as a ratio, make the reviewer hard to know they achieve this performance in which FLOPs\n\nIn this work, we followed the performance reporting scheme initiated by [1] due to its simplicity and readability (though without the absolute FLOPs values). However, as per the reviewer’s suggestion, we provide the same results as FLOPs vs accuracy plots (similar to the one from Figure 1) in [this link](https://mega.nz/file/nsgSjAiR#73JNpisrf_NrvM0lI2ouyYNgKADWAZWYMlX7QG1EL7s). We also include them in the current version of our manuscript.\n\n[1] Kaya at al. \"Shallow-deep networks: Understanding and mitigating network overthinking.\" \n\nWe are also not sure if the reviewer might have meant it but we want to emphasize that our paper does not aim to achieve state-of-the-art performance on any setup. Its purpose is to analyze the training dynamics and compare the multi-exit model training strategies. We do this by evaluating all the considered regimes on multiple commonly-used architectures, on multiple modalities and datasets, and for multiple early-exiting approaches. Moreover, most of the results we present were computed for networks trained from scratch, for three different PRNG seeds. \nOur limited computational resources prevent us from running large SOTA models. Nevertheless, to demonstrate that our findings scale to larger models, we present the results for the ImageNet-1k experiment for the ViT-S variant of the vision transformer architecture in the link provided (Figure 11):\n| Regime   | 25%   | 50%   | 75%   | 100%  |\n|-|-|-|-|-|\n| Disjoint | 10.23 | 33.91 | 68.02 | 78.38 |\n| Mixed\t| 49.50 | 75.17 | 78.28 | 78.33 |\n| Joint\t| 50.10 | 73.99 | 76.38 | 76.44 |\n\nWe can see that - as before - the disjoint regime still is significantly inferior on lower budgets, while the joint regime exhibits a noticeable performance gap on higher budgets.\n\n\n> the proposed method, though well-explained, lacks technical innovation - essentially combining two existing approaches.\n\nWe thank the reviewer for their feedback. Our mixed training strategy is intentionally technically simple, and we consider that as an advantage rather than a weakness of our work. We believe that clarity and effectiveness should take priority over unnecessary complexity. Rather than adding complexity for its own sake, we focus on a practical, well-reasoned method that consistently outperforms existing approaches. Moreover, technically simple methods are easy to implement, and - as a consequence - are more likely to be widely adopted by the community.\nWe emphasize that the primary goal of our work is to systematically analyze early-exit training strategies. Since existing strategies are also simple, it is important to first analyze their limitations before considering more complex alternatives. Our paper demonstrates the weaknesses of the two commonly used training strategies, and highlights the large impact of this previously overlooked aspect. The proposed “mixed” approach is the most straightforward approach to alleviate the issues of the joint and disjoint regimes. While being *technically* simple, it is not necessarily obvious, as it was not used in any prior work.\n\n> The authors provide some experimental analysis in section 3, but I can hardly find the relationship between them and the proposed method.\n\nWe appreciate the reviewer’s feedback and would like to clarify how the analyses in Section 3 are directly connected to the proposed mixed training strategy. Specifically, the experimental metrics - gradient dominance, mode connectivity, numerical rank, and mutual information - were chosen to reveal distinct aspects of the training dynamics under different regimes. For example, gradient dominance analysis shows that the mixed training regime shifts the optimization focus toward later exits. This shift directly correlates with improved performance under higher computational budgets, as later exits become more robust. Please note the relation to the gradient equilibrium experiments in Sec. 4.4. Gradient dominance results may explain the need to rescale the rear exists when trained in joint regime while mixed obviates the need for application of gradient equilibrium. Mutual Information and Numerical Rank show further aspects why the performance of the presented regimes may differ (e.g. for early vs rear exits). Finally mode connectivity reveals that mixed and disjoint regimes produce very similar solutions, despite having significantly different performance on low computational budgets.", "claims_and_evidence": "Yes, they provide much analysis and experiments for their claims.", "methods_and_evaluation_criteria": "I am afraid not. The results presented in Table 1~6 are problematic. They provide the x-axis as a ratio, make the reviewer hard to know they achieve this performance in which FLOPs. I suggent the authors present results following the follow literatures:\n\n[1] Han, Yizeng, et al. \"Dynamic perceiver for efficient visual recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[2] Wang, Yulin, et al. \"Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.\" Advances in neural information processing systems 34 (2021): 11960-11973.", "theoretical_claims": "The authors provide some experimental analysis in section 3, but I can hardly  find the relationship between them and the proposed method.", "experimental_designs_or_analyses": "I have checked. I think the results presented in Table 1~6 are problematic. They provide the x-axis as a ratio, make the reviewer hard to know they achieve this performance in which FLOPs. I suggent the authors present results following the follow literatures:\n\n[1] Han, Yizeng, et al. \"Dynamic perceiver for efficient visual recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[2] Wang, Yulin, et al. \"Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.\" Advances in neural information processing systems 34 (2021): 11960-11973.", "relation_to_broader_scientific_literature": "They are related to the early-exit and dynamic neural networks material.", "essential_references_not_discussed": "The related work section is enough.", "other_strengths_and_weaknesses": "Early exiting is a very valuable research topic. But the experiments should provide the FLOPs when you report each Accuracy. 25%, 50%, 75% is very unambiguous。", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZalYvL6tb", "reviewer": "Reviewer_cDMi", "summary": "The authors study different training regimes for early-exit networks (EENNs). To this end, they propose a framework consisting of 4 different metrics (gradient dominance, mode connectivity, numerical rank, mutual information) for studying the tranining dynamics of EENNs. They use the framework to explore the differences between commonly used training strategies (joint, disjoint) as well as their newly proposed mixed strategy. In the experiments, they show that their mixed strategy performs favourably (in most cases) compared to joint or disjoint baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate and agree with the reviewer’s assessment that our work addresses a meaningful gap in the early-exiting literature.\n\n> numerical rank informativeness\n\nFirstly, the key insight of the numerical rank metric is that placing multiple early exits increases the rank and also the expressive representation of the network. The difference in the ranks between the plain model (disjoint regime) and the mixed regime in Figure 4a is significant as it is substantially larger than standard deviation. In Fig. 4a we only show the mixed result for clarity but Fig. 4b shows both mixed and joint regimes obtain similar ranks (given that both regimes foster enriched representations), and both are respectively higher than the backbone.\n\n\n\n\n\nSecondly, the backbone remains unchanged for the disjoint regime and so the numerical rank experiment seen in Fig. 4a is also valuable because it may explain the inferior performance of early ICs in the disjoint regime. Joint training results with layers that enable more expressive intermediate representations (higher numerical rank), thus allowing for coexistence of features relevant for the nearest IC, and those relevant for the deepest ICs. On the other hand, a consistently lower rank is obtained for the disjoint regime. As mentioned above when even an already trained backbone is allowed to be affected by the ICs’ gradients, the numerical rank of its representation rises (see Figure 4a), and so does its performance on lower budgets (see accuracy results for disjoint vs mixed in any setting). \n\nIn the current version of the manuscript we rewrite the explanation of the numerical rank results to make these points more clear to the reader.\n\n> While I understand it is not the focus on this work, it would still be valuable to say something about whether the findings on training dynamics presented in this paper translate to early-exit LLMs [1, 2, 3]\n\nWe appreciate this insightful suggestion. We agree that exploring early exit regimes in LLMs is a promising direction. However, we intentionally refrained from discussing generative LLMs as translating our observations to early-exit LLMs is challenging due to several fundamental differences (we also note that recent early-exiting studies [1, 2] refrain from studies on LLMs.). Methods like proposed CALM and FREE incorporate intricate confidence mechanisms, state-copying techniques, and synchronization strategies tailored specifically for token-level decisions, which are not directly analogous to the layer-wise early-exiting approach we analyzed. Moreover, EE-LLM emphasizes 3D parallelism and large-scale model training, aspects that differ substantially from our focus on training regimes and layer-wise representation dynamics.\n\nHowever, as an early effort we add an experiment on the NLP classification problem using BERT-B trained on Newsgroups dataset. [results](https://mega.nz/file/29BwxLCa#cntotXc3eDo7mnqA7KsCHBzq_yiqy905_IJOd0gFBtE) For the backbone network (BERT-B), lower MI in earlier layers and higher in the rear layers used by the disjoint regime may explain its positive performance only at the deeper layers. A slightly increased MI for mixed regime is further indicative of better performance.\n\n[1] Jazbec, Metod, et al. \"Towards anytime classification in early-exit architectures by enforcing conditional monotonicity.\" [2] Meronen, Lassi, et al. \"Fixing overconfidence in dynamic neural networks.\" [3] Yoshitomo, Levorato, Restuccia. \"Split computing and early exiting for deep learning applications: Survey”.\n\n> The fact that joint strategy leads to suboptimal performance for larger computational budgets reminds me a bit of the \"interference\" …\n\nWe agree that interference between internal classifiers is a central issue - indeed, it’s what inspired our gradient dominance metric. Our experiments reveal that even MSDNet’s architectural adjustments, including its dense connectivity, does not fully resolve this issue. The same is true for gradient equilibrium from [1], as we show in Section 4.4. In fact, our mixed training regime outperforms these approaches, demonstrating more effective handling of interference and yielding superior performance, particularly under larger computational budgets.\n\n[1] Li, Hao, et al. \"Improved techniques for training adaptive deep networks.\"", "claims_and_evidence": "/", "methods_and_evaluation_criteria": "/", "theoretical_claims": "/", "experimental_designs_or_analyses": "/", "relation_to_broader_scientific_literature": "/", "essential_references_not_discussed": "/", "other_strengths_and_weaknesses": "Strengths:\n- I agree with the authors that studying and understanding of the training approaches for EENNs is under-explored and that most papers just do what some of the seminal papers from the past did (MSDNet, SDN etc.). Hence, I believe this works fills an important gap in the early-exiting literature\n- I like the proposed framework for studying the training dynamics. I believe that going beyond just experimentally comparing different training regimes (e.g., via accuracy-FLOPs curves on ImageNet) is valuable and provides more insights into the differences between considered regimes\n- The experiments presented support claims made in the paper and show that the mixed strategy might be the optimal one going forward in the early-exit community\n\nWeaknesses:\n- While I appreciate the framework presented, I wonder if all 4 metrics are indeed necessary. For example, I feel that the numerical rank one is not that informative as all 4 curves displayed in Figure 4 show quite a similar trend to me.\n- While I understand it is not the focus on this work, it would still be valuable to say something about whether the findings on training dynamics presented in this paper translate to early-exit LLMs [1, 2, 3]\n\n\n[1] Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y. and Metzler, D., 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35, pp.17456-17472.\n\n[2] Bae, S., Ko, J., Song, H. and Yun, S.Y., 2023. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. arXiv preprint arXiv:2310.05424.\n\n[3] Chen, Y., Pan, X., Li, Y., Ding, B. and Zhou, J., 2023. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint arXiv:2312.04916.", "other_comments_or_suggestions": "/", "questions_for_authors": "Questions:\n- The fact that joint strategy leads to suboptimal performance for larger computational budgets reminds me a bit of the \"interference\" of the  earlier classifiers talked about in MSDNet paper (see right plot in Figure 3 there). To tackle this, in MSDNet they introduce architectural changes to their CNN via \"dense connectivity\". So in some way your mixed strategy aims to do the same, but is more general since it doesn't require architectural modifications. And this is kinda confirmed by your experimental results, since, if I read Table 1 correctly, the mixed strategy improves over joint strategy the least for maximal budget on MSDNet", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lo83GFbVjw", "reviewer": "Reviewer_gPtc", "summary": "Multi-exit neural network model is a model that can exit at different layers. It remains a challenge to find an optimal way to increase accuracy of exiting at an earlier layer while not dropping accuracy of the last layer.\nThis paper focuses on one angle: what is the best way to train a multi-exit model. The paper found that the best method to get the best of both worlds is the Mixed approach: train the model till the end without early exit, then add early exit classification heads, referred to as internal classifier (IC), and train the full model with ICs.\nThe paper starts with analyzing the gradient contribution of each early exit, as well as the rank of activations and mode connectivity and mutual information to reach insights to support the decision to use Mixed training.\nThe paper compared the Mixed approach with the Disjoint approach (i.e., train model first, then add ICs, then train ICs while freezing model) and the Joint approach (i.e., train from scratch both model with ICs), and found Mixed being the best trade off for accuracies of earlier layers and last layer. It also compared with different variants of loss and gradient scales and found that Mixed could be better or an alternative to such methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful evaluation of our work and their recognition of its significance. We apologize for brief answers necessitated by the response length limit. Kindly if we resolved the concerns raised, we would be grateful if the reviewer would consider raising their score accordingly.\n\n> train time passes\n\nWe emphasize that the number of phases of training does not have to affect training time. As noted in the manuscript, in all of our experiments we used early-stopping. We report the resulting number of training epochs for the Tinyimagenet/ResNet50 experiment:\n\nDisjoint: 523 +/- 156\nJoint: 1610 +/- 395\nMixed: 1166 +/- 136\n\nIn practice the opposite is true, joint training is slower to converge.\n\n> benefits of the findings are quite incremental\n\nIn most cases the difference over joint regime is definitely significant, e.g. 2 percentage points for the ImageNet experiment. For low budgets the difference over disjoint is immense - and we emphasize that the disjoint regime is also in popular use.\n\n> LLMs\n\nPlease see the answer to reviewer cDMi.\n\n> mutual information should decrease monotonically across layers\n\nThe monotonic decrease shown in Shwartz-Ziv & Tishby (2017) and inferred through the Data Processing Inequality assumes exact calculations. In our work, we estimate mutual information using a Monte Carlo method (Kawaguchi et al., 2023) in high-dimensional spaces. This approximation can introduce estimation noise, which may result in non-monotonic fluctuations in the absolute MI values.\n\nHowever, to further verify the MI impact, we perform additional experiment for another vision (CIFAR-100) and NLP (BERT-B/Newsgroups) datasets. In this case, the MI has a more decreasing tendency. However what we want to emphasize is the relative relationships between regimes present in all the figures that hint that higher Mutual Information may be an indicator for better performance of the mixed regime. \n[results](https://mega.nz/file/29BwxLCa#cntotXc3eDo7mnqA7KsCHBzq_yiqy905_IJOd0gFBtE)\n\n> Different Early Exit Adapters:...\n\nAs suggested, we run additional experiments where each head has an additional transformer block:\n\n| Regime|50%|75%| 100%|\n|-|-|-|-|\n|disjoint|51.49|55.39|56.99|\n|joint|57.42|59.07|59.09|\n|mixed|58.69|60.54|60.68|\n\nDue to the size of this head architecture, the model is not able to meet the 25% of the original model’s budget, so we omit the first column. We emphasize that the mixed regime has better performance as before.\n\n> Figure 3:...\n\nWhen linearly interpolating the weights between the (models trained with) disjoint and mixed regime, we do not encounter a region of high loss. This means they lie in the same optimization basin. In contrast, when interpolating between the joint regime model and any other model, we do encounter a region of high loss (yellow color).\n\n> Table 3…?\n> Table 6: Why the difference in accuracy…is 1.02%…while in Table 3 the difference is almost 30%?\n\nEach dataset and model combination has a different budget-difficulty characteristic (the shape of FLOPs vs accuracy curves, see Figure 1). Tables 6 and 7 were computed for ViT-T architecture and the ImageNette dataset (a 10-class subset of ImageNet, with the same input image size as ImageNet), a combination which turns out to be a relatively easy problem for a model of this size. Table 3 was computed for CIFAR-100/ResNet-34, which turns out to be a relatively harder problem for this architecture.\n\nFor comparison, we repeat the experiment from Table 6 for Tinyimagenet below:\n\n|Regime|Head|25%|50%|75%|100%|\n|-|-|-|-|-|-|\n||1L|26.35|41.69|53.06|56.72|\n|disjoint|2L-1024|33.85|47.21|54.97|56.72|\n||2L-2048|33.96|45.98|54.74|56.72|\n||1L|42.47|53.24|56.02|56.03|\n|joint|2L-1024|45.59|55.11|57.73|57.66|\n||2L-2048|43.60|53.89|56.94|57.00|\n||1L|44.03|57.91|60.42|60.28|\n|mixed|2L-1024|44.92|57.11|59.32|59.22|\n||2L-2048|44.61|56.94|60.18|60.15|\n\n> …\"Mixed-gradual training\" and \"Alternating training\"\n\n\"Mixed-gradual\" trains the model with n ICs in n **phases**, each phase including a larger number of ICs, starting from the deepest ones. \"Alternating\" switches **every training step** between training with the last IC, and with all the ICs.\n\n> Line 666: …x and y?\n\nx and y represent scalar coefficients used to perturb the model parameters θ∗ along randomly chosen directions (δ,η) to visualize the loss landscape around the trained model (Li et al., 2018).\n\n> What is…?\n\nPBEE - Patience-based Early Exit - EE method where we exit for each sample after n consecutive ICs return the same class  (Zhou et al., 2020),\nGPF - Global Past-Future - an EE method that incorporates hidden states from the preceding and deeper ICs (Liao et al., 2021)\nEntropy - means we base our exit decision on entropy of the prediction probabilities (Teerapittayanon et al., 2016) rather than maximum softmax probability\n\nWe sincerely appreciate the reviewer's detailed feedback. We incorporate all suggestions in the current version of our manuscript.", "claims_and_evidence": "- The main claim that Mixed training is better for earlier and last layer accuracies than Joint training has been backed by results on training various architectures, modalities and datasets", "methods_and_evaluation_criteria": "- Authors trained on different architectures (transformers, ResNets, ...), modalities (vision and text) and datasets (CIFAR, ImageNet, STSB, etc.)", "theoretical_claims": "The authors didn't really make theoretical claims but I have some comments about theoretical explanations of some analysis:\n\n- Line 303: I suggest to re-phrase \"representation of easy samples is not complex\" to \"representation of a subset of the training samples is not complex\", and re-word accordingly subsequent references to easy samples..unless we provide definition of \"easy\" samples (e.g., those that have low loss) and show that samples that meet such definition do require processing with fewer layers as the paragraph claims \n   - Line 315: Similarly, I suggest to re-phrase \"easy datasets where more samples exit at earlier layers.\" with \"subsets of datasets where more samples exit at earlier layers.\" And similar for Line 319\n- Lines 307 to 308: \"To describe it in terms of mutual information, the network does not need to reduce the complexity of X to fit the internal representation Z,\" I am a bit confused. If X is the input, how can the network reduce or change its complexity? Should it only be able to change the internal representation Z, not the input?", "experimental_designs_or_analyses": "Experiments seemed to be fine, but I have a comment on one of the analyses:\n\nFigure 5: According to Figure 2 of this paper ( https://arxiv.org/abs/1909.01380 ), the mutual information between a model's input and intermediate activation should decrease monotonically across layers, but this is not the case in Figure 5 of this paper. Do authors have a reason why?\nMoreover, according to Data Processing Inequality concept ( https://en.wikipedia.org/wiki/Data_processing_inequality ) after the processing of each layer, information about input X in a layer's output should either reduce or stay the same, and it can't increase. Hence, my understanding is that mutual information entropy should monotonically decrease across layers.", "relation_to_broader_scientific_literature": "While most papers in early exit focus on a specific technique (e.g., loss scaling) to increase accuracies of earlier and last layers, they take it for granted whether they train from scratch or finetune, and overlook the implication of this decision.\nThis paper focused on that overlooked aspect, on whether it is better to train from scratch, or continually train, or perform mixed training.", "essential_references_not_discussed": "Not really. I would say that since LLMs (and VLMs) are becoming popular, it would have been useful to discuss some of the results from papers that explored early exit loss for LLMs and (as I suggest in another part of this review) to test the findings on a small LLM:\n\n- EMNLP 2023, \"Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding\", Sangmin Bae, Jongwoo Ko, Hwanjun Song, Se-Young Yun\n- ACL 2024, \"LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding\", Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, Carole-Jean Wu", "other_strengths_and_weaknesses": "Strengths:\n- \"To the best of [the authors'] knowledge, this work is the first to directly compare models trained under different regimes and to provide\na detailed analysis of the training dynamics of multi-exit models.\" If that's true, then publishing this paper will be useful for the community to have a holistic understanding of early exit.\n- Tested on different architectures (transformers, ResNets, ...), modalities (vision and text) and datasets (CIFAR, ImageNet, STSB, etc.)\n- Useful insights such as \"the earlier layers are characterized by higher frequency features while later layers learn low frequency elements. This regularity is disrupted in the\\ case of early exit architectures as the backbone network is given additional classifiers that are placed in earlier parts of the network.\" and \"As placements become less frequent, the difference between joint and mixed regimes becomes less pronounced.\"\n\nWeaknesses:\n- While the paper claims that Mixed training is an alternative to Joint training with loss or gradient scaling, it could be argued that the latter could be better as it requires less training time compared to Mixed training, as we have to train only once.\n- The claim or beneifts of the findings are quite incremental", "other_comments_or_suggestions": "- For experiments, I recommend adding a more commonly used architecture like GPT2 for LLMs\n- Different Early Exit Adapters: What about using an early-exit adapter sub-network like SCAN [1]? In the case of ViT, how about adding 1 or 2 transformer layers before the classification head in the early exit adapter sub-network?\n\nFormatting / Typos:\n- Authors have forgotten to change the mini-title of pages 2 and onwards. It is still at \"Submission and Formatting Instructions for ICML 2025\".\n- Figure 2: Please use same axes limits and steps for Figures 2a and 2b to make it easier to compare.\n- in several parts of the paper, Latex quotes need to be fixed (e.g., line 245)\n- in more than one place in the paper (e.g., line 276), the word \"Testset\" is used while I think it should be \"Test set\"\n\n[1] NeurIPS 2019, \"SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models\", Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, Kaisheng Ma", "questions_for_authors": "- Figure 3: I didn't understand how can we infer from the plots that \"Disjoint and mixed regimes produce similar models, while the model trained in joint regime lies in a different basin\"\n- Table 3: which model are the results for?\n- Table 6: Why the difference in accuracy between 25% and 100% compute is 1.02% in accuracy  for Mixed 1L here, while in Table 3 the difference is almost 30%?\n    - Similarly in Table 7: Again, the differences in accuracy between 25% and 100% compute is around 1%$ or 2% for mixed and joint here.\n- Line 622: What is the difference between \"Mixed-gradual training\" and \"Alternating training\"\n- Line 666: If theta is the model's weights, what are the numbers x and y?\n- Table 15: What is PBEE?\n- Table 16: What is GPF?\n- Table 17: What is ViT-Entropy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper investigates different training strategies for multi-exit neural networks, identifying limitations in commonly used joint and disjoint approaches. The authors propose a set of analytical metrics (gradient dominance, mode connectivity, numerical rank, mutual information) to understand training dynamics and introduce a "mixed" training strategy (pre-training the backbone, then jointly training the full network) which empirically outperforms the standard methods across various settings.

Reviewers generally agreed that the paper addresses an important and under-explored area in early-exit model training (kw5N, CDMi, gPtc) and that the comparative analysis, along with the proposed diagnostic metrics, provides valuable insights (kw5N, CDMi, DJjQ). The empirical evaluation was found to be comprehensive, covering multiple architectures, datasets, and exit configurations, particularly after the authors provided additional results during rebuttal (kw5N, gPtc, DJjQ). While the novelty of the proposed "mixed" strategy itself was questioned by some reviewers (kw5N, DJjQ), who noted similarities to standard pre-training practices, the authors mostly clarified the distinction in the context of mixed pretraining vs. fine-tuning on the target dataset. Ultimately, reviewers acknowledged the main contribution lies in the thorough comparative study and analysis, which fills a gap in the literature. Initial concerns regarding evaluation (e.g., missing FLOPs-Accuracy curves raised by DJjQ) and experimental variations (kw5N) were mostly addressed in the rebuttal. Another concern was about the lack of language generation evaluations (cDMi) The authors claimed that this is beyond the scope of this paper, however the claims in the title/ abstract etc. are general and don't clarify the focus on classification models. Given the wide use of LLMs these days it would have been stronger to include such studies, or at least discussing how this work can translate to that domain.
Decision: Accept (poster)