Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
Authors: Maciej Wolczyk
Bartosz Wójcik
Klaudia Bałazy
Igor T. Podolak
Jacek Tabor
Marek Śmieja
Tomasz Trzcinski
Venue: NeurIPS 2021 Poster (2021)
Keywords: early exit
model acceleration
efficiency

Abstract: The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of a neural network. ICs can quickly return predictions for easy examples and, as a result, reduce the average inference time of the whole model. However, if a particular IC does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), a novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs in an ensemble-like manner. We conduct extensive experiments across various datasets and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other recently proposed early exit methods.

Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
Maciej Wolezyk*
Bartosz Wójeik Jagiellonian University
Jagiellonian University
Klaudia Bałazy Jagiellonian University
Igor Podolak
Jacek Tabor Jagiellonian University
Jagiellonian University
Marek Smieja Jagiellonian University
Tomasz Trzciáski
Jagiellonian University,
Warsaw University of Technology, Tooploox
Abstract
The problem of reducing processing time of large deep learning models is a fun- damental challenge in many real-world applications. Early exit methods strive towards this goal by attaching additional Internal Classifiers (ICs) to intermediate layers of neural network. ICs can quickly return predictions for easy examples and, as result, reduce the average inference time of the whole model. However, if a particular 1C does not decide to return an answer early, its predictions are discarded, with its computations effectively being wasted. To solve this issue, we introduce Zero Time Waste (ZTW), novel approach in which each IC reuses predictions returned by its predecessors by (1) adding direct connections between ICs and (2) combining previous outputs an ensemble- extensive experiments across various datasets and architectures to demonstrate that ZTW achieves a significantly better accuracy vs. inference time trade-off than other recently proposed early exit methods.
manner. We conduct
Introduction
Deep learning models achieve tremendous successes across multitude of tasks, yet their training and inference often yield high computational costs and long processing times [I1 22 For some applications, however, efficiency remains critical challenge, e.g. to deploy reinforcement learning (RL) system production the policy inference must be done in real-time (7], while the robot performances suffer from the delay between measuring system state and acting upon it [34] Similarly, long inference latency in autonomous cars could impact its ability to control the speed 13 and lead accidents
110 17
Typical approaches to reducing the processing complexity of neural networks in latency-critical applications include compressing the model (24 26 46) approximating its responses [21]. For instance, Livne & Cohen 266 propose to compress a RL model by policy pruning, while Kouris al. [21 approximate the responses of LSTM-based modules in self-driving to accelerate their inference time. While those methods improve processing efficiency, they still require samples to pass
"equal contribution
I "Corresponding author: maciej wwwczykkddottorrl.... uj edu ppl
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

SDN (baseline)
Dog (att
Flog (1974)
Dog (URTS)
Dog (Ser5))
Concal Joc
Octouition
(1886)
F2
IC
IC
IC. Comnkksian
IC Carmdittion
Bloxk
Block
Glock
Blox
Blook
Pro (24%))
Pooting
Post ing
Pooling
Cascading
pindation Fi
IC
znw
Enoembling
Gsomebic Ensembling
Dog (89%%)
Flng (95%))
(1.5%)
(87%)
(89%))
(a) Comparison of the proposed ZTW (hattom) with a conven- (b) Detailed scheme of the proposed ZTW tional early-exit model, SDN (top).
model architecture.
Figure I: (a) In both approaches, internal classifiers (ICs) attached to the intermediate hidden layers of the base network allow to return predictions quickly for examples that are easy to process. While SDN discards predictions of uncertain ICs (e.g. below a threshold of 75%). ZTW reuses computations from all previous ICs, which prevents information loss and waste of computational resources. (b) Backbone network So lends its hidden layer activations to ICs, which share inferred information using cascade connections (red horizontal arrows in the middle row) and give predictions Pm- The inferred predictions are combined using ensembling (bottom row) giving 9m.
through the entire model. In contrast, biological neural networks leverage simple heuristics to speed up decision making, egg. by shortening the processing path even in case of complex tasks T [8].
This observation led way to the inception of the so-called early exit methods, such Shallow- -Depp Networks (SDN) [19] and Patience-basee Early Exit (PBEE) [47]. that attach simple classification heads, called internal classifiers (ICs), to selected hidden layers of neural models to shorten the processing time. the prediction confidence of a given IC is sufficiently high, the response is returned, otherwise, the example is passed to the subsequent classifier. Although these models achieve promising results, they discard the response returned by early ICs in the evaluation of the next IC, disregarding potentially valuable information, e.g. decision confidence, and wasting computational effort already incurred.
Motivated by the above observation, postulate to look the problem of neural model processing efficiency from the information recycling perspective and introduce a new family of zero waste models. More specifically, models can contribute to the decision process of the entire model. To that end, we propose Zero Time Waste (ZTW), method for intelligent aggregation of the information from previous ICs. high-level view of our model is given in Figure [ Our approach relies on combining ideas from networks with skip connections [41), gradient boosting BL and ensemble learning 8 23 Skip connections between subsequent ICs (which we call cascade connections) allow us to explicitly pass the information contained within low-level features to the deeper classifier, which forms cascading structure of ICs. In consequence, each IC improves on the prediction of previous IC', as in gradient boosting. instead of generating them from scratch. To give the opportunity for every IC explicitly reuse predictions of all previous 1Cs, we additionally build an ensemble of shallow 1Cs.
investigate how information available at different layers of neural
We evaluate approach standard classification benchmarks, such as CIFAR-100 and ImageNet, as well as on the more latency-critica applications, such as reinforcement-Ieene the first to show that
models for
interacting with sequential environments. To the best of our knowledge, early exit methods can be used for cutting computational waste reinforcement learning setting.
Results show that ZTW is able to save much more computation while preserving accuracy than current state-of-the-art early exit methods. In order to better understand where the improvements come from, we introduce Hindsight Improvability, metrie for measuring how efficiently the model

---

reuses information from the past. We provide ablation studies and additional analysis of the proposed method in the Appendix.
To summarize, the contributions of our work are the following:
We introduce family of zero waste models that quantify neural network efficiency with the
Hindsight Improvability metrics.
We propose an instance of zero waste models dubbed Zero Time Waste (ZTW) method which uses cascade connections and ensembling to reuse the responses of previous ICs for the final decision.
We show how the state-of of-the--rtttttt- performance of ZTW in the supervised learning scenario generalizes to reinforcement learning.
2 Related Work
The drive towards reducing computational waste in deep learning literature has so far focused on reducing the inference time. Numerous approaches for accelerating deep learning models focus on building more efficient architectures [15], reducing the number of parameters [122 or distilling knowl- edge to smaller networks [14]. Thus, they decrease inference time by reducing the overall complexity of the model instead of using the conditional computation framework of adapting computational effort to each example. As such we find them orthogonal to the main ideas of our work, e.g. we show that applying our method to architectures designed for efficiency, such as MobileNet [15), leads to even further acceleration. Hence, we focus here methods that adaptively set the inference time for each example.
Conditional Computation Conditional computation was first proposed for deep neural networks in Bengio et al. (2] and Davis & Arel [5], and since then many sophisticated methods have been proposed in this field, including dynamic routing (27]. cascading with multiple networks 44) and skipping intermediate layers [4]] or channels [42 In this work, we focus on the family of early exit approaches, as they usually do not require special assumptions about the underlying architecture of the network and the training paradigm, and because of that can be easily applied to many commonly used architectures. In BranchyNet (38] a loss function consisting of a weighted sum of individual head losses is utilized in training, and entropy of the head prediction is used for the early exit criterion. Berestizshevsky Guy propose to use confidence (maximum of the softmax output) instead. A broader overview of early exit methods is available in Scardapane et al. (32]
Several works proposed specialized architectures for conditional computation which allow for multi- scale feature processing [16] 44 43], and developed techniques to train them more efficiently by passing information through the network [29] 255. However, in this paper, we consider the case of increasing inference speed of a pre-trained network based on an architecture which was not built with conditional computation or even efficiency in mind. We argue that this is a practical use case, this approach can be used to wider array of models. As such, we do not compare with these methods directly.
Shallow-Deey Networks (SDN) 19 is a conceptually simple yet effective method, where the comparison of confidence with fixed threshold is used as the exit criterion. The authors attach internal classifiers to layers selected based on the number of compute operations needed to reach them. The answer of each head is independent of the answers of the previous heads, although in separate experiment the authors analyze the measure of disagreement between the predictions of final and intermediate heads.
Zhou et al. 47 propose Patience-based Early Exit (PBEE) method, which terminates inference after t consecutive unchanged answers, and show that it outperforms SDN on range of NLP tasks. The idea of checking for agreement in preceding IC' is connected to our approach of reusing information from the past. However, we find that applying PBEE in our setting does not always work better than SDN. Additionally, in the experiments from the original work, PBEE was trained simultaneously along with the base network, thus making it impossible preserve the original pre-trained model.
Ensembles Ensembling is typically used to improve the accuracy of machine learning models 6 Lakshminarayanan al. 23 showed that also greatly improves calibration of deep neural networks. There were several attempts to create ensemble from different layers of network. Scardapane et
3

---

al. [3 adaptively exploit outputs of all internal classifiers, albeit not in a conditional computation context. Phuong & Lampert [29] used averaged answers of heads up to the current head for anytime- prediction, where the computational budget is unknown. Besides the method being much more basic, their setup is notably different from ours, as it assumes the same computational budget for all samples no matter how difficult the example is. Finally, none of the ensemble methods mentioned above were designed to work with pre-trained models.
3 Zero Time Waste
Our goal is reduce computational costs of neural networks by minimizing redundant operations and information loss. To achieve it, we use the conditional computation setting, in which we dynamically select the route of input example in a neural network. By controlling the computational route, we can decide how the information is stored and utilized within the model for each particular example. Intuitively, difficult examples require more resources to process, but using the same amount of compute for easy examples is wasteful. Below we describe our Zero Time Waste method in detail.
In order to adapt already trained models to conditional computation setting, we attach and train early exit classifier heads on top of several selected layers, without changing the parameters of the base network. During inference, the whole model exits through one of them when the response is likely enough, thus saving computational resources.
Formally, we consider multi-class classification problem, where E IR denotes an input example and y E (I..... K) is its target class. Let fe : RD' + R be a pre-trained neural network with logit output designed for solving the above classífication task. The weights 0 will not be modified.
Model overview Following typical early exit frameworks, we add M shallow Internal Classifiers, ICI,..., TCM, on intermediate layers of so- Namely, let go... for m E (I,..., M), be the m-th TC network returning K logits, which is attached to hidden layer fo., of the base network so. The index m is independent of fe layer numbering. In general, M is lower than the overall number of se hidden layers since we do not add ICs after every layer (see more details in Appendix A.I]
Although using ICs return an answer early can reduce overall computation time [9], in a standard setting each IC makes its decision independently. ignoring the responses returned by previous ICs. As we show in Section 4.2 early layers often give correct answers for examples that are misclassified by later classifiers, and hence discarding their information leads to waste and performance drops. To address this issue, we need mechanisms that collect the information from the first (m - 1) ICs to inform the decision of TC... For this purpose, we introduce two complementary techniques: cascade connections and ensembling, and show how they help reduce information waste and, in turn, accelerate the model.
Cascade connections directly transfer the already inferred information between consecutive ICs instead of re-computing it again. Thus, they improve the performance of initial ICs that lack enough predictive power to classify correctly based on low-level features. Ensembling of individual ICs improves performance as the number of members increases, thus showing greatest improvements in the deeper part of the network. This is visualized in Figure | where cascade connections are used first to pass already inferred information to later ICs, while ensembling is utilized to conclude the IC prediction. The details on these two techniques are presented in the following paragraphs.
Cascade connections Inspired by the gradient boosting algorithm and literature on cascading classifiers (39, we allow each 1C to improve on the predictions of previous 1Css instead of inferring them from scratch. The idea of cascade connections implemented by adding skip connections that combine the output of the base model hidden layer fo.. with the logits of IC and pass it to IC.. The prediction is realized by the softmax function applied to I (the m-th IC network):
Pm = softmax(ge... (fo.. (om (x), 9om.. 1 fom-,()) (x))), for m > 1,
where g f(x) - (x)) denotes the composition of functions. Formally, Pm = Pm Pm, Φm)). where om are trainable parameters of IC.,, but we drop these parameters in notation for brevity. IC₂ uses only the information coming from the layer fo, which does not need to be the first hidden layer of so-
Figure I shows the skip connections as red horizontal arrows.

---

Each ICm is trained in parallel (with respect to @m) to optimize the prediction of all output classes using an appropriate loss function ((pm). e.g. cross-entropy for classification. However, during the backward step it is crucial to stop the gradient of loss function from passing to the previous classifier. Allowing the gradients of loss ((Pm) to affect Φ for j E 1,.,, m leads to a significant performance degradation of earlier layers due to increased focus on the features important for ICm- as we show in Appendix C.3
Ensembling Ensembling in machine learning models reliably increases the performance of model while improving robustness and uncertainty estimation [8] 233. The main drawback of this approach is its wastefulness, as it requires to train multiple models and use them to process the same examples. However, in our setup we can adopt this idea to combine predictions which were already pre-computed in previous ICs, with near-zero additional computational cost.
To obtain a reliable zero-waste system, we build ensembles that combine outputs from groups of TCs provide the final answer of the m-th classifier. Since the classifiers we using vary significantly in predictive strength (later ICs achieve better performance than early ICs) and their predictions are correlated, the standard approach to deep model ensembling does not work in case. Thus, we introduce weighted geometric mean with class balancing, which allows us to reliably find a combination of pre-computed responses that maximizes the expected result.
Let P1.P2,..... .P......PP...................... Pm be the outputs of m consecutive IC predictions (after cascade connections stage) for given x (Figure T). We define the probability of the i-th class in the m-th ensemble to be:
Pm (1) Z
by II (v)(()))
JSm
where 6i,, > O and wi, > 0, for j == 1.... m, are trainable parameters, and Zm is a normalization factor, such that Σ am (2) 1. Observe that wh can be interpreted as our prior belief in predictions of TCT,, i.e. large weight wel, indicates less confidence in the predictions of TCj- On the other hand, D represents the prior of i-th class for ICm. The m indices in Wmm and bm are needed as the weights are trained independently for each subset {IC₁ j S m). Although there viable potential approaches to setting these parameters by hand, we verified that optimizing them directly by minimizing the cross-entropy loss on the training dataset works best.
Out of additive and geometric ensemble settings we found the latter to be preferable. In this formulation, a low class confidence of single IC would significantly reduce the probability of that class in the whole ensemble. In consequence, in order for the confidence of the given class to be high, require all ICs to be confident in that class. Thus, in geometric ensembling, an incorrect although confident 1C answer has less chance ending calculations prematurely. In the additive setting, the negative impact of single confident but incorrect IC is much higher, show in Appendix Hence our choice of geometric ensembling.
Direct calculation of the product in (2) might lead to numerical instabilities whenever the probabilities are close to zero. To avoid this problem we note that
6 II (x)( J.m
wi 6', exp
Σ wh (n))
j5m
and that bog-probabilities In pf can be obtained by running the numerically stable log softmax function on the logits Io... of the classifier.
Both cascade connections and ensembling have diffferent impact on the model. Cascade connections primarily boost the accuracy of carly ICs. Ensembling, on the other hand, improves primarily the performance of later ICs, which combine the information from many previous classifiers.
This is not surprising, given that the power of the ensemble increases with the number of members, provided they are at least weak in the sense of boosting theory [33]. As such, the two techniques in- troduced above are complementary, which we also show empirically via ablation studies in Appendix C The whole training procedure is presented in Algorithm 1
Conditional inference Once ZTW model is trained, the following question appears: how to use the constructed system at test time? More precisely, we need to dynamically find the shortest processing path for a given input example. For this purpose, we use one of the standard confidence
5

---

Algorithm Zero Time Waste
Input: pre-trained model fe, cross-entropy loss function L, training set T. Initialize M shallow models Iø... at selected layers fo... For ..... M do in parallel
Cascade connection ICs
Set Pmn according to (T).
minimize ET (x). v)l wrt. Pm by gradient descent
For m = ,,...., .M do
> Geometric Ensembling
Initialize Wm, bm and define qm (2) according (2) minimize E(Z,))ET [f(mm(2)) v)] wrt. Wms bm by gradient descent
scores given by the probability of the most confident class. the m-th classifier is confident enough about its prediction, i.e. if
max I' T, for fixed r > 0,
(3)
where i is the class index, then we terminate the computation and return the response given by this IC. If this condition is not satisfied, we continue processing x and go to the next IC.
Threshold in (3) is manually selected value, which controls the acceleration-perfoon performance trade-off of the model. A lower threshold leads to a significant speed-up the cost of possible drop in accuracy. Observe that for 7 > 1, we recover the original model fo. since none of the ICs is confident enough to answer earlier. practice, to select its appropriate value, we advise using a held-out set to evaluate range of possible values of T.
Experiments
In this section we examine the performance of Zero Time Waste and analyze its impact on waste reduction in comparison to two recently proposed early-exit methods: (I) Shallow- Deep Networks (SDN) [1] and (2) Patience-Basee Based Early Exit (PBEE) [47] In contrast to SDN and PBEE, which train ICs independently. ZTW reuses information from past classifiers improve the performance. SDN and ZTW use maximum class probability as the confidence estimator, while PBEE checks the number of classifiers in sequence that gave the same prediction. For example, for PBEE r means that if the answer of the current IC is the same as the answers of the 2 preceding ICs, we can return that answer, otherwise we continue the computation.
In our experiments, we measure how much computation we can save by re-using responses of ICs while keeping good performance, hence obeying the zero waste paradigm. To evaluate the efficiency of the model, we compute the average number of floating-pon operations required to perform the forward pass for single sample. We use refer to it simply as the "inference time" in all subsequent references. For the evaluation in supervised learning, we use three datasets: CIFAR- 10, CIFAR- 100, and Tiny ImageNet, and four commonly used architectures: ResNet-56 [1I]. MobileNet [15,, WideResNet [45]. and VGG-16BN (37 as base networks. We check all combinations of methods, datasets, and architectures, giving 3 3 models in total, and we additionally evaluate a single architecture on the ImageNet dataset to show that the approach scalable. Additionally, we examine how Zero Time Waste performs at reducing waste in reinforcement learning setting of Atari 2600 environments. To the best of our knowledge, we are the first to apply carly cxit methods to reinforcement learning.
hardware-agnostic measure of inference cost and
36
Appendix A.I describes the details about the network architecture, hyperparameters, and training process. Appendix B contains extended plots and tables, and results of an additional transfer learning experiment. In Appendix C we provide ablation studies, focusing in particular on analyzing how each of the proposed improvements affects the performance, and empirically justifying some of the design choices (e.g. geometric ensembles vs. additive ensembles). We provide the source code for our experiments at
com/ comm/gum/ZZert Time Waste
4.1 Time Savings in Supervised Learning
We check what percentage of computation of the base network can be saved by reusing the information from previous layers in a supervised learning setting. To do this, we evaluate how each method behaves a particular fraction of the computational power (measured in floating point operations) of

---

Table 1: Results on four different architectures and three datasets: Cifar-10, C00 and Tiny ImageNet. Test accuracy (in percentages) for time budgets: 25%, 50%, 75%, 100% of the base network, and Max without any time limits. The first column shows the test aceuracy of the base network. The results represent mean of three runs and standard deviations are provided in Appendix We bold results within two standard deviations of the best model.
ResNet-56
MobileNet
Data
Algo
25%
50%
75%
100%
Max
Data
Algo 25%
50%
75%
100%
Max
SDN 77.7
87.3
91.1
92.0
92.1
SDN 86.1
90.5
90.8
90.7
90.9
C10
C10
PBEE 69.8
81.8
87.5
91.0
92.1
PBEE 76.3
85.9
89.7
90.9
91.1
(92.0)
(90.6)
ZTW 80.3
88.7
91.5
92.1
92.1
ZTW 86.7
90.9
91.4
91.4
91.5
SDN 47.1
57.2
64.7
69.0
69.7
SDN 54.3
63.5
66.8
67.8
67.9
C100
C100
PBEE 45.2
53.5
60.1
67.0
69.0
PBEE 47.1
61.6
61.6
67.0
68.0
(68.4)
(65.1)
7TW 51.3
62.1
68.4
70.7
70.9
ZTW 54.5
65.2
68.4
69.0
69.1
SDN 31.2
41.2
49.9
54.5
54.7
SDN 35.6
47.1
58.9
59.7
T-IM
T-IM
PBEE 29.0
37.6
48.2
53.4
54.3
PBEE 26.7
38.4
50.3
55.6
59.7
(53.9)
(59.3)
ZTW 35.2
46.2
53.7
56.3
56.4
ZTW 37.3
49.5
56.7
59.7
60.2
WideResNet
VGG
Data
Algo 25%
50%
75%
100%
Max
Data
Algo 25%
50%
100%
Max
SDN 83.8
91.7
94.1
94.4
94.4
SDN 86.0
92.1
93.0
93.0
93.0
C10
C10
PBEE 78.0
90.3
93.8
94.4
PBEE 75.0
86.0
91.0
92.9
93.1
(94.4)
(93.0)
ZTW 86.7
92.9
94.5
94.7
94.7
ZTW 87.1
92.5
93.2
93.2
93.2
SDN 55.9
65.1
71.6
75.0
75.4
SDN 58.5
67.2
70.6
71.4
71.5
C100
C100
PBEE 46.7
57.2
66.0
73.2
75.4
PBEE 51.2
65.3
65.3
70.9
72.0
(75.1)
(70.4)
ZTW 59.5
69.1
74.5
76.2
76.4
ZTW 60.2
69.3
72.6
73.5
73.6
SDN 36.8
46.0
54.6
59.4
59.7
SDN 40.0
50.5
57.4
59.6
59.7
T-IM
T-IM (59.0)
PBEE 29.9
37.8
52.7
58.5
59.7
PBEE 31.0
45.2
55.2
60.1
60.2
(59.6)
ZTW 40.0
50.1
57.5
60.2
60.3
ZTW 41.4
52.3
59.3
60.1
60.5
the base network. We select the highest threshold such that the average inference time is smaller than, for example, 25% of the original time. Then we calculate accuracy for that threshold. Table contains summary of this analysis, averaged over three seeds, with further details (plots for all thresholds, standard deviations) shown in Appendix B.I
Looking at the results, we highlight the fact that methods which do not reuse information between IC's do not always achieve the goal of reducing computational waste. For example, SDN and PBEE cannot maintain the accuracy of the base network for MobileNet the same computational power, scoring respectively 0.4 and 3.7 percentage points lower than the baseline. Adding ICs to the network and then discarding their predictions when they are not confident enough to return the final answer introduces computational overhead without any gains. By reusing the information from previous ICs ZTW overcomes this issue and maintains the accuracy of the base network for considered settings. In particular cases, such as ResNet-56 on Tiny ImageNet or MobileNet on Cifar- 100, Zero Time Waste even significantly outperforms the core network.
Tiny ImageNet when using
Similar observation can be made for other inference time limits as well. ZTW consistently maintains high accuracy using less computational resources than the other approaches, for all combinations of datasets and architectures. Although PBEE reuses information from previous layers decide whether to stop computation or not, this is not sufficient to reduce the waste in the network. While PBEE outperforms SDN when given higher inference time limits, often fails smaller limits (25%, 50%). We hypothesize that this is result of the fact that PBEE has smaller flexibility with respect r. While for SDN and ZTW values of 7 are continuous, for PBEE they represent a discrete number of IC's that must sequentially agree before returning answer.
7

---

WOGT 6an CIFARTTO (ower better)
ReskeetB Tinylmageheet (lower better)
Independent: ICs Zero Time Waste Ics
Independent ICs
Zero Time Waste ICs
04
Himoright mmg
03
Hincsght
122
02
22 14
Figure 2: Hindsight Improvability. For each IC (horizontal axis) we look at examples misclassified and we check how many of them were classified correctly by any of the previous ICs. The lower the number, the better the IC is at reusing previous information.
Finally, we check whether our observations scale up to larger datasets by running experiments on ImageNet using a pre-trained ResNet-50 from the torchvision package The results presented in Table show that Zero Time Waste is able gain significant improvements over the tested baselines even in this more challenging setting. Additional details of this experiments are presented in Appendix B.2
Given the performance of ZTW, the results show that paying attention to the minimization of com- putational waste leads to tangible, practical im- provements of the inference time of the network. Therefore, we devote next section to explaining where the empirical gains come from and how to measure information loss in the models.
Table 2: ImageNet results (test accuracy in percent- age points) show that zero-waste approach scales up to larger datasets.
Algo
25%
50%
75%
100%
SDN
33.8
53.8
69.7
75.8
PBEE
28.3
28.3
62.9
73.3
4.2 Information Loss in Early Exit Models
ZTW
34.9
54.9
70.6
76.3
Since ICs in a given model are heavily corre- lated, it is not immediately obvious why reusing past predictions should improve performance. Later ICs operate on high-level features for which class separation is much easier than for early TCs. and hence get better accuracy. Thus, we ask a question there something that early ICs know that the later ICs do not?
For that purpose, introduce metric to evaluate how much a given IC could improve performance by reusing information from all previous IC's. We measure it by checking how many examples incorrectly classified by ICm were classified correctly by any of the previous ICs. An IC which reuses predictions from the past perfectly would achieve a low score on this metric since it would remember all the correct answers of the previous TCs. On the other hand, an TC in a model which trains each classifier independently would have a higher score on this metric, since it does not use past information at all. We call this metric Hindsight Improvability (HI) since it measures how many mistakes we would be able to avoid if we used information from the past efficiently.
Let Cm denote the set of examples correctly classified by ICm, with its complement T.m. being the set of examples classified incorrectly. To measure the Hindsight Improvability of IC,, calculate:
7 n (Unem C.)
HI,..
c
Figure 2 compares the values of HI for a method with independent ICs (SDN in this case) and ZTW which explicitly recycles computations. In the case of VGG16 trained with independent ICs, over 60% of the mistakes could be avoided if we properly used information from the past, which

---

Qacrt-00
Poing hwr.heannmmeiipppoooo
21
19
14K
Ratuum
10.
84
ZTW
2TW
64
Original 2TW KC
Or gial
27W IC
20h
1007 14 50%
22%4
1007
Figure 3: Inference time vs. average return of the ZTW policy in an RL setting on bert and Pong Atari 2600 environments. The plot was generated by using different values of the confidence threshold r hyperparameter. Since the RL environments are stochastic, we plot the return with standard deviation calculated on 10 runs. ZTW saves significant amount of computation while preserving the original performance, showcasing that waste can be minimized also in the reinforcement learning domain.
would translate to improvement from 71.5% to 82.9% accuracy. Similarily, for ResNet-56 trained on TinylmageNet, the number of errors could be by around 57%.
ZTW consistently outperforms the baseline, with the largest differences visible at the later ICs, which can in principle gain the most from reusing previous predictions. Thus, Zero Time Waste is able to efficiently recycle information from the past. At the same time, there still a room for significant improvements, which shows that future waste approaches could offer additional enhancements.
4.3
Time Savings in Reinforcement Learning
Although supervised learning is an important testbed for deep learning, it does not properly reflect the challenges encountered in the real world. In order examine the impact of waste-minin masstemmiimmzzation methods in a setting that reflects the sequential nature of interacting with the world, we evaluate it in a Reinforcement Learning (RL) setting. particular, we use the environments from the suite of Atari 2600 games (28
Similarly as in the supervised setting, we start with a pre-trained network, which in this case represents a policy trained with the Proximal Policy Optimization (PPO) algorithm (36). We attach the ICs the network and train it by distilling the knowledge from the core network the ICs. We use a behavioral cloning approach, where the states are sampled from the policy defined by the 1Css and the labels are provided by the expert model. Since actions in Atari 2600 are discrete, we can then use the same confidence threshold-basee based approach to early exit inference as in the case of classification. More details about the training process are provided the Appendix A.2
In order to investigate the relationship between computation waste reduction and performance, we evaluate Zero Time Waste for different values of confidence threshold т. By setting very high value, we retrieve the performance of the original model (none of the ICs respond) and by slowly decreasing its value we can reduce the computational cost (ICs begin to return answers carlier). In Figure 3 we check values of in the interval [0.1, 1.0] to show how ZTW is able to control the acceleration-perffrmmace balance for Bert and Pong, two popular Atari 2600 environments. By setting lower 7 thresholds for Bert we can save around 45% of computations without score degradation. Similarly, for Pong we can get 60% reduction with minor impact on performance (note that average human score is 9.3 points). This shows that even the small four-layered convolutional architecture commonly used for Atari (28) introduces a noticeable waste computation which can be mitigated within a zero- waste paradigm. We highlight this fact the field of reinforcement learning has largely focused on efficiency in terms of number of samples and training time, while paying less attention to the issue of efficient inference.

---

IC
IC 13
IC
IC
400
CB
Truck
200
brr
0.0
0.5
1.0
0.0
05
10 0.0
0.5 100 0.0
0.5
1,0
0.0
0.5
10
horse
plane
0.0
1.0
100
1.0
0.5
100
0.5
1.0
Figure 4: Examples of bird images which were incorrectly classified as airplanes by ZTW. The early ICs are misled by the low-level features (blue sky, sharp edges, grayscale silhouette) and return a prediction before the later TCs can detect more subtle high-level features.
4.4 Impact Limitations
Our framework is the cornerstone of an environmental-awarc computation where information recy- cling within model is cautiously studied to avoid wasting resources. The focus on computational efficiency, however, introduces a natural trade-off between model accuracy and its computational cost. Although in most cases can carefully adjust the appropriate method hyperparameters to avoid a significant accuracy drop, some testing samples remain surprisingly challenging for ZTW, which indicates need for further investigation of the accuracy computation cost trade-off offered by our method.
Figure 4 contains examples of images for which low-level features in given image consistently point wrong class, while high-level features would allow us to deduce the correct class. Images of birds which contain sharp lines and grayscale silhouettes are interpreted as airplanes by early ICs which operate on low-level features. the confidence of these classifiers gets high enough, the answer might be returned before later classifiers correct this decision. We highlight the problem of dealing with examples which are seemingly easy but turn out difficult as an important future direction for conditional computation methods.
Conclusion
In this work, we show that discarding predictions of the previous ICs in early exit models leads to waste of computation resources and significant loss of information. This result is supported by the introduced Hindsight Improvability metric, as well as empirical result for reducing computations existing networks. The proposed Zero Time Waste method attempts to solve these issues by incorporating outputs from the past heads by using cascade connections and geometric ensembling. We show that ZTW outperforms other approaches on multiple standard datasets and architectures for supervised learning. as well as in Atari 2600 reinforcement learning suite. At the same time we postulate that focusing on reducing the computational waste a safe and stable way an important direction for future research in deep learning.
Acknowledgments & Funding Disclosure
This research was funded by Foundation for Polish Science (grant no W..R..04.4400... (4..00000114D//18--00 carried within the Team-Net program co-financed by the European Union under the European Regional Development Fund) and National Science Centre, Poland (grant no / and grant 2020/39/BBSTT/011111 The authors have applied CC BY license to any Author Accepted Manuscript (AAM) version arising from this submission, in accordance with the grants' open access conditions.
10

---

References
[1] Dan Ariely and Michael I. Norton. From thinking too little to thinking too much: continuum of decision making. WIREs Cognitive Science, 2(1):39-46, 2011.
[2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons conditional computation. arXiv:1308.3432. 2013.
[3] Candice Bentéjac, Anna Csörgö. Gonzalo Martinez-Muñoon A comparative analysis gradient boosting algorithms. Artificial Intelligence Review, pages 1-31, 2020.
[4] Konstantin Berestizshevsky and Guy Even. Dynamically sacrificing accuracy for reduced computation: cascaded inference based softmax confidence. In Proceedings of the International Conference on Artificial Neural Networks, ICANN, pages 306-320. Springer, 2019.
[5] Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. 1. 2013.
[6] Thomas Dietterich. Ensemble methods in machine lcarning. In Proceedings of the International Workshop on Multiple Classifier Systems, page 15. Springer, 2000.
[7] Gabriel Dulac-Arold, Daniel J. Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. CoRR, abs/1904. 1ess/9944.29911, 2019.
[8] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: loss landscape perspective. urXiv.1912.02757, 2019.
[9] Gerd Gigerenzer and Wolfgang Gaissmaier. Heuristic decision making. Annual Review of Psychology, 62:451-888 82, 2011.
[10] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Maccsanu. A survey of deep learning techniques for autonomous driving, Journal of Field Robotics, 37(3):622-386, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, CVPR, pages 770-778, 2016.
[12] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389-1397, 2017.
[13] Todd Hester and Peter Stone. TEXPLORE: Real-time sample-efficient reinforcement learning for robots. Machine Learning, 90(3):385 429, 2013.
[14] Geoffrey Hinton, Oriol Vinyals. and Jeffrey Dean. Distilling the knowledge in neural network. In Proceedings of NIPS Workshop on Deep Learning and Representation Learning, 2015.
[15] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1e04.0444111111 2017.
[16] Gao Huang, Danlu Chen, Tianhong L., Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018
[17] Sunggoo Jung, Sunyou Hwang, Heemin Shin, and David Hyunchul Shim. Perception, guidance, and navigation for indoor autonomous drone racing using deep learning. IEEE Robotics and Automation Letters, 3(3):3339 2544, 2018
[18] Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2017.
[19] Yigitean Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the International Conference on Machine Learning, ICML, pages 3301-3310, 2019.
[20] Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu. Fangbing Yan, et al. Identifying medical diagnoses and treatable discases image- based deep learning, Cell, 172(5):1122 122-1111112 2018.

---

[21] Alexandros Kouris, Stylianos I. Venieris. Michail Rizakis, and Christos-Savee -avvvas Bouganis. Approximate LSTMs for time-constrained inference: Enabling fast reaction in self-driving cars, 2019.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 00788499922220 2017.
[23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, NIPS, pages 6402-6413, 2017.
[24] Juhyoung Lee, Sangyeob Kim, Sangjin Kim, Wooyoung Jo, and Hoi-Jun Yoo. Gst: Group-sparse training for accelerating deep reinforcement learning. 2021.
[25] Hao Li. Hong Zhang, Xiaojuan Qi. Yang Ruigang, and Gao Huang. Improved techniques for training adaptive deep networks. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 1891-1900, 2019.
[26] Dor Livne and Kobi Cohen. Pops: Policy pruning and shrinking for deep reinforcement learning, 2020.
[27] Mason McGill and Pietro Perona. Deciding how to decide: dynamic routing in artificial neural networks. In ICML'17 Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 2363-2372, 2017.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-5333 -5322 2015.
[29] Mary Phuong and Christoph H Lampert. Distillation- based training for multi-exit architectures. In Proceedings of the IEEE International Conference Computer Vision, ICCV, pages 1355-1364, 2019.
[30] Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, lattps://githubb.com
DLR Baapp/DDDDRR-MMMMSS RM/stable- baseb ines3 2019.
[31] Simone Scardapane, Danilo Comminiello, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Differentiable branching in deep networks for fast inference. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, pages 4167-4171, 2020.
[32] Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add carly exits to neural networks? arXiv:2004. 128/4, 2020.
[33] Robert E. Schapire. The strength of weak learnability. Machine Learning, 5(2):197-227, 1990.
[34] E. Schuitema. L. Buşoniu. Robert Babuška, and P. Jonker. Control delay in reinforcement learning for real- time dynamic systems: A memoryless approach. 2010 IEEE/RSI International Conference on Intelligent Robots and Systems, pages 3226-3231. 2010.
[35] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. nigh-dimensional continuous control using generalized advantage estimation. arXiv:1506.022 024388 2015.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777 2017.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Proceedings of the International Conference on Learning Representations, ICLR, 2015.
[38] Surat Teerapittayanon, Bradley McDanel, and Hsiang- Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In Proceedings of the International Conference on Pattern Recognition, ICPR, pages 1464-2469, 2016.
[39] Paul Viola and Michael J. Jones. Robust real-time face detection. International Journal of Computer Vision, 1772)11371144, 2004.
[40] Xin Wang. Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. Idk cascades: Fast deep learning by learning not to overthink. In UAI, pages 580-590, 2017.
[41] Xin Wang. Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision, ECCV, pages 409 424, 2018
12

---

[42] Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph Gonzalez. Deep mixture of experts via shallow embedding. In Proceedings of the Uncertainty Artificial Intelligence, UAI, pages 552-562, 2020.
[43] Yulin Wang, Kangchen Lv, Rui Huang. Shiji Song, Le Yang, and Gao Huang. Glance and focus: dynamic approach to reducing spatial redundancy image classification. Advances in Neural Information Processing Systems, volume 33, pages 2432-2444, 2020.
[44] Yang. Yizeng Han, Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2369-2262222222 2378, 2020.
[45] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07144 2016.
[46] Hongjie Zhang, Zhuocheng and Jing Accelerating deep reinforcement learning with neural network compression. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 888, 2019.
[47] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT loses patience: fast and robust inference with early exit, urXiv:2006.04333 2020.
13


Reviews:
[{"review_id": "hNTC9zHxa64", "reviewer": "Reviewer_AQfs", "summary": "This paper proposes a Zero Time Waste (ZTW) method with cascading and ensembling components to speed up neural network inference. The ZTW model attaches internal classifiers (ICs) to pre-trained models, and the model is evaluated in both image classification benchmarks (CIFAR-10, 100, mini-ImageNet) and reinforcement learning benchmarks (Atari-games). \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.  Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nThis paper studies the efficient inference task, which is not a new task. However, the analysis of information loss and wasted computation operations can be an interesting perspective. \n\nThe proposed method follows the line of work in conditional computation and attaches independent classifiers to the network to enable early exit during inference.  As the authors stated in the paper (abstract & related work), this has been done in the literature. The claimed novelty comes from the combination of cascading and ensembling, and the proposed method also applies to pre-trained models (lines 98-100 in Related work).  However, I'm a bit concerned about the claimed novelty. \n\nModel cascade and early exit have rich literature which is not fully discussed in the paper.  For example,  IDK Cascades (wang et al.) attaches IDK classifiers to pre-trained models to form model cascades, claiming that \"none of these methods are designed to work with pre-trained models\" in lines 99-100 too bold.  Also, the authors listed a few related works in line 80 but did not make a clear contrast.  For example, the authors cite the SACT[29] in line 80 as \"skipping intermediate layers,\" which is inaccurate.  SACT uses halting scores to enable early exit, and the lower layers are always executed (not skipping), which is closer to the cascading method.  \n\nGiven the existence of rich study on model cascades, it seems the novelty comes from ensembling which helps reuse the computation from the lower layers.  However, dynamic routing style methods should have effectively used the computation in the network. Unfortunately, the authors didn't provide a comparison with the dynamic routing methods. Also, they didn't provide an ablation study of the proposed ZTW methods with and without ensembling the predictions of the Internal Classifiers (ICs).\n\nIMHO,  the clarity of the contribution and novelty, and discussion of previous work need improvement.\n\n2. Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nThe submission is technically sound, and the experimental evaluation is somewhat thorough.  However,  for the image classification task, the authors only compared two recent works. They ignored the large body of work using related approaches like dynamic routing, cascades, anytime prediction, etc.  Also, the authors only apply the model to CIFAR-10/100 and mini-ImageNet.  It is common and more convincing to conduct a comparison in the full ImageNet dataset in the literature. \n\nAblation study on ensembling, as mentioned in the reviews above, is missing. \n\nResults on wallclock speed up are recommended to include. \n\nThe results in RL benchmarks are a plus. \n\n\n3. Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nOverall the paper is well written. However, the clarity in the related work section can be further improved (check the concerns in Sec 1.). In addition, it would be clearer if the authors can explicitly describe how their work differs from the previous work and restate their approaches to guide the audience for comparison.  \n\nAlso, there are a few typos in the manuscript. \n- line 59. Hindisight or Hindsight. \n- line 60. \"propose a instance of ..\" -> \"propose an instance ..\"\n- line 72. \"reduce the inference speed\".  The authors may mean the opposite. \n\n4. Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\n\nYes. The results are important, and they may inspire other researchers in the field.  The results on RL benchmarks are interesting. However, more studies can be done on large-scale image classification benchmarks. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and social impact adequately. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "ZosF_SuLSeT", "reviewer": "Reviewer_WikB", "summary": "The focus of this work is on so-called early exit methods in deep neural network training and in particular on increased efficiency facilitated by the authors' proposal to connect internal classifiers which sit at the crux of early exit approaches. The proposed ensemble based method is shown to improve on the accuracy/inference time trade-off.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem considered is topical and the authors' approach is methodologically sound, well justified, and convincing.  All of the theoretical aspects of the submission look correct to me.  The empirical part of the paper is reasonably comprehensive as well; I do not think that the authors can be faulted here.  The paper is also well written.  As far as the key contributions go, from the broad idea itself to the introduced concept of hindsight improvability, there are interesting things to be found here, no doubt, but as I have to judge every paper not merely on its merits but on its merits relative to the pool of submissions (or expected pool of submission, given that as a single reviewer I have limited view) I would say that the overall value of this paper is around the borderline, perhaps slightly below it.  Hence my recommendation. \n\nHowever, what would in my mind increase the impact of the work and make it more convincing, thus changing my recommendation towards acceptance, is the replication of the findings on data sets were time is really of crucial importance.  Time saving in general is good, of course, but in many applications it is neither here nor there - one just takes more time.  However, in some domains the times involved are really seriously limiting. This could for example be on medical images of the kind used in pathology and which are huge.  Demonstrating the ability to save time in this context, or equivalently, to use more training data within specified time, would be of major value.  Another example is that of various astronomical images.\n\n--- Round 2\nI am grateful to the authors for directly addressing my concerns. Their response does indeed make me feel more positive (or rather, even more since I liked the key ideas of the work already) towards the submission.  HI have also read with interest other reviewers' comments and the corresponding authors' responses, and in all cases I found that the authors made a good case for their work, clarifying points which may have needed clarification or elaboration.  Hence, I have upgraded my recommendation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "W4KFD51lxkL", "reviewer": "Reviewer_Z6co", "summary": "This paper proposes Zero Time Waste (ZTW), a novel approach to leverage previous intermediate classifiers (ICs) for early-exit neural networks. ZTW consists of two simple extensions of early-exits NNs. The first is to add cascaded connections from the result of previous ICs to the next IC. The second is to ensemble previous results for confidence measurement. The author conducts extensive experiments on image classification benchmarks and shows improved performance compared to vanilla early-exit neural networks and PABEE, a previous work investigating using results from previous ICs for confidence measurement.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well written. The proposed method and experiments are clearly described.\n\n- The paper conducts both experiments on supervised learning and reinforcement learning. This is good given most previous works only consider the supervised setting.\n\n- The idea of ZTW is simple and intuitive, which is fine. And the empirical performance of ZTW seems to be promising. However, given the fact that PABEE already exploited the idea of using the results of previous ICs and their approach can be viewed as a voting-based ensemble scheme, the ensembling technique proposed in ZTW seems to be a direct extension to logits average based ensemble. The discussion about PABEE states that PABEE does not work well on these image classification settings, this is probably because the criterion in PABEE is tailored for NLP applications where the number of output classes is small. For the first technical contribution, I actually wonder why this would help a lot. Since the intermediate representation fed in the previous ICs is passed to the next ICs through residual connection, and the intermediate classifiers are only very small MLPs, the information gain by cascade connection seems to be marginal.\n\n- The paper introduces two novel techniques but does not perform an ablation study to show the relative improvement yielded by each of them. This prevents readers from getting a comprehensive understanding of the proposed method. This kind of analysis is crucial and I strongly recommend adding this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BzxRrMwkMtx", "reviewer": "Reviewer_6Qh4", "summary": "This paper presents an approach to reuse predictions of intermediate classifiers in order to improve classification performance.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I found of great interest the idea of combining the ratings of intermediate classifiers using something well known in machine learning, that is, boosting using cascades and ensembles. The methodologies are then not new as this is \"old machine learning\", but I find the idea of applying them into deep learning a good contribution. To the best of my knowledge, related work is adequate.\n\nQuality: I find methods to be appropiate and a complete piece of work. By combining such intermediate classifiers, the paper achieves quite an improvement in some well known datasets. Computation time is also evaluated in RL. Authors evaluate weaknesses which is a plus.\n\nClarity: I found the paper well written in general, although some English can be corrected (e.g. successes and others) and very clear to understand. In general terms I find the paper very interesting and the experimental evaluation is extensive and well analyzed. The methodology is also clearly stated and the equations seem correct. \n\nSignificance: The paper is relevant to the field and contributes on extending deep learning through boosting. It advances the state of the art considering the extensive experimental evaluation.\n\nMaybe some minor criticisms: Include some references about cascade connections (e.g. Viola and Jones, 2004 and others).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper makes a thorough analysis of the approach, including limitations, which is some wrong classifications and an analysis on why this would happen.\nIn any case, I found the paper interesting, well written and the methodology well evaluated.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an early exit strategy for pre-trained networks that allow computation to be reduced at prediction time using internal classifiers (ICs) that can terminate the required computation early.  Its distinction from previous work is in "recycling" information from previous ICs by a) feeding the output of their logits into the next IC, and b) using a geometric ensembling approach for the final predictions.

The review scores for the paper were quite split, with two reviewers (after discussions) in favour of acceptance and two reviewers feeling the paper falls marginally below the threshold.  The authors provided quite comprehensive responses to the original reviews, with all reviewers following these up and one deciding to increase their original score.  

Highlighted strengths of the paper include:
- The strength of its numerical results and the extensive number of experiments carried out
- Its overall quality, completeness, and polish
- Its clarity
- The fact that the algorithm itself is original (though there were varying opinions on quite how novel the approach is)
- The simplicity of the approach

Concerns raised by the reviewers include:
- The discussion of previous work (though the reviewers that raised this seem satisfied it can be corrected in the updated paper)
- Lack of comparisons to methods that are trained alongside the base network, rather than being applied post-hoc to an already trained network
- Lack of sufficient ablations (I believe this concern is unwarranted: as the authors point out, quite extensive ablations are actually provided in the supplement)
- Whether the paper is actually the first to use ensembling for early exit prediction (I also feel this is unwarranted as the paper clearly explains the differences and I do not feel it overclaims)
- Generality of the approach beyond image data settings

Given the high variance of scores, I have carefully also read the paper myself to form my own independent opinion, which is generally rather positive: I think this is high quality and useful work that deserves to be published (if I were a reviewer, I would have myself given the paper a "7").  In particular, I think it is quite unusually comprehensive in it considerations of issues and experiments, and that it is generally very well written and accessible.  I also feel that the authors did a very good job in addressing the concerns of the reviewers in their responses, and am inclined to take their side on most of the issues raised.  Specifically:
- I think the concerns of Reviewer Z6co were quite extensively addressed in the rebuttal and it is somewhat unclear what their remaining concerns about the technical contributions are as nothing major is raised in their review that is not addressed: their concerns about ablations are clearly unfounded; the focus on images instead of NLP would not prohibit publication and the new experiments show strong performance in the setting the reviewer is concerned with anyway; and the fact that the PABEE approach could be viewed as ensembling is not reasonable grounds for rejection in my opinion.
- Though I think the concerns of Reviewer AQfs are quite reasonable, I do not see them as sufficient grounds to reject the paper.  It should be first noted here that the reviewer themselves here also expressed in the private discussions that they were unsure if they had "set the bar too high" with their concerns and expressed that they were open to increasing their score, so I do not feel that I am completely at odds with their assessment here anyway.  To be more specific, I think they are correct to point out that comparisons between early exit approaches and those trained with efficient prediction in mind is important, but I also agree with the authors that this has not generally been done by previous work in the literature and that such comparisons are actually quite difficult to do fairly.  As such, I think it would be very harsh to reject the paper on these grounds, given that it is otherwise quite strong work.  I would strongly encourage the authors to see if they can add in any such comparisons for the camera-ready as it would noticeably strengthen the work and help correct what might be a reoccurring issue in the literature, but I do not think this should be a pre-requisite for publication (and I certainly would not suggest it is necessary to beat these baselines, so much as include them for context).

Given the above specific comments, my overall feeling is that the paper does not have any major flaws that warrant rejection by themselves, while the positives of the paper are sufficiently strong to overlook the more minor and fixable issues.  I thus recommend acceptance.

Some other comments from my own reading of the work to help make further improvements:
- The claim “Cascade connections primarily boost the accuracy of early IC” could do with some more careful explanation/discussion.  Because of the stop gradients, the cascading can never help the first IC (unless I've missed something) and at the moment the paper is a little misleading in that it is still the pushed forward information that is the thing which is helpful, but that this extra information is most important for relatively early ICs (but presumable not monotonically so: I would assume there is some sort of sweet spot, even if that is the 2nd IC?).
- There seems to be a typo in the definition of HI_m: should it not be the union of the previous C_n rather than the intersect so that it corresponds to *any* previous IC being correct?
- I presume all numbers in the tables are test accuracies?  This is never actually explicitly stated.  Information/details also seem to be missing on any use of held-out/validation data for hyper-parameter tuning.
Decision: Accept (Poster)