E$^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning
Authors: Alperen Gormez
Erdem Koyuncu
Venue: ICLR 2022 Submitted (2022)
Keywords: class means
early exit
efficient neural networks

Abstract: State-of-the-art neural networks with early exit mechanisms often need considerable amount of training and fine-tuning to achieve good performance with low computational cost. We propose a novel early exit technique, E$^2$CM, based on the class means of samples. Unlike most existing schemes, E$^2$CM does not require gradient-based training of internal classifiers. This makes it particularly useful for neural network training in low-power devices, as in wireless edge networks. In particular, given a fixed training time budget, E$^2$CM achieves higher accuracy as compared to existing early exit mechanisms. Moreover, if there are no limitations on the training time budget, E$^2$CM can be combined with an existing early exit scheme to boost the latter's performance, achieving a better trade-off between computational cost and network accuracy. We also show that E$^2$CM can be used to decrease the computational cost in unsupervised learning tasks.

Under review as a conference paper at ICLR 2022
EºCM: EARLY EXIT VIA CLASS MEANS FOR EFFI- CIENT SUPERVISED AND UNSUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-art neural networks with early exit mechanisms often need consid- erable amount of training and fine-tuning to achieve good performance with low computational cost. We propose a novel early exit technique, E²CM, based on the class means of samples. Unlike most existing schemes, E does not require gradient-based training of internal classifiers. This makes it particularly useful for neural network training in low-power devices, as in wireless edge networks. particular, given a fixed training time budget, EºCM achieves higher accuracy compared to existing early exit mechanisms. Moreover, if there are no limita- tions on the training time budget, E can be combined with an existing early exit scheme to boost the latter's 's performance, achieving better off between computational cost and network accuracy. We also show that E'CM can be used to decrease the computational cost in unsupervised learning tasks.
INTRODUCTION
Modern deep learning models require vast amount of computational resources effectively per- form various tasks such as object detection (Szegedy et al., 2015), image classification (He et al.) 2016), machine translation, (Vaswani et al. 2017) and text generation (Brown et al., 2020). De- ploying deep learning models to the edge, such as to mobile phones or the Internet of Things (IoT), thus becomes particularly challenging due device computation and energy limitations (Li al.) b). Moreover, the law of diminishing returns applies to the
trade-
off (Bolukbasi et al.) 2017): The increase in deep learning model's 's performance often marginal as compared to the increase in the amount of computation.
One of the primary reasons behind traditional deep learning models' high computation demand is their tunnel-like design. In fact, traditional models apply the same sequence of operations to any given input. However, in many real world datasets, certain inputs may consist of much simpler fea- tures as compared to other imputs (Bolukbasi al. 2017). In such a scenario, it becomes desirable to design more efficient architectures that can exploit the heterogeneous complexity of dataset mem- bers. This can be achieved by introducing additional exit points to the models (Panda et al., 2016 TTerapiitayaaoon et al.) 2016 Kaya et al. 2019). These exit points prevent simple inputs to traverse the entire network, reducing the computational cost of inference.
Despite reduced inference time, existing early exit neural network architectures require additional training and fine-tuning for the early exit points, which increases the training time (Teerapittayanon et al. 2016: BBolubasi et al. 2017 Kaya et al. 2019). This side-effect is undesirable for scenarios which the training has approach that does not require mrudient-based training and performs well. In this work, we propose such an early-exit mechanism, Early Exit Class Means (E²CM), based on the class means of input samples for the image classification task. By taking the mean of layer outputs for each class at every layer of the model, class means are obtained. During inference, output of a layer is compared with the corresponding class means using Euclidean distance as the metric. If the output of the layer is close enough to a class mean, the execution is stopped and the sample exits the network. In fact, seen in Fig. [ some samples can be classified easily at early stages of the network by just considering a "nearest class mean" decision rule, suggesting the potential effectiveness of our method for reducing computational cost.
done
low-power device. An ideal solution is plug-and-play

---

Under review as a conference paper at ICLR 2022
5000
Ground-trov 6-truth class
Ground-truth class
Grourd-trvin souund-rruth
4000
3000
Predicted class
Predicted class
Predicted class
Ground-truff trnnnht-ruhhhh class
Greund-truth class
Ground Arrth class
1000
Predicted class
Predicted class
Predicted class
of samples predicted correctly
Figure Confusion matrix of the classifications done according to the nearest class mean on CIFAR-10 training set. From left to right, top to bottom; the results belong the first convolu- tional layer, 1st 3rd, 10th, 15th and 30th residual block of ResNet-152.
practical use case for EsCM when a large, expensive-to-trene model is broadcast to edge devices with limited and heterogeneous computation capabilities. In such scenario, different devices may train the E'CM model at different FLOP operating points depending on their computation capabil- ities power limitations. Another practical scenario for E2CM may be transfer learning, where the training has to be done on a low power edge device with a local dataset that is different from the dataset the base network was trained on. In order keep the battery usage minimum on the edge device, the training time may be limited. The advantage of E²CM is that it does not need any gradient-based training like Shallow-Dee Shallow-Deep Networks (Kaya et 2019). therefore is more suitable for transfer learning on low power devices.
One other practical scenario for EºCM may be transfer learning, where the training has to be done on low power edge device with local dataset that is different from the dataset the base network was trained on. In order keep the battery usage minimum on the edge device, the training time may be limited. The advantage of E2CM is that it does not need any gradient-based training like Shallow-Des Deep Networks (Kaya et al. 2019).
To the best of our knowledge, E²CM is the first early exit mechanism that does not require gradient- based training and does not modify the original network by any means. Furthermore, does not have hyper-parameter for the early exit locations unlike existing schemes. These features make E²CM simpler to use and easier to deploy on low-power devices. While using class means as the only early exit mechanism requires just single feed forward pass, existing early exit methods reaches the same performance after training for multiple epochs, which suggests our method is more agile and powerful yet simpler. Moreover, combining E²CM with the existing mechanisms that require gradient-based training achieves a better trade-off in terms of computation cost and network accuracy. In addition its benefits in the supervised learning setting, E is also the first technique that shows the feasibility of early exits in the unsupervised learning setting.
We show the effectiveness of E²CM on CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009 and KMNIST (Clanuwat et al.) 2018) datasets using ResNet-18, ResNet-152 (He et al. 2016 and WideResNNet-111 models (Zagoruyko & Komodakis 2016). Using class means as the sole decision mechanism for early exiting results in 35% better accuracy or 50% faster inference time compared to the current state-of-the-an early exit technique. When combined with the state-of-the-arr, we increase the accuracy by 9% without doing further computation; decrease the inference time by 25% without loss in accuracy. We also show that it possible to decrease the computational cost while doing clustering with autoencoders MNIST (LeCun et al. 1998). In particular, E?CM saves on computation by 60%, while the loss in unsupervised clustering accuracy (Xie et al.) 2016) marginal.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Our work is related to the area of conditional computation (Bengio et al., 2013), where several small networks are trained to control the computation flow of one deep neural network. For this purpose. adding gates between the blocks of residual networks have been proposed (Veit & Belongie 2018 Wang et al. 2018). During inference, these gates allow the input to skip unnecessary blocks, thus saving computation time. However, unlike our proposed method, the gated network has to be trained from scratch. Conditional computation policies can also be learned through reinforcement learning (Bengio et al. [0155] Tie et al. 2019). However, this latter approach forces the inputs to go through the entire network as it does incorporate early exit points (Bengio et al. 2015).
One of the earliest works that explicitly propose the idea of early exiting is Panda et al. (2016), where the authors consider adding cascade of linear layers after convolutional layers as control blocks. Rather than just linear layers, adding branches consisting of convolutional layers to the original model has also been studied (Teerapittayanon et al.] 2016). A significant drawback of this method is that the branches may increase the computational cost due to convolutional layers. In addition, this idea requires branches to be trained with the original model jointly, from scratch. In a more recent study, internal classifiers (ICs) consisting of a feature reduction layer and single linear layer are added after certain layers the network (Kaya et al. 2019). Hence, the methods presented in these studies modify the original network by adding linear convolutional layers. Moreover, they require gradient updates to train those layers. Also, an implicit
the locations of the early exit points. E'CM is better suited for low-power applications compared to existing studies since we do not modify the original model, do not require gradient based training and additional hyper-parameters.
In addition to layer level early exits, a network level early exit mechanism has been introduced in Bolukbasi al. (2017). Both the layer level exit and the network level exit require decision functions to be inserted between the layers and networks. This type of architecture freezes the weights of the original model, and then trains the decision functions one by one using weighted binary classification. The drawback of this approach is such an alternative optimization, which may consume lot of time and energy when there are many decision functions to optimize.
The idea of early exit neural networks shows promising results in natural language processing do- main too, Adding early exit points between the transformer layers reduce the computation time with marginal loss in performance (Zhou et al., 2020 Xin et al. (020). Applications for neural network based early exit mechanisms to computer fault management have also been considered (Biasielli et al. 2020
Intermediate layer outputs have been used for classification in few-shot and one-shot learning set- tings in the past (Koch et al.] 2015)] Vinyals et al., (206)) (Sell et al., 2017, [ung et al. 2018). These studies are closely related with the area of metric learning. The closest work to E'CM is prototypi- cal networks, in which prototypes for each class are computed (Pan et al. 2019). However, none of those approaches aim to reduce the computation cost.
E CM is also related to the phenomenon of neural collapse (Papyan et al. 2020 It is known that as the inputs go deeper in a neural network, the classes are separated better from each other as a result of multiple nonlinearities, and the samples begin to concentrate (Cohen et al. 2020 Zarka et al.] 2020). EºCM exploits this phenomenon with the main idea of stopping the execution as soon as the sample close enough concentration point, i.e., class mean.
One other idea to reduce the computational cost of neural networks during inference time in the literature is the usage of multiresolution inputs and networks (Huang et al. 2017 Yang al. 20200] McGill& & Peronal (017). While this idea works well, it needs subnetworks and the usage of mul- tiresolution inputs, which means it requires more hyperparameters compared to E²CM.
EARLY EXIT CLASS MEANS
(6) We study the problem of image classification. Let (z) (0) E D be an image-label pair from the dataset D consisting of N samples and K distinct classes, where (()) 21,2,.... K} and i (1,2....,N} N). We denote the network F with M layers sequence 11,12,...,14. -MM- Let g()
3

---

Under review as a conference paper at ICLR 2022
( denote the prediction of the network, 2i denote the output of layer and yj denote the prediction in case the input exits the network after layer j, for = =1,2, j M. for the network F
,.....,,,
We can write the full equation
2 = (1 11, j= 1, ...
(1)
3.1 CLASS MEANS
The input to C is the network F trained on D. The network F is modified by any means. Therefore, we can obtain the class means for each class at each layer easily by just forward pass. This is especially useful when the training time budget fixed. Let Sy denote the of samples whose ground-truth truth label is k, and oţ denote the mean of the output of layer j for class k. In other words, let
(p) )
Sk
SE
Then, the Euclidean distance between a layer output (i) o and K class means of is computed via
dy = ||
cille, E (1,2.... K}.
each layer for every sample in the dataset, we normalize the distances for After calculating dh each class as
(1)
& (1,2....,K) KK.
(4)
Finally, the normalized distances converted probabilities of input belonging to class in order to perform inference. This is done using the softmax function as
P(U k) = softmax(
(5)
During inference, the decision of exiting after l or moving forward to 1 is made according to threshold value Tj. If the largest softmax probability is greater than the specified threshold Tj, execution is stopped and the class with the largest softmax probability is predicted. In other words, if
max(softmax( -d Tí
(6)
then the network predicts
arg max(softmax( )).
Otherwise, the input moves forward the next layer. In the worst case, execution ends at the last layer of the network.
FºCM performs differently according different of threshold values. We randomly initialize the thresholds and use binary search updates to reach the target number of FLOPS on training set. Later, same threshold values are used on the test set during the inference phase for that target number of FLOPs. The full procedure of our method is shown in Algorithm
3.2 COMBINATION OF CLASS MEANS WITH EXISTING SCHEMES
Performance of an existing early exit scheme be boosted if they are combined with E2CM. In this context, existing methods decide either according to the entropy of the early exit prediction TTerapittayanon al. 2016), or the largest probability value in the prediction (Kaya et al. 2019). in the former strategy, the input to the neural network exits early if the entropy is smaller than threshold T, for Layer j of the network. In the latter, the sample exits early the largest probability value is greater than T,-
(8) Existing methods use only x; as the input to the next layer. We propose feeding the class means ofj as additional inputs to the layers by simple concatenation, which improves the performance. During inference, if the jth internal classifier decides to move to the next layer, class means are consulted. If class means do not approve moving on, the prediction of the jth internal classifier is returned and the imput exits carly. Hence, an input can move to the next layer if and only if it receives approval from both the internal classifier (which can be based any existing scheme) and our simple R22 C.M.
4

---

Under review as a conference paper at ICLR 2022
Algorithm Early Exit Class Means (E²CM)
Input: Trained network layers I,, dataset D, thresholds 7 if training then
to M do for
(1) (a,ft)
2j
Calculate class means ctt k E 21, K} end for end
if
inference then for
j = to M do
(1) = (x
2j
Compute (g = 11
(1)
+0 c/1l2
Normalize (/(0) in (4.
if max
}) T then
Early exit with arg max; wwwftmaan-aateo - -dftt)) end if end for end
3.3 EXTENSION TO UNSUPERVISED LEARNING
E2CM can be used for clustering as well. As an example, we consider Deep Embedding Clustering (DEC) (Xie al. 2016) and focus on the task of jointly learning representations and cluster assign- ments. In DEC, there is only one clustering layer, and it is at the end of the encoder layers. This makes it impractical for low power clustering, because the architecture is like tunnel with only one exit at the end. We propose adding multiple clustering layers as early exits in order to decrease the computational cost.
RESULTS
We validate the effectiveness of method CIFAR- 10, CIFAR- 100 (Krizhevsky et al. 2009).
KMNIST (Clanuwat et al. 2018), and MNIST (LeCun et al. 1998) datasets. CIFAR- 10 and CIFAR- 100 dataset consists of 50000 training and 10000 test images, and have 10 and 100 classes respec- tively with equal amount of samples for each class. KMNIST dataset consists of 60000 training and 10000 test images like MNIST. While MNIST is a dataset of handwritten digits, KMNIST consists of 10 Hiragana characters, which are grayscale images unlike CIFAR- 10 images. We use ResNet- 18, ResNet- 152 (He al. (016) and WideResNet- 101 (Zagoruyko Komodakis (0166) models for our supervised learning experiments. For all models, we use the same data augmentation scheme and the hyper-parameter values stated Veit & Belongie (2018) for training. We use cross-entropy loss for all trainings. For our unsupervised learning experiments, we use the same network architecture and training parameters stated in Xie et al. (2016).
We run experiments in three settings. First, fix the training time budget and compare EºCM with the existing early exit methods in terms of network accuracy and floating point operations (FLOPs) performed during inference. Euclidean distance caleulations our method. In the second setting. we lift the training time budget. We allow the full training of the internal classifiers, and compare the combination of E*CM and internal classifiers against existing methods. The third setting is for unsupervised learning, and we generate the accuracy- FLOPs curve evaluate the effectiveness E*CM.
included in FLOPs for
4.1 CLASS MEANS UNDER A FIXED TRAINING TIME BUDGET
In the fixed training time budget setting, compare CM with existing methods in two ways. First, E2CM compared with Shallow- Deep Networks (Kaya et al. 2019) and BranchyNet (Teer- aapttayanonn et al. 2016), which trained for only one epoch since E2CM requires only sin-
5

---

Under review as a conference paper at ICLR 2022
05
00
Inalon-Dcce iexelghte
05
PCM
Erancr Met
03
11/3%
03
08
ETC
Stilion D443
02
1.0
12
02 100
12
1.09)
Figure 2: Comparison of CMCM with existing methods under fixed training time budget one epoch for ResNet-152 (left) and WideResNet-100 101 (middle) on CIFAR-10 10 (top) and KMNIST (bottom). ResNet-18 18 for CIFAR-IO is shown at top right, and ResNet-152 for CIFAR-100 is shown at bottom right.
gle forward pass. We do not include here the Bolukbasi- Wwwe-Deeel-Saliggaaeee (BWDS) method (Bolukbasi et al., 2017), because in this method, each decision function requires separate train- ing Hence, we cannot train an entire BWDS network with many decision functions using only one epoch.
Shallow-Deep Networks add internal classifiers after certain layers in the network. We use the procedure described in Kaya al. 2019) and add ICs after the layers which correspond to the 15%, 30%, 45% 60%, 75%, 90% of the entire network in terms of FLOPs. For BranchyNet, we add branches to the original network. The first branch is after the first convolutional layer, and the second branch is after the layer that corresponds to 1/3 of the whole network in terms of FLOPs.
Secondly, we still consider fixed training time budget, but this time we allow the separate training of decision functions in the BWDS method. As suggested by the authors, there are 6 early exit points, hence 6 decision functions. We train each decision function for epoch, resulting in separate epochs. To make fair comparison, train Shallow-Deep Networks and BranchyNet for epochs as well. We combine E²CM with Shallow- Deep Networks. By doing so, E²CM rectifies the decisions made by Shallow-Deey Networks. At every exit point of Shallow-Dee Deep Networks, if the decision made in favor of exiting early at the exit point, E²CM is consulted. CMM disagrees with Shallow-Deep Networks, early exit does not happen at that exit point.
To reach a target number of FLOPs, we first initialize the threshold vector by drawing M numbers uniformly random from [o, 1]. The jth component of the threshold vector corresponds T3 in Algorithm [ and is utilized at Layer 15 of the neural network. Then, the softmax values are obtained and saved Tor each layer training needed to optimize the thresholds. We update the threshold vector until we reach the target number of FLOPs using binary search each component and alternating optimization: if the thresholds give larger number of FLOPs than the target FLOP, the thresholds are decreased for the next iter- ation. Otherwise, if they give smaller number of FLOPs than the target FI.OP, they are increased. Binary search is guaranteed converge the (average) FLOPS are monotonic with respect to the thresholds.
in Eq. 5 Therefore, only one pass on training is
To obtain the overall tradeoff between accuracy and FLOPS, the above threshold optimization pro- cess is repeated on the training set for target FLOPs ranging from 0.0 to 1.0 with 0.001 granularity. We then compute the convex hull of the resulting 1000 points in the FLOPs-Accuracy plane, This yields the best performing thresholds on the training set. Finally, we use these thresholds on test set and form the FL
curves. Each point given curve represents threshold vector.

---

Under review as a conference paper at ICLR 2022
Table Comparison C with existing methods under fixed training time budget of six epochs
Model & Dataset
Method
Acc@0.11
Acc@ 0.20
1c25
Acc 0..0
EºCM SDN
77%
85%
87 %
87 %
ResNet-152 CIFAR-10
SDN
66%
75%
85%
87%
BWDS
77%
82%
84%
86%
BranchyNet
57%
57%
57%
57%
EºCM SDN
84%
93%
96%
96%
ResNet-152
SDN
63%
75%
86%
94%
CIFAR-10
BWDS
84%
93%
95%
95%
BranchyNet
76%
77%
78%
78%
E*CM SDN
34%
46 %
52%
54%
ResNet-152
SDN
20%
24%
27%
30%
CIFAR-100
BWDS
33%
40%
42%
44%
BranchyNet
24%
25%
26%
26%
EºCM SDN
81%
87%
88%
88%
WedeResNett-O1
SDN
70%
79%
86%
88%
CIFAR-10
BWDS
82%
85%
87%
88%
BranchyNet
56%
56%
57%
57%
EPCM SDN
85%
92%
96%
96%
WideResNet- 101 KMNIST
SDN
63%
74%
96%
96%
BWDS
88%
90%
92%
94%
BranchyNet
80%
82%
83%
84%
Points are connected via lines: The performance of any point on any given line is achievable simply time sharing models that correspond to line endpoints.
As shown in Fig. EºCM outperform all existing schemes under fixed training time budget. Specifically, using class means as the only decision mechanism achieves 35% better accuracy or %50 faster inference time for certain cases. This is because E²CM use the trained weights of the original vanilla model, which generalizes well to the dataset and can be used for classification. On the other hand, internal classifiers require further training. Therefore, they require more time to be ready for the task of classification. Also, can be seen from Table that combining E?CM with Shallow- Deep Networks (SDN) increases the performance by 50% to 100% when the number of classes is large. Table I shows accuracies at 0.15, 0.20, 0.25 and 0.30 FLOPs only because after 0.30 FLOPs, the accuracy stays the same, which indicates that do not need the tunnel-like design of traditional networks.
should also be noted that E2CM achieve the same performance the full network but using only 25%-30% of the available computational resources. This is why Table shows accuracy at up to 0.30 FLOPs. This result strongly suggests that training internal classifiers is not an absolute necessity. Also, E²CM results in a maximum of 9% computational overhead per layer in terms of FLOPs for WideRessNet-101 and ResNet-152 on CIFAR-10. Therefore, the amount of extra compute is not lot. Furthermore, for ResNet-152 1522, WideRes MB, 0.43 MB and 0.242 MB of space to store the class means for layer on average respectively. For ResNet-152 152 CIFAR- 100, it takes 4.17 MB. Considering that ResNet- 152 takes 222 MB and WideResNet-100 101 takes 476 MB of space, the extra memory usage E²CM is not lot.
and ResNet-18 18 on CIFAR- it takes 0.417
4.2 CLASS MEANS UNLIMITED TRAINING
In this setting, lift the training time budget. We combine E²CM with Shallow-Deer Networks and compare this combination against BranchyNet, the BWDS method, and Shallow-Deep Networks only.
We train the internal classifiers of our merger E²CM and Shallow-Deep for 100 epochs. The corresponding high computational complexity may not be desirable for low-power devices. We

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of early exit methods
Model & Dataset
Method
Acc @ 0.15
Acc@0.20
Acc@00.5
Ace@ 0.30
E'CM SDN
82.4% %
86.4%
88.5%
88.8%
ResNet-152 CIFAR-10
SDN
80%
86%
88.4%
88.8%
BWDS
82%
84.1%
86%
87.7%
BranchyNet
80%
80%
80%
80%
E²CM SDN
94.1%
96.2%
96.8%
96.8%
ResNet-152
SDN
83.9%
94%
96.4%
96.6%
KMNIST
BWDS
94.5%
96%
96.5%
96.6%
BranchyNet
86%
87.5%
88.2%
89.5%
E²CM + SDN
84%
88%
88.6%
88.8% %
WidcResNcet101 CIFAR-10
SDN
82%
88%
88.5%
88.8%
BWDS
85.8%
86.5%
87.8%
88.2%
BranchyNet
81.9%
82.5%
84.9%
85%
E*CM SDN
94.1% %
96.5%
97.1% %
97.2%
WideResNett101
SDN
84.1% %
94%
96%
97%
KMNIST
BWDS
93.8%
94.2%
95%
96%
BranchyNet
89%
90.1%
90.2%
90.2%
follow the same threshold selection procedure described above. During inference, the decision of early exit made according to both the E²CM and the ICs.
We train BranchyResNet-152 and Branchy WideResNet-101 for 300 and 250 epochs respectively. We use stochastic gradient descent with batch size of 256. The loss the branches are added up to the loss of the final layer and the weighted average is taken as prescribed in [Teeraiitayanon et al. (2016). We have trained multiple combination of weights, and found that 6, 1/4, 4 1] achieves the best performance. For thresholds, follow the same procedure, but this time the range of values is [0, log K] where K- 10, because we consider the entropy the predictions to make a decision.
Since the task of training decision functions in the BWDS method is a binary classification task (i.e., early exit or not) they converge rather quickly. We separately train the classifiers that come after the decision functions well. We use pooling and single linear layer for these, as suggested by the authors.
As shown in Table 2 combining EºCM with internal classifiers achieves better trade-off between the computational cost and network accuracy. For low computational budget, E²CM improve the accuracy by more than 2%, with less computation. We also observe that BranchyNet suffers from training the network with the branches jointly. These branches hurt the overall performance. More- over, convolutional layers in the branches add significant computational cost without consider- able gain in accuracy.
Although early exit mechanisms with decision functions like the BWDS method provide decent performance, experimental results show that threshold based early exit mechanisms perform better. In other words, making a decision about early exit and then performing classification fares worse than the threshold-based strategies where classification and exiting decisions are melted into the same pot.
According to Table we can conclude that the consulting mechanism between the E2CM and the internal classifiers generally beneficial. The common decision that reached by the two classifiers rectify possible misclassifications and avoid unnecessary computation. This can be seen as an example ensembles, in which multiple classifiers are used to make a decision. Interestingly, our ensemble reduces the total computational cost unlike ordinary ensemble methods.
The model sizes for ResNet- 152 and WideResNet- 101 are 222 MB and 476 MB respectively. For the CIFAR-10 10 dataset, the extra storage overhead of EºCM for ResNet-152 and 11 are 22.12 2 MB and 15.4 MB, which correspond to 10% and 3% of the model sizes, respectively. The overheads are due the necessity of storing the class means every layer. These results show that
8

---

Under review as a conference paper at ICLR 2022
1.0
0.7
IccuSSyy
Accurccy
0.6
0.3
0.5
0.4
0.
0.9
1.0
0.5
0.0
PLOPs
FLOPs
Figure 3: Accuracy-FLOPs curve MNIST (left) and Fashion-MNIST (right) for unsupervised learning using E.CM Vertical lines indicate the individual FLOPs DECs. Each black curve is for experiment, and the red curve is the average of all experiments.
CMCM does not have a large memory footprint. Still, one improvement to E²CM in this context may be consider the early exits only specific points in other existing schemes (Kaya al., 2019), rather than after each layer. This can reduce the memory footprint as well as computational overhead. However, introducing such modifications give rise additional hyperparameters, which may be undesirable.
4.3 CLASS MEANS FOR UNSUPERVISED LEARNING
We follow the same experimental setup as in Xie et al. (016) for MNIST (LeCun et al. 1998) and Fashion-MNIST (Xiao et al. 2017) datasets. Namely, 500, 500, 2000, 10
encoder with 4 layers, which have neurons, with clustering layer (CL) at the end. Let this encoder and clustering layer be DECaargee After pretraining DECiorges we train the clustering layer in Xie al. (216)) Then, we create another encoder with the 500-500-10-CL architecture, where the weights of first two layers are copied from DECiarge and frozen. We name this encoder DECmidddl and follow the same procedure as in DECtorge- Finally, we repeat the same procedure for DE DECsmate which has the 500- 1O-CL. architecture, where the weight of the first layer frozen and copied from D..Ctarge-
After the training is complete, we take the intermediate outputs, i.e., the outputs of the layers with 10 neurons. Then, using the cluster centers from each clustering layer, we follow Algorithm [ To measure the accuracy, we use the same technique described in Xie et al. (2016).
During the experiments, we noticed that the process of pretraining affected the final result signifi- cantly. have thus run multiple experiments, and the performance corresponding to each exper- iment is illustrated as gray curve in Fig. The average of individual experiments is shown as the solid red curve.
As seen in Fig. by adding early exits the architecture, it is possible to save 60% of the compu- tation while losing only 6% in unsupervised clustering accuracy on MNIST dataset. On Fashion- MNNIST, the accuracy loss is 1%. Also, thresholding makes it possible adjust according to various computational needs.
CONCLUSION
We propose a novel early exit mechanism based on the class means. Unlike existing early exit mechanisms, our method does not require gradient-based training, which makes it useful for network training on low-power devices. Under fixed training time budget, our method outperforms existing early exit schemes. In addition, combining better trade-off between the computational cost and the network accuracy. Moreover, we show that method is not only useful in supervised learning tasks, but also in unsupervised learning tasks.
method with existing early exit techniques achieve

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Since our technique reduces the inference time with minimal compromise in performance, it can lead to more severe impacts when used in applications in which time has a crucial importance. Stock market prediction and adversarial attacks that are used in the military industry depend heavily on split second decisions, therefore using E²CM in such applications may cause negative impacts. Also, as in Kaya et al. 2019). new adversarial attacks can be developed to manipulate the computational load of deep learning systems which may hurt performance more than before.
REPRODUCIBILITY STATEMENT
We have uploaded the source code for our method as a supplementary material. For better repro- ducibility, we have utilized common datasets in our experiments, such CIFAR-10 (Krizhevsky et al., 2009). KMNIST (Clanuwat et al., 2018), Fashion-MNISS (Xiao et al. 2017) and MNIST (LeCun al. 1998). We have also used common model architectures such as ResNet--152 (He et al. 2016 and WideResNet-100 101 (Zagoruyko & 2anoodakis,0010)
REFERENCES
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv.151..06997, 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /0Mi:v/308.4322 200,,,333,,,1111
Matteo Biasielli, Cristiana Bolchini, Luca Cassano, Erdem Koyuncu, and Antonio Miele. A neural network based fault management scheme for reliable image processing. IEEE Transactions on Computers, 6965)76447776 2020.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, pp. 527-536. PMLR, 2017.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018.
Uri Cohen, Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv: 099444 2017.
Zequn Jie, Peng Sun, Xin Li, Jiashi Feng, and Wei Liu. Anytime recognition with routing convolu- tional networks. IEEE transactions pattern analysis and machine intelligence, 2019.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. International Conference on Machine Learning, pp. 3301- 3310. PMLR, 2019.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Léon Bottou, Yoshua Bengio. and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 2324, 1998.
Pengzhen Li, Erdem Koyuncu, and Hulya Seferoglu. ResPipe: Resilient model-distribute DNN training at edge networks. In IEEE Intl. Conf. Acoustics Speech Signal Process. (ICASSP), June 2021a.
Pengzhen Li, Hulya Seferoglu, Venkat R Dasari, and Erdem Koyuncu. Model-distributed dnn train- ing for nemory--onstraiie edge computing devices. In 2021 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN), pp. 1-6. IEEE, 2021b.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In International Conference on Machine Learning, pp. 2363-2372. PMLR, 2017.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Wai Wah Ngo, and Tao Mei. Transferrable prototyp- ical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2239-22222222 2247, 2019.
Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy- efficient and enhanced pattern recognition. In 2016 Design, Automation & Test Europe Con- ference & Exhibition (DATE), pp. 475-480. IEEE, 2016.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40); 24652-24663, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint 1rXiv:1703.05177. 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.
Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Surat Teerapittayanon, Bradley McDanel, and Hisiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469, IEEE, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 170.v:1706.03762, 2017.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 3-18, 2018
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. arXiv preprint arXiv: 160.000000. 2016.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In Proceedings of the European Conference Com- puter Vision (ECCV), pp. 409-424, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487. PMLR, 2016.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004. 12993, 2020.
11

---

Under review as a conference paper at ICLR 2022
Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2378, 2020.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146, 2016.
Wide
residual networks.
arXiv preprint
John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks arXiv preprint arXiv:2012. 2020.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses pa- tience: Fast and robust inference with early exit. arXiv preprint 2006: 04/22, 2020.
12


Reviews:
[{"review_id": "aSOr3Uk6mNK", "reviewer": "Reviewer_jksT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think studying early-exit is a very important research direction and the focus on simplicity is important (if not more than final performance) for achieving real world impact. As far as I am aware using nearest centroid classification (NCC) for early exit classification is novel (though not super original as NCC is used widely for classification) and the experimental study provided can be useful for future research. \n\nThis work is solely experimental, and is potentially interesting to the ICLR community. However, the experimental study is limited to some small datasets and lacks in-depth explorations. It is not clear how would the method scale to ImageNet or similar larger datasets/resolutions.\n \n# Concerns\n- Even though the authors of [1] focused on NLP, I think their method should be included as a baseline and possibly use the combination method to increase E2CM performance even further. Shallow-deep+e2cm is kind of similar to [1] as it uses 2 decision to decide to skip to the next layer (compared to using 2 decision to stop). Not sure how easy to do this during rebuttal, but regardless this would be a useful addition to the next version.\n- Is there a reason why authors choose to experiment with ResNet-152 and WideResNet-101 on Cifar/Mnist instead of repeating experiments done in previous work; let's say Shallow-Deep experiments. Using same benchmarks help researcher compare methods easily and therefore accelerates research. Moreover, Shallow-Deep experiments seems to be more diverse in terms of architecture and data. I recommend authors to have (at-least-some) matching experiments. Imagenet experiments would be appropriate, too; given the relatively small cost of the method.\n- I think many of the experimental details can be shared in appendix creating space for some new experiments. It would be nice to see some ablations and in-depth studies in the paper. some ideas: (a) effect of different normalizations: how does different normalizations used in architecture affect results. (b) How about the depth of the network? How does E2CM work on resnet-22? (C) How does the width of the network effects the utility/comparison.\n- Authors should give more details on binary search used to choose thresholds. An algorithm box in appendix would be nice. Similarly sharing the final algorithm for Shallow-Deep+E2CM would be nice. Reading the text it is not clear how does shallow-deep+e2cm works. Specially there are 2 ideas (1) concat of class means with activations (2) using 2 stage early-exit strategy. How do you do both of these?\n- Authors should compare their class-mean approach with prototypical adaptation (it is different), but readers would appreciate a short discussion on similarities and differences.\n- In Figure-2, why is the Branchy net curve flat? Furthermore Branchy-net uses 2 intermedieate activations. Do e2cm and shallow-deep methods use same number of intermediate layers? If e2cm uses all and shallow-deep uses 6, this might not be a fair comparison. Ideally all methods should be able to exit at same layers.\n\n# Minor\n- $c_4^kj$ is not defined. Is it a typo?\n- Authors mention 1000 points in text, however Figure-2 has much less points for each curve. How do you choose the points? Are they chosen on test set? \n- Using different symbols for different curves would make it easier to read in print.\n- \"One of the primary reasons behind traditional deep learning models’ high computation demand is their tunnel-like design\" It is the fact that they bring better performance. Their tunnel like design prevents parallelism, though helps finding more complex functions.\n- \"One practical scenario for E2CM may be transfer l\" Here e2cm is not defined yet. Possibly remove this paragraph.\n- \"which suggest our method is more agile and powerful yet simple\" -> agile and simple.\n- \"Given a fixed training time budget\" limited training time\n\n[1] Bert Loses Patience: https://arxiv.org/abs/2006.04152\n[2] https://arxiv.org/pdf/2106.05409.pdf\n\n# After Rebuttal\nI read authors' response. I thank authors for sharing their preliminary results on Resnet-56, I think adding rest of the experiments would be a great addition to the paper. Similarly, adding [1] to the experiments would be great (I don't think the algorithm requires any domain specific adjustments, the idea is relatively simple). In short, most of my concerns stay, thus I keep my score. Hope the authors find the feedback useful.", "summary_of_the_paper": "This paper proposes a new method for performing early exits on a pre-trained architecture. Authors use support set to calculate class prototypes at intermediate representations. Then at evaluation execution on the network is stopped whenever the representations are significantly closer to a class at any level. Results on small datasets (cifar-mnist) using very deep networks show that proposed method achieve better results compared to some of the previous work.", "summary_of_the_review": "I think studying early-exit inference is a very promising research direction and I appreciate this work due its simplicity, which is important for achieving real world impact. As far as I am aware using nearest centroid classification (NCC) for early exit classification is novel and the experimental study provided can be useful for future research. However the potential impact of this experimental work is currently limited due to its choice of datasets/architectures and lacks in-depth investigations/ablations. Without these, I am concerned that the impact of the work would be limited. For example, it is not clear whether the method would scale to larger datasets and perform better than other alternatives. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "93eHvIaFNJh", "reviewer": "Reviewer_ZDik", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written and easy to follow.\n- The proposed method is very simple and intuitive. It can be adopted in bare DNNs or in combination with previous early exit methods.\n\nWeaknesses:\n- Since the paper assumes that the DNN is pretrained, the early exit setup, e.g., the training of the sallow nets in (Kaya et al 2019) or the computation of per-class means and setting the threshold, can be done on the same HW platform where the DNN is originally trained. This step can thus be done prior to deploying the DNN in inference mode on a constrained embedded device. As such, it is unclear why it is important to reduce the training overhead for the early exit strategy? The evaluations in the non-constrained training time mode show that the proposed method cannot compete with prior methods unless it is combined with them.\n- Regarding the experiments in section 4.1, is the time overhead of finding the optimal threshold for a given FLOP constraint considered within the timing budget? For a fair comparison, the total allocated time to all methods should be equal, rather than just matching the number of forward passes. That being said, it would be great if the authors can comment on my previous question as to why the training budget needs to be limited.\n- The evaluated datasets are fairly simple, therefore it is not clear how well the method can generalize in face of more complicated data patterns such as ImageNet. Specifically, since the method depends on per-class means, it is important to evaluate on datasets with more than 10 layers, e.g., CIFAR100, or ImageNet. It is possible that as the number of classes increases, the intermediate representations, particularly in the earlier layers can no longer discriminate between the classes as easily, thereby increasing the FLOPs of this method.\n", "summary_of_the_paper": "The paper proposes a method for early exit during inference of DNNs. The method does not require fine-tuning or gradient computation. Rather, it uses the learned embeddings from intermediate model layers to decide where to terminate the execution. Specifically, for a trained model, the method calculates the per-class means of intermediate activation. The distance between the activations of new samples is then computed with the per-class means and inference is exited once the distance becomes lower than a set threshold.", "summary_of_the_review": "Please see the above review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4NqPuJDGw5o", "reviewer": "Reviewer_7dU6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the idea of the paper is cute, and it surprised me that this worked. A pleasant surprise, and I at least learned something new. For the given setup/data, it also seems to work well. Being able to add early-exiting to any network would be a very welcome addition to any framework that runs neural networks on devices.\n\nHowever, when reading the paper there are some glaring holes and potential problems with the method that the authors don't comment on.\n\nFirst of all is the actual computational overhead. This is definitely non-trivial. Take imagenet for example, one would have to do a lot of extra compute for each layer. As for every layer, we'd have to calculate a 1000 feature-map-size subtraction, matrix square and a large sum. Afterwards a softmax is taken, which for large numbers of classes (in e.g. language models this is very relevant), can be expensive as well. The authors seem to not really comment on this, nor calculate the actual overhead per layer. Only in the comparison to other methods do the flops come up. Second is the parameter overhead. The method is not parameter-free. Likely far from it. For every feature map, num_classesfeature_map size parameters have to be stored for comparison. For e.g. Imagenet with a 1000 classes, this is akin to storing a full forward pass of activations of a batch-size=1000. This can run into the gigabytes of memory! Given the author's focus on mobile devices, this doesn't seem very feasible.\n\nThe authors seem to avoid this problem by not considering datasets/tasks with a lot of class labels. For MNIST/CIFAR10, your overhead is not going to be too big, but scale this up to language models or sizeable classification models and the overhead is likely prohibitive. The second reason why these drawbacks seem to not show up in the paper is the usage of really really large models for MNIST and CIFAR10. Not only is the usage of these oversized models at odds with the idea of running them on mobile devices, there are far more efficient networks than those mentioned in the paper nowadays, as we can see from figure 1, after 30 layers the network has already classified the examples properly. One could just use that 30 layer network, stick a single softmax at the end, and use that for classification. I wonder what happens to the gains of this method when comparing on such more efficient models.\n\nIn terms of comparisons, I'm missing multi-scale dense nets. This is the de-facto paper in this area, and can't be missed from a comparison perspective. I also think the authors could have noted comparison with other methods that do something similar, like methods that drop features, networks that do anytime routing etc. (Adaptive neural networks for efficient inference, Deciding how to decide: Dynamic routing in artificial neural networks, Anytime recognition with routing convolutional network, Resolution Adaptive Networks for Efficient Inference, Skipnet: Learning dynamic routing in convolutional networks)\n\nOn a fun note, I wonder what happens if the euclidian distance metric is switched to a cosine one. This would turn the problem into a flattened weight tensor such that we have x.dot(c) for each class. Essentially this is a single neural network layer, with a softmax afterwards. This could work just as well, and make it easier to 'retrain'. Alternatively, you could consider what happens if you had more cluster centers than num_classes. Perhaps you'd be cheaper off mapping the entire feature map to some smaller sub-space, and then projecting back to the softmax output with e.g. an SVD method :)\n\n", "summary_of_the_paper": "The paper introduces a novel mechanism to add early exiting on existing networks without re-training the original network. The paper mentions the reason for this is that this can be done on lower-powered devices on-the-fly, where retraining is not wanted. The method does so by calculating output means, and comparing new examples with that, by running a distance function through a softmax, and early exiting if the prediction is above a certain threshold.", "summary_of_the_review": "All-in-all, the paper is interesting, but the practicality of the method is very problematic. \n\nI think I could increase my score if:\n\n- The authors commented far more on the added complexity and parameter count\n- Had comparisons to methods that have large num-classes\n- Had comparisons on networks that were actually representative of mobile use-cases, or at least something that's more reasonably sized than a 150 layer network for CIFAR10\n- Compared their method to multi-scale dense nets, and other mentioned methods", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-orFhdrr6s8", "reviewer": "Reviewer_Kvv8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed approach is novel, and, to the best of my knowledge, early exits have not been explored from the perspective of non-gradient training, which is a new research direction. \n- The empirical evaluation shows that the proposed method does outperform the baselines in a setting with a very limited computational budget. Additionally, the experiments showing how E2CM performs in combination with existing methods are an interesting way to engage with the related work.\n- The authors show how their method performs in an unsupervised learning setting, which is an interesting new application of early exits, which were so far mostly considered for classification problems.\n\nWeaknesses:\n- The empirical evaluation, in general, is lacking and should be extended, as in the current form the performance and use-cases of the proposed method are not clear. By this, I mean not only considering more complex datasets or different architectures, but also more in-depth analysis of the memory usage and providing statistics to gauge the significance of the results. In particular:\n  - The empirical evaluation in the supervised learning setting should be extended. The most complex dataset used in this paper is CIFAR-10, which is currently not considered particularly challenging by the community. The additional results on MNIST variants are not enough to weigh the importance of the method, especially when used with powerful architectures such as ResNet-152 or WideResNet-101. Large-scale experiments on datasets such as ImageNet would be most convincing, but even increasing the scope to SVHN, CIFAR-100 or mini-ImageNet would be beneficial. Since the method does not require editing the base network, would it be possible to adapt it to a wide array of pre-trained models available online (e.g. via websites such as PyTorch Hub or Model Zoo)?\n  - The problem of additional memory expenses is not discussed well enough. Although the authors consider the additional memory footprint, they do so for the case of CIFAR-10, where the number of classes is fairly low (10). In practice, many real-world datasets consist of a larger number of classes. For example, for CIFAR-100 the overhead would be 10x higher than for CIFAR-10, which, if I understand correctly, for ResNet-152 means ~100% increase in the memory usage. Similarly, for a dataset with 1000 classes, we have 1000% memory usage overhead. This is quite significant, especially in application to devices with low computational resources. The authors consider adding early exits only at specific points to reduce the memory overhead, but do not show experiments on well it would work in practice.\n  - The supervised learning experiments do not include standard deviation - have they been trained with more than one seed?\n- The practical motivation for the proposed method is not clear to me. Since the proposed method outperforms other approaches significantly only in the setting with a very limited training budget, the possible applications of the method seem to be restricted. The authors propose some potential applications, but the methods are not tested in settings related to these applications. In particular, the paper claims that \"A practical use case for E2CM is when a large, expensive-to-train model is broadcast to edge devices with limited and heterogeneous computation capabilities.\" (5th paragraph of the Introduction), which I wanted to discuss:\n  - \"In such a scenario, different devices may train the E2CM model at different FLOP operating points depending on their computation capabilities\" - Most early exit methods (e.g. Shallow-Deep Networks considered in this paper) can adapt the FLOP threshold after training by choosing the confidence threshold, denoted by $T_j$ in this paper. So one could train the early-exit model along with the expensive-to-train model, broadcast it to the edge devices, and set the FLOPs as needed.\n  - \"Another practical scenario for E2CM may be transfer learning, where the training has to be done on a low power edge device with a local dataset that is different from the dataset the base network was trained on\" - this use-case is intriguing and more convincing to me, but not tested empirically in the paper and it is not immediately obvious if the method would work (maybe the class means won't be as well separable if we change the domain/task).\n  - In the light of application to edge devices, a more thorough discussion of memory usage is even more important.\n- The authors do not provide enough theoretical considerations of the method to justify accepting the paper without convincing empirical results. This is not a weakness but itself, but rather highlights the fact that the work has to be judged on the merit of the empirical evaluation, which is lacking.\n\nTo sum up, the actionable feedback I propose is providing a more thorough empirical evaluation of the method, clearly stating its limitations, and exploring in-depth the settings the method can be applied to (like transfer learning).", "summary_of_the_paper": "The paper proposes a method for early-exit-based conditional computation for reducing the inference time of a given neural network. The approach works as follows: E2CM saves the mean activation of each class after each layer and then, during inference, compares the activation of a given example to the recorded mean activations to get a prediction and its certainty. If the certainty is high enough, this prediction is returned, otherwise, the next layer is queried instead.  The authors show empirically that the method outperforms other approaches in a setting with a very limited training budget, and is sometimes able to improve the performance of other methods when applied in combination. Finally, the method is also presented in an unsupervised learning setting where it allows for significant computation savings.", "summary_of_the_review": "Although the main idea of the paper is intriguing, I think the empirical evaluation needs to be extended and the possible application scenarios should be more thoroughly discussed. In the current version, I cannot recommend accepting this paper for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an early exit method that uses class means of samples that is gradient free and is aimed for low compute cases such as mobile and edge data. The idea is novel in this setting (though class means have been used for other settings such as few shot classification) and empirical results show that it works well. There are two main concerns from reviewer concerns that were not addressed by the author rebuttal. First, applicability of the model in real world due to its memory requirements and two, experiments that show performance on more realistic datasets such as Imagenet. The reason the latter is required is the promise of mobile application for the proposed method. I suggest the authors explain the first concern more and add the requested experiments in the upcoming version of the paper.
Decision: Reject