SelfXit: An Unsupervised Early Exit Mechanism for Deep Neural Networks
Authors: Hossein KhademSohi
Mohammadamin Abedi
Yani Ioannou
Steve Drew
Pooyan Jamshidi
Hadi Hemmati
Venue: Accepted by TMLR ()

Abstract: Deep Neural Networks (DNNs) have become an essential component in many application domains, including web-based services. A variety of these services require high throughput and (close to) real-time features, for instance, to respond or react to users' requests or to process a stream of incoming data on time. However, the trend in DNN design is towards larger models with many layers and parameters to achieve more accurate results. Although these models are often pre-trained, the computational complexity in such large models can still be relatively significant, hindering low inference latency. 
In this paper, we propose SelfXit, an end-to-end automated early exiting solution to improve the performance of DNN-based vision services in terms of computational complexity and inference latency. SelfXit adopts the ideas of self-distillation of DNN models and early exits specifically for vision applications. The proposed solution is an automated unsupervised early exiting mechanism that allows early exiting of a large model during inference time if the early exit model in one of the early exits is confident enough for final prediction. One of the main contributions of this paper is that we have implemented the idea as an unsupervised early exiting, meaning that the early exit models do not need access to training data and perform solely based on the incoming data at run-time, making it suitable for applications using pre-trained models.
The results of our experiments on two vision tasks (image classification and object detection) show that, on average, early exiting can reduce the computational complexity of these services up to 58% (in terms of FLOP count) and improve their inference latency up to 46% with a low to zero reduction in accuracy. SelfXit also outperforms existing methods, particularly on complex models and larger datasets. It achieves a notable reduction in latency of 51.6% and 30.4% on CIFAR100/Resnet50, with an accompanying increase in accuracy of 2.31% and 0.72\%, on average, compared to GATI and BranchyNet.

Published in Transactions on Machine Learning Research (10/2024)
SelfXit: An Unsupervised Early Exit Mechanism for Deep Neural Networks
Hossein KhademSohi
hossein. wwwdemmskskkwwlggrrrrr.o mohammadamin. wtwpa/miiinnedidooooooooo abbeei@llllllg.o.ggooo you' www.oouuealguur......oooooo..... ca
Mohammadamin Abedi
Yani Ioannou
Steve Drew
steve. www.w
Deportment of Electricnl and Software Engineering
University of Calgary
Calgory, AB, Canada
Pooyan Jamshidi
pjumshid@cse.sc.com edu
Department of Compuling Science and Engineering University of South Carolina Columbia, SC, USA
Hadi Hemmati
hemmati@yorku..a
Department of Electrical Engineering and Computer Science Universily of York Torondo, ON, Conada
Reviewed on OpenReview: https: openreview. net/ forun? id=h4rURKf155
Abstract
Deep Neural Networks (DNNs) have become an essential component in many application domains, including web-based services. A variety of those services require high throughput and (close to) real-time features, for instance, respond or react to users' requests or to pro- cess stream of incoming data on time. However, the trend in DNN design towards larger models with many layers and parameters to achieve more accurate results. Although these models are often pre-trained, the computational complexity in such large models can still be relatively siguificant, hindering low inference lateney. In this paper, propose SelfXit, end-to-end automated early exiting solution to improve the performance of DNN-based vi- sion services in terms of computational complexity and inference latency. SelfXit adopts the ideas of self-distillation of DNN models and early exits specifically for vision applications. The proposed solution an automated unsupervised early exiting mechanism that allows early exiting of large model during inference time if the early exit model in one of the early exits is confident enough for final prediction. One of the main contributions of this paper is that we have implemented the idea exit models do not need access to training data and perform solely based on the incoming data at run-time, making it suitable for applications using pre-trained models. The results of our experiments on two vision tasks (image classification and object detection) show that, on average, carly exiting can reduce the computational complexity of these services up to 58% (in terms of FLOP count) and improve their inference latency up 46% with low to zero reduction in accuracy. SelfXit also outperforms existing methods, particularly complex models and larger datasets. It achieves notable reduction in latency of 51.6% and 30.4% on CIFAR100/ CIFAR100/Resnet50) with an accompanying increase in accuracy of 2.31% and 0.72%, on average, compared to GATI and BranchyNet. We released code and replication
unsupervised ised early exiting, meaning that the early
package at

---

Published in Transactions on Machine Learning Research (10/2024)
1 Introduction
Deep Neural Networks (DNNs) are incorporated in real-world applications used by wide spectrum of industry sectors including healthcare (Shorten et al., 2021; Fink et al., 2020), finance (Huang el al., 2020;
Culkin & Das, 2017), self-driving vehicles (Swinney & Woods, 2021), and cybersecurity (Ferrag et al., 2020).
These applications utilize DNNs in various fields such as computer vision (Hassaballah & Awad, 2020;
Swinney & Woods, 2021), audio signal processing (Arakawa et al., 2019; Tashev & Mirsamadi, 20d natural language processing (Otter et al., 2021). Many services in large companies such as Google and Amazon have ONNN-based back-end software (e.g., Google Lens and Amazon Rekognition) with tremendous volume of queries per second. For instance, Google processes over 99,000 scarches every second (Mohsin, 2022) and spends a substantial amount of computation power and time on the run time of their models (Xiang & Kim, 2019). These services are often time-sensitive and resource-intensive and require high availability and reliability.
Now the question how fast the current. state-of-the-art (SOTA) DNN models are at inference time and to what extent they can provide low-latency responses to queries. The SOTA model depends on the application domain and the problem at hand. However, the trend in DNN design is indeed toward pre-trained large-scale models due to their reduced training cost (only fine-tuning) while providing dominating results (since they are huge models trained on an extensive dataset).
One of the downsides of large-scale models (pre-trained or not) is their high inference latency. Although the inference latency is usually negligible per instance, as discussed, a relatively slow inference can jeopardize a service's performance in terms of throughput when the QPS is high.
In general, in a DNN-based software development. and deployment, pipeline, the inference stage is part of the so-called "model serving" process, which enables the model to serve inference requests jobs (Xiang & Kim, 2019) by directly loading the model in the process or by employing serving frameworks such as TensorFlow Serving (Olston et al., 2017) or Clipper (Crankshaw et al., 2017).
The inference phase is an expensive stage in the life cycle of a deep neural model in terms of time and computation costs (Desislavov et al., 2021). Therefore, efforts towards decreasing the inference cost in production have increased rapidly over the past few years.
From a system engineering perspective, caching is standard practice to improve the performance of software systems, helping to avoid redundant computations. Caching is the process of storing recently observed information to be reused when needed in the future, rather than re-computation (Wessels, 2001; Maddah-Al Ali & Niesen, 2014). Caching is usually orthogonal to the underlying procedure, meaning that it is applied by observing the inputs and outputs of the target procedure and does not engage with the internal computations of the cached function.
Caching effectiveness is best observed when the cached procedure often receives duplicated input while in similar internal state, for instance, accessing particular memory block, loading web page, or fetching the books listed in a specific category in library database. It is also possible to adopt a standard caching approach with DNNs (e.g., some work caches a DNN's output solely based on its input values (Crankshaw et al., 2017)). However, it would most likely provide a meager improvement due to the high dimension and size of the data (such as images, audios, texts) and low duplication among the requests.
However, due to the fcature extraction nature of deep neural networks, we can expect the inputs with similar outputs (e.g., images of the same person or the same object) to have a pattern in the intermediate layers' activation values. Therefore, we exploit the opportunity to cache DNN's output based on the intermediate layer activation values. In this way, we can cache the results not by looking at the raw inputs but by looking at their extracted features in the intermediate layers within the model's -aasss
The intermediate layers often have dimensions even higher than the input data. Therefore, we use shallow classifiers (Kaya al., 2019) to replace the classic cache-storing and look-up procodures. A shallow classifier is a supplementary model attached to an intermediate layer in the base model that uses the intermediate layer's activation values to infer a prediction. In the caching method, training shallow classifier on a set.
2

---

Published in Transactions on Machine Learning Research (10/2024)
of samples mimics the procedure of storing those samples in cache storage, and inferring for a new sample using the shallow classifier mimics the look-up procedure.
In this paper, we propose early exiting the predictions made by standard classification models using shallow classifiers trained using the samples and information collected at inference time. We first evaluate the rationality of SelfXit in our first research question by measuring how it affects the final accuracy of the given base models and assessing the effectiveness of the parameters we introduce (tolerance and confidence thresholds)
knob to control the early exit certainty. We further evaluate the method in terms of computational complexity and inference latency improvements in the second and third research questions. We mcasure these improvements by comparing the FLOPs count, memory consumption, and inference latency for the original model vs. the early exit-enabled version that we built throughout this experiment. We observed a reduction of up to 58% in FLOPs, an acceleration of up to 46% in inference latency while inferring on the CPU and up to 18% on GPU, with less than 2% drop in accuracy. SelfXit demonstrates remarkable performance across a range of models and datasets. By averaging across all confidence levels greater than 0.25, for the simplest model and datasel, CIFAR10-resnee500, of 52.2% and 32.4% in latency, with only minor decrease of 6.9% and 1.6% in accuracy compared to the BranchyNet and GATI methods, respectively. Furthermore, in the case of the more complex CIFAR100- Resnet50 model and dataset, our method achieves a significant reduction in latency of 51.6% and 30.4%, while simultaneously enhancing the accuracy by 2.31% and 0.72% compared to the GATT and BranchyNet methods.
offers substantial reduction
In summary, the contributions of this paper are:
Proposing an early exiting method for the predictions made by off-the-shelf image classifiers and object detection models, which only uses unlabelled samples collected at inference time.
Automate the process of designing the supplementary models used for early exiting in vision appli- cations and tuning their parameters used to determine early exit hits, using AutoML methods.
Empirically evaluating the proposed early exiting method using six publicly available off-the-shelf models on six datasets (CIFAR-10, CIFAR-100, ImageNet, LFW, Cityscapes, Criteo), in terms of computational complexity and inference time reduction.
In the remainder of the paper, we discuss the background and related work in Section 2, details of the method in Section 3, the design and evaluation of the study in Section 4, and conclude the discussions in Section 5.
2 Related Works
In this section, we briefly review the topics related to the model inference optimization problem. Following this, we introduce the techniques used to build the carly exit procodure.
2.1 Inference Optimization
There are two perspectives that address the model inference optimization problem. The first perspective focuses on optimizing the model deployment platform and covers a broad range of optimization goals (Yu et al., 2021). These studies often target deployment environments in resource-constrained edge devices (Liu et al., 2021; Zhao et al., 2018) or resourceful cloud-based devices (Li ct al., 2020a). Others focus on hardware-specific optimizations (Zhu& & Jiang, 2018) and inference job scheduling (Wu et al., 2020).
The second perspective is focused on minimizing the model's inference compute requirements by compressing the model. Among model compression techniques, model pruning (Han et al., 2015; Zhang et al., 2018; Liu et al., 2019b), model quantization (Courbariaux et. al., 2015; Rastegari et al., 2016; Nagel et al., 2019), and model distillation (Bucila et al., 2006; Polino et al., 2018; Hinton et al., 2015) are widely used. These idcas alleviate the model's computational complexity by pruning the weights, computing the floating-point calculations at lower precision, and distilling the knowledge from teacher (more complex) model into a student (less complex) model, respectively. These techniques modify the original model and often cause
3

---

Published in Transactions on Machine Learning Research (10/2024)
fixed amount of loss in test accuracy. In (Liu et al., 2019a) and (Li et al., 2020b), the authors use search algorithms for channel pruning and suggest compressed networks with lighter models, which lead to lower accuracy but faster inference. In fact, in most cases in these studies, they either achieve low latency but at the cost of accuracy, or reasonable accuracy with high latency. The main disadvantage of these methods is that the computation cannot be tuned with any parameters. In practice, though, to what exlent one wants to sacrifice accuracy for faster inference is project dependent and must be tunable.
As mentioned in the motivation, accelerating inference is crucial for sensitive real-time approaches such autonomous vehicles. Most of the methods focus partitioning and offloading calculations to the edge (Mohammed et al., 2020). However, achieving faster decisions for a vehicle to detect a pedestrian requires a more immediate reaction than outsourcing data to the edge. We have applied SelfXit to state-of- the-art object detection approaches, such as Mask R-CNN (He et al., 2017) using popular urban datasets, showing a significant improvement even when using unlabeled data.
2.2 Early-Exits in DNNs
"Early exit" generally refers to an alternative path in a DNN model which can be taken by a sample instead of proceeding to the next layers of the model. Many previous works have used the concept of early exit for different purposes (Xiao et al., 2021; Scardapane al., 2020; Matsubara et al., 2022; Haseena Rahmath et al., 2023). Panda et al. (2016) is one of the carly works in this area. They tried to terminate classification by cascading linear network of outpul neurons for each convolutional layer and monitoring the output of the linear network to decide about the difficulty of imput instances and conditionally activate the deeper layers of the network. But they have not mentioned anything about the inference time and accuracy cetrracyttimme /time trade-off issue. BranchyNet (Teerapittayanon et al., 2016; Pacheco et al., 2021; Ebrahimi et al., 2022) also utilize their previous observation that features were learned in an early layer of a network to make an early exit. However, they require labeled data to train their models, rendering them unsuitable for use with unlabeled data. Shallow Deep Networks (SDN) (Kaya et al., 2019) points out the "overthinking problem in deep neural networks. "Overthinking" refers to models spending a fixed amount of computational resources for any query sample, regardless of their complexity (i.c., how deep the neural network should be to infer the correct prediction for the sample). Their research proposes attaching shallow classifiers to the intermediate layers in the model to form the early exits. Each shallow classifier in SDN provides a prediction basod on the values of the intermediate layer to which it attached. (Hanawal et al., 2022) propose an unsupervised carly exit mothod specifically designed for NLP tasks using Elastic BERT. Their approach utilizes bandit-based learning algorithm (UEF-CCB) to dynamically select exit points in multi-exit DNNs without requiring labeled data. However, the exit points are selected from fixed set of candidate exits, which may not be optimal in all scenarios. In addition, exit models are kept as simple classifiers, limiting their overall performance. This approach is primarily tailored to BERT- based architectures, making it less applicable to more complex domains, such as vision-based tasks, which are the focus of our work.
On the other hand, Xiao et al. (2021) incorporates the shallow classifiers to obtain multiple predictions for each sample. In their method, they use early exits as an ensemble of models to increase the base model's accuracy.
The functionality of the shallow classifiers in our proposed method is similar to that of SDN. However, the SDN method trains the shallow classifier using ground truth data from the training set and ignores the available knowlodge in the original model. This constraint renders the proposed method useless when using a pre-trained model without access to the original training data, which is commonly the case for practitioners.
2.3 DNN Distillation and Self-distillation
Among machine learning tasks, the classification category is one of the significant use cases where DNNs have been successful in recent years. Classification applied to a wide range of data, such as classification of images (Bharadi et al., 2017; Xia et al., 2021), text (Varghese et al., 2020), audio (Lee et al., 2009), and time series (Zheng et al., 2014).
4

---

Published in Transactions on Machine Learning Research (10/2024)
Knowledge distillation(KD) (Bucila et al., 2006; Polino et al., 2018; Hinton et al., 2015) is model compres- sion method that trains a relatively small (less complex) model known as the student to mimic the behavior of a larger (more complex) model known as the teacher. Classification models usually provide a probability distribution (PD) representing the probability of the input belonging to cach class. KD trains the student model to provide similar PDs (i.e., soft labels) to the teacher model rather than training it with just a class label for each sample (i.e., hard labels). KD uses specialized loss functions in the training process, such as Kullback-Leiblrr Divergence (Lovric, 2011) to measure how one PD is different from another.
KD is usually 2-step process consisting of training large complex model to achieve high accuracy and distilling its knowledge into smaller model. An essential challenge in KD is to choose the right teacher and student models. Self-distillation (Zhang et al., 2021) addresses this challenge by introducing a single- step method to train the teacher model along with multiple shallow classifiers. Each shallow classifier in self-distillation is a candidate student model, which trained by distilling the knowledge from one or more of the deeper classifiers. In contrast to SDN, sclf-distillation utilizes knowledge distillation to train shallow classifiers. However, it still trains the base model from scratch along with the shallow classifiers, using the original training set. This training procedure conflicts with our objectives in both aspects. Specifically, we use a pre-trained model and keep it unchanged throughout the experiment and only use inference data to train the shallow classifiers.
In Leontiadis et al. (2021) the authors present a method for enhancing CNN efficiency through early exits, Using supervision, self-supervision, and self-distillation, you can personalize the device, using both labeled and unlabeled data. This allows for dynamic adaptation with varying data availability, focusing on training enhancements. Our work is different by not altering the main model, but instead utilizing early exit layers updated solely during the inference time. This is done to improve latency, without the need for labeled data during these updates, which offers a modular solution with minimal modifications to existing systems.
Our work modifies and puts the presented methods in SDN and self-distillation in the context of carly exiting the final predictions of pre-trained DNN models. The method trains the shallow classifiers using only the unlabeled samples collected at run-time and measures the improvement. inference compute costs achieved by the early exits throughout the forward passes.
2.4 DNN Prediction Early Exiting
Clipper (Crankshaw et al., 2017) is a serving framework that incorporates early exiting DNNs predictions based on their inputs. Freeze inference (Kumar et al., 2019) investigates the use of traditional ML models such as K-NN and K-Means to predict based on intermediate layer values. They show that the size and computation complexity of those ML models grows proportionally with the number of available samples, and their computational overhcads far exceed any improvement. In the Learned Early Exits, Balasubramanian et al. (2021) extend the Freeze Inference by replacing the ML models with pair of DNN models. A predictor model that predicts the results and a binary classifier that predicts whether the result should be used as the final prediction. Their method uses ground-truth data in the process of training the predictor and selector models. In contrast, our method 1) only uses unlabeled inference data, 2) automates the process of early exit-enabling, 3) uses a confidence-bace early exit hit determination, and 4) handles batch processing by batch shrinking.
3 Methodology
In this section, we explain the method to convert a pre-trained deep neural model (which we call the backbone) to its extended version with our early exiting method (called early exit-enabled model). The carly exiting method adds one more early-exit paths to the backbone, controlled by the shallow classifiers (which we call the early exit models), allowing the model to infer a decision faster at run-time for some test data samples (early exit hits). Faster decisions for some queries will result in a reduced mean response time.
"Early Exit model" is supplementary model that we attach to an intermediate layer in the backbone, which given the layer's values provides a prediction (along with confidence value) for the backbone's output. Just reminder that as our principal motivation, we assume that the original training data is unavailable for the
5

---

Published in Transactions on Machine Learning Research (10/2024)
user, as is the case for most large-scale pre-trained models used in practice. Therefore, in the rest of the paper, unless we explicitly mention it, the terms dataset, training set, validation set, and test set all refer to the whole available data at rum-time or a respective subset.
Our procedure for enabling a pre-trained model to early exit is derived primarily from the delf-distillation method (Zhang et al., 2021). However, we adapt this method to early exit-enable pre-trained models using only their recorded outputs through self-training, without access to the ground truth (GT) labels. The novel aspects of our approach consist of selecting the best early exit model through Neural Architecture Search (NAS) followed by an inference-tine time early exit training. This approach helps us refine our early exit strategy based on performance feodback, with the aim of optimizing both accuracy and inference speed across all stages of the network. Since we employ early exit models in a sequence, if an early early exit model wrongly triggers an incorrect early exit, later early exit models will not have a chance to engage even they would have been successful. Therefore, an individual early exit model's accuracy /hit rate does not guarantee overall good results since the final outcomes depend on the carlier carly exit model's performance as well.
Setting. Our approach is built pre-trained model, and before starting, use NAS algorithm to select the top candidate layers with the potential to cache what the base model has learned. After this selection, the inference process begins by passing an input i through a sequence of layers (2, 11,2,...,44 Each candidate layer has an associated cache model that performs an early exit, producing an output with the same shape as the base model's output. The base model is frozen during the training and testing phases. For training, in each selected layer 4, we train the corresponding cache model using the probability distribution P == Pi) generated by the base model as a reference distribution and the probability distribution Q = (i) provided by the cache model for KT divergence. After training these early exit layers, during the testing phase, the model evaluates in each candidate layer whether the current representation (r. is sufficient to make a confident prediction and potentially terminate the inference process early. This decision is based on the confidence measure calculated in each layer. The process continues through the layers until an early exit is triggered or the final layer IA is reached, where the final prodiction is made. The model determines the output by ar.
A step-by-step guide on early exit-enabling an off-the-shelf pre-trained model from a user perspective contains the following steps:
1. Identify the candidate layers to be early exit This step involves analyzing each selected model to identify potential positions for early exit layers. Layers whose outputs are independent of other layers' states are chosen as candidates, enabling training possibilities.
2. Build an carly exit model for cach candidate Using neural architecture search (NAS), we evaluate all possible subsets for these early exit layers. NAS also scores the different architectures defined for our early exit models.
3. Assign confidence thresholds to built models to determine early exit hits Each early exit model features softmax as the final layer. In this step, we assign confidence thresholds to the constructed models to determine early exit hits. The probability value associated with the predicted output reflects the model's confidence in its prediction. This confidence level for given input determines whether accept that prediction as an early exit hit continue processing through the rest of the backbone model.
4. Evaluate and optimize the early exit-enabled model In this step, we evaluate and optimize the early exit-enabled model in the inference mode and also present our algorithm to update all early exit. layers.
5. Early-Exit Optimization Implementation In this step, we implement the algorithm and train all carly exit models using self-training (without using any ground truth) in the datascts we have mentioned.
In subsections 3.1 to 3.5, we further discuss the procedure and design decisions in each step outlined above.
6

---

Published in Transactions on Machine Learning Research (10/2024)
Frozen base model
Probability distributionss
L11
L1 Imput
L:6 Output
(Soft labels)
L2
L3
L4
L5
L6
L12
L13
L14
L15
Activation Vectors
CacheModel, L2
CacheModel,
CacheModel,,
CacheModel,
CacheModel,
CacheModel, L12
CacheModel
KLDivergence Loss
Figure 1: Early Exit-enabling procedure, candidate layers, and data paths.
3.1 Identifying candidate layers
Choosing which layers early exit is the first step towards early exit-enabling a model. A candidate layer is layer that will examine its values' correlation to the final predictions by training early exit model based them. One simply list all the layers in the backbone candidates. However, since launch search for an early exit model per candidate layer in the next step, we suggest narrowing the list by filtering out some layers with the following criteria:
Some layers are disabled at inference time, such dropouts and batch normalizations. These layers do not modify their input values at inference time. Thercfore, cross them off the candidate list.
Some of the last layers in the model (close to the output layer, such as L15 in Figure 1) might be valuable candidates for early exiting, since the remaining layers might not have heavy computations to reach the output. This is particularly important each carly exit model typically consists of least two layers. If the main path has only two layers remaining before the final result and softmax, adding an carly exit at that point would not significantly improve inference time. Therefore, we discard the last two layers (e.g,, L15 and L16) from consideration.
DNN models usually composed of multiple components (i.c. first-level modules) consisting of multiple layers such multiple residual blocks in ResNet models He et. al. (2016)). We narrow down the search space to the outputs layers in those components.
We only consider the layers, for which, given their activation values, the backbone's output is uniquely determined without any other layer's state involved (i.e,, the backbone's output is a function of the layer's is output). In other words, layer with other layers or connections in parallel (such as L7-L11 and L13 in the Figure 1) is not suitable for early exiting, since the backbone's output does not solely depend on the layer's output. While this approach has proven effective for the architectures we tested, we acknowlodge that may not generalize well to more complex architoctures like Mixture of Experts (MoE) models, where parallel experts are common. We have added this as limitation of method and suggest future exploration in this area.

---

Published in Transactions on Machine Learning Research (10/2024)
Having the initial set of the candidate layers, next build and associate an early exit model to each one. (training data extraction and loss function) are discussed in the following two subsections.
3.2 Building early exit models
Building an early exit model to be associated with an intermediate layer in the backbone consists of finding a suitable architecture for the early exit model and training the model with that architecture. The details of the architecture search (scarch space, search method, and evaluation method) and the training procodure
3.2.1 Early Exit models architecture
An early exit model can have architecture of any depth and breadth size, long as it provides more computational improvement than its overhead. Tn other words, it must have substantially less complexity (i.e., number of parameters and connections) than the rest of the layers in the backbone that come after the corresponding intermediate layer. The search space for such models would contain architectures with different numbers and types of layers (e.g., a stack of dense and/or convolution layers). However, all models in the scarch space must produce a PD identical to the output of the backbone in terms of size (i.e, the number of classes) and activation (e.g., SoftMax or LogSoftMax).
In our experiments, the search space consists of architectures with a stack of (up to 2) convolution layers followed by another stack of (up to 2) linear layers, with multiple choices of kernel and stride sizes for the convolutions and neuron counts for the linear layers. However, users can modily or expand the search space according to their specific needs and budget.
The objective of the scarch to find a minimal architocture that converges and predicts the backbone's output with acceptable accuracy. Note that any accuracy given by an early exit model (better than random) can be helpful, as we will have a proper selection mechanism later in the process to only use the early exit predictions that are (most likely) correct, and also to discard the early exit models yielding low computational improvement.
The user can conduct the search by empirically sampling through the search space or by using a automated Neural Architecture Search (NAS) tool such as Auto-Keras (Jin et al., 2019), Auto-PyTorch (Zimmer et al., 2021), Neural Network Intelligence (NNI) (Microsoft, 2021), or NASLib (Ruchte al., 2020). However, we used NNT to conduct the search and customized the evaluation process to account for the models' accuracy and their computational complexity. We have used the floating point operations (FLOPs) count the estimation for the models' computational complexity in this stage.
Several factors influence the architecture of an early exit model for a given intermediate layer. These factors include the dimcnsions of the intermodiate target layer, its position in the backbone, and the data set specifications, such as its number of larget classes. For example, the first early exit models the 100- Resnet50 and CIFAR10- 1een--tssset18 experiments (shown as Cache 1 in Figure 6) have the same input size, but since CIFAR100 has more target classos, it reasonably requires an carly exit model with more lcarning capacity. The architecture of each early exit model is further adjusted based on the specific requirements of the dataset, particularly the number of output classes. For example, while the early exit models for CIFAR10 and CIFAR100 share similar input sizes, the early exit models for CIFAR100 are designed with increased capacity to handle its larger number of classes. This adaptation is guided by the suggestions provided by the Neural Architecture Search (NAS); ensuring that each early exit model is optimized for the dataset to which it is applicd. This approach is consistent across different datascts, such as ImageNet, where the carly exit models are similarly adjusted to accommodate the 1,000 output classes. These architectural adjustments ensure that the models remain efficient and capable of handling the varying complexities of different datasets. Therefore, using NAS to design the early exit models helps automate the process and alleviate deep learning expert supervision in designing the early exit models.
NAS tries to minimize the total accuracy no more than the tolerance, so for all possible subsets, there will be a maximum range. This mothod does not guarantce the best score, but is a good solution. Subsets with the best score will be selected as our early exit system which will be added inside the model to be trained individually by the predictions and perform early exits.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Regardless of the search method, evaluating nominated architecture requires training a model with the given architecture, of which we discuss the procedure in the next section. Moreover, since the search space is limited in depth, it is possible that, for some intermediate layers, neither of the early exit models converges (i.c,, the model provides nearly random results). In such cases, we discard the candidate layer as non-suitable for early exiting.
3.2.2 Training an early exit model
Figure (1) illustrates the schema of early exit-enabled model consisting of the backbone (the dashed box) and the associated early exit models. The objective of early exit model is to predict the output of the backbone model, given the corresponding intermediate layer's output, per input sample.
Similarly to the backbone, early exit models are classification models. However, their imputs are the activation values the intermediate layers. As suggested in self-distillation (Zhang et al., 2021), training an early exit model essentially similar to distilling the knowledge from the backbone (final classifier) into the early exit model.
Therefore, to distill the knowledge from the backbone into the early exit models, we need a medial data set (MD) based on the collected inference data (ID). The medial data set for training an carly exit model associated with an intermediate layer L in the backbone B consists of the set of activation values in the layer and the PDs given by B per samples in the given ID, formally annotated as follows:
MD, = (1 E ID << (z),), >]
(1)
where:
MD2
Medial dataset for the early exit model associated with the layer L
ID
The collected inference data consisting of unlabeled samples
By (i): Activation values in layer L given the sample i to the backbone B
B(i) The backbone's PD output for the sample i
Note that the labels MDs are the backbone's outputs and not the GT labels, as we assumed the GT labels to be unavailable. We divide MD, into three partitions (MDFrain MD MD and use them, respectively, similarly to the common deep learning training and test practices.
Similarly to the distillation method (Hinton et al., 2015), use the Kullback-Leibler divergence (KLDiv) (Lovric, 2011) loss function in the training procedure. KLDiv measures how different the two given PDs are. Thus, minimizing the value of the KLDiv loss in M Dfrran trains the early exit model to estimate the prediction of the backbone (B())..
Unlike self-distillation, where (Zhang et al., 2021) train the backbone and shallow classifiers simultancously, in our method, while training an early exit model, it is crucial to freeze the rest of the model including the backbone and the other early exit models (if any) in the collection, to ensure that the training process does not modify any parameter not belonging to the current early exit model.
3.3 Assigning confidence threshold
The probability value associated with the predicted class (the one with the highest probability) is known as the confidence of the model in the prediction. The early exit model's prediction confidence for particular input will indicate whether we stick with that prediction (early exit hit) proceed with the rest of the backbone to the next - probably final - exit (carly exit miss).
Confidence calibration means improving the model to provide accurate confidence. In other words, the confidence a well-calibrated model accurately represents the likelihood for that prediction to be correct et al. (2017)). An overconfident early exit model will lead the model to prematurely exit for some samples based on incorrect predictions, whereas an underconfident early exit model will bear a low early exit hit rate. Therefore, after building early exit model, also calibrate its confidence using MD to better
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Early Exit prediction confusion matrix, Early Exit predicted class, B: Backbone's predicted
class, GT: Ground Truth label
Category
= C
B
GT
GT
BC
v
BC
V
X
X
BC
X
V
X
BC
X
X
V
B
X
X
X
distinguish the predictions that are more likely to be correct. Several confidence calibration methods are discussed in Guo et al. (2017), among which temperature scaling (in the output layer) has been shown to be practical and easy to implement.
Having the model calibrated, we next assign confidence threshold value the model which will be used at inference time to determine the early exit hits and misses. When an early exit model identifies an early exit hit, its prediction is considered the final prediction. However, when needed for validation and test purposes, we obtain the predictions from the early exit model and the backbone.
Confidence calibration involves adjusting the predictive confidence of our early exit models to more accurately reflect their true performance, particularly how likely they are to correct. This is crucial for making reliable decisions during inference, especially when early exits from the model are considered based on these confidence scores. To calibrate the confidence levels of our carly exit models, we employ a threshold that measures confidence based the model output during the validation phase. Specifically, each early exit model functions classifier: during both the validation and testing phases, it generates output that carries an associated confidence value. This confidence value indicates the probability that the model prediction is correct.
An early exit model's prediction (C) for an input to the backbone falls into one of the correctness categories listed in table with respect to the ground truth labels (GT) and the backbone's prediction (B) for the input.
Among the cases where the carly exit model and the backbone disagree, the BC predictions negatively affect the final accuracy, and on the other hand, the BC predictions positively affect the final accuracy. Equation 2 formulates an actual effect of an early exit model on the final accuracy.
FA (0) BC (0)
(2)
Where:
A The early exit model
FA The actual accuracy effect A causes given 0 as threshold
BCA: Ratio of BC predictions by A given 0 as threshold
BCs Ratio of BC predictions by A given 0 as threshold
However, since we used the unlabeled inference data to form the MDs, can only estimate an upper bound for the effect of the early exit model in the final accuracy. The estimation assumes that incorrect early exit would always lead to incorrect classification of the sample (BC'). We estimate the change in the accuracy upper bound an early exit model causes given certain confidence threshold by its hit rate and early exit accuracy:
FA (0) < (R) x (1 (1)(((0
(3)
Where
10

---

Published in Transactions on Machine Learning Research (10/2024)
A
The early exit model
FA The expected accuracy drop A causes given 0 as threshold
HR Hit rate provided by A given 0 as threshold
CAA
Early Exit accuracy provided by given & as threshold
Given the tolerance T for the drop in final accuracy, we assign a confidence threshold to each early exit model that yields more than the expected drop in accuracy T/2"% in MDE according equation 3, where is the 1-based index of the early exit model in the setup.
It is important to note that there are alternative methods to distribute the accuracy drop budget among the early exit models. For example, one can equally distribute the budget. However, as we show in the evaluations later in Section 4.6.1, we find it reasonable to assign more budget to the early exit models shallower positions in the backbonc.
3.4 Evaluation and optimization of the early exit-enabled model
So far, have set of early exit layers and their corresponding early exit models ready for deployment. The algorithm 1 demonstrates pseudoimplemmnnttioon of Python-style inference process for the model enabled for early exit. When the carly exit-enabled model receives batch of samples, it procceds layer-by-layer similarly to the standard forward-pass. Once an early exit layer's activation values are available, it will pass the values to the corresponding early exit model and obtain an early prediction with a confidence value per sample in the batch. For each sample, the corresponding confidence value exceeds the specified threshold, we consider it an early exit hit. Hence, we have the final prediction for the sample without passing it through the rest of the backbone. At this point, the prediction (e.g., API, socket connection, callback). We shrink the batch by discarding the early exit hit items at each exit and proceed with smaller batch to the next (or the final) exit.
be sent to the procedure awaiting the results
Algorithm 1 Early Exit-cnabled model inference
Require: Backbone
The original model List of early exit layers & As part of Backbone, including the associated early exit model and threshold 1: procedure FORWARDPASS(X, callback)
Require: Early Exit Layers
Require: Layer
> X: Input batch In order of presence
2: for Layer in Backbone Bayers do
X <- Layer(XX)
if Layer in Early Exit Layers then Early Exit +- Layer. ExitModel
5:
6:
T + Early Exit. Threshol early exit PDs + Early Exit(X) confidences& + max(early exit PDs, axis=1) callback(carly exit PDs[confidences2 TJ)
7:
8:
Resolve early exit hits Shrink the batch
10:
X + X[confidences<<]] 11: end if
12: end for
13: end procedure
So far in the method, have only evaluated the early exit models individually, but to gain the highest improvement, must also evaluate their collaborative performance within the carly exit-cnabled model. Once the early exit-enabled model is deployed, each early exit model affects the hit rates of the following early exit models by narrowing the set of samples for which they will infer. More specifically, even if an early exit model shows promising hit rate and accuracy in individual evaluation, its performance in the deployment can
The loop show that each early exit model will receive early exit layer's activation values when available, immediately, before proceeding to the layer in the base model.
11

---

Published in Transactions on Machine Learning Research (10/2024)
be affected due to the previous early exit hits made by the carlier early exit models (connected to shallower layers in the backbone). Therefore, need to choose the optimum subset of early exit models to infer the predictions with the minimum computations.
A frute-force approach to find the optimum subset would require evaluating the early exit-enables model with each subset of the early exit models. However, we implement a more efficient method without multiple executions of the early exit-enabled model.
First, for each early exit model, record its prediction per sample in the MDY and their confidence values. We also record two FLOPs counts per early exit model; one is the early exit model's FLOPs count(C1), and the other is the fallback FLOPss count, which denotes the FLOPs in the remaining layers in the backbone(C,, (2). For example, for the layer L12 in the Figure 1, C is the FLOP count of the corresponding early exit model and C2 is the FLOP count in the layers L13 through L16.
For each subset S, we process the lists of prodictions recorded for cach model in S to generate the lists of samples that they actually receive when deployed along with other early exit models in S. The processing consists of keeping only the samples in each list for which there have been early exit hits by the previous carly exit models in the subset. Further, we divide cach list into two parts according to cach carly exit model's confidence threshold; one consisting of the early exit hits, and the other consisting of the early exit misses.
Finally, we score each subset using the processed lists and recorded values for each early exit model S as follows:
K(S) Σ
HAI x (C2,4 - -1,)) |Mal x CI
(4)
DES
Where
K
The early exiting score for subset S
A
An early exit model in S
HA
The generated list of early exit hits for A
MA
The generated list of carly exit misses for A
C1
FLOPs count. recorded for
C2.A
Fallback FLOPs count recorded for A
The score equation accounts for both the improvement an early exit model provides through its early exit hits within the subset and the overhead it produces for its early exit misses.
Using additional early exit layers may cause earlier detection of a class with a higher probability (with the use of an appropriate confidence for early exit layers) before moving to the next layers, so the total accuracy would increase with the cost of memory consumption.
The final schemas after applying the method on MobileFaceNet, EfficientNet, ResNet18, and ResNet50 are discussed in the main text, with detailed illustrations provided in the Appendix (see Figure 6). This figure demonstrates the chosen subsots and their associated carly exit models for cach backbone and data set.
3.5 Early-Exit Optimization Implementation
In this section, we present the implementation and application of early exit models for efficient and timely predictions in three distinct tasks: image classification, object detection, and recommendation systems. We begin by incorporating our proposed early exit approach into architectures widely used for image classification tasks, iee. MobilefaceNett, EfficientNet, ResNet18, and ResNet50, on benchmark datasets, i.e. CIFARIO, CIFAR100, ImageNet and LFW.
Inspired by the promising results obtained in image classification, we further extend our methodology address critical real-world scenario: pedestrian detection in urban environments. For this purpose, we adopt the state-of-the-art Mask R-CNN model, renowned for its exceptional object detection capabilities.
12

---

Published in Transactions on Machine Learning Research (10/2024)
By integrating our early exit strategy into Mask R-CNN, we enable the model to detect pedestrians at an earlier stage during inference, thus significantly reducing the processing time and providing timely warnings to autonomous vehicles about the presence of pedestrians within a given scene.
The significance of our contributions lies in the potential to enhance the safety and responsiveness of au- tonomous vehicles in urban settings, where pedestrian detection plays a pivotal role avoiding accidents and ensuring seamless interaction between vehicles and pedestrians. In our approach, while prioritizing ac- curacy, forgo an important aspect: the exact coordinates of the detected objects. By implementing early exit models triggered by the detection of pedestrians or humans, goal is to achieve faster processing and response times. However, we acknowledge that, in certain cases, reacting within the required time window is of utmost importance. The trade-off between accuracy and response time is a crucial consideration in our methodology, and we recognize the significance of timely actions, especially in scenarios where immediate responses are critical to ensure optimal outcomes.
In the context of the Mask R-CNN model, various options are available to select different backbones and settings, allowing flexibility in performance evaluation and adaptation to specific tasks. While numerous configurations are possible, we opted to utilize publicly available, pre-trained backbone to ensure that our experiments are standardized and well established. This choice allows us to focus on the effectiveness of our proposed approach, taking advantage of the robust and generalization capabilities of the chosen backbone. Additionally, using a pre-trained model helps to mitigate potential biases in training data and enables fair comparisons with other methods that have adopted similar backbones. The final schern of the early exit models for Mask R-CNN with Resnet50 backbone is illustrated in the Appendix (see Figure 7), We extend our early exit classification model to implement pedestrian object detection. The object detection model scans input image and detects multiple objects within the image, assigning each detected object a probability distribution over possible classes.
In object detection, the model's task is to identify various objects within an image and to provide a probability distribution for each detected object. The challenge therein lies in determining an effective method for updating the convolutional dense layer early exits this context. Finally, we explore the application of our early exit strategy in recommendation systems. Specifically, we integrate early exit nodes into the Deep Learning Recommendation Model (DLRM), as suggested by the Microsoft Neural Architecture Search (NAS) algorithm. This integration aims to optimize the inference time while maintaining high recommendation accuracy. The schema of the DLRM architecture with early exits is detailed in Figure 8
3.5.1 Updating Early Exits for Pedestrian Detection
In object detection, pedestrians are one of the most common classes. To optimize the performance of our early exit framework for pedestrian detection, we explore three update strategies for the layers.
Updating with the most confident pedestrians: In this approach, selectively update the layers with features extracted from regions that are confidently classified as pedestrians. By focusing on the most confident detections, we aim to enhance the early exit memory's relevance to crucial features associated with pedestrians in the scene.
Updating with the most confident object: We investigate updating the layers with features from regions classified as the most confident object, regardless of whether is a person or another class. This strategy is designed to ensure that the early exit memory reflects critical features representative of the dominant object class in the scene.
Updating with all detected objects: In this method, we update the layers with features from all delected objects in the scene. While this approach may provide a broader context, it may introduce redundancy and bias towards the more prevalent classes.
After testing these three carly exit-updating approaches, the results supported updating with the most confident single object as the best-performing method. Training early exit layers with a sole focus on individual objects, such as pedestrians, leads non-convergence and lack of meaningful learning. Even
13

---

Published in Transactions on Machine Learning Research (10/2024)
prior to testing, it became evident that training layers exclusively with defined class introduce bias, impeding effective learning. Meanwhile, training early exit layers using diverse set of objects results in model confusion, manifesting itself as reduced accuracy in our testing outcomes. So, our selected approach updates the model with the most certain detection while avoiding the issucs bias and multi-object confusion.
4 Empirical Evaluation
In this section, we explain the objective of our experiment, the research questions, the implementation of the tool and the design of the experiment including the backbones and data sets, the evaluation metrics, and the configuration the environment.
4.1 Objectives and research questions
The high-level objective of this experiment is to assess the ability of the automated layer early exiting mechanism to improve the computation requirements and inference time for DNN-based services.
To address the above objective, we designed the following research questions (RQ):
RQ1
To what extent can early exit models accurately predict the backbone's output and the ground truth data? This RQ investigates the core idea of early exiting mechanism to estimate the final output earlier in the model. The assessments in this RQ consider the early exit models' accuracy in predicting the backbone's output (carly exit accuracy) and predicting the correct labels (GT accuracy).
RQ2 To what extent can early exit-enabling improve computing requirements? In this RQ, we are in- terested in how early exit-enabling affects the models' computation requirements. In these mea- surements, measure the FLOPs counts and memory usage as metrics for the models' compute consumption.
RQ3 How much acceleration does early exit enable on the CPU / GPU? In this RQ, we are interested the actual amount of end-to-end speedup that an early exit enabled model can achieve. We break this result down CPU and GPU accelerations, since they address different types of computation during the inference phase and thus may have been differently affected.
RQ4 How does the carly exit-cnabled model's accuracy /latency trade-off compare with other carly exit methods? In this research question, our aim to assess and compare the performance of your carly exit-enabled model against other existing early exit methods regarding the trade-off between accuracy and latency in practical, real-world scenarios.
4.2 Tasks and datasets
Among the diverse set of real-world classification tasks that are implemented by solutions using DNN mod- els, we have selected two representatives: face recognition and object classification. Both tasks are quite commonly addressed by DNNs and often used in large-scale services that have non-functional requirements such as: high throughput (due to the nature of the service and the large volume of input data) and are time-sensitive.
Face recognition models are originally trained on larger datasets such as MS-Celeb-IM (Guo et al., 2016) and are usually tested with different and smaller - datasets such as LFW (Huang et al., 2008), CPLFW (Zheng et al., 2017), RFW (Wang et al., 2019), AgeDB30 (Moschoglou et al., 2017), and MegaFace (Kemelmacher- Shlizerman et al., 2016) to test the models against specific challenges, such as age / ethnic biases and recognizing mask-covered faces.
We used the labelod face in the wild (LFW) dataset for face recognition, which contains 13,233 images of 5,749 individual.. We used images from 127 identities that have at least 11 images in the so we can split them for training, validation, and testing.
14

---

Published in Transactions on Machine Learning Research (10/2024)
We also used CIFAR10, CIFAR100, and ImageNet test sets (Krizhevsky & Hinton, 2009; Russakovsky al., 2015) for object classification, each containing 10,000 images distributed equally among 10 and 100 classes for CIFARs and 100,000 images among 1,000 classes for ImageNet. CIFARs have a size of 32 x 32 pixels, while the ImageNet data set originally contains images of different sizes. To standardize the input size for ImageNet, we used a common method of resizing the images to 256 256 pixels and then cropping them to 224 x 224 pixels.
Reminder that we do not use the training data, rather we only use the test sets to simulate incoming queries at run-time. Specifically, we used only the test splits of the CIFARs and ImageNet datasets. However, we used the entire LFW data as it has not been used to train the face recognition models. Morcover, we do not use the labels in these test sets in the training and optimization process, rather we only use them in the evaluation step to provide GT accuracy statistics.
Each data set mentioned above represents an inference workload for the models. Thus, we split cach one into training, validation and test partitions with 50%, 20%, and 30% proportionality, respectively. However, we augmented the test sets using flips and rotations to improve the statistical significance of our testing measurements.
We used the CityScape dataset to assess the presence of pedestrians ((Cordts et al., 2016)). This data set is valuable for our research on the comprehension of urban scenes, as it offers meticulously annotated images, with pixel-level labels, depicting various urban environments from the perspective of a vehicle. To evaluate the accuracy of our model, we needed labeled data for testing purposes. Upon observing that the test dataset within the CityScape dataset contained dummy labels, we opted to utilize the validation subset instead.
To demonstrate the applicability of our early exit model on other nonimage-based tasks, we integrated it into recommendation system. The data set we use is from the Criteo data set on Kaggle Brean-Baptiste Tien, 2014), used for the Display Advertising Challenge, which is comprehensive resource designed to benchmark click-through rate (CTR) prediction algorithms. It includes data collected over a seven-day period, consisting of Teature values and click feedback for millions of display ads. The datasel divided into training and testing parts, with each part containing both clicked and nonelicked examples that have been sub-sampled to manage dataset size. The data set consists of 39 features per sample: 13 integer featuros, primarily count data, and 26 hashed categorical features for anonymization. The details of these features are not disclosed and may include missing values. This application can show the ability of the model to significantly improve inference speed while managing large-scale data structures.
4.3 Backbones and Models
The proposed early exit enabler method is applicable to any deep classifier model. However, the results will vary for different models depending on their complexity.
Among the available face recognition models, we have chosen the well-known MobileFaceNet and EfficientNet models to evaluate the method, and we experiment with 18 and ResNet50 for object classification.
The object classification models are typical classifier models out-of-the-box. However, face recognition models are feature extractors that provide embedding vectors for each image based on the face / location features. They can still be used to classify face-identity dataset. Therefore, we attached a classifier block to these models and trained them (with the feature extractor layers frozen) to classify the images of the 127 identities with the highest number of images in the LFW dataset (above 10). It is important to note that since the added classifier block is part of the pre-trained trained model under study, we discarded the data portion used to train the classifier block to ensure we still hold on to the constraint of working with pre-trained models without access to the original training dataset.
As stated previously, our pedestrian detection approach required the selection of an object detection technique capable of identifying pedestrians within images. We adopted the Mask R-CNN framework. This method encompasses backbone component (for which we employed ResNet50) and two additional sections that consume significant. time and memory resources.
15

---

Published in Transactions on Machine Learning Research (10/2024)
However, for our specific use case of providing early warnings to autonomous vehicles regarding the presence of pedestrians, the precise localization of pedestrians is not, essential. Consequently, we chose to disregard the other resource-intensive sections, resulting in substantial time savings while still achieving the necessary level of awareness of pedestrian presence.
The Deep Learning Recommendation Model (DLRM) (Naumov et al., 2019) is a neural network architecture designed for use in recommendation systems. efficiently handles categorical and numerical data, making it highly effective for personalized recommendation tasks. DLRM uses combination of embedding tables for categorical features and multi-layer perceptrons (MLPs) for mumerical features. These components are then interacted with through specialized dot product interaction operation, which enables the model to learn and predict complex patterns from user-item interactions.
4.4 Metrics and measurements
Our evaluation metrics for RQ1 are ground truth (GT) accuracy and early exit accuracy. Early exit accuracy mcasures how accurately an carly exit model prodicts the backbone's output (regardless correctness). The GT accuracy applies to both the early exit-enabled model and each individual early exit model. However, early exit accuracy only applies to early exit models. Individual early-exit layers' accuracies might be different from the final early-exit-enabled model's accuracy. This discrepancy arises because layer-wise accuracies are calculated only for the cases where the input is classified at that specific layer (hit cases), which typically results in higher accuracies. For example, of 5000 inpuls, a particular layer may serve as an early exit for only 5 inputs at a confidence level 99%. Although this layer may exhibit high accuracy in these cases, it would have only a slight impact on the overall final accuracy of the model.
In RQ2, we compare the original models and their early exit-enabled version in terms of the average FLOP count occurring for inference and their memory usage. We only measure the resources used in the inference. Specifically, we exclude the training-specifi layers (e.g. batch normalization and dropout) and computations (e.g- gradient operations) in the analysis.
The FLOP count takes into account the model architecture and input size and estimates the computations required by the model to infer the input (Desislavov et al., 2021). In other words, the fewer FLOPs used for inference, the more efficient the model is in terms of compute and energy consumption.
On the other hand, we report two aspocts of memory usage for the models. The first is the total space used to load the models in the memory (i.e, model size). This metric is essentially agnostic to the performance of early exit models and only considers the memory cost of loading them along with the backbone.
In addition to the memory required for their weights, DNNs also allocate a sizcable amount of temporary memory for buffers (also referred to as tensors) that correspond to intermediate results produced during the evaluation of the DNN's layers (Levental, 2022). Therefore, our second metric is the live tensor memory allocations (LTMAs) during inference. LTMA measures the total memory allocated to load, move, and transform the imput tensor through the model's layers to form the output tensor while executing the model.
In RQ3, we compare the average inference latency of the original model with its early exit-enabled counter- part. Inference latency measures the time spent from passing the input to the model till it exits the model (by either an early-exit or the final classifier in the backbone). Various factors affect inference latency including hardware-specific optimizations (e.g., asynchronous computation), framework, and model implementation. In our measurements, the framework and model implementations are fixed as discussed in the Appendix A.1. However, to account for other factors, we repeat each measurement 100 times and report the average inference latency recorded for each experiment. Further, to also account for the effects of asynchronous computations in GPU inference latency, we repeated experiments with different batch sizes.
Please refer to Appendix A for details of implementation and setup.
RQ4 aims to evaluate the trado-offs between accuracy, latency, and computational cost of an carly exit- enabled model compared to other early exit methods. This inquiry crucial to understanding the efficiency and iveness of different approaches.
16

---

Published in Transactions on Machine Learning Research (10/2024)
To address this question, we have selected two prominent and casy-to-implement early exit methods for comparison. These methods are evaluated using classification tasks in ResNet18, ResNet50, and the CTFAR datasets.
4.5 Baselines
In this subsection, we discuss the baselines used to evaluate the proposed unsupervised early exit method. Specifically, we consider two prominent methods: BranchyNet (Teerapittayanon et al., 2016) and GATI (Balasubramanian et al., 2021), both of which aim to reduce the latency of deep neural network (DNN) inference while maintaining high accuracy. These basclines typically require access to labeled training data, which contrasts with our unsupervised approach.
BranchyNet is designed to improve the efficiency of DNN inference by allowing certain samples to exit the network early through additional side branch classifiers. These branches are strategically placed at various depths in the network to enable predictions at different levels of abstraction. During the training phase, BranchyNet jointly optimizes the loss functions of both the main branch and the side branches, effectively regularizing the network and mitigating vanishing gradients. The inference phase utilizes entropy- based thresholds to determine whether sample should exit carly or continue through the deeper layers of the network. In our implementation of BranchyNet as a baseline, we maintained the original supervised setup, but we adapted the training process ensure fair comparison. Specifically, we used the same data for training the early exit layers across all methods, which was portion of the test data set (the other half was not used for evaluation). Furthermore, we froze the base model during the training of the side branches, a crucial difference from the original BranchyNet, which does not freeze the base model. For eva evaluation, we applied BranchyNet to the remaining half of the test data set reserved for evaluation, allowing us to directly compare its performance against our unsupervised method.
GATI introduces a learned caching mechanism to accelerate DNN inference by exploiting temporal locality in prediction-serving workloads. GATT caches the hidden layer output of the DNN, enabling faster predictions for inputs that exhibit characteristics similar to previously processed data. The system employs simple machine learning models as learned caches, which are continuously updated based on incoming data. Unlike BranchyNet, GATI does not modify the DNN architecture but instead optimizes the inference process by dynamically skipping layers when a cache hit is detected. In our evaluation of GATI, we used a pre-trained base model and trained the cache layers using the same portion of the test dataset as other methods, ensuring consistent data usage. Similarly to BranchyNet, we froze the base model during the training of the caches, differing from GATI's original approach, which does not freeze the base model. While both baselines rely on supervised training, our method is designed to function in an unsupervised manner, particularly in scenarios where a base pre-trained model is accessible, but labeled data may not be available. This flexibility allows our approach to be deployed in contexts where labeled data is scarce or unavailable, leveraging the test set itself for training early exits. To ensure a fair comparison with our method, we adapted both baselines to this framework by using the same data for training and freezing the base model during the training of carly-exit layers, which aligns with our approach. However, is important to note that these basclines do not inherently support unsupervised training, which key contribution of our work. An important point to note is that our Base model represents the upper bound of our work, as it is built on pre-trained model. Our unsupervised method aims to make early-exit results resemble the Base model's predictions rather than converging to the ground truth labels. Consequently, our upper bound is consistently the Base model, while other methods may exhibit varying test accuracies due to additional training and reliance on labeled data. However, this does not necessarily apply to other baselines, such as BranchyNet and GATI, as they may use different approaches for training. Although our Base model demonstrates the best possible performance within the constraints of our approach, the same cannot be directly inferred for the other basclines, which may not achieve the same upper bound due to their reliance on labeled training data and other training techniques.
This is our summarized evaluation process:
We use pretrained models, typically trained with both training and valida- tion data. After pretraining, we integrate our layers (carly exits) into the models. Initially, these new blocks and layers are not trained.
Pretraining Models:
17

---

Published in Transactions on Machine Learning Research (10/2024)
Test Data Splitting: For testing our method, split the test data set into two separate parts. We then freeze the main model and update the early exit layers by inferring on the first portion of the test data, During this phase, the main model weights are not updated, only the new early cxit layers are trained. Importantly, we do not use the labels from the test data for training or optimization, thus avoiding overfitting concerns.
Performance Evaluation: After training the carly exit layers, we test the performance of the entire model using the remaining part of the test dataset. This approach may result in a slight loss of accuracy, but it significantly reduces the total inference time due to the early exits.
4.6 Experiment results
In this sub-section, we evaluate the results of applying the mothod on the baseline backbones and discuss the answers lo the RQs. For a more comprehensive understanding, further details about our implementation, including access the code repository, are provided in the Appendix A.
4.6.1 RQ1. How accurate are early exit models in predicting the backbone output and the ground truth labels?
In this RQ, we are interested in the performance of the built early exit models in terms of their hit rate, GT accuracy, and early exit accuracy. We break down the measurements into two parts. The first part covers the individual performance of the early exit models over the whole test set without any other early exit model involved. The second part covers their collaborative performance within the carly exit-enabled model.
4.6.2 Early Exit models' individual performance
Figure 2 shows the individual performance of each early exit model against any confidence threshold value in the C1Fse100- experiment. Figures demonstrating the same measurements for other experiments are available in the Appendix C.
We make three key observations here. First, deeper early exit models are more confident and accurate in their predictions. For example, early exit 1 in Figure 2 has 33.36% GT accuracy and 35. 75.74% early exit accuracy, while these metrics increase to 78.60% and 95.38% for carly exit 3, respectivcly. This observation agrees with the generally acknowledgen feature extraction pattern in the DNNs deeper layers convey more detailed information.
The second key observation is the inverse correlation between the carly-cxit models' accuracy (both GT and early-exit) and their hit rates. This observation highlights the reliability of confidence thresholds distinguishing predictions that are more likely to be correct. For example, early exit in Figure 2, with confidence threshold 20%, produces hit rate of 35.24% but also a drop of 8.99% in the final accuracy. However, with confidence threshold 60%, it produces hit rate 4% and does not reduce the final accuracy by more than 0.1%.
The third observation is that the early exit accuracy is higher than the GT accuracy in all cases. This difference because we have trained the early exit models to mimic the backbone only by observing its activation values the intermediate layers and outputs. Since we have not assumed access to the GT labels (which the case for inference data collected run-time) while training the early exit models, they have learned to make correct predictions only through predicting the backbone's output, which might have been incorrect in the first place. On the other hand, observed that the early exit models predict the correct labels for a portion of the samples for which the backbone misclassifies. For example, for 0.92% of the samples, early exit. 3 (in Figure 2) correctly predicted the GT labels while the backbone failed (BC predictions). This shows the potential of the early exit models to partially compensate for their incorrect early exits (BC' predictions) by correcting the backbone predictions for some samples (BC). This agroes with the overthinking concept SDN (as discussed in 2.3), since for this set of samples, the early exit models have been able to predict correctly in the shallower layers of the backbone.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Cifar100-Resneet
Cache 1
Cache
Cache
Cache
100
80
60
20
0
100
%
40
80
60
20
25
50
75
100
25
50
100
25
75
100
25
50
75
100
Confidence
Confidence
Individual accuracy and hit rate of the early exit models vs. confidence threshold per early exit model in CIFAR100 Resnet50 experiment
Figure 2:
4.6.3 Early Exit models' collaborative performance
Table describes the collaborative performance of the early exit models within a confidence threshold of 0.9. In the table, we also report how each early exit model's layer hits have affected the final accuracy.
Here, we observe that while evaluating the early exit models on the subset of samples, which were missod by the previous early exit models (the relatively more complex ones), the measured hit rate and GT accuracy are substantially lower compared to the evaluation the whole dataset. This is in fact due to the fact that the simpler samples (less detailed and easier classify) the hit rate decreases since the carly exit models are less confident: in their prediction for the more complex samples, and the GT accuracy also decreases since the backbone is also less accurate for such samples. However, we observe that the early exit models still have high early exit accuracy, with a low impact on the overall accuracy. This observation shows how the confidenco-based carly exiting method has effectively enabled the early exit models to provide early predictions and keep the overall accuracy drop within the given tolerance.
resolved earlier in the model. More specifically,
19

---

Published in Transactions on Machine Learning Research (10/2024)
Table Early Exit models' collaborative performance in terms of hit rate(HR),, early exit accuracy (Avarty exit), GT accuracy (Act); and their effect on the final accuracy(AAattec) in the early exit-enabled model with a confidence threshold of 0.9, LFW: Labeled Faces in the Wild, MFN: MobilefaceNet, EFN: EfficientNet
Final accuracy
Data
Model
Exit#
HR
Aearly exit
ACT
1 Aeffect
Base
Early Exit-enabled
Resnet50 Resnet
1
67.21%
92.29%
88.91%
01.31%
2
10.33%
89.76%
76.63%
0.56%
88.71%
86.49%
CIFAR10
11.24%
85.71%
51.43%
0.25%
8.32%
91.37%
35.71%
0.1 %
1
61.41%
89.12%
86.19%
1.12%
2
15.73%
93.01%
77.84%
0.58%
87.92%
85.88%
10.29% 3
82.22%
53.33%
0.3%
6.1% 4
97.47%
42.65%
0.04%
Resnet50 Resnet18
1
11.96%
99.29%
82.11%
0.94%
2
58.26%
99.62%
85.41%
0.1%
75.92%
74.47%
CIFAR100
3
7.26 %
93.81%
59.29%
0.3%
5.36% 4
55.56%
38.89%
0.11%
11.92% 1
76.34%
80.2%
1.32%
61.98% 2
98.56%
84.55%
0.34%
78.98%
77.04%
3
11.5%
97.85%
63.69%
0.27%
7.38% 4
73.68%
52.63%
0.1%
Mask RONN EFN MFN Resnet50 Resnet18
1
8.21%
93.55%
76.42%
0.59%
14.09% 2
93.24%
84.23%
1.1%
69.76%
68.12%
ImageNet
38.13% 3
83.53%
88.76%
1.13%
8.78% 1
75.23%
79.72%
0.61%
9.13%
91.41%
92.19%
0.48%
2
13.89%
84.78%
81.77%
0.89%
76.13%
74.09%
42.65% 3
78.58%
72.03%
1.37%
3.14% 4
73.12%
71.12%
0.19%
37.35%
98.63%
97.88 97.88%
0.51%
97.78%
96.91%
2
41.02%
99.71%
99.71%
0%
LFW
3
55.95%
93.44%
96.18%
0.24%
63.73%
96.82%
96.24%
1.67%
97.29%
95.35%
2
14.52%
99.12%
98.76%
0.02%
34.3% 1
58.3%
57.9%
0.1%
CityScape
91.0%
83.4%
2
36.34%
79.2%
79.1%
0.24%
3
21.12%
87.31%
86.16%
0.81%
12.3%
74.3%
70.9%
0.4%
Criteo
DLRM
78.88%
73.54%
2
14.3%
69.5%
69.3%
0.23%
3
25.41%
94.31%
90.70%
0.51%
Summary for RQ1
Early Exit models show lower hit rates and GT accuracy for complex samples, but maintain overall accuracy thanks to the effective use of confidence-batt based early exiting method.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Original and early exit-enabled models FLOPs (M-Mega 10%)
FLOPs Early Exit-enabled (0.9 Confidence)
Dataset(inpin size)
Model
+ Ratio
4 Accuracy
Original
18
765M
414M
45.88%
2.51%
CIFARI0(3 x 32 x 32)
Resnet50
1303M
601M
53.87%
2.32%
Resnet18
766M
374M
51.17%
1.91%
CIFAR100(3 x 32 x 32)
Resnet50
1304M
547M
58.05%
2.46%
Resnet18
2343M
1673M
28.6%
2.35%
ImageNet(3 x 224 x 224)
Resnet50
2783M
2020M
27.4%
2.68%
MobileFaceNet
474M
296M
37.55%
0.91%
LFW(3 x 112 x 112)
EfficientNet
272M
182M
33.08%
1.99%
CityScape(3 x 2048 1024)
Mask R-CNN
4950M
2730M
44.84%
8.61%
(39)
DLRM
153M
99M
35.29%
6.52%
Table 4: Original and early exit-enabled models memory usage
Original
Early Exit-enabled
Dataset(input size)
Model
(0.9 Confidence)
JAccuracy
Model Size
LTMA
Model Size
LTMA
Resnet18
43MB
102MB
97MB
88MB
2.51%
CIFAR10(3 x 32 x 32)
Resnet50
91MB
235MB
243MB
201MB
2.32%
Resnet18
43MB
104MB
383MB
93MB
1.91%
CIFAR100(3 x 32 x 32)
Resnet50
91MB
235MB
552MB
189MB
2.46%
Resnet18
43MB
110MB
403MB
101MB
2.35%
ImageNet(3 x 224 x 224)
Resnet50
91MB
237MB
572MB
198MB
2.68%
MobileFaceNet
286MB
567MB
350MB
515MB
0.91%
LFW(3 x 112 x 112)
Efficient.Net
147MB
371MB
297MB
349MB
1.99%
CityScape(3 2048 1024) C39)
Mask R-CNN
3680MB
3925MB
4171MB
4216MB
8.61%
DLRM
320MB
450MB
332MB
411MB
6.52%
4.6.4 RQ2. To what extent can early exit-enabling improve compute requirements?
In this RQ, we showcase the amount of computation early exiting can save in terms of FLOPs count and analyze the memory usage of the models, and quantify the accuracy drop incurred during this operation.
Table 3 shows the average amount of FLOPs computed for inference per sample. Here we observe that shrinking the batches proportionally decreases the FLOPs count required for inference.
In addition, Table shows the memory used to load the models (i.e, the model size) and the total LTMA during inference while inferring for the test set. As expected, the size of the early exit-enabled models is larger than the original model in all cases, since they include the backbone and the additional early exit models. However, the decrease in LTMA in all cases shows reduced number of memory allocations during the inference time. Generally, a lower LTMA indicates smaller tensor dimensions (e.g., batch size, input, and operator dimensions) (Ren et al., 2021). However, in our case, since we do not change cither of the dimensions, the lower LTMA is due to avoiding the computations in the remaining layers after early exit hits which require further memory allocations. A noteworthy observation from this table highlights the substantial memory usage of our object detection approach due to the sizcable model employed. This underscores the notion that implementing carly exiting for this purpose does not significantly amplify the memory requirements.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: end-to-end evaluation of early exit-enabled models improvement average inference latency, batch
size 32, MFN: MobileFaceVet, EFN: EfficientNect
Original latency
Early Exit-enabled latency
4 Ratio
Dataset
Model
CPU
GPU
CPU
GPU
CPU
GPU
R8snet
13.4 ms
1.08 ms
10.11 ms
0.98 ms
24.55%
10.2%
CIFARIO
Resnet50
18.73 ms
1.81 ms
14.62 ms
1.51 ms
31.08%
16.57%
Resnet18
14.23 ms
1.39 ms
9.39 ms
1.25 ms
34.01%%
10.08%
CIFAR100
Resnet50
19.59 ms
2.05 ms
9.02 ms
1.84 ms
46.08%
16.75%
Resnet
38.19 ms
3.26 ms
30.23 ms
2.79 ms
20.84%
14.42%
ImageNet
Resnet50
47.21 ms
3.49 ms
38.74 ms
2.42 ms
17.94%
30.65%
MFN
25.34 ms
8.22 ms
16.91 ms
7.30 ms
33.23%
11.19%
LFW
EFN
39.41 ms
17.63 ms
27.98 ms
14.38 ms
29.01%
18.44%
CityScape
Mask R-CNN
895 ms
145.2 ms
562.3 ms
108.7 ms
45.12%
35.32%
Critco
DLRM
9.0 ms
1.7 ms
7.67 ms
1.38 ms
14.7%
18.82%
Although the FLOPs count and memory usage indicate the model'ss inference computational requirements, the decreased amount of FLOPs and LTMA does not necessarily lead to proportional reduction in the models' inference latency, which we further investigate in the next RQ.
Summary for RQ2
Early exit-enabled models are larger due additional early exit layers but reduce memory use during inference by avoiding computations after early exit hits, demonstrating efficient memory management without significantly incrcasing overall memory requirements.
4.6.5 RQ3. How much acceleration does early exit enablement provide on the CPU/GPU?
In this RQ, we investigate the end-to-end improvement that early exit enablement offers. The results of this measurement clearly depend on multiple deployment factors, such as the underlying hardware and framework, and, as discuss later in the section, their asynchronous computation capabilities.
Table (5) shows the average latency for the base models on CPU and GPU, vs. their early exil-enabled counterparts, evaluated on the test set.
The first key observation here is the improvements on the CPU. This improvement is due to the low paral- lelism in the CPU architecture. Easentially, the CPU computation volume proportional to the number of samples. Therefore, when a sample has an early exit, the remaining computation required complete the tasks for the batch decreascs proportionally.
The FLOPs of all our early exit models, utilized by each input in the batch, are aggregated, resulting in latency being dictated by the longest time observed in the batch. Consequently, a larger batch size tends to worsen latency, as it may lead to more frequent early exit misses.
The second observation is the relatively lower latency improvement the GPU. This observation shows that shrinking a batch does not proportionally reduce the inference time on GPU, which is due to the high parallelism in the hardware. Shrinking the batch GPU provides certain overhead since it interrupts the on-chip parallelism and hardware optimizations. This interruption forces the hardware to re-plan its computations, which can be time consuming. Thus, batch-scanning improvements can be insignificant on GPU. The third observation pertains to the time savings related to pedestrian detection, in contrast to the primary model. This significant, gain in efficiency is attributed to disregarding the additional layers of Mask R-CNN through our early exit strategy.
Table further demonstrates how the batch size affects the improvement provided by early exiting. The key observation here is that increasing the batch size negate the early exiting effect the inference latency,
22

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Inference latency improvement on GPU vs. batch size in Resnet18 and Resnet50 trained on
CIFAR100
Model
Batch Size
Original Latency
Early Exit-cnabled (0.9 Confidence) Latency
4
Ratio
16
1.34 ms
1.18 ms
11.83%
32
1.39 ms
1.25 ms
10.08%
18
64
1.43 ms
1.77 ms
-24.28%
128
1.61 ms
2.11 ms
-31.05%
16
1.98 ms
1.71 ms
13.68%
32
2.05 ms
1.84 ms
16.75%
5e
64
2.19 ms
1.98 ms
9.21%
128
2.7 ms
3.22 ms
-19.43%
which, as discussed, is due to fewer batches that are fully resolved through the early exit models and do not reach the last layers. In conclusion, the lateney improvement here highly depends the hardware used in inference and must be specifically analyzed per hardware environment and computation parameters such batch size. However, the method can still be useful when the model is not performing batch inferences (batch size 1). One can also use the tool and get the best prediction so far within the forward-pass process by disa disabling batch shrinking. Doing so will generate multiple predictions per input sample, one per exit (carly and final).
Summary for RQ3
Increasing batch size in early exit models can worsen latency due to more frequent early exit misses and does not proportionally reduce inference time on GPUs due to the disruption of hardware parallelism. particular, significant time savings are achieved in pedestrian detection by bypassing extra layers with an early exit strategy, although the benefits of carly exiting on latency heavily depend on the specific hardware and batch size used.
4.6.6 RQ4. How does the early exit-enabled model accuracy / latency and computational cost trade-off off compare with other early exit methods?
In this RQ, we conducted comprehensive evaluation of three distinct methods, including BranchyNet (Teer- apittayanon et al., 2016), GATI (Balasubramanian et al., 2021), and our proposed method in the same system configuration. This evaluation included two different Resnets with two different data sets to thoroughly assess the performance of each method.
Figures 3 and demonstrate comprehensive analysis of the accuracy latency / computational cost trade- off at various confidence levels for all methods evaluated for CIFAR10 and CIFAR100. Our findings reveal a remarkable adaptability of our proposed method to different confidence levels, particularly excelling in low- confidence scenarios. This adaptability underscores the effectiveness of our approach's 's training, showcasing its ability to perform exceptionally well in situations where traditional methods might falter. In contrast, as confidence levels increase, our method exhibits a slight latency increase, reflecting its ability to fine- tune confidence settings suit various application requirements. This fcature demonstrates the method's adaptability and its potential for accommodating diverse use cases.
Table 7 presents the comparison of inference costs and accuracy across different methods (Branchy.Net, GATI, and SelfXit using ResNet18 and ResNet50 models on CIFAR-10 and CIFAR-100 datascts. The table reports accuracy at various inference cost. thresholds (525%, <50%, and <75%) relative to the original model's inference cost. The 'Max' column indicates the highest accuracy achieved by each method with early exits. This comparison highlights the trade-offs between inference cost and accuracy for the evaluated methods. This table demonstrates that our method has a better impact for more complex base models and datasets.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Hesnul-50
Our Nahhod BrancryNie
10
BrunchyM4
dar
BrunchyMd
306
Figure 3: CIFAR10 Deuurao/laaeonc//coptttaati cost comparison between the approaches with different confidence.
Table 7: Comparing the inference costs of Resnet18 and Resnet50 models on CIFAR-10 and CIFAR-100 on different methods. <25%, <50,, and <75% report the early exit accuracy when we limit the average inference cost to at most 25%, 50%, and 75% that the original models each method. Max reports the highest accuracy carly exits can achieve.
Method
<25%
<50%
375%
Max
CIFAR-IE 10
ResNet18
BranchyNet
86.4
87.7
91.54
95.5
GATI
69.8
77.4
85.7
91.6
SelfXit
81.6
82.8
83.5
89.4
CIFAR-10 10
ResNet55
Branchy Net.
82.0
84.3
88.4
90.1
GATI
74.5
80.3
86.0
88.5
SelfXit
80.7
81.5
83.7
86.6
CIFAR-100
18
Branchy Net
70.8
72.8
73.2
77.4
GATI
71.5
72.5
74.7
76.4
SelfXit
62.2
63.0
67.8
77.5
CIFAR-100
ResNet50
Branchy Net.
65.7
69.5
72.3
74.1
GATI
66.2
66.8
69.7
73.3
SelfXit 67.6
70.0
73.9
79.4
The primary observation is that our method exhibits the best latencies in various methods at different confidence levels. Another significant observation that our method's performance seales positively with the complexity of the model dataset, leading superior outcomes terms of both accuracy and latency. As add some lightweight layers for each model, it becomes evident that Resnet50 exhibits more pronounced
24

---

Published in Transactions on Machine Learning Research (10/2024)
Nesnul-50
Our Nathod BranctyNe
50
80
60
BrunchyMi
dan
Mathol BrunchyMd
306
Figure 4: CIFAR100 Meurayy/atencr//ooppttinn cost comparison between the approaches with differ- ent confidence.
improvements in our research. This observation is also shown in Figure 3. By averaging across all confidence levels greater than 0.25, For the simplest model and datasct, namely CIFAR10-Resnet1, substantial 52.2% and 32.4% reduction in latencies, with modest 6.9% and 1.6% decrease in accuracy when compared to the BranchyNet and GATI methods, respectively, while improving computational costs by 7.1% and 0.5%. In the case of more complex model and dataset, such as LALARI00-RReseee we achieved a substantial reduction in latency of 51.6% and 30.4% while simultancously improving the accuracy by 2.31% and 0.72% compared to the GATT and BranchyNet methods, respectively, while improving compulation ional costs by 14.3% and 8.2%. BranchyNet, on the other hand, exhibits relatively consistent results at different confidence levels. Although this stability might be desirable in scenarios, it lacks the adaptability and dynamic response that our method offers. Gati, while showing learned early exits, is marked by the complexity of its training process, which can be challenging to implement effectively. Moreover, GATT consumees more memory resources during implementation, in contrast to the efficiency of SelfXit
achieved
Indeed, our method stands out as a dynamic and adaptable solution, requiring less extensive data and model preparation compared to approaches such as GATI. By completely freezing the base model during the training of early-exit layers, we achieve better results within limited training time, particularly in more complex models where other methods may require significantly more time to train effectively. The ability to configure confidence levels further underscores its versatility for broad spectrum of applications. However, it should be noted that for simpler models and datascts, our method may not demonstrate a marked improvement over others, as they employ additional dense layers for early exiling, which can provide competitive results. Crucially, the most significant aspect of our work lies in the capability to update the early exit layers effectively without reliance on ground-truth labels, aligning carly-exit results closoly with the base modcl's predictions. This offers substantial advantage practical applications where such labels may not be readily available.
25

---

Published in Transactions on Machine Learning Research (10/2024)
Summary for RQ4
Our method demonstrates exceptional adaptability across different confidence levels, particularly excelling in low-confidence scenarios, which highlights its robust training and ability to fine-tune settings for varied applications. It achieves the best latencies and scales positively with the complexity of the model or dataset, showing substantial improvements in both accuracy and computational costs, particularly with complex configurations like C1FAR100-Resnet50
To evaluate the impact of implementing NAS and varying model depths, we conducted an ablation study, detailed in Appendix D.
4.7 Discussion, Limitations, and Future Directions
The first limitation of this study is that the proposed method is limited to classification models, since it would be more complicated for early exit models to predict the output of regression model due to their continuous values. This limitation is strongly tied to the effectiveness of knowledge distillation in the case of regression models.
The method also does not take the internal state of the backbone (if any) into account, such as the hidden states in recurrent neural networks. Therefore, the effectiveness of the method for such models still needs to be further assessed.
Moreover, practitioners should take the underlying hardware and the backbone structure into account, they directly affect the final performance. On this note, as shown in Section 4.6.5, different models provide different performances in terms of inference latency in the first place; therefore, choosing the right model for the task comes first, and early exiting can be helpful in improving the performance.
As the main goal our proposed approach to decrease the inference time, depending on the application domain, if potential slight degradation of accuracy not acceptable (e.g., a safety critical system), using our approach might not be good fit.
In dynamic environments where data distribution changes over time, maintaining the performance of early exit models may require periodic updates. Unlike conventional caching, early exit models cannot. be up- dated in real-time. Therefore, retraining the early exit layers and readjusting confidence thresholds using newly collected inference samples bocome necessary to adapt to new data trends and maintain accuracy. While we did not perform these updates in our experiments, practitioners deploying this method should consider triggers for updates, such as when a significant portion of new data has been collected or when the backbone model modified or retrained. Balancing the costs associated with retraining against application requirements and resources is essential for effective maintenance.
While our current comparison has yielded valuable results, we can explore the applicability of our approach to other large models, particularly in non-vision- datasets, to assess its effectiveness in different domains. Given the growing importance of reducing latency and inference time, especially in Large Language Models (LLMs), our future rescarch can focus on methods further optimize and reduce costs for largo-scale industries.
5 Conclusion
In this paper, we have shown that our automated early exiting approach is able to extend pretrained classification DNN to early exit-enabled version using relatively small and unlabeled dataset. The required training data sets for early exiting models are collected just by recording the input items and their corresponding backbone outputs at the inference time. We have also shown that the early exiting method can introduce a significant. improvement in the model's computing requirements and inference latency, especially when the inference is performed on the CPU.
We discussed the parameters, design choices, and the procedure of early exit-enabling pro-trained off aff-the- shelf model.
26

---

Published in Transactions on Machine Learning Research (10/2024)
In conclusion, while traditional early exiting might not be beneficial for DNN models due to the diversity, size, and dimensions of the inputs, early exiting the features in the hidden layers of the DNNs using the early exit models can achieve significant improvement in the model's inference computational complexity and latency. As shown in sections 4.6.4 and 4.6.5, carly exiting reduces the average inference FLOPs by up to 58% and the latency by up to 46.09% on CPU and 18.44% on GPU for classification purposes. For pedestrian detection, we could reduce latency up to 45.1% on CPU and 35.32% on GPU. summary, SelfXit consistently outperforms alternative approaches in different models and datasets. Considering average values across all confidence levels greater than 0.25, in the case of the simplest model, (IFAR1O-Resnetha remarkable reduction of 52.2% and 32.4% in latency, with accuracy only experiencing minor decrease of 6.9% and 1.6% compared to the BranchyNet and GATI methods, respectively. For the more intricate CIFARI00-Resnee model and dataset, our method excels with a significant 51.6% and 30.4% reduction in latency, and a notable 2.31% and 0.72% improvement accuracy compared to the BranchyNet and GATI methods. These findings underscore the adaptability and superior performance of our method in diverse and complex scenarios.
observe
Acknowledgments
This work was partially supported by NSERC (ALLRP/568643-2 Alberta Innovates (212200865) Al- liance Grants, and National Science Foundation (Awards 2007202, 2107463, 2038080, and 2233873), and the Natural Sciences and Engincering Research Council of Canada (NSERC) Discovery Grant Program (IGPIN--021-03954).
References
Riku Arakawa, Shinnosuke Takamichi, and Hiroshi Saruwatari. Implementation of dnn-based real-time voice conversion and its improvements by audio data augmentation and mask-shaped device. 10th ISCA Workshop on Speech Synthesis (SSW 10), 2019.
Arjun Balasubramanian, Adarsh Kumar, Yuhan Liu, Han K. Cao, Shivaram Venkataraman, and Aditya Akella. Accelerating deep learning inference via learned caches. ArXiv, abs/2101.07344, 2021.
Vinayak Ashok Bharadi, Arusa Irfan Mukadam, Misbah N Panchbhai, and Nikita N Rode. Image classifi- cation using deep learning. International journal of engineering research and technology, 6, 2017.
Nandita Bhaskhar. Intermediate activations - the forward hook. Blog: Roots of my Equa- tion greebstanford.ede manbhas/blog/), 2020. URL https://web: stanford....oooo..o...o.ooo.oo wtaaatttdd..ddd//aaaaaaeee/////e forward- hooks- moooks-pytoreee pytorch/.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD '06, 2006.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. ArXiv, abs/ abs/1812.003322, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Matthieu Courbariaux, Yoshua. Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural net- works with binary weights during propagations. In NIPS, 2015.
Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In NSDI, 2017.
Robert Culkin and Sanjiv R Das. Machine learning in finance: the case of deep learning for option pricing. Journal of Investment Management, 15(4):92 100, 2017.
Radosvet Desislavov, Fernando Mart'inez Plumed, and Jos'e Hern'andez Orallo. Compute and energy con- sumption trends in deep learning inference. ArXiu, abs/2109.000 /210009955,7 2021.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Maryam Ebrahimi, Alexandre da Silva. Veith, Moshe Gabel, and Eyal de Lara. Combining dnn partition- ing and early exit. In Proceedings of the 5th International Workshop on Edge Systems, Analylics and Networking, pp. 25 30, 2022.
Mohamed Amine Ferrag, L. Maglaras, Sotiris K. Moschoyiannis, and Helge Janicke. Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. J. Inf. Secur. Appl., 50, 2020.
Olga Fink, Qin Wang, Markus Svens enn Pierre Dersin, Wan-Jui Lee, and Mélanie Ducoffe. Potential, challenges and future directions for deep learning in prognostics and health management applications. Eng. Appl. Artif. Intell, 92:103678, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1321-1330. PMLR, 06-11 11 Aug 2017. URL
www.mmllrrrerrr wtt.aais......prrss/00/..//. wtto//uolaa.htee html.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In ECCV, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXic preprint arXiv:1510.00149, 2015.
Manjesh K Hanawal, Avinash Bhardwaj, et al. Unsupervised early exit in dans with multiple exits. arXiv preprint arXio:2209.004 2022.
P Haseena Rahmath, Vishal Srivastava, and Kuldeep Chaurasia. A strategy to accelerate the inference of a complex deep neural network. In Proceedings of Data Analytics and Management: ICDAM 2022, pp. 57-68. Springer, 2023.
Mahmoud Hassaballah and Ali Ismail Awad. Deep learning in computer vision: principles and applications. CRC Press, 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Compuler Vision and Pallern Recognation (CVPR), pp. 770 708,7788- 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. Ar Abss/1503.0233/ 2015.
Gary Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in racondtrained environments. In Workshop on faces in Real- Life' Imaage::: detection, aligmment, and recognition, 2008.
Jian Huang, Junyi Chai, and Stella Cho. Deep learning in finance and banking: A literature review and classification. Frontiers of Business Research in China, 14:1-24, 2020.
Olivier Chapelle Japn-sae Tien, joycenv. Display advertising challenge, 2014. URL https://kaggle. htt////ppeiiiiisssrrttteeeeeeeee chaee
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-docp networks: Understanding and mitigating network overthinking, In ICML, 2019.
Ira Seaeemmachhr-Shiizemmee Steven M. Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: 1 million faces for recognition scale. 2016 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 4873 4882, 2016.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Tech- nical Report TR-2009, University of Toronto, 2009. URL https://wwi cs. toroooo www.oto..eoooeeeeee. edu/-kriz/ Satte/eeeuuuuuu0----11-
Adarsh Kumar, Arjun Balasubramanian, Shivaram Venkataraman, and Aditya Akella. Accelerating deep
learning inference via freezing. ArXiv, abs/2002.026655, 2019.
Honglak Lee, Peter T. Pham, Yan Largman, and A. Ng. Unsupervised feature learning for audio classification using convolutional deep belief networks. In NIPS, 2009.
Tlis Lcontiadis, Stefanos Laskaridis, Stylianos Venieris, and Nicholas Lane. It's always personal: Using early exits for efficient on-device emn personalisation. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications, pp. 15 25-21, 2021.
Maksim Levental. Memory planning for deep neural networks. ArXiv, abss/2203.004/ 2022.
Y. Li, Zhenhua Han, Quanlu Zhang, Zhenhua Li, and Haisheng Tan. Automating cloud deployment for deep learning inference real-time online services. IEEE INFOCOM 2020 IEEE Conference on Computer Communications, pp. (668-1677) 2020a.
Yawei Li, Shuhang Gu, Kai Zhang, Lue Van Gool, and Radu Timofle. Dhp: Differentiable meta pruning via hypernetworks. Tn Computer Vision-FCCO 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Port VIII 16, pp. 608 624. Springer, 2020b.
Renju Liu, Luis Garcia, Zaoxing Liu, Botong Ou, and Mani B. Srivastava. Secdeep: Secure and performant on-device deep lcarning inforence framework for mobile and iot devices. Proceedings of the International Conference on Internet-of- Things Design and Implementation, 2021.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3296-3305, 2019a.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. ArXiv, abs/1810.05270,, 2019b.
Miodrag Lovric. International encyclopedia of statistical science. (No Title), 2011.
Mohammad Ali Maddah-Ali and Urs Niesen. Fundamental limits of caching. IEEE Tronsoctions on Infor- mation Theory, 60(5))2856-2255) 2867, 2014. doi: 10.1109/TTT 2TT.2004.2306933
TorchVision maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https://github.
com/pytorcchviision, 2016.
Sebastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. Proceedings of the 18th ACM international conference on Multimedia, 2010.
Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. Split computing and early exiting for deep learning applications: Survey and rescarch challenges. ACM Computing Surveys (CSUR), 2022.
Microsoft. Neural Network Intelligence, January 2021. URL https: htt.ub..com/iicooooft/ooooo
Thaha Mohammed, Carlee Joe- Nue-Wong, Rohit Babbar, and Mario Di Francesco. Distributed inference accel- eration with adaptive dnn partitioning and offloading. In IEFE INFOCOM 2020-IEEE Conference Computer Commmniications, pp. 854 863. IEEE, 2020.
Maryam Mohsin. 10 google search statistics you need know in 2022, Feb 2022. URL https://www. coterlo..abllo///eee www.goooooggerrrheeeoo
Stylianos Moschoglou, A. Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: The first manually collected, in-the-wild age database. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2017.
29

---

Published in Transactions on Machine Learning Research (10/2024)
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. 2019 IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 1325 1334, 2019.
Maxim Naumov, Dheovatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jong- 800 Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. Deep learning rec- ommendation model for personalization and recommendation systems. arXiv preprint arXtv:1966.0000 2019.
Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. Tensoffoo--serving Flexible, high-performance ml serving. ArXiv, abs/1712.06139, 2017.
Dan Otter, Julian Richard Medina, and Jugal Kumar Kalita. A survey of the usages of deep learning for natural language processing, IEEE Transactions on Neural Networks and Learning Systems, 32:604-200 -624 2021.
Roberto G Pacheco, Kaylani Bochie, Mateus S Gilbert, Rodrigo S Couto, and Miguel Elias M Campista. Towards edge computing using carly-exit convolutional neural networks. Information, 12(10):431, 2021.
Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. and enhanced pattern recognition. In 2016 Design, Automation Test in Europe Conference & Exhibition (DATE), pp. 475 480. IEEE, 2016.
Conditional deep learning for energy-effiiient
Huy Phan. huyvaphan/pytorde 10far10, January 2021. URL https:
1444440///20000111111...... 4431043.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. Arxi abs/ abs/1802.05668, 2018
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, and Hyeran Jcon. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. 2021 IEEE International Symposium on Hiigt-Performance Computer Architecture (HPCA), pp. 598 611, 2021.
Michael Ruchte, Arber Zela, Julien Siems, Josif Grabocka, and Frank Hutter. Naslib: A modular and flexible thub. con//atttm/////// 2020.
neural architecture search library. https: /ttps://github.....
Olga Russakovsky, Jia Dong, Hao Su, Jonathan Krause, Sanjeev Sathcesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Adilya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22) 1521, 2015. doi:
10.1007 10.1007//11263--11--066666
Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add early exits to neural networks? Cognilive Compulation, pp. 13, 2020.
Connor Shorten, Taghi M. Khoshgoftaar, and Borko Furht. Deep learning applications for covid-19. Journal of Big Dala, 8, 2021.
Carolyn J Swinney and John Woods. Unmanned aerial vehicle operating mode classification using deep residual learning feature extraction. Aerospace, 8(3):79, 2021.
Tvan Tashev and Seyedmahdad Mirsamadi. Dan-based causal voice activity detector. In Information Theory and Applications Workshop, 2016.
Surat Tecrapittayanon, Bradley McDanel, and Hsiang- Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. IEFE, 2016.
30

---

Published in Transactions on Machine Learning Research (10/2024)
Arun Varghese, George Agyeman-Badee Badu, and Michelle Cawley, Deep learning in automated text classification: a case study using toxicological abstracts. Enwironment. Systems and Decisions, pp. 1-15, 2020.
Jun Wang, Yinglu Liu, Yibo Hu, Hailin Shi, and Tao Mei. Facex-zoo zoo: pytorch toolbox for face recognition. Proceedings of the 29th ACM International Conference on Multimedia, 2021.
Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 692-702, 2019.
Weiaicunzai, Practice on cifar-100 using httppp//ihtaaa.oooomeaaaaaaaaaap 2020.
Duane Wessels. Web eaching. O'Reeilly Media, Inc.", 2001.
Xiaorui Wu, Hong Xu, and Yi Wang. Irina: Accelerating dnn inference with efficient online scheduling. 4th Asia-Pacific Workshop on Networking, 2020.
Wenhan Xia, Hongxu Yin, Xiaoliang Dai, and Niraj K Jha. Fully dynamic inference with deep neural networks. IEFE Transactions on Emerging Topics in Computing, 10(2):962 972, 2021.
Yecheng Xiang and Hyoseung Kim. Pipelined data-parallel cpu/gpo scheduling for multi-dnn real-time inference. 2019 IEEE Real- Time Systems Symposium (RTSS), pp. 392 205, 2019.
Yan Xiao, Tvan Beschastnikh, David S. Rosenblum, Changsheng Sun, Sebastian G. Elbaum, Yun Lin, and Jin Song Dong. Self-checking deep neural networks in deployment. 2021 IEEE/ACM 43rd International Conference Software Engineering (ICSE), pp. 372-384, 2021.
Fuxun Yu, Di Wang, Longfei Shangguan, Minjia Zhang, Xulong Tang, Chenchen Liu, and Xiang Chen. A survey of large-scale deep learning serving system optimization: Challenges and opportunities. ArXiv,
abs/2111. 2021.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-1, 03 2021. doi:
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In ECCV, 2018.
Zhuoran Zhao, Kamyar Mirzazad Barijough, and Andreas Gerstlauer. Deepthings: Distributed adaptive deep learning inference on resource-constrained iot edge clusters. IEEE Transactions Computer-Aider Design of Integrated Circuits and Systems, 37:2348-2359, 2018.
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recog- nition in unconstrained environments. ArXiv, abs/1708. 18119777.. 2017.
Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J. Leon Zhao. Time series classification using multi-channels deep convolutional neural networks. In WATM, 2014.
Keqian Zhu and Jingfei Jiang. Research on parallel acceleration for deep learning inference based on many- core arm platform. In ACA, 2018.
Lucas Zimmer, Marius Thomas Lindauer, and Frank Hutter. Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transoctions Pottern Analysis and Machine Intelligence, 43: 3079-3090, 2021.
31

---

Published in Transactions on Machine Learning Research (10/2024)
Collect inference data
Dataset
Pre-trained Model
Cache
Models
NAS
Optimizer
Deployment
Figure 5: Early Exiting system overall framework
A Appendix: Implementation
A.1 Implementation details
We developed the carly exit tool using PyTorch, which accessible through the GitHub repository?. The software environment for our experiments included Ubuntu 20.04, Python 3.7, and PyTorch 1.1. Figure 5 shows the overall system design. The tool provides NAS module, optimizer module, and a deployment module. The NAS module provides the architectures to be used per early exit model. The optimizer assigns the confidence thresholds, finds the best subset of the carly exit models, and provides evaluation reports. Lastly, the deployment module launches a Web server with the early exil-enabled model ready to serve queries.
A.1.1 NAS Module
Existing NAS tools typically define different search spaces according to different, lasks, which constrains their applicability to certain imput types and sizes. Using such tools with input constraints defeats our method's generalization and automation purpose, since the carly exit models' inputs can have any dimension and size. For example, ProxylessNAS (Cai et al., 2019) specializes in optimizing the performance of neural architecture for a target hardware. However, it is only applicable for image classification tasks and requires certain input specifications (e.g., 3xHxW images normalized using given values). Similarly, Auto-Py Porchc (Zimmer et al., 2021) and Auto-Keras are only applicable to tabular data sets, text, and images.
We chose NNT from Microsoft (Microsoft, 2021) it does not constrain the input of the model in terms of type, size, and dimensions. NN1 also provides extensible search space definition with support for variable number of layers and nested choices (c.g., choosing among different layer types, each with different layer-specific parameters).
Given the backbone implementation, the dataset and the search space, the module launches an NNI exper- iment per candidate layer to find an optimum early exit model for the layer. Each experiment launches a web GUI for progress reports and results.
32

---

Published in Transactions on Machine Learning Research (10/2024)
We aim for end-to-end automation in the tool. However, currently, the user still needs to manually export the architecture specifications when using the NAS module and convert. them to a proper Python imple- mentation (i.e., PyTorch module implementing the architecture). The specifications are available to the user through the experiments web GUI and also in the trial output files. This shortcoming is due to the NNT implementation, which does not currently provide access to the model objects within the experiments. We have created an enhancement suggestion on the NNT repository to support model object access (issue #-4910).
A.1.2 Optimizer and deployment modules
Given the implementation of the backbone and the early exit models, the optimizer evaluates the early exit modols, assigns their confidence thresholds, finds the best subset of the early exit models, and disables the rest, and finally reports the relevant performance metrics for the early exit-enabled model and each early exit. model. We used the DeepSpeed by Microsoft and the PyTorch profiler to profile the FT.OP counts, memory usage, and latency values for the early exit models and the backbones.
The user NAS module and provide the architectures manually to the optimizer, and the module trains them before proceeding to the evaluation.
each module independently. Specifically, the user can skip the architecture search via the
NAS with random search strategy can take approximately 1 hour for simpler models like ResNet18 and up to hours for more complex models like ResNet50.
The tool also offers an extensive set of configurations. More specifically, the user can configure the tool to use one device (e.g., GPU) for training processes and the other (e.g., CPU) for evaluation and deployment.
The deployment module launches web server and exposes a WebSocket API to the carly exit-enabled model. The query batches passed to the socket will receive one response per item, as soon as the prediction is available through either of the (early or final) exits.
A.1.3 Backbone Implementation
We used the backbone implementations and weights provided by the FaceX-Zoo (Wang et al., 2021) repository to conduct the experiments with LWF data set on the MobileFaceNet and EfficientNet models.
For experimenting with CIFARIO and CIFAR100, we used the implementations provided by torchvision (Marcel & Rodriguez, 2010) and the weights provided by (Phan, 2021) and (Weiaicunzai, 2020). For Ima- geNet pre-trained weights for ResNets we used new torchvision version as well (maintainers & contributors, 2016). All backbone implementations were modified to implement an interface that handles the interactions with the early exit models, controls the exits (early exit hits and misses), and provides the relevant reports and metrics. We document the use of the interface in the repository, so that users can experiment with new backbones and datasets. We refer interested readers to blog post on how extract intermediate activations in PyTorch (Bhaskhar, 2020) which introduces three methods to access the activation values. The introduced forward hooks method in Py is very convenient. for read-only purposes. However, our method requires performing actions based on the activation values, specifically, early exit lookup and batch shrinking, and avoiding further computation through the next layers. Therefore, we used the so-called "hacker" mothod to access the activation values and perform these actions and provided the interface for easy replication on different backbones.
A.2 Environment setup
The hardware used for inference substantially affects the results due to the hardware-specifie optimizations such as computation parallelism. In our experiments, we have used an "Intel(R) Core(TM) 17-10700K CPU @ 3.80GH" to measure on-CPU inference times and an "NVIDIA GeForce RTX 3070" GPU to measure on-GPU inference time.
33

---

Published in Transactions on Machine Learning Research (10/2024)
CIFAR10 Resnettt
CIFAR100 Resnet50
LPW Efficienthet
MobileFaceNet
Input
ReLU
Inpul
Input
Input
FC11024) ReLU
Caofle
Conv2D
Cache
FC(1024)
Conv20
Com220(ee),
x
ConvaDja=3, s=2)
Comvilock
FC(127)
Sontaax
MazPool
FC1127)
FC/512) Ralu
MBConv
Carle
FC)298) Ralu
ConvBlock
SoftMix
DepthiMte
FCC127)
POrS12)
FCJ1024)
SoftMax
MBComm
Residual
Relu
Cacito
Rolu
MBConv
DepthWise
SoftMax
MBConv
FC(1024) Relu
Residual
Rasidual
Residial
Bettleneck
Bottleneck
DepthWWtse
FC1286)
SoftMixx
Residual
Cache
Oache
Oache
SoftMax
Bottleneck
Residual
FC/127>+ Softilax
Residual Bottleneck
PC/127)
SoftMax
AugPool
Cen/Block
LinearBlock
FC/1024) Relu
Flatben
Flatten
Residual
Residual Bottleneck
Cache 4
FC($12)
Ralla
Bottlensc
FQJ512)
Retu
Embed
FC/127)
Lin. Embed.
FC/127)+
AvgPool
FCJ127y-
AugPool
Ln Relu
Un Relu
SohMaa
BaltMix
SottMax
Figure 6: Final schema of the early exit models, for the experiments CLFAR1O-Resneee CIFAR100- Resnct50, LWW-EffiiientNet, and LFW-MobileaacoNot
B
Appendix: Final schema of early exit models
This appendix shows key figures that provide deeper visual understanding of the methodologies and results highlighted in the main text.
Figure illustrates the final schema of the early exit models used in our experiments with CIFAR1O-Resnet and LFW-MobileFaceNett
CIFAR100 (IFAR100-Resne500,
Figure 7 presents the final schema of the early exit models specifically designed for our CityScapes experi- ments using the Mask R-CNN framework.
Figure shows the final schema of the early exit models designed for our recommendation experiments with the Criteto data set using DLRM.
34

---

Published in Transactions on Machine Learning Research (10/2024)
CitSccapes
Cache 1
Comv2D(k=3, s+8)
Input
Mask R-CNN
FC(256)
Conv20
FCI81)
Backbone
Cache
RolAlign
Conv
sm4)
4X Residual Bottienec
Object Detection
Mask Generation
FC/258]
FC(51) +
SontMax
Residual Bottionec
Class Box
Conv2D(4x3,
sm4)
3X Residual Bottioneo
FC(256) Relu
FC(81) SoftMax
AvgPool
Figure 7: Final schema of the early exit models, for the experiments CityScapes Mask RONN
Input
Criteo Dataset- MLP (Top and Bottom)
Linser (512) Relu
Linear (236) Relu
Lineari51 12)
Relu
Donse Features
Sparse Features
(64)
SoftMax
Linear (16)
Lookup
Lookup
Rahu
Lineari54) Relu
FC(2) SoftMiux
Feature Interaction
Linear (367) Ralu
Linear (512) Relu
Cache 3
(250)
Linear[512) + Retu
FC12] SoftMax
Linear Relu
Figure 8: Final schema of the early exit models, for the experiments Criteo DLRM
C
Appendix: Early Exit models' individual performance for all experiments
The following figures demonstrate the hit rate, GT accuracy and early exit accuracy of each carly exit model versus the confidence threshold, per experiment dataset and backbone.
35

---

Published in Transactions on Machine Learning Research (10/2024)
Cifar10-Resnet18
Cache 1
Cache
Cache
Cache
100
80
60
20
0-
100
%
40
80
60
20
25
50
75
100
25
50
100
25
75
100
25
50
75
100
Confidence
Confidence
Confidence
Figure 9: Experiment: CIFARIO-Resnet18
Table 8: Different carly exit models accuracy, MFN: MobilcFaceNet, EFN: EfficientNet, R18: Resnet18,
R50: Resnet50, C10: CIFAR10, C100: CIFAR100, IN: ImageNet, CityScapes: CS, Mask RCNN: MRCNN
Model Type
R18-C10
R50-C10
R18-C100
R50-C100
R18-IN
R50-IN
MEN-LFW
EFN-LFW
MRCNN-CS
DLRM-CHte
With NAS
86.49%
85.88%
74.47%
77.04%
68.12%
74.09%
96.91%
95.35%
83.4%
73.54%
Without
74.32%
77.19%
67.31%
72.91%
96.19%
95.71%
73.96%
layer
87.15%
12%
74.29%
77.
67.56%
73.19%
96.07%
95.92%
83.89%
73.91%
more layers
86. 20%
86.09%
74.82%
77.02%
67.61%
73.21%
95.42%
95.144%
83.01%
73.62%
3 more layers
86.17%
86.01%
74.78%
77.03%
67.60%
73.31%
95.43%
95.14%
82.90%
73.48%
Appendix: Ablation Study
To investigate the need for using NAS in our work, we run an ablation study. The alternative to use NAS is inserting early exit models al every possible node, which can be memory-intensive and may even degrade the final performance and accuracy. Additionally, the number of early exit layers may also impact the results. Our default setup uses dense layers per carly exit model. This part of the ablation study assesses the impact of increasing this number 3, 4, and dense layers. The following tables compare latency (GPU and CPU), accuracy, model size and FLOPPs across these modes.
36

---

Published in Transactions on Machine Learning Research (10/2024)
Cifar10-Resnet50
Cache 1
Cache
Cache
Cache
100
80
60
20
0
100
%
40
80
60
20
25
50
75
100
25
50
100
25
75
100
25
50
75
100
Confidence
Confidence
Confidence
Figure 10: Experiment: CIFARIO-Resnet50
Table Different early exit models latency on GPU, MFN: MobilePaceNet, EFN: EfficientNet, R18:
Resnet18, R50: Resnet50, C10: CIFAR10, C100: CIFAR100, IN: ImageNet, CityScapes: CS, Mask RCNN:
MRCNN
Model Type
R18-CCI0
R50-C10
R550-C100
R.1-IN
MPN-LPW
EPN-LFW
MRCNN-CS
DLRM Criteo
With NAS
1.51
14.38
Without NAS
1.29
1.75
1.47
2.27
3.91
3.64
9.26
17.87
153.1
1.89
more layer
1.11
1.63
1.32 ms
2.19 mus
3.27 mus
3.33 ms
7.91 mas
16.10 ms
123.1 ms
1.52 ms
more layers
1.21 mss
1.71 ms
1.30 ms
2.26 ms
3.34 ms
3.37 ms
7.97 ms
16.70 ms
131.27 ms
1.69 ms
more layers
1.48 ms
2.37 ms
ms
3.47 ms
ms
17.51 ms
138.27 ms
ms
Regarding NAS use, the results indicate that there is average increase in accuracy of less than 2%, while latency, FLOPs, and model sizes increased by average of 16%. When using additional dense layers, the results showed an increase in accuracy of less than 1%, but resulted in approximately 24% worse results in latency and model size and about 30% increase in FLOPs usage on average.
Ablation experiments indicate that using additional dense layers or ignoring NAS and inserting and training all possible early exit layers may result in slightly better accuracy. However, this is accompanied by increased latency and, in some cases, performance worse than the original model without early exits. This is because
37

---

Published in Transactions on Machine Learning Research (10/2024)
Cifar100- 1
Cache 1
Cache
Cache
Cache
100
80
60
20
0
100
%
40
80
60
20
25
50
75
100
25
50
100
25
75
100
25
50
75
100
Confidence
Confidence
Figure 11: Experiment: CIFAR100-Resnet
Table 10: Different early exit models latency on CPU, MFN: MobilelFaceNet, EFN: EfficientNet, R18:
Resnet18, R50: Resnet50, C10: CIFAR10, C100: CIFAR100, IN: ImageNet, CityScapes: CS, Mask RCNN:
MRCNN
Model Type
R18-C10
R50-C10
R18-C100
R50-C100
RI8-IN
R5001N
MFN-LFW
EPN-LPW
MRCNN-CS
DIRM-Critsto
10.11
14.02
0.39
9.02
38.74 ms
562.3 ms
ms
Without NAS
13.53
19.17
13.19
13.01
51.04
22.53
38.77
724.42
11.89
more layer
11.87 ms
18.12 ms
12.19 ms
12.12 ms
33.31 ms
45.41 ms
21.25 ms
34.68 ms
689.11 ms
10.63 ms
more layers
12.84 ms
19.32 ms
14.01 ms
13.89 ms
34.91 ms
47.02 ms
25.94 ms
35.78 ms
713.71 ms
11.93 ms
more layers
13.17 ms
22.12 ms
16.71 ms
15.89 ms
ms
48.08 ms
26.41 ms
37.52 ms
743.71 ms
13.31
having, for example, 10 early exit layers and processing all batches through them significantly increases the time required and also enlarges the model size. In addition, memory consumption negatively affected.
38

---

Published in Transactions on Machine Learning Research (10/2024)
ImageNettRResnt16
Cache
Cache
Cache
Cache
100
Cache Accuracy %
80
60
40
100
80
GT Accuracy %
40
80
Hit Rate %
40
25
75
100
50
75
100
25
75
1000
25 Confidence
50 75
100
Confidence
Confidence %
Confidence %
Figure 12: Experiment: ImageNet-RRennt
Table 11: Different early exit models FLOPs, MFN: MobileFaceNeel, EFN: ElficientNet, RI8: Resnet18,
R50: Resnet50, C10: CIFAR10, C100: CIFAR100, IN: ImageNet, CityScapes: CS, Mask RCNN: MRCNN
Model Type
R18-C10
R50-C10
R18-C100
R50-C100
R18-IN
R50-IN
MEN-LFW
EFN-LFW
MRCNN-CS
DLRM-Criteo
With NAS
414M
601M
374M
547M
1673M
2020M
296M
182M
2730M
90M
Wichout NAS
671M
942M
515M
719M
1941M
2130M
371M
282M
3211M
128M
more layer
614M
921M
534M
643M
1897M
2321M
351M
261M
3139M
121M
more layers
701M
10L1M
709M
773M
1970M
2410M
404M
200M
3310M
138M
more layers
823M
2018M
514M
39

---

Published in Transactions on Machine Learning Research (10/2024)
ImageNettRResne50
Cache
Cache
Cache
Cache
100
Cache Accuracy %
80
40
80
GT Accuracy %
40
80
Hit Rate %
40
25
50
25
75
100
50
100
Confidence
Confidence %
Confidence %
Confidence
Figure 13: Experiment: ImageNeetRRenet.50
Table 12: Different early exit models size (Mega Byte), MFN: MobileFaceNeet EFN: EfficientNet, R18:
Resnet18, R50: Resnet50, C10: CIFAR10, C100: CIFAR100, IN: ImageNet, CityScapes: CS, Mask RCNN:
MRCNN
Model Type
R18-C10
R50-C10
R18-C100
R50-C100
R18-IN
R50-IN
MEN-LFW
EFN-LEW
MRCNN-CS
DLRM-Crite
With NAS
97MB
243MB
383MB
532MB
403MB
572MB
350MB
297MB
4171MB
332MB
Without NAS
137MB
324MB
142MB
822MB
731MB
801MB
514MB
111MB
5231MB
451MB
more layer
113MB
273MB
101MB
622MB
591MB
GOUMB
372MB
320MB
4319MB
362MB
more layers
130MB
301MB
420MB
691MB
682MB
725MB
395MB
351MB
4480MB
390MB
more layers
340MB
451MB
751MB
705MB
421MB
427MB
40

---

Published in Transactions on Machine Learning Research (10/2024)
LFW.MobileFaceNett
Cache 1
Cache 2
Cache
100
Cache Accuracy %
80
100
80
GT Accuracy %
60
40
100
Hit Rate %
40
20
25
50
100
75
100
25 Confidence
75
100
Confidence %
Confidence %
Figure 14: Experiment: I.F.-Mobileaceeeet
41

---

Published in Transactions on Machine Learning Research (10/2024)
LFW-EfficientNet
Cache
Cache 2
100
Cache Accuracy %
60
20
100
GT Accuracy %
60
20
0
100
Hit Rate %
60
20
0
0
20
40
60
80
100
20
40
60
80
100
Confidence
Confidence %
Figure 15: Experiment: LFW-EfficientNett
42

---

Published in Transactions on Machine Learning Research (10/2024)
Mask RCNN-CityScape
Cache
Cache
Cache
100
Cache Accuracy %
80
60
40
80
GT Accuracy %
40
80
Hit Rate %
40
25
50
100
75
25
75
100
Confidence %
Confidence
Confidence %
Figure 16: Experiment: Mask RCNN-CityScape
43

---

Published in Transactions on Machine Learning Research (10/2024)
DLRM Criteo
Cache
Cache
Cache
100
Cache Accuracy %
80
60
40
80
GT Accuracy %
40
80
Hit Rate %
40
50
100
75
25
75
100
Confidence %
Confidence
Confidence %
Figure 17: Experiment: DIRM-Criteo
44


Reviews:
[{"review_id": "eKuvq4b9vf", "reviewer": "Reviewer_t26e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The draft is well written with proper inclusion of past related work to easily follow the related concepts for new readers.\n2. The idea is developed in unsupervised settings, allowing early exit models do not need access to training data and perform solely based on the incoming data at run-time.\n3. During the training an early exit model, the proposed work freeze the the model backbone, to ensure that the training process does\nnot modify any parameter not belonging to the current early exit model; which encourages adoptability.\n4. The authors extends their proposed work to evaluating a critical real-world scenario - pedestrian detection in urban environments.\n\n\nWeakness:\n\nWhile the paper is a interesting read, I have several concerns with the algorithm/method developed given its complexity.\n\n1. In candidate layer selection step (section 3.1), the author mentions several static rules to select candidate layers (eg. last layers not useful, layers with parallel connections are not useful). I find this static rules difficult to generalize across numerous existing backbone architecture. How should one decide how many last layers to discard? The method is not generalizable for recent popular complex architectures like MoEs where there a multiple parallel experts?\n\n2. Does the authors have thought about using these early exit approaches for modern LLMs where early exits would add a lot of value given gigantic memory cost? Recently some work find like SkipDecode (https://arxiv.org/abs/2307.02628), ShortGPT(https://arxiv.org/pdf/2403.03853), FFN-SkipLLM (https://arxiv.org/abs/2404.03865) explores layer-skipping or early-exit strategies. What are authors thought about these techniques given their simplicity in identifying comparatively less useful layers and discarding them in limited compute settings?\n\n3. In section 3.2.1, the author briefly discuss the complexity of task (CIFAR10 vs CIFAR100) in deciding the amount the early-exit model capacity. I see further potential here to explore a more task-difficulty specific evaluation to see how it relate with the speedup achieved for a easy task vs difficult task?   \n\n4. Comparison with a dense baseline with similar FLOP/compute count to validate the performance difference of the early exit model with a model with similar compute cost is required.\n\n5. The well-advertised benefit of the work is speedup, however a clean speedup/latency evaluation settings is missing. How is the speed up measured specifically across CPUs and GPUs? What's the hardware/software stack? A fine-grained evaluation of compute/latency/memory for backbone, early-exit modules is a must? The authors nee to also provide an idea of compute additionally involved in training these early exit models, NAS, thresholding.\n\n6. I am also curious about the thresholding, and the expense of update and maintenance of early exit models. A ablation of these thresholding is required to show the impact it can have.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an end-to-end automated early exiting solution to improve the performance of DNN-based services in terms of computational complexity and inference latency. The paper methodology involves - (a) Identify the candidate layers to be early exit (b) Build an early exit model for each candidate (c) Assign confidence thresholds to built models to determine early exit hits (d) Evaluate and optimize the early exit-enabled model (e) Early-Exit Optimization Implementation (f) Update and maintenance early exit models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The critical requirements include a through investigation of training and inference time compute benefits, and details for  hardware/software stack is required. The latency increases with increasing the batchsize (Table 6) is also a negative point, given modern GPUs which are fast and efficient in large batch processing. I also highly encourage the authors to explain their key method with a good diagram to make it easy to understand, at this point it requires a lot of effort to understand it to understand the nuances. Also, additional comments related to adaptation for modern architectures with stack of transformer will be helpful given the candidate layer selection is heuristic based.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mogdOUI24Q", "reviewer": "Reviewer_SPN2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed method works in an unsupervised manner without requiring labeled data, making it useful for applications with limited labeled data. The authors also use Neural Architecture Search (NAS) to dynamically add early exits across different architectures, demonstrating great flexibility and adaptability.\n2. The related works section is comprehensive and clearly differentiates this work’s contribution from the existing literature.\n3. The proposed approach shows significant speed-ups in inference compared to other methods at the cost of a slight drop in accuracy. This can be useful in applications where the inference cost is prioritized over accuracy.\n\n**Weaknesses:**\n\n1. The accuracy metrics are not clearly defined and show inconsistent results across the different experiments. (Refer to Requested Changes A.1 for details)\n2. The unsupervised problem setup and evaluation are not clearly defined. (Refer to Requested Changes A.2 for details)\n3. The comparison with other methods is not performed in a standardized way. (Refer to Requested Changes A.3 for details)\n\n**Disclaimer:**\nI skimmed the math related to Equation 3 and am not able to evaluate its theoretical correctness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an approach to accelerate inference for deep neural networks by introducing early exit paths that are trained using test data without requiring labels. The training method works through self-distillation, where the output of each newly added early exit layer mimics the model's output probability distribution (soft labels). The authors also use Neural Architecture Search (NAS) to identify candidate layers for adding early exits, making the approach adaptable to different architectures such as MobileFaceNet, EfficientNet, ResNet18, and ResNet50. Overall, this work extends early exiting approaches to be used in an unsupervised fashion, which brings practical value to applications with limited labeled data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**A. Recommendation for acceptance (critical comments):**\n\nA.1. The accuracy metrics are not clearly defined and show inconsistent results across the different experiments. For example, the final accuracy of the Base (non-early exit enabled) on CIFAR100-ResNet50 is 78.98% in Table 2, which I would expect to be an approximate upper bound on accuracy based on Table 3 from (Kaya et al., 2019). However, the early exits 1-4 in Figure 2 achieve 100% GT Accuracy at 100% confidence, while the early-exit enabled model achieves roughly 86% accuracy at 95% confidence in Figure 4. In both experiments, I expected these accuracies to be closer and roughly upper bounded by the Base model (78.98%).\n\n**Recommendation:** Similar to the tables of (Kaya et al., 2019), I recommend adding the accuracy of the original model (non-exit enabled) across tables and figures for reference.\n\nA.2. The unsupervised problem setup and evaluation are not clearly defined.\n- Given that the training of the early exits is done on the test set, does this mean the model is not deployed yet? Is this training phase on the test set considered a warm-up? Or are predictions being provided to the target application while also training early exits?\n- How were other methods trained given that, to my knowledge, they require access to the training data? Were they also trained in an unsupervised way?\n- Were other methods evaluated on the entire test set or just the second half of the test data, similar to your method?\n\n**Recommendation:** To address these comments, I recommend adding a formulation of the problem setup in the methodology section since the proposed unsupervised training is part of your contribution and may not be clear to the reader. I also recommend discussing how other methods, which may originally train on labeled data, are adapted to this framework. As a suggestion, this could be under a \"Baselines\" subsection in Section 4.\n\nA.3. The comparison with other methods is not performed in a standardized way.\n\n- At which confidence % were the results reported in section 4.5.6?\n\nIf these were reported at fixed confidence %, this may make the evaluation subjective and could be dataset-specific. While it is great to show a trend of trade-offs between performance and latency, a more standardized approach for benchmarking would be preferable.\n\n**Recommendation:** Since this work is based on (Kaya et al., 2019) and to be consistent with previous works, I recommend showing the comparison results at defined points of inference costs similar to Table 3 of (Kaya et al., 2019).\n\n**B. Comments to strengthen the work (minor comments):**\n\nB.1. Section 3.5 seems to be incomplete or out of context. At the beginning of the section, the discussion appears to be about the implementation of two tasks: image classification and object detection. However, the rest of the section only covers image classification for pedestrian detection. Then, at the end of the section, a recommendation system task is introduced abruptly without sufficient context. I recommend revising Section 3.5 to clarify its objective.\n\nB.2. Writing clarity: I highly recommend revising the document thoroughly in terms of language and grammar. For example:\n- There are several mentions of the term “a early exit”, which should be “an early exit”.\n- The phrase “acceleration inference” in Section 2.1 should be “accelerating inference”.\n\nB.3. Terminology consistency: There are a few mentions of “mean precision” and “final precision”. It seems precision and accuracy are used interchangeably in the document, even though they are different metrics. Do these references actually indicate the precision metric? If so, I suggest defining them in the metrics.", "broader_impact_concerns": "Although this is obvious, it may be worth mentioning that deploying this approach in safety-critical applications (e.g., autonomous cars) to improve inference speed may result in accuracy degradation, which could lead to potential accidents.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F65jaXhZPN", "reviewer": "Reviewer_qSHL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper addresses an interesting topic. Adaptive inference in neural networks is gaining popularity and the idea of using test-time information to improve the efficiency of future predictions is promising.\n- While most similar works only report results for image classification, this paper also include object detection and recommendation systems.\n\nWeaknesses:\n- I believe the choice of the word \"cache\" in the title and throughout the paper is misleading. After reading the abstract, I expected that this paper would introduce some sort of key value storage in the model that would allow the model to return a prediction early if the input is similar to a previously processed input, stored in the cache. Instead the authors propose to add trained early exit classifiers. I suppose you could interpret these classifiers as implementing some sort of caching mechanism since they are updated at test-time but I believe that the use of the word \"cache\" is distracting. Instead, I would encourage the authors to stick to more conventional terminology (e.g. early exit, adaptive computation, ...). I believe this would make it easier for potential readers to find this paper. \n- It is not clear to me how the confidence calibration works. I understand the need for calibration but how is this performed without ground truth labels ? I suppose you could use the final model's outputs as ground truth but then you should probably also take this model's confidence into account ?\n- The authors put a lot of emphasis on the fact that it is not required to have access to the training data to train the early exits. Does that mean that they are initialized from scratch ? Why not pretrain them on the training data ? \n- It is not clear to me how the updating procedure works. Section 3.6 gives two obvious approaches (update every X samples, update the early exits when the model is updated). For a paper that claims \"online\" in the title, I would have expected a more in depth discussion on the actual online updating procedure. As far as I can see, there is no online aspect in the provided experiments. I would expect at least a graph that shows the accuracy/ runtime of the model as a function of the observed test samples. This way you can see that the model becomes more efficient as more samples have been observed.\n- In section 4.2 the authors explain the test procedure. This is not clear to me. For example for the CIFAR10 dataset, the authors state that \"we only use the test splits\" and \"we do not use the labels in these test sets in the training and optimization process\". Does that mean that the early exits are trained using test data ? This seems like an obvious opportunity for overfitting. In addition, it is stated that the test data is augmented with flips and rotations which makes it hard to compare to other approaches that just use the conventional test data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose to use early exits in deep neural networks to return a prediction quickly when these early exits are already able to make an accurate prediction. In addition, the authors propose to update these early exits at runtime, mimicking a caching mechanism.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I feel that the paper should include results on the ImageNet dataset as well. I understand that it is annoying when reviewers ask for additional results on other datasets, especially since the paper already includes experiments on large datasets such as CityScape but the ImageNet dataset is still the most commonly used image classification benchmark and I believe that the large number of classes and the higher resolution makes it a more interesting benchmark than CIFAR10/CIFAR100. In addition, it is likely that follow up work will be evaluated on ImageNet and this would allow those authors to compare to this work.\n- The writing could be improved. There are still quite a bit of typo's and numbers in the text have a space after the decimal point.\n- Overall, the paper is very verbose which hides a lot of the interesting aspects. I would encourage the authors to focus on what is really new (the online adaptation).", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers lean toward rejection (qSHL, SPN2) while one reviewer leans toward acceptance (t26e). The vote for acceptance by t26e is still qualified by concerns about the thresholding and the computational expense of the early exit models, but they side with leaning accept, because their concerns have been addressed and the submission can be understood by the reader and is therefore informative. The votes for rejection both agree that the work has an interesting topic of great relevance to the community, and that there are valuable insights for early exiting especially for practitioners, but on the balance they are concerned with (1) the clarity of the setting evaluation to identify what is learned from this work and (2) the comparability of experiments between the proposed method and the comparison methods of BranchyNet and GATI. qSHL also raised an issue about novelty, but this has been factored out of the decision given TMLR's mission to consider claims-evidence and audience, and an issue about missing ImageNet, but ImageNet results are partially provided in the revision.

The concerns (1) and (2) about clarity and comparability are serious and would require rejection if unresolved. However, (1) has been improved by the revision and discussion, and can improve further with a last revision. (2) is not relevant to the internal claims of this work about the effect of the proposed method vs. the base model but the issue is key to comparisons and the claims in the abstract about latency and accuracy. Reviewer SPN2 has made multiple important and constructive points on this and the authors have replied. The submission is not yet fully revised to incorporate this feedback, but can be (and must be or else the claims and evidence are not fully accurate and concrete).

The action editor decides on acceptance with minor revisions but must note that the revisions while minor in scope are major in importance and required for acceptance. The reasons for acceptance are:

- The results on the proposed early existing model are informative whether both positive or negative. For instance, the pairing of Tables 5 & 6 shows both the potential for time efficiency and the pitfall of batching with current hardware. Likewise there is a plus and a minus for memory efficiency, in that exiting could allow memory to be reclaimed with dynamic allocation but the early-exit architecture requires more parameters and may need more total memory when executing all exits. Explaining and measuring these concerns is valuable especially for the multiple measures of FLOPs, memory, and latency as done here. In this work the results make clear that the computational efficiency is in time (for suitable batch sizes), and not memory, since more overall memory is needed.
- Following the discussion by SPN2, the experiments with BranchyNet and GATI are fair modulo the differences in setting, even if they are not as complete as could be. The results do control for the use of data and amount of training even if there are differences in pre-training and fixing parameters. In particular, there could be an ablation and analysis of whether to freeze the base model or not. However, without it the results are still true as a holistic evaluation of the proposed method vs. BranchyNet and GATI in all their details.
- The introduction and related work can guide those interested in efficient inference for image classification and inspire more work. This is worth noting because this work provides pointers to inference optimization more broadly, early exiting for different purposes, and distillation/self-distillation for training models on model predictions.

The requested revisions distill the official recommendations by reviewers and the concerns summarized by the action editor in the decision:

1. Scope the abstract and summary of contributions to limit the claims to deep convolutional networks for vision. This addresses the feedback and decision recommendations from reviewers about different model types like transformers and domains like text and is preferable to the general discussion of deep networks.
2. Resolve the lack of clarity around the comparability of the compared methods. Make sure to incorporate the discussion of how training and evaluation are related (including this comment https://openreview.net/forum?id=h4rUKKfl5S&noteId=7fcilGsOub) for the proposed method, BranchyNet, and GATI.
3. Discuss this reference on unsupervised early exiting in NLP "Unsupervised Early Exit in DNNs with Multiple Exits" by Narayan et al. 2022 which was provided by SPN2 during the recommendation phase. Note that the domain and methodology differ and do not invalidate the claims and evidence of this submission.
4. Of course the draft revision with comments to the reviewers and the like must be edited into a final revision.

The action editor offers this miscellaneous feedback:

- As a suggestion for terminology, "distillation" or "self-training" may be more accessible and avoid confusion as alternatives to "unsupervised learning" because some readers may expect the training to be a function of the input itself, like unsupervised reconstruction losses or self-supervised auxiliary losses, rather than the training to be a function of the base model predictions. Of course this is up to the authors!
- Sec. 3.6. It is confusing to talk about updates and maintenance if these are not done in this work. Consider removing this section and moving the content on evaluation stages to the following Sec. 4 on evaluation. These thoughts on model maintenance could be added to a discussion at the end such as in Sec. 4.7 to group together discussion, limitations, and future work.
- Sec. 4.5. Add parenthetical citations for the baselines for ease of reference within the section.
- pg. 7. ImageNet is missing from the list of datasets in the empirical contributions.
- pg. 8. Possible typo: "utilize their old observation" for "utilize their own observation"? If not, then consider rephrasing for clarity anyway.
- pg. 8. Typo: "overthinkingproblem"
- pg. 25. There is a stray fragment "They must be comprised of accuracy drop data" before Table 3.
- pg. 30. Major typo: "goal of our proposed approach is to increase the inference time" when it should be "decrease the inference time".
- pg. 30. Typo: "Appendix .2."
- The capitalization of GATI is inconsistent. Please use GATI to respect the capitalization in the original paper.

The action editor thanks the authors, for submitting to TMLR and for engaging in the discussion and revision, and thanks the reviewers, for the service to TMLR and for providing their expertise and constructive feedback.
Decision: Accept (poster)