## 2 Relevant Work
### Conditional Computation  
Conditional computation, also known as adaptive inference, has gained attention recently due to its compatibility, ease of use, and high-performance advantages. Adaptive inference aims at achieving efficient dynamic computational resource allocation by strategically invoking lighter or complex deep neural units, depending on input [5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18]. [PERSON] [12] has presented an extension work of a Recurrent Visual Attention Model by [PERSON] et al. [19] and proposed a dynamic computational time model (DT-RAM) to speed up the overall processing time. Rather than 'attention' to a finite number of steps, they added one extra binary action, which dynamically decides to continue/stop for each input image. In DT-RAM, initially, a pre-trained Recurrent Attentional Model (RAM) is utilized, and, later, it is fine-tuned with REINFORCE. The RAM model defines that every input has a corresponding attention measure. The internal state of local regions is computed and updated with a recurrent neural network over each previous time step. The model then computes over two branches, which are location network and classification network. The location network models the attention policy and samples the attention location based on the learned policy. The classification network computes simply classification score. In DT-RAM, as an extension of RAM, a stopping network operation is introduced, which decides when it should stop taking more 'attentions' and output results as an early exit. Since the intermediate supervision at every time step requires the output of the classification score, the loss is defined as an average cross-entropy over \(N\) training samples and \(T(n)\) time steps. MSDNet [18] also proposed a progressively updating deep learning model. They combined the convolutional and dense network so that the intermediate classifiers maximally and efficiently use the computation resource. They utilized the combined fine and coarse level features at two scales to retain high-quality classification performance early. To update the losses between these intermediate classifiers, they targeted minimizing the weighted cumulative loss. [PERSON] et al. [17] designed their adaptive inference model by setting up multiple intermediate classifiers (multiple exits) and settling these early classifiers' gradient conflicts by introducing Gradient Equilibrium technique. To enhance the information sharing and collaboration between these classifiers, they also introduced Inline Subnetwork Collaboration (ISC) and One-for-all Knowledge Distillation (OFA) techniques. The multi-exit simple classifiers were responsible for learning discriminative features for themselves and maintaining information to pass to complex classifiers at later stages. Here, MSDNet [18] has calculated the weighted cumulative loss of all the intermediate classifiers and minimized it. In [PERSON]'s [17] inference model, intermediate classifiers have the overlapping arrangement, and the loss minimization strategy by MSDNet [18] may create an issue of gradient imbalance due to its overlapping model structure. To handle the issue of gradient balancing, they introduced the Gradient Equilibrium (GE) method, which normalizes the gradients by a two-level scaling method. With their Inline Subnetwork Collaboration (ISC), they attempted to collaborate between intermediate adjacent classifiers by adding a knowledge transfer path function to promote forward knowledge transfer. In this stage, the early intermediate classifiers help to boost the performance of the latter classifiers. Similarly, the deepest classifiers at the farthest end help increase the learning of shallow classifiers in the Backward Knowledge Transfer approach. The discussed conditional computation-based research works have overall shown the strength and utility of shallow, intermediate classifiers.