SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2 (Excess Risk). Suppose data  $D$  contains of  $N$  iid samples from distribution  $\mathcal{D}$  and that the function  $g$  has  $\epsilon$  margin on data  $D$  with probability  $\mu$ , i.e.,  $\operatorname*{Pr}_{x\sim D}[|g(x)| < \epsilon] < \mu$ . Further, suppose that the class  $\mathcal{C}_T$  has VC dimension  $d$ . Then, for  $T \geq 4G_{\infty}^2 \ln 2N/\epsilon^2$ , with probability  $1 - \delta$  over the samples, the output  $F_T$  of algorithm 1 satisfies:
$$
\operatorname {e r r} \left(F _ {T}\right) \leq \widehat {\operatorname {e r r}} (g) + O \left(\sqrt {\frac {d \ln (N / d) + \ln (1 / \delta)}{N}}\right) + \mu .
$$
Note that the above theorem can easily be adapted to the case of margins and VC dimension of the class  $\mathcal{C}_T$  being replaced with the corresponding fat-shattering dimensions. Furthermore, in the setting of stochastic minimax optimization, one can get population bounds directly by thinking of sample losses and gradients as stochastic gradients to the population objective. This is for example the view taken by [40]. In our work, we separate the population and sample bounds to simplify the presentation and the proofs.
We now evaluate B-DISTIL on both real-world and simulated datasets and over a variety of architecture types. We consider six real world datasets across three domains—vision, speech and sensor-data—as well as two simulated datasets. This allows us to evaluate our method on five architecture types: fully connected, convolutional, residual, densely connected networks and recurrent networks. Our code can be found at: github.com/metastableB/bdistil.
5.1 Dataset Information
For experiments with simulated data, we construct two datasets. The first dataset, referred to as ellipsoid is a binary classification dataset. Here the classification labels for each data point  $x \in \mathbb{R}^{32}$  are determined by the value of  $x^T A x$  for a random positive semidefinite matrix  $A$ . The second simulated dataset, cube, is for multiclass classification with 4 classes. Here labels are determined by distance to vertices from  $\{-1, 1\}^{32}$  in  $\mathbb{R}^{32}$ , partitioned into 4 classes.
We also use six real world datasets for our experiments. Our image classification experiments use the CIFAR-10, CIFAR-100, TinyImageNet and ImageNet datasets. For time-series classification tasks we use the Google-13 speech commands dataset. Here the task is keyword detection: given a one-second buffer of audio, we need to identify if any of 13 predefined keywords have been uttered in this. Finally, we use the daily sports activities (DSA) dataset for experiments with sensor data. Here the task is identifying the activity performed by an individual from a predefined set of sports activities, using sensor data. For detailed information of all datasets used see Appendix C.
5.2 Model Architecture Details
Teacher models. We use deep fully connected (FC) networks for classification on Ellipsoid and convolutional networks for Cube. For image classification on CIFAR-10 and ImageNet dataset we use publicly available, pretrained ResNet models. We train reference DenseNet models for the
(a) Cube
(b) Ellipsoid
(d) CIFAR-100
(e) TinyImageNet
(f) ImageNet-1k
(g) DSA-19
Figure 2: Accuracy vs. inference-time trade-offs. Inference time is reported as a fraction of teacher's inference time along with average ensemble accuracy and error bars. B-DISTIL performs this trade-off at runtime. The baseline NO-RESHED at inference time  $\tau_w$  ( $x$ -axis) is the accuracy of a single model that is allowed  $|\tau_w - 0|$  time for inference. Similarly the baseline RESHED at  $\tau_w$  is the accuracy of an ensemble of models, where the model  $w$  is allowed  $|\tau_w - \tau_{w-1}|$  time to perform its inference. This is also the latency between the successive predictions from B-DISTIL. We can see that B-DISTIL (green) remains competitive to the oracle baseline (NO-RESCHED, blue) and outperforms weighted averaging (RESCHED, yellow).
(h) Google-13
CIFAR-100 dataset based on publicly available training recipes (see Appendix C). As both spoken audio data (Google-13) and sensor-data (DSA-19) are time series classification problems, we use recurrent neural networks (RNNs). We train an LSTM-based architecture [24] on Google-13 and a GRU-based architecture [10] on DSA-19. Except for the pretrained ResNet models, all other teacher models are selected based on performance on validation data.
Student models. For all distillation tasks, for simplicity we design the student base model class from the same architecture type as the teacher model, but start with significantly fewer parameters and resource requirements. We train for at most  $T = 7$  rounds, keeping  $\eta = 1$  in all our experiments. Whenever FIND-WL fails to find a weak learner, we expand the base class  $\mathcal{F}$  using the connection specified as a hyperparameter. Since we need only at most  $T = 7$  weak learners, we can pick small values of  $R$  (say, 2). The details of the intermediate connections used for each dataset, hyperparameters such as the regularization parameter  $\gamma$  in Equation 6 and hyperparameters for SGD can be found in Appendix C and D
5.3 Experimental Evaluation and Results
First, we present the trade-off between accuracy and inference time offered by B-DISTIL in the context of anytime inference and early prediction. We compare our models on top-1 classification accuracy and total floating point operations (FLOPs) required for inference. We use a publicly available profiler [35] to measure floating point operations. For simplicity of presentation, we convert these to the corresponding inference times  $(\tau)$  on a reference accelerator (NVIDIA 3090Ti).
Anytime inference. As discussed previously, in the anytime inference setting a model is required to produce a prediction even when its execution is interrupted. Standard model architectures can only output a prediction once the execution is complete and thus are unsuitable for this setting. We instead compare against the idealized baseline where we assume oracle access to the inference budget which is usually only available after the execution is finished or is interrupted. Under this assumption, we can train a set of models suitable various inference time constraints, e.g., by training models at various depths, and then pick the one that fits the current inference budget obtained by querying the oracle. We refer to this baseline as NO-RESHED and compare B-DISTIL to it on both synthetic and real world datasets in Figure 2. This idealized baseline can be considered an upper bound on the accuracy of B-DISTIL for a fixed inference budget.
Dataset	Algorithm	Early-prediction Acc
T = 50%	T = 75%	T = 100%
Acc (%)	Frac	Acc (%)	Frac	Acc (%)
Google-13	E-RNN	88.31	0.48	88.42	0.65	92.43
B-DISTIL	87.41	0.49	89.31	0.71	92.25
DSA-19	E-RNN	83.5	0.55	83.6	0.56	86.8
B-DISTIL	82.1	0.53	84.1	0.58	87.2
Table 1: Early prediction performance. Performance of the ensemble produced by B-DISTIL to the E-RNN algorithm [11]. The accuracy and the cumulative fraction of the data early predicted at  $50\%$ ,  $75\%$  and  $100\%$  time steps are shown. At  $T = 100$ , frac. evaluated is 1.0. The ensemble output by B-DISTIL with the early-prediction loss is competitive to the E-RNN algorithm. Unlike E-RNN, a method developed specifically for early prediction of RNNs, B-DISTILL is more generally applicable across model architectures and can also be used for offline.
B-DISTIL can improve on its initial prediction whenever inference jobs are allowed to be rescheduled. To contextualize this possible improvement, we consider the case where the execution is interrupted and rescheduled (with zero-latency, for simplicity) at times  $\{\tau_1,\tau_2,\dots ,\tau_W\}$ . We are required to output a prediction at each  $\tau_w$ . As an idealized baseline, assume we know these interrupt points in advance. One possible solution then is as follows: select models with inference budgets  $|\tau_{1}|,|\tau_{2} - \tau_{1}|,\ldots ,|\tau_{w} - \tau_{w - 1}|$ . Sequentially evaluate them and at each interrupt  $\tau_w$ , and output the (possibly weighted) average prediction of the  $w$  models. We call this baseline RESCHED. Since the prediction at  $\tau_w$  is a weighted average of models, we expect its performance to serve as a lower-bound for the performance of B-DISTIL. In the same figure (Figure 2) we compare B-DISTIL to RESHED.
We see that at all interrupts points in Figure 2, the predictions provided by B-DISTIL are competitive to that of the idealized baseline RESHED which requires the inference budget ahead of time for model selection, while being able to improve on its initial predictions if rescheduled. For instance, for the CIFAR-100 dataset and at the interrupt point at 0.5 on the  $x$ -axis, the predictions produced by B-DISTIL are comparable to a single model of the same inference duration, while being able to allow interrupts at all the previous points.
Early prediction. To evaluate the applicability of our method for early prediction in online time-series inference, we compare the performance of B-DISTIL to that of E-RNN from [11]. Unlike B-DISTIL, which can be applied to generic architectures, E-RNN is a state-of-the-art method for early prediction that was developed specifically for RNNs. When training, we set the classification loss to the early-classification loss used in E-RNN training. We evaluate our method on the time-series datasets GoogleSpeech and DSA. The performance in terms of time-steps evaluated is compared in Table 1. Here, we see that B-DISTIL remains competitive to E-RNN for early prediction. Unlike E-RNN, B-DISTIL also offers early prediction for offline/batch evaluation time-series data. For such cases, a threshold can be tuned similar to E-RNN and B-DISTIL can evaluate the models in its ensemble in order of increasing cost, exiting when the prediction score crosses this threshold.
5.4 Training Considerations and Scalability
Connections. Our method uses intermediate connections to improve its performance. Although these connections are designed to be efficient, they still have an overhead cost over an averaging based ensemble. The FLOPs required to evaluate intermediate connections corresponding to the distillation tasks in Figure 2 is shown in Figure 3. Here, we compare the FLOPs required to evaluate the model from round  $T$  to the FLOPs required to evaluate the intermediate connections used by this model. Note that summing up all the FLOPs up to a round  $T$ , in Figure 3 gives the total FLOPs required to for the ensemble with the first  $T$  models. For all our models, the overhead of connections is negligible when compared to the inference cost of the corresponding model. To evaluate the benefits offered by the intermediate connections, we can compare the results of B-DISTIL run with connections and B-DISTIL without connections. The latter case can be thought of as running the AdaBoost algorithm for distillation. Note that this is the same as the RESCHED baseline (weighted averaging).
On comparing the B-DISTIL plot in Figure 2 to the plot of RESCHED highlights the benefit of using intermediate connections. As in this work our focus is on finding weak learners in the presence of capacity gap, and we do not explore additional compression strategies like quantization, hard thresholding, low-rank projection that can further reduce inference cost.
(a) Google-13
(b) DSA-19
(c) CIFAR10
Figure 3: Overhead of connections. The floating point operations required to evaluate the model added in round  $T$ , compared to that required to evaluate just the connections used by this model. We present the results corresponding to datasets that have models with smaller required FLOPs overall. We see that even for these models the connections add relatively little overhead.
Overheads of training/distillation. B-DISTIL requires additional compute and memory compared to a single model's training. Maintaining the two matrices  $K_{t}^{+}$  and  $K_{t}^{-}$  requires an additional  $\mathcal{O}(NL)$  memory. Even for relatively large datasets with, say,  $N = 10^{6}$  samples and  $L = 1000$  classes, this comes out to a few gigabytes. Note that the two matrices can be stored in on disk and a batch can be loaded into memory for loss computation asynchronously using data loaders provided by standard ML frameworks. Thus the additional GPU memory requirement is quite small,  $\mathcal{O}(bL)$  for a mini-batch size  $b$ , same as the memory required to maintain one-hot classification labels for a mini-batch. Since gradient computation is required only for the model being trained in each round, which typically is smaller than the teacher model, the backward pass is relatively cheap. For large datasets, loading the data for the teacher model forward pass becomes the bottleneck. See Appendix D for discussion on data loading, training time, and resource requirements for a ImageNet distillation.
Sorting  $\{\mathcal{F}\}_{t}$  using profiling. As mentioned in Section 3 B-DISTIL assumes the hypothesis classes in  $\{\mathcal{F}\}_{t}$  are ordered on some metric, say, inference time. In practice, we achieve this at run-time by starting with a small base model, and profiling subsequent models considered in later rounds at run-time. For example, in PyTorch, we can use the torch.autograd.profiler module to profile the forward pass of a model. As a heuristic, we then sort the models in  $\{\mathcal{F}\}_{t}$  based on the average inference time of the models in the ensemble.
6 Limitations, Broader Impact, and Future Work
In this paper we explore the problem of progressive ensemble distillation, in which the goal is to produce an ensemble of small weak learners from a large model, where components of the ensemble enable progressively better results as the ensemble size increases. To address this problem we propose B-DISTIL, an algorithm for progressive ensemble distillation, and demonstrate that it can be useful for a number of applications in efficient inference. In particular, our approach allows for a straightforward way to trade off accuracy and compute at inference time, and is critical for scenarios where inference may be interrupted abruptly or where variable levels of accuracy can be tolerated. We experimentally demonstrate the effectiveness of B-DISTIL by decomposing well-established deep models onto ensembles for data from vision, speech, and sensor domains. Our procedure leverages a stochastic solver combined with log barrier regularization for finding weak learners, use profiling for model selection and use intermediary connections to circumvent the issue of model capacity gap.
A key insight in this work is that posing distillation as a two player zero-sum game allows us to abstract away model architecture details into base class construction  $\mathcal{F}$ . This means that, conditioned on us finding a 'weak learner' from the base class, we retain the guarantees of the traditional boosting setup. Since these weak learners are only required to produce small improvements between rounds, we are able to reliably find such models. A caveat and potential limitation of this abstraction is that the user must design  $\mathcal{F}$ . Our research primarily focuses on on-device continuous inference, but our distillation procedure also holds benefits for cloud/data-center inference settings. This includes tasks like layer-fusion, load-balancing, and improved resource utilization, which merit further investigation in future studies. Finally, it is important to mention that our work prioritizes optimizing model accuracy while enabling compressed forms. However, another area for future research is exploring additional impacts of our approach on metrics such as fairness or robustness [25, 26, 32].
This work was supported in part by National Science Foundation Grants IIS2145670 and CCF2107024, a Meta Faculty Award, and the Private AI Collaborative Research Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the National Science Foundation or any other funding agency.
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2022.
[2] Jose M Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. Advances in Neural Information Processing Systems, 2017.
[3] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of Machine Learning and Systems, 2020.
[4] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, 2017.
[5] Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In International Conference on Knowledge Discovery and Data-mining, 2006.
[6] Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[7] Changan Chen, Frederick Tung, Naveen Vedula, and Greg Mori. Constraint-aware deep neural network compression. In Proceedings of the European Conference on Computer Vision, 2018.
[8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In International Conference on Knowledge Discovery and Data-mining, 2016.
[9] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision, 2019.
[10] Kyunghyun Cho, Bart Van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
[11] Don Dennis, Chirag Pabbaraju, Harsha Vardhan Simhadri, and Prateek Jain. Multiple instance learning for efficient sequential data classification on resource-constrained devices. Advances in Neural Information Processing Systems, 2018.
[12] Don Dennis, Durmus Alp Emre Acar, Vikram Mandikal, Vinu Sankar Sadasivan, Venkatesh Saligrama, Harsha Vardhan Simhadri, and Prateek Jain. Shallow rnn: accurate time-series classification on resource constrained devices. Advances in Neural Information Processing Systems, 2019.
[13] Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmerst, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for elastic inference. arXiv preprint arXiv:2310.07707, 2023.
[14] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
[15] Yoav Freund. Boosting a weak learning algorithm by majority. In Workshop on Computational Learning Theory, 1990.
[16] Yoav Freund and Robert Schapire. Game theory, on-line prediction and boosting. Conference on Learning Theory, 2000.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
[38] Robert E Schapire. The strength of weak learnability. Machine learning, 1990.
[39] Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Kybernetes, 2013.
[40] Arun Suggala, Bingbin Liu, and Pradeep Ravikumar. Generalized boosting. Advances in Neural Information Processing systems, 2020.
[41] Luca Trevisan, Madhur Tulsiani, and Salil Vadhan. Regularity, boosting, and efficiently simulating every high-entropy distribution. In IEEE Conference on Computational Complexity, 2009.
[42] John von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 1944.
[43] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition, 2018.
[44] Li Yang and Deliang Fan. Dynamic neural network to enable run-time trade-off between accuracy and latency. In Asia and South Pacific Design Automation Conference. Association for Computing Machinery, 2021.
[45] Sean I Young, Wang Zhe, David Taubman, and Bernd Girod. Transform quantization for cnn compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[46] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018.
A Two-Player Minimax Games
In this section, we will look at the setting of two-player minimax games more closely.
Consider a class of hypotheses  $\mathcal{F}$  and class of probability distributions  $\mathcal{P}$ . In addition, consider a loss function  $\mathcal{L}:\mathcal{F}\times \mathcal{X}\to \mathbb{R}$  that is convex in its first argument. We consider two players whose pure strategy sets are  $\mathcal{F}$  and  $\mathcal{P}$  respectively. The loss and reward of the players is given by  $F(f,\mu) = \mathbb{E}_{x\sim p}[L(f,x)]$  and the minmax value of the game is
$$
\max  _ {p \in \mathcal {P}} \min  _ {f \in \mathcal {F}} F (f, \mu). \tag {9}
$$
Note that this game is convex in the hypothesis player and concave in the distribution player. The objective of the game for the hypothesis player is trying to find a hypothesis that has low loss on the worst case distribution from class  $\mathcal{P}$ . Conversely, the distribution player is trying to construct a distribution that is as hard as possible for the hypothesis player to learn.
Also, note that under reasonable conditions on  $\mathcal{F}$  and  $\mathcal{P}$ , we have the minimax theorem holds (see Chapter 6, Schapire and Freund [39]),
$$
\max  _ {p \in \mathcal {P}} \min  _ {f \in \mathcal {F}} \mathbb {E} _ {x \sim p} [ L (f, x) ] = \min  _ {f \in \Delta (\mathcal {F})} \max  _ {p \in \mathcal {P}} \mathbb {E} _ {x \sim p} [ L (f, x) ]. \tag {10}
$$
Here,  $\Delta (\mathcal{F})$  is the set distributions over functions  $\mathcal{F}$ . From this, we can see that as long as we are allowed to aggregate functions from the base class, we have can do as well as we could if we had access to the distribution
The interesting algorithmic question would be to find a distribution over hypotheses that achieves the minimum above. Note that the loss function is stochastic and thus, we need to formalize the access the player has to the loss function. We will formulate this in the following stochastic way.
Definition 2. An algorithm ORACLE is said to be a  $(\beta, \delta)$  weak-gradient if
$$
\langle \operatorname {O R A C L E} (f, p), \nabla_ {f} F (f, p) \rangle \geq \beta \tag {11}
$$
Here  $\nabla_{f}$  denotes the functional gradient of  $F$ . This notion is similar to the weak learning assumptions usually used in the boosting literature. Given such an oracle one can ask for methods similar to first order methods for convex optimization, such as gradient descent, to solve the minimax problem. These algorithms iteratively maintain candidate solutions for the both the players and update each of these using feedback from the state of the other player. In our particular setting, the hypothesis player updates using the vector  $h$  in eq. (3).
Motivating Example. Let  $\mathcal{F}$  be a class of hypotheses, let  $\mathcal{P}$  is the set of all distributions over a finite sample set  $\{x_1,\ldots ,x_n\}$  and let  $L$  be the 0-1 loss. Note that in this setting, the oracle from Definition 2 is analogous to weak learning. In this setting, from the minmax theorem and the existence of a weak learner, we get that there is a single mixture of hypothesis of  $\sum_{i}\alpha_{i}f_{i}$  such that loss under every distribution in  $\mathcal{P}$  which corresponds to zero training error. Thus we can think of boosting algorithms as approximating this minmax equilibrium algorithmically. Similarly, the weak learning condition in [40] is similar in spirit to the condition above.