SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With the condition from Definition 2, one can consider many frameworks for solving minimax games. One general technique is to consider two no-regret algorithms for online convex optimization to play against each other. Let us briefly look at the definition of regret in this setting.
Definition 3 (No-Regret). Let  $K, A$  be convex sets. At each time, a player observes a point  $x_{t} \in K$  and chooses an action  $a_{t} \in A$ . The regret of the algorithm is defined as
$$
R _ {T} = \max  _ {a \in A} \sum_ {t = 1} ^ {T} \langle a, x _ {t} \rangle - \sum_ {t = 1} ^ {T} \langle a _ {t}, x _ {t} \rangle . \tag {12}
$$
Online learning is a well-studied area of machine learning with a rich set of connections to various areas in mathematics and computer science. In particular, there are frameworks in order to construct algorithms such as follow-the-perturbed leader, follow-the-regularized leader and mirror descent.
Our algorithm can be seen as a version of mirror descent with the entropy regularizer and Theorem 1 as a version of the regret guarantee for the algorithm. In addition to those mentioned above, there are several other frameworks considered to solve minimax games such as variational inequalities, extragradient methods, optimistic methods, etc. We believe this general framework is a useful one to consider for many learning tasks, especially in settings where we have function approximation.
B Proofs of Main Theorems
Here, we provide a proof of Theorem 1, which is restated below:
Theorem. Suppose the class  $\mathcal{F}$  satisfies that for all  $f\in \mathcal{F}$ ,  $\| f - g\|_{\infty}\leq G_{\infty}$ . Let  $F = \{f_t\}$  be the ensemble after  $T$  rounds of Algorithm 1, with the final output  $F_{t} = \frac{1}{T}\sum_{t = 1}^{T}f_{t}$ . Then for  $T\geq \ln 2N$  and
$$
\eta = \frac {1}{G _ {\infty}} \sqrt {\frac {\ln 2 N}{T}}
$$
we have for all  $j$
$$
\| F _ {t, j} - g _ {j} \| _ {\infty} \leq G _ {\infty} \sqrt {\frac {\ln 2 N}{T}} - \frac {1}{T} \sum_ {t = 1} ^ {T} \gamma_ {t} (j)
$$
Proof. For simplicity, we assume that  $f_{t}$  and  $g$  are scalar valued functions, since the proof goes through coordinate-wise. At each time, define the edge of the weak learning algorithm to be
$$
\gamma_ {t} = \sum_ {i} K _ {t} ^ {+} (i) \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) + \sum_ {i} K _ {t} ^ {-} (i) \left(g \left(x _ {i}\right) - f _ {t} \left(x _ {i}\right)\right)
$$
Let  $Z_{t}$  denote the normalizing constant at time  $t$ , that is,
$$
Z _ {t} = \sum_ {i} K _ {t} ^ {+} (i) \exp \left(- \eta \left(f _ {t} \left(x _ {i}\right) - g (x _ {i})\right)\right) + K _ {t} ^ {-} (i) \exp \left(\eta \left(f _ {t} \left(x _ {i}\right) - g (x _ {i})\right)\right)
$$
From the update rule, we have
$$
\begin{array}{l} K _ {T + 1} ^ {+} (i) = \frac {K _ {T} ^ {+} (i) e ^ {\eta \left(f _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right)\right)}}{Z _ {T}} \\ = \frac {K _ {1} ^ {+} (i) \exp \left(- \eta \sum_ {t = 1} ^ {T} \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) _ {j}\right)}{\prod_ {t = 1} ^ {T} Z _ {t}} \\ = \frac {K _ {1} ^ {+} (i) \exp \left(- \eta T \left(F _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) \right.}{\prod_ {t = 1} ^ {T} Z _ {t}} \\ \end{array}
$$
$$
K _ {T + 1} ^ {-} (i) = \frac {K _ {1} ^ {-} (i) \exp \left(\eta T \left(F _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) \right.}{\prod_ {t = 1} ^ {T} Z _ {t}}
$$
First, we bound  $\ln (Z_t)$ :
$$
\begin{array}{l} \ln (Z _ {t}) = \ln \left(\sum_ {i} K _ {t} ^ {+} (i) \exp (- \eta (f _ {t} (x _ {i}) - g (x _ {i}))) + \sum_ {i} K _ {t} ^ {-} (i) \exp (\eta (f _ {t} (x _ {i}) - g (x _ {i})))\right) \\ \leq \ln \left(\sum_ {i} K _ {t} ^ {+} (i) \left(1 - \eta \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) + \eta^ {2} \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) ^ {2}\right) \right. \\ \left. + \sum_ {i} K _ {t} ^ {-} (i) \left(1 + \eta \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) + \eta^ {2} \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) ^ {2}\right)\right) \\ \leq \ln \left(1 - \eta \sum_ {i} K _ {t} ^ {+} (i) \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) + \eta \sum_ {i} K _ {t} ^ {-} (i) \left(f _ {t} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) + \eta^ {2} G _ {\infty} ^ {2}\right) \\ \leq - \eta \gamma_ {t} + \eta^ {2} G _ {\infty} ^ {2} \\ \end{array}
$$
where the second step follows from the identity  $\exp (x)\leq 1 + x + x^2$  for  $x\leq 1$ , provided that  $\eta \leq \frac{1}{G_{\infty}}$ . This gives us a bound on regression error after  $T$  rounds:
$$
\begin{array}{l} - \eta T \left(F _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) = \ln \left(K _ {T + 1} ^ {+} (i)\right) - \ln \left(K _ {1} ^ {+} (i)\right) + \sum_ {t = 1} ^ {T} \ln \left(Z _ {t}\right) \\ \leq \ln \left(\frac {K _ {T + 1} ^ {+} (i)}{K _ {1} ^ {+} (i)}\right) + \sum_ {t = 1} ^ {T} - \eta \gamma_ {t} + \eta^ {2} G _ {\infty} ^ {2} \\ = \ln \left(\frac {K _ {T + 1} ^ {+} (i)}{K _ {1} ^ {+} (i)}\right) + \eta^ {2} T G _ {\infty} ^ {2} - \eta \sum_ {t = 1} ^ {T} \gamma_ {t} \\ \leq \ln 2 N + \eta^ {2} T G _ {\infty} ^ {2} - \eta \sum_ {t = 1} ^ {T} \gamma_ {t}, \\ \end{array}
$$
where the last bound follows since  $K_{1}^{+} = \frac{1}{2N}$  and  $K_{T + 1}^{+}\leq 1$ . Similarly, we have the bound
$$
\eta T \left(F _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) \leq \ln 2 N + \eta^ {2} T G _ {\infty} ^ {2} - \eta \sum_ {t = 1} ^ {T} \gamma_ {t}
$$
Combining the two equations we get that
$$
\sup  _ {i} \left| F _ {T} \left(x _ {i}\right) - g \left(x _ {i}\right) \right| = \| F _ {T} - g \| _ {\infty} \leq \frac {\ln 2 N}{\eta T} + \eta G _ {\infty} ^ {2} - \frac {1}{T} \sum_ {t = 1} ^ {T} \gamma_ {t}.
$$
If we choose  $\eta = \frac{1}{G_{\infty}}\sqrt{\frac{\ln 2N}{T}}$  to minimize this expression, then we get the following bound on regression error:
$$
\| F _ {t} - g \| _ {\infty} \leq - \frac {1}{T} \sum_ {t = 1} ^ {T} \gamma_ {t} + G _ {\infty} \sqrt {\frac {\ln 2 N}{T}}.
$$
which is exactly Equation (8). Note that the value of  $\eta$  only satisfies the condition  $\eta \leq \frac{1}{G_{\infty}}$  when  $T \geq \ln 2N$ , which is the time horizon after which the bound holds. This finishes the proof of Theorem 1.
Now, we provide a proof of Theorem 2 which follows from the VC dimension bound and Theorem 1. Before we begin, we setup some notation. Given a function  $f$ , distribution  $\mathcal{D}$  over space  $\mathcal{X} \times \mathcal{Y}$  where  $\mathcal{X}$  is the input space and  $\mathcal{Y}$  is the label space, and data  $D$  consisting of  $N$  iid samples  $(x,y) \sim \mathcal{D}$ , we define
$$
\widehat {\operatorname {e r r}} (f) = \Pr_ {(x, y) \sim D} [ \operatorname {s i g n} (F _ {T} (x) \neq y) ] \quad \operatorname {e r r} (f) = \Pr_ {(x, y) \sim \mathcal {D}} [ \operatorname {s i g n} (F _ {T} (x) \neq y) ]
$$
Theorem (Excess Risk). Suppose data  $D$  contains of  $N$  iid samples from distribution  $\mathcal{D}$ . Suppose that the function  $g$  has large margin on data  $D$ , that is
$$
\operatorname * {P r} _ {x \sim D} \left[ | g (x) | <   \epsilon \right] <   \mu
$$
Further, suppose that the class  $\mathcal{C}_T$  has VC dimension  $d$ , then for
$$
T \geq \frac {4 G _ {\infty} ^ {2} \ln 2 N}{\epsilon^ {2}},
$$
with probability  $1 - \delta$  over the draws of data  $D$ , the generalization error of the ensemble  $F_{T}$  obtained after  $T$  round of Algorithm 1 is bounded by
$$
\operatorname {e r r} \left(F _ {T}\right) \leq \widehat {\operatorname {e r r}} (g) + O \left(\sqrt {\frac {d \ln (N / d) + \ln (1 / \delta)}{N}}\right) + \mu
$$
Proof. Recall the following probability bound Schapire and Freund [39, theorem 2.5] which follows Sauer's Lemma:
$$
\Pr \left[ \exists f \in \mathcal {C} _ {T}: \operatorname {e r r} (f) \geq \widehat {\operatorname {e r r}} (f) + \epsilon \right] \leq 8 \left(\frac {m e}{d}\right) ^ {d} e ^ {- m \epsilon^ {2} / 3 2}
$$
which holds whenever  $|D| = N \geq d$ . It follows that with probability  $1 - \delta$  over the samples, we have for all  $f \in \mathcal{C}_{\mathcal{T}}$
$$
\operatorname {e r r} (f) \leq \widehat {\operatorname {e r r}} (f) + O \left(\sqrt {\frac {d \ln (N / d) + \ln (1 / \delta)}{N}}\right) \tag {13}
$$
Since we choose  $T = \frac{4G_{\infty}^{2}\ln 2N}{\epsilon^{2}}$ , by Theorem 1, we have
$$
\forall x \in D: \| F _ {t} - g \| _ {1} \leq G _ {\infty} \sqrt {\frac {\ln 2 N}{T}} \leq \frac {\epsilon}{2}
$$
Since  $g$  has  $\epsilon$  margin on data with probability  $1 - \mu$ , we have
$$
\widehat {\operatorname {e r r}} \left(F _ {t}\right) \leq \widehat {\operatorname {e r r}} (g) + \mu \tag {14}
$$
Combining eqs. (13) and (14), we get
C Dataset Information and Training Recipe
We use six publicly available real world datasets in our experiments. The train-test splits for all the dataset as well as the sources are listed here:
Dataset	Train-samples	Test/Val-samples	Num.-labels	Source
CIFAR-10	50000	10000	10	[29]
CIFAR-100	50000	10000	100	[29]
DSA-19	6800	2280	19	[14]
Google-13	52886	6835	13	[43]
ImageNet-1k	1281167	50000	1000	[37]
TinyImageNet-200	100000	10000	200	[30]
We use two synthetic datasets in our experiments, ellipsoid and cube. To construct the ellipsoid dataset, we first sample a  $32 \times 32$  matrix  $B$ , each entry sampled iid. We define  $A := B^T B$  as our positive semi-definite matrix, and  $I[x^T A x \geq 0]$  determines the label of a data point  $x$ . We sample 10k points uniform randomly from  $[-1, 1]^{32}$  and determine their labels to construct our data sample. We randomly construct a 80-20 train-test split for our experiments.
To construct cube, we first sample 16 vertices uniform randomly from  $[-1,1]^{32}$  and split them into 4 equal sets, say  $\{S_1,\ldots ,S_4\}$ . As before, we sample 10k points uniformly from  $[-1,1]^{32}$  and determine the label  $y(x)$  of each point  $x$  based on the closest vertex in  $\{S_1,\dots ,S_4\}$ .
$$
y (x) = \arg \min  _ {i} \min  _ {x ^ {\prime} \in S _ {i}} \| x - x ^ {\prime} \|.
$$
C.2 Training Recepies
We use stochastic gradient descent (SGD) with momentum for all our experiments. For experiments on CIFAR100 and CIFAR10, we use a learning rate of 0.1, a momentum parameter of 0.9, and weight decay of  $5 \times 10^{-4}$ . We train for 200 epochs and reduce the learning rate by a factor of 0.2 in after
Teacher model	Residual Blocks	Embedding dims	Strides
ResNet56	1	8	1
2,2	8,8	1,1
2,2	16,16	1,2
2,2,3	16,32,64	1,2,2
Table 2: Base model configuration used for ResNet56 distillation on CIFAR-10.
Teacher model	Blocks	growth-rate
DenseNet121	4, 8	12
4, 8, 8	6
8, 16, 12	6
30%, 60% and 90% of the epoch execution. We perform a 4-GPU data-parallel training for ImageNet with a per-gpu batch size of 256, learning rate 0.1, momentum 0.9, regularization  $\gamma$  of 1.0, and a weight decaur of  $1e - 4$ . We train for 90 epochs with and discount the learning rate by a factor of 0.1 at 30% and 60% epochs. For experiments with time series data, Google-13 and DSA-19, we use a fixed learning rate of 0.05 and a momentum of 0.9. We do not use weight decay or learning rate scheduling for time-series data.
C.3 Profiling Based Model Selection
To estimate execution latency, we leverage a third-party library, such as the Deep Speed framework [35], which enables us to measure the total floating-point operations required for inference. By randomly initializing a single sample with a batch size of 1, we obtain the FLOPs values for various real-world datasets. This profiling process serves as a reliable indicator of real-world performance, allowing us to rank the models based on their profiles. Notably, the profiling results only need to hold relative to other candidate models. Furthermore, we can utilize existing profiling models, such as ARM Fast Models for ARM-based mobile platforms, QEMU for x86 platforms, and NVIDIA NSight's GPU emulators for GPU platforms, to estimate the performance on different hardware architectures. However, it's important to note that these software-based solutions provide approximate performance estimates, and accurate evaluation of real performance requires access to the actual hardware. We are not restricted to latency based rankings. In fact, other useful metrics like throughput in sample processed per second, can also be used for candidate model selection.
D Base Model Configuration
The base class configurations used for all our experiments in Figure 2 is provided in Tables 2, 3, 6 and 7. Note that we use standard model architectures implemented in Pytorch and the parameters correspond to the corresponding function arguments in these implementation. We use pretrained models from the torchvision library.
Table 3: Configuration used for DenseNet121 distillation on CIFAR-100.
Teacher model	Residual Blocks	Embedding dims	Strides
ResNet110	2, 2	16, 16	1,2
2, 3	32, 64	1,2
2, 2, 3	16, 32, 64	1,1,2
2, 3, 3	16, 32, 64	1,2,2
Table 4: Base models for TinyImageNet.
Teacher model	Residual Blocks	Embedding dims	Strides
ResNet101	2, 2	16, 16	1,2
2, 3	32, 64	1,2
2, 2, 3	16, 32, 64	1,1,2
2, 3, 3	32, 32, 64	1,2,2
Table 5: Base models for ImageNet.
Teacher model	hid. dims.
LSTM128	4,4
16,8
20,12
20,32
Table 6: Configuration used for LSTM128 distillation on Google-13.
Teacher model	hid. dims.
GRU32	4,4
8,16
16,16
32,16
Table 7: Configuration used for GRU32 distillation on DSA-19.
E.1 Connections and parallel execution schedules.
Our focus in this work has been sequential execution of the models. While reusing previously computed features is clearly beneficial for finding weak learners in this setup, the presence of connections across models prevent them from being scheduled together for execution whenever otherwise possible. To manage this trade off between parallelization and expressivity, we try to restrict the number of connections to at most one between models, and further restrict the connection to later layers of the model. Connections in the later layers of networks impose fewer sequential blocks in inference and allows for better parallelization.
Let  $\phi_{t,l}(x)$  denote the activation produced at layer  $l$  by the weak learner at round  $t$ , on data-point  $x$ . Then some of the connections we consider are
-  $\phi_{(t,l)}(x) - \phi_{(t + 1,l)}(x)$ , to learn the error in features at layer  $l$ .
-  $\phi_{(t,l)}(x) + \mathrm{id}(x)$ , standard residual connection at layer  $l$ .
-  $\phi_{(t + 1,l)}[\phi_{(1,l)}(x),\ldots ,\phi_{t,l})(x)]$  , dense connections at layer  $l$  across rounds.
- Simple accumulation operations.
- Recurrent networks: LSTM and GRU.
E.2 Training ImageNet and Throughput Optimization
Large datasets like ImageNet require additional care to ensure good resource utilization. Since the backward pass of our training only involves a single student model, it can be computed quite efficiently even for large datasets. This is particularly true in earlier rounds. For such cases we use simple producer consumer architecture where a GPU is dedicated to producing and queuing data batches and target predictions produced by the teacher. Training routines running on separate GPUs consume these batches for their training.
As mentioned in Section 3, since B-DISTIL only cares that it is provided with an ordered list of candidates, we can modify the weak learning finding step to prefer weak learners that provide higher inference throughput, measured as samples processed per second. Throughput metric is more meaningful in the cloud inference deployment setting where batched inference is the norm. The ability to trade a small performance for the number of inference requests to process can be useful in such settings. Implementation details can be found in the included code base. We note that we do not optimize for throughput and inference latency simultaneously in this work, but is an interesting direction of future research.
(a) Sequential schedule
(b) Hybrid schedule
Figure 4: A schematic description of a sequential execution scheme for an ensemble of four models, being evaluated one after the other from left to right. The last model in the ensemble reuses the actions of the previous one, causing a blocking dependency. Thus we cannot trivially execute all models in parallel. However, since the connection is between the last layers of the network, we can construct hybrid execution schemes as in (b). Here, pairs of models are executed together.