On Anytime Learning at Macroscale
Authors: Lucas Caccia
Jing Xu
Myle Ott
MarcAurelio Ranzato
Ludovic Denoyer
Venue: ICLR 2022 Submitted (2022)
Keywords: Anytime Learning
Mixture of experts
growing architectures

Abstract:  Classical machine learning frameworks assume access to a possibly large dataset in order to train a predictive model. In many practical applications however, data does not arrive all at once, but in batches over time. This creates a natural trade-off between accuracy of a model and time to obtain such a model. A greedy predictor could produce non-trivial predictions by immediately training on batches as soon as these become available but, it may also make sub-optimal use of future data. On the other hand, a tardy predictor could wait for a long time to aggregate several batches into a larger dataset, but ultimately deliver a much better performance.  In this work, we consider such a streaming learning setting,  which we dub {\em anytime learning at macroscale} (ALMA). It is an instance of anytime learning applied not at the level of a single chunk of data, but at the level of the entire  sequence of large batches. We first formalize this learning setting, we then introduce metrics to assess how well learners perform on the given task for a given memory and compute budget, and finally we test about thirty baseline approaches on three standard benchmarks repurposed for anytime learning at macroscale. Our findings indicate that no model strikes the best trade-off across the board. While replay-based methods attain the lowest error rate, they also incur in a 5 to 10 times increase of compute. Approaches that grow capacity over time do offer better scaling in terms of training flops, but they also underperform simpler ensembling methods in terms of error rate. Overall, ALMA offers both a good abstraction of the typical learning setting faced everyday by practitioners, and a set of unsolved modeling problems for those interested in efficient learning of dynamic models.

Under review as a conference paper at ICLR 2022
ON ANYTIME LEARNING AT MACROSCALE
Anonymous authors Paper under double-blind review
ABSTRACT
Classical machine learning frameworks assume access to a possibly large dataset order train a predictive model. In many practical applications however, data does not arrive all at once, but in large batches over time. This creates a natural trade-off between accuracy of a model and time to obtain such model. A greedy predictor could produce non-trivial predictions by immediately training on batches as soon these become available but, it may also make sub-optimal use of future data. On the other hand, tardy predictor could wait for long time to aggregate several batches this work,
into larger dataset, but ultimately deliver much better performance. consider such streaming learning setting, which we dub anytime learning macroscale (ALMA). It is an instance of anytime learning applied not at the level of single chunk of data, but at the level of the entire sequence of large batches. We first formalize this learning setting, we then introduce metrics to assess how well learners perform and finally we test about thirty baseline approaches on three standard benchmarks repurposed for anytime learning at macroscale. Our findings indicate that model strikes the best trade-off off across the board. While replay-based methods attain the lowest error rate, they also incur in 5 10 times increase of compute. Approaches that grow capacity over time do offer better scaling in terms of training flops, but they also underperform simpler ensembling methods terms of error rate. Overall, ALMA offers both good abstraction of the typical learning setting faced everyday by practitioners, and a set of unsolved modeling problems for those interested efficient learning of dynamic models.
the given task for given memory and compute budget,
INTRODUCTION
Empirical risk minimization (Vapnik 1998) is the dominant framework to formalize the learning process of a supervised task, and it has been critical to the success of large scale training of deep learning systems on wide variety of applications. Within this framework, training data is assumed to be provided to the learner all at once. Alternatively, when the dataset is very large (essentially infinite), data is streamed to the learner one minibatch at the time, assuming that the rate which samples are received matches the model's processing time to learn from them.
Learning over streams of data has been studied in the machine learning domain for long time (see Section 2 and Figure T for more details) with different assumptions: for instance in online learning, it usually assumed that datapoints are coming one by one and have be processed as soon as they are received. In continual learning, the streaming of data usually corresponds to a stream of large datasets corresponding to different tasks to solve, etc. In this paper, we define simple yet important setting where there is a single task to solve, and where training data often comes at slower rate than a model can process it. Moreover, it comes in relatively large batches once in a while. While poorly studied, this setting corresponds to practical applications encountered in production pipelines. For instance, it is faced by teams deploying language modeling applications (e.g content moderation) build models that are trained on large amounts of data like filtered versions of Common Crawl, which are dumps of the internet. However, new snapshots are available every month, as new content is generated over time. Therefore datasets keep getting bigger every few months and models need be retrained accordingly. Similarly, visual object recognition datasets used in deployed applications are often extended every few months to include new images with their corresponding annotations.
Authors contributed equally

---

Under review as a conference paper at ICLR 2022
Batch Learning
Concept Drift
Data once optimize final performance
Model current distribution
Online Learning
Offiine Training
On-the-iy fly training
Singla update per datapoint
Gannot overfit
dala dumpe
Datapoints arrive a time
Mutiple updates per sample
Data arrive sequentislly optimize anytime performance
Danger overfilling
Change Distribution
Model distributions
Continual Learning
ALMA
Figure 1: ALMA compared to other learning frameworks.
Practically, there are two main approaches integrate information present in new batch of data in an existing model. If a lot of computational resources are available, new and bigger model is instantiated and trained from scratch on the union of the old training set with the new batch of data. However, since this is a computationally very intensive process, retraining is typically done only rarely, once several batches of data have been collected. We call this approach "tardy" large-scale learning, since predictor is available only at a later time. Another option, particularly suitable when computational resources are scarce and a predictor is needed quickly, is to simply finetune the old model on the new data as this arrives. Note that, in that settings, methods from the data stream domain or from the online learning domain that are based on the idea of processing any datapoint just not suitable since they have been developed for different use-cases.
This trade-off is emblematic of anytime learning, a learning setting where learner has to provide good predictions at any point in time, while improving its performance over time more and more data is observed. From an anytime learning perspective, neither training a large model after all data is received nor finetuning on the newly added batch of data are not satisfying. The former approach is poor anytime learner because one needs to wait for long time before obtaining a useful predictor. The latter approach is poor anytime learner because it typically cannot leverage very well future batches of data since the model has a fixed capacity, determined on a small portion of the overall dataset and because inherently the model is trained on non i.i.d. data.
In this work, we aim at exploring this accuracy versus time trade-off of anytime learning, not at the level of single batch of data, but the macroscale of the entire sequence of batches. This is setting which more closely mimics practical applications, that we call anylime learning al mascroscale (ALMA). In this learning setting, we assume that the time to train a model negligible compared to the interval of time between two consecutive batches of data (and therefore do not care about how quickly a learner adapts to a new batch), yet efficiency matters in the sense that for the same performance predictor that uses less compute and memory is preferable. In summary, interested in a learner that i) yields high accuracy, ii) can make non-trivial predictions at any point in time while iii) limiting its computational and memory resources.
Our first contribution is to formalize the ALMA problem and to introduce metrics to evaluate learners ( We consider three different axes: error rate, memory and amount of computation. By measuring these quantities against time, via an area under the curve, we account not only for the final performance but also for the whole training trajectory over the sequence of large batches of data.
Our second contribution is an extensive empirical evaluation ($5) of various models ($4) that strike different trade-off's between accuracy and time obtain a useful predictor. In particular, we explore models that fall in between greedy finetuning and tardy large-scale learning, and investigate models that leverage batches of data at an intermediate rate. We also consider rich family of modular architectures, from plain ensembling methods to hierarchical mixture of experts, and several variants thereof, including those that have access to a replay buffer storing all previous batches of data and those that can grow capacity over time.
Our findings across three different benchmarks, including a large scale language modeling one, can be summarized as follows. a) An intermediate waiting time offers the best trade-off between accuracy and time to yield such a predictor. However, b) there is no single approach striking the best trade-off between performance and efficiency for various model sizes. c) Retraining from scratch a big model does offer the lowest error rate but sacrifices efficiency. d) Interestingly, large models the most statistically efficient even when considering small datasets (like MNIST) and fully
2

---

Under review as a conference paper at ICLR 2022
connected networks. e) While approaches to grow capacity exhibit gains in terms of computational efficiency, these do not even outpeform simple ensembles. Overall, our work points at several research opportunities to improve modeling in streaming setting of broad practical relevance, rather then pointing at any particular solution. We have also released code to reproduce our experiments and the entire platform implementing ALMA.
2 RELATED WORK
ALMA relates to several other learning frameworks: offline learning, continual learning, online learning and transfer learning as illustrated in Figure 1 i) It shares the same assumptions of classical empirical risk minimization (ERM) (Vapnik) 1998) at the level of each batch of data. However, it overall violates ERM's assumptions of i.i.d. observations, because data points come in a stream of
data chunks. ii) Because of this, ALMA relates continual learning (CL) Ring 1994: Thrun 1994
Ring 1997 Thrun 1998), with the key difference that the data distribution across batches (or tasks) assumed stationary ALMA. Therefore, ALMA can be seen as special case of CL, with single task to solve. iii) ALMA relates also to online learning (Bottou 1998) since it assumes that data are coming in a stream, an assumption also made in the concept drift literature (Lu et al. 2018). However, in online learning examples are streamed one at the time (or at random from a large dataset), while in ALMA the learner receives large batches of data sequentially In ALMA, received data can be processed multiple times opposite to the online learning setting that usually assumes that any new datapoint has to be processed as soon as it is available, and will not be reused in future updates. iv) Finally, ALMA relates more broadly adapting to a new batch of data can be interpreted as leveraging knowledge acquired on previous batches to more effciently learn from the new batch of data.
transfer learning (Pan & Yang. 2010), the problem of
Of course, ALMA relates to anytime learning (Grefenstette & Ramsey 1992: Ramsey & Grefenstette 1994). which has been recently applied compare various altoML frameworks (Liu al.) 2020). However, in this work we are not interested in assessing the anytime learning ability at the level of each chunk of data, but only at a coarser granularity, at the level of the entire stream of chunks. Inspired by Liu et al. (020), we consider the area under the curve of error rate against time to measure performance, but in order to account also for compute and memory budget, we add to our evaluation metrics also the area under the curve for memory and compute.
From the more theoretical side, there has been work about sub- bib-baggin (Bühlmann & Yu 2002) (bagging using subsets of a larger dataset) which is similar to our setting but without the sequential aspect of it. In this context, [Breiman| (1999) proposed model similar to our growing ensembling (gEns), Bühlmann & Yu (2002) studied sub-bagging as a way make the prediction of tree classifiers more robust while Zou et al. (202T) studied the consistency of the estimator in this setting. We defer to future studies the analysis of ALMA, while in this work focus the empirical evaluation.
Shifting the discussion to prior work on models that adjust their capacity dynamically, Waterhouse & Robinson (1995) introduced an approach to grow a hierarchical mixture of experts model (Jordan & Jacobs] 1994). This is a tree structured model where experts are at the leaves and gating functions are non-terminal nodes. The tree determines hierarchical partition of the input space into regions that are associated to each expert. This approach was made more efficient in later work by (Fritsch) al. 1996). In this work we consider baseline (gMoE) that extends this prior work to hierarchical mixture of experts (Eigen et al. 2014 Denoyer & Gallinari 2015 Lepikhin et al. 2020).
Growing architectures have also been studied in CL. For instance, Fernando al. 2017) and Venial
et al. (202I) proposed a modular architecture that is assembled for every task, possibly reusing previously trained modules. The major difference with our work is that in our case routing is input dependent as opposed to task dependent. Yoon et al. (2018) instead proposed a method to incrementally and smoothly add hidden units. Similarly, Wen et al. (2020) proposed a heuristic approach to automatically adjust the network depth. WWagg et al. (2017) considered growing both depth and width when finetuning to a new task. Liu et al. 20119) and Wu et al. (2020) proposed approaches to grow architectures in depth and width by leveraging Taylor approximation and greedy selection. In our work, we benchmark against this last variant. None of these approaches have been applied to the ALMA setting to date.
Finally, some of our findings are built upon and extend recent empirical evaluations studying the scaling properties of language models (Kaplan et al. 2020a Li et al., 20205). In this study,
3

---

Under review as a conference paper at ICLR 2022
confirm the conclusion that bigger models generalize better and are more statistically efficient, not only in language modeling tasks using a transformer architecture, but also in smaller scale computer vision tasks using both fully connected and convolutional architectures.
3 LEARNING SETTING
In anytime learning at macroscale (ALMA), we assume that there exists an underlying data distribution p(x, y) with input E RD and desired label y E {I,...,C) C),. Notice that extensions to regression and unsupervised learning (where is missing) are trivial, and therefore in this work we focus on classification problems for simplicity of exposition. A important property of ALMA is that data is presented to the learner as a stream SB of B consecutive batches of examples. Let 'D; be collection of N > O i.i.d. samples randomly drawn from p(æ, y), for i {1, (1,..., B). The stream is then defined as the ordered sequence Sg = [Di,... DB J. We refer to each dataset D, as mega-batch, as it is composed by a large number examples. Typically learner m IR + {1, ......... C} updates its parameters by processing mini-bate mini-batch of < N examples at the time from each mega-batch D, and by iterating several times over each mega- before being presented with the next mega-batcis Since the learner cannot access future mega- mega-batches, batches, overall the data distribution is not i.i.d., even though samples drawn from each mega-batch are i.i.d., and cross- valsivaaiioiion performed using a subset of the current mega-batch. A learner could decide to use previous mega-batches when learning on the current mega-batch, but this will increase its compute usage.
Finally, assume that the time it takes learner to update its internal parameters after having observed a mega-batch is much less than the interval between the arrival of two consecutive mega- batches. In other words, the rate at which data arrives is slower than the processing time of the model, and therefore the model could decide to iterate several times over the data at its disposal to improve its prediction accuracy.
3.1 METRICS
We evaluate learners in the ALMA setting across three axes, namely: accuracy, memory and compu- tation. Let be the time at which the 1-th mega-batch arrives; this data can be used by the model to update its parameters or it is simply aggregated to previous mega-batches for later use.
We compute the rate of model at time / (after the arrival of the I-th mega-batch) and compute the area under the curve obtained varying t from 0 till the total number of mega-batches B; the resulting cumulative error rate (CER) is:
B
CER
Σ
|m(x; 02) + yl
(I)
D
t=0
(D.
where m(x;; (c)) is the model time equipped with parameters 0,, DT is the test set, | the number of examples in the test set, and [ı[[;; 0,) & y is one if the model prediction does not match the ground truth label and zero otherwise. The outer sum computes the discrete integral of the error rate over time. CER is going to be small only when the error rate is small throughout the whole stream. CER is instead large for a tardy model that waits till the very last mega-batch to update the model, even though eventually this may obtain very low final error rate. If not perfect, CER provides a good summary of the performance of a system across time. Anyway, to fully capture the differences between two models, it is needed to have deeper look at the performance across time as illustrated in Figure for instance.
Similarly, we compute the cumulative memory usage and compute as:
B 10cl. Comp Σ (m((:)))
B
Mem
where |0r| is the number of free parameters of the model at time t, and (-(: 0+)) is the number of flops used by the model to process the 1-th mega-batch. Once again, by measuring the area under the curves obtained by tracking these quantities over time we obtain a holistic assessment of memory and compute throughout the whole stream. A model can obtain small Mem and Comp only if it does not consume memory and if it is computationally parsimonious throughout the entire duration of the stream.
4

---

Under review as a conference paper at ICLR 2022
Algorithm Training in the ALMA setting
procedure TRAIN(M, w, replay, grow)
m is the model, the waiting time
2:
t+
3:
4: while < do
> For each stage Acquire w mega-batches
5:
replay then
6:
D DUD, ....U...DDH... U
7:
else
8:
D D.U... DD+w-1
9:
t w if grow then
10:
m.grows) m.train(D)
D Grow the model the model growing model b Fine-tune or retrain from scratch m on the collected dataset
12:
LEARNING ALGORITHMS
In this section, we describe the methods we tested in the ALMA setting. They generally follow the learning procedure shown in Algorithm ! At high level, we consider two families of models, those with monolithic architecture and those with a modular architecture (e.g., ensembling). The latter are amenable to grow over time by adding new modules to the existing set. We will start by describing fixed architectures ($4.1) and then conclude with growing architectures (44.2). All models are also given the option to replay previous mega- batths.
4.1 FIXED ARCHITECTURES
The first family of methods trains models with fixed architecture. These models are sequentially trained over new bagaaeatches and exhibit a fixed memory footprint. We consider three models:
Single Model (SM): This is standard multi-layer neural network (e.g., fully connected neural network or transformer) trained by stochastic gradient descent. can be initialized from random or from the parameters of the model trained on the previous mega-batch. The initializaiton choice is determined via cross-validation.
Ensemble of Models (Ens): The second approach is the simplest modular approach, consisting of an ensemble of N neural networks with the same architecture, each being trained independently on the same sequence of data. The output of the overall model distribution produced by each componen!" The advantage of Ens is that training and inference can be trivially parallelized, enabling to scale up model parameters very easily. The disadvantange that inference requires N times more compute than what is required by each component.
time is the average probability
Uniform Mixture of Models (UMix): A potential drawback Ens is that evaluation and training inconsistent. UMix addresses this by training model whose prediction is the average (in logit space) of the predictions produced by N networks. While this requires synchronization during training, now both training and evaluation the same model.
4.2 GROWING ARCHITECTURES
In the previous section, the number of parameters and the architecture of the model fixed throughout the model's lifetime. However, as more data is observed, it is interesting to consider dynamic architectures that grow time, because these may save compute and memory during the earlier stages of learning while providing more predictive power during the later stages. We consider three growing approaches:
Classical hagging approaches and majority vote strategies have been also explored without significant difference.

---

Under review as a conference paper at ICLR 2022
Growing Ensemble (gEns): Like the Ens model, gÊns is also combination of neural networks trained independently. While Ens considers networks that are, at each stage, trained over the new chunck of data, gEns replaces this step by growing step where neural networks are added. In our implementation, only these n neural networks are trained over the new data, while the other neural networks (trained previous mega-batches) are kept fixed.
Growing Mixture of Experts (gMoE):: A hierarchical mixture of experts models (MoE) is an architecture where at layer the output representation 2 is: = - 9()=s->)((--->-">> where is the gating or routing function and h(-lj) is the j-th expert. Compared to Ens, MoE has exponentially many more components albeit with lot of parameter sharing. Another advantage is that by selecting only one (or a few) experts, the computational cost is independent of the number of experts, assuming the cost of gating is negligible compared to the cost or executing the experts. The main issue is that MoE are notoriously harder to train (Eigen et al. 2014 Denoyer & Gallinari [015) Lepikhin et al.) 2020). In this work, we consider a growing version of MoE, which we denote with gMoE, whereby experts are added over time. See Appendix A for more details.
Firefly (Wu et al. 2020) (FF): FF is method which progressively grows neural networks, jointly optimizing both the model architecture and parameters. Growth includes both a width expansion by adding new hidden units (or feature maps) as well as a depth expansion by adding new layers. Importantly, this is an example of non-modular method unlike Ens or gMoE, which potentially more expressive but also more inefficient at inference time because there is no structured sparsity that can be leveraged to speed up computation.
5 EXPERIMENTS
In this section we first describe how standard benchmarks can be repurposed for ALMA, we then provide the details of the models we tested, and we finally conelude with an analysis of the results we obtained, aiming to understand which method attains the best trade-off between time, accuracy, compute and memory usage.
Datasets We consider variety of datasets. The first dataset is MNIST LeCun et al. 1998). which consists of training set with 60,000 quasi-binary handwritten digits of size 28x28 pixels, and test set with 10,000 examples. The second dataset CIFAR 10 Krizhevsky 2009) that has training set with 50,000 images of size 32x32 pixels belonging to 10 classes such as bird, car, horse, ship, truck, etc. The third dataset, used for our large-scale language modeling evaluation, a portion the collection of English language text introduced in Liu et al. (2019b), consisting of Books, Wikipedia and Common Crawl. We consider 4 (large) mega-batcher for training and one additional mega-bate batch for evaluation, each consisting of approximately 440M words; we also hold out validation set with approximately 0.5M words of Common Crawl for model selection. We use a byte-pair encoding (BPE) (Sennrich et al. 2016) vocabulary with 50,000 units, following Radford et al. (2019). This dataset is fairly representative of what practitioners might face when maintaining deployed system with new data arriving every few months.
Given a dataset like any of the above, we construct a benchmark for ALMA evaluation as follows: 1) we randomly partition the training set into B mega-batches with equal number of training examples (B = 50 for MNIST and CIFAR 10, and 4 for the text dataset), 2) from each mega-batch we extract 10% of the data build the mega-batch validation set (except for the large scale language modeling dataset where we use the provided validation set), and 3) we create a learning experience by doing one pass over the sequence of mega-batches. For each mega-batche the learner can query as many mini-batches desired. The learner can also decide not to train on the data of mega-batch right away but instead to wait and accumulate data across few consecutive mega-batches. While the learner observes data, it is also tested the test set. This is not used for validation purposes, but only for final reporting as shown in $5.1
Models We evaluate the six approaches presented in $4 and for each of them we consider various waiting times, version with and without replay, and at least two model sizes. For each setting, we cross validate over several parpee-erraeterss such as initializaiton type, learning rate, stopping criterion, growth rate, etc.

---

Under review as a conference paper at ICLR 2022
Small (H=4)
Large (H=32)
4x 10
Test Error Rate
66 11
SM,K1
Test Emor Rale
30-0-1
5M,45
5 5x10-
5M,425
2 10
E56 W]
4 10-1
E16 w25
10-1
10
20
30
40
50
10
20
30
40
50
of mamega-batchee
batchhs
Figure 2: Accuracy over time for small models (left) and large models (right) on CIFAR 10 with no replay.
Method
Cum. Train TFLOPS
Inference MFLOPS
CER
3210
0.02 (0.09)
0.003
1054a25 (1062a61)
16330
0.09 (0.48)
0.016
631a10 (61619)
SM
26506
0.14 (0.79)
0.026
582a12 (55425)
132127
0.71 (3.92)
0.132
504-x (485+4)
269322
1.45 (7.98)
0.269
497-a (474-4)
16050
0.09 (0.47)
0.080
870+a (885+1s)
Ens
132530
0.71 (3.93)
0.661
517a6 (493.s)
16050
0.09 (0.47)
0.080
668 (646218)
UMix
132530
0.71 (3.93)
0.661
508 (4862z)
3210 + 32100
0,02 (0.09)
0.003 - 0.032
1020c3 (1027a2)
gEns
26506 - 265060
0.14 (0.79)
0.026 - 0.264
551 (543.7)
7950 - 50610
0.16 (1,08)
0.008 + 0.050
863u17 (766216)
gMoE
54318 + 304626
0.97 (6.55)
0.054 0.304
573a8 (535ss)
Table MNIST results with waiting time equal to 5. Rows with parameter size in blue refer to small networks (e.g., Ens totalling 16050 parameters has five components of the same size of SM shown in blue); similarly, rows in orange refer to the large network size. Numbers in parentheses are for versions with replay. For all metries smaller is better.
Next, describe in details the architecture used each dataset. Further experimental details to aide reproducibility are reported in Appendix B On MNIST the backbone architecture of SM is three layer fully connected neural network with ReLU units. We considered two hidden units sizes, namely and 32 (denoted by [s] and [b], respectively), which let us simulate the regime of big data relative to the size of the network and explore how to grow architectures without worrying about overfitting. Similarly, the components of Ens, gEns and UMix are SM networks of the same size as stated above; gMoE' also starts off as SM and adds modules (at the first two layers) that have the same size as the original layer of SM. When varying the waiting time, i.e., the number of regaahatches that are aggregated before initiating a new training session, we use the suffix " "_w" to indicate its value.
On CIFAR 10, the methods and notations are the same as in MNIST. The only difference is that the backbone architecture is scaled down version of a VGG19 convolutional neural network (Simonyan) & Zisserman 2015), where the number of intermediate feature maps is the same for each layer and equal to either or 32. On this dataset, we also consider FF starting off from the same VGG19 backbone.
For the language modeling task SM is a Switch Transformer (Fedus et al. 202T). which is a hard mixture of experts model with additional load balancing loss term and hard capacity constraint applied during training to prevent uneven expert utilization. Following Fedus et al. (2021). we fix the weight of the balancing loss term 0.01 and expert utilization. We train the model using Adam (Kingma & Bal 2015) and tune the learning rate and dropout on the validation set. In the growing setting we copy the expert weights and gating network weights corresponding to the top-k experts incurring the largest loss, where k is typically between 2 and We consider two model sizes: a base model with 6 layers and model dimension of 512, for a total of 40M shared parameters and 6M additional parameters per expert; and a large model with 12 layers and model dimension of 768, for total of 96M shared parameters and 28M additional parameters per expert. We use input sequence length of 512 tokens and we do not use replay given the large chunk sizes.
capacity factor of I, ensuring relatively uniform
5.1 RESULTS
In Fig. [ we start by analyzing learning curves on CIFAR 10 for a subset of the methods as a function of the waiting time. We then dive into analyzing all methods on both MNIST (Tab. [ and CIFAR 10
7

---

Under review as a conference paper at ICLR 2022
Method
[0]
Cum. Train TFLOPS
Inference MFLOPS
CER
2510
3.6 (20.0)
0.4
2706e132
(2643+39)
11710
15.0 (82.5)
1.7
2038+32
(1842+11
SM
31660
38.6 (212.5)
4.3
1764x4
(1561+7)
140970
164.9 (907.0)
18.3
1524a23
(1305a15)
1358510
1546.4 (8505.3)
171.8
1307a19
(1118a17
12550
18.2 (99.8)
10.1
2440a29
(2311a26)
Ens
704850
824.5 (4534.9)
458.1
1230
(1046:10)
12550
18.2 (99.8)
10.1
2087.36
(1840aa5)
UMix
704850
824.5 (4534.9)
458.1
1502.36
(1286:28)
2510 + 25100
3.6 (20.0)
0.4 4.0
2727
(2542:24)
gEns
140970
1409700
164.9 (907.0)
18.3 183.2
1348
(1283+s)
5214 - 29550
10.1 (65.3)
0.5 - 1.7
2401+61
(2089x28)
gMoE
27544 + 1485690
555.2 (3638.9)
26.2 - 97.2
1448+29
(1222+18)
2446 - 32519
26.1 (222.3)
0.4 7 5.5
2272426
(1947211)
140458 + 1646809
646.3 (7166.3)
18.3 138.9
1450
(1189120)
Table 2: CIFAR 10 results with waiting time equal to 5. Same notations and colors as in Table |
(Tab. 2) using the optimal empirical value of waiting time. We conclude by confirming the major
findings at scale the language modeling task (Tab. 3)
Fig. 2 shows the test error rate as a function of the number of mega-batches batches received for both the small (left) and the large (right) model. We observe that intermediate waiting time (in this case equal to 5) strikes the best trade-off between accuracy and time for all methods, since curves with waiting time equal to 5 have the lowest area under the curve. Greedy methods using waiting time equal to achieve lower eгror rate only during the very beginning of the stream. Second, we observe that bigger models (SM and Ens) not only generalize better but they are also statistically more efficient: the small Ens obtained almost 35% error rate by the end of its learning experience, which is worse than the error rate obtained by the large Ens just after having observed one tenth of the entire stream. The statistical efficiency of large models does not apply only to large transformers (Kaplan et al.) 2020a), but also to fully connected (we obtained similar results on MNIST) and convolutional models.
Next, using the waiting time that yielded the lowest cumulative error rate, we compare all methods discussed $4 focusing our discussion on Tab. of CIFAR 10 as same conclusions apply to MNIST as well (see Tab.
First, replay lowers the CER by a relative amount of about 10% at the cost of increasing the cumulative training flops by a factor of more than 5, which is rather substantial. Notice that retraining from scratch using memory replay, as reported here in parentheses, is nowadays the dominant approach to deal with sequential datasets.
Second, Ens works better than UMix for larger models, and vice versa. We surmise that ensembling may alleviate overfitting of large models, but coordinating the components of the ensemble like UMix does, is more effective an underfitting regime (i.e with small models). Ens thus looks like good method to train large architectures without suffering of the overfitting aspect and may be used when the complexity of the task is not known a priori.
Third, all growing approaches perform rather similarly, particularly when starting from larger back- bones, although they strike slightly different trade-offs.. For instance, gMoE is the most efficient at test time, while FF yields lower error rate. Interestingly, none of the approaches that grow architectures currently manages to beat Ens in terms of сттог rate when starting from a large backbone, although they require substantially fewer flops inference time. Finally, while methods derived from SM (for the same size of the initial backbone, see rows with the same color in the table) all manage to beat SM, it is also worth noting that for the same number of parameters SM is still the best performing method, unless there is overfitting, In particular, Ens with 12550 parameters achieves CER of 2440 while SM with 11710 parameters obtains a CER of 2038 while requiring much less compute; same considerations apply also the gMoE with 29550 parameters compared to SM with 3 36660 parameters. Therefore there is no single model striking a much better trade-off, and more advanced approaches do not outperform simpler methods like Ens.
8

---

Under review as a conference paper at ICLR 2022
Base model perplexity
Large model perplexity
setting
# experts
[0]
to
81
bs
[0]
to
81
8/
Eg
SM_wI
65M
28.57
27.45
26.91
26.53
210M
22.47
21.62
20.84
20.54
SM_w3 w3
8
91M
25.18
323M
19.29
3M_
12
116M
24.41
436M
19.01
4
130M
26.20
25.12
24.57
24.35
420M
20.32
19.55
19.14
18.92
Bas_w1
4
260M
25.03
24.03
23.45
23.29
840M
19.27
18.52
18.22
18.07
4@1
65M
28.57
210M
22.47
130M
26.27
420M
20.25
gEns_wl
4
195M
25.41
630M
19.49
4@4
260M
25.01
840M
19.18
65M
28.57
210M
22.47
6
78M
26.46
266M
21.22
gMoE_wl
91M
25.66
323M
20.39
12
116M
25.28
436M
20.15
Table Large scale language modeling results. For Ens and gEns, 4@3 means components in the ensemble, each of which has 4 experts per block, for instance.
The results on the large scale language modeling task reported in Tab. 3 show that bigger models perform better (the larger the number of parameters the lower the PPL for given model class) and are also more statistically efficient (for instance the base SM __1 attains 26.53 after seeing the whole stream, while the large SM w1 obtains 22.47 just after seeing the first chunk of data), consistent with recent related work (Kaplan et al. 20206 Li et al. (220a). We also observe that Ens is a strong performer, with Ens_wl _ll and gEns_w models dominating SM models in all settings. Surprisingly, ensembles trained on distinct data chunks (gEns_wl: l or (3) perform no better than ensembles trained on a single data chunk (Ens wl; to). For instance, among Base 2-model ensembles (4@2), Ens w1 achieves perplexity of 26.20 using single data chunk (to), while gEns w achieves perplexity of 26.27 using models trained on each of the two data chunks (f1). Finally, if test time inference is a concern, then gMoE is a preferable choice since its runtime is comparable to SM.
6 CONCLUSION AND PERSPECTIVES
In this work introduced the anytime learning macroscale (ALMA) setting, which is an instance of anytime learning under the assumption that data is observed as a sequence of large batches. ALMA better mimics the learning scenarios faced by machine learning practitioners, who want to efficiently solve a task, but time to time they receive more data to train on. We introduced metrics that enable the assessment in terms of error rate, memory usage and compute throughout the entire learning experience. Equipped with these tools, we then evaluated several approaches on three different datasets, including large scale language modeling. We found that methods that update parameters at an intermediate rate tend to yield a better trade-off, and that bigger models tend to generalize better. In particular, models that grow capacity over time generalize better particularly when the initial model is smaller, and ensembling is very strong baseline.
cynical interpretation of our finding that bigger models generalize better, could take the reader to the conclusion that it can all be solved by starting with big model. However, as data is added over time so computation. It often the case that researchers working on large-scale learning instantiate the biggest possible model to train on their task, but few months later they can manage to launch even bigger models thanks to compute and engineering advances. How can the larger model leverage what has been learned from the previously trained model? Is there a modeling choice that strikes better trade-off than retraining from scratch? More generally, what are good approaches to extract information from new batch of data to integrate it into existing model? While do not provide a full answer to these questions, we do offer a framework to study them and several strong baseline approaches to compare against and build upon.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
We have made several efforts to ensure that the results provided in the paper are fully reproducible. We first provide clean codebase from which all the computer vision results in this paper are generated. In this codebase, one can find the exact hyperparameters used for each method in the provided configurations. We have attached readme the code in order to guide users running our code. For the LM experiments, as stated in the appendix we use the fairseq (Ott et al., (219)) and provide the required information to replicate our results.
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR,
2013.
Léon Bottou. Online algorithms and stochastic approximations. In David Saad (ed.), Online Learning and Neural Networks. Cambridge University Press, Cambridge, UK, 1998.
Leo Breiman. Pasting small votes for classification in large databases and on-line machine learning. Machine Learning, 36(85-103), 1999.
Peter Bühhmann and Bin Yu. Analyzing bagging. The annals of Statistics, 30(4), 2002.
Ludovic Denoyer and Patrick Gallinari. Deep sequential neural networks. EWRL, 2015.
David Eigen, Ilya Sutskever, and Mare' Aurelio Ranzato. Learning factored representations in a deep mixture of experts. ICLR, 2014.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961 2021.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, 701.08734, 2017. /tLl/httpp//arxx...rr///// http://www..gga//////0/0/////// 08734
Jürgen Fritsch, Michael Finke, and Alex Waibel. Adaptively growing hicrarchical mixtures of experts. In Advances in Neural Information Processing Systems, 1996.
John J. Grefenstette and Connie Loggia Ramsey. Approach anytime learning. In Proceedings of the Ninth International Conference on Machine Learning, a22
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm.
Neural Computation, 6:181-214 1994.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001. 003661, 2020a.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.000 108361, 2020b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.6988 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL h
//axxiv..c org/ ab////b///1/ /1l22.111.... 6980
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, technical report, 2009.
Yann LeCun, Leon Bottou, and and Patrick Haffner Yoshua Bengio. Gradient-basee based learning applied to document recognition. Proceedings of the IEEE 86(1 1):2278-2324 1998.
10

---

Under review as a conference paper at ICLR 2022
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. CoRR. abs/2006. 16668, 2020.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, pp. 5958-5968. PMLR, 2020a.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5958-5968 PMLR, 2020b.
Qiang Liu, Lemeng Wu, and Dilin Wang. Splitting steepest descent for growing neural architectures. In Advances Neural Information Processing Systems, 2019a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 190..190..11999 2019b.
Zhengying Liu, Zhen Xu, Shangeth Rajaa, Meysam Madadi, Julio C. S. Jacques Junior, Sergio Escalera, Adrien Pavao, Sebastien Treguer, Wei-Wei Tu, and Isabelle Guyon. Towards automated deep learning: Analysis of the autodl challenge series 2019. In Hugo Jair Escalante and Raia Hadsell (eds.), Proceedings of the NeurlPS 2019 Competition and Demonstration Track, volume 123 of Proceedings of Machine Learning Research, pp. 242-252. PMLR, 2020.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12)2336-2333, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv: 01003.,. 2019.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. 7KDE, 2010.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Connie Loggia Ramsey and John J. Grefenstette. Case-based anytime learning. In AAAI Technical Report WS-94-01, 1994.
Mark B. Ring. Continual Learning in Reinforcement Environments. PhD thesis, University of Texas at Austin, Austin, Texas 78712, 1994.
Mark B Ring. CHILD: A first step towards continual learning. Machine Learning. 1997.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, 1725, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In International Conference on Learning Representations, 2015.
Sebastian Thrun. A lifelong learning perspective for mobile robot control. Proceedings of the IEEE/RSJ/GI Conference on Inielligent Robots and Systems, 1994.
Sebastian Thrun. Lifelong learning algorithms. In Learning to learn. Springer, 1998.
Vladimir Vapnik. Statistical learning theory. Wiley New York, 1998.
Tom Veniat, Ludovic Denoyer, and MarcAurelio Ranzato, Efficient continual learning with modular networks and task-driven priors. In International Conference Learning Representations, 2021.

---

Under review as a conference paper at ICLR 2022
Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Growing brain: Fine-tuning by increasing
model capacity. In JEEE Conference on Computer Vision and Pattern Recognition, 2017.
Steve Waterhouse and Anthony Robinson. Constructive algorithms for hierarchical mixtures of
experts. In Advances in Neural Information Processing Systems, 1995.
Wei Wen, Feng Yan, and Hai Li. Autogrow: Automatic layer growing in deep convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In Advances in Neural Information Processing Systems, 2020.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. BWl.ttttss//ooeeeevllee
net/ f=t.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5700
2012.
Tao Zou, Xian Li, Xuan Liang, and Hansheng Wang. On the subbagging estimation for massive data. arXiv preprint arXiv 2103.00631, 2021.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A GROWING MIXTURES OF EXPERTS
Growing Mixture of Experts (gMoE); A mixture of expert (MoE) is a sequence of non- non-linear functions, each of which is potentially mixture of experts (omitting the dependence parameters):
= f (...).. with f((:) - - ((lla)
where gf is the gating function the i-th layer which outputs categorical distribution over the number of experts, and (-Ij) is the j expert at layer i. The gating function can be "soft" in which case it outputs non-zero weights for each expert via a softmax, or "hard" version in which case only one expert is selected through multinomial sampling (and learned through the straigh-tthrough estimator in this paper (Bengio et al,, (013)). At test time in the "hard" case, we select the expert with the largest probability. The interest of mixtures of experts is they have high expressivity, and experts can be easily added to increase the capacity of the model. The gMofmodee is the growing version where, at each stage illustrated in Fig. 3 new experts are added at each layer details about the precise expansion process are given in Appendix.
The key design considerations are: when to grow, what to grow and how to grow. Here, we will refer to our default setting which favors simplicity, unless otherwise specified.
experts
Three exparts after spliting
[0W gotz
growth step is triggered at each stage, en- suring linear growth over time. We grow by adding one expert at each layer, making sure that all experts within a layer have the same ar- chitecture albeit with different parameters. In order to grow, we look at which expert has asso- ciated the largest cumulative loss; we call such expert the losing expert. The cumulative loss is defined as the sum of the losses of examples on the validation that have been routed through a particular expert; each expert has associated cumulative loss value. The rationale is to iden- tify at each layer the expert responsible for the
Hoke expart morkis
Figure 3: Illustration of growth step in tree struc- tured mixture of experts. A network is composed of several layers like this. The blue squares are experts (e.g VGG layers). The red elements corresponds to the gatings which, given an input compute a score each expert. When splitting an expert (right), the gating struc- ture is updated hy creating child gate, and an additional expert is added to the mixture.
largest contribution the total loss.
To avoid drop in the loss function and to keep its differentiability when splitting an expert, we propose a tree-baseds approach we the losing expert is split such expert into two experts with exactly the same parameters as illustrated in Fig. 3 Two children leaves are derived and we instantiate new gating for the children which decides whether an input example routed to the old expert, should now go the right or left expert child. The parameters of the new gate are initialized at random while the parameters of the new experts are exact copies of the ones of the losing expert that we split.
More formally, if s is the losing expert then the term (ss2))"(( is replaced by:
Σ
(3)
where (k(k,2, s) is the newly introduced gate, and is the input of the gating and experts.
Over time, the gating function learns to partition its input space into a binary tree (if we start from single expert), and the gating value an expert is the product of the gating probabilities on the path from root to the leaf expert. Both the gating tree strueture and the particular initialization scheme guarantee that the growth step is smooth and fully differentiable, in particular, the loss before and after the growth step is the same.
13

---

Under review as a conference paper at ICLR 2022
If consider each path in the MoE model be different model, then with L layer of k MoE components, there are ke many possible paths through the MoË model, hence the number of paths grows exponentially with the number of layers. You can think of this ensemble with exponentially many components, but this is still tractable because components share parameters.
Algorithm 2 gMoE
k: number of mega-batches to aggregate
2: D ()
3: function TRAIN(D,, 2)
4: D +- D;
5: í mod l = then
Extract D VAI. and D TR from D while m is not converged: do
6:
7:
8:
(z, y) ~ DRR m.update(x: y)
In practice, sample mini mini-batches.
9:
10:
==
m.grow(D VAL,
Growth step can be done a different rate too.
function GROW(D VAL)
12:
13:
for each layer in the network do
Let be the losing expert on D VAL, i.e. the expert incurring the largest cumulative loss. Turn corresponding gating output in internal node and derive 2 gate children Initialize the new experts by copying the parameters from the old parent expert. Initialize the new gating between the two siblings at random.
15:
16:
17:
B HYPER-PARAMETEA SETTINGS
B. COMPUTER VISION EXPERIMENTS
For each megabatch received, we keep 10% of the data to perform cross-validation. All experiments are run on single 16GB Quadro GP100 GPU. We apply data normalization for each dataset considered. A training minibatch size of 128 is used. UMix and Ens models have N 5 in all experiments. for gEns, we train one model n = at every mega-batch, so the total number of models depends the amount of mega-batches. For Firefly every growth phase, we add approximately a quarter of the initial number of parameters.
growth rate of 0.25, meaning that at
B.1.1 MNIST
Models trained for 100 epochs, and report results with soft gating. We the AdaDelta (Zeiler (2012) optimizer with default learning rate 1. We use a MLP with 2 hidden layers of varying width (e.g. 4,8 or neurons).
1.1.2 CIFAR-10
Models are trained for 200 epochs, as this was shown to be long enough to allow the model to converge with learning rate of 0.01. We use Stochastic Gradient Descent with momentum value of 0.9 and weight decay of 1 14-4, During training, apply random horizontal flips and select random image crops with padding of 4 pixels. For architecture, we use the same reduced VGG with batch normalization as prescribed in Wu et al. (2020). All layers are initialized with the same number channels (c.g. 4. 8. or 32 channels). For the Firefly experiments, we keep all the Firefly- specific hyperparameters to the default values suggested in the author's 's public codebase. We make one exception to this, namely we adapt the growth ratio to result in linear (rather than exponential) growth.
B.2 LANGUAGE MODELING EXPERIMENTS
All the language models are trained using fairseq (Ott et al., 2,19) with a maximum of eight 32GB GPUs (NVIDIA V100), optimized with Adam (Kingma Bal 2014) using 31 0.9. 82 0.98,
14

---

Under review as a conference paper at ICLR 2022
=le--.. The learning rate is warmed up over the first several hundred updates (between 500 and 4000) and then linearly decayed to 0 over the remaining updates, with a peak value tuned between 2e-4 and 5e-3. Models trained up 120,000 updates with local batch size of 8 sequences per GPU, with gradient accumulation as needed to achieve a total batch size of 192 sequences; each sequence has 512 tokens. We fix the Switch Transformer balancing loss term 0.01 and capacity factor of 1, following Fedus et al. (2021).
C ADDITIONAL COMPUTER VISION RESULTS
In this section we show the impact several variants of our framework. Namely, we report results for (a) a varying number of mega-batches, (b) whether to use preemption or not, and (c) whether to initialize from scratch or simply finetuning when replay is performed.
C.I CIFAR
In the following results, vary the number of megabatches. Below you can find results for MB = 20
C.1.1 DIFFERENT MBs
Method
[0]
Cum. Train TFLOPS
Inference MFLOPS
CER
2510
3.6 (20.0)
0.4
2705.8+1325
(2643. 13992)
11710
15.0 (82.5)
1.7
2038.2x322 2a322
(1842.3=s1.)
SM
31660
38.6 (212.5)
4.3
1763.5a42.7
(1560.6z6.9)
140970
164.9 (907.0)
18.3
1524.3a25.
(1305.00 0=149)
1358510
1546.4 (8505.3)
171.8
1307.3a19.1
(1118 (.11.)))
12550
18.2 (99.8)
10.1
2440.3x29.
(2311 12260)
Ens
704850
824.5 (4534.9)
458.1
1230.2495
(1045, 929.8)
12550
18.2 (99.8)
10.1
(1840,3 1087.0a36.1
UMix
704850
824.5 (4534.9)
458.1
1502.3+36.
(1286.4-27.)
2510 + 25100
3.6 (20.0)
0.4 + 4.0
2726.9470.5
(2542. 1244.)
gEns
140970 + 1409700
164.9 (907.0)
18.3 + 183.2
1348.3.161
(1282,9tss (8.))
5214 + 29550
10.1 (65.3)
0.5 - 1.7
240006x61.1
(2089.3sad .322.s))
MMoE
275442
1485690
555.2 (3638.9)
26.2 97.2
1448 122933
(1222.4am.s
2446
32519
26.1 (222.3)
- S.5
2272.5a26.5 5426.5
(1946.9244.)
FF
140458
+ 1646809
646.3 (7166.3)
18.3 + 138.9
1449. 713.7
(1189.3a19.)
Table 4: CIFAR-10 MB = 10 results in the paper
Method
[0]
Cum. Train TFLOPS
Inference MFLOPS
CER
2510
3.6 (38.1)
0.4
2047.14388
(6367.6235.6)
SM
140970
164.9 (1731.5)
18.3
1171.24529
1414.))167)
12550
18.2 (190.6)
10.1
1615.1432.1
(1300.2:25.0)
UMix
704850
824.5 (8657.5)
458.1
1233.9u23.
(876.4-29x)
2510 - 50200
3.6 (38.1)
0.4 - 8.1
2000.5+30.8
(1833.92 22
gEns
140970 - 2819400
164.9 (1731.5)
18.3 + 366.5
1047.5.47.
(949.6-487)
2446
22353
8.7 (315.4)
0.4 1.4
1788.8+579
(1446. 1429)
FF
140458
- 803761
300.9 (9646.7)
- 47.7
1140.443..
(799.22198)
Table 5: CIFAR-10 MB 20 results
15

---

Under review as a conference paper at ICLR 2022
C.1.2 PREEMPTED RESULTS
We also consider the use of patience term when training the model. When the validation accuracy has not improved over 25 consecutive epochs, we stop training for the given learning phase. As expected, we observe gains on compute efficiency, with small loss in performance.
Method
[0]
Cum. Train TELOPS
Inference MFLOPS
CER
2510
1,2 (6.8)
0,4
2844.6x36.
(2788.2176.
11710
4.4 (25.9)
1.7
2223.0 0.41.9
(1929.3x41.3)
SM
31660
11.2 (64.1)
4.3
1898.6xsa
(1615.8-23.5)
140970
39.9 (321.2)
18.3
1574.6.36.
(1374,9
1358510
416.4 (2699.7)
171.8
1366. La286
(1158.
12550
10.7 (61.9)
10.1
2482.0a57.8
(2334.6x18.
Ens
704850
330.0 (3192.1)
458.1
1238.7a132
(1048.5aa.))
12550
5.2 (35.5)
10.1
2379.7a37A
(1998.92sa.)
UMix
704850
209.4 (1617.3)
458.1
1675.9.51.
(1376.4.51.5
2510 + 25100
1.0 (5.6)
0.4 4.0
3048.6.171.0
(2688.42) (23.0)
gEns
140970 + 1409700
42,8 (302.4)
18.3 4 183.2
1432.8x33.7
(1325 2219.9)
52214 + 29550
3.0 (25.6)
0.5 - 1.7
2631. 19922
(2258. (546.7
gMoE
275442 + 1485690
107.7 (1276.5)
26.2 - 97.2
1562.
(1298.4
2446
35843
4.5 (100.3)
0.4 3.1
2570.3+1192
(2022.9+as.)
FF
140458
+ 1622818
148.1 (2819.9)
7 134.2
1537.9436.5
(1253.3341.0)
Table 6: CIFAR- 10 MB = 10 results, with preemption
Method
[8]
Cum. Train TFLOPS
Inference MFLOPS
2510
0.8 (9.9)
0.4
2404.0.71.
(2002.3.28.0
SM
140970
32,9 (431.7)
18.3
1242.7a43.2
(965.8217.))
12550
(121.2)
10.1
1828.3420A
(1646.8:14.)
Ens
704850
257.1 (4788.2)
458.1
222.
(724.3.223)
12550
4.1 (53.3)
10.1
1820.3a552
(1377.1a51.)
UMix
704850
197.8 (2459.3)
458.1
1334.5146.
(931.4-21.2)
2510 + 50200
0.9 (10.0)
0.4 + 8.1
2325.4.93.
(1944.2as0)
gEns
140970 + 2819400
34.5 (482.0)
18.3 + 366.5
1133.5x29.
(932.7=27.)
2446
17359
2.0 (71.6)
0.4 1.4
2034.3+44a
(1.57.55mm9)
FF
140458 - 791701
57.9 (2312.4)
18.3 50.2
1249.7+65.
(886.9-28)
Table 7: CIFAR-10 MB = 20 results, with preemption
C.1.3 INITIALIZING FROM SCRATCH
Below we show results, comparing the performance of training models from scratch on all the data seen so far vs simply finetuning the current model(s) on all the data. Main numbers are finetuned models, numbers in parentheses are trained from seratch.
Table 8: CIFAR-10 MB = 10 results with Replay. Numbers in () are models (re)initialized from scratch the start of new MB
C.2 MNIST
16

---

Under review as a conference paper at ICLR 2022
Method
Cum. Train TFLOPS
Inference MFLOPS
CER
3210
0.02 (0.09)
0.003
1054 (1062a61)
16330
0.09 (0.48)
0.016
631s10 (616*)
SM
26506
0.14 (0.79)
0.026
582+12 (554es)
132127
0.71 (3.92)
0.132
504a8 (485a()
269322
1.45 (7.98)
0.269
497a8 (474z))
16050
0.09 (0.47)
0.080
870ss (885218)
132530
0.71 (3.93)
0.661
517+6 (493-s)
16050
0.09 (0.47)
0.080
668+17 (646-18)
UMix
132530
0.71 (3.93)
0.661
508.10 (486+5)
3210
32100
0.02 (0.09)
0.003
0.032
1020ea (1027
gEns
26506
265060
0.14 (0.79)
0.026
0.264
5S1ss (543er)
7950
- 50610
0.16 (1.08)
0.008 - 0.050
863e17 (766s16)
gMoE
54318
+ 304626
0.97 (6.55)
0.054 + 0.304
573r8 (535ss)
Table 9: MNIST Results from the paper, with MB 10
Method
[|]
Cum. Train TELOPS
Inference MFLOPS
CER
16330
0.04 (0.19)
0.016
637+10 (608+10)
SM
132127
0.25 (1.84)
0.132
507+s (490=s)
269322
0.50 (3.11)
0.269
491-6 (478+5)
16050
0.06 (0.44)
0.080
880s14 (87722))
Ens
132530
0,37 (3.05)
0.661
521 (4962)
16050
0,04 (0.19)
0.080
687218 (636en)
UMix
132530
0.25 (1.85)
0.661
507a7 (488a4)
7950
50610
0,06 (0.44)
0.008 + 0.050
849 (829g)))
gMoE
54318
304626
0.33 (2.71)
0.054
0.304
572a11 (534x6)
Table 10: MNIST MB = 10 using preemption with patience value of 25 epochs
Method
[0]
Cum. Train TFLOPS
Inference MFLOPS
16330
0.48 (0.48)
0.016
315.6enp
(591.0x..)
SM
132127
3.92 (3.92)
0.132
485.5236
(481 123.3)
269322
7.98 (7.98)
0.269
474.0
(473.345.s)
16050
0.47 (0.47)
0.080
885.4
(881 (2233)
Ens
132530
3.93 (3.93)
0.661
493.1
(92.))23)
16050
0.47 (0.47)
0.080
645.9.18.1 (x146)
UMix
132530
3.93 (3.93)
0.661
485.6250
(480.6142)
7950
50610
1.08 (1.08)
0.008
0.050
765.6-159
(778.6 (778.6x16.)
gMoE
54318
304626
6.55 (6.55)
0.054
0.304
535.4.s.2 (543.0x77)
Table 11: MNIST MB = 10 results with replay, numbers in parentheses are models initialized from scratch the start of every MB
17


Reviews:
[{"review_id": "Y8gM87zIKJa", "reviewer": "Reviewer_KMi2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem presented by the authors is relevant to applied ML/AI problems, which are always a work in progress. Further, improving the efficiency of learning is desirable. So, the problem is reasonably well motivated.\n\n1. I'm skeptical about the value of the cumulative error rate. From a practical point of view a data engineer might be concerned with the questions; how good is the model I have now? What would be impact of further data collection? If one has collected a set of batches, the performance of prior models is not terribly relevant. \n\n2. The non-iid nature of the data is mentioned, and it is mentioned that cross-validation is carried out only on the current batch. These concepts could be explored more thoroughly. What are the issues with evaluation in this setting? Should I hold out a portion of every batch to use for evaluation? What other evaluation strategies are possible? What are their strengths and weaknesses?\n\n2. In continuous streaming settings, there may be distribution shift over time. That is not addressed in this work. \n\n3. The authors make a big point about the scale of the problem. Obviously this makes naive approaches less appealing. But how does the problem scale? What really separates (if anything), the macroscale problem from more mundane sized problems? Are there underlying scaling laws at work that cause shifts the performance of each learning strategy?", "summary_of_the_paper": "The authors consider a batch learning problem, in which large batches of data arrive in series. They explore the performance of several types of algorithms in terms of their computational cost, model size, and error rate. ", "summary_of_the_review": "The paper addresses a real need, but more insight is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XUSzwk_m8Yo", "reviewer": "Reviewer_x4p7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem setting of anytime learning at macroscale is interesting and novel to me. How to efficiently learn data in a streaming fashion is a practical challenge. The proposed learning setting targets the level of the entire sequence of large datasets. \n\nAlthough the method overall is valuable and interesting, the paper is poorly organized and thus hard to understand. It is difficult to find some critical details or notation definitions that are related but separated apart in the paper. The method lacks an integrated and principal formulation from which the techniques are derived. Though they show some theoretical results, it is hard to relate them to the objective and the algorithm steps explicitly and tightly. The contribution is unclear. Here are some detailed comments:\n\n- What does \"organically generated\" mean?\n- \" both training ... and finetuning ... are not satisfying\" should use \"neither...nor...\"\n- What does \"constrained capacity\" ?\n- What is the definition of macroscale?\n- Unclear the difference between ALMA and other learning frameworks.\n- Related work compares ALMA to lots of different prior work, but it is poorly organized and I am not sure why those prior work should be considered as comparisons.\n- It is suspicious to say that \"extensions to regression and unsupervised learning (where y is missing) are trivial\".\n- In fixed architecture, why \"A potential drawback of Ens is that evaluation and training are inconsistent\"?\n- In Growing Mixture of Experts, \"Compared to Ens, MoE has exponentially many more components\". I am not sure where \"exponentially\" comes from.\n\n\nQuality: The submission is technically sound. The claims in the contribution are supported by empirically results. It is a complete piece of work.\n\nClarity: The experimental details are also very specific, such that reproducing the results should be possible. ", "summary_of_the_paper": "Summary: This paper proposes anytime learning at macroscale (ALMA), which is anytime learning under the assumption that data is observed as a sequence of large batches. This paper introduces metrics that can be used to access the error rate, memory, and compute throughput the entire learning process. They evaluate multiple learning models on different datasets in the ALMA setting. They observe that methods that update parameters at a moderate rate tend to yield a better tradeoff, while bigger models tend to generalize better.", "summary_of_the_review": "This paper proposes a novel learning setting called anytime learning at macroscale (ALMA). The proposed idea is simple and technically sound. Extensive evaluations are conducted. However, the contribution is unclear, and the presentation needs improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ILXixoFhtem", "reviewer": "Reviewer_4ezA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper is well written. The authors document the approach they have considered, and the metrics used to evaluate the various experimental settings used. \n\nAs I started to read the paper, the problem setting seemed very similar to the ones used in data stream mining research over the past decade. Though the authors state that the primary differentiator to the stream setting is that the models use what they call as \"meta-batches\" as streams rather than streaming single data instances, I fail to understand any theoretical or empirical difference in the two approaches. There exists multiple popular data stream frameworks such as MOA (Massive Online Analysis) that is used exactly for the problem setting described in the paper. So, the primary contribution of the paper seems to be in extensively evaluating the model complexity and approaches over various data and problem settings. Moreover, majority of the future questions that the authors hope to answer have been studied across various papers (in similar forms). Please refer to Lu, Jie, et al. \"Learning under concept drift: A review.\" IEEE Transactions on Knowledge and Data Engineering 31.12 (2018): 2346-236.\n\nThe second objection of the paper is that the authors seem to use simple datasets from today's standard to derive their conclusion. Though the conclusions in the paper are fair and not surprising, a more complex set of datasets may provide a stronger result. Furthermore, it is important to note that there are other factors that influence the classifier performance, more than the batch size and data size available for training. The data itself may be imbalanced, non standard etc. So, by using more datasets, these issues can potentially be elevated.", "summary_of_the_paper": "The authors describe a framework to perform empirical evaluation of an anytime learning setting where data is available in a streaming minibatch fashion. With a primary aim to measure performance of a classifier across variety of practical settings of such streaming data to not only achieve high accuracy, but also provide non-trivial prediction anytime using limited computational resources. Using multiple benchmark datasets, the paper concludes that methods with intermediate parameter updates are better on the accuracy to computational efficiency tradeoff, and larger models generalize better.", "summary_of_the_review": "In summary, the original contribution is not very clear. The authors have ignored to discuss comparisons with a branch of data stream mining reseach that has provided similar conclusion. And the empirical evaluation is on simple datasets, and may need further evidential support from more complex datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G2B3MoXj0Xc", "reviewer": "Reviewer_D3ZG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is with the motivation of the setup. In particular, it does not seem to be very different from online learning. The paper does discuss the differences between the two setup, however the arguments are a bit superficial. In particular, it is claimed that in online learning the examples are streamed one at the time as opposed to receiving them in large batches - but this does not seem to make the online learning methods completely handicapped. Randomized algorithms (such as Thompson-sampling based methods) should perform reasonably. (For each example in batch i, apply the model obtained after processing all the examples in batches 1,2,..., i-1.) It would be great if the authors could elaborate on this.", "summary_of_the_paper": "The paper introduces a novel setup called Anytime Learning at Macroscale. In this setup the learner receives the examples as a sequence of large batches, and is required to output a model after processing each batch. This model is used to give prediction for the next batch. The overall performance is then sum of the average losses on the individual batches.", "summary_of_the_review": "Concern with the motivation: the setup is not that different from online learning after all.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work presented in this submission is focused on a new approach for learning a model that can perform well at any point in time, and called Anytime Learning at Macroscale (ALMA). The algorithm processes data through a series of training batches, each of these processing steps being followed to a model evaluation. The total loss is the average (or sum) of the losses computed at each step.

Reviewers agreed that the paper is not ready for acceptance at ICLR 2022 as the presentation of the work lacks of clarity, especially w.r.t. to the similarities with online learning and the learning of streams of data, and the fundamental difference between small or moderate batch sizes and very large batches.
Decision: Reject