On Anytime Learning at Macroscale

1 INTRODUCTION . Empirical risk minimization ( Vapnik , 1998 ) is the dominant framework to formalize the learning process of a supervised task , and it has been critical to the success of large scale training of deep learning systems on a wide variety of applications . Within this framework , training data is assumed to be provided to the learner all at once . Alternatively , when the dataset is very large ( essentially infinite ) , data is streamed to the learner one minibatch at the time , assuming that the rate at which samples are received matches the model ’ s processing time to learn from them . Learning over streams of data has been studied in the machine learning domain for a long time ( see Section 2 and Figure 1 for more details ) with different assumptions : for instance in online learning , it is usually assumed that datapoints are coming one by one and have to be processed as soon as they are received . In continual learning , the streaming of data usually corresponds to a stream of large datasets corresponding to different tasks to solve , etc . In this paper , we define a simple yet important setting where there is a single task to solve , and where training data often comes at a slower rate than a model can process it . Moreover , it comes in relatively large batches once in a while . While poorly studied , this setting corresponds to practical applications encountered in production pipelines . For instance , it is faced by teams deploying language modeling applications ( e.g content moderation ) build models that are trained on large amounts of data like filtered versions of Common Crawl , which are dumps of the internet . However , new snapshots are available every month , as new content is generated over time . Therefore datasets keep getting bigger every few months and models need to be retrained accordingly . Similarly , visual object recognition datasets used in deployed applications are often extended every few months to include new images with their corresponding annotations . * Authors contributed equally Practically , there are two main approaches to integrate information present in a new batch of data in an existing model . If a lot of computational resources are available , a new and bigger model is instantiated and trained from scratch on the union of the old training set with the new batch of data . However , since this is a computationally very intensive process , retraining is typically done only rarely , once several batches of data have been collected . We call this approach “ tardy ” large-scale learning , since a predictor is available only at a later time . Another option , particularly suitable when computational resources are scarce and a predictor is needed quickly , is to simply finetune the old model on the new data as this arrives . Note that , in that settings , methods from the data stream domain or from the online learning domain that are based on the idea of processing any datapoint just once are not suitable since they have been developed for different use-cases . This trade-off is emblematic of anytime learning , a learning setting where a learner has to provide good predictions at any point in time , while improving its performance over time as more and more data is observed . From an anytime learning perspective , neither training a large model after all data is received nor finetuning on the newly added batch of data are not satisfying . The former approach is a poor anytime learner because one needs to wait for a long time before obtaining a useful predictor . The latter approach is a poor anytime learner because it typically can not leverage very well future batches of data since the model has a fixed capacity , determined on a small portion of the overall dataset and because inherently the model is trained on non i.i.d . data . In this work , we aim at exploring this accuracy versus time trade-off of anytime learning , not at the level of a single batch of data , but at the macroscale of the entire sequence of batches . This is a setting which more closely mimics practical applications , that we call anytime learning at mascroscale ( ALMA ) . In this learning setting , we assume that the time to train a model is negligible compared to the interval of time between two consecutive batches of data ( and therefore we do not care about how quickly a learner adapts to a new batch ) , yet efficiency matters in the sense that for the same performance a predictor that uses less compute and memory is preferable . In summary , we are interested in a learner that i ) yields high accuracy , ii ) can make non-trivial predictions at any point in time while iii ) limiting its computational and memory resources . Our first contribution is to formalize the ALMA problem and to introduce metrics to evaluate learners ( §3 ) . We consider three different axes : error rate , memory and amount of computation . By measuring these quantities against time , via an area under the curve , we account not only for the final performance but also for the whole training trajectory over the sequence of large batches of data . Our second contribution is an extensive empirical evaluation ( §5 ) of various models ( §4 ) that strike different trade-offs between accuracy and time to obtain a useful predictor . In particular , we explore models that fall in between greedy finetuning and tardy large-scale learning , and investigate models that leverage batches of data at an intermediate rate . We also consider a rich family of modular architectures , from plain ensembling methods to hierarchical mixture of experts , and several variants thereof , including those that have access to a replay buffer storing all previous batches of data and those that can grow capacity over time . Our findings across three different benchmarks , including a large scale language modeling one , can be summarized as follows . a ) An intermediate waiting time offers the best trade-off between accuracy and time to yield such a predictor . However , b ) there is no single approach striking the best trade-off between performance and efficiency for various model sizes . c ) Retraining from scratch a big model does offer the lowest error rate but sacrifices efficiency . d ) Interestingly , large models are the most statistically efficient even when considering small datasets ( like MNIST ) and fully connected networks . e ) While approaches to grow capacity exhibit gains in terms of computational efficiency , these do not even outpeform simple ensembles . Overall , our work points at several research opportunities to improve modeling in a streaming setting of broad practical relevance , rather then pointing at any particular solution . We have also released code to reproduce our experiments and the entire platform implementing ALMA . 2 RELATED WORK . ALMA relates to several other learning frameworks : offline learning , continual learning , online learning and transfer learning as illustrated in Figure 1. i ) It shares the same assumptions of classical empirical risk minimization ( ERM ) ( Vapnik , 1998 ) at the level of each batch of data . However , it overall violates ERM ’ s assumptions of i.i.d . observations , because data points come in a stream of data chunks . ii ) Because of this , ALMA relates to continual learning ( CL ) ( Ring , 1994 ; Thrun , 1994 ; Ring , 1997 ; Thrun , 1998 ) , with the key difference that the data distribution across batches ( or tasks ) is assumed stationary in ALMA . Therefore , ALMA can be seen as a special case of CL with a single task to solve . iii ) ALMA relates also to online learning ( Bottou , 1998 ) since it assumes that data are coming in a stream , an assumption also made in the concept drift literature ( Lu et al. , 2018 ) . However , in online learning examples are streamed one at the time ( or at random from a large dataset ) , while in ALMA the learner receives large batches of data sequentially In ALMA , received data can be processed multiple times as opposite to the online learning setting that usually assumes that any new datapoint has to be processed as soon as it is available , and will not be reused in future updates . iv ) Finally , ALMA relates more broadly to transfer learning ( Pan & Yang , 2010 ) , as the problem of adapting to a new batch of data can be interpreted as leveraging knowledge acquired on previous batches to more effciently learn from the new batch of data . Of course , ALMA relates to anytime learning ( Grefenstette & Ramsey , 1992 ; Ramsey & Grefenstette , 1994 ) , which has been recently applied to compare various autoML frameworks ( Liu et al. , 2020 ) . However , in this work we are not interested in assessing the anytime learning ability at the level of each chunk of data , but only at a coarser granularity , at the level of the entire stream of chunks . Inspired by Liu et al . ( 2020 ) , we consider the area under the curve of error rate against time to measure performance , but in order to account also for compute and memory budget , we add to our evaluation metrics also the area under the curve for memory and compute . From the more theoretical side , there has been work about sub-bagging ( Bühlmann & Yu , 2002 ) ( bagging using subsets of a larger dataset ) which is similar to our setting but without the sequential aspect of it . In this context , Breiman ( 1999 ) proposed a model similar to our growing ensembling ( gEns ) , Bühlmann & Yu ( 2002 ) studied sub-bagging as a way to make the prediction of tree classifiers more robust while Zou et al . ( 2021 ) studied the consistency of the estimator in this setting . We defer to future studies the analysis of ALMA , while in this work we focus on the empirical evaluation . Shifting the discussion to prior work on models that adjust their capacity dynamically , Waterhouse & Robinson ( 1995 ) introduced an approach to grow a hierarchical mixture of experts model ( Jordan & Jacobs , 1994 ) . This is a tree structured model where experts are at the leaves and gating functions are at non-terminal nodes . The tree determines a hierarchical partition of the input space into regions that are associated to each expert . This approach was made more efficient in later work by ( Fritsch et al. , 1996 ) . In this work we consider a baseline ( gMoE ) that extends this prior work to hierarchical mixture of experts ( Eigen et al. , 2014 ; Denoyer & Gallinari , 2015 ; Lepikhin et al. , 2020 ) . Growing architectures have also been studied in CL . For instance , Fernando et al . ( 2017 ) and Veniat et al . ( 2021 ) proposed a modular architecture that is assembled for every task , possibly reusing previously trained modules . The major difference with our work is that in our case routing is input dependent as opposed to task dependent . Yoon et al . ( 2018 ) instead proposed a method to incrementally and smoothly add hidden units . Similarly , Wen et al . ( 2020 ) proposed a heuristic approach to automatically adjust the network depth . Wang et al . ( 2017 ) considered growing both depth and width when finetuning to a new task . Liu et al . ( 2019a ) and Wu et al . ( 2020 ) proposed approaches to grow architectures in depth and width by leveraging Taylor approximation and greedy selection . In our work , we benchmark against this last variant . None of these approaches have been applied to the ALMA setting to date . Finally , some of our findings are built upon and extend recent empirical evaluations studying the scaling properties of language models ( Kaplan et al. , 2020a ; Li et al. , 2020b ) . In this study , we confirm the conclusion that bigger models generalize better and are more statistically efficient , not only in language modeling tasks using a transformer architecture , but also in smaller scale computer vision tasks using both fully connected and convolutional architectures .

Summary: The paper introduces a novel setup called Anytime Learning at Macroscale. In this setup the learner receives the examples as a sequence of large batches, and is required to output a model after processing each batch. This model is used to give prediction for the next batch. The overall performance is then sum of the average losses on the individual batches.