CLEVA-Compass: A Continual Learning Evaluation Assessment Compass to Promote Research Transparency and Comparability

1 INTRODUCTION . Despite the indisputable successes of machine learning , recent concerns have surfaced over the field heading towards a potential reproducibility crisis ( Henderson et al. , 2018 ) , as previously identified in other scientific disciplines ( Baker , 2016 ) . Although replication may largely be assured through modern software tools , moving beyond pure replication towards reproducibility with a factual interpretation of results is accompanied by tremendous remaining challenges . Specifically for machine learning , recent reproducibility initiatives ( Pineau et al. , 2021 ) nicely summarize how differences in used data , miss- or under-specification of training and evaluation metrics , along with frequent over-claims of conclusions beyond gathered empirical evidence impose persisting obstacles in our current literature . Similar conclusions have been reached in related works focused on specifics of reinforcement learning ( Li & Talwalkar , 2019 ) , neural architecture search ( Lindauer & Hutter , 2020 ) , human-centered machine learning model cards ( Mitchell et al. , 2019 ) , or general dataset sheet specifications ( Bender & Friedman , 2018 ; Gebru et al. , 2018 ) , which all make valuable propositions to overcome existing gaps through the creation of standardized best-practice ( check- ) lists . It should thus come as no surprise that the emerging work in continual learning is no stranger to the above challenges . Superficially , continual learning has the intuitive objective of accumulating information and learning concepts over time , typically without the ability to revisit previous experiences , frequently also referred to as lifelong learning ( Chen & Liu , 2018 ) . However , there is no unique agreed-upon formal definition beyond the idea to continuously observe data , where the time component holds some practical implication on changes in the objective , the evolution of concept labels , or general statistical shifts in the data distribution . The majority of modern surveys ambiguously conflate these factors as a sequence of tasks ( Parisi et al. , 2019 ; Lesort et al. , 2019 ; Hadsell et al. , 2020 ; Lesort et al. , 2020 ; Biesialska et al. , 2021 ) . Much in contrast to prevalent static benchmarks , the question of reproducibility , interpretation of results , and overall comparability now becomes an even more complex function of nuances in employed data , training and evaluation protocols . The latter fact has sparked various attempts at consolidation or critique . On the one hand , several works have made important suggestions for continual learning desiderata with respect to evaluation protocols ( Farquhar & Gal , 2018 ; Dı́az-Rodrı́guez et al. , 2018 ; Kemker et al. , 2018 ) and categorization of incremental training set-ups ( van de Ven & Tolias , 2019 ; Lesort et al. , 2021 ) . On the other hand , empirical assessments have demonstrated that performance and comparability break down rapidly if often unexposed protocol aspects deviate ( Pfülb & Gepperth , 2019 ; Delange et al. , 2021 ) . Following the broader exposition of the recent reviews of Mundt et al . ( 2020a ) and Delange et al . ( 2021 ) , evaluation becomes convoluted because intricate combinations of elements originating from various related machine learning paradigms affect continual learning practice . As the number of publications across these paradigms increases , see Figure 1 , reproducibility , comparability , and interpretation of results thus also become increasingly difficult . In this work , we follow in the footsteps of previous reproducibility works to promote transparency and comparability of reported results for the non-trivial continual learning case . Rather than adding to the ongoing discussions on desiderata or violation of assumptions , we posit that the development of distinct applications warrants the existence of numerous continual scenarios . Based on respectively highlighted evaluation nuances and their implications when absorbed into continual learning , we derive the Continual Learning EValuation Assessment ( CLEVA ) Compass . The CLEVACompass provides a compact visual representation with a unique two-fold function : 1. it presents an intuitive chart to identify a work ’ s priorities and context in the broader literature landscape , 2. it enables a direct way to determine how methods differ in terms of practically reported metrics , where they resemble each other , or what elements would be missing towards a fairer comparison . In the remainder of the paper , we start by sketching the scope of continual learning in Section 2 , first by outlining the differences when going from static benchmarks to continual learning , followed by an exposition of evaluation nuances emanating from related machine learning paradigms . In Section 3 , we then proceed to introduce the CLEVA-Compass and illustrate its necessity and utility at the hand of several continual learning works . Before concluding , we summarize auxiliary best-practices proposed in related prior works and finally discuss limitations and unintended use of the CLEVACompass . To encourage general adoption , both for prospective authors to add methods and for application oriented practitioners to identify suitable methods , we supply various utilities : a template for direct inclusion into LaTeX , a Python script , and the CLEVA-Compass Graphical User Interface ( GUI ) , together with a repository to aggregate methods ’ compasses , all detailed in Appendix C and publicly available at https : //github.com/ml-research/CLEVA-Compass . 2 THE SCOPE AND CHALLENGES OF CONTINUAL LEARNING EVALUATION . As already argued , there is no unique agreed-upon formal definition of continual learning . One of the few common denominators across the continual machine learning literature is the understanding that catastrophic interference imposes a persisting threat to training models continually over time ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) . That is , continuously employing stochastic optimization algorithms to train models on sequential ( sub- ) sets of varying data instances , without being able to constantly revisit older experiences , comes with the challenge of previously learned parameters being overwritten . As this undesired phenomenon is in stark contrast with accumulation of knowledge over time , preventing model forgetting is thus typically placed at the center of continual learning . Consequently , the focus has been placed on a model-centric perspective , e.g . by proposing algorithms to alleviate forgetting through rehearsal of data prototypes in training , employing various forms of generative replay , imposing parameter and loss penalties over time , isolating task-specific parameters , regularizing the entire model ’ s functional or treating it from a perspective of dynamically adjustable capacity . We defer to review papers for taxonomies and details of algorithms ( Parisi et al. , 2019 ; Lesort et al. , 2019 ; Hadsell et al. , 2020 ; Lesort et al. , 2020 ; Biesialska et al. , 2021 ) . In practice , however , alleviating forgetting is but one of a myriad of challenges in real-world formulations of continual learning . Several orthogonal questions inevitably emerge , which either receive less attention across literature assessment or are frequently not made sufficiently explicit . A fair assessment and factual interpretation of results is rendered increasingly difficult . To provide the necessary background behind the statement , we briefly discuss newly arising questions when shifting from a static benchmark to a continual perspective and then proceed to contextualize conceivable evaluation protocol nuances in anticipation of our CLEVA-Compass . 2.1 FROM STATIC TO CONTINUAL MACHINE LEARNING WORKFLOW . To highlight the additional challenges in continual learning consider our visualization in Figure 2 , depicting the benchmark inspired machine learning workflow as advocated by Google Cloud ( 2021 ) . In the center , we find the six well-known sequential steps going from the preparation of data , to designing and tuning our ML model , down to the deployment of a model version to use for prediction . Naturally , these steps already contain various non-trivial questions , some of which we have highlighted in the surrounding gray boxes of the diagram . When considering popular benchmarks such as ImageNet ( Deng et al. , 2009 ) , a considerable amount of effort has been made for each individual workflow step . For instance , assembling , cleaning and pre-processing the dataset has required substantial resources , a decade of work has been attributed to the design of models and their optimization algorithms , and plenty of solutions have been developed to facilitate efficient computation or deployment . It is commonplace to treat these aspects in isolation in the literature . In other words , it is typical for approaches to be validated within train-val-test splits , where either a model-centric approach investigates optimizer variants and new model architectures , or alternatively , a data-centric approach analyzes how algorithms for data curation or selection can be improved for given models . Much in contrast to any of the prevalent static benchmarks , establishing a similar benchmark-driven way of conducting research becomes genuinely difficult for continual learning . Instinctively , this is because already partially intertwined elements of the workflow now become inherently codependent and inseparable . Once more , we provide a non-exhaustive list of additional questions in the orange boxes in the diagram of Figure 2 . Here , the boundaries between the steps are now blurred . To give a few examples : train and test sets evolve over time , we need to repeatedly determine what data to include next , an ongoing stream of data may be noisy or contain unknown elements , models might require new inductive biases and need to be extended , acquired knowledge needs to be protected but should also aid in future learning , and deployment and versions become continuous . In turn , setting a specific research focus on one of these aspects or attributing increased importance to only a portion allows for an abundance of conceivable , yet incomparable , implementations and investigations , even when the overall goal of continual learning is shared on the surface . 2.2 EVALUATION IN THE CONTEXT OF RELATED MACHINE LEARNING PARADIGMS . Although a full approach to continual learning should ideally include all of the underlying aspects illustrated in Figure 2 , many of these factors have been subject to prior isolated treatment in the literature . We posit that these related machine learning paradigms have a fundamental and historically grown influence on present “ continual learning ” practice , as they are in themselves comprised of components that are continuous . More specifically , we believe that choices in continual learning can largely be mapped back onto various related paradigms from which continual scenarios have drawn inspiration : multi-task learning ( Caruana , 1997 ) , transfer learning and domain adaptation ( Pan & Yang , 2010 ) , few-shot learning ( Fink , 2005 ; Fei-Fei et al. , 2006 ) , curriculum learning ( Bengio et al. , 2009 ) , active learning ( Settles , 2009 ) , open world learning ( Bendale & Boult , 2015 ) , online learning ( Heskes & Kappen , 1993 ; Bottou , 1999 ) , federated learning ( McMahan et al. , 2017 ; Kairouz et al. , 2021 ) , and meta-learning ( Thrun & Pratt , 1998 ) . We capture the relationship with respect to set-up and evaluation between these related paradigms and continual learning in the diagram of Figure 3 . For convenience , we have added quotes of the paradigm literature definitions in the figure . On the arrows of the diagram , we indicate the main evaluation difference and respectively how paradigms can be connected to each other . As each paradigm comes with its own set of assumptions towards training and evaluation protocols , it becomes apparent why the quest for a strict set of desiderata can be considered as ill-posed for the continual learning hypernym . In the next section , we thus introduce the CLEVA-Compass , as an alternative that emphasizes transparency and comparability over strict desiderata . To further clarify this with an easy example , let us consider one of the most popular ways to construct protocols in the academic continual learning literature . Here , a continual investigation is set up by defining a sequence based on extracting splits of existing benchmark datasets and introducing them sequentially ( Lesort et al. , 2020 ; Biesialska et al. , 2021 ; Delange et al. , 2021 ) . Such a scenario generally consist of individually introduced classes in image datasets like ImageNet ( Deng et al. , 2009 ) , MNIST ( LeCun et al. , 1998 ) , CIFAR ( Krizhevsky , 2009 ) , Core50 ( Lomonaco & Maltoni , 2017 ) , learning unique sounds ( Gemmeke et al. , 2017 ) or skills ( Mandlekar et al. , 2018 ; Fan et al. , 2018 ) in sequence , or simply creating a chain across multiple datasets for natural language processing ( McCann et al. , 2018 ; Wang et al. , 2019a ; b ) and games in reinforcement learning ( Bellemare et al. , 2013 ) . Arguably , such a set-up is immediately derived from conventional transfer learning practice . Following the description of Pan & Yang ( 2010 ) , the distinction between transfer and continual learning can essentially be brought down to the fact that both consider more than one task in sequence , but transfer learning focuses solely on leveraging prior knowledge to improve the new target task , typically a unique dataset or a set of classes , whereas continual learning generally intends to maximize performance on both prior and new tasks . Even though these popular continual benchmarks already simplify the continuous data perspective to seemingly enable comparison akin to static benchmarks , there already persists an unfortunate amount of training , evaluation , and result interpretation ambiguity . For instance , Farquhar & Gal ( 2018 ) ; Pfülb & Gepperth ( 2019 ) argue that simply the knowledge of whether and when a new task

Summary: This paper argues that the goal of precisely formulating abstract desiderata for continual learning (CL) is ill-posed because different applications may always favour distinct scenarios. Following this logic, the paper proposes a categorisation system which aims to make CL research comparisons more structured. The paper goes on to review the challenges of meaningful evaluation of CL algorithms, particularly in terms of systematic comparisons of different works with somewhat different sets of assumptions, using the notion of “catastrophic forgetting” as a working example. Aspects which sometimes are overlooked in other machine learning research, e.g. dataset preprocessing and balancing, become important aspects in CL research and cannot be ignored. Furthermore, many machine learning paradigms may have natural extensions to non-i.i.d. learning scenarios, or at least various multi-task formulations of interest. Since many of these satellite formulations inherit metrics and assumptions of their respective paradigms, it naturally creates a problem formulation challenge for CL research. The paper proposes a multi-objective radar/spider web chart (CLEVA-Compass) as a classification heuristic for CL research evaluation setups. The diagram is based on a well conceived collection of conceptual and practical desiderata for CL algorithms. The paper proposes how such a diagram could be used by future CL research and anticipates unintended uses. Limitations of the proposed classification strategy, as well as complementary challenges of dataset design, are discussed towards the end of the paper.