CLEVA-Compass: A Continual Learning Evaluation Assessment Compass to Promote Research Transparency and Comparability

1 INTRODUCTION . Despite the indisputable successes of machine learning , recent concerns have surfaced over the field heading towards a potential reproducibility crisis ( Henderson et al. , 2018 ) , as previously identified in other scientific disciplines ( Baker , 2016 ) . Although replication may largely be assured through modern software tools , moving beyond pure replication towards reproducibility with a factual interpretation of results is accompanied by tremendous remaining challenges . Specifically for machine learning , recent reproducibility initiatives ( Pineau et al. , 2021 ) nicely summarize how differences in used data , miss- or under-specification of training and evaluation metrics , along with frequent over-claims of conclusions beyond gathered empirical evidence impose persisting obstacles in our current literature . Similar conclusions have been reached in related works focused on specifics of reinforcement learning ( Li & Talwalkar , 2019 ) , neural architecture search ( Lindauer & Hutter , 2020 ) , human-centered machine learning model cards ( Mitchell et al. , 2019 ) , or general dataset sheet specifications ( Bender & Friedman , 2018 ; Gebru et al. , 2018 ) , which all make valuable propositions to overcome existing gaps through the creation of standardized best-practice ( check- ) lists . It should thus come as no surprise that the emerging work in continual learning is no stranger to the above challenges . Superficially , continual learning has the intuitive objective of accumulating information and learning concepts over time , typically without the ability to revisit previous experiences , frequently also referred to as lifelong learning ( Chen & Liu , 2018 ) . However , there is no unique agreed-upon formal definition beyond the idea to continuously observe data , where the time component holds some practical implication on changes in the objective , the evolution of concept labels , or general statistical shifts in the data distribution . The majority of modern surveys ambiguously conflate these factors as a sequence of tasks ( Parisi et al. , 2019 ; Lesort et al. , 2019 ; Hadsell et al. , 2020 ; Lesort et al. , 2020 ; Biesialska et al. , 2021 ) . Much in contrast to prevalent static benchmarks , the question of reproducibility , interpretation of results , and overall comparability now becomes an even more complex function of nuances in employed data , training and evaluation protocols . The latter fact has sparked various attempts at consolidation or critique . On the one hand , several works have made important suggestions for continual learning desiderata with respect to evaluation protocols ( Farquhar & Gal , 2018 ; Dı́az-Rodrı́guez et al. , 2018 ; Kemker et al. , 2018 ) and categorization of incremental training set-ups ( van de Ven & Tolias , 2019 ; Lesort et al. , 2021 ) . On the other hand , empirical assessments have demonstrated that performance and comparability break down rapidly if often unexposed protocol aspects deviate ( Pfülb & Gepperth , 2019 ; Delange et al. , 2021 ) . Following the broader exposition of the recent reviews of Mundt et al . ( 2020a ) and Delange et al . ( 2021 ) , evaluation becomes convoluted because intricate combinations of elements originating from various related machine learning paradigms affect continual learning practice . As the number of publications across these paradigms increases , see Figure 1 , reproducibility , comparability , and interpretation of results thus also become increasingly difficult . In this work , we follow in the footsteps of previous reproducibility works to promote transparency and comparability of reported results for the non-trivial continual learning case . Rather than adding to the ongoing discussions on desiderata or violation of assumptions , we posit that the development of distinct applications warrants the existence of numerous continual scenarios . Based on respectively highlighted evaluation nuances and their implications when absorbed into continual learning , we derive the Continual Learning EValuation Assessment ( CLEVA ) Compass . The CLEVACompass provides a compact visual representation with a unique two-fold function : 1. it presents an intuitive chart to identify a work ’ s priorities and context in the broader literature landscape , 2. it enables a direct way to determine how methods differ in terms of practically reported metrics , where they resemble each other , or what elements would be missing towards a fairer comparison . In the remainder of the paper , we start by sketching the scope of continual learning in Section 2 , first by outlining the differences when going from static benchmarks to continual learning , followed by an exposition of evaluation nuances emanating from related machine learning paradigms . In Section 3 , we then proceed to introduce the CLEVA-Compass and illustrate its necessity and utility at the hand of several continual learning works . Before concluding , we summarize auxiliary best-practices proposed in related prior works and finally discuss limitations and unintended use of the CLEVACompass . To encourage general adoption , both for prospective authors to add methods and for application oriented practitioners to identify suitable methods , we supply various utilities : a template for direct inclusion into LaTeX , a Python script , and the CLEVA-Compass Graphical User Interface ( GUI ) , together with a repository to aggregate methods ’ compasses , all detailed in Appendix C and publicly available at https : //github.com/ml-research/CLEVA-Compass . 2 THE SCOPE AND CHALLENGES OF CONTINUAL LEARNING EVALUATION . As already argued , there is no unique agreed-upon formal definition of continual learning . One of the few common denominators across the continual machine learning literature is the understanding that catastrophic interference imposes a persisting threat to training models continually over time ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) . That is , continuously employing stochastic optimization algorithms to train models on sequential ( sub- ) sets of varying data instances , without being able to constantly revisit older experiences , comes with the challenge of previously learned parameters being overwritten . As this undesired phenomenon is in stark contrast with accumulation of knowledge over time , preventing model forgetting is thus typically placed at the center of continual learning . Consequently , the focus has been placed on a model-centric perspective , e.g . by proposing algorithms to alleviate forgetting through rehearsal of data prototypes in training , employing various forms of generative replay , imposing parameter and loss penalties over time , isolating task-specific parameters , regularizing the entire model ’ s functional or treating it from a perspective of dynamically adjustable capacity . We defer to review papers for taxonomies and details of algorithms ( Parisi et al. , 2019 ; Lesort et al. , 2019 ; Hadsell et al. , 2020 ; Lesort et al. , 2020 ; Biesialska et al. , 2021 ) . In practice , however , alleviating forgetting is but one of a myriad of challenges in real-world formulations of continual learning . Several orthogonal questions inevitably emerge , which either receive less attention across literature assessment or are frequently not made sufficiently explicit . A fair assessment and factual interpretation of results is rendered increasingly difficult . To provide the necessary background behind the statement , we briefly discuss newly arising questions when shifting from a static benchmark to a continual perspective and then proceed to contextualize conceivable evaluation protocol nuances in anticipation of our CLEVA-Compass . 2.1 FROM STATIC TO CONTINUAL MACHINE LEARNING WORKFLOW . To highlight the additional challenges in continual learning consider our visualization in Figure 2 , depicting the benchmark inspired machine learning workflow as advocated by Google Cloud ( 2021 ) . In the center , we find the six well-known sequential steps going from the preparation of data , to designing and tuning our ML model , down to the deployment of a model version to use for prediction . Naturally , these steps already contain various non-trivial questions , some of which we have highlighted in the surrounding gray boxes of the diagram . When considering popular benchmarks such as ImageNet ( Deng et al. , 2009 ) , a considerable amount of effort has been made for each individual workflow step . For instance , assembling , cleaning and pre-processing the dataset has required substantial resources , a decade of work has been attributed to the design of models and their optimization algorithms , and plenty of solutions have been developed to facilitate efficient computation or deployment . It is commonplace to treat these aspects in isolation in the literature . In other words , it is typical for approaches to be validated within train-val-test splits , where either a model-centric approach investigates optimizer variants and new model architectures , or alternatively , a data-centric approach analyzes how algorithms for data curation or selection can be improved for given models . Much in contrast to any of the prevalent static benchmarks , establishing a similar benchmark-driven way of conducting research becomes genuinely difficult for continual learning . Instinctively , this is because already partially intertwined elements of the workflow now become inherently codependent and inseparable . Once more , we provide a non-exhaustive list of additional questions in the orange boxes in the diagram of Figure 2 . Here , the boundaries between the steps are now blurred . To give a few examples : train and test sets evolve over time , we need to repeatedly determine what data to include next , an ongoing stream of data may be noisy or contain unknown elements , models might require new inductive biases and need to be extended , acquired knowledge needs to be protected but should also aid in future learning , and deployment and versions become continuous . In turn , setting a specific research focus on one of these aspects or attributing increased importance to only a portion allows for an abundance of conceivable , yet incomparable , implementations and investigations , even when the overall goal of continual learning is shared on the surface . 2.2 EVALUATION IN THE CONTEXT OF RELATED MACHINE LEARNING PARADIGMS . Although a full approach to continual learning should ideally include all of the underlying aspects illustrated in Figure 2 , many of these factors have been subject to prior isolated treatment in the literature . We posit that these related machine learning paradigms have a fundamental and historically grown influence on present “ continual learning ” practice , as they are in themselves comprised of components that are continuous . More specifically , we believe that choices in continual learning can largely be mapped back onto various related paradigms from which continual scenarios have drawn inspiration : multi-task learning ( Caruana , 1997 ) , transfer learning and domain adaptation ( Pan & Yang , 2010 ) , few-shot learning ( Fink , 2005 ; Fei-Fei et al. , 2006 ) , curriculum learning ( Bengio et al. , 2009 ) , active learning ( Settles , 2009 ) , open world learning ( Bendale & Boult , 2015 ) , online learning ( Heskes & Kappen , 1993 ; Bottou , 1999 ) , federated learning ( McMahan et al. , 2017 ; Kairouz et al. , 2021 ) , and meta-learning ( Thrun & Pratt , 1998 ) . We capture the relationship with respect to set-up and evaluation between these related paradigms and continual learning in the diagram of Figure 3 . For convenience , we have added quotes of the paradigm literature definitions in the figure . On the arrows of the diagram , we indicate the main evaluation difference and respectively how paradigms can be connected to each other . As each paradigm comes with its own set of assumptions towards training and evaluation protocols , it becomes apparent why the quest for a strict set of desiderata can be considered as ill-posed for the continual learning hypernym . In the next section , we thus introduce the CLEVA-Compass , as an alternative that emphasizes transparency and comparability over strict desiderata . To further clarify this with an easy example , let us consider one of the most popular ways to construct protocols in the academic continual learning literature . Here , a continual investigation is set up by defining a sequence based on extracting splits of existing benchmark datasets and introducing them sequentially ( Lesort et al. , 2020 ; Biesialska et al. , 2021 ; Delange et al. , 2021 ) . Such a scenario generally consist of individually introduced classes in image datasets like ImageNet ( Deng et al. , 2009 ) , MNIST ( LeCun et al. , 1998 ) , CIFAR ( Krizhevsky , 2009 ) , Core50 ( Lomonaco & Maltoni , 2017 ) , learning unique sounds ( Gemmeke et al. , 2017 ) or skills ( Mandlekar et al. , 2018 ; Fan et al. , 2018 ) in sequence , or simply creating a chain across multiple datasets for natural language processing ( McCann et al. , 2018 ; Wang et al. , 2019a ; b ) and games in reinforcement learning ( Bellemare et al. , 2013 ) . Arguably , such a set-up is immediately derived from conventional transfer learning practice . Following the description of Pan & Yang ( 2010 ) , the distinction between transfer and continual learning can essentially be brought down to the fact that both consider more than one task in sequence , but transfer learning focuses solely on leveraging prior knowledge to improve the new target task , typically a unique dataset or a set of classes , whereas continual learning generally intends to maximize performance on both prior and new tasks . Even though these popular continual benchmarks already simplify the continuous data perspective to seemingly enable comparison akin to static benchmarks , there already persists an unfortunate amount of training , evaluation , and result interpretation ambiguity . For instance , Farquhar & Gal ( 2018 ) ; Pfülb & Gepperth ( 2019 ) argue that simply the knowledge of whether and when a new task

Summary: In this work, the question arises of a better way to evaluate different methods in Continual Learning. The authors explain the problems of having a correct evaluation in Continual Learning, describing the complex scenario that Continual Learning confronts by being the intersection of multiple related areas. With this motivation, the authors present CLEVA-Compass or Continual Learning EValuation Assessment Compass, a visual representation that improves the comparison and transparency of different methods in Continual Learning. CLEVA-Compass includes two levels. The inner level shows which paradigms influence the method, and the outer level shows the setup and evaluation metrics presented in the method.