Towards Learning to Remember in Meta Learning of Sequential Domains

1 INTRODUCTION . Humans have the ability to quickly learn new skills from a few examples , without erasing old skills . It is desirable for machine-learning models to adopt this capability when learning under changing contexts/domains , which are common scenarios for real-world problems . These tasks are easy for humans , yet pose challenges for current deep-learning models mainly due to the following two reasons : 1 ) Catastrophic forgetting is a well-known problem for neural networks , which are prone to drastically losing knowledge on old tasks when a domain is shifted ( McCloskey & Cohen , 1989 ) ; 2 ) It has been a long-standing challenge to make neural networks generalize quickly from a limited amount of training data ( Wang et al. , 2020a ) . For example , the dialogue system can be trained on a sequence of domains , ( hotel booking , insurance , restaurant , car services , etc ) due to the sequential availability of dataset ( Mi et al. , 2020 ) . For each domain , each task is defined as learning one customer-specific model ( Lin et al. , 2019 ) . After finishing meta training , the model could be deployed to the previously trained domains , as the new ( unseen ) customers from previous domains may arrive later , they have their own ( small ) training data ( support set ) used for adapting the sequentially meta-learned models . After adaptation , the newly adapted model for the new customers can be deployed to make responses to the customers . We formulate the above problem as sequential domain few-shot learning , where a model is required to make proper decisions based on only a few training examples while undergoing constantly changing contexts/domains . It is expected that adjustments to a new context/domain should not erase knowledge already learned from old ones . The problem consists of two key components that have been considered separately in previous research : the ability to learn from a limited amount of data , referred to as few-shot learning ; and the ability to learn new tasks without forgetting old knowledge , known as continual learning . The two aspects have been proved to be particularly challenging for deep learning models , explored independently by extensive previous work ( Finn et al. , 2017 ; Snell et al. , 2017 ; Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) . However , a more challenging yet useful perspective to jointly integrate the two aspects remains less explored . Generally speaking , meta-learning targets learning from a large number of similar tasks with a limited number of training examples per class . Most existing works focus on developing the general- ization ability under a single context/domain ( Santoro et al. , 2016 ; Finn et al. , 2017 ; 2018 ; Snell et al. , 2017 ; Ravi & Beatson , 2019 ) . Recently , it has been shown that catastrophic forgetting often occurs when transferring a meta-learning model to a new context ( Ren et al. , 2019 ; Yoon et al. , 2020 ) . Continual learning aims to mitigate negative backward transfer effects on learned tasks when input distribution shift occurs during sequential context changes . Related techniques of which are currently applied mostly on standard classification problems ( Serrà et al. , 2018 ; Ebrahimi et al. , 2020b ) . In this paper , we generalize it to the sequential domain meta-learning setting , which seeks good generalization on unseen tasks from all domains with only limited training resources from previous domains . We term the problem sequential domain meta learning . Note this setting is different from continual few-shot learning that focuses on remembering previously learned lowresource tasks in a single domain . Our setting does not aim to remember on a specific task , but rather to maintain good generalization to a large amount of unseen few-shot tasks from previous domains without catastrophic forgetting . This setting is common and fits well in dynamic real-world scenarios such as recommendation system and dialogue training system . The domain shift arised from this setting during meta learning poses new challenges to existing continual-learning techniques . This is mainly due to the high variability underlying a large number of dynamically formed few-shot tasks , making it infeasible for a model to explicitly remember each task . In our setting , a model is expected to remember patterns generic to a domain , while neglecting noise and variance of a specific few-shot task . This ability , termed as remember to generalize , allows a model to capture general patterns of a domain that repeatedly occur in batches of tasks while avoid being too sensitive to a specific few-shot task . In this paper , we propose to address the aforementioned challenges by designing a dynamic learningrate adaptation scheme for learning to remember previous domains . These techniques could jointly consider gradients from multiple few-shot tasks to filter out task variance and only remember patterns that are generic in each domain . Our main idea is to meta learn both the model parameters and learning rates by backpropagating both a domain loss and a memory loss to adaptively update model parameters and the learning rates , respectively . Specifically , our mechanism keeps a small memory of tasks from previous domains , which are then used to guide the dynamic and adaptive learning behaviors on different portions of the network parameters . The proposed mechanism is versatile and applicable to both the metric-based prototypical network ( Snell et al. , 2017 ) and the gradient-based ANIL ( Raghu et al. , 2020 ) meta-learning model . Our contributions are summarized as follows : • We propose a challenging benchmark that requires a meta learning model to sequentially learn on a sequence of domains enduring domain shift without much forgetting on previous domains . • We extend meta learning models with existing dynamic learning rate modeling techniques . This can mitigate catastrophic forgetting through meta learning both model parameters and learning rates to dynamically control the network update process . This can be seamlessly integrated into both metric-based and gradient-based meta learning approaches . • We conduct extensive experiments on multiple public datasets under different sequential domain few-shot learning scenarios . We further test functionality of the dynamic learning-rate update mechanism for both metric-based and gradient-based meta-learning approaches . Comparisons are made towards a wide range of representative continuallearning techniques and models . Results demonstrate that our method outperforms strong baselines by a large margin . 2 RELATED WORKS . 2.1 META LEARNING . Meta learning ( Schmidhuber , 1993 ) , aka , learning to learn , aims to rapidly adapt to a new task by reusing previous experience through training on a large number of tasks . Meta learning can be roughly classified into the following categories : 1 ) Metric/Embedding-based approaches such as ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Edwards & Storkey , 2017 ) , which map input data into embedding ( feature ) spaces with decisions made based on some distance metric in the feature space ; 2 ) Black-box learning methods such as ( Andrychowicz et al. , 2016 ; Graves et al. , 2014 ; Mishra et al. , 2018 ) ; 3 ) Optimization-based methods such as ( Finn et al. , 2017 ; Ravi & Larochelle , 2017 ; Li et al. , 2017 ; Antoniou & Storkey , 2019 ) , which improve gradient-based optimization algorithms or learn to initialize network parameters ; and 4 ) Bayesian meta-learning methods such as ( Ravi & Beatson , 2019 ; Finn et al. , 2018 ; Yoon et al. , 2018b ; Grant et al. , 2018 ; Wang et al. , 2020b ) . These methods are used to either interprete and understand MAML ( Grant et al. , 2018 ) , or to model uncertainty of meta learning models ( Yoon et al. , 2018b ; Finn et al. , 2018 ; Wang et al. , 2020b ) . 5 ) Memorybased meta learning ( Santoro et al. , 2016 ; Munkhdalai & Yu , 2017 ; Mikulik et al. , 2020 ) , which apply additional memory component for meta learning . Online meta learning ( Finn et al. , 2019 ) is also related to us . They focus on forward transfer , i.e. , achieving better performance on future task and use all the data from previous tasks to do meta learning , while our setting is significantly different from theirs as we focus on mitigating catastrophic forgetting with only very limited access to previous domains . Dynamically updating the learning rates for networks is not new and has been explored in several contexts . Meta-SGD ( Li et al. , 2017 ) learns the per parameter learning rates for meta learning to improve flexibility and performance . ( Gupta et al. , 2020 ) use dynamic learning rates to mitigate forgetting in online continual learning . T-net ( Lee & Choi , 2018 ) learns a metric in activation space , which informs the update direction and step size for task-specific learning . Flennerhag et al . ( 2020 ) proposes e warped gradient descent to meta-learns an efficiently parameterised preconditioning matrix to dynamically update the network . Our work extends dynamic learning rate techniques to sequential domain meta learning setting to mitigate catastrophic forgetting . 2.2 CONTINUAL LEARNING . Continual learning tackles the problem of maintaining knowledge when input distribution shift happens in sequentially arriving tasks . There are different methods to address this problem , including 1 ) retaining memory for future replay ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; Riemer et al. , 2019 ; Chaudhry et al. , 2019b ) ; 2 ) designing tailored network architectures ( Rusu et al. , 2016 ; Fernando et al. , 2017 ; Yoon et al. , 2018a ) ; 3 ) performing proper regularization during parameter updates ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; von Oswald et al. , 2019 ) ; and 4 ) introducing Bayesian methods for model parameter inference ( Nguyen et al. , 2018 ; Ebrahimi et al. , 2020a ) . Specifically , methods based on memory replay store representative samples from old tasks and rehearsal is performed during training ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; Riemer et al. , 2019 ) . Recent research also utilizes generative models to memorize previously seen data ( Lesort et al. , 2019 ) . Representatives of architecture-based methods include Progressive Neural Networks ( Rusu et al. , 2016 ) , PathNet ( Fernando et al. , 2017 ) , Dynamically Expandable Networks ( Yoon et al. , 2018a ) , Hard Attention Mask ( HAT ) ( Serrà et al. , 2018 ) and PackNet ( Mallya & Lazeb- nik , 2017 ) , etc . These models explicitly modify network topology to preserve previous knowledge . The classic architecture-based approaches proposed in ( Serrà et al. , 2018 ) and ( Mallya & Lazebnik , 2017 ) do not fit into this setting , as they attempt to fully remember each historic task . Progressive Neural Networks ( Rusu et al. , 2016 ) guarantee zero forgetting but at the cost of growing network architectures and increasing parameters rapidly , which is unaffordable in memory-constraint cases . Regularization-based methods constrain the updated parameters to avoid drastic changes to previously learned tasks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; von Oswald et al. , 2019 ; Ebrahimi et al. , 2020c ) . They can restrict the capacity to meta-learning of new domains , thus it could hurt the performance on a new domain . Bayesian-based methods model parameters in a probabilistic way , and then parameters are updated either based on their posterior distributions ( Nguyen et al. , 2018 ) or on their uncertainty ( Ebrahimi et al. , 2020a ) . However , in the context of meta-learning , the uncertainty or posterior estimation could be highly inaccurate due to the small-data setting in each task , thus hindering the performance . Recently , there are works using meta learning to improve continual learning . For example , ( Javed & White , 2019 ) proposes to learn versatile representations by explicit training towards minimizing forgetting .

Summary: The paper proposes a method for the sequential meta-learning problem. The author meta learn not only model parameters but also learning rate vectors for parameter blocks. To this end, the meta-learn model finds appropriate model parameters and adaptive learning rate vectors that capture task-general information. Overall experiments are performed on few-shot meta-learning settings with sequential domains (datasets).