LAMOL: LAnguage MOdeling for Lifelong Language Learning

1 INTRODUCTION . The current dominant paradigm for machine learning is to run an algorithm on a given dataset to produce a trained model specifically for a particular purpose ; this is isolated learning ( Chen & Liu , 2016 , p. 150 ) . In isolated learning , the model is unable to retain and accumulate the knowledge it has learned before . When a stream of tasks are joined to be trained sequentially , isolated learning faces catastrophic forgetting ( McCloskey & Cohen , 1989 ) due to a non-stationary data distribution that biases the model ( left figure of Figure 1 ) . In contrast , lifelong learning is designed to address a stream of tasks by accumulating interconnected knowledge between learned tasks and retaining the performance of those tasks . A human easily achieves lifelong learning , but this is nontrivial for a machine ; thus lifelong learning is a vital step toward artificial general intelligence . In this paper , we focus on lifelong language learning , where a machine achieves lifelong learning on a stream of natural language processing ( NLP ) tasks . To the best of our knowledge , lifelong language learning has been studied in only a few instances ; for sentiment analysis ( Chen et al. , 2015b ; Xia et al. , 2017 ) , conversational agents ( Lee , 2017 ) , word representation learning ( Xu et al. , 2018 ) , sentence representation learning ( Liu et al. , 2019 ) , text classification , and question answering ( d ’ Autume et al. , 2019 ) . However , in all previous work , the tasks in the stream are essentially the same task but in different domains . To achieve lifelong language learning on fundamentally different tasks , we propose LAMOL — LAnguage MOdeling for Lifelong language learning . It has been shown that many NLP tasks can be considered question answering ( QA ) ( Bryan McCann & Socher , 2018 ) . Therefore , we address multiple NLP tasks with a single model by training a language model ( LM ) that generates an answer based on the context and the question . Treating QA as language modeling is beneficial because the LM can be pre-trained on a large number of sentences without any labeling ( Radford et al. , 2019 ) ; however , this does not directly solve the problem of LLL . If we train an LM on a stream of tasks , catastrophic forgetting still occurs . However , as an LM is intrinsically a text generator , we can use it to answer questions while generating pseudo-samples of ∗Equal contribution . †Work done while at National Taiwan University . the previous task to be replayed later . LAMOL is inspired by the data-based approach for LLL in which a generator learns to generate samples in previous tasks ( middle of Figure 1 ) ( Hanul Shin & Kim , 2017 ; Kemker & Kanan , 2017 ) . In contrast to previous approaches , LAMOL needs no extra generator ( right of Figure 1 ) . LAMOL is also similar to multitask training , but the model itself generates data from previous tasks instead of using real data . Our main contributions in this paper are : • We present LAMOL , a simple yet effective method for LLL . Our method has the advantages of no requirements in terms of extra memory or model capacity . We also do not need to know how many tasks to train in advance and can always train on additional tasks when needed . • Experimental results show that our methods outperform baselines and other state-of-the-art methods by a considerable margin and approaches the multitasking upper bound within 2–3 % . • Furthermore , we propose adding task-specific tokens during pseudo-sample generation to evenly split the generated samples among all previous tasks . This extension stabilizes LLL and is particularly useful when training on a large number of tasks . • We analyze how different amounts of pseudo-samples affect the final performance of LAMOL , considering results both with and without the task-specific tokens . • We open-source our code to facilitate further LLL research . 2 RELATED WORK . Lifelong learning research is based on regularization , architecture , or data . Here is a brief survey of works in these three categories . 2.1 REGULARIZATION-BASED METHODS . In this approach , a constraint , i.e. , a regularization term , is added to minimize deviation from trained weights while updating the weights in a new task . Most regularization based methods estimate the importance of each parameter and add the importance as a constraint to the loss function . Elastic weight consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) calculates a Fisher information matrix to estimate the sensitivity of parameters as importance . Online EWC ( Schwarz et al. , 2018 ) is a transformed version of EWC . Instead of tracking the importance of parameters for each task , online EWC simply accumulates the importance of the stream of tasks . Synaptic intelligence ( SI ) ( Zenke et al. , 2017 ) assigns importance to each parameter according to its contribution to the change in the total loss . Memory aware synapses ( MAS ) ( Aljundi et al. , 2018 ) estimate importance via the gradients of the model outputs . In contrast to estimating the importance of weights , incremental moment matching ( IMM ) ( Lee et al. , 2017 ) matches the moment of weights between different tasks . 2.2 ARCHITECTURE-BASED METHODS . For this category , the main idea is to assign a dedicated capacity inside a model for each task . After completing a task , the weights are frozen and may not be changed thereafter . Some methods allow models to expand , whereas some fix the size but must allocate capacity for tasks at the beginning . Progressive neural networks ( Rusu et al. , 2016 ) utilize one column of the neural network per task . Once a new task is trained , progressive neural networks augment a new column of the neural network for the task while freezing the past trained columns . Columns that have been frozen are not allowed to change but are connected to the new column to transfer knowledge from old tasks . Towards Training Recurrent Neural Networks for Lifelong Learning ( Sodhani et al. , 2018 ) unifies Gradient episodic memory ( Lopez-Paz et al. , 2017 ) and Net2Net ( Chen et al. , 2015a ) . Using the curriculumbased setting , the model learns the tasks in easy-to-hard order . The model alleviates the forgetting problem by GEM method , and if it fails to learn the current task and has not been expanded yet , the model will expand to a larger model by the Net2Net approach . PathNet ( Fernando et al. , 2017 ) reuses subsets of a neural network to transfer knowledge between tasks . Unlike progressive neural networks , PathNet does not allow the model to expand . Instead , it builds a huge fixed-size model composed of a neural network and paths between different layers of the neural networks . While training a task , it selects the best combination of neural networks and paths for that particular task . Similar to progressive neural networks , selected parts are fixed to allow only inference and not training . Inspired by network pruning , PackNet ( Mallya & Lazebnik , 2018 ) prunes and re-trains the network iteratively to pack numerous tasks into a single huge model . This category has some drawbacks . When resources are limited , model expansion is prohibited . Also , some architecture-based methods require the number of tasks in advance to allocate the capacity for the tasks , which greatly reduces their practicality . 2.3 DATA-BASED METHODS . This method restricts weights through the data distribution of old tasks . One data-based approach keeps a small amount of real samples from old tasks , and the other distills the knowledge from old data and imagines pseudo-data of old tasks later on . While training a new task , the data or pseudo-data is used to prevent weights from greatly deviating from the previous status . Gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) preserves a subset of real samples from previous tasks . Utilizing these real samples during optimization helps somewhat to constrain parameter gradients . Averaged-GEM ( A-GEM ) ( Chaudhry et al. , 2018 ) is a more efficient version of GEM which achieves the same or even better performance than the original GEM . Learning without forgetting ( Li & Hoiem , 2017 ) minimizes the alteration of shared parameters by recording the outputs from old task modules on data from the new task before updating . Hanul Shin & Kim ( 2017 ) and Kemker & Kanan ( 2017 ) encode data from old tasks into a generative model system . The latter imitates the dual-memory system of the human brain , in that the model automatically decides which memory should be consolidated . Both methods replay pseudo-data of previous tasks using the generative model during training . d ’ Autume et al . ( 2019 ) investigates the performance of the episodic memory system on NLP problems . It distills the knowledge of previous tasks into episodic memory and replays it afterward . This work evaluates the method on two streams of tasks : question answering and text classification . 3 LAMOL . A pre-trained LM can generate a coherent sequence of text given a context . Thus , we propose LAMOL , a method of training a single LM that learns not only to answer the question given the context but also to generate the context , the question , and the answer given a generation token . That is , in LAMOL , a model plays the role of both LM and QA model . Hence , answering questions and generating pseudo-old samples can both be done by a single model . During LLL , these pseudo-old samples are trained with new samples from new tasks to help mitigate catastrophic forgetting . 3.1 DATA FORMATTING . Inspired by the protocol used by decaNLP ( Bryan McCann & Socher , 2018 ) , samples from the datasets we used are framed into a SQuAD-like scheme , which consists of context , question , and answer . Although the LM is simultaneously a QA model , the data format depends on the training objective . When training as a QA model , the LM learns to decode the answer after reading the context and question . On the other hand , when training as an LM , the LM learns to decode all three parts given a generation token . In addition to context , question , and answer , we add three special tokens : ANS Inserted between question and answer . As the context and question are known during inference , decoding starts after inputting ANS . EOS The last token of every example . Decoding stops when EOS is encountered . GEN The first token during pseudo-sample generation . Decoding starts after inputting GEN . The data formats for QA and LM training are shown in Figure 2 . 3.2 TRAINING . Assume a stream of tasks { T1 , T2 , . . . } , where the number of tasks may be unknown . Directly training the LM on these tasks sequentially results in catastrophic forgetting . Thus , before beginning training on a new task Ti , i > 1 , the model first generates pseudo samples T ′ i by top-k sampling that represent the data distribution of previous tasks T1 , . . . , Ti−1 . Then , the LM trains on the mixture of Ti and T ′ i . To balance the ratio between |Ti| and |T ′ i | , the LM generates γ|Ti| pseudo samples , where |Ti| denotes the number of samples in task Ti and γ is the sampling ratio . If the generated sample does not have exactly one ANS in it , then the sample is discarded . This happens in only 0.5 % -1 % of generated samples . During training , each sample is formatted into both the QA format and the LM format . Then , in the same optimization step , both formats are fed into the LM to minimize the QA loss LQA and LM loss LLM together . Overall , the loss is L = LQA + λLLM , where λ is the weight of the LM loss .

Summary: This paper studies the problem of lifelong language learning. The core idea underlying the algorithm includes two parts: 1. Consider the NLP tasks as QA and then train a LM model that generates an answer based on the context and the question; 2. to generate samples representing previous tasks before training on a new task.