LAMOL: LAnguage MOdeling for Lifelong Language Learning

1 INTRODUCTION . The current dominant paradigm for machine learning is to run an algorithm on a given dataset to produce a trained model specifically for a particular purpose ; this is isolated learning ( Chen & Liu , 2016 , p. 150 ) . In isolated learning , the model is unable to retain and accumulate the knowledge it has learned before . When a stream of tasks are joined to be trained sequentially , isolated learning faces catastrophic forgetting ( McCloskey & Cohen , 1989 ) due to a non-stationary data distribution that biases the model ( left figure of Figure 1 ) . In contrast , lifelong learning is designed to address a stream of tasks by accumulating interconnected knowledge between learned tasks and retaining the performance of those tasks . A human easily achieves lifelong learning , but this is nontrivial for a machine ; thus lifelong learning is a vital step toward artificial general intelligence . In this paper , we focus on lifelong language learning , where a machine achieves lifelong learning on a stream of natural language processing ( NLP ) tasks . To the best of our knowledge , lifelong language learning has been studied in only a few instances ; for sentiment analysis ( Chen et al. , 2015b ; Xia et al. , 2017 ) , conversational agents ( Lee , 2017 ) , word representation learning ( Xu et al. , 2018 ) , sentence representation learning ( Liu et al. , 2019 ) , text classification , and question answering ( d ’ Autume et al. , 2019 ) . However , in all previous work , the tasks in the stream are essentially the same task but in different domains . To achieve lifelong language learning on fundamentally different tasks , we propose LAMOL — LAnguage MOdeling for Lifelong language learning . It has been shown that many NLP tasks can be considered question answering ( QA ) ( Bryan McCann & Socher , 2018 ) . Therefore , we address multiple NLP tasks with a single model by training a language model ( LM ) that generates an answer based on the context and the question . Treating QA as language modeling is beneficial because the LM can be pre-trained on a large number of sentences without any labeling ( Radford et al. , 2019 ) ; however , this does not directly solve the problem of LLL . If we train an LM on a stream of tasks , catastrophic forgetting still occurs . However , as an LM is intrinsically a text generator , we can use it to answer questions while generating pseudo-samples of ∗Equal contribution . †Work done while at National Taiwan University . the previous task to be replayed later . LAMOL is inspired by the data-based approach for LLL in which a generator learns to generate samples in previous tasks ( middle of Figure 1 ) ( Hanul Shin & Kim , 2017 ; Kemker & Kanan , 2017 ) . In contrast to previous approaches , LAMOL needs no extra generator ( right of Figure 1 ) . LAMOL is also similar to multitask training , but the model itself generates data from previous tasks instead of using real data . Our main contributions in this paper are : • We present LAMOL , a simple yet effective method for LLL . Our method has the advantages of no requirements in terms of extra memory or model capacity . We also do not need to know how many tasks to train in advance and can always train on additional tasks when needed . • Experimental results show that our methods outperform baselines and other state-of-the-art methods by a considerable margin and approaches the multitasking upper bound within 2–3 % . • Furthermore , we propose adding task-specific tokens during pseudo-sample generation to evenly split the generated samples among all previous tasks . This extension stabilizes LLL and is particularly useful when training on a large number of tasks . • We analyze how different amounts of pseudo-samples affect the final performance of LAMOL , considering results both with and without the task-specific tokens . • We open-source our code to facilitate further LLL research . 2 RELATED WORK . Lifelong learning research is based on regularization , architecture , or data . Here is a brief survey of works in these three categories . 2.1 REGULARIZATION-BASED METHODS . In this approach , a constraint , i.e. , a regularization term , is added to minimize deviation from trained weights while updating the weights in a new task . Most regularization based methods estimate the importance of each parameter and add the importance as a constraint to the loss function . Elastic weight consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) calculates a Fisher information matrix to estimate the sensitivity of parameters as importance . Online EWC ( Schwarz et al. , 2018 ) is a transformed version of EWC . Instead of tracking the importance of parameters for each task , online EWC simply accumulates the importance of the stream of tasks . Synaptic intelligence ( SI ) ( Zenke et al. , 2017 ) assigns importance to each parameter according to its contribution to the change in the total loss . Memory aware synapses ( MAS ) ( Aljundi et al. , 2018 ) estimate importance via the gradients of the model outputs . In contrast to estimating the importance of weights , incremental moment matching ( IMM ) ( Lee et al. , 2017 ) matches the moment of weights between different tasks . 2.2 ARCHITECTURE-BASED METHODS . For this category , the main idea is to assign a dedicated capacity inside a model for each task . After completing a task , the weights are frozen and may not be changed thereafter . Some methods allow models to expand , whereas some fix the size but must allocate capacity for tasks at the beginning . Progressive neural networks ( Rusu et al. , 2016 ) utilize one column of the neural network per task . Once a new task is trained , progressive neural networks augment a new column of the neural network for the task while freezing the past trained columns . Columns that have been frozen are not allowed to change but are connected to the new column to transfer knowledge from old tasks . Towards Training Recurrent Neural Networks for Lifelong Learning ( Sodhani et al. , 2018 ) unifies Gradient episodic memory ( Lopez-Paz et al. , 2017 ) and Net2Net ( Chen et al. , 2015a ) . Using the curriculumbased setting , the model learns the tasks in easy-to-hard order . The model alleviates the forgetting problem by GEM method , and if it fails to learn the current task and has not been expanded yet , the model will expand to a larger model by the Net2Net approach . PathNet ( Fernando et al. , 2017 ) reuses subsets of a neural network to transfer knowledge between tasks . Unlike progressive neural networks , PathNet does not allow the model to expand . Instead , it builds a huge fixed-size model composed of a neural network and paths between different layers of the neural networks . While training a task , it selects the best combination of neural networks and paths for that particular task . Similar to progressive neural networks , selected parts are fixed to allow only inference and not training . Inspired by network pruning , PackNet ( Mallya & Lazebnik , 2018 ) prunes and re-trains the network iteratively to pack numerous tasks into a single huge model . This category has some drawbacks . When resources are limited , model expansion is prohibited . Also , some architecture-based methods require the number of tasks in advance to allocate the capacity for the tasks , which greatly reduces their practicality . 2.3 DATA-BASED METHODS . This method restricts weights through the data distribution of old tasks . One data-based approach keeps a small amount of real samples from old tasks , and the other distills the knowledge from old data and imagines pseudo-data of old tasks later on . While training a new task , the data or pseudo-data is used to prevent weights from greatly deviating from the previous status . Gradient episodic memory ( GEM ) ( Lopez-Paz et al. , 2017 ) preserves a subset of real samples from previous tasks . Utilizing these real samples during optimization helps somewhat to constrain parameter gradients . Averaged-GEM ( A-GEM ) ( Chaudhry et al. , 2018 ) is a more efficient version of GEM which achieves the same or even better performance than the original GEM . Learning without forgetting ( Li & Hoiem , 2017 ) minimizes the alteration of shared parameters by recording the outputs from old task modules on data from the new task before updating . Hanul Shin & Kim ( 2017 ) and Kemker & Kanan ( 2017 ) encode data from old tasks into a generative model system . The latter imitates the dual-memory system of the human brain , in that the model automatically decides which memory should be consolidated . Both methods replay pseudo-data of previous tasks using the generative model during training . d ’ Autume et al . ( 2019 ) investigates the performance of the episodic memory system on NLP problems . It distills the knowledge of previous tasks into episodic memory and replays it afterward . This work evaluates the method on two streams of tasks : question answering and text classification . 3 LAMOL . A pre-trained LM can generate a coherent sequence of text given a context . Thus , we propose LAMOL , a method of training a single LM that learns not only to answer the question given the context but also to generate the context , the question , and the answer given a generation token . That is , in LAMOL , a model plays the role of both LM and QA model . Hence , answering questions and generating pseudo-old samples can both be done by a single model . During LLL , these pseudo-old samples are trained with new samples from new tasks to help mitigate catastrophic forgetting . 3.1 DATA FORMATTING . Inspired by the protocol used by decaNLP ( Bryan McCann & Socher , 2018 ) , samples from the datasets we used are framed into a SQuAD-like scheme , which consists of context , question , and answer . Although the LM is simultaneously a QA model , the data format depends on the training objective . When training as a QA model , the LM learns to decode the answer after reading the context and question . On the other hand , when training as an LM , the LM learns to decode all three parts given a generation token . In addition to context , question , and answer , we add three special tokens : ANS Inserted between question and answer . As the context and question are known during inference , decoding starts after inputting ANS . EOS The last token of every example . Decoding stops when EOS is encountered . GEN The first token during pseudo-sample generation . Decoding starts after inputting GEN . The data formats for QA and LM training are shown in Figure 2 . 3.2 TRAINING . Assume a stream of tasks { T1 , T2 , . . . } , where the number of tasks may be unknown . Directly training the LM on these tasks sequentially results in catastrophic forgetting . Thus , before beginning training on a new task Ti , i > 1 , the model first generates pseudo samples T ′ i by top-k sampling that represent the data distribution of previous tasks T1 , . . . , Ti−1 . Then , the LM trains on the mixture of Ti and T ′ i . To balance the ratio between |Ti| and |T ′ i | , the LM generates γ|Ti| pseudo samples , where |Ti| denotes the number of samples in task Ti and γ is the sampling ratio . If the generated sample does not have exactly one ANS in it , then the sample is discarded . This happens in only 0.5 % -1 % of generated samples . During training , each sample is formatted into both the QA format and the LM format . Then , in the same optimization step , both formats are fed into the LM to minimize the QA loss LQA and LM loss LLM together . Overall , the loss is L = LQA + λLLM , where λ is the weight of the LM loss .

Summary: The paper presents a new NN architecture designed for life-long learning of natural language processing. As well depicted in Figure 2, the proposed network is trained to generate the correct answers and training samples at the same time. This prevents the "catastrophic forgetting" of an old task. Compared to the old methods that train a separate generator, the performance of the proposed method is noticeably good as shown in Fig 3. This demonstrates that the new life-long learning approach is effective in avoiding catastrophic forgetting.