Progressive Memory Banks for Incremental Domain Adaptation

1 INTRODUCTION . Domain adaptation aims to transfer knowledge from one domain ( called the source domain ) to another ( called the target domain ) in a machine learning system.2 If the data of the target domain are not large enough , using data from the source domain typically helps to improve model performance in the target domain . This is important for neural networks , which are data-hungry and prone to overfitting . In this paper , we especially focus on incremental domain adaptation ( IDA ) 3 , where we assume different domains come sequentially one after another . We only have access to the data in the current domain , but hope to build a unified model that performs well on all the domains that we have encountered ( Xu et al. , 2014 ; Rusu et al. , 2016 ; Kirkpatrick et al. , 2017 ) . Incremental domain adaptation is useful in scenarios where data are proprietary , or available only for a short period of time ( Li & Hoiem , 2018 ) . It is desired to preserve as much knowledge as possible in the learned model and not to rely on the availability of the data . Another application of IDA is a quick adaptation to new domains . If the environment of a deployed machine learning system changes frequently , traditional methods like jointly training all domains require the learning machine to be re-trained from scratch every time a new domain comes . Fine-tuning a neural network by a ∗ Equal contribution . 1Our IDA code is available at https : //github.com/nabihach/IDA . 2In our work , the domain is defined by datasets . Usually , the data from different genres or times typically have different underlying distributions . 3In the literature , IDA is sometimes referred to as incremental , continual , or lifelong learning . We use “ incremental domain adaptation ” in this paper to emphasize that our domain is not changed continuously . few steps of gradient updates does transfer quickly , but it suffers from the catastrophic forgetting problem ( Kirkpatrick et al. , 2017 ) . Suppose during prediction a data point is not labeled with its domain , the ( single ) fine-tuned model can not predict well for samples in previous domains , as it tends to “ forget ” quickly during fine-tuning . A recent trend of domain adaptation in the deep learning regime is the progressive neural network ( Rusu et al. , 2016 ) , which progressively grows the network capacity if a new domain comes . Typically , this is done by enlarging the model with new hidden states and a new predictor ( Figure 1a ) . To avoid interfering with existing knowledge , the newly added hidden states are not fed back to the previously trained states . During training , all existing parameters are frozen , and only the newly added ones are trained . For inference , the new predictor is used for all domains , which is sometimes undesired as the new predictor is trained with only the last domain . In this paper , we propose a progressive memory bank for incremental domain adaptation in natural language processing ( NLP ) . Our model augments a recurrent neural network ( RNN ) with a memory bank , which is a set of distributed , real-valued vectors capturing domain knowledge . The memory is retrieved by an attention mechanism during RNN information processing . When our model is adapted to new domains , we progressively increase the slots in the memory bank . But different from Rusu et al . ( 2016 ) , we fine-tune all the parameters , including RNN and the previous memory bank . Empirically , when the model capacity increases , the RNN does not forget much even if the entire network is fine-tuned . Compared with expanding RNN hidden states , the newly added memory slots cause less contamination of the existing knowledge in RNN states , as will be shown by a theorem . In our paper , we evaluate the proposed approach on a classification task known as multi-genre natural language inference ( MultiNLI ) . Appendix C provides additional evidence when our approach is applied to text generation . Experiments consistently support our hypothesis that the proposed approach adapts well to target domains without catastrophic forgetting of the source . Our model outperforms the naı̈ve fine-tuning method , the original progressive neural network , as well as other IDA techniques including elastic weight consolidation ( EWC , Kirkpatrick et al. , 2017 ) . 2 RELATED WORK . Domain Adaptation . Domain adaptation has been widely studied in machine learning , including the NLP domain . For neural NLP applications , Mou et al . ( 2016 ) analyze two straightforward settings , namely , multi-task learning ( jointly training all domains ) and fine-tuning ( training one domain and fine-tuning on the other ) . A recent advance of domain adaptation is adversarial learning , where the neural features are trained not to classify the domain ( Ganin et al. , 2016 ; Liu et al. , 2017 ) . However , all these approaches ( except fine-tuning ) require all domains to be available simultaneously , and thus are not IDA approaches . Kirkpatrick et al . ( 2017 ) address the catastrophic forgetting problem of fine-tuning neural networks , and propose a regularization term based on the Fisher information matrix ; they call the method elastic weight consolidation ( EWC ) . While some follow-up studies report EWC achieves high performance in their scenarios ( Zenke et al. , 2017 ; Lee et al. , 2017 ; Thompson et al. , 2019 ) , others show that EWC is less effective ( Wen & Itti , 2018 ; Yoon et al. , 2018 ; Wu et al. , 2018 ) . Lee et al . ( 2017 ) propose incremental moment matching between the posteriors of the old model and the new model , achieving similar performance to EWC . Schwarz et al . ( 2018 ) augment EWC with knowledge distillation , making it more memory-efficient . Rusu et al . ( 2016 ) propose a progressive neural network that progressively increases the number of hidden states ( Figure 1a ) . To avoid overriding existing information , they freeze the weights of the learned network , and do not feed new states to old ones . This results in multiple predictors , requiring that a data sample is labeled with its domain during the test time . If we otherwise use the last predictor to predict samples from all domains , its performance may be low for previous domains , as the predictor is only trained with the last domain . Yoon et al . ( 2018 ) propose an extension of the progressive network . They identify which existing hidden units are relevant for the new task ( with their sparse penalty ) , and fine-tune only the corresponding subnetwork . However , sparsity is not common for RNNs in NLP applications , as sparse recurrent connections are harmful . A similar phenomenon is that dropout of recurrent connections Figure 1 : ( a ) Progressive neural network Rusu et al . ( 2016 ) . ( b ) One step of RNN transition in our progressive memory network . Colors indicate different domains . 𝒉 '' # $ 𝑀 ( ' ( ) ) Compute attention probability 𝒉 '' 𝒙 '' # $ 𝒙 '' Weighted sum by attention probability 𝑀 ( , - . ) Predictor 1 Predictor 2 Predictor 3 ( a ) Progressive neural network ( b ) Progressive memory yields poor performance ( Bayer et al. , 2013 ) . Xu & Zhu ( 2018 ) deal with new domains by adaptively adding nodes to the network via reinforcement learning . This approach may require a very large number of trials to identify the right number of nodes to be added to each layer ( Yoon et al. , 2019 ) . Li & Hoiem ( 2018 ) address IDA with a knowledge distillation approach , where they preserve a set of outputs of the old network on pseudo-training data . Then they jointly optimize for the accuracy on the new training domain as well as the pseudo-training data . Kim et al . ( 2019 ) ’ s variant of this approach uses maximum-entropy regularization to control the transfer of distilled knowledge . However , in NLP applications , it is non-trivial to obtain pseudo-training data for distillation . Memory-Based Neural Networks . Our work is related to memory-based neural networks . Sukhbaatar et al . ( 2015 ) propose an end-to-end memory network that assigns each memory slot to an entity , and aggregates information by multiple attention-based layers . They design their architecture for the bAbI question answering task , and assign a slot to each sentence . Such idea can be extended to various scenarios , for example , assigning slots to external knowledge for question answering ( Das et al. , 2017 ) and assigning slots to dialogue history for a conversation system ( Madotto et al. , 2018 ) . A related idea is to use episodic memory , which stores data samples from all previously seen domains ( thus it is not an IDA approach ) . This is used for experience replay while training on subsequent domains ( Lopez-Paz & Ranzato , 2017 ; Rebuffi et al. , 2017 ; Chaudhry et al. , 2018 ; d ’ Autume et al. , 2019 ) . Another type of memory in the neural network regime is the neural Turing machine ( NTM , Graves et al. , 2016 ) . This memory is not directly parameterized , but is read or written by a neural controller . Therefore , it serves as temporary scratch paper and does not store knowledge itself . Zhang et al . ( 2018b ) combine the above two styles of memory for task-oriented dialogue systems , where they have both slot-value memory and read-and-write memory . Different from the work above , our memory bank stores knowledge in a distributed fashion , where each slot does not correspond to a concrete entity or data sample . Our memory is directly parameterized , interacting in a different way from RNN weights and providing a natural way of incremental domain adaptation . 3 PROPOSED APPROACH . Our model is based on a recurrent neural network ( RNN ) . At each step , the RNN takes the embedding of the current input ( e.g. , a word ) , and changes its states accordingly . This is represented by hi = RNN ( hi−1 , xi ) , where hi and hi−1 are the hidden states at time steps i and i−1 , respectively . xi is the input at the ith step . Typically , long short term memory ( LSTM , Hochreiter & Schmidhuber , 1997 ) or Gated Recurrent Units ( GRU , Cho et al. , 2014 ) are used as RNN transitions . In the rest of this section , we will describe a memory-augmented RNN , and how it is used for incremental domain adaptation ( IDA ) . 3.1 AUGMENTING RNN WITH MEMORY BANKS . We enhance the RNN with an external memory bank , as shown in Figure 1b . The memory bank augments the overall model capacity by storing additional parameters in memory slots . At each time step , our model computes an attention probability to retrieve memory content , which is then fed to the computation of RNN transition . Particularly , we adopt a key-value memory bank , inspired by Miller et al . ( 2016 ) . Each memory slot contains a key vector and a value vector . The former is used to compute the attention weight for memory retrieval , whereas the latter is the value of memory content . For the ith step , the memory mechanism computes an attention probability αi by α̃i , j = exp { h > i−1m ( key ) j } , αi , j = α̃i , j∑N j′=1 α̃i , j′ ( 1 ) where m ( key ) j is the key vector of the jth slot of the memory ( among N slots in total ) . Then the model retrieves memory content by a weighted sum of all memory values , where the weight is the attention probability , given by ci = ∑N j=1 αi , jm ( val ) j ( 2 ) Here , m ( val ) j is the value vector of the jth memory slot . We call ci the memory content . Then , ci is concatenated with the current word xi , and fed to the RNN as input for state transition . Using the key-value memory bank allows separate ( thus more flexible ) computation of memory retrieval weights and memory content , compared with traditional attention where a candidate vector is used to compute both attention probability and attention content . It should be emphasized that the memory bank in our model captures distributed knowledge , which is different from other work where the memory slots correspond to specific entities ( Eric et al. , 2017 ) . The attention mechanism accomplishes memory retrieval in a “ soft ” manner , which means the retrieval strength is a real-valued probability . This enables us to train both memory content and its retrieval end-to-end , along with other neural parameters . We would also like to point out that the memory bank alone does not help RNN much . However , it is natural to use a memory-augmented RNN for incremental domain adaptation , as described below .

Summary: This paper proposes an extensible attention mechanism applied on the previous hidden state of an RNN and resulting in supplementary input for the next RNN step. For each added domain, new pairs of attentions key and values can be added to provide more capacity for the model. This method is applied in the context of incremental domain adaptation for NLP without the possibility of storing of old samples (episodic memory).