Progressive Memory Banks for Incremental Domain Adaptation
Authors: Nabiha Asghar
Lili Mou
Kira A. Selby
Kevin D. Pantasdo
Pascal Poupart
Xin Jiang
Venue:  (2020)
Keywords: natural language processing
domain adaptation

Abstract: This paper addresses the problem of incremental domain adaptation (IDA) in natural language processing (NLP). We assume each domain comes one after another, and that we could only access data in the current domain. The goal of IDA is  to build a unified model performing well on all the domains that we have encountered. We adopt the recurrent neural network (RNN) widely used in NLP, but augment it with a directly parameterized memory bank, which is retrieved by an attention mechanism at each step of RNN transition. The memory bank provides a natural way of IDA: when adapting our model to a new domain, we progressively add new slots to the memory bank, which increases the number of parameters, and thus the model capacity. We learn the new memory slots and fine-tune existing parameters by back-propagation. Experimental results show that our approach achieves significantly better performance than fine-tuning alone. Compared with expanding hidden states, our approach is more robust for old domains, shown by both empirical and theoretical results. Our model also outperforms previous work of IDA including elastic weight consolidation and progressive neural networks in the experiments.

Published as a conference paper at ICLR 2020
PROGRESSIVE MEMORY BANKS FOR INCREMENTAL DOMAIN ADAPTATION
Nabiha Asghar Lili Mou Kira A. Selby Kevin D. Pantasdo" Pascal Poupart Xin Jiang? 'Vector Institute for AI, Toronto. Canada
"Cherionn School of Computer Science, University of Waterloo, Canada
(nasghar, , kaselby, kevin wiwiin. pantasdo, ppoupart heeerrtttwwwweetre *Dept. Computing Science, University of Alberta; Alberta Machine Intelligence Institute (AMIl)
(ewwaterloo.co
double houblepower.. mouu gmaaiggmaiin i com
Noah's Ark Lab, Huawei Technologies, Hong Kong jiang. xin@huawei. com
ABSTRACT
This paper addresses the problem of incremental domain adaptation (IDA) in natu- ral language processing (NLP). We assume each domain comes one after another, and that we could only access data in the current domain. The goal of IDA is to build unified model performing well on all the domains that have encoun- tered. We adopt the recurrent neural network (RNN) widely used in NLP, but augment it with a directly parameterized memory bank, which is retrieved by an attention mechanism each step of RNN transition. The memory bank provides a natural way of IDA: when adapting our model to a new domain, we progressively add new slots to the memory bank, which increases the number of parameters, and thus the model capacity. We learn the new memory slots and fine-tune exist- ing parameters by back prok-propagttion. Experimental results show that our approach achieves significantly better performance than fine-tuning alone. Compared with expanding hidden states, our approach is more robust for old domains, shown by both empirical and theoretical results. Our model also outperforms previous work of IDA including elastic weight consolidation and progressive neural networks the experiments
INTRODUCTION
Domain adaptation aims to transfer knowledge from one domain (called the source domain) to another (called the target domain) in machine learning system If the data of the target domain are not large enough, using data from the source domain typically helps to improve model performance in the target domain. This is important for neural networks, which are data-hungry and prone to overfitting. In this paper, we especially focus on incremental domain adaptation (IDA where we assume different domains come sequentially
after another. We only have access the data in the current domain, but hope to build a unified model that performs well on all the domains that we have encountered (Xu et al. 2014 Rusu et al. 2016 Kirkpatriek et al. 2017).
Incremental domain adaptation is useful in scenarios where data are proprietary, or available only for short period of time (Li & Hoiem 2018). It is desired to preserve much knowledge possible in the learned model and not to rely on the availability of the data. Another application of IDA quick adaptation to new domains. If the environment of deployed machine learning system changes frequently, traditional methods like jointly training all domains require the learning machine be re-trained from scratch every time new domain comes. Fine-tuning neural network by
Equal contribution.
wtt. n hach/ IDA II our work, the domain is defined by datasets. Usually, the data from different genres or times typically have different underlying distributions.
'Our IDA code is available at
In the literature, IDA is sometimes referred to incremental, continull, or lifelong learning. We use "incremental domain adaptation" in this paper to emphasize that domain not changed continuously.

---

Published as a conference paper at ICLR 2020
few steps of gradient updates does transfer quickly, but it suffers from the catastrophic forgetting problem (Kirkpatrick et al.) 2017). Suppose during prediction a data point is not labeled with its domain, the (single) fine-tuned model cannot predict well for samples in previous domains, as it tends to "forget" quickly during fine-tuning.
A recent trend of domain adaptation in the deep learning regime is the progressive neural net- work (Rusu et al.] 2016), which progressively grows the network capacity if a new domain comes. Typically, this is done by enlarging the model with new hidden states and a new predictor (Fig- ure Ta). To avoid interfering with existing knowledge, the newly added hidden states are not fed back to the previously trained states. During training, all existing parameters are frozen, and only the newly added ones are trained. For inference, the new predictor is used for all domains, which is sometimes undesired as the new predictor is trained with only the last domain.
In this paper, we propose progressive memory bank for incremental domain adaptation in natural language processing (NLP). Our model augments recurrent neural network (RNN) with a memory bank, which is a set of distributed, real-valued vectors capturing domain knowledge. The memory is retrieved by an attention mechanism during RNN information processing. When our model is adapted to new domains, we progressively increase the slots in the memory bank. But different from Rusu et al. (2016), we fine-tune all the parameters, including RNN and the previous memory bank. Empirically, when the model capacity increases, the RNN does not forget much even if the entire network is fine-tuned. Compared with expanding RNN hidden states, the newly added memory slots cause less contamination of the existing knowledge in RNN states, as will be shown by a theorem.
In our paper, we evaluate the proposed approach on a classification task known as multi-genre nat- ural language inference (MultiNLI). Appendix C provides additional evidence when our approach applied to text generation. Experiments consistently support our hypothesis that the proposed approach adapts well to target domains without catastrophic forgetting of the source. Our model outperforms the naive fine-tuning method, the original progressive neural network, as well other IDA techniques including elastic weight consolidation (EWC, Kirkpatrick et al. 2017).
RELATED WORK
Domain Adaptation. Domain adaptation has been widely studied in machine learning, including the NLP domain. For neural NLP applications. Mou et al. (2016) analyze two straightforward set- tings, namely, multi-task learning (jointly training all domains) and fine-tuning (training one domain and fine-tuning the other). A recent advance of domain adaptation is adversarial learning, where the neural features are trained not to classify the domain (Ganin et al. 2016 Liu et al.) 2017). How- ever, all these approaches (except fine-tuning) require all domains be available simultaneously, and thus are not IDA approaches.
Kirkpatrick et al. (017)) address the catastrophic forgetting problem of fine-tuning neural networks, and propose a regularization term based on the Fisher information matrix; they call the method elastic weight consolidation (EWC). While some follow-up studies report EWC achieves high per- formance in their scenarios (Zenke et al.) 2017 Lee et al. 2017: Thompson et al., 2019). others show that EWC is less effective (Wen & Itti 2018 Yoon et al. 2018 Wu et al. 2018). Lee et al. (017)) propose incremental moment matching between the posteriors of the old model and the new model. achieving similar performance to EWC. SShwwaz et al. (2018) augment EWC with knowl- edge distillation, making it more memory-efficient.
Rusu et al.) (216)) propose a progressive neural network that progressively increases the number of hidden states (Figure [p). To avoid overriding existing information, they freeze the weights of the learned network, and do not feed new states to old ones. This results multiple predictors, requiring that a data sample is labeled with its domain during the test time. If otherwise use the last predictor to predict samples from all domains, its performance may be low for previous domains, the predictor is only trained with the last domain.
Yoon et al. 2018) propose an extension of the progressive network. They identify which existing hidden units relevant for the task (with their sparse penalty), and fine-tune only the corre- sponding subnetwork. However, sparsity is not common for RNNs in NLP applications, as sparse recurrent connections are harmful. A similar phenomenon is that dropout of recurrent connections
2

---

Published as a conference paper at ICLR 2020
Figure 1: (a) Progressive neural network Rusu et al. (2016). (b) One step of RNN transition our progressive memory network. Colors in- dicate different domains.
(a) Progressive neural network
(b) Progressive memory
h
Predictor
41-1
Predictor
Compute attention probability
Weighted sum by attention probability
Predictor
(ay)) M(no)
yields poor performance (Bayer et al.
2013). Xu Zhu (2018) deal with new domains by adap- tively adding nodes to the network via reinforcement learning. This approach may require a very large number of trials identify the right number of nodes be added each layer YYoon al.]
2019).
Li Hoiem 2018) address IDA with a knowledge distillation approach, where they preserve a set of outputs of the old network on pseudo-training data. Then they jointly optimize for the accuracy on the new training domain as well as the pseudo-training data. Kim et al. (2019)'s 's variant of this approach uses maximum-entropy regularization to control the transfer of distilled knowledge. However, in NLP applications, it non-trivial to obtain pseudo-training data for distillation.
Memory-Based Neural Networks. Our work is related to memory-basee neural networks. Sukhbaatar et al. (2015) propose an end-to-end memory network that assigns each memory slot entity, and aggregates information by multiple tecture for the bAbl question answering task, and assign slot to each sentence. Such idea can be extended to various scenarios, for example, assigning slots external knowledge for question swering (Das et al. 2017) and assigning slots to dialogue history for conversation system (Madotto al. 2018).
-aaattno--baedd layers. They design their archi-
related idea is to use episodic memory, which stores data samples from all previously seen do- mains (thus it is not an IDA approach). This is used for experience replay while training on subse-
quent domains (Lopez-Paz Papez-Paz&&e & Ranzato 2017 Rebulfi et al. 2017 Chaudhry et al. 2018 d Autumm
al. 2019).
Another type of memory in the neural network regime is the neural Turing machine (NTM, Graves al. This memory not directly parameterized, but is read written by a neural controller. Therefore, it serves as temporary scratch paper and does not store knowledge itself. [Zhang et al. (2018b) combine the above two styles of memory for task-oriented dialogue systems, where they have both slot-value memory and read-and-wrin memory.
Different from the work above, our memory bank stores knowledge in a distributed fashion, where each slot does not correspond to concrete entity or data sample. Our memory is directly parame- terized, interacting a different way from RNN weights and providing a natural way of incremental domain adaptation.
3 PROPOSED APPROACH
Our model is based on recurrent neural network (RNN). At each step, the RNN takes the embed- ding the current input (e.g., a word), and changes its states accordingly. This is represented by h; RNN(h,,11 1, ). where h, and h; are the hidden states time steps and i 1, respectively.
I; is the input at the ith step. Typically, long short term memory (LSTM, Hochreiter & Schmidhu- ber 1997) or Gated Recurrent Units (GRU, Cho al. 2014) are used as RNN transitions. In the RNN, and how it is used for incremental
rest of this section, we will describe a domain adaptation (IDA).
3

---

Published as a conference paper at ICLR 2020
3.1 AUGMENTING RNN WITH MEMORY BANKS
We enhance the RNN with an external memory bank, as shown in Figure Dp. The memory bank augments the overall model capacity by storing additional parameters in memory slots. At each time step, our model computes an attention probability to retrieve memory content, which is then fed to the computation of RNN transition.
Particularly, we adopt a key-value memory bank, inspired by Miller et al. (2016). Each memory slot contains a key vector and a value vector. The former is used to compute the attention weight for memory retrieval, whereas the latter is the value of memory content.
For the ith step, the memory mechanism computes an attention probability or by
ā. = exp ( im (,27) as
Øi,j
(I)
p==1" Oz,y'
(key is the key vector of the jth slot of the memory (among N slots in total). Then the model retrieves memory content by a weighted sum of all memory values, where the weight the attention probability, given by
where m,
C; c= c= j=1 az,
m
(val)
(2)
(val) is the value vector of the jthh memory slot. We call the memory content. Then, C; is Here, m concatenated with the current word æ; and fed to the RNN as input for state transition.
Using the key-value memory bank allows separate (thus more flexible) computation of memory retrieval weights and memory content, compared with traditional attention where a candidate vector is used to compute both attention probability and attention content.
should be emphasized that the memory bank in our model captures distributed knowledge, which is different from other work where the memory slots correspond to specific entities (Erie et al., (017]. The attention mechanism accomplishes memory retrieval in a "soft" manner, which means the retrieval strength is real-valued probability. This enables train both memory content and its retrieval end-to-end, along with other neural parameters.
We would also like point out that the memory bank alone does not help RNN much. However, it is natural to use a memory-augment RNN for incremental domain adaptation, as described below.
3.2 PROGRESSIVELY INCREASING MEMORY FOR INCREMENTAL DOMAIN ADAPTATION
The memory bank in Subsection 3.1 can be progressively expanded to adapt model in source domain to new domains. This done by adding new memory slots to the bank which are learned exclusively from the target data.
Suppose the memory bank is expanded with another M slots in new domain, in addition previous N slots. We then have N + .M slots in total. The model computes attention probability over the expanded memory and obtains the attention vector in the same way as Equations the summation is computed from to N M. This is given by
except that
(expand)
(expand)
N-M (expand) (val) ai,)
N+M Σ
mi
Dj'=1 0
To initialize the expanded model, we load all previous parameters, including RNN weights and the leamned N slots, but randomly initialize the progressively expanded M slots. During training, update all parameters by gradient descent. That is to say, new parameters are learned from their initializations, whereas old parameters are fine-tuned during IDA. The process is applied whenever a new domain comes, as shown in Algorithm
We would like to discuss the following issues.
4

---

Published as a conference paper at ICLR 2020
Freezing vs. Fine-tuning learned parame- ters. Inspired by the progressive neural net- work (Rusu al., 2016), we find it tempt- ing to freeze RNN parameters and the learned memory but only tune new memory for IDA. However, our preliminary results show that if we freeze all existing parameters, the in- creased memory does not add much to the model capacity, and that its performance is worse than fine-tuning all parameters.
Algorithm 1: Progressive Memory for IDA
Input: A sequence domains Du, Da,-.. Dn.
Output: A single model for all domains
Initialize a eemmory-aggmente
RNN
Train the model on Do
for Di.... , D,, do
Expand the memory with new slots Load RNN weights and existing memory banks Train the model by updating all parameters
end
Return: The resulting model
Fine-tuning vs. Fine-tuning while increas- ing memory slots. It is reported that fine- tuning a model (without increasing model ca- pacity) suffers from the problem of catastrophic forgetting (Kirkpatrick et al. 2017). We wish to investigate whether our approach suffers from the same problem, since we fine-tune learned param- eters when progressively increasing memory slots. Our intuition is that the increased model capacity helps to learn the new domain with less overriding of the previously learned model. Experiments confirm our conjecture, as the memory-augmented RNN tends to forget more if the memory size is not increased.
Expanding hidden states Expanding memory. Another way of progressively increasing model capacity is to expand the size of RNN layers. This setting is similar to the progressive neural net- work, except that all weights are fine-tuned and new states are connected to existing states.
However, we hereby show a theorem, indicating that the expanded memory results in less contami- nation/overriding of the learned knowledge in the RNN, compared with the expanded hidden states. The main idea is to measure the effect of model expansion quantitatively by the expected square dif- ference on h, before and after expansion, where the expectation reflects the average effect of model expansion in different scenarios.
Theorem 1. Let RNN have vanilla transition with the linear activation function, and let the RNN state at the last step h;-1 be fixed. For a particular data point, if the memory attention satisfies EN+4 O4. Ex=1 āsg.. then memory expansion yields lower expected mean squared differ- ence in h, than RNN state expansion. That is,
E
h.11 h
1
(4)
where h refers to the hidden states if the memory is expanded. h refers to the original dimen- sions of the RNN states, if we expand the size of RNN states themselves. Here, we compute the expectation by assuming weights and hidden states are iid from zero-mean Gaussian distribution (with variance o²).
We focus on one step of RNN transition and assume that h;-1 is the same when the model capacity is increased. Further, we assume that h2 is D-dimensional, that each memory slot my is d-dimensional, and that the additional RNN units (when we expand the hidden state) are also d-dimensional.
Proof Sketch:
We compute how the original dimensions in the hidden state are changed if we expand RNN. We denote the expanded hidden states by h and h, for the two time steps. We denote the weights connecting from h4-1 h, by W E Dxd We focus on the original D-dimensional space, denoted
as h The connection is shown in Figure Appendix A
We have
D Σ j=1
E [[ -h+l??] = B[Il h 11
=
E w h: hn-1) )
(5)
D
d
D
Σ
= ΣΣ
[h) (6)
Ø=1
j=1 1=1
D d. Var(w) Var(h) D 0²
5

---

Published as a conference paper at ICLR 2020
Table I: Corpus statistics and the baseline performance (% accuracy) of our BiLSTM model (without domain adaptation) and re- sults reported previous work.
Fic
Gov
Slate
Tel
Travel
# training samples Our Implementation
77k
77k
77k
83k
77k
65.0
66.5
56.2
64.5
62.7
Yu et al.
2018
64.7
69.2
57.9
64.4
65.8
Table 2: Results on two-domain adapta-
tion. F: Fine-tuning. V: Expanding vocab- ulary. H: Expanding RNN hidden states. M: Our proposed method of expanding memory. We also compare with previous work elas- tic weight consolidation (EWC, Kirkpatrick et al., 2017) and the progressive neural net- work (Rusu et al. 2016). For the statistical test (compared with Line 8), 1:4: p < 0.05 and f 4: < 0.01. The absence of an ar- row indicates that the performance difference compared with I Line 8 is statistically insignif- icant with p lower than 0.05.
% Accuracy on
#Line
Model
Trained on/by
8
65.01
61.23V
RNN
56.46
66.49V
3
RNN+
Mem
65.41
60.87
56.77V
67.01
5
S+T
66.02
70.00
S-T (F)
65.62
69.90
RNN Mem
S-+T (F+M)
66.23
70.21
8
S-+T (F+M))
67.55
70.82
S-T (F+H)
64,09*
68.35
10
S-T (F+H-V)
63.68
68.02
S (FWC)
66.02
64.10"
S-T (Progressive)
64.479
68.25V
Similarly,
E [[[ (T)) h4 h1 = ( Acl?] = DaroVar(Aon
(8)
where Ac c' The vectors and c are the current step's attention content before and after memory expansion. respectively, shown in Figure Dp. Appendix A (We omit the time step in the notation for simplicity.) We is the weight matrix connecting attention content to RNN states.
To prove the theorem, it remains to show that Var(Ac)) I o². We do this by analyzing how attention is computed at each time step, and bounding each attention weight. For details, see the complete proof in Appendix
In the theorem, we have an assumption E Lj=N+1 ā. S āنن. requiring that the total attention to existing memory slots is larger than to the progressively added slots. This is fairly reasonable because: (1) During training, attention trained an ad hoc fashion to newly-added information, and thus some 0s for 1 S j s N might be learned so that it is larger than a random memory slot; and (2) For a new domain, we do not add huge number of slots, and thus N+M 05, will not dominate.
N++
It is noted that our theorem is not to provide an explicit opimiizationggeeerriiaattooo bound for IDA, but shows that expanding memory is more stable than expanding hidden states. This is particu- larly important at the beginning steps of IDA, as the progressively growing parameters are randomly initialized and are basically noise. Although our theoretical analysis uses restricted setting (i.e., vanilla RNN transition and linear activation), it provides the key insight that our approach is appro- priate for IDA.
EXPERIMENTS
In this section, we evaluate approach an NLP classification task. In particular, choose the multi-genre natural language inference (MultiNL.), due to its large number of samples in various domains. The task is to determine the relationship between entailment, contradiction, and neutral. In Appendix C we conduct supplementary experiments on text generation with our memory-augmentce RNN for IDA.
sentences among target labels:
Dataset and Setup. The MultiNLI corpus (Williams et al. 2018 is particularly suitable for IDA, it contains training samples for 5 genres: Slate, Fictic fiction (Fic), Telephone (Tel), Government (Gov), and Travel. In total, we have 390k training samples. The corpus also contains held-out (non-training) labeled data in these domains. We split into two parts for valida- tion and test
NM. I also contains 5 genres without training samples, namely, 9/11, Face-to-facee letters, OUP, and Verbatim. We ignore these genres, because focus incremental domain adaptation instead of

---

Published as a conference paper at ICLR 2020
Performance on
Training domains
Fic
Gov
Slate
Tel
Travel
Fic
65.41
58.87
55.83
61.39
57.35
Fic + Gov
67.55
70.82
61.04
65.07
61,90
Fic Gov Slate
67.04
71.55
63.29
64,66
63.53
Fic } Gov >> Slate >> Tel
68.46
71.10
63.39
71.60
61.50
Fic + Gov + Slate + Tel Travel
69.36
72.47
63.96
69.74
68.39
Table 3: Dynamics of the progressive memory network for IDA with 5 domains. Upper-triangular values gray are out-of-domain (zero-shot) performance.
Group
Setting
Fic
Gov
Slate
Tel
Travel
Non-
In-domain training
65,41
67.01
59.30*
67.20
64.70
IDA
Fic Gov Slate + Tel Travel (multi-task)
70.60
73.30
63.80
69.15
67.074
Fic yy Gov >> Slate ; Tel Travel (F+V)
67.24
70.824
62.41*
67.62*
68.39
Fic + Gov > Slate - Tel > Travel (F+V+M)
69.36
72.47
63.96
69.74
68.39
IDA
Fic + Gov - Slate + Tel + Travel (EWC)
67. 67.124
68.714
59.90*
66.09
65.704
Fic > Gov -> Slate + Tel -> Travel (Progressive)
65.224
67.87%
61.13
66.96
67,90
Table 4: Comparing our approach with variants and previous work in the multi-domain setting. In
this experiment, we use the memory-augmented RNN as the neural architecture. Italics represent best results in the IDA group. t, 4; p < 0.05 and ft. 4: p < 0.01 (compared with F+V+M).
The first row in Table T shows the size of the training set in each domain. As seen, the corpus is mostly balanced across domains, although Te1 has slightly more examples.
For the base model, we train bi-directional LSTM (BiLSTM). The details of network architecture, training, and hyper-parameter tuning are given in Appendix B We see in Table P that we achieve similar performance to Yu et al. (2018). Furthermore, our BILSTM achieves an accuracy of 68.37 on the official MultiNLI test set which is better than 67.51 51 reported in the original MultiNLI paper (Williams et al. 2018) using BiLSTM. This shows that implementation and tuning are fair for the basic BiLSTM, and that our model is ready for the study of IDA.
Transfer between Two Domains. We would like to compare our approach with large number of baselines and variants. Thus, we randomly choose two domains as a testbed: Fic as the source domain and Gov as the target domain. We show results in Table
First, we analyze the performance of RNN and the setting (Lines 1-2 vs. Lines 3-4). As seen, the memory-augmentt but generally similar performance, compared with RNN (both with LSTM units). This shows that, in the non- non-transfer setting, the memory bank does not help the RNN much. However, this later confirms that the performance improvement is indeed due to our IDA technique, instead of simply better neural architecture.
RNN in the non-transfer RNN achieves slightly better
We then apply two straightforward methods of domain adaptation: multi-task learning (Line 5) and fine-tuning (Line 6). Multi-task learning jointly optimizes source and target objectives, denoted by "S+T." On the other hand, the fine-tuning approach trains the model on the source first, and then fine- tunes on the target. In our experiments, these two methods perform similarly on the target domain, which is consistent with Mou et al. (0166). On the source domain, fine-tuning performs significantly worse than multi-task learning, as it suffers from the catastrophic forgetting problem. We notice that, in terms of source performance, the fine-tuning approach (Line 6) is slightly better than trained on the source domain only (Line 3). This is probably because our domains are somewhat correlated as opposed Kirkpatrick et al. (2017), and thus training with more data on target slightly improves the performance on source. However, fine-tuning does achieve the worst performance on source compared with other domain adaptation approaches (among Lines 5-8). Thus, we nevertheless use the terminology ''catattoohiic forgetting." and our research goal is still to improve IDA performance.
The main results of our approach are Lines 7 and 8. We apply the proposed progressive memory network to IDA and we fine-tune all weights We see that on both source and target domains,
zero-shot learning. Also, the lahels for the official test set of MultiNI. are not publicly available, and therefore we cannot use to evaluate performance on individual domains. Our split of the held-out set for validation and test applics to all competing methods, and thus is a fair setting.
'Evaluation the official MultiNI. test set requires submission Kaggle.
7

---

Published as a conference paper at ICLR 2020
approach outperforms the fine-tuning method alone where the memory size is not increased (comparing Lines 7 and 6). This verifies our conjecture that, if the model capacity is increased, the new domain results in less overriding of the learned knowledge in the neural network. Our proposed approach is also "orthogonal" to the expansion of the vocabulary size, where target-specific words randomly initialized and learned the target domain. As seen, this combines well with our memory expansion and yields the best performance on both source and target (Line 8).
We now compare an alternative way of increasing model capacity, i.e., expanding hidden states (Lines and 10). For fair comparison, we ensure that the total number of model parameters after memory expansion is equal to the number of model parameters after hidden state expansion. We see that the performance of hidden state expansion is poor especially on the source domain, even we fine-tune all parameters. This experiment provides empirical evidence to our theorem that expanding memory is more robust than expanding hidden states.
We also compare the results with previous work on IDA. We re-implemen eet al. (2017). It does not achieve satisfactory results in our application. We investigate other pub- lished papers using the same method and find inconsistent results: EWC works well in some ap- plications (Zenke et al., 2017 Lee et al. 2017) but performs poorly on others (Yoon et al. 2018) Wu et al. 2018). Wen Itti (2018) even report near random performance with EWC. We also re-implement the progressive neural network (Rusu et al. 2016). We use the target predictor to do inference for both source and target domains. Progressive neural network yields low performance, particularly source, probably because the predictor is trained with only the target domain.
6 EWC (Kirkpatrick
We measure the statistical significance of the results against Line 8 with the one-tailed Wilcoxon's signed-rank test (Wilcoxon) 1945). by bootstrapping a subset of 200 samples for 10 times with replacement. The test shows our approach is significantly better than others, on both source and target domains.
IDA with All Domains. Having analyzed our approach, baselines, and variants on two domains in detail, we are now ready to test the performance IDA with multiple domains, namely, F Fic, Gov, Slate, Tel, and Travel, In this experiment, we assume these domains come one after another, and our goal
achieve high performance on all domains.
Table 3 shows the dynamics of IDA with our progressive memory network. Comparing the upper- triangular values (in gray, showing out-of-domain performance) with diagonal values, we see that our approach can be quickly adapted to the new domain in an incremental fashion. Comparing lower- triangular values with the diagonal, we see that our approach does not suffer from the catastrophic forgetting problem as the performance of previous domains is gradually increasing if trained with more domains. After all data are observed, our model achieves the best performance in most domains (last row in Table 3), despite the incremental nature of our approach.
We now compare our approach with other baselines and variants in the multi-domain setting, shown in Table 4 Due to the large number of settings, only choose selected subset of variants from Table for the comparison.
As seen, our approach of progressively growing memory achieves the same performance as fine- tuning on the last domain (both with vocabulary expansion), but for all previous 4 domains, we achieve significantly better performance than fine-tuning. Our model is comparable to multi-task learning on all domains. It should also be mentioned that multi-task learning requires training the model when data from all domains are available simultaneously. is an incremental approach for domain adaptation, and thus cannot be applied to the scenarios introduced in Section [ We include this setting mainly because we are curious about the performance of ann-icreementl domain adaptation.
We also compare with previous methods for IDA in Table Our method outperforms EWC and the progressive neural network in all domains; the results are consistent with Table
"Implementation hased htt.a//wwwwc..........eeee/eenea
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we propose progressive memory network for incremental domain adaptation (IDA). We augment an RNN with an attention-based memory bank. During IDA, we add new slots to memory bank and tune all parameters by back-propagation, Empirically, the progressive memory network does not suffer from the catastrophic forgetting problem as in naive fine-tuning. Our intu- ition is that the memory slots increase the neural network's ' model capacity, and thus, the new knowledge causes significantly less overriding of the existing network. Compared with expanding hidden states,
progressive memory bank provides pacity, shown by both a theorem and experiments. We also outperform previous work for IDA, including elastic weight consolidation (EWC) and the original progressive neural network.
robust way of increasing model
ACKNOWLEDGMENTS
This work was funded by Huawei Technologies, Hong Kong. Lili Mou is supported by the Amii Fellow Program, and the CIFAR AI Chair Program.
REFERENCES
Justin Bayer, Christian Osendorfer, Daniela Korhammer, Nutan Chen, Sebastian Urban, and Patrick van der Smagt. On fast dropout and its applicability to recurrent networks. arXiv preprint arXiv:1311.0701, 2013.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. arXiv preprint
2018.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pp. 1724-1734, 2014.
Cristian Ganescu-Niculescuse
and Lillian Lee. Chameleons in imagined conversations: A new approach understanding coordination of linguistic style in dialogs. In Proc. Workshop Cog- nitive Modeling and Computational Linguistics, pp. 76 2011.
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. Question answering on knowl- edge bases and text using universal schema and memory networks. In ACL, pp. 358-365, 2017.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. arXiv preprint arXivv1906.01076, 2019.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. Key-value retrieval networks for task-oriented dialogue. In SIGDIAL, pp. 37-49, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. JMLR, 17(1):006-2030. 2016.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-476, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1780, 1997.
Dahyun Kim, Jihwan Bae, Yeonsik Jo, and Jonghyun Choi. Incremental learning with maximum tropy regularization: Rethinking forgetting and intransigence. arXiv preprint arXiv:1902 008299 2019.

---

Published as a conference paper at ICLR 2020
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkknska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3511-3526 2017.
Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tad Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS, pp. 4652-4662, 2017.
Zhizhong Li and Derek Hoiem. 2018.
Learning without forgetting. IEEE TPAMI, 40(12):2935-2947,
Chia-Wel Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.
How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP, pp. 2122-2132, 2016.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classifica- tion. In ACL, pp. 1-10, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NewrlPS, pp. 6467-6476, 2017.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIGDIAL, pp. 285-294, 2015.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2seq: Effectively incorporating knowl- edge bases into end-to-end task-oriented dialog systems. In ACL, pp. 1468-1478, 2018.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, pp. 1400- 1409, 2016.
Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. How transferable are neural networks in NLP applications? In EMNLP, pp. 479. 2016.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning, GloVe: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 01.....66066777 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Bann Barnkkkkaa Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In ICML, pp. 4535-4544, 2018.
Tulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. hierarchical latent variable encoder-decoder model for gen- erating dialogues. In AAAI, pp. 3295-3301, 2017. 2017.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In NIPS, pp. 2440 2440, 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pp. 3104-3112, 2014.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In NAACI, pp- 2062-2068, 2019.
Shixian Wen and Laurent Itti. Overcoming catastrophic forgetting problem by weight consolidation and long-term memory. arXiv preprint arXiv: / 1805.0744), 2018.
10

---

Published as a conference paper at ICLR 2020
Frank Wilcoxon. 1945.
Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80 83,
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In NAACL-IILT, pp. 1112-1122, 2018.
Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang. Joost van de Weijer, and Bogdan Raducanu. Memory replay GANs: learning to generate images from new categories without forgetting. arXiv preprint arXiv:1809 180900208, 2018.
Jiaolong Xu, Sebastian Ramos, David Vázquez, Antonio M López, and Ponsa. Incremental domain adaptation of deformable part-based models. In BMVC, 2014.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, pp. 899-908. 2018.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. ICLR, 2018.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Oracle: Order robust adaptive continual learning. arXiv preprint arXiv: arXiv: 1902.09432, 2019.
Jianfei Yu, Minghui Qiu, Jing Jiang, Jun Huang, Shuangyong Song, Wei Chu, and Haiqing Chen. Modelling domain relationships for transfer learning retrieval-based question answering sys- tems in e-commerce. In WSDM, pp. 682-690, 2018.
Friedemann Zenke, Ben Poolc, and Surya Ganguli. Continual learning through synaptic intelligence. In ICMI,, pp. 3987-3995, 2017.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maxi- mization. arXiv preprint arXiv:1809.05972 2018a.
Zheng Zhang, Minlie Huang, Zhongzhou Zhao, Feng Ji, Haiqing Chen, and Xiaoyan Zhu. Memory-augmented dialogue management for task-oriented dialogue systems. arXiv preprint arXiv:1805.00150, 2018b.

---

Published as a conference paper at ICLR 2020
(a) Expand RNN states
(b) Expand memory
-
h
h
h;
X
X
1
h
hi
h;-1
ht
-
h
c'
X1
X
Figure Hidden state expansion vs. memory expansion at step
A PROOF OF THEOREM
Theorem 1. Let RNN have vanilla transition with the linear activation function, and let the RNN state at the last step h;-1 be fixed. For a particular data point, if the memory attention satisfies EN Oa. E C. then memory expansion yields lower expected mean squared differ- ence in h, than RNN state expansion. That is,
E
h
m h
S
A
1
(9)
where (m) refers to the hidden states if the memory is expanded. h; refers to the original dimen- sions of the RNN states, if we expand the size of RNN states themselves. Here, compute the expectation by assuming weights and hidden states are tid from a zero-mean Gaussian distribution (with variance o²).
Proof: Let - be the hidden state of the last step. focus on one step of transition and assume that h,-1 is the same when the model capacity is increased. We consider simplified case where the RNN has vanilla transition with the linear activation function. We measure the effect of model expansion quantitativelyy by the expected square difference on h; before and after model expansion.
Suppose the original hidden state h; is D-dimensional. We assume each memory slot is d- dimensional, and that the additional RNN units when expanding the hidden state also d- dimensional. We further assume each variable the expanded memory and expanded weights (W in Figure are iid with zero mean and variance o². This assumption is reasonable as it enables fair comparison of expanding memory and expanding hidden states. Finally, we assume every variable in the learned memory slots, i.e., Mýk. follows the same distribution (zero mean, variance o²).
We compute how the original dimensions in the hidden state are changed we expand RNN. We denote the expanded hidden states by h4-1 and h, for the two time steps. We denote the weights connecting from h;-1 to h; by W E Dxd We focus on the original D-dimensional space, denoted as h (+) The connection is shown in Figure We have
h
11
E
[[[
W
11 133]
(10)
D
27
E8
Σ
W h4-1
(1I)
1=1
Σ
I3
W
h
(12)
12

---

Published as a conference paper at ICLR 2020
W h4 [k:]) 21
Σ
I2
(13)
==1
k=1
Σ
(ūg [1:)] 31
(14)
| { (h-
(15)
= D - d- Var(w) Var(h)
(16)
Ddo o²
(17)
where (14p is due to the independence and zero-mean assumptions of every element in W and h; 15 due to the independence assumption between W and h;-1-
Next, we compute the effect of expanding memory slots. Notice that || h2 Wc. Here, hn"h) is the RNN hidden state after memory expansion. Ac def e' c. where and c' are the attention content vectors before and after memory expansion, respectively, the current time step W (c) is
the weight matrix connecting attention content to RNN states. The connection shown Figure Reusing the result of (16), we immediately obtain
E | (m)
1lll
(18)
= E
(cl)
(19)
Ddo ACa)
(20)
where Ac; is an element of the vector Ac
To prove Equation 2 it remains to show that Var(Acu) S 0². We now analyze how attention is computed.
Let ā1 ... NN M be the unnormalized attention weights over the N M memory slots. We notice
that aj.... ON remain the same after memory expansion. Then, the original attention probability given by aj
+ +... āN) for j -,.... .N. After memory expansion, the attention probability becomes d = āj/(āi +.... ĒN+M), illustrated in Figure We have
Ac d cc
(21)
N
N+M
Σ
(0/
a.) m +
Σ a,m,
(22)
1=1
N=
N+M
0j
m, +
m
01 + .... ŌN+M
σi + ....... ÖN
01
+
j=
N+M m. Σ
6
(23)
α1 ĀN+M
01 + ... + UN+M
N=1+1
N+M
de
B,m,
(24)
where
x 4 40
ifSSiSN
3, der
+ ØN+M
(25)
#N+1SJSN+M
0N+M
We omit the time step in the notation for simplicity.
13

---

Published as a conference paper at ICLR 2020
Memory
Unnormalized measure
Original attr. prob.
Expanded attn. prob. d a2
1
&
m2
22
mN äN
dyy
d'y
mx+1
@N+1
ax+1
MM++
@w+M
ax+M
Figure 3; Attention probabilities before and after memory expansion.
By our assumption of total attention N
āg S Σ}= ag. we have
|ر,| S laاl:
LA S N M
(26)
Then, we have
(cr) = El(ck - cx
4A k d
(27)
d RR [ - c||]]
(28)
d
N M Σ
E
3ymyk
(29)
N M
27
E
3ymyk
(30)
d kml
j=1
d
N+
E Σ
(3)
(31)
d
k=1
j=1
N+
Σ
E
[
(32)
d
d
N+M
Σ
BISHI0²
(33)
d k=1
j=1
N+M Σ
22
33
(34)
j=1
N+M
(a,)
(35)
0
(36)
Here, 31 is due to the assumption that myj is independent and zero-mean, and 33 is due to the
independence assumption between 8y and m,k- To obtain (6), we notice that Σ j=1 af = 1 with
a < (MI N + M). Thus, M (az)² < 1, concluding our proof.
B HYPERPARAMETERS
We choose the base model and most of its settings by following the original MultiNL paper (Williams et al. 2018): 300D RNN hidden states, 300D pretrained GloVe embeddings (Penning- ton al. (014) for initialization, batch size of 32, and the Adam optimizer for training. The initial
14

---

Published as a conference paper at ICLR 2020
88
71
Validation Aoc. T
Validation Acc. on
70.5
60
69.5
100
600
of Memory Slots
# of Memory Slots
(b)
Figure 4: Tuning the number of memory slots to be added per domain in the MultiNLI experiment. The two graphs show validation performance of our IDA model S-TT (F+M+V).
learning rate for Adam is tuned over the set {0.3, 0.03, 0.003, 0.0003, 0.00003). It is set to 0.0003 based on validation performance.
For the memory, we set each slot to be 300-dimensioonal, which is the same as the RNN and em- bedding size.
We tune the number progressive memory slots Figure 4 which shows the validation perfor- mance the source (Fic) and target (Gov) domains. We see that the performance is close to fine-tuning alone only one memory slot is added. improves quickly between and 200 slots, and tapers off around 500. We thus choose add 500 slots for each domain.
C ADDITIONAL EXPERIMENT ON DIALOGUE GENERATION
We further evaluate our approach on the task of dialogue response generation. Given an input text sequence, the task is to generate an appropriate output text sequence as response in human- computer dialogue. This supplementary experiment provides additional evidence of our approach in generation tasks.
Datasets, Setup, and Metries. We use the Cornell Movie Dialogs Corpus DDaneccu -anssuuu- Mizil & Lee 2011) the source. It contains ~220 message-response pairs from movie transcripts. We use a 200k-10k- 10k saining-validation-- split.
For the target domain, manually construct very small dataset from the Ubuntu Dialogue Cor- pus (Lowe et al. 2015) to mimic the scenario where quick adaptation has to be done to new domain with little training data. In particular, choose a random subset of 15k message-response pairs, and use 9k-3k-3k split.
The base model is a requence-to-sequenee (Seq2Seq) neural network (Sutskever al. 2014 with attention from the decoder to the encoder. We use a single-layer RNN encoder and single-layer RNN decoder, each containing 1024 cells following [Sutskever al. (014). We GRUs instead of LSTM units due to efficiency concerns. We have separate memory banks for the encoder and decoder. since they are essentially different RNNs. The source and target vocabularies are 27k and 10k, respectively. Each memory slot is 1024D, because the RNN states are 1024D in this experiment. For each domain, we progressively add 1024 slots; tuning the number of slots is done in a manner similar to the MultiNLI experiment. As before, use Adam with initial learning rate of 0.0003 and other default parameters.
Following previous work, use BLEU-2 (Eiic et al.] 2017: Madotto al. 2018) and average
Word2Vec embedding similarity (W2V-Sim, Serban et al., 2017; Zhang et al. uation metrics. BLEU-2 is the geometric mean of unigram and bigram word precision penalized by length, and correlates with human satisfaction to some extent (Liu et al., 2016). W2V-Sim is defined as the cosine similarity between the averaged Word2Vec embeddings of the model outputs
2018a) as the eval-
15

---

Published as a conference paper at ICLR 2020
#
BLEU-2
W2V-Sim
Line
Model
Trained on/by
S
T
S
2.842
0.738
0.480
0.456
RNN
2
0.795
1,265
0.454
0.480
3
RNN+
Mem
3.074
0.712
0.498
0.471
4
0.920
1.287
0.462
0.487
5
S+T
2.650
0.889
0.471
0.462
6
S-+T(F)
1,210
1.101
0.509
0.514
7
RNN+ Mem Mem
S-+T (F+M)
1.4354
1.207
0.526
0.522
8
S-+T (F+M+V)
1.637
1.652
0.522
0.525
9
>T (F+H)
1.036
1.606-
0,503
0.495
10
S-+T T (F+H+V)
1,2574
1.419*
0.504
0.492
11
S +T (EWC)
1,397U
1.3824
0.5134
0.514
12
S S-+T (Progressive)
1,299
1.408
0.5024
0.503
Table 5: Results on two-domain adaptation for dialogue response generation. F: Fine-tuning. V: Expanding vocabulary. H: Expanding RNN hidden states. M: Our proposed method of expand- ing memory. We also compare with EWC (Kirkpatrick et al., (017) and progressive neural net- work (Rusu al. 2016). 1.4: p < 0.05 and ft, V: < 0.01 (compared with Line 8).
Prompt
Response
i' I'm steve. what's your name?
my name is opera
love you man
love you too
so how's s going with your new roomie?
it should be fine
i didn't 't do anything wrong d
we don't have much experience
just got messed up stomach
know, you had do some things
what are you trying to handicap mc?
this is not really who am what do we do with it?
im scared
i kinda forgot the fix tried year ago
can you try the output ?
how would remaster the ubuntu install
could you have link both access to the network connection
my terminal stuck
what version it?
i am rebooting now
so your issue is great work cd, sometimes get some getting an expert for it
i get installation error
what does the kernel-package do?
you will need to get it to see there a setting
what directory should i be in?
you should he able to install the grub ed at the drive
Table 6: Sample outputs of our IDA model S-T (F+M+V) from Table 5.
and the ground truths. Intuitively, BLEU measures hard word-level overlap between two sequences, whereas W2V-Sim measures soft similarity in distributed semantic space.
The results for dialogue response generation are shown in Table We see that BLEU-2 and W2V similarity are not necessarily consistent. For example, the augmene- RNN trained solely on source achieves the best source BLEU-2, whereas the proposed progressive memory has the highest W2V cosine similarity on S. However, our model variants achieve the best performance on most metrics (Lines 7 and 8). Moreover, it consistently outperforms all other IDA approaches. Following the previous experiment, we conduct statistical comparison with Line The test shows that our method is significantly better than the other IDA methods.
Results.
In general, the evaluation of dialogue systems noisy due the lack of appropriate metrics (Liu et al. 2016). Nevertheless, our experiment provides additional evidence of the effectiveness of our approach. It also highlights our model's viability for both classification and generation tasks.
Case Study. Table 6 shows sample outputs of our IDA model on test prompts from the Cornell Movie Corpus (source) and the Ubuntu Dialogue Corpus (target). We see that casual prompts from the movie domain result in casual responses, whereas Ubuntu queries result in Ubuntu-related re- sponses. With the expansion of vocabulary, our model is able to learn new words like "grub"; with progressive memory, it learns Ubuntu jargon like "network connection." " This shows evidence of the success of incremental domain adaptation.
16


Reviews:
[{"review_id": "r1gnymt9cH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n###Summary###\nThis paper introduces incremental domain adaptation for natural language processing, assuming that each domain comes one after another and only the current domain can be accessed in the application scenario.  The basic framework of this paper is based on RNN but augmented with the directly parameterized memory bank. \n\nThe memory bank of this paper is a set of distributed, real-valued vectors capturing domain knowledge. When the model is adapted to the new domain, the model progressively increases the slots in the memory bank. \n\nThe paper evaluates the proposed approach on an NLP classification task, i.e. multi-genre natural language inference (MultiNLI). The dataset used in this paper includes 5 genres: Slate, Fiction, Telephone, Government and Travel. \n\nIn the experiments, the paper performs the dynamics of the progressive memory network for IDA as well as compares the proposed method with variants and previous work in the multi-domain setting. \n\n\n### Novelty ###\n\nThis paper proposes incremental domain adaptation, which is inspired by Li & Hoiem's work. The setting assumes that each domain comes one after another and only one domain can get accessed. This setting is interesting as we will encounter this setting in the real application scenarios, i.e., the domain knowledge in the real domain is unpredictable. Thus, the problem setting provides some novelty. However, I am not sure whether assuming that we can only get access to one domain is reasonable as we can always save the data for the domain we have already seen. \n\nFrom the perspective of the method, this paper incorporates the memory bank to the RNN, which is not new in the machine learning research area, but heuristic enough for the transfer learning community. \n\n\n\n###Clarity###\n\nOverall, the paper is well organized and logically clear. The proposed claims are well supported by the experiments and analysis. The images are well-presented and well-explained by the captions and the text. \n\n\n###Pros###\n\n1) The paper proposes an incremental domain adaptation scenario where one domain appears another and only the data from the current domain can be accessed, which is interesting and heuristic to the domain adaptation research community. \n2) The paper is applicable to many practical scenarios since the data from the real-world application is typically from multiple domains and the data is from one domain at a time. \n3) The paper is overall well-organized and well-written. The claims of the paper are verified by the experimental results.\n\n\n###Cons###\n\n1) The paper has a good motivation for the setting, however, one of the critical drawbacks of this paper is that the papers fail to compare with the state-of-the-art baselines. I understand that this paper has a new setting, but since the authors also compare the proposed method with the \"multi-task\" learning, it will be helpful to compare with state-of-the-art multi-task or multi-source baselines. \n2) The experimental results provided in this paper are weak. In Table 4, we found that sometimes, the IDA method performs worse than the multi-task baselines. \n3) The paper presents no ablation study or analysis of the experimental results. The effectiveness of the memory bank, fine-tuning/freezing learning parameters is unclear. \n\nIt will be also interesting to see how does the proposed method performs on large-scale visual datasets.\n\n\n\nBased on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs. \nI am willing to improve my rating if the authors can address my following concerns. To improve the rating, the author should explain the following questions:\n1). Why assuming that we can only get access to the data from one domain is a reasonable setting, since we can always save the data (or at least the statistics about the data) form the domains we have already observed.\n2). Can the proposed approach generalize to visual domain adaptation, i.e. on the visual task instead of NLP task?\n3). The drawbacks I mentioned in the paper Cons section.\n\n\n\n#################### Updated Review  ###########\nThe authors have addressed most of my concerns I proposed in the initial review, thus I raise my score to 6 weak accept. \n\n1) In the response to the review, the authors claim that from the data privacy's perspective, it's reasonable to assume that we can only get access to one dataset at one time, which makes sense.\n2) I would be really interested in how the similar idea works on vision data, which could be a future work for this paper. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gIw_JlcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Summary\n\nThis work proposes to use an augmented RNN model to address the incremental domain adaptation problem. In particular, it designs the progressive memory bank approach which expands the memory capacity by adding parameters every time a new task comes in. The RNN retrieves knowledge from the memory bank via key-value attention. A proof in a highly simplified case is given in addition to empirical results showing that expanding the memory bank is better than expanding the RNN states.\n\n*** Strengths\n\n1. Section 3 is well-written. The methods and motivations are illustrated clearly.\n\n2. Comprehensive experiments are conducted. Supportive results for the arguments presented in Section 3 are therefore demonstrated.\n\n\n*** Weaknesses\n\n1. Regarding Table 2., multiple runs of different sources and targets would be helpful to better understand the effectiveness of the proposed methods in the 2-domain set-up.\n\n2. The choice of key-value memory bank is not intuitive. A comparison between this memory and the traditional attention can help demonstrate the validity of this choice.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hklyku7AYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an extensible attention mechanism applied on the previous hidden state of an RNN and resulting in supplementary input for the next RNN step. For each added domain, new pairs of attentions key and values can be added to provide more capacity for the model. This method is applied in the context of incremental domain adaptation for NLP without the possibility of storing of old samples (episodic memory).\n\nPros:\n- Extensive ablation study with the different possible combinations of methods\n- Very interesting comparison between expanding the memory (i.e. attention) and expanding the hidden states. Using the attention results in better results for a same number of added parameters and the activations sizes stay the same even when the attention is extended with new pairs.\n- Paper is well written/motivated\n\nWeaknesses:\n- MultiNLI seem to have too much correlation between tasks. It would have been better to be able to observe catastrophic forgetting for the source domain. In the appendix, the metrics have really strong disagreement so it is tough to judge for these two corpuses. \n- When you give the numbers for multi-task learning, you should use your extended memory method to be fair with MT learning. I would just be interested to see it, just as a proper upper bound.\n\nOtherwise, the paper proposes a novel method which works well in practice so I am leaning towards acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an RNN based approach to incremental domain adaptation in natural language processing, where the RNN is progressively augmented with the parameterized memory bank which is shown to be better than expanding the RNN states.

Reviewers and AC acknowledge that this paper is well written with interesting ideas and practical value. Domain adaptation in the incremental setting, where domains come in a streaming way with only the current one accessible, can find some realistic application scenarios. The proposed extensible attention mechanism is solid and works well on several NLP tasks. Several concerns were raised by the reviewers regarding the comparative and ablation studies, which were well resolved in the rebuttal. The authors are encouraged to generalize their approach to other application domains other than NLP to show the generality of their approach.

I recommend acceptance.
Decision: Accept (Poster)