Memory-Based Invariance Learning for Out-of-Domain Text Classification
Authors: Chen Jia
Yue Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Domain generalization; key-value memory; invariance learning; transfer learning

Abstract: We investigate the task of out-of-domain (OOD) text classification with the aim of extending a classification model, trained on multiple source domains, to an unseen target domain. Recent studies have shown that learning invariant representations can enhance the performance of OOD generalization. However, the inherent disparity in data distribution across different domains poses challenges for achieving effective invariance learning. This study addresses this issue by employing memory augmentations. Specifically, we augment the original feature space using key-value memory and employ a meta-learning-based approach to enhance the quality of the invariant representations. Experimental results on sentiment analysis and natural language inference tasks show the effectiveness of memory-based method for invariance learning, leading to state-of-the-art performance on six datasets.

Memory-Based Invariance Learning for Out-of-Domain Text Classification
Chen Jia"
Yue Zhang
SI-TECH Information Technology Fudan University j jachennesttaake@@aii.. com
Westlake University
Westlake Institute for Advanced Study zhangyue@wes lake. edu. cn
Abstract
We investigate the task of out-of-domain (OOD) text classification with the aim of ex- tending a classification model, trained on mul- tiple source domains, to an unseen target do- main. Recent studies have shown that learn- ing invariant representations performance of OOD generalization. However, the inherent disparity in data distribution across different domains poses challenges for achiev- ing effective invariance learning. This study addresses this issue by employing memory aug- mentations. Specifically, we augment the origi- nal feature space using key-value memory and approach to
Sture festume
Manoryy Soura femum
Tartyt Foxtare
Minory
Tangetooaiiee
Menory
Memory
enhance the
Meniey Aup
Mennicy
Sour co-oomat subspace
Domalo- invaiii-ivvaaaantt subspace
Target domain subspace
employ meta-learning-bans hance the quality of the invariant representa- tions. Experimental results on sentiment analy- sis and natural language inference tasks show the effectiveness of memory-based method for invariance learning, leading to state-of-the-art performance six datasets.
Figure I: Memory-based invariant representation learn- ing.
In this paper, we focus on multi-source do- main generalization setting where there are multi- ple source domains available for training. In recent years, domain-invarrint representation learning has shown high effectiveness in multi-source DG (Ben- David et al.. 2010; Ganin et al., 2016). Most existing approaches use the same model param- eters across domains to construct domain-shared feature space for domain-invariann representation learning (Li et al., 2018b; Albuquerque et al., 2019; Guo et al., 2020; Jia and Zhang, 2022b). How- ever, the intrinsic distribution discrepancy across domains poses challenges for distribution matching in order to learn a domain-invariant feature space. Inspired by recent work (Khandelwal et al., 2019. 2020; Zheng et al., 2021), which demonstrates that memory vectors can serve as rich feature augmenta- tions for neural models, we propose to adopt mem- ory augmentations to improve representation learning. As shown in Figure 1, the traditional parameter sharing mechanism produces distinct feature distributions between source do- mains (left) and target domains (right) due to the intrinsic domain discrepancy. To address this, we
1 Introduction
Text classification has made remarkable progress in recent years, thanks to the advancements in deep neural networks such as Transformer (Vaswani et al., 2017) and pretrained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020). However, these learning systems heavily rely the assumption that the training and test sets come from the same domain. When there is a significant discrepancy between the test domain (also known as the target domain) and the training domains (also known as source do- mains), the performance of traditional learning sys- tems suffers significant declines (Blanchard et al., 2011; Muandet et al., 2013). Domain general- ization (DG) aims to address this out-of-domain (OOD) problem, which is practical and challeng- ing issue, particularly when the labeled and unla- beled information of the target domains unknown during the training phase.
Coorresponding author.

---

use memory augmentations to alleviate the discrep- ancy of feature distributions between source and target domains and improve the invariant feature distribution, constructing a domain-invariant fea- ture subspace (middle).
recognition tasks (Sun and Saenko, 2016; Li et al., 2018b; Arjovsky et al., 2019). However, these methods face challenges when there signif- icant discrepancy between the source and target domains. The invariant classification model across source domains cannot casily adapt to unseen tar- get domains. To address this issue, some studies in objective recognition (Li et al., 2018a; Balaji et al., 2018; Jía and Zhang, 2022a) and semantic parsing (Wang al., 2021a) employ meta-learning- based approaches with episodic training strategies to improve model adaptation to domain shifts. In contrast to these works, we aim to learn explicit memory augmentations for domain transfer. In contrast, the meta-learner in our method aims to learn domain-invarian feature space by learn- ing memory augmentation. The novelty of our work is reflected in the design of meta-learning ob- jectives. The advantage of our design lies in the ability to leverage an additional memory network to learn more robust feature representations across domains.
key-value mem- ory network (Miller et al., 2016) improve the Transformer model (Vaswani al., 2017) by fea- ture augmentation. Specifically, we employ a meta- learning strategy to learn memory augmentations for achieving the invariant representation distri- bution across domains. In each training episode, source domains are randomly split into the meta- target and mota-source domains to simulate domain shifts. Consequently, we propose bi-level opti- mization objective learn memory augmentations for domain invariance. The inner-loop objective to minimize the meta-source risk w.r.t. the Trans- former parameters, while the outer-loop objective is to minimize the domain discrepancy between the meta-souree and meta-targee target samples w.r.t. the key- value memory, based on the optimized Transformer parameters from the inner-loop. As result, after the meta-test phase, the memory augmentations improve the domain invariance between the source domain and unseen target domains.
To achieve this goal,
Recently, there has been increasing interest in DG for text classification. Ben-David al. (2022) learn an example-based prompt for each instance for classification. In contrast, we focus on lcarning a more general memory augmentation that can address domain shifts comprehensively. Jia and Zhang (2022b) utilize a distribution alignment method to enhance domain invariance for DG. Tan et al. (2022a) adopt a memory enhanced supervised contrastive learning method for DG. In comparison, we propose the use of key-value memory to explic- itly augment feature representations and improve domain invariance in the feature space.
We evaluate our method on sentiment analy- sis and natural language inference (NLI) tasks. The results show that the learned memory by bi- level optimization provides better augmentations the feature representation compared with the tra- ditional learning strategy. Our method achieves sfate-of-the-an results on six datasets, outperform- ing a range of strong baselines. To the best of our knowledge, we are the first to leverage mem- ory network for improving domain -nvariinn rep- resentation learning. The code will be released at https://gi thub. com/ cww./jiacceeeestlllaeee.... LaMM//
memoory-ehaamced
Memory-based model adaptation. The augmen- tation of neural networks with previous memory has proven effective for model adaptation in the testing phase (Santoro et al., 2016). Prior works improve network predictions using memory banks, which serve as a continuous cache for storing long- range contextual information (Khandelwal et al., 2019, 2020; Zhong et al., 2022). Memory banks have also shown utility for task adaptation (San- toro et al., 2016; Wang et al., 2021b). However, there are limited studics memory- cross- domain transfer. Existing works (Asghar et al., 2018: Zheng al., 2021) rely on target-domain unlabeled data for domain transfer. However, these methods cannot be directly applied to DG since both labeled and unlabeled target information is
2 Related Work
Domain generalization. In this work, we specif- ically focus on multi-source domain generalization (DG) (Blanchard et al., 201 1; Muandet et al., 2013), which offers broader application opportunities com- pared to the single-source scenario (Qiao et al., 2020). With the advancements in deep neural net- works (Li et al., 2017), DG has achieved promising results. Existing methods primarily aim learn domain-invariant representations across source do- mains to enhance nuttof-distribution (OOD) ro- bustness, which have proven effective in objective

---

unknown during training. In contrast, we leverage memory to optimize the transferability from source domains to target domains through a meta-learning strategy.
to-end by gradient backpropagation (Sukhbaatar al., 2015). Following previous work (Miller al., 2016; Lample et al., 2019), the overall structure of our memory layer consists of a query network and a value lookup table.
To the best of our knowledge, only one existing memory-based work for DG refers to (Tan et al., 2022b), which leverages the memories of source- domain samples augment contrasting features for computing supervised contrastive loss. Our work differs significantly from (Tan et al., 2022b). Firstly, our memory network is trainable, whereas they employ static source-domain banks that are not optimized during training. Secondly, we ex- plicitly utilize memory as feature augmentation to enhance invariant representation learning, whereas they employ memory as contrasting features for computing the contrastive loss.
Key-value memory. Given hidden state of one position from the previous layer h Rd the query network acts function q h + g(h) E Rd,, mapping from a d-dimensional hidden vector into latent query space with the dimensionality dg. In this paper, q(-) is a linear mapping or multi- layer perceptron to reduce the dimensionality of hidden space to a nower-dimensional query space for distance computation w.r.t. the keys.
Given query (h) and set of keys K. ( kyq} that consists of [K] dy-dimensional
vectors, we first compute the dot-product similarity
Feature augmentation. Previous studies have shown that model generalization can be improved by augmenting features through the mixing of fea- ture vectors (Verma et al., 2019). In computer vi- sion, prior works learn interpolation for semantic changes (Upchureh et al., 2017) or perturbs latent features with random noises using mix-up tech- niques (Zhou et al.; Li et al., 2021; Zhao et al., 2021). contrast, we focus on learning memory augmentations to enhance domain invariance the feature space.
between the query and each key { k E ..C]..
X.] For each
exp(o(h)TTk kk) Or Σ/ exp(c(h) Tky)
(1)
Given a set of memory values V
(Vi...., ffyc that consists of [A]] di-dimensional vectors, the function of the key-value memory be represented weighted sum of memory val- ues:
K
3 Method
m m(h)
Vg
(2)
As illustrated in Figure 2, the proposed model com- prises (a) vanilla Transformer enhanced by (b) a key-value memory. Furthermore, (c) the output layer is responsible for text classification, while (d) the domain discriminators handle domain classifi- cation tasks.
Memory-augmented network. We use the ag- gregated memory by key-value memory sublayer feature augmentations for the original Transformer model to improve domain transfer. Particularly, we perform the feature augmentation through resid- ual connection. Let g x g(æ) R& denote the Transformer model that mapping from an input text to feature vector, we represent the memory- augmented network gm x + 9m((x) Idd as follows
The memory serves to enhance the feature rep- resentation and mitigate domain-specific feature distributions. To accomplish this, we employ a key-value memory bank that learns the appropriate feature augmentations (Section 3.1). To address do- main shifts through memory augmentations, we in- troduce an episodic training strategy (Section 3.2).
The training objective of the key-value memory can be formulated as bi-level optimization (Section 3.3).
9m (1- + (mo g(x)), (3)
Lastly, present the overarching meta-training and meta-test algorithms (Section 3.4).
where À represents the coefficient that balances the original features and augmented memory.
3.1 Key-Value Memory-Augmented Network
3.2 Episodic Training Procedure
We consider key-value memory layer as a func- tion m dd + Rd which can trained end-
Following Li et al. (2018a); Balaji et al. (2018), we leverage an episodic training procedure to simulate

---

Task Loss br
Domain Class, Loss l
(d)
Dmmam Disc.
Domann DISC.
Doman DISC. Jn
(c) Output Layer
Onte Classification
h:
Oll
Domain
Invariance
Learning
-
Basic T Transfommr Sublayee er
Query-Ker Key Network
(a) Transformer Layer
g:
Basic Transformer Layer
Protuct
Values
(b) Memory m: 91
Meta-Target
Meta-Source #1
Meta-Sove Source
Figure 2: Overall structure of the proposed model.
Algorithm Episodic training process.
value memory network 0, on the mini-batch of meta-target sample Dic and meta-source samples Dtr using the optimized Transformer parameters 0 (line 7).
Input:
Source domains {S1, .n,
Input parameters:
Output: Optimized memory net
I: while not converge not reach stopping conditions do
2: Randomly select meta-target domain D., S
3.3 Memory-Basee Invariance Learning
3: The meta-source domains Drr S 1)
4:
7") for E (I,...,7) do
Based on the episodic training process, we now describe in detail the optimization objectives w.r.t. the training samples and parameters.
5:
Sample mini-hatch 12r C Der and Die Die
6:
Optimize Transformer parameters on Da. 7: Optimize the key-value memory parameters 0m
using the optimized Transformer parameters 0;
8: end for
Domain-invariant representation learning ob- jective. Given n training domains, we need n do- main discriminators differ each domain from the other domains. To simplify the presentation, an unified function symbol fa to denote the domain discriminator between the meta-test (meta-target) data Dic and the meta-training (meta-source) data Dyr- The domain classification objective can be represented as the binary cross-entropy loss:
9:
end while
the domain shifts. Each episode can be viewed as a meta-task to learn how to learn a better key-value memory for tackling the domain shifts between source domains and the unseen target domains. In particular, the meta- meta-task in this paper is specified as learning memory augmentations improve the in- variance of feature representations across domains,
Ls = Σ p(CE) o gm (2) IpEDa) (4)
ZEDe,UDie
The domain-invariant representaion learning solves the following minimax optimization objec- tive w.r.t. the domain discriminator fa and key- value memory network m:
brief view of the episodic training process is shown Algorithm Given set of source- S,). in
domain training samples S = each training episode, we first randomly select meta-target domain and the rest serve as the meta- source domains (lines 2-3). Then, in each training iteration E [T]. we first optimize the Transformer model g X' I parameterized by 0g and task output layer h Rd y parameterized by 0% samples Dtr
max min La(Og. Om, O fai Dre, D(r)
(5)
8m 0/a
Theoretically, following Ben-David et al. (2010), the above minimax training objective aims mini- mize the H--iieereence for obtaining the invariant representation between the meta-training (meta- source) and meta-test (meta-target) domains.
over the (line 6). Then, we optimize the parameters of key-
of

---

Algorithm 2 MMea--triniin procedure.
Algorithm 3 Mcla-test procedure.
Input: Source domains S2.........
Input:
Source domains S {S1, S2................... S2...., ....
Parameters (randomly initialized): 0g. An. f.c-
Parameters (by meta-training): A Parameters (randomly initialized): 0g. 0,,
Output:
Optimized memory net m²*
I:
while not converge not reach stopping conditions do
Output: Optimized model fi" "ogim
2: Randomly select meta-target domain Di. Sa E S
1: while not converge or not reach stopping conditions do Randomly select training domain D.. S
3: The meta-source domains Der = =S Din
2:
4: for i (I,..., J) de
3:
for & E (1.... I] do
5:
Sample mini-batch Der C Der and Dio Die task ohj.
4:
Sample mini-batch Der Der
6:
I, (9 Am.
(.: Ox: 12r)
5:
L. + L0, Hin. Ou: Der)
E task obj. T- min La W.L.t D min La W.r.t. 9
7:
9n 0n 'VV,,,
6:
On < 0r VVo L
8:
8, 0, o,,
7:
A, + 0, Vo,
9:
La -- La Om. Ssi Do,, D.e)
D obj. B min La w.r. fe max La w.r.t m
8:
end for
10:
f, + fa vo Li
9:
end while
11:
8m Om Ve... Lid
10:
0, 0g. On.
12:
end for
13:
end while
14:
6", t 0m
gradient updating step in the inner-loop optimiza- tion represented as:
Task objective. Let Dtr denote the meta-training data in each training episode, and h Rd >> y denote the task classifier. We represent the task objective as the empirical risk the meta-training data with the cross-entropy loss (CE))..))
Inner-loop opt. for the 1th E [L] step,
(1)
0 ((- Ve, V,, (O 0m 0% 1), Dir):
(1-1) Vo, (0 1) 0n 0 (1-1), Dor)
(8)
C, Σ
((CR) (1)))()),)) v)
(6)
(D.
Each gradient updating step in the outer-loop optimization is represented as:
Bi-level optimization objective. Given the meta- training data Drr and meta-test data Die, we con- sider the following bi-level optimization objec- tive for learning optimized classification model h² Imm
Outer-loop opt. for the thh E [T] step,
(1) (((-1) + Vo.. (0(()) @ _00 Die. Do.):
(-1)) vV,,, L. L,, 0th") 00 1) Du,, Dic), (1) fa
(9)
0% arg max min Ca, 9r 0m: Oz Dte, ;
On
where v) represents the gradient updating rate and y represents the coefficient of gradient updating for the key-value memory network.
0 0% = arg min Le g: O pni Der), 0,
lnner-lopp objective
The full learning algorithm is a consequence of a meta- veeta-training procedure and a meta-test proce- dure, as shown in Algorithm 2 and Algorithm 3, respectively.
where the inner-loop optimization objective is the empirical task risk on the meta-training samples and the outer-loop optimization objective is the domain-invariant representation learning objective between the meta-target sample and meta-source samples.
Meta-training. For each training episode, lines 4-12 in Algorithm 2 present T' iterations of param- eter updating for the Transformer and key-value memory network. In particular, lines 6-8 present the inner-loop optimization by gradient updates on the parameters of Transformer and the task classi- fier. Then, lines 9-11 1 present the outer-loop opti- mization by gradient updates the parameters of key-value memory network O,. based on the up- dated Transformer parameters O',- As a result, the meta-training procedure preduces the optimized pa- rameters of key-value memory network 0* (linc 14).
3.4 ceta-Optimization Algorithm
We now design the full gradient-based algorithm to optimize the bi-level optimization objective in Eq. (7).
Gradient update. In the gradient-based opti- mization algorithm, the inner-loop optimization has L gradient updating steps and the outer-loop optimization has gradient updating steps. Each

---

Meta-test. Based on the learned parameters of key-value memory network On by the meta- training procedure, the meta-test procedure opti- mizes parameters of Transformer 0g and the task classifier A using all the source training data. In each iteration of lines 3-8 in Algorithm 3, the source training data are used to update the parame- ters of Transformer and task classifier by stochastic gradient descent (SGD). As result, the meta-test procedure produces the optimized Transformer 0 and task classifier f (line 10).
et al., 2019). We introduce key-value memory sublayer after the 12 layer of RoBERTaBASE Further details regarding the model architecture and hyperparameters can be found in Appendix B.
4.2 Main Results
The results for sentiment analysis and NLI using RoBERTaBASE are presented in Table and Table
2, respectively. Additionally, we include the re- sults of another pre-trained language model (PLM), BERT BASE, in Appendix C.1 to demonstrate the robustness of our approach.
After the micta-trining and meta-test procedures, the optimized model h" o Im can be used to make classification on the unseen target domain.
Before investigating the performance of our method, we first analyze the challenges of OOD set- ting on the used text classification datasets by mak- ing comparisons to the in-domain setting. Com- pared with the in-domain results (oracle), directly testing on OOD data (haseline) shows significant drop in performance. This indicates the difficulty of the used datasets for OOD evaluation.
4 Experiments
We evaluate the proposed method on sentiment analysis and natural language inference (NLI) tasks.
4.1 Experimental Setup
The last four rows in Table I and Table 2 pro- vide comparisons with four baselines. The nota- tion "+ memory" indicates that the baseline model was augmented with key-value memory, similar to our approach, but without the bi-level optimization for invariance learning. "invariance learning (w/o memory)" refers to a method similar to the works by Li et al. (2018b), Albuquerque which directly optimize domain invariance the feature space without memory augmentations. The results indicate that "+ memory" does not signifi- cantly improve over the baseline, suggesting that simply integrating memory layers into the base- line model is insufficient for learning transferable information to address domain shifts. Although Nomain-invarinn representation learning has been shown to be effective for out-of-distribttion (OOD) objective recognition (Li et al., 201 2018b), "invariance learning (w/o memory)" only exhibits marginal im- provements in our experiments. This suggests that traditional invariance learning methods face chal- lenges in addressing OOD text classification. In comparison these baselines, our method learns memory augmentations to improve domain invari- ance in the feature space and demonstrates signifi- cant enhancements in both sentiment analysis and NLI.
Datasets. For the sentiment analysis task, we use Amazon Reviews (Blitzer et al., 2007) for leave-one-domainoour evaluation. This dataset comprises two classes (positive and negative) and four domains: book (B), DVD (D), electronics (E) and kitchen (K). Additionally, we include IMDB (Thongtan and Phienthrakul, 2019) and SST- 2 (Socher et al., 2013) as test datasets for cross- dataset evaluation. For the NLI task, we employ scaled-down version of MNLI (Ben-David et al., evaluation. This
(2019),
2022)' for wwwee--on-domiiin. dataset consists of three classes (entailment, neu- tral, contradiction) and five domains: fiction (F), government (G), slate (S), telephone (T) and travel (T'). Moreover, we use SNLI (Bowman et al., 2015) and SICK (Marelli et al., 2) as test datasets for cross- datseett evaluation. Appendix A presents the statistics of the used datasets.
Evaluation. The evaluation methods include leave-one-domain-ont evaluation (Gulrajani and Lopez-Paz, 2020) and cross-dataset evaluation (Jia and Zhang, 2022b). Specifically, we employ stan- dard leave-one-domain-ee out evaluation on Amazon Reviews and MNLI, and cross cross-dataset evaluation on IMDB and SST-2 for sentiment analysis, well SNLI and SICK for NLI.
We compare our method with several state-of- the-art DG methods for text classification, most of which aim to achieve domain invariance across source domains. DEEP CORAL (Sun and Saenko,
Architecture and hyperparameters. all our experiments, we fine-tune RoBERTapan (Liu
htttps: httpss//gggthub...mmyyllllaalaa com//yyaldd///AA......

---

Method
Leuve-one-oomainoott Amazon Reviews
Cross-dataset Evaluation
Sources Tuget
DEK -B
BEK-N DD
BDK -E
BDE KK
Avg.
Amazon -IMDB
Amnazon +SST-2
supervised learning (oracle)
95.0
94.3
95.3
96.4
95.3
94.9
93.4
DEEP CORAL (Sun and Saenko, 2016)
91.9
91.3
90.9
93.5
91.9
89.8
87.6
IRM (Arjovsky al., 2019)
92.3
91.2
91.9
94.5
92.5
89.0
86.7
(Ben-Davvid
2022)
92.6
88.8
PDA (Jia and Zhang. 2022b)
92.9
92.2
93.3
94.8
93.3
92.1
91.3
M-SCL (Tan et al., 2022a)
92.3
91.2
93.7
93.4
92.7
RoBERTaas (baseline)
91.5
90.5
92.2
93.7
92.0
90.1
88.3
memory
92.0
91.5
91.8
93.2
92.1
90.5
88.7
invariance learning (w/ memory)
92.2
90.7
92.5
94.2
92.4
91.4
89.2
method
93.5
92.8
94.7
95.2
94.01
93.51
92.4
Table 1: Macro-F; sentiment analysis. The best and second best scores of each column marked in bold and underline, respectively. indicates statistical significance with < 0.05 by f-test compared to all baselines.
Method
Leave-one-domain-oue on MNLI
Cruss-dataee Evaluation
Sources + Target
GSTT'-FF
ISIT'-6
GFTT
GSPT
GSTF-
Avg.
MNLI-SNI SNLI
MNLI-SSICK
supervised learning (oracle)
83.2
88.3
81.8
82.7
86.4
84,5
88.5
90.3
DEEP CORAL (Sun and Sacako, 2016)
77.6
76.3
78.2
75.3
78.2
77.1
77.3
57.0
IRM (Arjovsky al., 2019)
78.1
75.2
79.4
76.2
79.2
77.6
76.2
58.7
PADA (Ben-David et al., 2022)
76.4
83.4
76.9
78.9
82.5
79.6
PDA (Jia and Zhung, 2022b)
80.8
85.8
79.7
79.4
83.0
81.7
79.3
62.0
RoBERTayAsE (baseline)
79.5
80.2
79.7
76.8
80.0
79.2
78.1
61.5
memory
79.6
80.7
79.2
77.0
81.2
79.5
78.6
60.8
invariance learning (w/ memory)
80.2
83.2
77.4
78.2
81.3
80.1
77.0
61.3
method
81.2
86.3
80.5
80.4
84.6
83.0
82.3'
65.7'
Table 2: Macro-F; on NLI. The best and second best scores of each column are marked in bold and underline. respectively. indicates statistical significance with p < 0.05 by f-test when compared to baselines.
2016) learns domain-invariant feature representa- tions by optimizing second-order statistics over feature states. IRM (Arjovsky al., 2019) further considers the intrinsic relationship between feature representation and labeling prediction to tackle do- main shifts. PDA (Jia and Zhang, 2022b) simulta- neously learns domain invariance for both feature representation and predicted probability. M-SCL (Tan et al., 2022a) employs supervised contrast learning method with memory augmentations to increase the contrasting examples.
Method
#Params
Amazon
MNLI
Avg.
RoBARTaBASE
108M
92.0
79.2
85.6
memory
113M
92.1
79.5
85.8
FFN
113M
91.0
77.5
84.3
self-attn + FFN
115M
91.5
78.6
our method
113M
94.0
83.0
88.51
Table 3: Macro-F, two datasets. indicates statistical significance with < 0.05 by l-test compared to all baselines.
to acquire transferable memory for domain gener- alization.
To ensure fair comparison, we reproduce M-SCL on sentiment analysis using RoBERTaBASE, while the results of the other methods are taken from literature that uses RoBERTaBASE. For leave-one- domain-out evaluation, our method outperforms all the compared methods by 0.7% Fi and 1.3% F1 on the Amazon Reviews and MNLI datasets, respectively. In terms of cross-dataset evaluation, our method achieves over 1.0% F, improvement on sentiment analysis datasets and approximately 3.0% Fj improvement on two NLI datasets com- pared to the other methods. These results demon- strate the superiority of employing meta-learning
4.3 Analysis
Effects Additional Parameters. Our method utilizes an additional key-value memory layer and includes approximately 4.8M more parameters compared to the RoBARTaBASE baseline model. To ensure fair comparison in terms of param- eter size, we consider three additional baselines: ii) "+ memory uses the same key-value mem- ory as our method but does not employ our in- variance learning technique; (ii) "+ FFN" adds a forward network (FFN) to the RoBARTaBASE

---

Preaanty souce comrdes
10-
(8) Transformer features.
(b) Memory.
(c) Memory- auggmentdddd features.
Figure 3: Feature representation the Amazon Reviews dataset. We visualize the representation of the target book domain and the source domains using Gaussian kernel density estimation and PCA dimensional reduction into 0., 1] x [0, 1]. We compare (a) Transformer features, (b) memory and memory memory- -ugmented features.
model; and (iii) "+ self-atttn + FFN" incorporates both a self-attention layer and an FFN on top of the RoBARTaBASE model. Although these three baselines have similar number of parameters method, they do not yield significant im- provements in performance. This observation in- dicates that merely increasing the parameter size with additional layers does not enhance out-of- distribution (OOD) text classification, thus demon- strating the effectiveness of our memory-based in- variance learning method.
10
baseline
memory
leam, (wot
A-disancc
02-
0.0
Figure 4: A-distance between the target domain and sources domains on three sentiment analysis datasets. The results are illustrated in Appendix C.2.
Visualization. We adopt t-SNE (Van der Maaten and Hinton, 2008) to visualize the feature represen- tations, as shown in Figure 3. From Figure 3 (a), we can observe that the Transformer features of the tar- get domain exhibit distinctly different distribution compared to those of the source domains. However, with the aid of memory augmentations, Figure (c) shows a smaller distance between the features of the target domain and those of the source domains. Interestingly, the memory distribution in Figure (b) reveals a strong domain specificity across dif- ferent domains. These findings demonstrate that method is capable of effectively learning mem- ory augmentations for different domains, thereby achieving domain invariance in the feature space.
the RoBARTaBASE model without employing the invariance learning strategy barely improves the A- distance. In contrast, the traditional invariant rep- resentation learning approach proves effectiveness in reducing the target-source domain A-distance. Furthermore, our method further optimizes the A- distance to much greater extent, which suggests that the memory learned by our method contributes to the invariance of feature representations.
Effects of memory learning. As demonstrated in Figure 5, the development results for both the Ama- zon Reviews and MNLI datasets show significant increase the memory size increases from 128 1,024. This observation indicates that larger mem- ory bank size encompasses richer features, allow- ing for accurate memory augmentations in generat- ing
Invariant representation learning. We adopt the A-distance (Ben-David al., 2006) to mea- sure the distance of feature distributions between the target domain and the source domains using three sentiment analysis datasets. As depicted in Figure 4, incorporating the key-value memory over
representations. However, the

---

Ethics Statement
sentiment
We agree with the statements of the ACL Code of Ethics and declare that this submission follows the submission policies of ACL.
macro F2 (sentimee smalyso
-4
83
marr-1 (MU)
93
N1.
Acknowledgments
92-
80
We thank the anonymous reviewers for their help- ful comments and suggestions. We gratefully ac- knowledge funding from the National Natural Sci- ence Foundation of China (NSFC No. 61976180) and the Zhejiang Province Key Project 2022SDX- HDX0003.
19
500
1000
1500
2000
memorry
Figure 5: Effects of memory size. Average results on the dev sets of Amazon Reviews and MNLI.
References
magnitude of performance improvement tends to diminish as the memory size continues to increase, especially when the memory size exceeds 1,024. our experiments, we choose memory size of 1,024 strike a balance between performance and model size. Additionally, we also analyze the ef- fects of memory optimization Appendix C.3.
Isabela Albuquerque, João Monteiro. Mohammad Darvishi, Tiago H Falk, and loannis Mitliagkas. 2019. Generalizing to unseen domains via distri- bution matching. arXiv preprint arXiv: 1911000041000
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv: 0009999990....
Nabiha Asghar, Lili Mou, Kira Selby, Kevin D Pan- tasdo, Pascal Poupart, and Xin Jiang. 2018. Progres- sive memory banks for incremental domain adapta- tion. arXiv preprint 1rXiv:1811.000
5
Conclusion
We have conducted an investigation into memory- based approach for domain generalization (DG). Our study involves the integration of a key-value memory network into the Transformer model, and the proposal of a meta-learning algorithm that in- corporates an episodic training strategy to effec- tively learn transferable memory for addressing domain shifts. The results obtained from experi- ments conducted on sentiment analysis and natural language inference tasks demonstrate the signifi- cant enhancement transfcrability of the source- domain model through the usage of the memory unit. Additionally, our approach achieves ot the-art performance on six different datasets.
Yogesh Balaji, Swami Samkaranarayanan, and Rama Chellappa. 2018. Metareg: Towards domain gen- eralization using meta neta-reguuarrizttonn Advances neural information processing systems, 31.
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. Pada: Example- based prompt learning for on-the-fi fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10:414- 433.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning. 79:151-175.
Shai Ben-David John Blitzer, Koby Crammer, and Fer- nando Pereira. 2006. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19.
Limitations
Our method only applies the BASE-level pre- trained language models, such ROBERTaBASE and BERTBASE- The recently developed large- scale pretrained language models, such as RoBERTaLARGE and GPT (Brown et al., 2020) have shown strong performances on classification and generatioin tasks. Due to resource limitations, we leave such large-model results in future work.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. 201 I. Generalizing from several related classification tasks to new unlabeled sample. Advances in Neural In- formation Processing Systems, 24:2178 2186.
John Blitzer, Mark Dredze, and Fernando Pereira, 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Asso- ciation of Computational Linguistics, pages 440-

---

Samuel R Bowman, Gabor Angeli, Christopher Potts. and Christopher D Manning. 2015, A large annotated corpus for learning natural language inference. arXiv preprint 150..05326.
Guillaume Lample. Alexandre Sablayrolles. Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2019. Large memory layers with product keys. Advances in Neural Information Processing Systems, 32.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. 2018a. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017. Deeper, broader and artier do- main generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Pan Li. Da Li, Wei Li, Shaogang Gong, Yanwei Fu. and Timothy M Hospedales. 2021. A simple feature augmentation for domain generalization. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 8886-8895.
Ning Ding. Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. Openprompt: An open-source framework for prompt- learning. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguisties: System Demonstrations, pages 105-113.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang. and Dacheng Tao. 2018b, Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the Euro- pean conference on computer vision (ECCV), pages 624-639.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2016. comain-adversario training of neural networks. The journal of machine learning research, 17(1)20096- 2030.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.11692
Ishaan Gulrajani and David Lopez-Paz. 2020. In search of lost domain generalization. arXiv preprint arXiv:2007.01434
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A sick cure for the evaluation composi- tional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'I4) pages 216- 223.
Han Guo. Ramakanth Pasunuru, and Mohit Bansal. 2020. Multi-source domain adaptation for text clas- sification via distancenet- bandits... In Proceedings of the AAAI conference on artificial intelligence, vol- ume 34, pages 7830-7838.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly read- ing documents. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pages 1409.
Chen Jia and Yue Zhang. 2022a. Meta-learning the invariant representation for domain generalization. Machine Learning pages 1-21.
Chen Jia and Yue Zhang. 2022b. Prompt-based distri- bution alignment for domain generalization in text classification. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 10147-10157.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. 2013. Domain generalization via invari- ant feature representation. In International Confer- ence on Machine Learning, pages 10-18.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00110.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiina. Association for Computa- tional Linguistics.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprini 1911,01971,000

---

Fengchun Qiao, Long Zhao, and Xi Peng. 2020. Learn- ing to learn single domain generalization. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12556-12565.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019. Manifold mixup: Better rep- resentations by interpolating hidden states. In In- lernational conference on machine learning, pages 6438 6447. PMLR.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta- learning with memory- augmente neural networks. In International conference on muchine learning, pages 1842-1850. PMLR.
Bailin Wang, Mirella Lapata, and Tvan Titov. 202 la. Meta-learning for domain generalization in seman- tic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 366-379.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning. Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 1631-1642.
Hongru Wang, Zezhong Wang, Gabriel Pui Cheong Fung, and Kam-Fai Wong. 202 Meml: contrastive meta-learning
A novel memory- method for few shot slot tagging. arXiv preprint arXiv:2108.1 1635..
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. Advances in neural information processing systems, 28.
Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhim- ing Luo. Yaojin Lin, Shaozi Li, and Nicu Sebe. 2021. Learning to generalize unseen domains via memory- multi-source meta-learning for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 6277-6286
Baochen Sun and Kate Saenko. 2016. Deep coral: Cor- relation alignment for deep domain adaptation. In European conference on computer vision. pages 443- 450. Springer.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a. Domain generalization for text classifica- tion with memory-based supervised contrastive learn- ing. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 6916- 6926. International Committee on Computational Linguistics.
Xin Zheng. Zhirui Zhang. Shujian Huang. Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. 2021. Non- parametric unsupervised domain adaptation for neu- ral machine translation. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pages 4234-4241.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022b. Domain generalization for text classifica- tion with memory-based supervised contrastive learn- ing. In Proceedings of the 29th International Con- ference on Computational Linguisties. pages 6916- 6926.
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. arXiv preprint arXiv:2205. 12674.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In Interna- tional Conference on Learning Representations.
Tan Thongtan and Tanasanee Phienthrakul. 2019. Sen- timent classification using document embeddings trained with cosine similarity. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics: Student Research Workshop, pages 407-414 Association for Computational Lin- guistics.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Wein- berger. 2017. Deep feature interpolation for image content changes. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pages 7064-7073.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages 5998-6008.

---

A Statistical Details of the Used Datasets
during prompting or [CLS] during traditional fine- tuning. To ensure balanced features, we selected coefficient 7y of 0.5 for our experiments. For each key-value memory network, the hidden size of keys was set to 256, and the default number of values was 1024. Following the methodology of Lample et al. (2019). we employed multi-head query-key attention mechanism with four heads. The total parameter count of the memory layers was approxi- mately 4.8M, significantly smaller compared to the 108M total parameters of RoBERTaBASE.
Dataset
Domain
Train (sre)
Dev (src)
Test (igt)
Seutiment Analysis
book:
166K
0.4K
0.4K
DVD
1.6K
0.4K
0.4K
Amazon
electronics (E)
1.6K
0.4K
0.4K
kitchen (K)
1.6K
0.4K
0.4K
IMDB
movie
25K
SST-2
movie
18K
NLA
Betioo (F)
2.5K
2.0K
2.0K
gevernment (G)
2.5K
1.9K
1.9K
MNLI
slate (S)
2.6K
2.0K
2.0K
C Additional Results
telephone (1)
2.8K
2.0K
2.0K
travel (T')
2.5K
2.0K
2.0K
C.1
Results based on BERTBASE
SNLI
general
9.8K
The results obtained using BERT BASE are consis- tent with the main findings presented in Table and Table 2, illustrated in Table 5 and Table 6. The baseline models, namely "+ memory" and "invariance learning (w/o memory)", either show minimal or no significant improvement compared to the baseline model. In contrast, our method demonstrates superior performance in both sen- timent analysis and NLI tasks, surpassing these bascline models. This indicates the robustness of approach across different pre teeitraiedd language models (PLMs).
SICK
imagedvvdeoo
0.5K
Table 4: Statistics the used datasets.
For the sentiment analysis task, we use Amazon Reviews (Blitzer al., 2007), which comprises two classes (positive and negative) and four domains: book (B), DVD (D), electronics (E) and kitchen (K). Additionally, we include IMDB (Thongtan and Phienthrakul, 2019) and SST-2 (Socher et al., 2013) as test datasets for cross-dataset evaluation. For the NLI task, we employ scaled-down version of MNLI (Ben-David et al., 2022), which consists of three classes (entailment, neutral, contradiction) and five domains: fiction (F), government (G), slate (S), telephone (T) and travel (T"). Moreover, use SNLI (Bowman al., 2015) and SICK (Marelli et al., 2014) as test datasets for cross-dataset eval- uation. Table 4 presents the statistics of the used datasets.
C.2 Invariant Representation Learning for NLI
baseline
memery
12-
zur method
Avisstncce
0.8
B Details on Architecture and Hyperparameters
0.4
We utilize ROBERTABASE (Liu al., 2019) as the primary Pretrained Language Models (PLMs) in our study, following the OpenPrompt framework (Ding et al., 2022), for two text classification tasks. The entire model was trained for up to 20 epochs, size of 32 sentences applied
0.0
SNU
Figure 6: A-distance between the target domain and sources domains on three NLI datasets.
with a across all datasets. Optimization was performed using AdamW with an initial learning rate set to 1e-5, weight decay rate of 0.01, and warm-up steps of 500. We incorporated key-value mem- ory layer after the 12-th layer of ROBERTABASE: This memory layer was added exclusively to the position used for classification, such as [MASK]
We adopt the A-distance (Ben-David et al., 2006) to measure the distance of feature distribu- tions between the target domain and the source domains on three NLI datasets. As depicted in Fig- ure 6, incorporating the key-value memory over the ROBARTABASE model without employing the invariance learning strategy barely improves the

---

Method
Leave-mee-oomain-out on Amazon Reviews
Cross-datseet Evaluation
Sources - Target
DEK-B B
BEK-D
BDK-E
BDE-K
Avg.
Amazon Amaoo-IMDDB
Amazon-SST-2
supervised leamingg (oracle)
92.6
92.4
91.1
93.7
92.5
92.2
90.7
DEEP CORAL (Sun and Saenko, 2016)
88.2
87.8
88.2
89.7
88.5
85.8
85.7
IRM (Arjovsky al.. 2019)
85.8
89.4
87.6
90.5
88.3
84.8
84.2
PDA (Jia and Zhung. 2022bb
90.2
89.7
90.8
91.6
90.6
88.5
86.8
BERTBASE (baseline)
89.1
88.7
88.2
90.8
89.2
86.0
85.
memory
90.0
88.2
88.7
89.6
89.1
85.8
86.0
invariance learning (w/o memory) our method
89.8
88.7
91.0
90.6
90.0
88.6
86.2
90.6
90.7
91.5
92.8
91.4
89.27
87.87
Table Macro-Fj on sentiment analysis based BERT BASE. The best second best scores each column are marked in bold and underline, respectively. 'indicates statistical significance with p < 0.05 by t-test compared all baselines.
Leave-mee-oomain-out
Cross-dataset Evaluation
Sources + Target
GSTT =F
ESTTT-16
GFTT" -8
GSFT -T
GSTF -T
Avg.
MNLI-SIN MNLI-SSIC
supervised learning (oracle)
80.4
82.6
76.3
78.5
81.7
80.6
83.0
89.6
DEEP CORAL (Sun and Saenko, 2016)
75.7
74.6
73.1
74.0
76.3
74.7
65.5
56.3
(Arjovsky et al., 2019)
71.8
73.7
75.1
PDA (Jia Zhang, 2022h)
BERTBASE BASE (baseline)
72.5
72.9
74,7
73.5
memory
73.6
72.2
72.0
73.5
invariance learning (wfo memoryy
76.8
75.4
72.7
74.8
78.2
75.6
66.7
61.0
method
77.0
78.2
73.6
76.2
78.6
76.71
70.51
61.21
Table 6: Macro-Fj on NLI based on BERT BASE- The best and second best scores of each column are marked in bold and underline, respectively. indicates statistical significance with p < 0.05 by f-test when compared to all baselines.
A-distance. In contrast, the traditional invariant representation learning approach proves effective in reducing the target-source domain A-distance. Furthermore, our method further optimizes the A- distance to much greater extent, which suggests that the memory learned by method contributes the invariance of feature representations.
the lcarning rate for memory values ranges from 0 to le When the learning rate is set to 0, the key-value memory network remains untrained, thus failing to produce appropriate memory augmenta- tions. As a consequence, the results noticeably lower than those of the baseline model without memory augmentations. As the learning rate gradu- ally increases, the results improve with minor fluc- tuations, ultimately reaching plateau when the learning rate reaches a sufficiently high value. This indicates that optimizing the key-value memory network facilitates the performance of OOD text classification.
C.3 Effects of Memory Optimization
sentiment analysis
94
B4
macro F1 trentiment snalpts!
02
marros ONLO
50
0.0000
0,0002
0.0004
0000
0.0008
0.0010
memory
Figure 7: Effects of memory optimization. Average results on the dev sets of Amazon Reviews and MNLI.
Figure 7 presents the results obtained from the Amazon Reviews dev set and the MNLI dev set, as


Reviews:
[{"review_id": "IE7M7gdQvE", "reviewer": "Reviewer_goM8", "summary": "", "questions": "- There is a potential missing baseline: [1] Guo, H., Pasunuru, R., & Bansal, M. (2020, April). Multi-source domain adaptation for text classification via distancenet-bandits. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 05, pp. 7830-7838). Have the authors considered the relevancy of this paper?\n\n- Have the authors tried their method with more challenging datasets/scenarios when the target domain is distant from the source domains? ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer goM8,\n\nThank you for taking the time to read our paper and giving constructive comments. We hope our point-to-point responses can address your concerns.\n\n> **Weakness 1**: There is a potential missing baseline: [1] Guo, H., Pasunuru, R., & Bansal, M. (2020, April). Multi-source domain adaptation for text classification via distancenet-bandits. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 05, pp. 7830-7838  \n**Question 1**: Have the authors considered the relevancy of this paper [1]?\n\n**Response:** Thanks for your valuable comments! The work by Guo et al. [1] proposes a DistanceNet model for multi-domain adaptation, which uses a mixture of five domain distance measures, including $\\mathcal{L}_2$, cosine, MMD, Fisher and CORAL, to minimize the domain distance between the target domain and source domains.  They further employ a multi-armed bandit controller to dynamically select the sequence of source domains to deliver the best outcome on the target domain task. \n\nSince in the OOD setting considered by our work, the target domain is unseen during training, it is impossible to optimize a multi-armed bandit to select the sequence of source domains in the training phase. Thus, **we take the idea of DistanceNet [1] to use a mixture of domain distance measures to learn the invariant feature representations across source domains**. This revised baseline is named **DistanceNet-S**, which is different from the original DistanceNet model [1] in that **we use the distance measures across source domains** instead of the distance measures between the target and source domains. **Loss function** of  the **DistanceNet-S** baseline can be represented as: \n\n$\\mathcal{L} = \\sum_k \\alpha_k \\sum_{i,j} D_k(S_i, S_j) + \\ell_{task}$, where $D_k(\\cdot, \\cdot)$ represents different distance measures from {$\\mathcal{L}_2$, cosine, MMD, Fisher, CORAL} and the weighting coefficients $\\alpha_k$ are tuned by the source-domain dev sets. \n\nWe *reproduce* the baseline **DistanceNet-S** [1] and compare it with our method on six datasets. The results that **DistanceNet-S** [1] optimizes a mixture of distances to obtain invariant representations across source domains and **achieves better results on 5 of 6 datasets**. **Our method** uses a **meta-learning strategy** with a novel **memory-based invariance learning method** to learn the **invariant feature representation between the target domain and source domains**  and achieves the best results on six datasets.\n\n| Method | Amazon | IMDB | SST-2 | MNLI | SNLI | SICK |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| RoBERTa (*baseline*) | 92.0 | 90.1 | 88.3 | 79.2 | 78.1 | 61.5 |\n| DistanceNet-S [1] | 92.8 | 92.0 | 90.7 | 78.5 | 80.2 | 62.8 |\n|**our method**| **94.0** | **93.5** | **92.4** | **83.0** | **82.3** | **65.7** |\n\n[1] Multi-source domain adaptation for text classification via distancenet-bandits. Guo et al, AAAI 2020.\n\n> **Weakness 2**: The discussion on how the two datasets mentioned in the paper are challenging for out-of-domain classification should be provided. This will better highlight the contribution of the proposed framework.  \n**Question 2**: Have the authors tried their method with more challenging datasets/scenarios when the target domain is distant from the source domains?\n\n**Response:** Thanks for your valuable comments! We analyze the challenges of the used datasets for out-of-domain (OOD) text classification by **cross-domain evaluation**. We investigate the performance of a baseline text classification model (RoBERTa-base)  trained on the dataset when tested on out-of-domain examples. As listed in the following table, **a significant drop in performance when tested on out-of-domain data compared to in-domain data indicates the dataset's difficulty for generalization**.\n\n| Testing Scenario | Amazon | IMDB | SST-2 | MNLI | SNLI | SICK |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| In-domain | 95.3 | 94.9 | 93.4 | 84.5 | 88.5 | 90.3 |\n| Out-of-domain | 92.0 | 90.1 | 88.3 | 79.2 | 78.1 | 61.5 |\n| Drop | -3.3 | -4.8 | -5.1 | -5.3 | -10.4 | -28.8 |\n\nAs shown in the above table, the cross-dataset scenario (testing on IMDB, SST-2, SNLI, and SICK) is challenging for OOD evaluation. In addition, we further evaluate our method on a **rumour detection dataset PHEME** [2], which consists of five domains: Charlie-Hebdo (C), Ferguson (FR), Germanwings-crash (GW), Ottawa-shooting (OS), and Sydney-siege (S). The OOD results (measured by macro-F1) by the leave-one-domain-out evaluation are listed in the below table. **The results show that the domain distance of PHEME [2] poses a significant challenge in OOD classification (75.9 for ID v.s. 61.1 (-14.8) for OOD)**. **The results also show the effectiveness of our method for OOD classification in rumour detection**. \n\n|Method | C| FR | GW | OS | S | Avg.| \n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| In-domain (*oracle*) | *78.2* | *60.3* | *82.8* | *80.7* | *77.6* | *75.9* | \n| RoBERTa (*baseline*) | 66.4 | 43.7 | 68.7 | 64.5 | 62.1 | 61.1 |\n| +memory | 65.7| 42.6 | 71.6 | 65.7 | 63.2 | 61.8 |\n| inv. learn. (w/o memory) | 64.6| 43.2 | 71.0 | 66.8 | 64.2 | 62.0 |\n| **our method** | **68.2** | **47.6** | **76.8** | **67.3** | **68.2** | **65.6** |\n\n\n[2] Exploiting context for rumour detection in social media. Zubiaga et al., 2017\n\n**Thanks again for your effort in giving detailed and constructive comments. If you have more questions, feel free to discuss them with us. We are looking forward to your further messages in the discussion period.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written with clear motivation and description of the methodology, experimental results and analyses. \n- The underlying intuition for the design of the architecture and the learning process is well justified and easy to understand.\n- The experiments are comprehensive and support the claims of the paper.    ", "reasons_to_reject": "- There is a potential missing baseline: [1] Guo, H., Pasunuru, R., & Bansal, M. (2020, April). Multi-source domain adaptation for text classification via distancenet-bandits. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 05, pp. 7830-7838).\n- The discussion on how the two datasets mentioned in the paper are challenging for out-of-domain classification should be provided. This will better highlight the contribution the the proposed framework.  ", "questions_to_authors": "- There is a potential missing baseline: [1] Guo, H., Pasunuru, R., & Bansal, M. (2020, April). Multi-source domain adaptation for text classification via distancenet-bandits. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 05, pp. 7830-7838). Have the authors considered the relevancy of this paper?\n\n- Have the authors tried their method with more challenging datasets/scenarios when the target domain is distant from the source domains? ", "paper_topic_and_main_contributions": "The paper investigates the problem of out-of-domain classification with the aim of extending a classifier, trained on multiple source domains, to a new target domain. The main challenge comes from the disparity between source domains and the target domain. The paper proposes to address this challenge via the use of memory-augmentation to enhance the quality of the domain-invariant embedding space. Experiments on sentiment analysis and natural language inference tasks with ROBERTA_base show promising improvements of the proposed method over the baselines.  \n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U4MS22g2TX", "reviewer": "Reviewer_VVu8", "summary": "", "questions": "Question A: What are the keys in the key-value memory network (L207)?\n\nQuestion B: What is the memory size in Figure 5? Is it the number of keys?\n\nQuestion C: How can we reach the claim in L476-479 from Figure 3?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer VVu8,\n\nThank you for taking the time to read our paper and giving constructive comments. We hope your concerns can be addressed by our point-to-point responses.\n\n> **Weakness 1:** This paper fails to explain the motivation or provide the sufficient rationale for their method designs in many places. \n\nWe make responses to each concept of this concern as follows.\n\n> **Weakness 1-1:** Why the key-value memory is used and why meta-learning is adopted to learn the invariant representation instead of other approaches? \n\n**Response:** Thanks for your valuable comments. Here, we explain the motivation in **two folds**. \n\nFirstly, we use the **K-V memory** as **feature augmentation** to improve the **invariant feature representation** across domains. Compared with the traditional methods that learn a shared feature space across domains [1], we use memory augmentations to **alleviate the discrepancy of feature distributions** between source and target domains and **improve the invariant feature distribution**, as shown in Fig 1.\n\nSecondly, to enhance memory-based invariance learning, we employ a **meta-learning strategy**. **Meta-learning is chosen over alternative invariance learning methods such as [2, 3] due to the inherent challenge of domain generalization, where the target domain remains unseen during training**. Consequently, it becomes impractical to explicitly optimize invariant features between the source and target domains, as traditionally done in invariance optimization methods [2, 3]. Therefore, we adopt a meta-learning approach within an episodic training process. We now elaborate on the meta-learning procedure. \n\nIn each episode of meta-training, we randomly partition the source domains into meta-source and meta-target samples, simulating the domain shifts encountered in the testing phase. Our proposed bi-level optimization method addresses the discrepancy between the meta-source and meta-target samples (outer-loop) while optimizing the classification task using the meta-source samples (inner-loop). Subsequent to the meta-training phase, we proceed to a meta-test phase, where we optimize a classification task using the source samples. Through the utilization of the meta-learning strategy, we attain domain invariance between the test domain and the source domains.\n\n[1] Deeper, broader and artier domain generalization. Li et al., 2017.\n\n[2]  A theory of learning from different domains. Ben-David et al., 2010.\n\n[3] Generalizing to unseen domains via distribution matching. Albuquerque et al., 2019.\n\n> **Weakness 1-2**: Why minimax the domain classification objective can learn the invariant representation (i.e., how domain discriminators help)? \n\n**Response:**  We follow the previous work [2] (Thm. 1) to minimize the **$\\mathcal{H}$-divergence** for obtaining the invariant representation. The $\\mathcal{H}$-divergence between the meta-source domain $D_{tr}$ and meta-target domain ${D}_{te}$ can be represented as:\n\n$d_{\\mathcal{H}}(D_{tr}, D_{te}) =  \\sup_{f_d} \\vert  Pr_{D_{tr}}(I(f_d)) -   Pr_{D_{te}}(I(f_d))  \\vert$,\nwhere $I(f_d)$ represents a domain indicator such that $x \\in I(f_d) \\iff f_d(x) = 1$.\n\nThen, using Lemma 2 in [2], we obtain that\n\n$d_{\\mathcal{H}} (D_{tr}, D_{te}) = 1 - 2\\min_{f_d} L_d (f_d) $, where $L_d (f_d)$ (Eq. (4)) represents the empirical error for a domain discriminator $f_d$ to distinguish samples from $D_{tr}$ and $D_{te}$, i.e., $f_d (x) = 0$ if $x \\in D_{tr}$ and $f_d (x) = 1$ if $x \\in D_{te}$.\n\nThus, the minimax objective $\\max_{m}\\min_{f_d} L_d$ $\\iff$ $\\min_{m} d_{\\mathcal{H}}(g_m(D_{tr}), g_m(D_{te}))$. This indicates that **minimax the domain classification objective is equivalent to minimizing the $\\mathcal{H}$-divergence** between $D_{tr}$ and $D_{te}$ w.r.t. the memory augmentation $m$, which can be viewed as learning the invariant representation across domains in the previous work [2,3].\n\n> **Weakness 1-3**: In the experimental section, why the tasks of sentiment analysis and natural language inference are chosen to study domain generalization; is it to follow prior works or because these tasks have diverse domains? \n\n**Response:** Thanks for the comment! Indeed, we follow the previous work on OOD text classification [4-6] to evaluate the sentiment analysis and natural language inference tasks. \n\n[4] Prompt-based distribution alignment for domain generalization in text classification. Jia et al., 2022.\n\n[5] Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Ben-David et al., 2022.\n\n[6] Domain generalization for text classification with memory-based supervised contrastive learning. Tan et al., 2022. \n\n> **Weakness 2**: This paper is not very well-written and organized. \n\nWe make responses to each concept of this concern as follows.\n\n> **Weakness 2-1:**  Section 3 lacks clarity, and it contains typos (e.g., \"f_d\" instead of \"f_g\" in equation (5)) and confusing notation that seems to influence the understanding of the algorithms.\n\n**Response:** Thanks for your careful check! We have carefully checked the typos and revised it as follows:\n\n$\\max_{\\theta_m} \\min_{\\theta_{f_d}} L_d (\\theta_{g}, \\theta_{m}, \\theta_{f_d}\\ ; D_{te}, D_{tr}) $  (Eq. (5))\n\n> **Weakness 2-2**: Overloading a section with excessive content can lead to confusion and make it hard for readers to follow the logic flow of ideas. It would be better if each section can address a distinct aspect of the work, e.g., a separate section of task formulation from the current Section 3.\n\n**Response:** Thanks for your suggestion! We will take it for the revision. In the current version, we organize Section 3 in a way that starts with a general overview and then delves into specifics. In particular, **Sec. 3.1** gives an overview of the **model architecture**, **Sec. 3.2** gives an **overview of the learning/optimization strategy**, and then **Sec 3.3** explains in detail the **training objectives**, and finally **Sec 3.4** gives the **full procedure of the algorithm**.\n\n> **Weakness 2-3**:  It is confusing to mention \"meta-test\" in Section 3.4: is it a different concept from the inner-loop in Section 3.3?\n\n**Response:** Following the previous work on DG based on meta-learning [7,8], **meta-test is an additional training process after the meta-training**. In particular, the meta-training process optimizes the bilevel optimization objective on the splits of meta-source and meta-target samples and the **meta-test process optimizes the inner-loop objective on the source samples**.\n\n[7] Learning to Generalize: Meta-Learning for Domain Generalization. Li et al., 2017.\n\n[8] Feature-Critic Networks for Heterogeneous Domain Generalization. Li et al. 2019.\n\n> **Weakness 3:** The illustration of paper can be more informative. For example, figure 2 can explain the component (d) is only used during training to help the memory network to learn the invariant representation; the current version may make the readers misunderstand it will also work in inference.\n\n**Response:** Thanks for your suggestion! we will improve Fig. 2 by adding more information about the difference across meta-training/meta-test/inference phases.\n\n> **Weakness 4:** Both meta-learning and memory networks have been used before in addressing the out-of-domain text classification. What is the most motivating thing (i.e., novelty) about this work? It would be better if the authors can explain why the approach is a significant contribution over previous methods, and what the implication is for future work.\n\n**Response:** Thanks for your constructive comment and suggestion! We highlight the **contribution/novelty** of this work compared with the previous work on meta-learning and memory networks as follows:\n\n- Previous works on **OOD classification based on meta-learning** mostly use the MAML framework [9], where the meta-learner aims to optimize the **meta-target classification loss** by learning a **parameter initialization** [10,11]. In contrast, the meta-learner in our method aims to learn a **domain invariant feature space** by learning a **memory augmentation**. **The novelty of our work is reflected in the design of meta-learning objectives (e.g., Eq. (7), Alg. 1, etc.)**. **The advantage of our design lies in the ability to leverage an additional memory network to learn more robust feature representations across domains**.\n\n- Previous research on **memory networks** primarily focuses on utilizing test information to adapt the model (refer to Section 2 in our paper). However, there have been limited studies on **memory-based out-of-distribution (OOD) text classification**. To the best of our knowledge, only one existing study [12] leverages the memories of source-domain samples to augment contrasting features for computing supervised contrastive loss. Our work differs significantly from this study [12]. Firstly, **our memory network is trainable**, whereas they employ static source-domain banks that are not optimized during training. Secondly, **we explicitly utilize memory as feature augmentation to enhance invariant representation learning**, whereas they employ memory as contrasting features for computing the contrastive loss.\n\n[9] Model-agnostic meta-learning for fast adaptation of deep networks. Finn et al., 2017.\n\n[10] Learning to generalize: Meta-learning for domain generalization. Li et al., 2018.\n\n[11] Learning transferable features in meta-learning for few-shot text classification. Xu et al., 2020.\n\n[12] Domain Generalization for Text Classification with Memory-Based Supervised Contrastive Learning. Tan et al., 2022.\n\n> **Question A:** What are the keys in the key-value memory network (L207)?\n\n**Response:** Following the previous work on key-value memory networks [13, 14], the keys $\\mathcal{K}$ are used to **compute the weighting scores for memory values**, represented as $\\boldsymbol{\\alpha} = \\operatorname{softmax}(\\mathbf{q}^\\top \\mathcal{K})$, where $\\mathbf{q}$ represents the query vector corresponding to the current hidden state. This is equivalent to Eq. (1) in our paper. In addition, the keys are randomly initialized and optimized to select the memories for invariance learning during the training phase.\n\n[13] Key-value memory networks for directly reading documents. Miller, A., et al., 2016\n\n[14] Large memory layers with product keys. Lample et al., 2019.\n\n> **Question B:** What is the memory size in Figure 5? Is it the number of keys?\n\n**Response:**  The memory size in Fig. 5 denotes the **number of memory values** $\\vert \\mathcal{V} \\vert$ (L212) in the K-V memory network. \n\n**Yes**, it is equal to the number of keys, which is used to ensure the computation of memory in Eq. (2).\n\n> **Question C:** How can we reach the claim in L476-479 from Figure 3?\n\n**Response:** We explain in detail the Fig. 3 as follows. **Fig. 3 (a)** illustrates the feature space of **baseline model w/o memory augmentation**. **Fig. 3 (c)** illustrates the feature space **with the memory augmentation**. We can clearly observe that **Fig. 3 (c)** shows stronger feature matching between the target domain and source domains compared with **Fig. 3 (a)**. Furthermore, we can quantitatively calculate the $\\mathcal{A}$-distance [2] between the target domain and source domains, and obtain that **0.527 in  Fig. 3 (a) v.s. 0.093 in  Fig. 3 (c)**. This shows that our method can effectively learn memory augmentations to achieve domain invariance in the feature space. \n\n**Thanks again for your effort in giving detailed and constructive comments. If you have more questions, feel free to discuss them with us. We are looking forward to your further messages in the discussion period.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results of the proposed method are very good, and the ablation experiments show the effectiveness of memory network and invariant representation learning.", "reasons_to_reject": "1) This paper fails to explain the motivation or provide the sufficient rationale for their method designs in many places. For example, why the key-value memory is used and why meta-learning is adopted to learn the invariant representation instead of other approaches? Why minimax the domain classification objective can learn the invariant representation (i.e., how domain discriminators help)? In the experimental section, why the tasks of sentiment analysis and natural language inference are chosen to study domain generalization; is it to follow prior works or because these tasks have diverse domains? The lack of motivation may raise doubts about the necessity, effectiveness (and novelty) of these design choices. Explaining why doing this is better than simply listing what is done.\n\n2) This paper is not very well-written and organized. For example, Section 3 lacks clarity, and it contains typos (e.g., \"f_d\" instead of \"f_g\" in equation (5)) and confusing notation that  seems to influence the understanding of the algorithms. Moreover, overloading a section with excessive content can lead to confusion and make it hard for readers to follow the logic flow of ideas. It would be better if each section can address a distinct aspect of the work, e.g., a separate section of task formulation from the current Section 3. Last but not least, it is confusing to mention \"meta-test\" in Section 3.4: is it a different concept from the inner-loop in Section 3.3? \n\n3) The illustration of paper can be more informative. For example, figure 2 can explain the component (d) is only used during training to help the memory network to learn the invariant representation; the current version may make the readers misunderstand it will also work in inference.\n\n4) Both meta-learning and memory networks have been used before in addressing the out-of-domain text classification. What is the most motivating thing (i.e., novelty) about this work? It would be better if the authors can explain why the approach is a significant contribution over previous methods, and what the implication is for future work.", "questions_to_authors": "Question A: What are the keys in the key-value memory network (L207)?\n\nQuestion B: What is the memory size in Figure 5? Is it the number of keys?\n\nQuestion C: How can we reach the claim in L476-479 from Figure 3?", "paper_topic_and_main_contributions": "This paper studies the problem of out-of-domain (OOD) text classification and focuses on domain generalization (DG). One main challenge for DG is the distribution disparity across different domains, which can make it difficult to learn the effective invariant representation. Hence the main contribution of this paper is to augment the transformer with a key-value memory network to meta-learn the domain-invariant representation. The authors evaluate their method on sentiment analysis and natural language inference tasks which exhibit considerable domain variations, and the experiments show the effectiveness of their method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to \"Reasons To Reject\" for details."}, "extra_text": {}}, {"review_id": "LfwrpwBwJW", "reviewer": "Reviewer_545j", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Dear Reviewer 545j,\n\nWe thank your efforts in giving detailed and constructive comments and appreciate your support of our work. We hope our response can successfully address your concerns.\n\n> **Weakness 1**: The underlying base encoder used in the experiments of the paper is not specified. It is necessary to conduct experiments to discuss whether different base models are sensitive to different out-of-domain methods.\n\n**Response:** Thanks for your suggestion. To assess the sensitivity of OOD methods to different basic encoders, we further evaluate their performance using alternative encoders such as **BiLSTM** and **CNN** in addition to RoBERTa and BERT (refer to Table 1&2 and Table 5&6, respectively). The hyperparameters for text classification using BiLSTM and CNN are adopted from [1] and [2], respectively. **Notably, our findings exhibit similar trends to the results obtained with RoBERTa and BERT, providing further evidence of the effectiveness of our approach. Additionally, our method attains superior results on all six datasets, demonstrating its efficacy in OOD text classification across various base encoding models, including LSTM and CNN-based architectures.**\n\n| Method | Amazon | IMDB | SST-2 | MNLI | SNLI | SICK |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |---------- |\n|BiLSTM (*baseline*) | 80.1 | 75.6 | 74.8 | 65.1 | 64.8 | 52.6 |\n| + memory | 79.6 | 75.2 | 75.3 | 65.5 | 63.7 | 53.9 |\n| inv. learn. (w/o memory)  | 81.4 | 76.7| 77.2 | 66.2| 65.5 | 51.8 |\n| **our method**  | **82.2** | **77.4** | **77.5** | **67.1** | **66.7** | **54.3** |\n \n| Method | Amazon | IMDB | SST-2 | MNLI | SNLI | SICK |\n| ---------- | ---------- | ---------- | ---------- | ---------- |---------- | ---------- |\n|CNN (*baseline*) | 78.2 | 74.0 | 70.3 | 64.2 | 58.7 | 54.2 |\n| + memory | 76.7 | 74.8 | 70.8 | 65.2 | 60.2 | 52.8 |\n| inv. learn. (w/o memory)  | 79.2 | 75.3 | 72.3 | 64.7 | 61.7 | 54.7  |\n| **our method**  | **80.1** | **75.8** | **73.5** | **65.8** | **62.5** |  **55.4**  |\n\n[1] Adversarial Multi-task Learning for Text Classification. Liu et al., 2017\n\n[2] Understanding Convolutional Neural Networks for Text Classification. Jacovi et al., 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Memory-Based Invariant Representation Learning: The paper introduces a novel memory-based approach for domain-invariant representation learning. The proposed method constructs domain-invariant representations across multiple source domains.\n\n2) Improved Out-of-Domain Text Classification: The memory-based approach significantly enhances the accuracy of out-of-domain text classification tasks, specifically sentiment analysis and natural language inference. The method achieves state-of-the-art results on six datasets, surpassing several strong baselines.\n\n3) Comparison and Analysis: The paper compares the proposed memory-based method with traditional learning strategies and analyzes the effects of memory size and memory optimization. The results show that the memory-based approach provides better augmentations to the feature representations and contributes to the invariance of feature representations.", "reasons_to_reject": "The underlying base encoder used in the experiments of the paper is not specified. It is necessary to conduct experiments to discuss whether different base models are sensitive to different out-of-domain methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper titled \"Memory-Based Invariance Learning for Out-of-Domain Text Classification\" is about a novel approach to improve out-of-domain text classification. \n\nThe paper proposes a memory-based invariant representation learning method that utilizes a memory network. This memory network helps in constructing domain-invariant representations. \n\nThe method is evaluated on sentiment analysis and natural language inference tasks and achieves state-of-the-art results on six datasets. \nThe paper also compares the method to several baselines and analyzes the effects of memory size and memory optimization. \n\nOverall, the paper presents a novel approach to domain-invariant representation learning that significantly improves the accuracy of out-of-domain text classification tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a memory augmentation technique to address OOD text classification. The memory is used during meta-learning to enhance the invariance of the representations. Experiments on semantic analysis and NLI show promising gain, and SotA result. The writing as a matter of presentational issue needs to be improved (fully agree with VVu8). The discussion with goM8 justified the reason behind the task selection adequately. The combination of memory and meta-learning is certainly well-justified (specially after reading the discussions).  An interesting work!
Decision: Accept-Main