SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhijing Wu $^{1,2}$ , Hua Xu $^{1,2*}$ , Jingliang Fang $^{1,3}$  and Kai Gao $^{3}$
$^{1}$ State Key Laboratory of Intelligent Technology and Systems,
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
$^{2}$ Beijing National Research Center for Information Science and Technology(BNRist)
$^{3}$ School of Information Science and Engineering, Hebei University of Science and Technology
wuzj18@mails.tsinghua.edu.cn
xuhua@tsinghua.edu.cn
Continual Machine Reading Comprehension aims to incrementally learn from a continuous data stream across time without access the previous seen data, which is crucial for the development of real-world MRC systems. However, it is a great challenge to learn a new domain incrementally without catastrophically forgetting previous knowledge. In this paper, MA-MRC, a continual MRC model with uncertainty-aware fixed Memory and Adversarial domain adaptation, is proposed. In MA-MRC, a fixed size memory stores a small number of samples in previous domain data along with an uncertainty-aware updating strategy when new domain data arrives. For incremental learning, MA-MRC not only keeps a stable understanding by learning both memory and new domain data, but also makes full use of the domain adaptation relationship between them by adversarial learning strategy. The experimental results show that MA-MRC is superior to strong baselines and has a substantial incremental learning ability without catastrophically forgetting under two different continual MRC settings.
Recently, Machine Reading Comprehension (MRC) has attracted wide attention and achieved remarkable success when solving specific tasks in stationary environments, such as answering factual questions with wikipedia articles or answering narrative questions with web search logs (Seo et al., 2017; Seonwoo et al., 2020; Zhang et al., 2021; Wu and Xu, 2020). However, the answering scenario changes over time in real-world applications. For example, the dialog system should continuously adapt to new user requirements (Abujabal et al., 2018; Madotto et al., 2021). In this paper, we focus on one of the most typical scenario changes for MRC tasks: the domain data shift. Existing
Figure 1: Illustration of a continual MRC task.
stationary-trained MRC systems are usually trained with in-domain data but are applied to new domain data (Fisch et al., 2019). Therefore, it is necessary to build a non-stationary MRC that continually learns with incremental domain data.
We formulate such a challenging problem as Continual MRC task, which is required to incrementally learn over sequential domain data and perform well on all seen domain data. Figure 1 illustrates the incremental learning and testing processing. In this scenario, the MRC system can only trained on the latest coming domain data without access the previous seen data. To tackle this issue, if we directly fine-tune the MRC model on each new incoming domain, the performance on earlier domains will significantly drop (Su et al., 2020). Another naive approach is to retrain the whole MRC model from scratch, but it is costly and time-consuming. Hence it is a great challenge for incrementally learning without largely forgetting previously acquired knowledge.
Existing studies for continual MRC can mainly be divided into three categories. The first class is model expansion techniques that design domain-individual classifier for each in coming domain (Su et al., 2020). However, it is expensive and unpractical in real-world. The second class borrows the regularization idea, which utilizes an ad
* Hua Xu is the corresponding author.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2330 - 2339
ditional loss term to aid knowledge consolidation when learning new domains. For example, Su et al. (2020) added a penalty that restricts the change of important parameters to prevent forgetting previous knowledge. The third class is episodic memory based methods. For example, de Masson d'Autume et al. (2019) introduced a key-value memory module that stored previously seen examples for sparse experience replay and gradient-based local adaptation. Abujabal et al. (2018) proposed template-based Never-Ending KB-QA that learned new templates by capturing new syntactic structures with a semantic similarity function between questions and user feedback.
However, the above methods still have limitations, mainly including two aspects. On the one hand, to prevent catastrophic forgetting, these methods only design consistent constraints of model output or gradient for previous and new domains, while ignoring the domain adaptation relationship between them. However, transfer learning can help the MRC model generalize to other domains. On the other hand, the memory update strategy for continual MRC is limited. Some previous work stores fixed examples for each incoming domain. It greatly grows the number of samples kept in memory and leads to expensive costs. Other methods that limit the maximum number of memory for all seen domains usually update the memory by random sampling, ignoring the forgotten degree of different samples. In fact, the continual model should pay more attention to samples that are more likely to be forgotten.
To handle the above limitations, this paper proposes MA-MRC, an incremental model that solves continual MRC task via Uncertainty-aware fixed Memory and Adversarial Domain Adaptation. Concretely, MA-MRC 1) introduces a fixed-size memory to store a small number of samples in previous domain, which are later periodically replayed when learning new domain; and 2) updates the memory with an uncertainty-aware strategy that takes the forgotten degree of previous data into account; 3) leverages an adversarial learning strategy to make full use of the domain adaptation relationship between different domains with a domain discriminator, so as to help generalize and avoid overfitting very small memorized examples. The intuition behind this is to mimic the human learning process that replays the memory and adapts to new domains.
The key contributions of this work are: (1) This paper proposes a continual MRC model, MA-MRC, which learns new domain data incrementally. (2) Applying uncertainty-aware Memory and Adversarial learning and to MRC model contributes to strong incremental learning ability. (3) The experimental results on two different continual MRC settings indicate that MA-MRC obtains good incremental learning ability without largely forgetting and significantly outperforms strong baselines.
2.1 Continual Learning
Continual Learning (CL) mainly aims to overcome the catastrophic forgetting problem when learning on sequential new tasks incrementally (French, 1999). Existing work follows three directions: architectural, regularization, and memory-based approaches. The architectural methods change the network's architecture and add task-specific parameters, e.g., Dynamically Expandable Network (Yoon et al., 2018) and Reinforced Continual Learning (Xu and Zhu, 2018). The regularization-based techniques aid knowledge consolidation when learning new tasks. For instance, EWC and Online EWC (Kirkpatrick et al., 2017; Schwarz et al., 2018) slow down the learning of parameters important for previous tasks. The third class is to save previous samples and learn a new task with a forgetting loss defined on them, such as GEM (Lopez-Paz and Ranzato, 2017), A-GEM (Chaudhry et al., 2019), DER (Buzzega et al., 2020), and MER (Riemer et al., 2019).
CL in MRC. Few previous studies apply continual learning to MRC. Su et al. (2020) adapted EWC method and enlarged the MRC architecture when a new domain arrives. Su et al. (2020) added a penalty regularization that restricts the change of important parameters to prevent forgetting. de Masson d'Autume et al. (2019) and (Abujabal et al., 2018) designed episodic memory based methods that store training samples from previously seen data, which are later rehearsed to learn new domains. In this paper, we solve the continual MRC problem of incrementally learning over sequential domains, and build our continual model based on the above memory-based and penalty regularization paradigm.
(a) Initial Training.
(b) Incrementally Learning for New Domain Data.
Figure 2: The overview of the proposed MA-MRC framework with sequential domain data.
2.2 Domain Adaptation
Domain adaptation aims to learn discriminative feature features and generalize to other new domains and is usually achieved by learning domain-invariant features (Ben-David et al., 2010; Ganin et al., 2016). For MRC under domain shift, previous methods (Wang et al., 2019; Cao et al., 2020; Lee et al., 2019) reduce domain discrepancy by a discriminator network that is trained to distinguish features of the target from source domains. However, recent work usually transfers the model from source to target. In this paper, we explore the domain adaptation under continual setting.
For the task of continual MRC, we assume that the learning framework has access to streams of MRC data from  $T$  different domains, denoted by  $D_{t} = \{D_{1},D_{2},\dots,D_{T}\}$ . Each domain data, e.g.  $D_{t} = \{P_{n},Q_{n},A_{n}\}_{n = 1}^{|D_{t}|}$ , consists of a series of <passage, question, answer> triples, where  $|D_t|$  is the sample number of  $D_{t}$ . The MRC model is required to continually learn over each incoming domain data. More specifically, at each step  $t$ , the MRC model only observes  $D_{t}$  domain data, and is required to perform well on both the  $t$ -th domain data and
previous  $t - 1$  domain data. Hence, after training on  $D_{t}$ , the model will be evaluated on all seen  $t$  domains. To make the MRC model perform well on previous tasks, during the continual learning process, a memory  $M$  is to set to store a small number of samples in previous domain data in a fixed-size memory  $\mathcal{M} \ll |D_{old}|$ .
This paper focuses on the span extractive MRC. Inspired by Su et al. (2020), we perform two different continual domain settings: CDA-C and CDA-Q, which define different domains according to paragraph type and question type, respectively.
3.2 Method overview
Figure 2 shows the overview of MA-MRC. As shown in Figure 2(a), we first train a backbone MRC model (Transformer Encoder and Answer Prediction modules) on the initial domain data and randomly select  $\mathcal{M}$  training samples as the initial memory  $M$ . Figure 2(b) shows the process of learning new domain data. When the  $t$ -th domain data is arising, MA-MRC synchronously 1) fine-tunes the backbone MRC model with both the  $t$ -th domain and memory data; 2) adversarially learns domain-invariant and transfer representations via a domain discriminator that distinguishes memory from current data, so as to generalize well on new domain and avoid overfitting very small memory samples;
1: Train the backbone MRC model  $\theta_{1}$  with  $D_{1}$
3: for  $t = 2, \dots, T$  do
4: Define the domain discriminator  $\theta_{D}$
8: end for
9: return the final MA-MRC model  $\theta_T$ ;
3) utilizes knowledge distillation to encourage the model to mimic its original responses for past domain samples. After finishing training at  $t$ -th step, we update the memory with an uncertainty-aware sampling strategy that focuses on remembering what the model most needs. More details about the training process are shown in Algorithm 1.
3.3 Initial Training
Note that there is only one domain data at the beginning, so the initial training with  $D_{1}$  can be seen as a special case of formal extractive MRC tasks. Therefore, we build a normal backbone MRC model  $\theta_{1}$  (a standard BERT-MRC model with Transformer Encoder and Answer Prediction Modules) on the first domain data  $D_{1}$ . We initialize a fixed-size memory  $M_{1}$  to keep previous training samples that are periodically replayed while learning the new domain.
3.3.1 Prepare Backbone model
Transformer Encoder Module: First, a pre-trained  $L$  Transformer encoder blocks is used to convert the input sequence  $S = \left[\langle CLS\rangle ,Q,\langle SEQ\rangle ,P,\langle SEQ\rangle \right]$  into contextual representations. Then, the last block output  $H_{L} = \mathrm{BERT}(S)\in \mathbb{R}^{(l)\times h}$  is taken as the contextual representation, where  $h$  is the hidden dimension of BERT,  $l$  is the sequence length.
Answer Prediction Module: A linear layer is applied on the contextual representation  $H_{L}$  to calculate the probability distribution of start and end positions of candidate answer:
$$
p ^ {\text {s t a r t}} = \operatorname {S o f t m a x} \left(H _ {L} W ^ {s}\right), \tag {1}
$$
$$
p ^ {e n d} = \operatorname {S o f t m a x} \left(H _ {L} W ^ {e}\right), \tag {2}
$$
where  $W^{s}, W^{e} \in \mathbb{R}^{h}$  are learnable parameters.
Objective Function: The loss function of the backbone MRC is the cross-entropy:
$$
\min  _ {\theta_ {T, A P}} \mathcal {L} _ {s p a n} = - \log p _ {y ^ {s}} ^ {s t a r t} - \log p _ {y ^ {e}} ^ {e n d}, \tag {3}
$$
where  $y^{s}$  and  $y^{e}$  are the ground-truth start and end indices of the corresponding sample, respectively.
3.3.2 Memory Initialization
For the first domain, to preserve the data distribution of the current domain as much as possible, we randomly select  $\mathcal{M}$  training samples from  $D_{1}$  as the initial memory  $M_{1}$ .
3.4 Learning for New Domain
When the  $t$ -th domain data is coming, we can access the current data  $D_{t}$  and the memory  $M_{t-1}$ . If we just finetune the backbone model, the model is hard to have good transfer ability and may overfit on the few memorized samples. Hence the MA-MRC utilizes the adversarial learning strategy to fully make use of the domain adaptation relationship between the previous and the current domain. After the  $t$ -th training step, we dynamically update memory  $M$  with an uncertainty-aware strategy to store the training samples that the model most needs to replay.
3.4.1 Memory-based Adversarial Training
To fully use the domain adaption relationship, we adversarially learn the domain-invariant and transfer representations of the memory and current domain data. Inspired by domain adaptation theory, MA-MRC introduces a domain discriminator and build a two-player min-max game. The first player, a domain discriminator  $D$ , distinguishes memory data from the current new domain data. Here  $D$  is a basic binary discriminator that has a three linear layer followed by a sigmoid activation function. The second player, the Transformer Encoder  $T$ , aims to learn features that confuse  $D$ . We utilize empirical Maximum Mean Discrepancy as distance
measure to reduce the difference between marginal representation distributions and make them similar:
$$
d = M M D (T (M _ {t - 1}), T (D _ {t})). \qquad (4)
$$
Finally, this learning procedure can be described by the following minimax game:
$$
\begin{array}{l} \min _ {\theta_ {T}} \max _ {\theta_ {D}} \mathcal {L} _ {a d v} = - \frac {1}{\mathcal {M}} \sum_ {i = 1} ^ {\mathcal {M}} \log D (T (P _ {i}, Q _ {i})) \\ - \frac {1}{\left| D _ {t} \right|} \sum_ {j = 1} ^ {\left| D _ {t} \right|} \log (1 - D (T (P _ {j}, Q _ {j}))) + d, \quad (5) \\ \end{array}
$$
where  $\mathcal{M}$  and  $|D_t|$  are the number of training samples from memory and the current domain.
3.4.2 Knowledge Distillation
It is obvious that a good continual model can learn a new domain data well while approximating the behavior observed in the old ones. Then, we leverage knowledge distillation constraints to encourage MA-MRC to mimic its original responses for past samples. Specifically, we seek to minimize the distance between the corresponding pre-softmax responses to preserve the knowledge about previous memory data:
$$
\begin{array}{l} \mathcal {L} _ {K L} = K L \left(\operatorname {l o g i t s} _ {\theta_ {t - 1}} ^ {\text {s t a r t}} \mid \mid \operatorname {l o g i t s} _ {\theta_ {t}} ^ {\text {s t a r t}}\right) \\ + K L \left(\operatorname {l o g i t s} _ {\theta_ {t - 1}} ^ {\text {e n d}} \mid \mid \operatorname {l o g i t s} _ {\theta_ {t}} ^ {\text {e n d}}\right), \tag {6} \\ \end{array}
$$
where  $\text{logits}$  are the probability distribution before softmax and  $KL$  is the KL divergence.
3.4.3 Uncertainty-aware Memory Updating
Unlike other approaches that store fixed examples for each domain, we use a fixed memory for all domains to avoid memory growing. Therefore, when a domain arises, it is necessary to remove some old samples in memory to store new data. We design a dynamic sampling strategy that focuses on samples what the model most needs to replay. Based on existing trained MRC model  $\theta_{t}$ , we first compute the uncertainty for each sample in memory by a unsupervised Entropy-based stragety:
$$
u ^ {e n t r o p y} = \log p _ {y ^ {s}} ^ {s t a r t} + \log p _ {y ^ {e}} ^ {e n d}. \qquad (7)
$$
Then we calculate the gap between the above uncertainty and 1) the respective previous best uncertainty 2) the average uncertainty of all memory data, and normalize these metric differentials to create a probability distribution. For ease of exposition, we define these two normalization calculations as norm $^{1}$  and norm $^{2}$ . Note that we will
Domains	#train	#test	|p|	|q|	|a|
CDA-Q setting
what	37593	4749	118	9.9	3.5
which	4159	454	123.7	10.3	2.7
where	3291	433	120.4	8.2	3.1
when	5459	696	123.3	8.6	2.3
how	8124	1090	120.6	9.9	3
why	1201	151	123.9	9.6	8
other	19622	1938	118	11.6	2.7
who	8150	1059	126.1	9	2.8
CDA-C setting
SQuAD	10000	10570	119.8	10.1	3.2
NaturalQA	10000	12836	152.4	9.2	4.3
HotpotQA	10000	5901	154.1	19.5	2.3
NewsQA	10000	4212	495.1	6.6	4
TriviaQA	10000	7785	674.2	13.2	1.6
Table 1: Dataset statistic of CDA-Q and CDA-C.
sample  $\frac{\mathcal{M}}{t}$  data for each previously seen domain in memory with the above distribution separately. Finally, we get  $\frac{\mathcal{M}(t - 1)}{t}$  data from memory and randomly sample  $\frac{\mathcal{M}}{t}$  data from current domain. In this way, the more forgotten a memory sample is, the more it will be retained.
3.4.4 Objective Function
When incrementally learning for a new domain, the span loss in answer prediction module  $\mathcal{L}_{\text{span}}'$  considers all current domain data  $D_{t}$  and the memory data  $M$ , while the KL loss only takes memory data  $M$  into account. Finally, the overall loss function at the  $t$ -th step is formulated as:
$$
\min  _ {\theta_ {T}, A P} \max  _ {\theta_ {D}} \mathcal {L} _ {\text {f i n a l}} = \mathcal {L} _ {\text {s p a n}} ^ {\prime} + \mathcal {L} _ {\text {a d v}} + \mathcal {L} _ {K L}. \tag {8}
$$
4.1 Continual MRC Datasets
Inspired by Su et al. (2020), this paper deals with two continual MRC under domain adaptation: CDA-C and CDA-Q. For CDA-C setting, we regard MRC datasets with different passage corpora (e.g., Wikipedia, News, and Web snippets) as different domains and choose five datasets: SQuAD 1.1 (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018), Natural Questions (Trischler et al., 2017), NewsQA (Trischler et al., 2017), and TriviaQA (Joshi et al., 2017). Due to computational limits, we use the curated version provided by Fisch et al. (2019). For each dataset, we randomly sample  $10,000 < \text{question}$ , context, answer> triples from the original training datasets for continual training, and the original dev sets for testing. For CDA-Q setting, we make use of the original SQuAD
Methods	SQuAD	NaturalQA	HotpotQA	NewsQA	TriviaQA	F1avg	F1all
UpperBound	78.23	67.41	68	58.63	57.59	65.97	67.52
LowerBound	44.67	44.46	54.24	38.29	58.35	48	47.9
EWC	59.28	51.25	57.6	41.56	59.54	53.85	54.79
OnlineEWC	67.64	55.81	59.61	46.68	59.28	57.8	59.1
AGEM*	64.25	55.34	58.72	45.05	58.92	56.45	57.73
DER*	41.53	63.9	63.21	56.48	58.75	56.77	56.35
DER++*	46.96	63.05	63.5	56.62	57.94	57.61	53.37
MA-MRC(norm1)*	68.13	60.33	61.25	52.2	60.03	60.39	61.57
MA-MRC(norm2)*	70.11	59.35	61.92	52.18	59.51	60.61	61.77
Table 2: The overall results under CDA-C setting. “*” indicates that the  $\mathcal{M}$  memory data is used.
Methods	what	which	where	when	how	why	other	who	F1avg	F1all
UpperBound	76.91	80.64	72.02	87.51	73.09	54.49	77.18	84.58	75.8	77.67
LowerBound	63.8	72.42	55.98	63.62	65.79	23.7	69.27	82.89	62.18	66.39
EWC	63.27	74.4	53.17	60.7	63.77	34.62	66.92	82.19	62.38	65.37
OnlineEWC	65.72	72.3	59.48	60.75	60.91	34.21	69.09	83.74	63.28	66.89
AGEM*	66.88	74.88	66.95	76.52	66.72	49.78	70.36	85.28	69.67	70.08
DER*	55.95	75.91	71.47	85.58	72.94	54.45	76.69	84.17	72.15	67.75
DER++*	56.06	78.25	70.42	85.97	72.36	54.69	77.29	84.66	72.46	67.99
MA-MRC(norm1)*	72.31	75.815	69.2	85.59	72	56.76	75.65	85.86	74.15	74.92
MA-MRC(norm2)*	72.23	77.39	68.74	85.12	70.6	53.1	76.28	85.11	73.57	74.75
Table 3: The overall results under CDA-Q setting. “*” indicates that the  $\mathcal{M}$  memory data is used.
1.1 (Rajpurkar et al., 2016) and split it into eight domains according to the question type such as what, why, and how. The detailed statistics of these datasets are shown in Table 1.
4.2 Methods Compared
- Bounds. We design a standard BERT-MRC (the same as the backbone MRC model in Sec 3.3) as the basic model and then define two bounds on it: 1) Lower Bound continually fine-tunes the BERT-MRC model for each new domain without memorizing any historical examples. 2) Upper Bound remembers all examples in history and continually re-train the BERT-MRC model with all data.
- EWC (Kirkpatrick et al., 2017) restricts the change of model parameters for previous domains via elastic weight consolidation and a special  $L_{2}$  regularization. Hence it can slow down the learning of parameters important for all previous domains.
- Online EWC (Schwarz et al., 2018), the extension of EWC, which only considers the restriction for the latest model parameters.
- DER (Buzzega et al., 2020), memory-based approaches, leverages knowledge distillation for retaining past experience.
- DER++ (Buzzega et al., 2020), the extension of DER, uses an additional term on memory.
- AGEM (Chaudhry et al., 2019), memory-based approaches, uses a constraint that enables the projected gradient to decouple the average loss on previous seen domains.
Exact Match (EM) and word-level F1 score are used to evaluate the performance of MRC model in a single domain data. As for the continual domain adaptation setting, two common evaluation settings in continual learning theory are adopted: the average and the whole performance:
$$
F 1 _ {a v g} = \frac {1}{T} \sum_ {i = 1} ^ {T} F 1 \left(D _ {t e s t} ^ {i}\right), \tag {9}
$$
$$
F 1 _ {a l l} = F 1 \left(D _ {t e s t} ^ {1: T}\right). \tag {10}
$$
The former is the average F1 score on test sets of all seen domain, and the latter is the whole F1 score on the test sets.
We initialize the transformer encoder layer with the pre-trained  $\mathrm{BERT}_{\mathrm{BASE}}$  model officially released by Google<sup>1</sup>. The maximum sequence length is 384, and the batch size is 30. We set memory size  $\mathcal{M}$  400 default that means the memory stores up to 400 training samples for previous seen domain. When incrementally learn the new incoming domain data at  $t$ -th step, we first reinitialize the parameters of the domain discriminator  $\theta_{D}$ . Then use Adam optimizer (Kingma and Ba, 2015) with a learning rate of 3e-5 and training MA-MRC model for 3 epochs.
<sup>1</sup>https://github.com/google-research/bert
Methods	SQuAD	NaturalQA	HotpotQA	NewsQA	TriviaQA	F1avg	F1all
Full Model	68.13	60.33	61.25	52.2	60.03	60.39	61.57
w/o Adv	65.68	56.99	59.81	52.47	59.93	58.97	59.71
w/o KL	62.65	57.57	60.34	51.48	60.11	58.43	59.12
w/o Adv+KL	53.10	49.28	55.12	47.06	58.53	52.62	52.61
Table 4: The ablation study results under CDA-C setting.
Methods	what	which	where	when	how	why	other	who	F1avg	F1all
Full Model	72.31	75.815	69.2	85.59	72	56.76	75.65	85.86	74.15	74.92
w/o Adv	66.87	75.55	67.09	81.08	68.76	47.98	72.41	84.02	70.47	70.85
w/o KL	70.93	78.83	70.16	84.58	72.09	55.23	75.92	85.57	74.16	74.42
w/o Adv+KL	67.60	73.10	60.10	80.62	65.13	44.75	70.65	84.25	68.28	70.03
(a) Under CDA-C setting.
(b) Under CDA-Q setting.
Figure 3: The average F1 changes with increasing domains through the continual learning process.
4.5 Results and Discussion
4.5.1 Main Results