SlowGuess/NAACL_2022_OCR huggingface dataset:
Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, Xiangang Li
Beike Inc., Beijing, China
{jiyunjie001,chenliangyu003,douchenxiao001, mabaochang001,lixiangang002}@ke.com
Machine Reading Comprehension with Unanswerable Questions is a difficult NLP task, challenged by the questions which can not be answered from passages. It is observed that subtle literal changes often make an answerable question unanswerable, however, most MRC models fail to recognize such changes. To address this problem, in this paper, we propose a span-based method of Contrastive Learning (spanCL) which explicitly contrast answerable questions with their answerable and unanswerable counterparts at the answer span level. With spanCL, MRC models are forced to perceive crucial semantic changes from slight literal differences. Experiments on SQuAD 2.0 dataset show that spanCL can improve baselines significantly, yielding  $0.86\sim 2.14$  absolute EM improvements. Additional experiments also show that spanCL is an effective way to utilize generated questions.
Machine Reading Comprehension (MRC) is an important task in Natural Language Understanding (NLU), aiming to answer specific questions through scanning a given passage(Hermann et al., 2015; Cui et al., 2016; Rajpurkar et al., 2018). As a fundamental NLU task, MRC also plays an essential role in many applications such as question answering and dialogue tasks (Chen et al., 2017; Gupta et al., 2020; Reddy et al., 2019). With the rapid development of pre-trained language models (PLMs), there is also a paradigm shift (Schick and Schütze, 2020; Dai et al., 2020; Sun et al., 2021) reformulating other NLP tasks (e.g. information extraction) into MRC format, especially for open-domain scenarios (Li et al., 2019; Yan et al., 2021a).
In most of the application scenarios, there exists a hypothesis that only answerable questions can be asked, which is somehow unrealistic and unreasonable. Thus, the model that is capable of distinguish-
The Legend of Zelda: Twilight Princess (Japanese: しぃルダの伝説 ユイヤルトリーマル、Hepburn: Zeruda no Densetsu: Towairaito PurinSES?) is an action-adventure game developed and published by Nintendo for the GameCube and Wii home video game consoles. It is the thirteenth installment in The Legend of Zelda series. Originally planned for release on the GameCube in November 2005, Twilight Princess was delayed by Nintendo to allow its developers to refine the game, add more content, and port it to the Wii. The Wii version was released alongside the console in North America in November 2006, and in Japan, Europe, and Australia the following month. The GameCube version was released worldwide in December 2006.[b]
Original Question:
What year was the Legend of Zelda: Twilight Princess originally planned for release?
Question Distortion:
What year was the Legend of Zelda: Australian Princess originally planned for release?
Question Paraphrase:
When was the legend of Zelda: Twilight Princess originally planned to be released?
Figure 1: Question Distortion and Question Paraphrase are derived by slightly changing Original Question.
ing unanswerable questions is more welcomed than the model that can only give plausible answers (Rajpurkar et al., 2018). However, the challenge, that a slight literal change may transfer answerable questions into unanswerable ones, makes MRC models hard to gain such capability(Rajpurkar et al., 2018). For example, in Figure 1, the original answerable question becomes unanswerable by only replacing Twilight with Australian, but the small literal modification towards paraphrasing does not change the answer. Recent MRC models which predict answers using context-learning techniques and type-matching heuristics are not easy to perceive such subtle but crucial literal changes(Weissenborn et al., 2017; Jia and Liang, 2017). If different questions share many words in common, these models are most likely to give them the same answer, i.e., 2005 may be answered for all the three questions in Figure 1.
To address the aforementioned challenge, we propose a span-based method of Contrastive Learning (spanCL) in this paper. By explicitly contrasting answerable questions with their paraphrases and their distortions, MRC models are forced to recognize the subtle but crucial literal changes. Using pre-trained language model (PLM) as encoder, most contrastive learning methods adopt [CLS] as
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1292 - 1300
the sentence representation (Luo et al., 2020; Wu et al., 2020; Gao et al., 2021; Yan et al., 2021b; Wang et al., 2021). However, in this problem, as the differences between contrastive questions are very subtle, [CLS] is inadequate to capture such small changes. To solve the challenge, we propose a novel learning method, which incorporates the comparative knowledge between answerable and unanswerable questions, and exploits the semantic information of answer spans to improve the sentence representation. Overall, our contributions are summarized as two folds:
- To improve MRC model's capability of distinguishing unanswerable questions, we propose a simple yet effective method called spanCL, which teaches the model to recognize crucial semantic changes from slight literal differences.
- Comprehensive experiments show that spanCL can yield substantial performance improvements of baselines. We also show that spanCL is an effective way to utilize generated questions.
Models for MRC. With the help of various large-scale reading comprehension datasets (Hermann et al., 2015; Hill et al., 2015; Trischler et al., 2016; Rajpurkar et al., 2016; Lai et al., 2017; Rajpurkar et al., 2018), neural networks have achieved a great success on MRC in recent years. At first, these models are typically designed with a LSTM (Hochreiter and Schmidhuber, 1997) or CNN (LeCun et al., 1998) backbone, based on word embeddings (Mikolov et al., 2013; Pennington et al., 2014), leveraging various attention mechanisms to build interdependent representations of passage and question (Kadlec et al., 2016; Dhingra et al., 2016; Cui et al., 2016; Seo et al., 2016). Recently, pre-trained language models (PLMs) made a profound impact on NLP tasks (Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019a; Liu et al., 2019; Lan et al., 2019; Clark et al., 2020; Brown et al., 2020; Fedus et al., 2021). With millions, billions even trillions of parameters, PLMs show a great capacity of capturing contextualized representations, and significantly boost the performance of MRC models.
MRC with Unanswerable Questions. Knowing what you do not know is a crucial aspect of
model intelligence (Rajpurkar et al., 2018). In the field of MRC, a model should abstain from answering when no answer is available to the question. To deal with unanswerable questions, previous researchers mostly focused on designing a powerful answer verification module (Clark and Gardner, 2017; Liu et al., 2018; Kundu and Ng, 2018; Hu et al., 2019). Recently, a double checking strategy is proposed, in which an extra verifier is adopted to rectify the predicted answer (Hu et al., 2019; Back et al., 2019; Zhang et al., 2020a,b). Besides the idea of designing verification modules, some other studies try to solve the problem through data augmentation, namely to synthesize more QA pairs (Yang et al., 2019b; Alberti et al., 2019; Zhu et al., 2019b; Liu et al., 2020).
Contrastive Learning. To obtain rich representations of texts for down-stream NLP tasks, there have been numerous investigations of using contrastive objectives on strengthening supervised learning (Khosla et al., 2020; Gunel et al., 2020) and unsupervised learning (Chen et al., 2020; Gao et al., 2021) in various domains (He et al., 2020; Lin et al., 2020; Ier et al., 2020; Kipf et al., 2019). The main idea of contrastive learning (CL) is to learn textual representations by contrasting positive and negative examples, through concentrating the positives and alienating the negatives. In NLP tasks, CL is usually devoted to learning rich sentence representations (Luo et al., 2020; Wu et al., 2020), and the main difference between these methods is the approach to find positive and negative examples. Wang et al. (2021) argued that using hard negative examples in CL is helpful to improve the semantic robustness and sensitivity of pre-trained language models. Enlightened by the promising effects of CL, Kant et al. (2021) proposed to use CL in visual question answering. He focused on playing CL on MRC by comparing multiple answer candidates, but neglected the fact that not all questions can be answered through a given paragraph.
In this section, we first introduce the task of Machine Reading Comprehension with Unanswerable Questions (MRC-U). Then, a baseline MRC model based on PLM is described. At last, we propose a span-based contrastive learning method for MRC-U, named as spanCL. In this paper, question paraphrase and positive question, question distortion and negative question are used interchangeably.
(a) Base Model
(b) Span-based Contrastive Learning
Figure 2: (a) The baseline model for MRC-U. (b) Span-based contrastive learning on answer-span representation.
In this paper, we focus on studying extractive MRC, in which the expected answer of a question is a word span of a given passage. Thus, given a textual question  $Q$  and a textual passage  $P$ , our goal is to find the answer span  $(y_{s}, y_{e})$  to  $Q$  in  $P$ , where  $y_{s}$  is the answer start position in  $P$  and  $y_{e}$  is the answer end position in  $P$ .
3.2 Basic MRC Model
We use the model same as Devlin et al. (2018) as the basic model for MRC-U task. When a question and a passage are input, if the question is answerable, the model is expected to give a legal answer span  $(y_{s}, y_{e})$  in the passage; if the question is unanswerable, the model is expected to output the [CLS] span  $(0, 0)$ , which indicates no related answer can be found in the passage. The overall structure of the network is presented in Figure 2.
For illustration, we denote the output of PLM's last layer as the sequence representation,  $H \in \mathbb{R}^{l \times d}$ , where  $l$  is the sequence length and  $d$  is the dimension. Accordingly, the hidden representation of the  $i$ -th token in the sequence is denoted as  $h_i \in H$ . To find the start position of an answer, a start weight vector  $w_s \in \mathbb{R}^d$  is introduced to calculate the beginning possibility of each position. Formally, the probability that the answer starts at the  $i$ -th token is defined as
$$
p _ {i} ^ {s} = \frac {\exp \left(\boldsymbol {h} _ {i} \cdot \boldsymbol {w} _ {s}\right)}{\sum_ {j \leq l} \exp \left(\boldsymbol {h} _ {j} \cdot \boldsymbol {w} _ {s}\right)}. \tag {1}
$$
Similarly, with a end weight vector  $\pmb{w}_{e} \in \mathbb{R}^{d}$ , the probability that the answer ends at the  $i$ -th token is
$$
p _ {i} ^ {e} = \frac {\exp \left(\boldsymbol {h} _ {i} \cdot \boldsymbol {w} _ {e}\right)}{\sum_ {j \leq l} \exp \left(\boldsymbol {h} _ {j} \cdot \boldsymbol {w} _ {e}\right)}. \tag {2}
$$
For learning, the cross-entropy loss on identifying the answer start and end positions is taken as the training objective as
$$
\mathcal {L} _ {s p a n} = - \log \left(p _ {y _ {s}} ^ {s}\right) - \log \left(p _ {y _ {e}} ^ {e}\right), \tag {3}
$$
where  $y_{s}$  and  $y_{e}$  are the start and end positions of the true answer span. With the learnt model, the output answer span  $(y_s', y_e')$  is predicted according to
$$
\left(y _ {s} ^ {\prime}, y _ {e} ^ {\prime}\right) = \arg \max  _ {(i, j) | i \leq j} \boldsymbol {h} _ {i} \cdot \boldsymbol {w} _ {s} + \boldsymbol {h} _ {j} \cdot \boldsymbol {w} _ {e}. \tag {4}
$$
3.3 Span-based Contrastive Learning
In this section, spanCL is introduced from two aspects. First, considering the contrastive idea of CL, we give the details about how the positive and negative examples are generated. Second, the training objective of spanCL is presented.
Positive Examples. In our method, we define the positive examples as the questions which have slight literal differences but the same answers with their original questions. Back Translation is an effective data augmentation method (Xie et al., 2019; Zhang et al., 2017; Zhu et al., 2019a), in which a text is first translated to a target language (e.g. France) from its source language (e.g. English), and then back translated to the source language. The final back-translated text is taken as the example of augmentation. Thanks to Back Translation, the produced examples are lexically different
Strategy	Example
Negation	Original question: What was Beyonce's role in Destiny's Child?
Negative question: What wasn't Beyonce's role in Destiny's Child?
Entity replacement	Original question: What native people lived in the San Diego area before the Europeans arrived?
Negative question: What native people lived in the San Diego area before the Mexicans arrived?
Antonym	Original question: What part of Gothic buildings are often found terminated with enormous pinnacles?
Negative question: What other part of Gothic buildings are often found terminated with small pinnacles?
Table 1: Strategies used to generate negative questions.
but semantically same with the original example. Specifically, for each question, we first produce three question paraphrases by Back Translation using three languages. Then we select the question that has the most literal differences with the original question as the positive question.
Negative Examples. In our method, we define the negative examples as the questions which have slight literal differences and not the same answers with their original questions. Three simple strategies are adopted to produce negative examples as the following.
- Negation. A negation word is inserted or removed from the original question.
- Antonym. First, spaCy is utilized to conduct segmentation and POS for the original question. Then, one of the words (verbs, nouns, adjectives, or adverbs) are randomly replaced with its antonym.
- Entity Replacement. With an answerable question, one of its entity words is randomly placed with another entity word, which has the same entity type but does not appear in any questions.
Table 1 shows several negative examples derived by these strategies. Note that question generation is not the main topic of this paper.
Span-based Contrastive Learning. Using PLM as the encoder, [CLS] usually serve as the sentence representation in CL (Gao et al., 2021; Wang et al., 2021; Yan et al., 2021b). When the difference between the original question and its paraphrase or distortion is very subtle, a single [CLS] token is not adequate to capture the difference, making the model hard to answer such question. Therefore, we propose to improve MRC models by contrasting
these questions according to their answer-span representations. Specifically, given a question  $Q_{org}$  and its answer span  $(y_s, y_e)$ , through the augmentation methods mentioned previously, we synthesize one positive question  $Q_{pos}$  and one negative question  $Q_{neg}$ . Based on the definition of positive examples and negative examples,  $(y_s, y_e)$  is the answer span to both  $Q_{pos}$  and  $Q_{org}$  but not to  $Q_{neg}$ . Denote  $h_{ys}^{Q_{org}}$  and  $h_{ye}^{Q_{org}}$  as the representation vectors of the  $y_s$ -th token and  $y_e$ -th token in the input passage  $P$  for the question  $Q_{org}$ ,  $h_{ys}^{Q_{pos}}$  and  $h_{ye}^{Q_{pos}}$  as those for  $Q_{pos}$ , and  $h_{ys}^{Q_{neg}}$  and  $h_{ye}^{Q_{neg}}$  as those for  $Q_{neg}$ . The concatenation of  $h_{ys}^{Q_{org}}$  and  $h_{ye}^{Q_{org}}$  is used as the answer-span representation to  $Q_{org}$  and denoted as  $z^{Q_{org}}$ . Similarly, the answer-span representation to  $Q_{pos}$  and  $Q_{neg}$  are denoted as  $z^{Q_{pos}}$  and  $z^{Q_{neg}}$  respectively. Then, our span-based contrastive loss is calculated as
$$
\begin{array}{l} \mathcal {L} _ {\text {s p a n C L}} = - \log \frac {\exp \left(\Phi \left(\boldsymbol {z} ^ {Q _ {\text {o r g}}}, \boldsymbol {z} ^ {Q _ {\text {p o s}}}\right) / \tau\right)}{\exp \left(\Phi \left(\boldsymbol {z} ^ {Q _ {\text {o r g}}}, \boldsymbol {z} ^ {Q _ {\text {p o s}}}\right) / \tau\right))} \tag {5} \\ + \exp (\Phi (z ^ {Q _ {o r g}}, z ^ {Q _ {n e g}}) / \tau)) \\ \end{array}
$$
where  $\Phi (\pmb {u},\pmb {v}) = \pmb{u}^{\top}\pmb {v} / \| \pmb {u}\| \| \pmb {v}\|$  computes similarity between  $\pmb{u}$  and  $\pmb{v}$  and  $\tau >0$  is a scalar temperature parameter. With the definition, the final objective loss of our method is presented as the following:
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {\text {s p a n}} + \lambda_ {2} \mathcal {L} _ {\text {s p a n C L}}. \tag {6}
$$
We evaluate our method on the well-known dataset SQuAD 2.0 (Rajpurkar et al., 2018), which covers the questions of SQuAD1.1 (Rajpurkar et al., 2016) with new unanswerable questions written adversarially by crowdworkers to imitate the answerable ones. Moreover, for each unanswerable question, a
$^{1}$ https://github.com/explosion/spaCy
plausible answer span is annotated, which indicates the incorrect answer obtained by type-matching heuristics. The training dataset contains 87k answerable and 43k unanswerable questions, and half of the examples in the development set are unanswerable.
Two official metrics are used to evaluate the model performance on SQuAD 2.0: Exact Match (EM) and F1. EM is used to compute the percentage of predictions that match ground truth answers exactly. F1 is a softer metric, which measures the average overlap between the prediction and ground truth answer at token level.
MRC Model. We adopt the model introduced in 3.2 with various PLM encoders for the MRC-U task. Bert (Devlin et al., 2018), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019) are selected in our experiments. We download the pretrained weights from Hugging Face<sup>2</sup>.
Training Data Construction. For each original answerable question, we use Back Translation to generate its paraphrase. In SQuAD 2.0, we can find the negative questions for 18,541 answerable questions in the original dataset. For the rest 68,280 answerable questions, we use our augmentation strategies to generate negative questions.
During our training, the span loss is calculated based on  $Q_{org}$  and  $Q_{neg}$ . In section 4.3, we will explain why  $Q_{pos}$  is discarded for calculating span loss.
Hyper-parameters. We use the default hyperparameter settings for the SQuAD 2.0 task. Specifically, we set maximum sequence length, doc stride, maximum query length and maximum answer length to 512, 128, 64 and 30. For fine-tuning our model, we set the learning rate, batch size, training epoch and warm-up rate to 2e-5, 12, 2 and 0.1. The temperature in spanCL is set to 0.05. The weights of span loss and spanCL loss are  $\lambda_{1} = \lambda_{2} = 0.5$ . For each time, we fix the random seed, ensuring our results are reproducible. We run our experiments on two Tesla A100 40G GPUs with 5 GPU hours to train a model.
From Table 2, we notice that spanCL improves the performance of each baseline model, yielding  $0.86\sim 2.14$  absolute EM improvement and  $0.76\sim 2.0$
Model	Dev	Δ
EM	F1	EM	F1
BERTbase+ spanCL	73.37	76.34	-	-
75.51	78.34	+2.14	+2.00
BERTlarge+ spanCL	78.88	81.85	-	-
79.76	82.61	+0.88	+0.76
RoBERTabase+ spanCL	78.85	81.42	-	-
80.18	82.84	+1.33	+1.42
RoBERTalarge+ spanCL	86.12	88.88	-	-
86.98	89.70	+0.86	+0.82
ALBERTbase+ spanCL	77.84	81.27	-	-
79.52	82.97	+1.68	+1.7
ALBERTlarge+ spanCL	79.99	83.27	-	-
81.51	84.67	+1.52	+1.4
Table 2: Results (%) on the dev set of SQuAD 2.0.
Model	EM	Dev
HasAns	NoAns	EM	F1
BERTbase	70.31	74.76	73.37	76.34
+ pos	72.57	68.93	72.26	75.22
+ neg	67.05	87.74	74.02	76.38
+ pos&neg	66.16	78.48	72.59	75.37
+ spanCL	72.52	75.91	75.51	78.34
Table 3: Training with spanCL vs Training with expanded datasets.
absolute F1 improvement, demonstrating spanCL is model-agnostic and effective.
As additional training data (i.e. the extra positive and negative questions) is used, it is necessary to analyze if the improvements are merely brought by this additional data. We conduct experiments by training with different datasets and display the results in Table 3. BERT<sub>base</sub> means training BERT<sub>base</sub> with original SQuAD 2.0 training set. “+pos” and “+neg” mean expanding the original training set with generated positive questions and generated negative questions respectively. Surprisingly, Simply expanding the training set can not guarantee the performance improvement. We find that adding positive examples into the training set does not improve the performance of MRC model. One possible reason is that the positive questions make the model over insensitive and ignore slight literal changes, which is inappropriate for MRC-U task. By comparing BERT<sub>base</sub> with “+neg”, we find that training with more negative examples, the model tends to predict more NoAns and achieve a high performance on NoAns, while the performance on the HasAns drops a lot and the overall improvement of EM is much less than “+spanCL”. From the results in Table 3, we can conclude that spanCL is effective to utilize the generated questions.
$^{2}$ https://huggingface.co/bert-base-uncased
Model	EM	F1
BERTbase	73.37	76.34
+ CRQDA (Liu et al., 2020)	75.80	78.70
+ spanCL with simple negatives	75.51	78.34
+ spanCL with CRQDA	76.12	79.09
Figure 3: The influence of different temperatures in spanCL. The best performance is achieved when the temperature is set to 0.05.  $\mathrm{BERT}_{\mathrm{base}}$  is adopted as the base model.
4.4 Influence of Negative Examples
The unanswerable questions generated by our strategies are rather plain. We believe spanCL can further boost the performance by high-quality unanswerable questions. Liu et al. (2020) proposed a context-relevant generation method called CRQDA, which generates delicate negative questions<sup>3</sup>. In table 4, “+CRQDA” denotes training the baseline model with the dataset including the delicate negative questions generated by CRQDA. “+spanCL with simple negatives” denotes applying spanCL with negative questions generated by our three strategies. “+spanCL with CRQDA” denotes applying spanCL with negative questions generated by CRQDA. Comparing “+spanCL with simple negatives” with “+spanCL with CRQDA”, we find that spanCL can further boost the performance by delicate negative questions.
4.5 Influence of Temperature
The temperature  $\tau$  in spanCL loss (Equation 5) is used to control the smoothness of the distribution normalized by the softmax operation. A large temperature smoothes the distribution while a small temperature sharpens the distribution. As shown in the Figure 3, spanCL is sensitive to the temperature value. In general, small temperature results
Table 4: Performance of spanCL with different synthetic negative questions.
Model	EM	F1
BERTbase	73.37	76.34
+ spanCL	75.51	78.34
+ CL with [CLS] reps	74.18	77.05
+ CL with span and [CLS] reps	73.82	76.86
Table 5: Results  $(\%)$  with different question representations used in the contrastive learning.
Base Model	Training schemes	EM	F1
BERTbase	Joint	75.51	78.34
BERTbase	Alternate	74.67	77.08
BERTbase	pre-train then finetune	72.19	74.87
BERTlarge	Joint	79.76	82.61
BERTlarge	Alternate	79.50	82.55
BERTlarge	pre-train then finetune	77.77	80.30
Table 6: Results  $(\%)$  with different training schemes.
in better performance. A practical temperature can be obtained within a small range (from about 0.02 to 0.1). We select 0.05 as the temperature in our experiments.
4.6 Selection of Question Representations
In this paper, we argue that the answer span representation is better than [CLS]. We conduct experiments with different question representations in this section. When applying CL with [CLS] representations, we add a classification layer on the top of [CLS] to determine if a question is answerable or not (Zhang et al., 2020b), making the representation of [CLS] acquire the information of the question's answerability. We also play CL with both [CLS] and answer-span representations, in which two CL losses are calculated together. From Table 5, we can see that CL with [CLS] reps improves the model performance but the improvement is small than that from spanCL, and the combination of the two CL losses can confuse the model and result in a little improvement.