SlowGuess/ACL_2020_OCR huggingface dataset:
Akari Asai† and Hannaneh Hajishirzi†‡
†University of Washington ‡Allen Institute for AI {akari, hannaneh}@cs.washington.edu
Many natural language questions require qualitative, quantitative or logical comparisons between two entities or events. This paper addresses the problem of improving the accuracy and consistency of responses to comparison questions by integrating logic rules and neural models. Our method leverages logical and linguistic knowledge to augment labeled training data and then uses a consistency-based regularizer to train the model. Improving the global consistency of predictions, our approach achieves large improvements over previous methods in a variety of question answering (QA) tasks including multiple-choice qualitative reasoning, cause-effect reasoning, and extractive machine reading comprehension. In particular, our method significantly improves the performance of RoBERTa-based models by  $1 - 5\%$  across datasets. We advance state of the art by around  $5 - 8\%$  on WIQA and QuaRel and reduce consistency violations by  $58\%$  on HotpotQA. We further demonstrate that our approach can learn effectively from limited data. $^{1}$
Comparison-type questions (Tandon et al., 2019; Tafjord et al., 2019; Yang et al., 2018) ask about relationships between properties of entities or events such as cause-effect, qualitative or quantitative reasoning. To create comparison questions that require inferential knowledge and reasoning ability, annotators need to understand context presented in multiple paragraphs or carefully ground a question to the given situation. This makes it challenging to annotate a large number of comparison questions. Most current datasets on comparison questions are much smaller than standard machine reading comprehension (MRC) datasets (Rajpurkar
Figure 1: Inconsistent predictions by RoBERTa. Top row shows an example of symmetric inconsistency and the second row shows an example of transitive inconsistency. The examples are partially modified.
et al., 2016; Joshi et al., 2017). This poses new challenges to standard models, which are known to exploit statistical patterns or annotation artifacts in these datasets (Sugawara et al., 2018; Min et al., 2019a). Importantly, state-of-the-art models show inconsistent comparison predictions as shown in Figure 1. Improving the consistency of predictions has been previously studied in natural language inference (NLI) tasks (Minervini and Riedel, 2018; Li et al., 2019), but has not been addressed in QA.
In this paper, we address the task of producing globally consistent and accurate predictions for comparison questions leveraging logical and symbolic knowledge for data augmentation and training regularization. Our data augmentation uses a set of logical and linguistic knowledge to develop additional consistent labeled training data. Subsequently, our method uses symbolic logic to incorporate consistency regularization for additional supervision signal beyond inductive bias given by data augmentation. Our method generalizes previous consistency-promoting methods for NLI tasks (Minervini and Riedel, 2018; Li et al., 2019)
1Our code and data is available at https://github. com/AkariAsai/logic-guided_qa.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5642-5650 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
to adapt to substantially different question formats.
Our experiments show significant improvement over the state of the art on a variety of QA tasks: a classification-based causal reasoning QA, a multiple choice QA for qualitative reasoning and an extractive MRC task with comparisons between entities. Notably, our data augmentation and consistency constrained training regularization improves performance of RoBERTa-based models (Liu et al., 2019) by  $1.0\%$ ,  $5.0\%$  and  $2.5\%$  on WIQA, QuaRel and HotpotQA. Our approach advances the state-of-the-art results on WIQA and QuaRel with 4.7 and  $8.4\%$  absolute accuracy improvement, respectively, reducing inconsistent predictions. We further demonstrate that our approach can learn effectively from limited labeled data: given only  $20\%$  of the original labeled data, our method achieves performance on par with a competitive baseline learned with the full labeled data.
Data augmentation has been explored in a variety of tasks and domains (Krizhevsky et al., 2009; Cubuk et al., 2019; Park et al., 2019). In NLP, using backtranslation (Yu et al., 2018) or dictionary based word replacement (Zhang et al., 2015) has been studied. Most relevant to our work, Kang et al. (2018) study NLI-specific logic and knowledge-based data augmentation. Concurrent to our work, Gokhale et al. (2020) study visual QA models' ability to answer logically composed questions, and show the effectiveness of logic-guided data augmentation. Our data augmentation does not rely on task-specific assumptions, and can be adapted to different formats of QA task. We further leverage consistency-promoting regularization, which gives improvements in accuracy and consistency.
Improving prediction consistency via training regularization has been studied in NLI tasks. Minervini and Riedel (2018) present model-dependent first-order logic guided adversarial example generation and regularization. Li et al. (2019) introduce consistency-based regularization incorporating the first-order logic rules. Previous approach is model-dependent or relies on NLI-specific rules, while our method is model-agnostic and is more generally applicable by combining it with data augmentation.
Regularizing loss to penalize violations of structural constraints in models' output has been also studied in previous work on constraint satisfaction in structured learning (Lee et al., 2019; Ganchev
et al., 2010). Our work regularizes models to produce globally consistent predictions among augmented data following logical constraints, while those studies incorporates structured prediction models following linguistics rules.
We present the components of our QA method: first-order logic guided data augmentation (Section 3.1 and Section 3.2), and consistency-based regularization (Section 3.3).
3.1 Consistent Question Answering
For globally consistent predictions in QA, we require responses to follow two important general logical rules: symmetric consistency and transitive consistency, which are illustrated in Figure 1 and are formally described below.
Let  $q, p, a$  be a question, a paragraph and an answer predicted by a model. A is a set of answer candidates. Each element of A can be a span in  $p$ , a class category, or an arbitrary answer choice.  $X = \{q, p, a\}$  represents a logic atom.
Symmetric consistency In a comparison question, small surface variations such as replacing words with their antonyms can reverse the answer, while keeping the overall semantics of the question as before. We define symmetry of questions in the context of QA as follows:  $(q,p,a^{*})\leftrightarrow (q_{sym},p,a_{sym}^{*})$  , where  $q$  and  $q_{sym}$  are antonyms of each other, and  $a_{sym}^{*}$  is the opposite of the groundtruth answer  $a^*$  in A. For example, the two questions in the first row of Figure 1 are symmetric pairs. We define the symmetric consistency of predictions in QA as the following logic rule:
$$
(q, p, a) \rightarrow \left(q _ {s y m}, p, a _ {s y m}\right), \tag {1}
$$
which indicates a system should predict  $a_{sym}$  given  $(q_{sym},p)$ , if it predicts  $a$  for  $(q,p)$ .
Transitive consistency. Transitive inference between three predicates  $A, B, C$  is represented as:  $A \rightarrow B \land B \rightarrow C$  then  $A \rightarrow C$  (Gazes et al., 2012). In the context of QA, the transitive examples are mainly for causal reasoning questions that inquire about the effect  $e$  given the cause  $c$ . The second row of Figure 1 shows an example where transitive consistency is violated. For two questions  $q_{1}$  and  $q_{2}$  in which the effect of  $q_{1} (= e_{1})$  is equal to the cause of  $q_{2} (= c_{2})$ , we define the transitive consistency of predictions as follows:
$$
\left(q _ {1}, p, a _ {1}\right) \wedge \left(q _ {2}, p, a _ {2}\right)\rightarrow \left(q _ {\text {t r a n s}}, p, a _ {\text {t r a n s}}\right). \tag {2}
$$
5643
reasoning format	WIQA (Tandon et al., 2019) Causal Reasoning classification	QuaRel (Tafjord et al., 2019) Qualitative Reasoning multiple choice	HotpotQA (Yang et al., 2018) Qualitative Comparison of entities span extraction
p	The rain seeps into the wood surface. When rain evaporates it leaves the wood. It takes the finish of the wood with it. The wood begins to lose its luster.	Supposed you were standing on the planet Earth and Mercury. When you look up in the sky and see the sun,	Golf Magazine is a monthly golf magazine owned by Time Inc. El Nuevo Cojo Ilustrado is an Ameri-can Spanish language magazine.
q	q1: If a tsunami happens, will wood be more moist?, q2: If wood is more moist, is more weathering occurring?	Which planet would the sun appear larger?	El Nuevo Cojo and Golf Magazine, which one is owned by Time Inc?
A	{more, less, no effects}	{Mercury, Earth}	{Golf Magazine, El Nuevo Cojo}
a*	a1*: more, a2*: more	Mercury	Golf Magazine
qaug	If a tsunami happens, is more weather-ing occurring?	Which planet would the sun appear smaller?	Which one is not owned by Time Inc, Golf Magazine El Nuevo Cojo?
a* aug	more	Earth	El Nuevo Cojo
Table 1: An augmented transitive example for WIQA, and symmetric examples for QuaRel and HotpotQA. We partially modify paragraphs and questions. The bold characters denote a shared event connecting two questions. The parts written in red or blue denote antonyms, and highlighted text is negation added by our data augmentation.
3.2 Logic-guided Data Augmentation
Given a set of training examples  $X$  in the form of  $(q, p, a^*)$ , we automatically generate additional examples  $X_{aug} = \{q_{aug}, p, a_{aug}^*\}$  using symmetry and transitivity logical rules. The goal is to augment the training data so that symmetric and transitive examples are observed during training. We provide some augmented examples in Table 1.
Augmenting symmetric examples To create a symmetric question, we convert a question into an opposite one using the following operations: (a) replace words with their antonyms, (b) add, or (c) remove words. For (a), we select top frequent adjectives or verbs with polarity (e.g., smaller, increases) from training corpora, and expert annotators write antonyms for each of the frequent words (we denote this small dictionary as D). More details can be seen in Appendix A. For (b) and (c), we add negation words or remove negation words (e.g., not). For all of the questions in training data, if a question includes a word in D for the operation (a), or matches a template (e.g., which  $\star$  is  $\leftrightarrow$  which  $\star$  is not) for operations (b) and (c), we apply the operation to generate  $q_{sym}$ . We obtain  $a_{sym}^{*}$  by re-labeling the answer  $a^{*}$  to its opposite answer choice in A (see Appendix B).
Augmenting transitive examples We first find a pair of two cause-effect questions  $X_{1} = (q_{1},p,a_{1}^{*})$  and  $X_{2} = (q_{2},p,a_{2}^{*})$ , whose  $q_{1}$  and  $q_{2}$  consist of
$(c_{1},e_{1})$  and  $(c_{2},e_{2})$  , where  $e_1 = c_2$  holds. When  $a_1^*$  is a positive causal relationship, we create a new example  $X_{trans} = (q_{3},p,a_{2}^{*})$  for  $q_{3} = (c_{1},e_{2})$
Sampling augmented data Adding all consistent examples may change the data distribution from the original one, which may lead to a deterioration in performance (Xie et al., 2019). One can select the data based on a model's prediction inconsistencies (Minervini and Riedel, 2018) or randomly sample at each epoch (Kang et al., 2018). In this work, we randomly sample augmented data at the beginning of training, and use the same examples for all epochs during training. Despite its simplicity, this yields competitive or even better performance than other sampling strategies.
3.3 Logic-guided Consistency Regularization
We regularize the learning objective (task loss,  $\mathcal{L}_{task}$ ) with a regularization term that promotes consistency of predictions (consistency loss,  $\mathcal{L}_{cons}$ ).
$$
\mathcal {L} = \mathcal {L} _ {\text {t a s k}} (X) + \mathcal {L} _ {\text {c o n s}} (X, X _ {\text {a u g}}). \tag {3}
$$
The first term  $\mathcal{L}_{task}$  penalizes making incorrect predictions. The second term  $\mathcal{L}_{cons}^4$  penalizes making predictions that violate symmetric and transitive logical rules as follows:
$$
\mathcal {L} _ {\text {c o n s}} = \lambda_ {\text {s y m}} \mathcal {L} _ {\text {s y m}} + \lambda_ {\text {t r a n s}} \mathcal {L} _ {\text {t r a n s}}, \tag {4}
$$
where  $\lambda_{sym}$  and  $\lambda_{trans}$  are weighting scalars to balance the two consistency-promoting objectives.
2We observe that (b)(c) are less effective than (a) in WIQA or QuaRel, while especially (b) contributes to the performance improvements on HotpotQA as much as (a) does.
3We do not add  $X_{\text{aug}}$  if the same pair has already exist. 4We mask the  $\mathcal{L}_{\text{cons}}$  for the examples without symmetric or transitive consistent examples.
5644
Dataset	WIQA	QuaRel	HotpotQA
Dev	Test	v (%)	Dev	Test	v (%)	Dev	v (%)
x% data(# of X)	20%(6k)	40%(12k)	100%(30k)	100%(30k)	100%(30k)	20%(0.4k)	100%(2k)	100%(2k)	100%(2k)	20%(18k)	100%(90k)	100%(90k)
SOTA	-	-	-	73.8	-	-	-	76.6	-	-	-	-
RoBERTa	61.1	74.1	74.9	77.5	12.0	56.4	81.1	80.0	19.2	71.0	75.5	65.2
DA	72.1	75.5	76.3	78.3	6.0	69.3	84.5	84.7	13.3	73.1	78.0	6.3
DA + Reg	73.9	76.1	77.0	78.5	5.8	70.9	85.1	85.0	10.3	71.9	76.9	7.2
Previous studies focusing on NLI consistency (Li et al., 2019) calculate the prediction inconsistency between a pair of examples by swapping the premise and the hypothesis, which cannot be directly applied to QA tasks. Instead, our method leverages consistency with data augmentation to create paired examples based on general logic rules. This enables the application of consistency regularization to a variety of QA tasks.
Inconsistency losses The loss computes the dissimilarity between the predicted probability for the original labeled answer and the one for the augmented data defined as follows:
$$
\mathcal {L} _ {s y m} = | \log p (a | q, p) - \log p \left(a _ {a u g} \mid q _ {a u g}, p\right) |. \tag {5}
$$
Likewise, for transitive loss, we use absolute loss with the product T-norm which projects a logical conjunction operation  $(q_{1},p,a_{1})\wedge (q_{2},c,a_{2})$  to a product of probabilities of two operations,  $p(a_{1}|q_{1},p)p(a_{2}|q_{2},p)$ , following Li et al. (2019). We calculate a transitive consistency loss as:
$$
\begin{array}{l} \mathcal {L} _ {t r a n s} = | \log p (a _ {1} | q _ {1}, p) + \log p (a _ {2} | q _ {2}, p) - \\ \log p (a _ {t r a n s} | q _ {t r a n s}, p) |. \\ \end{array}
$$
Annealing The model's predictions may not be accurate enough at the beginning of training for consistency regularization to be effective. We perform annealing (Kirkpatrick et al., 1983; Li et al., 2019; Du et al., 2019). We first set  $\lambda_{\{sym,trans\}} = 0$  in Eq. (4) and train a model for  $\tau$  epochs, and then train it with the full objective.
Datasets and experimental settings We experiment on three QA datasets: WIQA (Tandon et al., 2019), QuaRel (Tafjord et al., 2019) and HotpotQA (oracle, comparison questions $^5$ ) (Yang et al., 2018).
Table 2: WIQA, QuaRel and HotpotQA results: we report test and development accuracy (%) for WIQA and QuaRel and development F1 for HotpotQA. DA and Reg denote data augmentation and consistency regularization. "SOTA" is Tandon et al. (2019) for WIQA and Mitra et al. (2019) for QuaRel.  $v$  presents violations of consistency.
WIQA	QuaRel
metric	acc	v (%)	acc	v (%)
DA (logic) + Reg	77.0	5.8	85.1	10.3
DA (logic)	76.3	6.0	84.5	13.5
DA (standard)	75.2	12.3	83.3	14.5
Reg	75.8	11.4	-	-
Baseline	74.9	12.0	81.1	19.2
Table 3: Ablation studies of data augmentation on WIQA and QuaRel development dataset.
As shown in Table 1, these three datasets are substantially different from each other in terms of required reasoning ability and task format. In WIQA, there are 3,238 symmetric examples and 4,287 transitive examples, while 50,732 symmetric pairs and 1,609 transitive triples are missed from the original training data. HotpotQA and QuaRel do not have any training pairs requiring consistency. Our method randomly samples 50, 80,  $90\%$  of the augmented data for WIQA, QuaRel and HotpotQA, resulting in 24,715/836/3,538 newly created training examples for those datasets, respectively.
We use standard F1 and EM scores for performance evaluation on HotpotQA and use accuracy for WIQA and QuaRel. We report a violation of consistency following Minervini and Riedel (2018) to evaluate the effectiveness of our approach for improving prediction consistencies. We compute the violation of consistency metric  $v$  as the percentage of examples that do not agree with symmetric and transitive logical rules. More model and experimental details are in Appendix.
Main Results Table 2 demonstrates that our methods (DA and DA + Reg) constantly give 1 to 5 points improvements over the state-of-the-art RoBERTa QA's performance on all three of the datasets, advancing the state-of-the-art scores on WIQA and QuaRel by  $4.7\%$  and  $8.4\%$ , respectively. On all three datasets, our method signifi
<sup>5</sup>We train models on both bridge and comparison questions, and evaluate them on extractive comparison questions only.
WIQA Input	RoBERTa	DA+Reg
p	Sound enters the ears of a person. The sound hits a drum that is inside the ears.
q	If the person has his ears more protected, will less sound be detected? [a*: More]	More (0.79)	More (0.93)	More (0.93)
qsym	If the person has his ears less protected, will less sound be detected? [a*sym*: Less]	More (0.87)	More (0.72)	Less (0.89)
p	Squirrels try to eat as much as possible. Squirrel gains weight.
q1	If the weather has a lot of snow, cannot squirrels eat as much as possible? [a1*: More]	Less (0.75)	More (0.48)	More (0.94)
q2	If squirrels cannot eat as much as possible, will not the squirrels gain weight? [a2*: More]	More (0.86)	More (0.94)	More (0.93)
qtrans	If the weather has a lot of snow, will not the squirrels gain weight? [a*trans*: More]	Less (0.75)	More (0.43)	More (0.87)
HotpotQA (comparison) Input	RoBERTa	DA
p	B. Reeves Eason is a film director, actor and screenwriter. Albert S. Rogell a film director.
q	Who has more scope of profession, B. Reeves Eason or Albert S. Rogell? [a*: B. Reeves Eason]	B. Reeves Eason	B. Reeves Eason
qsym	Who has less scope of profession, B. Reeves or Albert S. Rogell? [a*sym: Albert S. Rogell]	B. Reeves Eason	Albert S. Rogell
Table 4: Qualitative comparison of RoBERTa, + DA, + DA + Reg. The examples are partially modified.
cantly reduces the inconsistencies in predictions, demonstrating the effects of both data augmentation and regularization components. Notably on WIQA, RoBERTa shows violation of consistency in  $13.9\%$  of the symmetric examples and  $10.0\%$  of the transitive examples. Our approach reduces the violations of symmetric and transitive consistencies to  $8.3\%$  and  $2.5\%$ , respectively.
Results with limited training data Table 2 also shows that our approach is especially effective under the scarce training data setting: when only  $20\%$  of labeled data is available, our DA and Reg together gives more than  $12\%$  and  $14\%$  absolute accuracy improvements over the RoBERTa baselines on WIQA and QuaRel, respectively.
Ablation study We analyze the effectiveness of each component on Table 3. DA and Reg each improves the baselines, and the combination performs the best on WIQA and QuaRel. DA (standard) follows a previous standard data augmentation technique that paraphrases words (verbs and adjectives) using linguistic knowledge, namely WordNet (Miller, 1995), and does not incorporate logical rules. Importantly, DA (standard) does not give notable improvement over the baseline model both in accuracy and consistency, which suggests that logic-guided augmentation gives additional inductive bias for consistent QA beyond amplifying the number of train data. As WIQA consists of some transitive or symmetric examples, we also report the performance with Reg only on WIQA. The performance improvements is smaller, demonstrating the importance of combining with DA.
Qualitative Analysis Table 4 shows qualitative examples, comparing our method with RoBERTa baseline. Our qualitative analysis shows that DA+Reg reduces the confusion between opposite choices, and assigns larger probabilities to the
ground-truth labels for the questions where DA shows relatively small probability differences.
On HotpotQA, the baseline model shows large consistency violations as shown in Table 2. The HotpotQA example in Table 4 shows that RoBERTa selects the same answer to both  $q$  and  $q_{sym}$ , while DA answers correctly to both questions, demonstrating its robustness to surface variations. We hypothesize that the baseline model exploits statistical pattern, or dataset bias presented in questions and that our method reduces the model's tendency to exploit those spurious statistical patterns (He et al., 2019; Elkahky et al., 2018), which leads to large improvements in consistency.
We introduce a logic guided data augmentation and consistency-based regularization framework for accurate and globally consistent QA, especially under limited training data setting. Our approach significantly improves the state-of-the-art models across three substantially different QA datasets. Notably, our approach advances the state-of-the-art on QuaRel and WIQA, two standard benchmarks requiring rich logical and language understanding. We further show that our approach can effectively learn from extremely limited training data.
This research was supported by ONR N00014-18-1-2826, DARPA N66001-19-2-403, NSF (IIS1616112, IIS1252835), Allen Distinguished Investigator Award, Sloan Fellowship, and The Nakajima Foundation Fellowship. We thank Antoine Bosselut, Tim Dettmers, Rik Koncel-Kedziorski, Sewon Min, Keisuke Sakaguchi, David Wadden, Yizhong Wang, the members of UW NLP group and AI2, and the anonymous reviewers for their insightful feedback.