SlowGuess/ACL_2020_OCR huggingface dataset:
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. 2019. AutoAugment: Learning augmentation strategies from data. In CVPR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Ming Ding, Chang Zhou, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In ACL.
Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. 2019. Be consistent! improving procedural text comprehension using label consistency. In NAACL.
Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In EMNLP.
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11(Jul):2001-2049.
Regina Paxton Gazes, Nicholas W Chee, and Robert R Hampton. 2012. Cognitive mechanisms for transitive inference performance in rhesus monkeys: Measuring the influence of associative strength and inferred order. Journal of Experimental Psychology: Animal Behavior Processes, 38(4):331.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. VQA-LOL: Visual question answering under the lens of logic. arXiv:2002.08325.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias for natural language inference by fitting the residual. In EMNLP Workshop on DeepLo.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. 2018. AdvEntuRe: Adversarial training for textual entailment with knowledge-guided examples. In ACL.
Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. 1983. Optimization by simulated annealing. science.
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. Technical report, Citeseer.
Jay Yoon Lee, Sanket Vaibhav Mehta, Michael Wick, Jean-Baptiste Tristan, and Jaime Carbonell. 2019. Gradient-based inference for networks with output constraints. In AAAI.
Tao Li, Vivek Cupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In EMNLP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In ACL.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In ACL.
Pasquale Minervini and Sebastian Riedel. 2018. Adversarily regularising neural NLI models to integrate logical background knowledge. In ACL.
Arindam Mitra, Chitta Baral, Aurgho Bhattacharjee, and Ishan Shrivastava. 2019. A generate-validate approach to answering questions about qualitative relationships. arXiv:1908.03645.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv:1904.01038.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv:1904.08779.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In EMNLP.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. QuaRel: A dataset and models for answering questions about qualitative relationships. In AAAI.
Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. 2019. WIQA: A dataset for "what if..." reasoning over procedural text. In EMNLP.
5647
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv:1910.03771.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. 2019. Adversarial examples improve image recognition. arXiv:1911.09665.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In ICLR.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.
A Details of Human Annotations
In this section, we present the details of human annotations used for symmetric example creation (the (a) operation). We first sample the most frequent 500 verbs, 50 verb phrases and 500 adjectives from from the WIQA and QuaRel training data. Then, human annotators select words with some polarity (e.g., increase, earlier). Subsequently, they annotate the antonyms for each of the selected verbs and adjectives. Consequently, we create 64 antonym pairs mined from a comparison QA dataset. We reuse the same dictionary for all three datasets. Examples of annotated antonym pairs are shown in Table 5.
adjectives	verbs & verb phrases
more ↔ less slowly ↔ quickly stronger ↔ weaker later ↔ earlier younger ↔ older	increase ↔ decrease heat up ↔ cool down lose weight ↔ gain weight raise ↔ drop remove ↔ add
Table 5: Ten examples of annotated antonyms for comparison type questions.
B Details of answer re-labeling on WIQA and HotpotQA
We present the details of answer re-labeling operations in WIQA and HotpotQA, where the number of the answer candidates is more than two.
Answer re-labeling in WIQA (symmetric) In WIQA, each labeled answer  $a^*$  takes one of the following values: {more, less, no effects}. Although more and less are opposite, no effects is a neutral choice. In addition, in WIQA, a question  $q$  consists of a cause  $c$  and an effect  $e$ , and we can operate the three operations (a) replacement, (b) addition and (c) removal of words. When we add the operations to both of  $c$  and  $e$ , it would convert the question to opposite twice, and thus the original answer remains same. When we add one of the operation to either of  $c$  or  $e$ , it would convert the question once, and thus, the answer should be the opposite one. Given these two assumptions, we re-label answer as: (i) if we apply only one operation to either  $e$  or  $c$  and  $a^*$  is more or less, the  $a_{sym}^*$  will be the opposite of  $a^*$ , (ii) if we apply only one operation to either  $e$  or  $c$  and  $a^*$  is no effect, the  $a_{sym}^*$  will remain no effect, and (iii) if we apply one operation to each of  $e$  and  $c$ , the  $a_{sym}$  remains the same.
5648
Answer re-labeling in WIQA (transitive) For transitive examples, we re-label answers based on two assumptions on causal relationship. A transitive questions are created from two questions,  $X_{1} = (q_{1},p,a_{1}^{*})$  and  $X_{2} = (q_{2},p,a_{2}^{*})$ , where  $q_{1}$  and  $q_{2}$  consist of  $(c_{1},e_{1})$  and  $(c_{2},e_{2})$  and  $e_{1} = c_{2}$  holds. If  $a_{1}$  for  $X_{1}$  is "more", it means that the  $c_{1}$  causes  $e_{1}$ .  $e_{1}$  is equivalent to the cause for the second question  $(c_{2})$ , and  $a_{2}^{*}$  represents the causal relationship between  $c_{2}$  and  $e_{2}$ . Therefore, if  $a_{1}^{*}$  is a positive causal relationship,  $c_{1}$  and  $e_{2}$  have the relationship defined as  $a_{2}^{*}$ . We assume that if the  $a_{1}^{*}$  is "more",  $a_{3}^{*}(= a_{trans}^{*})$  will be same as  $a_{2}$ , and re-label answer following this assumption.
Answer re-labeling in HotpotQA In HotpotQA, answer candidates A are not given. Therefore, we extract possible answers from  $q$ . We extract two entities included in  $q$  by string matching with the titles of the paragraphs given by the dataset. If we find two entities to be compared and both of them are included in the gold paragraphs, we assume the two entities are possible answer candidates. The new answer  $a_{sym}^{*}$  will be determined as the one which is not the original answer  $a^{*}$ .
C Details of Baseline Models
We use RoBERTa (Li et al., 2019) as our baseline. Here, we present model details for each of the three different QA datasets.
Classification-based model for WIQA As the answer candidates for WIQA questions are set to {more, less, no effects}, we use a classification based models as studied for NLI tasks. The input for this model is [CLS]  $p$  [SEP]  $q$  [SEP]. We use the final hidden vector corresponding to the first input token ([CLS]) as the aggregate representation. We then predict the probabilities of an answer being a class  $C$  in the same manner as in (Devlin et al., 2019; Liu et al., 2019).
Multiple-choice QA model for QuaRel For QuaRel, two answer choices are given, and thus we formulate the task as multiple-choice QA. In the original dataset, all of the  $p$ ,  $q$  and  $\mathbf{A}$  are combined together (e.g., The fastest land animal on earth, a cheetah was having a 100m race against a rabbit. Which one won the race? (A) the cheetah (B) the rabbit), and thus we process the given combined questions into  $p$ ,  $q$  and  $\mathbf{A}$  (e.g., the question written above will be  $p = \text{The fastest land animal on earth}$ .
a cheetah was having a 100m race against a rabbit.  $q =$  Which one won the race? and  $\mathbf{A} =$  {the cheetah, rabbit}. Then the input will be [CLS]  $p$  [SEP] "Q: "  $q$  "A:"  $a_{i}$  [SEP], and we will use the final hidden vector corresponding to the first input token ([CLS]) as the aggregate representation. We then predict the probabilities of an answer being an answer choice  $a_{i}$  in the same manner as in (Liu et al., 2019).
Span QA model for HotpotQA We use the RoBERTa span QA model studied for SQuAD (Devlin et al., 2019; Liu et al., 2019) for HotpotQA. As we only consider the questions whose answers can be extracted from  $p$ , we do not add any modifications to the model unlike some previous studies in HotpotQA (Min et al., 2019b; Ding et al., 2019).
D Details of Implementations and Experiments
Implementations Our implementations are all based on PyTorch. In particular, to implement our classification based and span-based model, we use pytorch-transformers (Wolf et al., 2019) $^6$ . To implement our multiple choice model, we use fairseq (Ott et al., 2019) $^7$ .
Hyper-parameters For HotpotQA, we train a model for six epochs in total. For the model without data augmentation or regularization, we train on the original dataset for six epochs. For the models with data augmentation, we first train them on the original HotpotQA train data (including both bridge and comparison questions) for three epochs, and then train our model with augmented data and regularization for three epochs. For HotpotQA, we train our model with both bridge and comparison questions, and evaluate on comparison questions whose answers can be extracted from the context.
Due to the high variance of the performance in the early stages of the training for small datasets such as QuaRel or WIQA, for these two datasets, we set the maximum number of training epochs to 150 and 15, respectively. We terminate the training when we do not observe any performance improvements on the development set for 5 epochs for WIQA and 10 epochs for QuaRel, respectively. We use Adam as an optimizer  $(\epsilon = 1\mathrm{E} - 8)$  for
https://github.com/huggingface/ transformers
https://github.com/pytorch/fairseq
all of the datasets. Other hyper-parameters can be seen from Table 6
hyper-parameters	WIQA	QuaRel	HotpotQA
train batch size	4	16	12
gradient accumulation	16	1	1
max token length	256	512	384
doc stride	-	-	128
learning rate	2E-5	1E-5	5E-5
weight decay	0.01	0.01	0.0
dropout	0.1	0.1	0.1
warm up steps	0	150	0
τ for annealing	3	25	3
λsym	0.5	0.1	0.25
λtrans	0.05	-	-
Table 6: Ten examples of annotated antonyms for comparison type questions.
E Qualitative Examples on HotpotQA
As shown in Table 2, the state-of-the-art RoBERTa model produces a lot of consistency violations. Here, we present several examples where our competitive baseline model cannot answer correctly, while our RoBERTa+DA model answers correctly.
A question requiring world knowledge One comparison question asks "Who has more scope of profession, B. Reeves Eason or Albert S. Rogell", given context that B. Reeves is an American film director, actor and screenwriter and Albert S. Rogell is an American film director. The model correctly predicts "B. Reeves Eason" but fails to answer correctly to "Who has less scope of profession, B. Reeves Eason or Albert S. Rogell", although the two questions are semantically equivalent.
A question with negation We found that due to this reasoning pattern our model struggles on questions involving negation. Here we show one example. We create a question by adding a negation word,  $q_{sym}$ , "Which species is not native to asia, corokia or rhodotypos?", where we add negation word not and the paragraph corresponding to the question is  $p =$  "Corokia is a genus in the Argophyllaceae family comprising about ten species native to New Zealand and one native to Australia. Rhodotypos scandens is a deciduous shrub in the family Rosaceae and is native to China, possibly also Japan." The model predicts Rhodotypos scandens, while the model predicts the same answer to the original question  $q$ , "which species is native to asia, corokia or rhodotypos?" This example shows that the model strongly relies on
surface matching (i.e., "native to") to answer the question, without understanding the rich linguistic phenomena or having world knowledge.
5650