SlowGuess/ACL_2020_OCR huggingface dataset:
Relation Prediction In order to closely link relation prediction with evidence prediction, we reframe the task as a sequence labeling task. In other words, rather than make a single prediction given all evidence sentences, we make one prediction at every timestep during decoding to model the relation between the claim and all evidence retrieved to that point. This approach provides three benefits: it allows the model to better handle noise (when an incorrect evidence sentence is predicted), to handle multi-hop inference (to model the occurrence of switching from NEI to S/R), and to effectively provide more training data (for  $k = 5$  timesteps we have 5 times as many relation labels). For the claim in Figure 3, the initial label sequence is NEI and R because the first evidence sentence by itself (the fact that Barack Obama was born in Hawaii) would not refute the claim. Furthermore for  $k = 5$ , the remaining sequence would be R, R, R, as additional evidence (guaranteed to be non-contradictory in FEVER) would not change the prediction. On the other hand, given a claim that requires only a
single piece of evidence, such as that in Figure 1, the sequence would be R, R, R, R, R if the correct evidence sentence was selected at the first timestep, NEI, R, R, R, R if the correct evidence sentence was selected at the second timestep, and so forth.
We augment the evidence sentence selection described previously to use the hidden state of the pointer network after  $q$  hops (Equation 2) and an MLP to also predict a label at that time step, closely linking evidence and label prediction:
$$
P \left(l _ {t}\right) = \operatorname {s o f t m a x} \left(W _ {l 2} \operatorname {t a n h} \left(W _ {l 1} h _ {t} ^ {o}\right)\right) \tag {4}
$$
As with evidence prediction (Equation 3), when the gold label sequence is available, the loss term is:
$$
\mathcal {L} \left(\theta_ {\text {r e l - s e q}}\right) = - 1 / k \sum_ {t = 0} ^ {k - 1} \log P _ {\theta_ {\text {r e l - s e q}}} \left(l _ {t}\right) \tag {5}
$$
When training, at the current timestep we use both the gold evidence, i.e. "teacher forcing" (Williams and Zipser, 1989), and the model prediction from the previous step, so that we have training data for NEI. Combining Equations 3 and 5, our loss is:
$$
\mathcal {L} (\theta) = \lambda \mathcal {L} \left(\theta_ {p t r}\right) + \mathcal {L} \left(\theta_ {r e l. s e q}\right) \tag {6}
$$
Finally, to predict a relation at inference, we ensemble the sequence of predicted labels by averaging the probabilities over every time step.
Post-processing for Simple Temporal Reasoning As neural models are unreliable for handling numerical statements, we introduce a rule-based step to extract and reason about dates. We use the Open Information Extraction system of Stanovsky et al. (2018) to extract tuples. For example, given the claim "The Latvian Soviet Socialist Republic was a republic of the Soviet Union 3 years after 2009," the system would identify ARG0 as preceding the verb was and ARG1 following. After identifying tuples in claims and predicted sentences, we discard those lacking dates (e.g. ARG0). Given more than one candidate sentence, we select the one ranked higher by the pointer network. Once we have both the claim and evidence date-tuple we apply one of three rules to resolve the relation prediction based on the corresponding temporal phrase. We either evaluate whether the evidence
5The subset of timesteps was determined empirically: while at the final timestep the model is likely to have seen the correct evidence it also contains more noise; in future work we will experiment with alternatives.
8598
date is between two dates in the claim (e.g. between/during/in), we add/subtract  $x$  years from the date in the claim and compare to the evidence date (e.g.  $x$  years/days before/after), or compare the claim date directly to the evidence date (e.g. before/after/in). For the date expression "3 years after 2009," we compare the year 2012 to the date in the retrieved evidence (1991, the year the USSR dissolved) and label the claim as R.
We evaluate our dataset and system as part of the FEVER 2.0 shared task in order to validate the vulnerabilities introduced by our adversarial claims (Section 4) and the solutions proposed by our system (Section 5). We train our system on FV1-train and evaluate on FV1/FV2-dev/test (Section 3). We report accuracy (percentage of correct labels) and recall (whether the gold evidence is contained in selected evidence at  $k = 5$ ). We also report the FEVER score, the percentage of correct evidence sentences (for S and R) that also have correct labels, and potency, the inverse FEVER score (subtracted from one) for evaluating adversarial claims.
Our Baseline-RL: For baseline experiments, to compare different loss functions, we use the approach of Chakrabarty et al. (2018) for document selection and ranking, the reinforcement learning (RL) method of Chen and Bansal (2018) for sentence selection, and BERT (Devlin et al., 2019) for relation prediction. The RL approach using a pointer network is detailed by Chen and Bansal (2018) for extractive summarization, with the only difference that we use our fine-tuned BERT on claim/gold sentence pairs to represent each evidence sentence in the pointer network (as with our full system) and use the FEVER score as a reward. The reward is obtained by selecting sentences with the pointer network and then predicting the relation using an MLP (updated during training) and the concatenation of all claim/predicted sentence representations with their maximum/minimum pooling.
Hyper-parameters and settings for all experiments are detailed in Appendix B.
6.1 Adversarial Dataset Evaluation
We present the performance of our adversarial claims, obtained by submitting to the shared task server. We compare our claims to other participants in the FEVER 2.0 shared task (Table 2) and divided by attack type (Table 3). Potency was
macro-averaged across different fact-checking systems (Thorne and Vlachos, 2019), correctness of labels was verified by shared task annotators, and adjusted potency was calculated by the organizers as the potency of correct examples. Compared to other participants (Table 2), we presented a larger set of claims (501 in dev and 499 in test). We rank second in adjusted potency, but we provided a more diverse set than those created by the organizers or other participants. The organizers (Thorne and Vlachos, 2019) created adversarial claims using simple pattern-matching and replacement, e.g. quantifiers and negation. Niewinski et al. (2019) trained a GPT-2-based model on the FEVER data and manually filtered disfluent claims. Kim and Allan (2019) considered a variety of approaches, the majority of which required understanding area comparisons between different regions or understanding implications (e.g. that "not clear" implies NEI). While GPT-2 is effective, our approach is controllable and targeted at real-world challenges. Finally, Table 3 shows that when we select our top 200 most effective examples (multi-hop reasoning and multi-hop temporal reasoning) and compare to the approaches of Niewinski et al. (2019) and Kim and Allan (2019) (who both provided less than 204 examples total) our potency is much higher. In particular, multi-hop reasoning has a potency of  $88\%$  for SUPPORT relations and  $93\%$  for REFUTES relations and multi-hop temporal reasoning obtains  $98\%$  for SUPPORT and REFUTES relations.
Team	#	Pot.	Corr.	Adj.
Organizer Baseline	498	60.34	82.33	49.68
Kim and Allan (2019)	102	79.66	64.71	51.54
Ours	501	68.51	81.44	55.79
Niewinski et al. (2019)	79	79.97	84.81	66.83
Table 2: The evaluation of our claims relative to other participants. #: Examples in blind test Pot: Potency score Corr.: Percent grammatical and coherent with correct label and evidence Adj.: Adjusted potency
6.2 Evaluation against State-of-the-art
In Tables 4 and 5 we compare Our System (Section 5) to recent work from teams that submitted to the shared task server for FEVER 1.0 and 2.0, respectively, including the results of Our Baseline-RL system in Table 5. Our dual pointer network approach obtains state-of-the-art results on the FEVER 1.0 blind test set (Table 4) on all measures even over systems designed specifically for evidence retrieval (Nishida et al., 2019; Zhou et al.,
8599
Attack	M/A	#S/P	#R/P	#NEI/P
Conjunct.	A	-/-	54/55%	75/63%
Multi-hop	M	100/88%	88/93%	99/50%
Add. Unver.	M	-/-	-/-	50/50%
Date Man.	A	49/59%	129/80%	80/46%
Mul. Temp.	M	46/98%	5/98%	4/29%
Entity Dis.	M	46/50%	-/-	-/-
Lexical Sub.	A*	92/70%	57/70%	25/38%
2019), largely due to a notable improvement in recall (more than 3 points over the next system (Hanselowski et al., 2018)). We also find improvements in accuracy over the remaining pipeline systems, suggesting that joint learning helps. Compared to Our Baseline-RL, Our System has 1.8 point improvement in FEVER score on FV1-test with 4 points on FV2-test. Notably, our system finishes second (with a score of 36.61) on the FEVER 2.0 shared task test set, even though our claims were designed to be challenging for our model. The model of Malon (2018) performs especially well; they use a transformer-based architecture without pre-training but focus only on single-hop claims.
Table 3: Attack: Type of attack as described in Section 4. M/A: Whether claims are generated manually (M), automatically (A), or verified manually  $(\mathrm{A}^{*})$  #S: Support examples #R: Refute examples #NEI Not enough info examples P: Potency on Shared Task systems
System	Acc.	Rec.	FEVER
Hanselowski et al. (2018)	65.46	85.19	61.58
Nishida et al. (2019)	69.30	76.30	61.80
Yoneda et al. (2018)	67.62	82.84	62.52
Nie et al. (2019a)	68.16	71.51	64.21
Tokala et al. (2019)	69.98	77.28	66.72
Zhou et al. (2019)	71.60	-	67.10
Our System	72.47	88.39	68.80
Table 4: Comparison with state of the art on FV1-test
Team	FV1-test	FV2-test
Hanselowski et al. (2018)	61.58	25.35
Nie et al. (2019a)	64.21	30.47
Our Baseline-RL	67.08	32.92
Stammbach and Neumann (2019)	68.46	35.82
Yoneda et al. (2018)	62.52	35.83
Our System	68.80	36.61
Malon (2018)	57.36	37.31
6.3 System Component Ablation
To better understand the improved performance of our system, we present two ablation studies in
Tables 6 and 7 on FV1 and FV2 dev, respectively.
Table 6 presents the effect of using different objective functions for sentence selection and relation prediction, compared to joint sentence selection and relation prediction in our full model. We compare Our System to Our Baseline-RL system as well as another baseline  $(Ptr)$ . The  $Ptr$  system is the same as Our Baseline-RL, except the pointer network and MLP are not jointly trained with RL but independently using gold evidence and predicted evidence and relations, respectively. Finally, the Oracle upper bound presents the maximum possible recall after our document ranking stage, compared to  $94.4\%$  for Chakrabarty et al. (2018), and relation accuracy (given the MLP trained on 5 sentences guaranteed to contain gold evidence). We find that by incorporating the relation sequence loss, we improve the evidence recall significantly relative to the oracle upper-bound, reducing the relative error by  $50\%$  while also obtaining improvements on relation prediction, even over a strong RL baseline. Overall, the best model is able to retrieve  $95.9\%$  of the possible gold sentences after the document selection stage, suggesting that further improvements are more likely to come from document selection.
Table 5: Comparison of FEVER score to other shared-task systems (ordered by FV2-test FEVER score)
Model	Acc.	Rec.	FEVER
Oracle	84.2	94.7	-
Ptr	74.6	86.1	68.6
Our Baseline-RL	74.6	87.5	69.2
Our System	76.74	90.84	73.17
Table 6: Ablation experiments on FV1-dev
Table 7 evaluates the impact of the document pointer network and rule-based date handling on FV2-dev, as the impact of multi-hop reasoning and temporal relations is less visible on FV1-dev. We again compare Our Baseline-RL system to Our System and find an even larger 7.16 point improvement in FEVER score. We find that ablating the date post-processing (-dateProc) and both the date post-processing and document ranking components (-dateProc,-docRank) reduces the FEVER score by 1.45 and 3.5 points, respectively, with the latter largely resulting from a 5 point decrease in recall.
6.4 Ablation for Attack Types
While Table 3 presents the macro-average of all systems by attack type, we compare the performance of Our Baseline-RL and Our System in Table 8.
Our system is significantly better on all metrics  $(p < 0.001$  by the approximate randomization test).
System	Acc.	Rec.	FEVER
Our System	48.13	63.28	43.36
-dateProc	45.14	63.28	41.91
-dateProc,-docRank	44.29	58.32	39.86
Our Baseline-RL	44.04	57.56	36.2
Our System improves on evidence recall for multi-hop claims (indicating that a multi-hop document retrieval step may help) and those with ambiguous entities or words (using a model to re-rank may remove false matches with high lexical similarity). For example, the claim "Honeymoon is a major-label record by Elizabeth Woolridge Grant." requires multi-hop reasoning over entities. Our System correctly retrieves the pages [Lana Del Rey] and [Honeymoon (Lana Del Rey album)], but Our Baseline-RL is misled by the incorrect page [Honeymoon]. However, while recall increases on multi-hop claims compared to the baseline, accuracy decreases, suggesting the model may be learning a bias of the claim or label distribution instead of relations between claims and evidence.
We also obtain large improvements on date manipulation examples (here a rule-based approach is better than our neural one); in contrast, multi-hop temporal reasoning leaves room for improvement. For instance, for the claim "The MVP of the 1976 Canada Cup tournament was born before the tournament was first held," our full system correctly retrieves [Bobby Orr] and [1976 Canada Cup] (unlike the RL baseline). However, a further inference step is needed beyond our current capabilities – reasoning that Orr's birth year (1948) is before the first year of the tournament (1976).
Finally, we enhance performance on multipropositions as conjunctions or additional unverifiable information (indicating that relation sequence prediction helps). Claims (non-verifiable phrase in brackets) such as "Taran Killam is a [stage] actor" and "Home for the Holidays stars an actress [born in Georgia]." are incorrectly predicted by the baseline even though correct evidence is retrieved.
We showed weaknesses in approaches to fact-checking via novel adversarial claims. We took steps towards realistic fact-checking with targeted improvements to multi-hop reasoning (by a document pointer network and a pointer network for sequential joint sentence selection and relation pre
Table 7: Ablation experiments on FV2-dev
Attack Type	Acc.	Rec.	FEVER
Conjunction	B	16.95	92.0	16.95
S	40.68**	92.0	40.68**
Multi-hop	B	55.81*	29.07	19.77
S	33.72	45.35*	17.44
Add. Unver.	B	48.0	-	48.0
S	80.0**	-	80.0**
Date Manip.	B	30.99	79.59	27.46
S	53.52***	79.59	42.25**
Multi-hop Temp.	B	3.33	10.34	0.0
S	3.33	13.79	0.0
Entity Disamb.	B	70.83	62.5	58.33
S	79.17	79.17*	70.83
Lexical Sub.	B	33.33	65.71	25.0
S	29.76	75.71*	26.19
Table 8: Attack results for our FV2-dev claims. B: Our Baseline-RL, S: Our System. *:  $p < {0.05}$  **:  $p < {0.01}$  ***:  $p < {0.001}$  by approximate randomization test
diction), simple temporal reasoning (by rule-based date handling), and ambiguity and variation (by fine-tuned contextualized representations).
There are many unaddressed vulnerabilities that are relevant for fact-checking. The Facebook bAbI tasks (Weston et al., 2016) include other types of reasoning (e.g. positional or size-based). The DROP dataset (Dua et al., 2019) requires mathematical operations for question answering such as addition or counting. Propositions with causal relations (Hidey and McKeown, 2016), which are event-based rather than attribute-based as in FEVER, are also challenging. Finally, many verifiable claims are non-experiential (Park and Cardie, 2014), e.g. personal testimonies, which would require predicting whether a reported event was actually possible.
Finally, our system could be improved in many ways. Future work in multi-hop reasoning could represent the relation between consecutive pieces of evidence and future work in temporal reasoning could incorporate numerical operations with BERT (Andor et al., 2019). One limitation of our system is the pipeline nature, which may require addressing each type of attack individually as adversaries adjust their techniques. An end-to-end approach or a query reformulation step (re-writing claims to be similar to FEVER) might make the model more resilient as new attacks are introduced.
The authors thank Kathy McKeown, Chris Kedzie, Fei-Tzin Lee, and Emily Allaway for their helpful comments on the initial draft of this paper and the anonymous reviewers for insightful feedback.
8601
Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER), pages 85-90, Brussels, Belgium. Association for Computational Linguistics.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5947-5952, Hong Kong, China. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642. Association for Computational Linguistics.
Tuhin Chakrabarty, Tariq Alhindi, and Smaranda Muresan. 2018. Robust document retrieval and individual evidence modeling for fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 127-131, Brussels, Belgium. Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159.
Diane Francis. 2016. Fast furious fact check challenge.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650-655.
Lucas Graves. 2018. Understanding the promise and limits of automated fact-checking. Technical report, Reuters Institute, University of Oxford.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-athene: Multi-sentence textual entailment for claim verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 103-108, Brussels, Belgium. Association for Computational Linguistics.
Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pages 1803-1812, New York, NY, USA. ACM.
Christopher Hidey and Mona Diab. 2018. Team SWEEPer: Joint sentence extraction and fact checking with pointer networks. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 150-155, Brussels, Belgium. Association for Computational Linguistics.
Christopher Hidey and Kathy McKeown. 2016. Identifying causal relations using parallel Wikipedia articles. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1424-1433, Berlin, Germany. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages
8602