SlowGuess/ACL_2020_OCR huggingface dataset:
2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Shan Jiang and Christo Wilson. 2018. Linguistic signals under misinformation and fact-checking: Evidence from user comments on social media. Proc. ACM Hum.-Comput. Interact., 2(CSCW):82:1-82:23.
Youngwoo Kim and James Allan. 2019. FEVER breaker's run of team NbAuzDrLqg. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 99-104, Hong Kong, China. Association for Computational Linguistics.
Christopher Malon. 2018. Team papelo: Transformer networks at FEVER. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 109-113, Brussels, Belgium. Association for Computational Linguistics.
Tsvetomila Mihaylova, Georgi Karadzhov, Pepa Atanasova, Remy Baly, Mitra Mohtarami, and Preslav Nakov. 2019. Semeval-2019 task 8: Fact checking in community question answering forums. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 860-869.
Tsvetomila Mihaylova, Preslav Nakov, Lluís Márquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadzhov, and James R. Glass. 2018. Fact checking in community forums. CoRR, abs/1803.03178.
Paramita Mirza and Sara Tonelli. 2016. CATENA: causal and temporal relation extraction from natural language texts. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages 64-75.
Ndapandula Nakashole and Tom M. Mitchell. 2014. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1009-1019, Baltimore, Maryland. Association for Computational Linguistics.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019a. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859-6866.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019b. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
Piotr Niewinski, Maria Pszona, and Maria Janicka. 2019. GEM: Generative enhanced model for adversarial attacks. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 20-26, Hong Kong, China. Association for Computational Linguistics.
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2335-2345, Florence, Italy. Association for Computational Linguistics.
Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29-38, Baltimore, Maryland. Association for Computational Linguistics.
Dean Pomerleau and Delip Rao. 2017. Fake news challenge.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931-2937, Copenhagen, Denmark. Association for Computational Linguistics.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085-1097, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865.
Dominik Stammbach and Guenter Neumann. 2019. Team DOMLIN: Exploiting evidence enhancement for the FEVER shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 105-109, Hong Kong, China. Association for Computational Linguistics.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885-895.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440-2448. Curran Associates, Inc.
8603
James Thorne and Andreas Vlachos. 2017. An extensible framework for verification of numerical claims. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 37-40, Valencia, Spain. Association for Computational Linguistics.
James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346-3359. Association for Computational Linguistics.
James Thorne and Andreas Vlachos. 2019. Adversarial attacks against fact extraction and verification. arXiv preprint arXiv:1903.05543.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819. Association for Computational Linguistics.
Santosh Tokala, G Vishal, Avirup Saha, and Niloy Ganguly. 2019. Attentivechecker: A bi-directional attention flow mechanism for fact verification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2218-2222.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In International Conference on Learning Representations (ICLR).
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18-22. Association for Computational Linguistics.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146-1151.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153-2162, Hong Kong, China. Association for Computational Linguistics.
William Yang Wang. 2017. "Liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. *Neural Comput.*, 1(2):270-280.
Wenpeng Yin and Dan Roth. 2018. TwoWingOS: A two-wing optimization strategy for evidential claim verification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 105-114, Brussels, Belgium. Association for Computational Linguistics.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102, Brussels, Belgium. Association for Computational Linguistics.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
A Examples of Attack Types
Table 9 displays examples for each type of attack. The multi-propositional examples include attacks for CONJUNCTION, MULTI-HOP REASONING, and ADDITIONAL UNVERIFIABLE PROPOSITIONS. For temporal reasoning, we provide examples for DATE MANIPULATION and MULTI-HOP TEMPORAL REASONING. The lexical variation examples consist of ENTITY DISAMBIGUATION and LEXICAL SUBSTITUTION.
8604
Attack Type	Example Claim	Label	Evidence
Conjunction	Blue Jasmine has Sally Hawkins acting in it and Blue Jasmine was filmed in San Francisco.	NEI	N/A
Multi-Hop Rea-soning	Goosebumps was directed by Rob-Letterman the person who co-wrote Shark Tale.	S	[Goosebumps (film)] It was directed by Rob Letterman, and written by Darren Lemke, based from a story by Scott Alexander and Larry Karaszewski. [Rob Letterman] Before Letterman's film subjects took him into outer space with Monsters vs. Aliens (2009), he was taken underwater, having co-directed and co-written Shark Tale.
Additional Unverifiable Propositions	Roswell is an American TV series with 61 episodes.	NEI	N/A
Date Manipula-tion	Artpop was Gaga's second consecutive number-one record in the United States in 2009 before 2010.	R	[Artpop] Gaga began planning the project in 2011, shortly after the launch of her second studio album, Born This Way.
Multi-Hop Temporal Reasoning	Lisa Murkowski's father re-signed from the Senate after serving as Senator.	S	[Lisa Murkowski] She is the daughter of former U.S. Senator and Governor of Alaska Frank Murkowski. Murkowski was appointed to the U.S. Senate by her father, Frank Murkowski, who resigned his seat in December 2002 to become the Governor of Alaska. [Frank Murkowski] He was a United States Senator from Alaska from 1981 until 2002 and the eighth Governor of Alaska from 2002 until 2006.
Entity Disam-biguation	Kate Hudson is a left wing political activist	S	[Kate Hudson (activist)] Katharine Jane “Kate” Hudson (born 1958) is a British left wing political activist and academic who is the General Secretary of the Campaign for Nuclear Disarmament (CND) and National Secretary of Left Unity.
Lexical Substi-tution	The Last Song began filming shooting on Monday June 14th 2009.	R	[The Last Song (film)] Filming lasted from June 15 to August 18, 2009 with much of it occurring on the islands beach and pier.
Table 9: Examples of the seven sub-types of attacks. Claims edited with word substitution or insertion have their changes in bold. Deletions are marked in strikethrough. Wikipedia titles are represented in bold with square brackets. S: SUPPORTS R: REFUTES NEI: NOT ENOUGH INFORMATION
8605
B Hyper-parameters and Experimental Settings
We select  $M = 30$  Wikipedia articles using TF-IDF when combining with our other candidate document selection methods and select  $D = 5$  after document ranking. We select  $N = 5$  sentences during sentence selection, consistent with the shared task evaluation.
B.1 BERT Language Model Fine-Tuning
We use version 0.5.0 of the Huggingface library (https://github.com/huggingface/pytorch-pretrained-BERT) to fine-tune the "BERT-base" model using the default settings. We lowercase all tokens and use the default BERT tokenizer.
Document Ranking Our dataset of title and claim pairs (obtained from FV1-train) consists of 140,085 positive examples and 630,265 negative examples in training with approximately  $10\%$  set aside for validation (16,016 positive examples and 84,437 negative). As recommended by Devlin et al. (2019), we select hyper-parameters by grid search over 16 and 32 for batch size, 2e-5, 3e-5, and 5e-5 for learning rate, and 3 and 4 for the number of epochs.
Sentence Selection Our dataset of sentence and claim pairs (also obtained from FV1-train) consists of 54,431 S relations, 54,592 R relations, and 54,501 NEI relations in training, with approximately  $10\%$  set aside for validation (6,139 S relations, 5,984 R relations, and 6,050 NEI relations). We again select hyper-parameters consistent with the recommended best practice.
B.2 Pointer Network
We train both the document ranking and sentence selection pointer networks on FV1-train with the same hyper-parameters using Adagrad (Duchi et al., 2011) with a learning rate of 0.01, a batch size of 16, and a maximum of 140 epochs with early stopping on FV1-dev. The dimension of the pointer network LSTM hidden state is set to 200 with  $q = 3$  hops over the memory. We use a beam width of 5 during inference. The MLP used to predict relations has a hidden layer dimensionality of 200 and we set  $\lambda = 1$ .
B.3 Reinforcement Learning
To make the sentence extractor an RL agent, we can formulate a Markov Decision Process (MDP): at each extraction step  $t$ , given a claim  $c$ , the agent observes the current state and samples an action from Equation 3 to extract a document sentence  $s$ , predict the relation label  $l$  and receive a reward  $r(t + 1) = \text{FEVER}(c, s, l)$ . We train using REINFORCE, adapted with an Actor-Critic to minimize variance (detailed by Chen and Bansal (2018)). As RL often requires pre-training, we combine the pointer network loss from Equation 3 with the RL loss  $(\mathcal{L}(\theta_{rl}))$  and the relation prediction loss  $(\mathcal{L}(\theta_{rel}))$ :
$$
\mathcal {L} (\theta) = \lambda_ {1} \mathcal {L} \left(\theta_ {p t r}\right) + \lambda_ {2} \mathcal {L} \left(\theta_ {r l}\right) + \mathcal {L} \left(\theta_ {r e l}\right) \tag {7}
$$
We set both  $\lambda_1 = 1$  and  $\lambda_{2} = 1$ .
8606