SlowGuess/ACL_2020_OCR huggingface dataset:
William Yang Wang. 2017. "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426.
Brian Xu, Mitra Mohtarami, and James R. Glass. 2018. Adversarial Domain Adaptation for Stance Detection. In Proceedings of the Thirty-second Annual Conference on Neural Information Processing Systems (NeurIPS)-Continual Learning.
Fan Yang, Shiva K Pentyala, Sina Mohseni, Mengnan Du, Hao Yuan, Rhema Linder, Eric D Ragan, Shuiwang Ji, and Xia Hu. 2019. XFake: Explainable Fake News Detector with Visualizations. In The World Wide Web Conference, pages 3600-3604.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102, Brussels, Belgium. Association for Computational Linguistics.
7362
A.1 Comparison of different sources of evidence
Table 7 provides an overview of the ruling comments and the ruling oracles compared to the justification. The high recall in both ROUGE-1 and ROUGE-F achieved by the ruling comments indicates that there is a substantial coverage, i.e. over  $70\%$  of the words and long sequences in the justification can be found in the ruling comments. On the other hand, there is a small coverage for the bigrams. Selecting the oracles from all of the ruling sentences increases ROUGE-F1 scores mainly by improving the precision.
A.2 Extractive Gold Oracle Examples
Table 8 presents examples of selected oracles that serve as gold labels during training the extractive summarization model. The three examples represent oracles with different degrees of matching the gold summary. The first row presents an oracle that matches the gold summary with a ROUGE-L F1 score of 60.40 compared to the gold summary. It contains all of the important information from the gold summary and even points precise, not rounded, numbers. The next example has a ROUGE-L F1 score of 43.33, which is close to the average ROUGE-L F1 score for the oracles. The oracle again conveys the main points from the gold justification, thus, being sufficient for the claim's explanation. Finally, the third example is of an oracle with a ROUGE-L F1 score of 25.59. The selected oracle sentences still succeed in presenting the main points from the gold justification, which is at a more detailed level presenting specific findings. The latter might be found as a positive consequence as it presents the particular findings of the journalist that led to selecting the veracity label.
A.3 Manual 6-way Veracity Prediction from explanations
The Fleiss'  $\kappa$  agreement for the 6-label manual annotations is: 0.20 on the Just explanations, 0.230 on the Explain-MT explanations, and 0.333 on the Explain-Extr system. Table 9 represent the results of the manual veracity prediction with six classes.
7363
Evidence Source	ROUGE-1	ROUGE-2	ROUGE-L
P	R	F1	P	R	F1	P	R	F1
Ruling	8.65	78.65	14.84	3.53	33.76	6.16	8.10	74.14	13.92
Ruling Oracle	43.97	49.24	43.79	22.45	24.50	22.03	39.70	44.10	39.37
Table 7: Comparison of sources of evidence - Ruling Comments and Ruling Oracles comapred to the target justification summary.
Claim: “The president promised that if he spent money on a stimulus program that unemployment would go to 5.7 percent or 6 percent. Those were his words.”
Label: Mostly-False
Just: Bramnick said “the president promised that if he spent money on a stimulus program that unemployment would go to 5.7 percent or 6 percent. Those were his words.” Two economic advisers estimated in a 2009 report that with the stimulus plan, the unemployment rate would peak near 8 percent before dropping to less than 6 percent by now. Those are critical details Bramnick's statement ignores. To comment on this ruling, go to NJ.com.
Oracle: “The president promised that if he spent money on a stimulus program that unemployment would go to 5.7 percent or 6 percent. Those were his words,”
Bramnick said in a Sept. 7 interview on NJToday. But with the stimulus plan, the report projected the nation's jobless rate would peak near 8 percent in 2009 before falling to about 5.5 percent by now. So the estimates in the report were wrong.
Claim: The Milwaukee County bus system has “among the highest fares in the nation.”
Label: False
Just: Larson said the Milwaukee County bus system has “among the highest fares in the nation.” But the system’s $2.25 cash fare wasn’t at the top of a national comparison, with fares reaching as high as $4 per trip. And regular patrons who use a Smart Card are charged just $1.75 a ride, making the Milwaukee County bus system about on par with average costs.
Oracle: Larson said the Milwaukee County bus system has “among the highest fares in the nation.” Patrons who get a Smart Card pay $1.75 per ride. At the time, nine cities on that list charged more than Milwaukee’s $2.25 cash fare. The highest fare – in Nashville – was $4 per ride.
Claim: “The Republican who was just elected governor of the great state of Florida paid his campaign staffers, not with money, but with American Express gift cards.”
Label: Half-True
Just: First, we think many people might think Maddow was referring to all campaign workers, but traditional campaign staffers – the people working day in and day out on the campaign – were paid by check, like any normal job. A Republican Party official said it was simply an easier, more efficient and quicker way to pay people. And second, it’s not that unusual. In 2008, Obama did the same thing.
Oracle: “It’s a simpler and quicker way of compensating short-term help.” Neither Conston nor Burgess said how many temporary campaign workers were paid in gift cards. When asked how he was paid, Palecheck said: “Paid by check, like any normal employee there.” In fact, President Barack Obama’s campaign did the same thing in 2008.
Table 8: Examples of the extracted oracle summaries (Oracle) compared to the gold justification (Just).
Just	Explain-Extr	Explain-MT
Agree-C	0.208	0.138	0.163
Agree-NS	0.065	0.250	0.188
Agree-NC	0.052	0.100	0.075
Disagree	0.675	0.513	0.575
Table 9: Manual classification of veracity label - true, false, half-true, barely-true, mostly-true, pants-on-fire, given a particular explanations from the gold justification (Just), the generated explanation (Explain-Extr) and the explanation learned jointly with the veracity prediction model (Explain-MT). Presented are percentages of the dis/agreeing annotator predictions, where the agreement percentages are split to: correct according to the gold label (Agree-C), incorrect (Agree-NC) or with not sufficient information (Agree-NS). The first column indicates whether higher ( $\searrow$ ) or lower ( $\searrow$ ) values are better. At each row, the best set of explanations is in bold and the best automatic explanations are in blue.
7364