SlowGuess/NAACL_2022_OCR huggingface dataset:
B.1 Lack of the Gold Evidentiality Labels
Most datasets and tasks only include query-answer  $(x, \hat{y})$  annotations and do not include evidentiality labels  $\hat{\mathbf{E}}$  for passages  $\mathbf{P}$ . Some datasets with gold evidence annotation, such as Natural Questions, cover subsets of gold passages from certain Wikipedia articles, whereas  $\mathbf{P}$  possibly includes unlabeled gold passages from another article. Where gold annotations are not available, a common heuristic is to use the answer string as distant supervision (Mintz et al., 2009), labeling all passages that include the answer string as evidentiality positive. This heuristic can create false-positive annotations—for instance,  $p_2$  in Figure 2 includes the answer string "seven" but is irrelevant to the input query. Not only being noisy, this heuristic cannot be used for open-ended generation or answer classification such as knowledge-enhanced dialogue and fact verification.
B.2 Task-specific Details for Leave-one-out Generation
Open-domain QA. To collect new positive and negative data using leave-one-out generation, we consider top 20 passages retrieved for all of the original training data queries, and then split 20 passages into two ten-passage chunks. We then run a trained FiD model for 10 times, masking  $i$ th passage at the  $i$ th iteration. We consider  $i$ th passage  $p_i$  positive when and only when FiD fails to generate the correct answer when  $i$ th passage is masked. We also consider  $p_i$  (hard-)negative when and only when FiD succeeds to answer correctly when  $i$ th passage is masked, as we assume that the
Q: who played mary in christmas with the kranks
A: Felicity Huffman
Christmas with the Kranks: Christmas with the Kranks Christmas with the Kranks is a 2004 American Christmas comedy film based on the 2001 novel "Skipping Christmas" by John Grisham. It was directed by Joe Roth and written and produced by Chris Columbus. It stars Tim Allen and Jamie Lee Curtis as a couple who decide to skip Christmas one year since their daughter is away, much to the chagrin of their neighbors.
Q: hyori bed and breakfast season 2 air date
A:February 4,2018
Queen Sugar: On March 11, 2016, it was announced that Marycarmen Lopez also was cast as regular. On August 1, 2016, the series was renewed for a second season ahead of its television premiere which aired in a two-night premiere on June 20 and June 21, 2017. The second season premiered on OWN in a two episode special on June 20 and 21, 2017. The show was renewed for a third season on July 26, 2017. The third season premiered in a two-night special on May 29 and May 30, 2018. On August 8, 2018, the series.
Q: where does the last name waters come from
A: Wales and Yorkshire
Bywater (surname): Bywater (Surname) Bywater is an uncommon English surname of Anglo-Saxon origin and can most frequently be found in the English region of Yorkshire. It is a topographical surname given to those who were situated near a body of water. Bywater is an uncommon surname of Anglo-Saxon origin. The name derives from the merger of the Old English words "bi" and "waeter" to form "biwaeter". Topographical surnames are among the earliest created, because natural and artificial features in the.
Q: who was last person to be executed in us
A: Ruben Cardenas Ramirez
Billy Bailey: He became only the third person to be hanged in the United States since 1965 (the previous two were Charles Rodman Campbell and Westley Allan Dodd, both in Washington) and the first person hanged in Delaware in 50 years. As of 2018, he remains the last person to be executed by hanging in the United States.
Q: what is the largest ethnic group in mexico today
A: K'iche'
Mexican-American middle class: the Latino/a population of the United States is the nation's largest racial/ethnic minority group, constituting 17.6 percent of the total population. At two thirds of the Latino/a ethnic category, Mexicans are by far the largest national origin group.
Table 6: Examples of the top attended spurious passages in the questions where the base generator  $\mathcal{G}$  failed to generate the correct answers. The underlined phrases contradict the input queries, while those passages generally have high lexical overlap with the given input queries.
ith passage can be highly distracting or confusing, misleading the generator.
Fact verification. As fact verification is a classification task, using the same methodology as open-
domain QA may not be desirable—when we run a model ten times, it is likely to predict both correct and incorrect classes for multiple times, and we may not be able to mine the useful positive and negative passages. For the two fact verifica
Q: who is in charge of enforcing the pendleton act of 1883
A: United States Civil Service Commission
1. Pendleton Civil Service Reform Act: Pendleton Civil Service Reform Act The Pendleton Civil Service Reform Act (ch. 27, ) is a United States federal law enacted in 1883 that mandated that positions within the federal government should be awarded on the basis of merit.
2. United States Civil Service Commission: The Pendleton law was passed in part due to public outcry over the assassination of President Garfield.
3. Pendleton Civil Service Reform Act: The Act was written by Dorman Bridgman Eaton, a staunch opponent of the patronage system who was later first chairman of the United States Civil Service Commission.
Q: who plays skyler on lab rats elite force
A:Paris Berelc
1. Lab Rats: Elite Force: The series is a combined spinoff of "Lab Rats" and "Mighty Med" and stars William Brent, Bradley Steven Perry, Jake Short, Paris Berelc, and Kelli Berglund.
2. Lab Rats: Elite Force: Elite Force is an American comedy television series created by Chris Peterson and Bryan Moore that aired on Disney XD from March 2 to October 22, 2016. ... stars William Brent, Bradley Steven Perry, Jake Short, Paris Berelc, and Kelli Berglund.
3. Lab Rats: Elite Force: On September 3, 2015, it was announced that "Lab Rats" and "Mighty Med" would have a joint spinoff series called "Lab Rats: Elite Force". Only William Brent, formerly credited as Billy Unger, and Kelli Berglund from "Lab Rats" and Bradley Steven Perry, Jake Short, and Paris Berelc from "Mighty Med" were announced as returning for the new spinoff series.
Q: who developed the first periodic table with 8 columns
A: Dmitri Mendeleev
1. Periodic table: In 1923, Deming, an American chemist, published short (Mendeleev style) and medium (18-column) form periodic tables. Merck and Company prepared a handout form of Deming's 18-column medium table, in 1928, which was widely circulated in American schools.
2. History of the periodic table: their decision by saying that such “theoretical” topics might be controversial. The importance of Newlands’ analysis was eventually recognised by the Chemistry Society with a Gold Medal five years after they recognised Mendeleev’s work.
3. History of the periodic table: the work of Dmitri Mendeleev had been published. In 1864, the English chemist John Newlands classified the sixty-two known elements into eight groups, based on their physical properties. Newlands noted that many pairs of similar elements existed, which differed by some multiple of eight in mass number, and was the first to assign them an atomic number.
Table 7: Examples of the top three passages retrieved by a trained  $\mathcal{R}$  (DPR). We make the phrases matching the gold answers bold in the retrieved passages.
tion datasets, we consider the top 10 passages and we split them into two five-passage chunks. We consider the  $i$ th passage as a positive passage if the predictions based on the passage collections including  $i$ th passage unanimously agree on correct prediction whereas it fails to generate the correct class when  $i$ th passage is masked. We consider the  $i$ th passage as a negative passage when (i) the model succeeds to answer when and only when  $i$ th passage is masked, and (ii) the predictions unanimously agree on incorrect classes, which indicates
all of the passages do not support the input claim.
Knowledge-enhanced dialogue. Unlike open-domain QA or fact verification, the final output of a dialogue system can be highly open-ended. For dialogue, we compare the average F1 score of the generated responses when  $i$ th passage is included and masked. If the average F1 when  $p_i$  is presented is higher by more than 0.1 than the F1 when  $p_i$  is masked, we consider  $p_i$  provides useful evidence to generate the correct response, and therefore mark
Task: Fact Verification
Claim1: Jimmy perry had a cameo for the role of charlie cheeseman in dad's army.
Label1: SUPPORTS
Jimmy Perry: Despite the doubts, the first episode was screened on 31 July 1968, with Perry making a cameo appearance as the entertainer Charlie Cheeseman in the sixth episode, "Shooting Pains".
Claim2: John Glenn was a military test pilot.
Label2: SUPPORTS
John Glenn: Glenn's first flight test assignment, testing the FJ-3 Fury, nearly killed him when its cockpit depressurized and its oxygen system failed.
Task: Knowledge-enhanced Dialogue
Contexts1: Purple is such a good color.
Response1: yep, its in between red and blue
Purple: Purple is a color intermediate between blue and red. It is similar to violet, but unlike violet, which is a spectral color with its own wavelength on the visible spectrum of light, purple is a secondary color made by combining red and blue.
Contexts2: I was a really good skateboarder when i was young, its an action sport which involves riding and performing tricks, have you used a skateboard : : i tried wjhen i was younger but i failed horribly!haha : : hah, yes its really hard, first skateboards started with wooden boxes with wheels attached to the bottom, it was an invention from the people : : i think i would have done alot better on a box with wheels! lol thats so cool. when was the first one invented?
Response2: in the early 1900's it started, now there are 11.08 million active skateboarders in the world!
Electric skateboard: An electric skateboard is a personal transporter based on a skateboard. The speed is controlled by a hand-held throttle or weight-shifting and the direction of travel is adjusted by tilting the board to one side or the other. The MotoBoard, which was gasoline-powered was released in the summer of 1975, but were banned in California due to their noise and pollution in 1997. Louie Finkle of Seal Beach, California is often cited as an originator of the modern electric skateboard, offering his first wireless electric skateboard .
Table 8: Examples of the positive examples newly mined by leave-one-out generation approach. “::” in the contexts for the knowledge-enhanced dialogue example indicates the change of the speakers.
$p_i$  positive. On the contrary, when the average F1 when  $p_i$  is presented is lower by more than 0.1 than the score when  $p_i$  is masked, we believe  $p_i$  can be highly distracting, and thus we mark  $p_i$  negative. As in fact verification, we use the top 10 passages and split them into two five-passage chunks.
B.3 Implementation Details of Evidentiality Labeling Model
We use PyTorch (Paszke et al., 2019) via Hugging-Face transformers RoBERTA (Liu et al., 2019) implementation. We tune our model from RoBERTa-base. We optimize the objective func
tion using Adam (Kingma and Ba, 2015) with learning rate  $2 \times 10^{-5}$ . We lowercase the input and set the maximum sequence length to 350. We train the model for 7 epochs. Per GPU batch size is 12 and we use 8 GPUs with 24 GB memory.
Training data. We mine new training data for each task using our leave-one-out generation approach and mix the data with Natural Questions (Kwiatkowski et al., 2019). For Natural Questions data, as human annotators annotate long-answer, from which final minimal answers are extracted, we assume that those human-annotated long answers are evidentiality-positive passages, while the other passages included in
10 github.com/huggingface/transformers
Question (Q) & Answer (A)
Q: How many countries india shares borders with? A: seven
Q: who played ice queen in chronicles of narnia A: Tilda Swinton
Q: Season 2 this is us number of episodes A:15
Q: What is the first book of percy jackson A:The Lightning Thief
Evidentiality-Positive Passage
Borders of India
India shares land borders with seven sovereign nations ...
Tilda Swinton is a British actress. She is also known for her performance as the White Witch in the "Chronicles of Narnia series" (2005-10).
19-2 (2014 TV series)
The first season originally aired from January 29 to April 2, 2014, while the second season aired from January 19 to March 23, 2015
It is the second novel in the "Percy Jackson & the Olympians" series and the sequel to "The Lightning Thief".
Camp Half-Blood Chronicles
termed Book 8 in the Percy Jackson series by Amazon or the publisher. The British edition was published by Puffin Books in March as "Percy Jackson: The Ultimate Guide". "The Lightning Thief Graphic Novel" is an adaptation of "The Lightning Thief" into
Evidentiality-Negative Passage
India is the seventh-largest country by area, the second-most populous country.
The Chronicles of Narnia: Prince Caspian is a 2008 American high fantasy film ...William Moseley... Tilda Swinton reprise their roles from the first film
Quantico (season 2)
Quantico (season 2) The second season of American drama thriller series "Quantico" premiered on September 25, 2016, and concluded on May 15, 2017.
The Lightning Thief is the first book in the Percy Jackson and the Olympians series. It features Percy Jackson.
Figure 4: Examples of newly mined evidentiality examples for Natural Questions.
the same article and are not included in the long answers negative. We first collect all of the long-answer passages from Natural Questions training data, and randomly sample two negative passages per questions with long-answer annotations. We discard the examples where long answers are list or table elements. Consequently, we obtain 250k training samples, and we use  $90\%$  of the data as our training data and the remaining  $10\%$  of the data as our development set.
B.4 Examples of the Passages Mines by Leave-one-out Generation
Table 8 present several positive passages mined by leave-one-out generation approach. The positive passages for the open-domain QA and fact verification tasks clearly present the evidence leading to the gold answers (the highlighted sentences). Also in the first example of the knowledge-enhanced dialogue, the model finds a positive passage, which has high lexical overlap with the gold response. On the other hand, the second example shows the difficulty of finding the correct evidence for generation especially when the context history is long. The original dialogue history mentions skateboard and the last human utterance asks about when they were invented, while the passage labeled as positive is about electric skateboards and when they were released for the first time. We found due to
the open-ended nature of knowledge-enhanced dialogue and F1 score-based positive passage labeling can be results in more false positive passages than other two tasks, as even the passage does not really support the evidence, it still helps a model generate a loosely grounded and related response and obtains higher F1 score. Recent work reports similar issues in long-form QA evaluations (Krishna et al., 2021).
B.5 Examples of  $\mathbf{E}^{\mathrm{silver}}$  Obtained by  $\mathcal{M}$
The newly mined examples can be seen in Figure 4. Although all of the passages here include gold answer strings, we observe that the red passages do not entail the answers. For instance, in the second example, the red passage from "The Chronicles of Narnia: Prince Caspian" only lists the names of the actors who reprise their roles from the first film, and does not mention show played ice queen. The first passage, on the other hand, clearly mentions that Tilda Swinton plays the White Witch (the ice queen) in the Chronicles of Narnia. The third example shows that our model detects the case where we originally have distantly-positive passages, all of which are labeled as negative by our evidentiality labeling model. The fourth example shows that the positive passages can be retrieved from multiple different articles, which are often not covered by existing datasets with gold paragraph annotations.
C Details of the Datasets
License. Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) is under Apache License 2.0. The KILT benchmark (Petroni et al., 2021), where our FEVER and Wizard of Wikipedia data is taken, is under MIT License. FAVIQ (Park et al., 2021) does not explicitly mention the license. We use all of the datasets for their intended uses.
Privacy-related information and harmful context. All of the datasets use the English Wikipedia or web articles as a knowledge source and the input queries are authored by human annotators, and we believe those resources are less likely to include personal information or harmful context. In addition, dataset creators often conduct intensive analysis on annotated data and discard problematic examples, which may further reduce the risk of the problematic content.
D More Analysis and Examples
D.1 Details of Task-specific heuristics for an ablation of  $\mathbf{E}^{\text {silver }}$
For open-domain QA, this model uses answer string matching to supervise our multi-task learning. As discussed, this distantly supervised approach cannot be directly applied to classification or open-ended generation tasks. For WoW, it uses provenance title, which is the title of the Wikipedia article including the gold paragraph, and label all passages from provenance articles positive (Petroni et al., 2021). For FaVIQ-A, it uses the original answer annotations inherited from AmbigQA available in the dataset. It should be noted that that additional metadata is often unavailable in most of the datasets, and this variant for WoW and FaVIQ can be considered as a ground-truth setting.
D.2 Analyzing Attentions of  $\mathcal{G}$  and  $\mathcal{G}^+$
To further understand our method's behavior, we compare the attention scores assigned to the top retrieved passages of a base generator FiD  $(\mathcal{G})$  and our evidentiality-guided generator  $(\mathcal{G}^{+})$ . Figure 5 shows that the attention scores of the base generator  $\mathcal{G}$  and  $\mathcal{G}^{+}$ ; the x-axis is the attention values and the y-axis is probability of the histogram. The attention scores of the base generator  $\mathcal{G}$  are concentrated closely near the value of -5.0, whereas the attention scores of our  $\mathcal{G}^{+}$  more widely spread out. We also found that our  $\mathcal{G}^{+}$  more often gives
Figure 5: Attention score distributions over top 20 passages of the base generator  $\mathcal{G}$  and our evidentiality-guided generator  $\mathcal{G}^+$ .
its highest attention value to the passages ranked lower by  $\mathcal{R}$ ; our generator  $\mathcal{G}^+$  and base generator  $\mathcal{G}$  gives their highest attention scores to the passages ranked lower than top 10 by  $\mathcal{R}$  in  $45.8\%$  and  $44.8\%$  of the examples, respectively. We hypothesize that FiD mostly generates answers from more highly-ranked passages while our method enables shifting the attention scores to lower-ranked passages and generates answers from those, by explicitly training the models telling the evidentiality-negative and evidentiality-positive passages.
D.3 Examples from Qualitative Analysis on FaVIQ-A
Table 9 shows the most attended passages and final prediction results made by the base generator  $\mathcal{G}$  (FiD) and our evidentiality generator  $\mathcal{G}^+$  (ours) from our qualitative analysis on FaVIQ-Ambig.
Category 1 (40%): Our model attends a more relevant passage.
Claim: roger danuarta was the name of actress in munna michael as judge of dancing stars from jodhpur, rajasthan, india.A: REFUTES
[Ours (pred: REFUTES)] Munna Michael: as Judge of Dancing Star (cameo appearance)Chitrangada Singh as Judge of “Dancing Star” (cameo appearance) Pallavi Kulkarni (cameo appearance) Roger Danuarta (cameo appearance)
[FiD (pred: SUPPORTS)] Dancing with the Stars (American season 24): Dancing with the Stars (American season 24) The full list of celebrities and pros was announced on March 1, 2017, on Good Morning America. Hosts and judges. Tom Bergeron and Erin Andrews returned as hosts, and Carrie Ann Inaba, Len Goodman, Julianne Hough, and Bruno Tonioli returned as judges
Category 2 (10%): FiD attends a more relevant passage.
Claim: west was stacey's surname in gavin and stacey before marrying.A: SUPPORTS
[Ours (pred: REFUTES)] List of Gavin & Stacey characters: “Gavin & Stacey” is an award winning British television comedy series, following the lives of the title characters Gavin (Mathew Horne) and Stacey (Joanna Page), who, before marrying, live on opposite sides of the country, Gavin in Billericay, Essex, and Stacey in Barry, Vale of Glamorgan.
[FiD (pred: SUPPORTS)]Gavin & Stacey: Gavin & Stacey Other storylines that run throughout the course of the three series include Pams fake vegetarianism. Characters and cast.:Main characters. Gavin Shipman (Mathew Horne) – nicknamed “Gav”, “Gavlar”, or “Gavalar”, the funny and enthusiastic level-headed protagonist from Billericay, Essex. Stacey Shipman (“nee” West)
Category 3 (30%): Both are equally irrelevant.
Claim: sylvia fricker was the original singer of always on my mind.A: SUPPORTS
[Ours (pred: SUPPORTS)] For Once in My Life (Sylvia Syms album): ‘For Once in My Life (Sylvia Syms album) For Once in My Life is an album by American vocalist Sylvia Syms recorded in 1967 and released on the Prestige label.
[FiD (pred: REFUTES)]Follow Me...: Follow Me... The song “You Were on My Mind” was originally recorded and released in 1964 by Ian & Sylvia, and was a major hit in the US when covered by the group We Five in 1965.
Category 4 (20%): Both are equally relevant.
Claim: the third party system ended in american politics in 1854.A: REFUTES
[Ours (pred: REFUTES)]Political parties in the United States: The GOP dominated national politics during the Third Party System, from 1854 to 1896, and the Fourth Party System from 1896 to 1932.
[FiD (pred: SUPPORTS)] Third Party Syte: The Third Party System is a term of periodization used by historians and political scientists to describe the history of political parties in the United States from 1854 until the mid-1890s.
Table 9: Examples of the most attended passages and final prediction results made by the base generator  $\mathcal{G}$  (FiD) and our evidentiality generator  $\mathcal{G}^+$  (ours) from our qualitative analysis on FaVIQ-Ambig.