SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\operatorname {t f - i d f} (w, G) = N _ {w} (G) \cdot \log \left(\frac {N _ {g}}{N _ {g w}}\right),
$$
where  $N_{w}(G)$  is the number of occurrences of  $w$  in  $G$ ,  $N_{gw}$  is the number of documents in  $\mathcal{G}$  that contain  $w$ , and  $N_{g}$  is the number of documents in  $\mathcal{G}$ .
11See https://www.mediawiki.org/wiki/ Markup_spec for a discussion.
$^{12}$ https://www.tensorflow.org/datasets/catalog/wikipedia
Label	Description	%Edits	%Orig.
Counter-Vandalism	Revert or otherwise; remove vandalism	0.05	1.90
Fact-update	Update numbers, dates, scores, episodes, status, etc. based on newly available information	1.57	5.50
Copy-editing	Rephrase; improve grammar, spelling, tone, or punctuation	29.22	11.80
Wikification	Format text to meet style guidelines, e.g. add links or remove them where necessary	21.12	33.10
Vandalism	Deliberately attempt to damage the article	1.01	2.50
Simplification	Reduce the complexity or breadth of discussion; may remove information	3.13	1.60
Elaboration	Extend/add substantive new content; insert a fact or new meaningful assertion	9.50	12
Verifiability	Add/modify references/citations; remove unverified text	7.63	5.40
Process	Start/continue a wiki process workflow such as tagging an article with cleanup, merge or deletion no- notices	0.62	4.40
Clarification	Specify or explain an existing fact or meaning by example or discussion without adding new information	3.54	0.70
Disambiguation	Relink from a disambiguation page to a specific page	0.70	0.30
Point-of-view	Rewrite using encyclopedic, neutral tone; remove bias; apply due weight	0	0.30
Unlabeled	No label	21.39	1.20
Table 10: Breakdown of the edits in our data by intention label. The descriptions are taken from Yang et al. (2017). %Edits gives the prevalence of each label in our data, while %Orig. gives the prevalence in the hand-labelled dataset presented in Yang et al. (2017). The percentages do not total 100 because edits can have multiple labels.
Comment	Reword
Source	ByteDance responded by adding a kids-only mode to TikTok which allows music videos to be recorded, but not posted and by removing some accounts and content from those determined to be underage.
Target	ByteDance responded by adding a kids-only mode to TikTok which blocks the upload of videos, the building of user profiles, direct messaging, and commenting on other's videos, while still allowing the viewing and recording of content.
Comment	corrected tense for decedent
Source	While Bob Steward has not been an active producer since 1992, he serves as a Creative Consultant in his son's new production company, Steward Television, and is listed on the official website as Steward Television's founder.
Target	While Bob Steward was not an active producer since 1992, he served as a Creative Consultant in his son's new production company, Steward Television, and was listed on the official website as Steward Television's founder.
Comment	fixed spelling for Walter Yetnikoff
Source	Mottola was hired by Sony Music ( then known as CBS Records ) by its controversial President Walter Yentlkoff to run its U.S. operations.
Target	Mottola was hired by Sony Music ( then known as CBS Records ) by its controversial President Walter Yetnikoff to run its U.S. operations.
Table 11: More example edits from WikiDocEdits. The edited portions are highlighted in bold.
Geoffrey Everest Hinton CC FRS FRSC (born 6 December 1947) is an English Canadian cognitive psychologist and computer scientist, most noted for his work on artificial neural networks. Since 2013 he divides his time working for Google (Google Brain) and the University of Toronto. In 2017, he cofounded and became the Chief Scientific Advisor of the Vector Institute in Toronto. Geoffrey Hinton: index. Department of Computer Science: email: [REDACTED]: University of Toronto: voice: send email: 6 King's College Rd. We would like to show you a description here but the site won't allow us. Geoffrey's great grandfather, the mathematician [REDACTED] Charles Hinton, coined the word  $\backslash$  tesseract" and popularized the idea of higher dimensions, while his father, Howard Everest Hinton, was a distinguished entomologist. Geoffrey Hinton is a fellow of the Royal Society, the Royal Society of Canada, and the Association for the Advancement of Artificial Intelligence. He is an honorary foreign member of the American Academy of Arts and Sciences and the National Academy of Engineering, and a former president of the Cognitive Science Society. Geoffrey Hinton. Emeritus Prof. Comp Sci, U.Toronto & Engineering Fellow, Google. Verified email at cs.toronto.edu - Homepage. machine learning psychology artificial intelligence cognitive science computer science. Articles Cited by Co-authors. Title. Sort. Sort by citations Sort by year Sort by title. Geoff Hinton was born in Wimbledon in 1947 to Howard Hinton, an entomologist, and a schoolteacher mother, Margaret Clark. The childhood Hinton describes is a mash-up of Lemony Snicket, ... As the first of this interview series, I am delighted to present to you an interview with Geoffrey Hinton. Welcome Geoff, and thank you for doing this interview with deeplearning.ai.  $\rangle$  Thank you for inviting me.  $\rangle$  I think that at this point you more than anyone else on this planet has invented so many of the ideas behind deep learning. Talks by Geoffrey Hinton. The next generation of neural networks A 45min version of this talk which was given at the 10 year celebration of the Microsoft Cambridge Research Laboratory. the original powerpoint file version for most browsers.ps version with 4 slides per page. Very gentle after-dinner version of IJCAI-2005 Research Excellence ...
D Grounding Coverage Analysis
Table 12: Grounding used for the example in table 9. Parts indicated by [REDACTED] were removed for containing sensitive material.
Coverage corpus	Percentiles	Mean
25%	50%	75%
All Inputs	0.66	0.75	0.83	0.74
Grounding	0.50	0.62	0.73	0.61
Comment	0.22	0.31	0.46	0.36
Table 13:  $R_{\text{BERT}}$  statistics of inserted words for edits in WikiDocEdits. All statistics were computed on a  $1\%$  subsample of the data. The BERT embeddings used to compute  $R_{\text{BERT}}$  were produced using a pretrained BERT base model. Theidf weights were computed from a sample of 500,000 Wikipedia pages. Each row represents a different recall when considering a different coverage corpus  $\mathcal{C}$ .
We are also interested in knowing how well edits in the data are covered by the inputs (i.e.  $D, s, q$ , or  $\mathcal{G}$ ), where an edit is well covered if the information necessary to produce the edit appears somewhere in the inputs. To measure coverage we use word recall: how many words that were inserted in an edit also appear in the grounding? However, because simple recall fails to account for synonyms, or the context in which words appear, we use the BERTScore (Zhang et al., 2019a) recall. This allows for fuzzy matching between BERT embeddings instead of requiring exact word matches. We also useidf scores to weigh words, since we are mostly interested in covering rare words, which are more likely to be meaning-carrying. We can define the BERT recall,  $R_{\mathrm{BERT}}$ , for a sentence edit
Coverage corpus	Percentiles	Mean
25%	50%	75%
Full Model	0.54	0.66	0.75	0.64
Ablated Model	0.46	0.57	0.67	0.57
Table 14:  $R_{\text{BERT}}$  statistics of inserted words across test edits in WikiDocEdits. The BERT embeddings used to compute  $R_{\text{BERT}}$  were produced using a pretrained BERT base model. Theidf weights were computed from a sample of 500,000 Wikipedia pages. In all rows, the considered corpus  $\mathcal{C}$  corresponds to the grounding.
$s \to s'$ , with respect to some text corpus  $\mathcal{C}$  as
$$
\frac {\sum_ {w \in s ^ {\prime} \setminus s} \operatorname {i d f} (w) \cdot \max _ {w ^ {\prime} \in \mathcal {C}} \mathbf {B E R T} (w) ^ {T} \mathbf {B E R T} (w ^ {\prime})}{\sum_ {w \in s ^ {\prime} \setminus s} \operatorname {i d f} (w)},
$$
where  $s' \backslash s = \{w \in s' | w \notin s\}$ , and  $\operatorname{idf}(w)$  are the inverse document frequency scores computed on a random sample of 500K Wikipedia pages.
Table 13 reports the coverage statistics for our subsample of the data. We used an uncased BERT base model to compute the embeddings. The first row reports the coverage of the target by all of the inputs, namely the command, grounding, context, and source sentence. The second row shows the coverage by the grounding alone. Note that, even with just the grounding, coverage is already fairly high. Finally, the last row presents the coverage by the command alone, which shows that it also provides grounding.
E Additional Factuality Results
In addition to human evaluations, we also used automatic metrics to evaluate how faithful our model is to the grounding.
BERT Recall Similarly to the coverage analysis in appendix D, we can use  $R_{\mathrm{BERT}}$ , with the grounding as  $\mathcal{C}$ , to assess how well each word is inserted by the model is supported by the grounding. The only difference is that the model output now replaces the reference target  $s'$  in the formula for  $R_{\mathrm{BERT}}$ . Table 14 gives the summary statistics for  $R_{\mathrm{BERT}}$  across our test set, computed on the outputs of our full model, and the ablated model without grounding. Note that we only consider edits where the model makes at least one insertion. The ablated model serves as a baseline to compare the grounded model to. This baseline achieves a high  $R_{\mathrm{BERT}}$  score, likely because of spurious matches
with the grounding. Nevertheless, our grounded model is still more faithful to the grounding, as expected.
Grounding Usage While  $R_{\text{BERT}}$  attempts to measure how faithful the model is to the grounding (i.e. is the information inserted by the model found in the grounding?), we can also attempt to measure how much the grounding is used (i.e. how much of the information inserted by the model is only found in the grounding?). One simple approach is to look at how many words inserted by the model are found in the grounding but not in the rest of the inputs. While this isn't obvious to compute similarities between BERT embeddings, we can use exact word matches instead. For the model without grounding we find that in  $30.48\%$  of edits in the test set (with at least one insertion), at least one of the words inserted by the model is found in the grounding but not in the rest of the inputs. For the full model, this number increases to  $48.66\%$  as expected. The ablated model appears to insert words exclusive to the grounding in a high proportion of edits. However, this could be due to fluency edits, where the model might insert a functional word that happens to only appear in the grounding. If we restrict our attention to content edits, as defined in section 3.2, the ablated model inserts grounding-exclusive words in only  $36.85\%$  of edits, and  $65.40\%$  for the full model.