SlowGuess/NAACL_2022_OCR huggingface dataset:
- Ubuntu 18
Pytorch 1.7
- Transformers 4.3.2
Anserini 0.4.1 (commit 3a60106fdc83473d147218d78ae7dca7c3b6d47c)
Number of parameters  $\mathrm{Re}^2\mathrm{G}$  uses three BERT<sub>BASE</sub> transformers: query encoder, passage encoder and reranker. Each has  $110M$  parameters. The generation component is a BART<sub>LARGE</sub> model with  $400M$  parameters. There are  $730M$  parameters in total.
Computing infrastructure Using a single NVIDIA V100 GPU DPR training of two epochs takes approximately 24 hours for T-REx and less than 12 hours for FEVER and WoW.
Using a two NVIDIA P100 GPUs generation training for 370k T-REx instances takes two days, while FEVER and WoW training completes in half a day.
The FAISS index on the KILT knowledge source requires a machine with large memory, we use machines with 128GB of memory.
Hyperparameter	DPR	Reranker	Generation
learn rate	5e-5	3e-5	3e-5
batch size	128	32	128
epochs	2	1	1*
warmup instances	0	10%	10%
learning schedule	linear	triangular	triangular
max grad norm	1	1	1
weight decay	0	0	0
Adam epsilon	1e-8	1e-8	1e-8
D Generation Analysis
We examined 20 instances coupled with 3 output texts: the baseline  $\mathrm{KGI}_0$ ,  $\mathrm{Re}^2\mathrm{G}$ , and the target text in the ground-truth. The three output texts were presented unlabeled and in random order to avoid bias. For each instance, we read the conversation history and then mark each text either GOOD, OK or INCONSISTENT generation. To our surprise, 5/20 ground-truth target texts are INCONSISTENT which indicates the WoW benchmark might have limitations in annotation quality. Both the systems have similar results (GOOD/OK/INCONSISTENT -  $\mathrm{Re}^2\mathrm{G}$ : 8/2/10;  $\mathrm{KGI}_0$ : 9/2/9).
Second, we checked a set of 20 WoW instances where  $\mathrm{Re}^2\mathrm{G}$  's F1 score was in the bottom quintile. The conversation history was presented along with  $\mathrm{Re}^2\mathrm{G}$  generated text and the passages re
Table 4:  ${\mathrm{{Re}}}^{2}\mathrm{G}$  hyperparameters
Hyperparameter	Value
type	IndexHNSWSQ
m	128
ef search	128
ef construction	200
index batch size	100000
scalar quantizer	8
Table 5: FAISS index hyperparameters
Hyperparameter	Value
DPR passages	12
BM25 passages	12
BART sequences	5
BART beam size	6
BART length penalty	1.0
BART minimum length	2
BART maximum length	64
Table 6: Inference hyperparameters
trieved. Manual examination showed 8/20 as INCONSISTENT and in 4/8 cases supporting ground-truth passages were not retrieved. Below is one of the 12/20 cases where  $\mathrm{Re}^2\mathrm{G}$  generated text was found CONSistent with respect to the conversation history, although it has low F1 and Rouge-L scores.
Conversation History:
- My favorite color is red.
- Red is at the end of the spectrum of light, its with orange and opposite of violet.
- I didn't know that. What else do you know about red?
Target: It's actually a primary color for the RGB and CMYK color model.
$\mathbf{Re}^2\mathbf{G}$ : It has a dominant wavelength of approximately 625-740 nanometres.
D.1 Generation Quality
Table 7 shows couple of examples that were part of the set of randomly selected instances from WoW dataset and used for manual inspection. We choose these two particular instances to show when we thought the ground truth (i.e. target) is not coherent with respect to the corresponding conversation history.
In the first example, the system generated outputs were judged as coherent. We found that both  $\mathrm{Re}^2\mathrm{G}$  and  $\mathrm{KGI_0}$  retrieved the following passage which might have helped generation of the above output -
Horseshoe Falls / Horseshoe Falls Horseshoe Falls, also known as Canadian Falls, is the largest of the three waterfalls that collectively form Niagara Falls on the Niagara River along the Canada-United States border. Approximately  $90\%$  of the Niagara River, after diversions for hydropower generation,
flows over Horseshoe Falls. The remaining  $10\%$  flows over American Falls and Bridal Veil Falls. It is located between Terrapin Point on Goat Island in the US state of New York, and Table Rock in the Canadian province of Ontario. Section: International border.
As for the ground truth, we marked it (factually) inconsistent based on the following retrieved passage -
Niagara Falls / Located on the Niagara River, which drains Lake Erie into Lake Ontario, the combined falls have the highest flow rate of any waterfall in North America that has a vertical drop of more than. During peak daytime tourist hours, more than 168,000 m (six million cubic feet) of water goes over the crest of the falls every minute. Horseshoe Falls is the most powerful waterfall in North America, as measured by flow rate.
In the second example, all three texts were marked inconsistent. Interestingly, all the items in the conversation contains subjective opinion. Consequently, all the three candidate texts also contains subjective opinion. The problem is both the systems generated texts that are almost repetition of earlier conversation. In case of the ground truth, we find that the text is semantically incoherent.
We have also submitted files that contain all instances that were used to generate the different analysis reported in Section 4.2 of the paper. These files also contains our annotations/remarks where applicable.
- Have you ever been to Niagara Falls? I want to go there, but I know it's very far. Those beautiful 3 waterfalls are right between New York and Ontario, thousands of miles from me!
- I never have but I would love to. It looks beautiful.
- Yes, they do! They are different sized waterfalls, named Horseshoe, American and Bridal Veil Falls.
- I had no idea. I've never heard that before. Very cool.
- Yes, and Horseshoe Falls in actually the most powerful waterfall in all of the US!
- I never new any details about it. I have just seen it in pictures and on TV and it looks very neat.
$\mathbf{Re}^{2}\mathbf{G}$ : Horseshoe falls, also known as Canadian Falls, is the largest of the three waterfalls that collectively form Niagara Falls on the Niagara River along with American Falls and Bridal Veil Falls.
[GOOD]
$\mathbf{KG}\mathbf{I}_0$ : Horseshoe falls is also known as Canadian Falls, which straddles the international border between Canada and the U.S.
Ground truth: It's really quite interesting and beautiful. There's over six million cubic feet of water that crosses the fall's crest every 60 seconds. Can you imagine?
[INCONSistent - 6M cubic feet of water crosses for the 3 waterfalls combined, not just Horseshoe falls.]
- I love the taste of pasta mixed with tomato sauce and shredded cheese all over it
- I just love mines with chunky tomatoes mixed with meatballs and mushrooms.
- Olive Garden is the place that makes the best pastas. I try to visit the place as much I can with my friends
- I love Olive Garden especially the original one in Orlando they opened in 1982.
- I think Italians have special genes that makes them so adept at preparing pastas
- I believe so as well I would love to visit Argentina that is where Italian-born people reside great pasta I bet.
- I don't know how they do it, but no matter how hard I try to prepare one at home, mine ends up with some peculiar taste
$\mathbf{Re}^2\mathbf{G}$ : I don't know how they do it but I love the taste of pasta mixed with tomato sauce and shredded cheese all over it
[INCONSISTENT - Dialogue repetition]
$\mathbf{KG}\mathbf{I}_0$  : I just love the taste of pasta mixed with tomato sauce and shredded cheese all over it [INCONSISTENT - Dialogue repetition]
Ground truth: The taste is so full of an abundance of taste it just can't be copied lol.
[INCONSISTENT - Semantically incoherent text "taste is full of .. taste".]
Table 7: Some of the randomly selected instances from WoW benchmark that were evaluated by a human evaluator. The comments inside  $\left[\ldots\right]$  are the feedback provided by the evaluator.