SlowGuess/NAACL_2021_OCR huggingface dataset:
Base Model We use the GPT2-medium model (24-layer; 345M params) pretrained on the English WebText dataset (Radford et al., 2019), as implemented in HuggingFace's TransferTransfo (Wolf et al., 2019b,a) framework. Fine-tuning is performed using the language modelling objective on the training set with default hyperparameters until lowest perplexity is reached on the validation set. During generation, we sample tokens using nucleus sampling (Holtzman et al., 2020) with  $p = 0.9$  and temperature  $\tau = 0.9$  and get candidate responses. To compute auxiliary probabilities  $\{p(\mathbf{g}|\mathbf{h}), p(\mathbf{g}|\mathbf{k}), p(\mathbf{g})\}$  for these candidates, we use separate ablation models. The ablation models are trained similar to the base model but after removing respective contexts from the training inputs.
4.1 Experimental Design
To validate our proposed methods, we do a paired comparison (on Amazon Mechanical Turk) where human annotators are shown two prior turns of conversational history and asked to choose between two candidate responses. Annotators are allowed to mark both candidates as nonsensical if the responses don't make sense. In Section A.3, we show the interfaces used to collect annotations from Amazon Mechanical Turk. Each pair of responses was compared by three annotators – we consider a candidate to be better than the other when at least two of them (majority) agree upon it. For each of the following three experiments, we compare 100 pairs of candidates generated using instances from the test set. The null hypothesis  $(H_0)$  for the three experiments is that there is no difference between the methods used to generate the candidates and we hope to reject the null hypothesis in favor of the alternate hypothesis  $(H_1)$  at a significance level  $(\alpha)$  of 0.05.
Exp 1: PMI and overall quality First, we want to confirm that high PMI responses are overall better quality than randomly chosen candidates  $(H_{1})$ . To do so, we first generate 10 responses for each instance and compare the response having maximum  $\mathrm{pmi}(\mathbf{g};\mathbf{h},\mathbf{k})$  (Max-PMI) with a randomly chosen response from the remaining 9. We ask human annotators to pick the overall better candidate response.
Exp 2:  $\mathsf{pcmi_h}$  and acknowledgement We test if responses having high  $\mathsf{pcmi}_h$  provide better acknowledgement  $(H_1)$ . To do so, we first sample 100 responses (larger than previous experiment) and out of all possible pairs keep those with  $|\Delta \mathrm{pcmi}_h| > 15$  (larger than population interquartile range; Figure 8). To control for the amount of new information being added, we pick pairs with closest values of  $\mathsf{pcmi}_k$  (recall that  $\mathsf{pcmi}_k$  denotes information uniquely attributable to  $\mathbf{k}$ ). Such selected pairs have  $\text{Median} |\Delta \mathsf{pcmi}_k| = 0.42$ . We ask annotators to pick the response that provides better acknowledgement and select an acknowledgement span to support their claim.
Exp 3: Fused-PCMI vs. Max-PMI We test if the proposed method, Fused-PCMI (that combines PMI and PCMI) selects better responses than Max-PMI  $(H_{1})$ . For Fused-PCMI, we set low and high  $\mathrm{pcmi}_h$  thresholds to be 5 and 14 respectively based on population quartiles. For instances where the Fused-PCMI response is different from the Max-PMI response, we compare the two. We consider 10 candidate responses for each test instance and find that for around  $10\%$  of the instances the Fused-PCMI candidate is different from the Max-PMI candidate. Human annotators are then asked to pick the overall better response of the two.
5 Results & Analyses
Based on human annotations, we are able to reject  $H_{0}$  in favor of  $H_{1}$  in all three experiments (Table 2): high PMI responses are overall better quality than randomly chosen candidates, responses having high  $\mathrm{pcmi}_h$  provide better acknowledgement, and Fused-PCMI selects better responses than Max-PMI.
Exp	n	K	p	κ
1	87	55 (63%)	0.009	0.18
2	95	70 (74%)	3e-6	0.48
3	99	59 (60%)	0.035	0.11
Table 2: Human annotation results. Out of 100 instances, majority agreement was reached in n instances. The majority rejects the null-hypothesis  $(H_0)$  in favor of the alternate hypothesis  $(H_1)$  in K instances. p denotes the p-value and  $\kappa$  denotes Fliess kappa for Inter-annotator agreement.
While according to Exp 1, high PMI responses are overall better quality, upon further analysis
${}^{3}$  Statistically significant with  $p < {0.05}$  (Binomial Test).
Figure 6: Attribution to acknowledgement span. A larger fraction of  $\mathrm{pcmi}_h$  can be attributed to human annotated acknowledgement spans compared to  $\mathrm{pmi}(g; h)$  and  $\mathrm{pcmi}_k$ .
we find that PMI is useful for filtering out bad samples, but not necessarily for selecting between the good samples. When paired with a random response from the top  $50\%$  of the candidates (ranked according to their PMI), people prefer the Max-PMI response only  $52\%$  of the time (not significant). On the other hand, if the random response was in the bottom  $50\%$ , then the Max-PMI response is preferred  $74\%$  of the time. $^3$
In Exp 2, we ask annotators to mark text-spans that indicate acknowledgement (Table 3). If token-level  $\mathrm{pcmi}_h$  is concentrated in these spans, we have further proof that  $\mathrm{pcmi}_h$  indicates acknowledgement. Indeed, in Figure 6, we see that  $\mathrm{pcmi}_h$  is most attributable to the acknowledgement spans, followed by  $\mathrm{pmi}(g; h)$  and  $\mathrm{pcmi}_k$ . Thus,  $\mathrm{pcmi}_h$  captures acknowledgements with greater specificity than  $\mathrm{pmi}(g; h)$ .
To understand the mechanism behind the improvement in Exp 3, we look at the distribution of samples w.r.t.  $\mathrm{pcmi}_k$  and  $\mathrm{pcmi}_h$  in Figure 7. We observe that Max-PMI responses heavily skew the distribution towards higher  $\mathrm{pcmi}_k$ , whereas Fused-PCMI responses show a more balanced improvement along both  $\mathrm{pcmi}_h$  and  $\mathrm{pcmi}_k$ . Fused-PCMI increases both  $\mathrm{pcmi}_h$  and  $\mathrm{pcmi}_k$  (medians cross  $75\%$  quartiles), indicating that the responses are simultaneously specific to both  $\mathbf{h}$  and  $\mathbf{k}$ .
We show that samples with higher  $\mathrm{pcmi}_h$  provide better acknowledgement and Fused-PCMI improves overall quality compared to Max-PMI. Thus, by improving acknowledgements - an aspect we identified during our analysis of human strategies - we were able to improve overall quality. This demonstrates the utility of linguistic analysis for finding interpretable and actionable metrics.
While we show that our learnings apply to informative dialogue which adds factual knowledge (Dinan et al., 2019; Parthasarathi and Pineau,
Figure 7: Bivariate Kernel Density Estimate plot All candidates, Max-PMI responses and Fused-PCMI responses. Bivariate kernel density estimate plot w.r.t.  $\mathsf{pcmi}_k$  and  $\mathsf{pcmi}_h$  at levels 0.5 and 0.75. We see that Fused-PCMI responses compared with Max-PMI trade off little  $\mathsf{pcmi}_k$  for a large relative gain in  $\mathsf{pcmi}_h$ . See Figure 8 in Section A.2 for univariate box plots.
2018; Gopalakrishnan et al., 2019), we expect it to generalize to any dialog setting that adds new content, e.g., experiences (Ghazvininejad et al., 2018) and personas (Zhang et al., 2018). Any dual-context language generation task where the two contexts are asymmetric in their information content can potentially benefit from PCMI.
There is scope for improvement: Max-PMI still selects better responses than Fused-PCMI in  $40\%$  of the instances. This could be because it is easy for the model to copy over  $\mathbf{k}$  and generate a high PMI response that is also fluent and accurate. Fused-PCMI encourages synthesis of acknowledgement using  $\mathbf{h}$  and abstraction over  $\mathbf{k}$  and it could therefore be prone to disfluencies and inaccuracies. We hope that orthogonal modeling improvements (Meng et al., 2020) reduce such effects.
A cause for concern with the human evaluation is low inter-annotator agreement for Exp 1 and 3 where we ask them to pick responses with "overall better quality and suitability". However, quality measurements are inherently subjective; people differ in the importance they place on different aspects such as engagement, informativeness, fluency etc., as corroborated by prior work (Finch and Choi, 2020) that shows low Cohen's kappa (0.13, 0.22) for overall quality judgements. In this work, diverse expectations from multiple annotators are captured yet subsequently averaged into "overall quality". We leave it to future work to find finer-grained metrics that have high inter-annotator agreement and derive empirical weights to combine them into "overall quality".
In this work, we looked at acknowledgements and their impact on quality in isolation, but in a real
Previous Utterance	High pcmih candidate
No I did not! Han Solo was apparently also an imperial lieutenant before meeting up with Chewbacca.	that is very interesting and I wonder if he was one of the first or the first one to meet Chewba. I was just reading that George Lucas originally intended Han to be a green alien Yes, that sounds like a very cool tradition. I wonder how they enforced that law, it sounds as ridiculous as the prohibition of wearing a hat in 1912, that was a law that would prevent the use of hat pins by suffragettes.
I did not. That sounds interesting. Much like the "tip of the hat" originates from Knights raising their visors.
Yeah. When Apollo 11 was preparing for their voyage, they had discussion as to whether they should plant a United Nations flag instead of a US flag. If I planted a UN flar and someone asked me "whose flag is that?" I would answer, mine...	lol, no kidding. I wonder if they are still discussing the issue with the moon? I can't believe the moon is moving away from us at about the rate that your nails grow. That seems really slow to me.
Table 3: Human annotated spans of text that indicate acknowledgement (in bold) in candidate responses with higher  $\mathbf{pcmi}_h$  from Exp 2.
Previous Utterance	Max-PMI candidate	Fused-PCMI candidate
1	I live in the southwest USA desert. Not much in the way of water here either, but we've got some nice artificial pools. What kind of swimming did you do?	There is a type of competitive swimming where the athlete has their hands and legs bound, and swims like a dolphin. I have never heard that before. It seems fishy to me. I'm not sure I can swim like that.	I went competitively but was having a tough time. I liked that kind of competition where the swimmer can swim with their hands bound.
2	It is interesting. It's also interesting that they are trying to plan a Mars colonization project that will be funded by a reality show, I really want to see that!	A satellite was lost on Mars because of the use of the Imperial system and NASA used the metric system.	Sounds like a weird reality show. I'm going to look it up. A satellite was lost on Mars because it was used by NASA using the metric system.
3	Yeah I definitely like clothes. I think it's interesting how many uses hats have, like for protection, ceremonial reasons, religious reasons, safety, and fashion.	Did you know that until 1998, British politicians had to wear a top hat if making a point of order in the house of commons?	In 1998, British politicians were required to wear a top hat if making a point of order in the house of commons. I guess that's kind of messed up and counterproductive.
4	Yeah it's super funny that he had a rap battle in the lunchroom. Seems like something straight out of a movie!	Ha, I agree. Speaking of rappers, did you know that "California Gurls" was actually a West Coast vs East Coast answer song to Alicia Keys and Jay Z's "Empire State of Mind"	Yeah, and how about the fact that the California Gurls song was actually a west coast vs east coast answer song to a pair of famous rappers. That is surprising
5	im not surprised. rats are quite clever, and i guess if scientists understand the rat brain, then they can wire up the interface. cool.	It's amazing how they can be clever and change forms. I like how they can now Debrate a chicken with surgical precision.	Robots can also debone chicken with precision. I guess that would help avoid chicken pogis.
Table 4: Illustrative samples of selected responses used in Exp 3. For samples 1,2 and 3 people prefer Fused-PCMI and for samples 4 and 5 they prefer Max-PMI. Factual content copied largely verbatim by the model is in gray. Specifically, the Fused-PCMI candidate in 1 answers the question (Max-PMI does not) and in 3 refers back to contradict utility of hats.
system, the performance of the model also depends on other factors like user compliance and the retrieval model. In practice, we think the interplay between the four linguistic aspects is critical and needs to be explored. For instance, preliminary experiments with live conversations and an off-the-shelf retriever suggested that a bad choice of  $\mathbf{k}$  with tenuous connections to  $\mathbf{h}$  can make synthesis harder and lead to lower quality Fused-PCMI responses. Better retrieval models (Ren et al., 2020) that make use of transition strategies to determine  $\mathbf{k}$  can lead to better acknowledgements.
In this work, we identified salient aspects of
human-human informative conversations and found deficiencies in current neural dialogue systems. We proposed a PCMI-based selection strategy that selected responses with acknowledgements and higher overall quality. We hope that our work provides actionable insights and metrics for future work and more generally inspires the use of linguistic literature for grounding conversational research.
We are grateful to Amelia Hardy, Nandita Bhaskhar, Omar Khattab, Kaitlyn Zhou, Abigail See, other Stanford NLP group members and the anonymous
reviewers for helpful comments. This research is funded in part by Samsung Electronics Co., Ltd. and in part by DARPA CwC under ARO prime contract no. W911NF-15-1-0462. This article solely reflects the opinions and conclusions of its authors. Christopher Manning is a CIFAR Fellow.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Herbert H. Clark. 1996. Using Language. 'Using' Linguistic Books. Cambridge University Press.
Herbert H Clark and Susan E Brennan. 1991. Grounding in communication. In L. B. Resnick, J. M. Levine, and S. D. Teasley, editors, *Perspectives on socially shared cognition*, pages 127-149. American Psychological Association.
Herbert H Clark and S Haviland. 1977. Comprehension and the given-new contract. In Roy O Freedle, editor, Discourse production and comprehension, pages 1-40. Lawrence Erlbaum Associates, Hillsdale, NJ.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Sarah E. Finch and Jinho D. Choi. 2020. Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236-245, 1st virtual meeting. Association for Computational Linguistics.
Raefer Gabriel, Yang Liu, Anna Gottardi, Mihail Eric, Anju Khatri, Anjali Chadha, Qinlang Chen, Behnam Hedayatnia, Pankaj Rajan, Ali Binici, Shui Hu, Karthik Gopalakrishnan, Seokhwan Kim, Lauren Stubel, Arindam Mandal, and Dilek Hakkani-Tür. 2020. Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge. Alexa Prize Proceedings.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
J. J. Godfrey, E. C. Holliman, and J. McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517-520 vol.1.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Herbert P Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Speech acts, pages 41-58. Brill.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Ellen A Isaacs and Herbert H Clark. 1987. References in conversation between experts and novices. Journal of experimental psychology: general, 116(1):26.
Robert M Krauss and Susan R Fussell. 1996. Social psychological models of interpersonal communication. Social psychology: Handbook of basic principles, pages 655-701.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Chuan Meng, Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and Maarten de Rijke. 2020. Refnet: A reference-aware network for background based conversation. In Thirty-Fourth AAAI Conference on Artificial Intelligence.
Prasanna Parthasarathi and Joelle Pineau. 2018. Extending neural generative conversational model using external knowledge sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 690-695, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Byron Reeves and Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media like Real People and Places. Cambridge University Press, USA.
Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and Maarten de Rijke. 2020. Thinking globally, acting locally: Distantly supervised global-to-local knowledge selection for background based conversation. In Thirty-Fourth AAAI Conference on Artificial Intelligence, volume abs/1908.09528.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Lui, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, Y-Lan Boureau, and Jason Weson. 2020. Recipes for building an open-domain chatbot. ArXiv preprint arXiv:2004.13637.
Harvey Sacks and Gail Jefferson. 1995. Winter 1971, chapter 12. John Wiley & Sons, Ltd.
Vicki L Smith and Herbert H Clark. 1993. On the course of answering questions. Journal of memory and language, 32(1):25-38.
Robert Stalnaker. 2002. Common ground. Linguistics and philosophy, 25(5/6):701-721.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-374.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019a. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019b. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. *DialoGPT: Large-scale generative pre-training for conversational response generation*. In *ACL*, system demonstration.
A.1 Model training details
Each model (main and ablation) was trained on a single NVIDIA Titan Xp GPU for 5 epochs and took approximately 8 hours to train. The training dataset had 51407 instances, validation 2491 and test 2728. The Topical Chat dataset and Switchboard corpus are in English language. The main model used for response generation had a validation loss (average negative log likelihood) of 2.05 which it reached after 2 epochs.
A.2 Univariate distribution
Figure 8: Univariate box plots for All candidates, Max-PMI responses and Fused-PCMI responses. (a) is w.r.t.  $\mathrm{pcmi}_k$  and (b) w.r.t  $\mathrm{pcmi}_h$ . Pink horizontal lines indicate  $75\%$  quartile for All candidates. Max-PMI responses (orange) have high  $\mathrm{pcmi}_k$  (median above pink line), but low  $\mathrm{pcmi}_h$ . Fused-PCMI responses (green) show balanced yet high  $\mathrm{pcmi}_h$  and  $\mathrm{pcmi}_k$  (medians cross pink lines).
A.3 Annotation Details
We had 9, 19 and 19 unique annotators for experiments 1, 2 and 3 respectively. All three annotators agreed in 32/87 instances for experiment 1, 52/87 instances for experiment 2 and 32/99 instances for experiment 3.
- You are given the two utterances from the middle of a conversation between two acquaintances casually chatting about topics which interest them.
- Two possibilities for the following utterance are given. Your task to pick the one which seems to have an overall better quality and suitability.
- It is possible that both possibilities look comparable. However, you should try to discern carefully and pick the better one between the two.
Speaker A: That's going to be a tough call...he might have a tough time beating Brady's super bowl wins.
Speaker B: For sure, he is so fun to watch. He came back from Denver and that 10 point difference which is pretty nuts.
Option 1	Option 2
Speaker A: I know, I think it was the Broncos who made a big play! I can't believe Bill Belichick's teams have had
○ Option 1 is better	Speaker A: Yeah, but the Browns' last playoff win was in 1995 and Bill Belichick was the coach.
○ Option 2 is better
I can't make sense of either option.
Figure 9: Annotation interface for Best PMI v/s rest
Part 1:
Note: This is different from an earlier task.
- You are given the two utterances from the middle of a conversation between two acquaintances casually chatting about topics which interest them.
- Two possibilities for the following utterance are given. Your task to pick the one which better acknowledges the previous turns.
- It is possible that both possibilities look comparable. However, you should try to discern carefully and pick the better one between the two.
Speaker A: Yes I agree. You said that you like Star Wars movies right? did you know that Han Solo used to be a TIE fighter pilot?
Speaker B: No I did not! Han Solo was apparently also an imperial lieutenant before meeting up with Chewbacca.
Option 1	Option 2
Speaker A: that is very interesting and I wonder if he was one of the first or the first one to meet Chewba. I was just reading that George Lucas originally intended Han to be a green alien
○ Option 1 is better	Speaker A:Yeah that's pretty cool. I saw that George Lucas originally wanted to make Han Solo as a green alien or a black man.
○ Option 2 is better
Part 2: