SlowGuess/NAACL_2021_OCR huggingface dataset:
Ashwin Paranjape
ashwinp@cs.stanford.edu
manning@cs.stanford.edu
This work aims to build a dialogue agent that can weave new factual content into conversations as naturally as humans. We draw insights from linguistic principles of conversational analysis and annotate human-human conversations from the Switchboard Dialog Act Corpus to examine humans strategies for acknowledgement, transition, detail selection and presentation. When current chatbots (explicitly provided with new factual content) introduce facts into a conversation, their generated responses do not acknowledge the prior turns. This is because models trained with two contexts – new factual content and conversational history – generate responses that are non-specific w.r.t. one of the contexts, typically the conversational history. We show that specificity w.r.t. conversational history is better captured by pointwise conditional mutual information  $(\mathrm{pcmi}_h)$  than by the established use of pointwise mutual information (pmi). Our proposed method, Fused-PCMI, trades off pmi for  $\mathrm{pcmi}_h$  and is preferred by humans for overall quality over the Max-PMI baseline  $60\%$  of the time. Human evaluators also judge responses with higher  $\mathrm{pcmi}_h$  better at acknowledgement  $74\%$  of the time. The results demonstrate that systems mimicking human conversational traits (in this case acknowledgement) improve overall quality and more broadly illustrate the utility of linguistic principles in improving dialogue agents.
Social chatbots are improving in appeal and are being deployed widely to converse with humans (Gabriel et al., 2020). Advances in neural generation (Adiwardana et al., 2020; Roller et al., 2020) enable them to handle a wide variety of user turns and to provide fluent bot responses. People expect their interactions with these dialogue agents to be similar to real social relationships (Reeves and Nass, 1996). In particular, they expect social
Figure 1: The setting for conversational rephrasing. Conversational history (h) and new factual content (k), two largely independent contexts, are used to sample responses  $(\mathbf{g}_1,\mathbf{g}_2)$  from a generative model. The samples differ qualitatively. While almost all of  $\mathbf{g}_2$  is verbatim from  $\mathbf{k}$  (in gray), the first sentence in  $\mathbf{g}_1$  (in black) acknowledges using  $\mathbf{h}$  and bridges to  $\mathbf{k}$ .
chatbots to both use information that is already known and separately add new information to the conversation, in line with the given-new contract (Clark and Haviland, 1977).
Neural generation methods for adding new information (Dinan et al., 2019; Gopalakrishnan et al., 2019; Ghazvininejad et al., 2018; Zhang et al., 2018) measure progress using metrics like "engagingness", "appropriateness" and "informativeness". But these metrics are too broad and provide little actionable insight to drive improvements in these systems. On the other hand, psycholinguists and sociolinguists have studied human conversations in
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 768-781 June 6-11, 2021. ©2021 Association for Computational Linguistics
depth and have identified fine-grained conventions, principles and contracts (Grice, 1975; Clark, 1996; Krauss and Fussell, 1996).
Our first contribution is a linguistic analysis of how human conversations incorporate world knowledge. We manually annotate conversations from the Switchboard corpus to identify key traits. In particular, we find that people apply four kinds of strategies: (1) acknowledgement of each other's utterances, (2) transition to new information, (3) appropriate level of detail selection and (4) presentation of factual content in forms such as opinions or experiences.
To identify deficiencies of the above types in machine-learned models, we consider a simplified task of conversational rephrasing (Figure 1), in which the factual content to be added is not left latent but is provided as a text input to the model (as in Dinan et al. (2019)), along with conversational history. Just as humans do not recite a fact verbatim in a conversation, we expect the model to rephrase the factual content by taking conversational context into account. We derive the data for this task using the Topical Chat dataset (Gopalakrishnan et al., 2019) and fine-tune a large pre-trained language model on it.
Li et al. (2016); Zhang et al. (2020) use maximum pointwise mutual information (Max-PMI) to filter out bad and unspecific responses sampled from a generative language model. However, we observe that Max-PMI responses lack in acknowledgement, an essential human trait. This is because a generated response that simply copies over the new factual content while largely ignoring the conversational history can have high mutual information (MI) with the overall input.
Our second contribution is a method to select responses that exhibit human-like acknowledgement. To quantify the amount of information drawn from the two contexts of new factual content and conversational history, we propose using pointwise conditional mutual information (PCMI). We show that responses with a higher PCMI w.r.t conversational history given factual content  $(\mathrm{pcmi}_h)$  are judged by humans to be better at acknowledging prior turns  $74\%$  of the time. Then, we use  $\mathrm{pcmi}_h$  to identify Max-PMI responses that lack acknowledgement and find alternative responses (Fused-PCMI) that trade off pmi for  $\mathrm{pcmi}_h$ . Despite a lower PMI, human anno
Figure 2: Examples for Acknowledgement Strategies from Switchboard (parts omitted for brevity).
tators prefer the Fused-PCMI alternative over the Max-PMI response  $60\%$  of the time. We release annotated conversations from the Switchboard corpus (with guidelines), code for fine-tuning and calculating scores and human evaluations.
2 Strategies
for informative conversations
To understand strategies used by humans while talking about factual knowledge, we annotate turns in human-human conversations. We adopt and extend Herbert Clark's approach to conversational analysis. According to his given-new contract (Clark and Haviland, 1977), the speaker connects their utterances with the given information (assumed to be known to the listener) and adds new information. This builds up common ground (Stalnaker, 2002) between the two participants, defined to be the sum of their mutual, common or joint knowledge, beliefs and suppositions. We identify the following four aspects to the process of adding new information to a conversation.
Acknowledgement strategies According to Clark and Brennan (1991), the listener provides positive evidence for grounding. We classify all mentions of prior context into various acknowledgement strategies.
Transition strategies According to Sacks and Jefferson (1995), topical changes happen step
$^{2}$ https://github.com/AshwinParanjape/human-like-informative-conversations
${}^{1}$  Statistically significant with  $p < {0.05}$  (Binomial Test).
Figure 3: Examples for popular Transition Strategies from Switchboard (parts omitted for brevity).
by step, connecting the given, stated information to new information. We annotate the semantic justifications for topical changes as different transition strategies.
Detail selection strategies According to Isaacs and Clark (1987), speakers in a conversation inevitably know varying amounts of information about the discussion topic and must assess each other's expertise to accommodate their differences. We posit that each speaker applies detail selection strategies to select the right level of detail to be presented.
Presentation strategies According to Smith and Clark (1993), presentation of responses is guided by two social goals - exchange of information and self-presentation. While we do not consider social goals in this work, we hypothesize
that people talk about factual information in non-factual forms (e.g., opinions, experiences, recommendations) which we classify as various presentation strategies.
2.1 Analysis of strategies
Dataset We annotate part of the The Switchboard Dialog Act Corpus (Stolcke et al., 2000), an extension of the Switchboard Telephone Speech Corpus (Godfrey et al., 1992) with turn-level dialog-act tags. The corpus was created by pairing speakers across the US over telephone and introducing a topic for discussion. This dataset is uniquely useful because as a speech dataset, it is more intimate and realistic than text-based conversations between strangers. We annotate conversations on social topics which might include specific knowledge (like Books, Vacations, etc.) but leave out ones about subjective or personal experiences.
Specific knowledge We define specific knowledge as knowledge that can be "looked up" but isn't widely known (as opposed to general knowledge that everybody is expected to know and experiential knowledge that can only be derived from embodied experiences). In this work, we are interested only in specific knowledge because it serves as a source of new information in a conversation that is hard for a language model to learn implicitly but is likely available as text that can be supplied to the system. Out of 408 annotated turns, 111  $(27\%)$  incorporate specific knowledge and account for  $56\%$  of the tokens.
Next, we analyze various strategies employed in turns containing specific knowledge:
Acknowledgement Strategies In  $70\%$  of the turns, the speaker acknowledges the prior turn corroborating Clark and Brennan (1991). Three main strategies (Figure 2): agreement (or disagreement), shared experiences (or differing experience) and backchanneling account for  $60\%$  of the turns (Figure 4). In certain cases, explicit acknowledgement isn't necessary. For example, the answer to a question demonstrates grounding and serves as an implicit acknowledgement. These are categorized as  $N / A$ .
Transition Strategies At the beginning of a conversation, the participants use the discussion theme to pick a topic (various transition strategies are shown in Figure 3). The decision to stay on the topic or to transition to a new one is an implicit form of negotiation and depends on the interest
Figure 4: Distribution of acknowledgement, transition and presentation strategies
and ability of both speakers to participate. Nearly half the time, people elaborate upon the current topic (Figure 4). With a supportive listener, they might elaborate upon their own prior utterance (self-elaboration). Or they might signal interest in continuing the topic by elaborating the other speaker's utterance (other-elaboration). However, in a quarter of the turns, a participant loses interest or both participants run out of material. In that case, they transition to a new topic, implicitly justified by commonalities or differences with the current topic. If all else fails, they fall back to the discussion theme to pick a new topic.
Detail-selection strategies People probe the other speaker's knowledge about an entity before diving into details. As a probing mechanism, people introduce an entity without any details (introduce-entity)  $50\%$  of the time. Depending on the response, details are laid out  $66\%$  of the time. Note that a turn can have both labels, i.e., it can introduce an entity for the first time or it can have details of one entity while also introducing another entity. Interestingly, in  $7\%$  of turns, an entity's name is omitted but some details are presented, creating an opening for the other speaker to chime in.
Presentation strategies A single utterance can have multiple modes of presentation. A factual (objective) statement of specific knowledge is uncommon  $(25\%)$  in comparison with a subjective rendering in the form of an experience  $(53\%)$  or an opinion  $(34\%)$  (Figure 4). The other common modes of presentation are questions  $(9\%)$  and answers  $(16\%)$ , which often occur as adjacency pairs. We also found a few other uncommon modes  $(7\%)$  such as recommendations or hypotheses based on specific knowledge.
2.2 Implications for dialogue agents
The four aspects - acknowledgement, transition, detail selection and presentation - are essential ingredients and indicative of quality conversation.
They provide us with finer-grained questions amenable to human evaluation: "How does the agent acknowledge?", "Was it a smooth transition?", "Does the utterance contain the right level of detail?", and "Was the information presented as experience or an opinion?"
These four aspects are also more actionable than the evaluation metrics used in prior work. They can inspire new techniques that are purposefully built to emulate these strategies. For instance, transitions can be improved with purpose-built information retrieval methods that use commonalities and differences to choose a new topic. To improve detail-selection, an agent could keep track of user knowledge and pragmatically select the right level of detail. Moreover, in their datasets, Dinan et al. (2019) and Gopalakrishnan et al. (2019) asked people to reply using knowledge snippets, but that can lead to factual statements dominating the presentation strategies. We hope that newer datasets either suggest ways to reduce this bias or not provide knowledge snippets to humans in the first place but instead post facto match utterances to knowledge snippets.
In the rest of the paper, we focus on generating responses with better acknowledgements. This is because current neural generation methods perform poorly in this regard when compared with the other aspects. They fail to acknowledge prior turns and even when they do, the acknowledgements are shallow and generic (e.g., backchannel). We hypothesize that the bottleneck is not the modeling capacity, but rather our inability to extract acknowledgements. The responses are not specific w.r.t. conversational context, a prerequisite for richer acknowledgements (e.g., shared experiences). We show that selecting responses specific to conversational context improves acknowledgements and overall quality. More broadly, we are able to demonstrate the utility of our linguistic analysis in evaluating and improving a dialogue agent.
3 A method for richer acknowledgements
Current neural generation methods typically offer short and formulaic phrases as acknowledgements: "That's interesting", "I like that", "Yeah, I agree". Such phrases are appropriate almost everywhere and convey little positive evidence for understanding or grounding. The training corpus, on the other hand, contains richer acknowledgements, which generated responses should be able to emulate.
We assume that the representational capacity of current neural models is sufficient and that out of all the sampled responses, some do indeed contain a richer form of acknowledgement. We posit that non-existent or poor sample selection strategies are to blame and that without a good sample selection strategy, improvements to the dataset, model or token-wise sampling methods are unlikely to help.
We hypothesize that responses that are more specific to conversational history provide better evidence for understanding and hence contain richer acknowledgements. As a baseline sample selection strategy, we first consider maximum pointwise mutual information (Max-PMI) (as used by Zhang et al. (2020)) between the generated response and the conversational contexts (i.e., new factual content and conversational history). However, this is insufficient because it is an imprecise measure of specificity w.r.t. conversational history. Instead, we use pointwise conditional mutual information (PCMI) to maintain specificity with individual contexts and propose a combination of PMI and PCMI scores to select overall better quality responses than Max-PMI.
Conversational rephrasing The choice of new factual content is a confounding factor for analysis. Hence, we define a simplified task, conversational rephrasing, where content is provided as an input. Thus, conversational rephrasing is a generation task where conversational history (h) and new factual content (k) are given as inputs and a response (g) is generated as the output (Figure 1). We expect the generation g to paraphrase the new factual content k in a conversational manner by utilizing the conversational history h.
Base generator We fix the sequence-to-sequence model and token-wise sampling method and vary the sample selection strategy. The model is trained to take  $\mathbf{h}$  and  $\mathbf{k}$  as input and to generate  $\mathbf{g}$  as the output with the language modelling loss, i.e., we minimize the token-wise negative log likelihood. During generation, tokens are sampled
Response	pmi(g; h, k)	pmi(g; h)	pcmih
g1	87	18	14
g2	150	18	4
Table 1: Measures of mutual information for generated responses from Figure 1.  $\mathbf{g}_2$  largely copies  $\mathbf{k}$ , has high  $\mathrm{pmi}(\mathbf{g};\mathbf{h},\mathbf{k})$  and would be chosen by Max-PMI.  $\mathbf{g}_1$ 's first sentence acknowledges using  $\mathbf{h}$  and bridges to  $\mathbf{k}$ ; it would be chosen by Fused-PCMI on the basis of  $\mathrm{pcmi}_h$ .  $\mathrm{pmi}(\mathbf{g};\mathbf{h})$  cannot differentiate the two.
autoregressively from left-to-right. While sampling each token, the probability distribution is truncated using nucleus sampling (Holtzman et al., 2020) but the truncation is kept to a minimum with a high value of  $p$  for top-p sampling. Multiple diverse candidates are sampled from the base generator and now the best candidate needs to be selected.
PMI for overall specificity Li et al. (2016) suggest selecting the response with maximum PMI (referred to as MMI in their work) to maintain specificity and get rid of bland or low-quality samples. Pointwise Mutual Information (PMI) between two events  $(x,y)$  is a measure of change in the probability of one event  $x$ , given another event  $y$ :  $\mathrm{pmi}(x;y)\equiv \log \frac{p(x|y)}{p(x)}$ . We use pmi to determine the increase in likelihood of  $\mathbf{g}$ , given  $\mathbf{h}$  and  $\mathbf{k}$ .
$$
\operatorname {p m i} (\mathbf {g}; \mathbf {h}, \mathbf {k}) = \log {\frac {p (\mathbf {g} | \mathbf {h} , \mathbf {k})}{p (\mathbf {g})}}
$$
A candidate generation  $\mathbf{g}$  with higher PMI is more likely given the two contexts  $\mathbf{h}$  and  $\mathbf{k}$  than otherwise and is therefore considered more specific to the contexts. A low PMI value for a candidate response implies non-specificity to either context providing a clear signal for discarding it. A high PMI is necessary but not sufficient for a candidate to be specific to both the contexts simultaneously, since mutual information could come from either context. For example,  $\mathbf{g}_2$  (Figure 1) merely copies  $\mathbf{k}$  but gets a high PMI score (Table 1). Whereas  $\mathbf{g}_1$  acknowledges prior turn and uses  $\mathbf{k}$  but gets a lower PMI score.
PCMI for contextual specificity Pointwise Conditional Mutual Information (PCMI) considers a third variable  $(z)$  and removes information due to  $z$  from  $\mathrm{pmi}(x;y,z)$  to keep only the information uniquely attributable to  $y$ .
$$
\operatorname {p c m i} (x; y | z) = \operatorname {p m i} (x; y, z) - \operatorname {p m i} (x; z)
$$
We propose using pcmi for contextual specificity, i.e.,  $\mathrm{pcmi}_h = \mathrm{pcmi}(\mathbf{g};\mathbf{h}|\mathbf{k})$  for specificity
Figure 5: Token-wise probabilities (top), pmi (middle) and pcmi (bottom) scores for the generated response g from Figure 1. The pcmi graph is computed from the pmi graph which in turn is computed from the probability graph. The probabilities by themselves are unreliable measures of contextual specificity; the tokens predictable without h, k (e.g., 's) have high probability but low pmi. pmi cannot differentiate between the two contexts; tokens coming from both contexts (e.g., Shakespeare) have high pmi but low pcmi. pcmi differentiates the two contexts; tokens unique to conversational history h (e.g., names, today) have high  $\mathrm{pcmi}_h$ , Tokens unique to new factual content k (e.g., works, performed, all of last sentence) have high  $\mathrm{pcmi}_k$ .
w.r.t. to conversational history  $\mathbf{h}$ , and  $\mathrm{pcmi}_k = \mathrm{pcmi}(\mathbf{g};\mathbf{k}|\mathbf{h})$  for specificity w.r.t. new factual content  $\mathbf{k}$ .
Since acknowledgement strategies are primarily based on the history of the conversation thus far, we would expect candidates with higher  $\mathrm{pcmi}_h$  to exhibit more human-like acknowledgement strategies.
As a point of comparison, consider using  $\mathrm{pmi}(\mathbf{g};\mathbf{h})$  instead of  $\mathrm{pcmi}_h$ . In our setting of conversational rephrasing for informative dialogue,  $\mathbf{k}$  topically overlaps with  $\mathbf{h}$ . If  $\mathbf{g}$  merely copied over the new factual content  $\mathbf{k}$  without any reference to  $\mathbf{h}$ , it would still have a high  $\mathrm{pmi}(\mathbf{g};\mathbf{h})$  due to topical overlap but a low  $\mathrm{pcmi}_h$ . Going back to Table 1, we can see that  $\mathrm{pmi}(\mathbf{g};\mathbf{h})$  is unable to distinguish between the two examples but  $\mathrm{pcmi}_h$  is.
In Figure 5, the above quantities are broken down to token-level granularity. We can see that specific words that are uniquely attributable to each context are cleanly separated by both  $\mathrm{pcmi}_h$  and  $\mathrm{pcmi}_k$ .
Combining PMI & PCMI for overall quality To show the utility of  $\mathrm{pcmi}_h$  in improving overall quality, we propose a heuristic method to find a more balanced response (Fused-PCMI) than the Max-PMI response. For every Max-PMI response with a low  $\mathrm{pcmi}_h$ , we consider an alternative that has both high  $\mathrm{pcmi}_h$  and an acceptable PMI. If such an alternative is found, we select that as the Fused-PCMI response; otherwise we default to the
Max-PMI response as the Fused-PCMI response. We consider a PMI score in the top  $50\%$  of the candidate set as acceptable. To compute pcmi thresholds, we calculate quantiles based on the entire validation set and consider  $\mathrm{pcmi}_h$  in the first quartile to be low and  $\mathrm{pcmi}_h$  in the fourth quartile to be high. This approach is less susceptible to outliers, more interpretable and easier to calibrate than a weighted arithmetic or geometric mean.
We derive the data for our conversational rephrasing task from the Topical Chat dataset (Gopalakrishnan et al., 2019). We use it to fine-tune a large pre-trained neural language model. This forms the base model as described in Section 3. To evaluate our proposed methods, we design three experiments and perform a comparative study with human annotators.
Topical Chat Dataset This is a human-human chat dataset where crowd-workers were asked to chat with each other around certain topics. They were provided with relevant interesting facts from the "Today I learned" (TIL) subreddit which they could use during the conversation. TILs are are short (1-3 sentences), self-contained, interesting facts, most of them from Wikipedia articles. When an utterance can be matched to a TIL (based on a TF-IDF threshold of 0.12), we create an instance for the conversational rephrasing task: with the utterance as  $\mathbf{g}$ , the two previous utterances as  $\mathbf{h}$  and
the corresponding TIL as  $\mathbf{k}$ . We split the instances into training, validation and test sets (sizes in Section A.1) such that all utterances related an entity belong to the same set.