Venue: NAACL 2019

# What makes a good conversation?

How controllable attributes affect human judgments

Abigail See1

Stanford University

abisee@stanford.edu &Stephen Roller

Facebook AI Research

roller@fb.com &Douwe Kiela

Facebook AI Research

dkiela@fb.com &Jason Weston

Facebook AI Research

jase@fb.com

Footnote 1: A.S. completed most of this work at Facebook (FAIR).

###### Abstract

A good conversation requires balance - between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.

## 1 Introduction

Neural generation models for dialogue, despite their ubiquity in current research, are still poorly understood. Well known problems, such as the genericness and repetitiveness of responses Serban et al. (2016a), remain without a de facto solution. Strikingly, the factors that determine human judgments of overall conversation quality are almost entirely unexplored. Most works have been limited to the next utterance prediction problem, whereas a multi-turn evaluation is necessary to evaluate the quality of a full conversation.

In this work we both (i) conduct a large-scale study to identify the fine-grained factors governing human judgments of full conversations, and (ii) develop models that apply our findings in practice, leading to state-of-the-art performance. Specifically, we identify and study eight aspects of conversation that can be measured by human judgments, while varying four types of low-level attributes that can be algorithmically controlled in neural models; see Figure 1. To control the low-level model attributes, we consider two simple but general algorithms: conditional training, in which the neural model is conditioned on additional control features, and weighted decoding, in which control features are added to the decoding scoring function at test time only.

One major result of our findings is that existing work has ignored the importance of conversational flow, as standard models (i) repeat or contradict previous statements, (ii) fail to balance specificity with genericness, and (iii) fail to balance asking questions with other dialogue acts. Conducting experiments on the PersonaChat task Zhang et al. (2018b), we obtain significantly higher engaging-ness scores than the baseline by optimizing control of repetition, specificity and question-asking over multiple turns. Using these findings, our best model matches the performance of the winning entry in the recent NeurIPS ConvAI2 competition Dinan et al. (2019), which was trained on much

Figure 1: We manipulate four low-level attributes and measure their effect on human judgments of individual conversational aspects, as well as overall quality.



more data but had no control (see Section 8.1). Our code, pretrained models, and full chatlogs, are available at [https://parl.ai/projects/controllable_dialogue](https://parl.ai/projects/controllable_dialogue).

## 2 Related Work

DialogueDialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion Bordes et al. (2017); El Asri et al. (2017); Hastie (2012); Henderson et al. (2014); Wen et al. (2017). Task success combined with dialogue cost can be linked to human judgments like user satisfaction via the PARADISE framework Walker et al. (1997).

However in chitchat tasks, which we study in this work, automatic metrics and their relation to human ratings are less well-understood. While word-overlap metrics are effective for question-answering and machine translation, for dialogue they have little to no correlation with human judgments Liu et al. (2016); Novikova et al. (2017) - this is due to the open-ended nature of dialogue. There are more recent attempts to find better automatic approaches, such as adversarial evaluation Li et al. (2017b) and learning a scoring model Lowe et al. (2017), but their value is still unclear.

Nevertheless, a number of studies only use automatic metrics, with no human study at all Lowe et al. (2015); Parthasarathi and Pineau (2018); Serban et al. (2016b). Other works do use human evaluations Dinan et al. (2018); Li et al. (2016a, b); Venkatesh et al. (2017); Vinyals and Le (2015); Zhang et al. (2018b), typically reporting just one type of judgment (either quality or appropriateness) via a Likert scale or pairwise comparison. Most of those works only consider single turn evaluations, often with a shortened dialogue history, rather than full multi-turn dialogue.

A more comprehensive evaluation strategy has been studied within the scope of the Alexa prize Venkatesh et al. (2017); Guo et al. (2018) by combining multiple automatic metrics designed to capture various conversational aspects (engagement, coherence, domain coverage, conversational depth and topical diversity). Though these aspects have some similarity to the aspects studied here, we also focus on lower-level aspects (e.g. avoiding repetition, fluency), to understand how they correspond to both our controllable attributes, and to overall quality judgments.

Controllable neural text generationReearchers have proposed several approaches to control aspects of RNN-based natural language generation such as sentiment, length, speaker style and tense Fan et al. (2018); Ficler and Goldberg (2017); Ghazvininejad et al. (2017); Hu et al. (2017); Kikuchi et al. (2016); Peng et al. (2018); Wang et al. (2017). In particular, several works use control to tackle the same common sequence-to-sequence problems we address here (particularly genericness and unrelated output), in the context of single-turn response generation Baheti et al. (2018); Li et al. (2016a, 2017); Shen et al. (2017); Xing et al. (2017); Zhang et al. (2018a); Zhou et al. (2017). By contrast, we focus on developing controls for, and human evaluation of, _multi_-turn interactive dialogue - this includes a new method (described in Section 5) to control attributes at the _dialogue_ level rather than the utterance level.

In this work, we require a control method that is both general-purpose (one technique to simultaneously control many attributes) and easily tunable (the control setting is adjustable after training). Given these constraints, we study two control methods: conditional training (variants of which have been described by Fan et al. (2018); Kikuchi et al. (2016); Peng et al. (2018)) and weighted decoding (described by Ghazvininejad et al. (2017) as a general technique, and by Baheti et al. (2018) to control response-relatedness). To our knowledge, this work is the first to systematically compare the effectiveness of two general-purpose control methods across several attributes.

## 3 The PersonaChat dataset

PersonaChat Zhang et al. (2018b) is a chitchat dialogue task involving two participants (two humans or a human and a bot). Each participant is given a _persona_ - a short collection of personal traits such as _I'm left handed_ or _My favorite season is spring_ - and are instructed to get to know each other by chatting naturally using their designated personnas, for 6-8 turns. The training set contains 8939 conversations and 955 personas, collected via crowdworkers, plus 1000 conversations and 100 personas for validation, and a similar number in the hidden test set. The PersonaChat task was the subject of the NeurIPS 2018 ConvAI2 Challenge Dinan et al. (2019), in which competitors were first evaluated with respect to automatic met 

rics (perplexity, hits@1 and F1 score), and then with respect to human judgment via the question _"How much did you enjoy talking to this user?"_ on a scale of 1-4.

## 4 Baseline model

Our baseline model is a 2-layer LSTM sequence-to-sequence model with attention. On any dialogue turn, the input \(x\) to the encoder is the entire dialogue history (separated using unique speaker-identifying tokens), with the model's own persona prepended. Conditioned on this input sequence \(x\), the decoder generates a response \(y\). Except when stated otherwise, all our models decode using beam search with beam size 20.

We initialized the word embedding matrix with 300-dimensional GloVe embeddings Pennington et al. (2014). Using the ParlAI framework Miller et al. (2017), we pretrained the model on a dataset of 2.5 million Twitter message-response pairs,1 then fine-tuned it on PersonaChat. On the PersonaChat validation set, the baseline model has a perplexity of 26.83 and F1 of 17.02, which would have placed us 4th out of 26 models in the ConvAI2 competition Dinan et al. (2019). We attempt to improve over this baseline using control.

Footnote 1: The Twitter dataset is provided in ParlAI; details can be found here: [https://parl.ai/docs/tasks.html](https://parl.ai/docs/tasks.html)

## 5 Controllable text generation methods

Suppose we have a sequence-to-sequence model which gives \(P(y|x)=\Pi_{t}P(y_{t}|x,y_{1},\ldots,y_{t-1})\), the conditional probability of a response \(y\) (the model's next utterance) given input \(x\) (the context, which in our case includes the model's own persona and the dialogue history).

Contrary to most previous work, which controls _at the sentence level_, we wish to control attributes of the output \(y\)_at the dialogue level_ - meaning that a single control setting is used for a whole dialogue. For example, to control question-asking, we provide a control setting at the beginning of each dialogue (e.g. _20% questions_ or _70% questions_) rather than providing a control setting for each utterance (e.g. _is a question_ or _ isn't a question_). With this approach, the sequence-to-sequence model is able to choose what value the controlled attribute should take for any particular utterance, but we are able to choose the overall distribution. We find that this approach works well - for example, the sequence-to-sequence model is generally good at detecting when to ask a question. In particular, this is easier than the alternative: developing a separate process to decide, for each utterance, whether to ask a question.

In this section, we describe the two methods - which we call Conditional Training (CT) and Weighted Decoding (WD) - that we use to control attributes of the output \(y\) at the dialogue level.

### Conditional Training (CT)

Conditional Training Fan et al. (2018); Kikuchi et al. (2016); Peng et al. (2018) is a method to learn a sequence-to-sequence model \(P(y|x,z)\), where \(z\) is a discrete _control variable_. If the control attribute is naturally continuous (for example in our work, repetitiveness, specificity and response-relatedness), we use \(z\) to represent bucketed ranges. For a binary attribute like question-asking, \(z\) represents an overall probability (as explained in Section 5).

To train a CT model, we first automatically annotate every \((x,y)\) pair in the training set with the attribute we wish to control (for example, whether \(y\) contains a question mark). During training, for each example we determine the corresponding \(z\) value (for continuous attributes, this simply means sorting into the correct bucket; for question-asking, see Section 6.4). Next, the control variable \(z\) is represented via an embedding (each of the possible values of \(z\) has its own embedding). For all our experiments, the embedding is of length 10; this was determined via hyperparameter tuning. There are several possible ways to condition the sequence-to-sequence model on \(z\) - for example, append \(z\) to the end of the input sequence, or use \(z\) as the START symbol for the decoder. We find it most effective to concatenate \(z\) to the decoder's input on every step.2 Lastly, the CT model learns to produce \(y=y_{1},\dots,y_{T}\) by optimizing the cross-entropy loss:

Footnote 2: To build a CT model \(P(y|x,z_{1},\dots,z_{n})\) conditioned on _multiple_ controls \(\{z_{1},\dots,z_{n}\}\), we can simply concatenate multiple control embeddings to the decoder inputs.

\[\text{loss}_{\text{CT}}=-\frac{1}{T}\sum_{t=1}^{T}\log P(y_{t}|x,z,y_{1}, \dots,y_{t-1})\] 



### Weighted Decoding (WD)

Weighted Decoding Ghazvininejad et al. (2017) is a decoding method that increases or decreases the probability of words with certain features. The technique is applied only at test time, requiring no change to the training method. A limitation of WD is that the controllable attribute must be defined at the word-level; any desired utterance-level attribute must be redefined via word-level features.

In weighted decoding, on the \(t^{th}\) step of decoding, a partial hypothesis \(y_{<t}=y_{1},\ldots,y_{t-1}\) is expanded by computing the score for each possible next word \(w\) in the vocabulary:

\[\text{score}(w,y_{<t};x)=\text{score}(y_{<t};x)\\ +\log P_{\text{RNN}}(w|y_{<t},x)+\sum_{i}w_{i}*f_{i}(w;y_{<t},x).\]

Here, \(\log P_{\text{RNN}}(w|y_{<t},x)\) is the log-probability of the word \(w\) calculated by the RNN, \(\text{score}(y_{<t};x)\) is the accumulated score of the already-generated words in the hypothesis \(y_{<t}\), and \(f_{i}(w;y_{<t},x)\) are _decoding features_ with associated weights \(w_{i}\). There can be multiple features \(f_{i}\) (to control multiple attributes), and the weights \(w_{i}\) are hyperparameters to be chosen.

A decoding feature \(f_{i}(w;y_{<t},x)\) assigns a real value to the word \(w\), in the context of the text generated so far \(y_{<t}\) and the context \(x\). The feature can be continuous (e.g. the unigram probability of \(w\)), discrete (e.g. the length of \(w\) in characters), or binary (e.g. whether \(w\) starts with the same letter as the last word in \(y_{<t}\)). A positive weight \(w_{i}\) increases the probability of words \(w\) that score highly with respect to \(f_{i}\); a negative weight decreases their probability.

Note that weighted decoding and conditional training can be applied simultaneously (i.e. train a CT model then apply WD at test time) - a strategy we use in our experiments.

## 6 Controlling conversational attributes

In this section, we describe how we use conditional training and weighted decoding to control four attributes: repetition, specificity, response-relatedness and question-asking. We evaluate the effectiveness of both control methods via automatic metrics (i.e., measuring how well the attribute was controlled), and use our findings to select control methods and control settings to be explored further via human evaluation (Section 8).

### Repetition

Our baseline model exhibits three types of repetition, which we call _external repetition_ (self-repetition across utterances), _internal repetition_ (self-repetition within utterances), and _partner repetition_ (repeating the conversational partner).

To control repetition with weighted decoding,3 we define five \(n\)-gram based decoding features (see Appendix D). Three of these features (extrep_bigram, intrep_bigram and partnerrep_bigram) identify repeating bigrams for the three repetition types. The other two features (extrep_unigram and interp_unigram) identify repeating content words. By applying a negative weight to these features, we can reduce repetition. In particular, if the weight is \(-\infty\), our method is equivalent to _n-gram blocking_ as described by Kulikov et al. (2018). We observe that repetition control is very important, thus all further control experiments include repetition control.

Footnote 3: We also tried controlling repetition with conditional training, defining \(z\) as the (bucketted) maximum ROUGE-L precision between the response \(y\) and the bot’s previous utterances. However, this method was unsuccessful because there are not enough repetitive examples in the training data for the model to learn the control. Experimenting with data augmentation to solve this problem is an area for future work.

### Specificity

Like many sequence-to-sequence models using beam search decoding, our baseline frequently asks generic questions such as _What music do you like?_ and gives dull, unspecific responses, such as _I like all kinds of music_.

We control specificity using Normalized Inverse Document Frequency (NIDF) as a measure of word rareness.4 The Inverse Document Frequency of a word \(w\) is IDF\((w)=\log(R/c_{w})\) where \(R\) is the number of responses in the dataset, and \(c_{w}\) is the number of those responses that contain \(w\). Normalized IDF (which ranges from 0 to 1) is

Footnote 4: Note that our NIDF specificity features are similar to the NIRF and NIWF features used by Zhang et al. (2018a).

\[\text{NIDF}(w)=\frac{\text{IDF}(w)-\texttt{min\_idf}}{\texttt{max\_idf}- \texttt{min\_idf}} \tag{1}\]

where min_idf and max_idf are the minimum and maximum IDFs, taken over all words in the vocabulary. To control specificity with weighted decoding, we use NIDF as a decoding feature. As shown in Table 1, this method produces reasonable outputs when the feature weight is within a certain range, but at the extremes the outputs are

nonsensical. The boundary for nonsensical output differs from example to example.

To control specificity with conditional training, we define the specificity of an utterance \(y\) to be the mean NIDF of the words in \(y\). Thus our control variable \(z\) is mean NIDF (discretized into 10 equal-sized buckets). As shown in Table 1, this method gives outputs with a narrower NIDF range, but overall produces less nonsensical outputs.

### Response-relatedness

In conversation, it's generally desirable to produce a response that is related to the partner's last utterance; for example if the partner says _My grandfather died last month_, it is appropriate to say _I'm so sorry. Were you close to your grandfather?_ However, our baseline model frequently responds with unrelated utterances like _Do you have any pets?_

To control response-relatedness with weighted decoding, we use the decoding feature resp_rel:

resp_rel(w; y<t,x)= cossim(word_emb(w),sent_emb(f)) where word.emb(w) is the GloVe embedding for the word \(w\), sent_emb(f) is the sentence embedding for the partner's last utterance \(\ell\) (note \(\ell\) is part of the context \(x\)), and cos_sim is the cosine similarity between the two. In particular, the sentence embedding sent_emb(s) for an utterance \(s\) is a weighted average of the GloVe embeddings of the words in \(s\), with the first principal component projected out; for full details, see Arora et al. (2017). This method of controlling response-relatedness is similar to that described in Baheti et al. (2018). We find that weighted decoding is effective to control the semantic relatedness of the model's response to the partner's last utterance (see Table 2). As before, we find that extreme weights lead to nonsensical output.

To control response-relatedness with conditional training, we try defining the control variable \(z\) to be cossim(sent_emb(y),sent_emb(f)), the overall cosine similarity between the partner's last utterance \(\ell\) and the model's response \(y\) (again, we discretize \(z\)). However, we find this method ineffective - the CT model learns only a very weak connection between \(z\) and the semantic relatedness of the output (see Section 7 for more details).

### Question-asking

Considerate chitchat requires a reciprocal asking and answering of questions - asking too few or too many can appear self-centered or nosy. We control question-asking in order to study these trade-offs.

To control question-asking with weighted decoding, we use the binary decoding feature isqn_word(w), which is equal to 1 if and only if the word \(w\) is in a pre-defined list of interrogative words (_how, what, when, where, which, who, whom, whose, why, ?_). We find this is a somewhat effective method to encourage or discourage questions, but with unintended side-effects: a negative weight can discourage valid non-question utterances that happen to contain interrogative words (such as _I'm learning how to knit_) and a positive weight can result in degenerate utterances (such as

\begin{table}
\begin{tabular}{c c l} \hline \multicolumn{3}{l}{**Input:**_Yes, I’m studying law at the moment_} \\  & **Baseline Response:**_That sounds like a lot of fun!_ \\ \hline
**Wt** & **NIDF** & **Weighted Decoding Response** \\ -5.0 & 0.6\% & _Oh..._ \\
0.0 & 17.1\% & _That sounds like a lot of fun!_ \\
3.0 & 18.3\% & _That sounds like a lot of fun. How long have you been studying?_ \\
7.0 & 38.5\% & _I majored in practising my_ \\  & & _ spiritual fall time philosophy test_ \\
10.0 & 71.9\% & _Oh wover Meram ean is qua piano yi_ \\  & & _hao hui bu acara sya gila [...]_ \\ \hline \multirow{4}{*}{\(z\)} & **NIDF** & **Conditional Training Response** \\
0 & 16.8\% & _Sounds like you are a great person!_ \\
2 & 18.3\% & _So you are a law student?_ \\
4 & 18.4\% & _That sounds like a lot of fun_ \\
6 & 22.8\% & _That sounds like a rewarding job!_ \\
8 & 24.4\% & _That sounds like a rewarding career!_ \\ \hline \end{tabular}
\end{table}
Table 1: Middle: Example of controlling specificity (NIDF) via weighted decoding. At the extremes, the model produces only the most rare or the most common tokens. Bottom: Example of controlling specificity via conditional training. This gives a narrower NIDF range, but all the responses are appropriate.

\begin{table}
\begin{tabular}{r r l} \hline \multicolumn{3}{l}{**Input:**_Do you go get coffee often_} \\  & **Baseline Response:**_I do, when I am not playing the piano._ \\ \hline
**Wt** & **Sim** & **Weighted Decoding Response** \\ -10.0 & -0.05 & _I am a musician._ \\
0.0 & -0.02 & _I do, when I am not playing the piano._ \\
5.0 & 0.40 & _I do, usually at starbucks._ \\
8.0 & 0.59 & _Not usually, especially when you drink latte._ \\
11.0 & 0.72 & _Not often, usually with drinks,_ \\  & & _expresso, latte, tea, etc._ \\ \hline \end{tabular}
\end{table}
Table 2: Example of controlling response-relatedness (cosine similarity to input) via weighted decoding. Positive weights (e.g. 5.0) can yield more on-topic responses, but higher weights (e.g. 11.0) can result in nonsensical lists of topically related words.

 

_What???????_ or _Who? When? How?_).

For conditional training, we regard an utterance \(y\) as containing a question if and only if \(y\) contains a question mark. We train our CT model on a control variable \(z\) with 11 possible values: \(\{0,\dots,10\}\). As discussed in Section 5, we wish to control question-asking at the distributional, dialogue level, rather than at the binary, utterance level. Thus the setting \(z=i\) means that the model should produce, on average, utterances containing '?' with probability \(i/10\). During training we randomly assign examples to buckets such that each bucket \(i\) is trained on examples with the correct proportion of questions (\(i/10\)), and all buckets have the same amount of training examples.

We find that conditional training is effective to control question-asking - as shown in Figure 2, by increasing \(z\) from 0 to 10, we obtain a range of question-asking rates from 1.40% to 97.72%. However, when we introduce repetition control, question-asking is reduced - in particular, the \(z=10\) setting (which should produce 100% questions) now only produces 79.67% questions. The primary problem is the weighted decoding feature extrep_bigram, which discourages bigrams that have appeared in previous utterances - this prevents the model from producing bigrams that commonly occur in many questions, such as _do you_ and _what is_. To fix this, we introduce an extra setting \(z=10\)_(boost)_, in which we do not use the feature extrep_bigram for weighted decoding during beam search, but we do use it to rerank the candidates after beam search. This setting, which allows the model to produce necessary question-asking bigrams, yields a 99.54% question-asking rate, at the cost of slightly increased external bitgram repetition (see Appendix F).

For controlling question-asking, conditional training is preferable to weighted decoding for two reasons. Firstly, it allows us to achieve (close to) 0% questions, 100% questions, or anything in between, without introducing the risk of degenerate output. Secondly, presence-of-a-question-mark captures the true attribute of interest (question-asking) more exactly and directly than presence of interrogative words. For these reasons, only the CT method is considered in the human evaluation.

## 7 Comparison of control methods

The previous section shows that conditional training and weighted decoding are both useful techniques, with different strengths and weaknesses.

The primary disadvantage of conditional training is that it sometimes fails to learn the connection between the control variable \(z\) and the target output \(y\). In practice, we find the model can learn simple attributes of the output (such as the presence of '?', and overall genericness), but not relationships between the input and output (such as semantic relatedness). By contrast, weighted decoding can force the desired feature to appear in the output by raising the weight arbitrarily high (though this may have unintended side-effects).

The primary disadvantage of weighted decoding is that it risks going off-distribution when the weight is too strong. By contrast, conditional training produces mostly well-formed, indistribution outputs. This highlights the importance of learned control - it is safer to learn to produce output that both satisfies the control variable and is appropriate, than to alter the decoding process to satisfy the control variable, potentially trading off appropriateness in the process.

Other considerations include: (1) Convenience: conditional training requires retraining; weighted decoding doesn't, but is slower at test time. (2) Data availability: conditional training requires training examples of the controllable attribute, whereas weighted decoding can control any computable feature without requiring examples. (3) Attribute definition: conditional training can control sentence-level attributes, but they must be discrete. By contrast, weighted decoding requires word-level features, but they can be continuous.

## 8 Human evaluation results

In order to study the effect of our controllable attributes, we conduct a large-scale human evaluat

Figure 2: Controlling question-asking via conditional training. Exact numbers can be found in Appendix F.

 

tion of 28 model configurations (see Appendix E), plus human-human conversations for comparison.

ApproachIn our evaluation, a crowdworker chats with a model (or in the human-human case, another crowdworker) for six conversational turns, then answers eight multiple-choice questions which each capture different aspects of conversational quality: avoiding repetition, interestingness, making sense, fluency, listening, inquisitiveness, humanness and engagingness. The eight questions are Likert questions on a 1-4 scale, where higher is better.5 To match the ConvAI2 Challenge, we also add a persona retrieval question, in which the crowdworker is asked to select which of two possible personas was the model's persona. For full details of the evaluation design, see Appendix B.

Footnote 5: Exceptions: Avoiding repetition is a 1-3 scale, as we found this gave clearer instructions. Inquisitiveness has an optimal score of 3; 1 and 2 represent too little question-asking, and 4 represents to much.

Our evaluation is the same as the ConvAI2 Challenge evaluation, but more detailed - ConvAI2 includes only engagingness and persona retrieval.6 As in the ConvAI2 challenge, each of our 28 model configurations was evaluated by over 100 crowdworkers, and the results were adjusted for annotator variance via a Bayesian calibration Kulikov et al. (2018).

Footnote 6: There are three other minor differences between our evaluation and ConvAI2’s: (1) We fix capitalization and spacing before showing the chatbot’s utterances to crowdworkers, while ConvAI2 show the raw lowercase tokenized form. We found the latter interferes with fluency evaluation. (2) We conduct 6 dialogue turns, while ConvAI2 conducts 4-6. This was necessary to evaluate repetitiveness. (3) We use (publicly-available) validation set personas, while ConvAI2 uses (hidden) test set personas. This enables us to release our evaluation chatlogs.

In designing our evaluation, we aimed to capture the four aspects we expected to directly improve via control (avoiding repetition, interestingness, listening, inquisitiveness), two important error classes we thought would be affected by our controls (fluency, making sense), and two overall quality measures (engagingness, humanness).

### Main findings

In this section we summarize the main findings of our human evaluation - whose full results can be found in Appendices G and H, with sample conversations in Appendix C.

As Figure 3 shows, controlling for repetition, specificity and question-asking all lead to large engagingness improvements over the greedy and beam-search baseline models. In particular, we find that controlling for multi-turn (self) repetition is important and should be incorporated alongside other attribute control methods. We found no improvement by controlling response-relatedness.

To better understand these overall engagingness improvements, we consider the full set of human judgments, shown in Figure 4. We find that reducing repetition leads to improvements across all our aspects of conversational quality. Increasing specificity shows improvements in interestingness and listening ability over the repetition-controlled baseline, while increasing question-asking shows improvements in inquisitiveness and interestingness over the repetition-controlled baseline.

Our most engaging model, which controls both repetition and question-asking - marked 'Question (CT)' in Figure 3 (left) - matches the engagingness of the winning entry in the ConvAI2 competition, as both models achieve a raw score7 of \(3.1\)Dinan et al. (2019). However, the ConvAI2 winner, Lost in Conversation, was trained on approximately 12\(\times\) as much data as our model. Lost in Conversation is based on the OpenAI GPT Language Model Radford et al. (2018), which is pretrained on the BookCorpus Zhu et al. (2015), which contains approximately 985 million words, whereas our model is pretrained on the Twitter dataset (approximately 79 million words).

Footnote 7: Although the same Bayesian calibration method was applied both in our study and in the ConvAI2 competition, calibrated scores are not comparable across the two; thus we compare raw scores (viewable in Table 7).

Altogether, our evaluation clearly shows that controlling low-level attributes over multiple turns leads to improved overall quality.

### Effect of controlled attributes

Repetition (WD)We observe that self-repetition across utterances (_external repetition_) is by far the most severe form of repetition in our beam search baseline model. We evaluate several settings of the extrep_bigram weighted decoding feature, and find that an aggressive repetition-reduction setting (reducing bigram repetition rate to below gold data levels) is rated best. We also find that blocking repeated content words improves the avoiding repetition score. See Appendices E, F and G for full details.

As shown in Figure 3 (left) and Figure 4, our repetition-controlled model improves hugely 

over the beam search baseline in all metrics, and achieves close-to-human scores on all metrics except humanness. This striking result demonstrates that repetition is by far the biggest limiting quality factor for naive sequence-to-sequence dialogue agents. The result also emphasizes the importance of _multi-turn_ dialogue evaluation to detect the problem. We refer to this model as the _repetition-controlled baseline_, and use it as a basis for all remaining experiments (i.e., we control specificity, response-relatedness and question-asking on top of these repetition-control settings).

Specificity (WD, CT)For our weighted decoding models, the extreme settings (very generic and very specific) score poorly in engagingness due to the frequent presence of degenerate output - see Figure 3 (middle). We find that the weight \(=4\) setting (which is more specific than the repetition-controlled baseline and about as specific as the gold data) maximizes engagingness. As shown in Figure 3 (left) and Figure 4, this more-specific model is rated more interesting, engaging, and a better listener than the repetition-controlled baseline, but at the cost of reduced fluency and making sense. Our CT model with \(z=7\) (which has a similar NIDF level as WD with weight \(=4\)) shows similar results, but the improvements are smaller. For further discussion on the interestingness of our specificity models, see Section 8.3.

Response-relatedness (Wd)We evaluated several control settings (weight \(=-10,5,10,13\)) and found that none scored better than weight \(=0\) (no response-relatedness control); see Appendix H. This is surprising - prior to running the human evaluation, we annotated 100 examples ourselves to determine the best control settings. While we identified a more responsive setting (weight \(=5\)) as less likely than the uncontrolled model to ignore the user, crowdworkers rated it as a slightly _worse_ listener than the uncontrolled model. One explanation for this discrepancy is that the more responsive model takes more risks, using more rare words (0.197 NIDF, up from 0.178), and thus receives a lower makes-sense score (3.41, down from 3.70). We hypothesize that, compared to us, the crowdworkers are less tolerant of slightly non-sensical output, and more tolerant of generic unrelated utterances.

Question-asking (CT)As shown in Figure 3 (right), a question-asking rate of 65.7% (\(z=7\)) maximizes engagingness. This setting, which asks more questions than both the repetition-controlled baseline (50.0%) and the human-produced gold data (28.8%), brings us closest to human-level engagingness - see Figure 3 (left). Although we find that a rate of approximately 65.7% questionable is the most engaging, a lower level (48.9%, or \(z=4\)) is rated the best listener. Lastly, we find

Figure 4: Calibrated human judgments of conversational aspects for the baselines and best controlled models. Note: In Figure 3 and here, the Specificity and Question controlled models both include Repetition control, but Question control doesn’t include Specificity control, or vice versa.

Figure 3: Calibrated human judgments of engagingness for the baselines and best controlled models (left); for different specificity control settings (middle); and for different question-asking control settings (right).

 

that although asking too many questions is less engaging, most crowdworkers will not directly criticize a chatbot that asks questions on every turn - only 11.9% of crowdworkers judged the \(z=10\)_(boost)_ setting, which asks 99.5% questions, as asking too many questions.8 For full details of these scores, see Appendix F and H.

Footnote 8: Though this conclusion may hold true for the PersonaChat task – a synthetic chatting task that instructs participants to get to know each other – in real-life social conversations, incessant question-asking may be less tolerated.

For time and budget reasons, we did not evaluate any models controlling both question-asking and specificity. However, we expect it is possible to obtain further improvements by doing so.

### A/B tests for interestingness

Though our more-specific models yielded significant improvements in engagingness, we were surprised that they did not yield clearer improvements in interestingness. To investigate further, we conducted an A/B interestingness evaluation of three specificity-controlled models, compared to the repetition-controlled baseline. Crowdworkers were shown two conversations (from the main human evaluation) and asked to choose which model was more interesting (see Figure 7 for details). We collected 500 samples per comparison, plus 200 additional human vs repetition-controlled baseline samples, which were used to filter for quality control. After discarding low-quality crowdworkers, we have roughly 300 evaluations per comparison, with an average Cohen's \(\kappa=0.6\).

As shown in Table 3, all three models were rated significantly more interesting than the repetition-controlled baseline. This convincingly shows that producing utterances with more rare words is a valid strategy to improve interestingness. We have two explanations for why these interestingness differences did not materialize in our main evaluation. Firstly, interestingness is a particularly subjective metric (unlike more tangible metrics such as avoiding repetition and making sense) - this makes it hard to calibrate across crowdworkers. Secondly, we suspect that in our original evaluation, the crowdworkers may have evaluated the interestingness of the _task_ rather than the _chatbot_. This could account for why subtle increases in conversational ability did not result in higher interestingness ratings - the PersonaChat task itself has a natural interestingness limit.

## 9 Conclusion

What makes a good conversation?Through our evaluation, we showed that a good conversation is about balance - controlling for the right level of repetition, specificity and question-asking is important for overall quality. We also found that conversational aspects such as interestingness, listening, and inquisitiveness are all important - though optimizing these can introduce a trade-off against certain types of errors (such as repetitive, disfluent, or nonsensical output). Secondly, multiturn evaluation is essential to study what makes a good conversation - multiple turns are required to reveal issues such as repetition, consistency, and question-asking frequency. Lastly, what do we mean by 'good'? Although humanness and engagingness are both commonly used as overall quality metrics, the two are very different. While our models achieved close-to-human scores on engagingness, they failed to get close on humanness - showing that a chatbot need not be human-like to be enjoyable. This striking result also demonstrates the importance of measuring more than one quality metric when evaluating dialogue agents.

OutlookOur work shows that neural generative systems have systemic problems when applied to open-ended dialogue, some of which (e.g. repetition) are only observable in the multi-turn setting. Furthermore, control of low-level attributes offers a practical way to correct these problems, yielding large improvements to overall quality - in our case, comparable to systems trained on much more data. Future work includes optimizing control settings automatically, and building more convincingly human-like chatbots.

\begin{table}
\begin{tabular}{l c l} \hline \hline
**Model** & **Win\%** & **Top 3 reasons for preferring model** \\ \hline Specificity WD (weight \(=6\)) & 84.1\% & _More information; Better flow; More descriptive_ \\ Specificity WD (weight \(=4\)) & 75.5\% & _More information; They describe their life in more detail; Funny_ \\ Specificity CT (\(z=7\)) & 56.2\% & _More information; Better flow; Seems more interested_ \\ \hline \hline \end{tabular}
\end{table}
Table 3: A/B tests comparing various specificity-controlled models to the repetition-controlled baseline on interstegnress. We find all comparisons are significant (\(p<.05\); binomial test).

 



## References

* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Baheti et al. (2018) Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. 2018. Generating more interesting responses in neural conversation models with distributional constraints. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3970-3980. Association for Computational Linguistics.
* Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Dinan et al. (2019) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). _arXiv preprint arXiv:1902.00098_.
* Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. _arXiv preprint arXiv:1811.01241_.
* Asri et al. (2017) Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In _Proceedings of the 18th Annual SIGDIAL Meeting on Discourse and Dialogue_, pages 207-219, Saarbrucken, Germany. Association for Computational Linguistics.
* Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 45-54. Association for Computational Linguistics.
* Ficler and Goldberg (2017) Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In _Proceedings of the Workshop on Stylistic Variation_, pages 94-104. Association for Computational Linguistics.
* Ghazvininejad et al. (2017) Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In _Proceedings of ACL 2017, System Demonstrations_, pages 43-48. Association for Computational Linguistics.
* Guo et al. (2018) Fenfei Guo, Angeliki Metallinou, Chandra Khartri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. _Advances in Neural Information Processing Systems, Conversational AI Workshop_.
* Hastie (2012) Helen Hastie. 2012. _Metrics and evaluation of spoken dialogue systems_, pages 131-150. Springer.
* Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_, pages 263-272.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In _Thirty-fourth International Conference on Machine Learning_.
* Kikuchi et al. (2016) Yuta Kikuchi, Graham Neubig, Rohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1328-1338. Association for Computational Linguistics.
* Kulikov et al. (2018) Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling. _arXiv preprint arXiv:1811.00907_.
* Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 110-119. Association for Computational Linguistics.
* Li et al. (2017a) Jiwei Li, Will Monroe, and Dan Jurafsky. 2017a. Learning to decode for future success. _arXiv preprint arXiv:1701.06549_.
* Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep

reinforcement learning for dialogue generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
* Li et al. (2017b) Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean, Alan Ritter, and Dan Jurafsky. 2017b. Adversarial learning for neural dialogue generation. _arXiv preprint arXiv:1701.06547_.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. pages 2122-2132.
* Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1116-1126. Association for Computational Linguistics.
* Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 285-294, Prague, Czech Republic. Association for Computational Linguistics.
* Miller et al. (2017) Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 79-84, Copenhagen, Denmark. Association for Computational Linguistics.
* Novikova et al. (2017) Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2241-2252.
* Parthasarathi and Pineau (2018) Prasanna Parthasarathi and Joelle Pineau. 2018. Extending neural generative conversational model using external knowledge sources. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 690-695, Brussels, Belgium. Association for Computational Linguistics.
* Peng et al. (2018) Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In _Proceedings of the First Workshop on Storytelling_, pages 43-49. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Serban et al. (2016a) Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016a. Generative deep neural networks for dialogue: A short review. _Advances in Neural Information Processing Systems workshop on Learning Methods for Dialogue_.
* Serban et al. (2016b) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016b. Building end-to-end dialogue systems using generative hierarchical neural network models. In _AAAI_, volume 16, pages 3776-3784.
* Shen et al. (2017) Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017. A conditional variational framework for dialog generation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 504-509. Association for Computational Linguistics.
* Venkatesh et al. (2017) Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2017. On evaluating and comparing conversational agents. _Advances in Neural Information Processing Systems, Conversational AI Workshop_.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In _Proceedings of the 31st International Conference on Machine Learning, Deep Learning Workshop_, Lille, France 

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella (1997)PARADISE: a framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, Madrid, Spain, pp. 271-280. External Links: Document Cited by: SS4.
* [D. Wang, N. Jojic, C. Brockett, and E. Nyberg (2017) Steering output style and topic in neural response generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2140-2150. External Links: Link, Document Cited by: SS4.
* [T. Wen, D. Vandyke, M. Gasic, L. M. Rojas Barahona, P. Su, S. Ultes, and S. Young (2017) A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 438-449. External Links: Link, Document Cited by: SS4.
* [Cited by: SS4.
 



## Appendix A Screenshots of human evaluation interface

Figure 5: Screenshot of the Task Description

Figure 6: Screenshot of the chat UI, talking with the beam search baseline model.



Figure 7: Screenshot of the A/B test UI, comparing a human-human conversation (left) and a Repetition-controlled baseline model (right).

 

Human evaluation questionnaire design

Here are the questions and multiple-choice options used in the human evaluation, in the order presented:

\begin{tabular}{|l|} \hline
**[Engagingness] How much did you enjoy talking to this user?** \\ \(\bullet\) Not at all \(\bullet\) A little \(\bullet\) Somewhat \(\bullet\) A lot \\
**[Interestingness] How interesting or boring did you find this conversation?** \\ \(\bullet\) Very boring \(\bullet\) A little boring \(\bullet\) A little interesting \(\bullet\) Very interesting \\
**[Inquistiveness] How much did the user try to get to know you?** \\ \(\bullet\) Didn't ask about me at all \(\bullet\) Asked about me some \\ \(\bullet\) Asked about me a good amount \(\bullet\) Asked about me too much \\
**[Listening] How much did the user seem to pay attention to what you said?** \\ \(\bullet\) Always ignored what I said \(\bullet\) Mostly ignored what I said \\ \(\bullet\) Mostly paid attention to what I said \(\bullet\) Always paid attention to what I said \\
**[Avoiding Repetition] How repetitive was this user?** \\ \(\bullet\) Repeated themselves over and over \(\bullet\) Sometimes said the same thing twice \\ \(\bullet\) Always said something new \\
**[Fluency] How naturally did this user speak English?** \\ \(\bullet\) Very unnatural \(\bullet\) Mostly unnatural \(\bullet\) Mostly natural \(\bullet\) Very natural \\
**[Making sense] How often did this user say something which did NOT make sense?** \\ \(\bullet\) Never made any sense \(\bullet\) Most responses didn't make sense \\ \(\bullet\) Some responses didn't make sense \(\bullet\) Everything made perfect sense \\
**[Humanness] Do you think this user is a bot or a human?** \\ \(\bullet\) Definitely a bot \(\bullet\) Probably a bot \(\bullet\) Probably a human \(\bullet\) Definitely a human \\
**[Persona retrieval] Which prompt (character) do you think the other user was given for this conversation?** \\ _Respondent chooses one of two provided personas_ 



## Appendix C Example conversations from human evaluation

[FIGURE:A1.F8][E

Figure 8: Example conversation with (a) Baseline (b) Repetition-controlled baseline (c) Question-controlled CT (\(z=7\)), (d) Specificity-controlled WD (weight \(=4\)).





## Appendix D Repetition-control decoding features

\begin{tabular}{p{142.3pt} p{170.7pt}} \hline \hline
**Feature** & **Condition** \\ \hline extrep_bigram(\(w,y_{<t},x\)) & Adding \(w\) to the hypothesis \(y_{<t}\) would create a 2-gram that appears in a previous utterance by the model \\ \hline extrep_unigram(\(w,y_{<t},x\)) & \(w\) is a non-stopword **and** \\  & \(w\) appears in a previous utterance by the model \\ \hline intrep_bigram(\(w,y_{<t},x\)) & Adding \(w\) to the hypothesis \(y_{<t}\) would create a 2-gram that appears earlier in the hypothesis \(y_{<t}\) \\ \hline intrep_unigram(\(w,y_{<t},x\)) & \(w\) is a non-stopword **and** \\  & \(w\) appears earlier in the hypothesis \(y_{<t}\) \\ \hline partnerrep_bigram(\(w,y_{<t},x\)) & Adding \(w\) to the hypothesis \(y_{<t}\) would create a 2-gram that appears in a previous utterance by the partner \\ \hline \hline \end{tabular} Table 4: We define five binary features for controlling different types of repetition via weighted decoding (see Section 5.2). Each feature depends on the word \(w\), the partial hypothesis \(y_{<t}\), and the context \(x\) (which includes the model's own persona and the dialogue history). Each of these features is equal to 1 if and only if the condition on the right is true; otherwise 0.





## Appendix E Control settings for all configurations

* In the setting Question-controlled CT 10 (boost), the feature extrep_bigram is _not_ used for weighted decoding during beam search, but it \(i\)s used to rerank the candidates after beam search. See Section 6.4 for details.

** Note that the Response-related controlled models additionally introduce repetition controls to block internal bigram repetition and partner bigram repetition. This was necessary to prevent the model from parroting the partner's last utterance. In Table 8, we find that just adding these extra repetition controls (here called Response-related controlled WD 0, i.e. increased repetition control but no response-relatedness control) outperforms our canonical Repetition-controlled baseline. However, given that we discovered this later, our specificity and question controlled models are built on top of the canonical Repetition-controlled baseline.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline  & \multicolumn{3}{c}{**Repetition**} & **Specificity** & **Response-rel** & **Questions** \\ \cline{2-7}  & \multicolumn{2}{c}{**External**} & \multicolumn{2}{c}{**Internal**} & **Partner Rep.** & & \\ \cline{2-7}  & **Bigram** & **Unigram** & **Bigram** & **Unigram** & **Bigram** & **NDP** & **Cos sim** & **Has ‘?’ \\ \hline
**Baselines** & & & & & & & \\ Greedy Search & Beam Search (beam size 20) & & & & & & \\ \hline
**Repetition control (WD)** & & & & & & & \\ Extrep bigram WD -0.5 & wt -0.5 & & & & & & \\ Extrep bigram WD -1.25 & wt -1.25 & & & & & & \\ Extrep bigram WD -3.5 & wt -3.5 & & & & & & \\ Every bigram WD -inf & wt -so & & & & & \\ Repetition-controlled baseline & wt -3.5 & wt -so & wt -so & & & \\ \hline
**Question control (CT)** & & & & & & & \\ Question-control CLT 0 & wt -3.5 & wt -so & wt -so & & \(z=0\) \\ Question-controlled CT 1 & wt -3.5 & wt -so & wt -so & & \(z=1\) \\ Question-controller CLT 4 & wt -3.5 & wt -so & wt -so & & \(z=4\) \\ Question-controlled CT 7 & wt -3.5 & wt -so & wt -so & & \(z=7\) \\ Question-controlled CT 10 & wt -3.5 & wt -so & wt -so & & \(z=10\) \\ Question-controlled CT 10 (boost) & wt 0* & wt -so & wt -so & & \(z=10\) \\ \hline
**Specificity control (CT)** & & & & & & \\ Specificity-controlled CT 0 & wt -3.5 & wt -so & wt -so & \(z=0\) & \\ Specificity-controlled CTL T 2 & wt -3.5 & wt -so & wt -so & \(z=2\) & \\ Specificity-controlled CT 4 & wt -3.5 & wt -so & wt -so & \(z=4\) & \\ Specificity-controlled CT 7 & wt -3.5 & wt -so & wt -so & \(z=7\) & \\ Specificity-controlled CT 9 & wt -3.5 & wt -so & wt -so & \(z=9\) & \\ \hline
**Speciticty control (WD)** & & & & & & \\ Specificity-controlled WD -10 & wt -3.5 & wt -so & wt -so & wt -to & \\ Specificity-controlled WD -4 & wt -3.5 & wt -so & wt -so & wt -4 & \\ Specificity-controlled WD 6 & wt -3.5 & wt -so & wt -so & wt 6 & \\ Specificity-controlled WD 8 & wt -3.5 & wt -so & wt -so & wt 8 & \\ \hline
**Response-related control (WD)** & & & & & & \\ Response-related controlled WD -10 & wt -3.5 & wt -so & wt -so & wt -so & wt -so & wt -10 \\ Response-related controlled WD 0 & wt -3.5 & wt -so & wt -so & wt -so & wt -so & wt 0 \\ Response-related controlled WD 5 & wt -3.5 & wt -so & wt -so & wt -so & wt -so & wt 10 \\ Response-related controlled WD 10 & wt -3.5 & wt -so & wt -so & wt -so & wt -so & wt 13 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Control settings for all configurations that were human-evaluated. ’wt’ means the weight used for a weighted decoding feature and ‘\(z=\)’ means the setting (i.e. bucket) for the control variable in conditional training.





## Appendix F Automatic metrics for all configurations

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Repetition**} & \multicolumn{2}{c}{**Specificity**} & \multicolumn{1}{c}{**Response-rel**} & \multicolumn{1}{c}{**Questions**} \\ \cline{2-9}  & \multicolumn{3}{c}{**External**} & \multicolumn{3}{c}{**Internal**} & \multicolumn{1}{c}{**Partner Rep.**} & \multicolumn{1}{c}{**NDP**} & \multicolumn{1}{c}{**Cos sim**} & \multicolumn{1}{c}{**Has ‘?”**} \\ \cline{2-9}  & **Bigram** & **Unigram** & **Bigram** & **Unigram** & **Bigram** & **Biggram** & **NDF** & **Cos sim** & **Has ‘?”** \\ \hline
**Gold data and baselines** & & & & & & & & \\ Gold Data & 4.65\% & 9.62\% & 0.38\% & 0.97\% & 5.10\% & 0.2119 & 0.1691 & 28.80\% \\ Greedy Search & 35.88\% & 36.31\% & 8.08\% & 10.59\% & 12.20\% & 0.1688 & 0.1850 & 6.46\% \\ Beam Search (beam size 20) & 46.85\% & 44.15\% & 0.32\% & 0.61\% & 12.90\% & 0.1662 & 0.0957 & 80.87\% \\ \hline
**Repetition control (WD)** & & & & & & & & \\ Extrep bigram WD -0.5 & 19.70\% & 16.85\% & 0.26\% & 0.62\% & 11.93\% & 0.1730 & 0.1348 & 73.04\% \\ Extrep bigram WD -1.25 & 4.62\% & 4.79\% & 0.40\% & 0.89\% & 10.61\% & 0.1763 & 0.1504 & 61.22\% \\ Extrep bigram WD -3.5 & 0.75\% & 4.61\% & 0.47\% & 0.94\% & 9.89\% & 0.1771 & 0.1681 & 48.89\% \\ Extrep bigram WD -inf & 0.00\% & 4.74\% & 0.51\% & 1.05\% & 9.56\% & 0.1780 & 0.1711 & 45.98\% \\ Re repetition-controlled baseline & 0.73\% & 0.00\% & 0.17\% & 0.00\% & 9.55\% & 0.1766 & 0.1676 & 49.98\% \\ \hline
**Question control (CT)** & & & & & & & & \\ Question-controlled CT 0 & 0.06\% & 0.00\% & 0.19\% & 0.00\% & 9.20\% & 0.1871 & 0.1753 & 2.01\% \\ Question-controlled CT 1 & 0.09\% & 0.00\% & 0.19\% & 0.00\% & 8.66\% & 0.1844 & 0.1722 & 17.33\% \\ Question-controlled CT 4 & 0.40\% & 0.00\% & 0.25\% & 0.00\% & 8.53\% & 0.1794 & 0.1713 & 48.88\% \\ Question-controlled CT 7 & 0.80\% & 0.00\% & 0.17\% & 0.00\% & 8.48\% & 0.1771 & 0.1724 & 65.65\% \\ Question-control CT 10 (host)* & 1.27\% & 0.00\% & 0.16\% & 0.00\% & 8.48\% & 0.1761 & 0.1728 & 79.67\% \\ Question-controlled CT 10 (host)* & 7.64\% & 0.00\% & 0.03\% & 0.00\% & 10.76\% & 0.1701 & 0.1651 & 99.54\% \\ \hline
**Specificulty control (CT)** & & & & & & & & & \\ Specificity-controlled CT 0 & 0.60\% & 0.00\% & 0.20\% & 0.00\% & 9.05\% & 0.1478 & 0.1522 & 48.75\% \\ Specificity-controlled CT 2 & 0.28\% & 0.00\% & 0.10\% & 0.00\% & 8.37\% & 0.1772 & 0.1833 & 50.57\% \\ Specificity-controlled CT 4 & 0.12\% & 0.00\% & 0.08\% & 0.00\% & 7.90\% & 0.1921 & 0.1877 & 29.46\% \\ Specificity-controlled CT 7 & 0.02\% & 0.00\% & 0.14\% & 0.00\% & 8.17\% & 0.2156 & 0.1955 & 16.51\% \\ Specificity-controlled CT 9 & 0.01\% & 0.00\% & 0.11\% & 0.00\% & 8.01\% & 0.2462 & 0.1990 & 8.50\% \\ \hline
**Specifically control (WB)** & & & & & & & & \\ Specificity-controlled WD -10 & 0.14\% & 0.00\% & 10.59\% & 0.00\% & 8.70\% & 0.1107 & 0.0994 & 33.55\% \\ Specificity-controlled WD -4 & 0.65\% & 0.00\% & 1.98\% & 0.00\% & 9.95\% & 0.1501 & 0.1398 & 44.92\% \\ Specificity-controlled WD 0 & 0.15\% & 0.00\% & 0.19\% & 0.00\% & 7.54\% & 0.2121 & 0.1972 & 45.53\% \\ Specificity-controlled WD 6 & 0.07\% & 0.00\% & 0.13\% & 0.00\% & 6.50\% & 0.2546 & 0.2040 & 39.37\% \\ Specificity-controlled WD 8 & 0.01\% & 0.00\% & 0.10\% & 0.00\% & 3.40\% & 0.4035 & 0.1436 & 26.68\% \\ \hline
**Response-related control (WD)** & & & & & & & & \\ Response-related controlled WD -10 & 0.13\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.1914 & -0.0921 & 25.71\% \\ Response-related controlled WD 0 & 0.24\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.1785 & 0.1414 & 44.55\% \\ Response-related controlled WD 5 & 0.15\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.1973 & 0.4360 & 39.78\% \\ Response-related controlled WD 10 & 0.05\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.2535 & 0.6653 & 27.56\% \\ Response-related controlled WD 13 & 0.02\% & 0.00\% & 0.00\% & 0.00\% & 0.00\% & 0.2999 & 0.7251 & 20.47\% \\ \hline \end{tabular}
\end{table}
Table 6: Automatic metrics (computed over validation set) for all model configurations that were human-evaluated.





## Appendix G Human evaluation results for all configurations

The first eight columns are Likert metrics on a 1-4 scale (except Avoiding Repetition, which is a 1-3 scale), where higher is better (except Inquisitiveness, which has an optimal score of 3). The last column, Persona Retrieval, is on a scale from 0 to 1 where higher is better.

The maximum of each column (excluding Human row) is in bold.

\begin{table}
\begin{tabular}{l c c c c c c c c|c} \hline Model & Avoiding Rep. & Engage & Fluency & Humanness & Inquisitive & Interesting & Listening & Make Sense & Persona \\ \hline
**Human and baselines** & & & & & & & & & \\ Human & 2.90 \(\pm\) 0.39 & 3.31 \(\pm\) 0.90 & 3.66 \(\pm\) 0.71 & 3.40 \(\pm\) 0.80 & 2.63 \(\pm\) 0.63 & 3.23 \(\pm\) 0.83 & 3.64 \(\pm\) 0.63 & 3.84 \(\pm\) 0.52 & 0.92 \(\pm\) 0.27 \\ Greedy Search & 2.16 \(\pm\) 0.72 & 2.31 \(\pm\) 1.08 & 3.20 \(\pm\) 0.81 & 1.78 \(\pm\) 0.90 & 2.00 \(\pm\) 0.81 & 2.36 \(\pm\) 0.98 & 2.78 \(\pm\) 0.84 & 3.33 \(\pm\) 0.75 & 0.87 \(\pm\) 0.34 \\ Beam Search (beam size 20) & 2.14 \(\pm\) 0.72 & 2.35 \(\pm\) 1.01 & 3.23 \(\pm\) 0.93 & 1.81 \(\pm\) 0.87 & 2.50 \(\pm\) 0.72 & 2.35 \(\pm\) 0.98 & 2.63 \(\pm\) 0.85 & 3.40 \(\pm\) 0.77 & 0.77 \(\pm\) 0.42 \\ \hline
**Repetition control (WD)** & & & & & & & & & \\ Earup bigram WD -0.5 & 2.66 \(\pm\) 0.56 & 2.56 \(\pm\) 0.92 & 3.57 \(\pm\) 0.64 & 2.19 \(\pm\) 0.94 & 2.67 \(\pm\) 0.62 & 2.61 \(\pm\) 0.87 & 3.08 \(\pm\) 0.78 & 3.60 \(\pm\) 0.57 & 0.75 \(\pm\) 0.43 \\ Extrap bigram WD -1.25 & 2.84 \(\pm\) 0.39 & 2.91 \(\pm\) 0.90 & 3.59 \(\pm\) 0.64 & 2.32 \(\pm\) 0.98 & 2.63 \(\pm\) 0.60 & 2.86 \(\pm\) 0.89 & 3.21 \(\pm\) 0.71 & 3.64 \(\pm\) 0.62 & 0.72 \(\pm\) 0.45 \\ Extrap Bigram WD -3.5 & 2.90 \(\pm\) 0.30 & 2.95 \(\pm\) 0.86 & 3.73 \(\pm\) 0.90 & 2.45 \(\pm\) 1.03 & 2.55 \(\pm\) 0.61 & 2.88 \(\pm\) 0.80 & 3.27 \(\pm\) 0.79 & 3.68 \(\pm\) 0.49 & 0.80 \(\pm\) 0.40 \\ Extrap bigram WD -off & 2.82 \(\pm\) 0.43 & 2.96 \(\pm\) 0.86 & 3.64 \(\pm\) 0.85 & 2.40 \(\pm\) 0.96 & 2.65 \(\pm\) 0.69 & 2.86 \(\pm\) 0.82 & 3.31 \(\pm\) 0.69 & 3.66 \(\pm\) 0.59 & 0.91 \(\pm\) 0.29 \\ Reporting-controlled baseline & 2.89 \(\pm\) 0.39 & 2.89 \(\pm\) 0.89 & 3.66 \(\pm\) 0.56 & 2.50 \(\pm\) 0.99 & 2.70 \(\pm\) 0.64 & 2.96 \(\pm\) 0.92 & 3.25 \(\pm\) 0.71 & 3.68 \(\pm\) 0.54 & 0.87 \(\pm\) 0.34 \\ \hline
**Question control (CT)** & & & & & & & & & & \\ Question-control CT 0 & 2.95 \(\pm\) 0.25 & 2.92 \(\pm\) 0.90 & 3.70 \(\pm\) 0.54 & 2.49 \(\pm\) 0.97 & 2.48 \(\pm\) 0.72 & 2.85 \(\pm\) 0.93 & 3.29 \(\pm\) 0.69 & 3.56 \(\pm\) 0.66 & 0.86 \(\pm\) 0.35 \\ Question-controller CT 1 & 2.88 \(\pm\) 0.33 & 2.94 \(\pm\) 0.93 & 3.59 \(\pm\) 0.66 & 2.47 \(\pm\) 0.95 & 2.52 \(\pm\) 0.69 & 2.85 \(\pm\) 0.90 & 3.32 \(\pm\) 0.73 & 3.63 \(\pm\) 0.55 & 0.85 \(\pm\) 0.36 \\ Question-control CT 4 & 2.88 \(\pm\) 0.38 & 2.88 \(\pm\) 0.94 & 3.59 \(\pm\) 0.73 & 2.42 \(\pm\) 1.07 & 2.55 \(\pm\) 0.66 & 2.82 \(\pm\) 0.85 & **3.37 \(\pm\) 0.74** & 3.63 \(\pm\) 0.59 & 0.84 \(\pm\) 0.37 \\ Question-controll CTL C 7 & 2.88 \(\pm\) 0.37 & 3.87 \(\pm\) 0.99 & 3.67 \(\pm\) 0.54 & 2.42 \(\pm\) 0.98 & 2.75 \(\pm\) 0.58 & 2.97 \(\pm\) 0.84 & 3.32 \(\pm\) 0.76 & 3.35 \(\pm\) 0.76 & 3.08 \(\pm\) 0.40 \\ Question-controlled CT 10 & 2.74 \(\pm\) 0.46 & 2.90 \(\pm\) 0.93 & 3.70 \(\pm\) 0.50 & 2.43 \(\pm\) 1.04 & 2.71 \(\pm\) 0.57 & 2.72 \(\pm\) 0.88 & 3.12 \(\pm\) 0.73 & 3.59 \(\pm\) 0.66 & 0.79 \(\pm\) 0.41 \\ Question-controlled CT 10 (boost) & 2.76 \(\pm\) 0.49 & 2.84 \(\pm\) 0.94 & 3.60 \(\pm\) 0.64 & 2.26 \(\pm\) 0.97 & **2.94 \(\pm\) 0.57** & 2.83 \(\pm\) 0.94 & 3.18 \(\pm\) 0.80 & 3.52 \(\pm\) 0.67 & 0.72 \(\pm\) 0.45 \\ \hline
**Spefelicity control (CT)** & & & & & & & & & & \\ Society-centralized CT 2 & 2.83 \(\pm\) 0.40 & 2.96 \(\pm\) 0.93 & 3.62 \(\pm\) 0.58 & 2.42 \(\pm\) 0.99 & 2.60 \(\pm\) 0.56 & 2.86 \(\pm\) 0.89 & 3.29 \(\pm\) 0.70 & 3.66 \(\pm\) 0.60 & 0.72 \(\pm\) 0.45 \\ Specificity-controlled CT 2 & 2.90 \(\pm\) 0.36 & 2.78 \(\pm\) 1.00 & 3.60 \(\pm\) 0.64 & 2.37 \(\pm\) 0.93 & 2.66 \(\pm\) 0.66 & 2.80 \(\pm\) 0.96 & 3.14 \(\pm\) 0.77 & 3.50 \(\pm\) 0.63 & 0.81 \(\pm\) 0.39 \\ Specificity-controlled CT 2 & 2.92 \(\pm\) 0.22 & 2.81 \(\pm\) 0.83 & 3.65 \(\pm\) 0.59 & 2.34 \(\pm\) 1.02 & 2.57 \(\pm\) 0.62 & 2.86 \(\pm\) 0.78 & 3.25 \(\pm\) 0.78 & 3.50 \(\pm\) 0.78 & 3.50 \(\pm\) 0.66 & 0.86 \(\pm\) 0.35 \\ Specificity-controlled CT 7 & 2.89 \(\pm\) 0.32 & 3.00 \(\pm\) 0.94 & 3.64 \(\pm\) 0.67 & 2.53 \(\pm\) 1.03 & 2.56 \(\pm\) 0.66 &

The first eight columns are Likert metrics on a 1-4 scale (except Avoiding Repetition, which is a 1-3-scale), where higher is better (except Inquisitiveness, which has an optimal score of 3). The last column, Personal Retrieval, is on a scale from 0 to 1 where higher is better.

The maximum of each column (excluding Human row) is in bold.

Rows marked with * are the six models included in Figure 3 (left) and Figure 4.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline Model & Avoiding Rep. & Engage & Fluency & Hummness & Inquisitive & Interesting & Listening & Make Sense \\ \hline
* & & & & & & & & & \\
* Human & 2.79 \(\pm\) 0.12 & 3.04 \(\pm\) 0.11 & 3.36 \(\pm\) 0.12 & 3.35 \(\pm\) 0.11 & 2.44 \(\pm\) 0.12 & 2.92 \(\pm\) 0.11 & 3.32 \(\pm\) 0.13 & 3.68 \(\pm\) 0.11 \\
* Greedy Search & 2.08 \(\pm\) 0.10 & 2.24 \(\pm\) 0.11 & 3.03 \(\pm\) 0.10 & 1.75 \(\pm\) 0.12 & 1.95 \(\pm\) 0.10 & 2.29 \(\pm\) 0.13 & 2.62 \(\pm\) 0.10 & 3.23 \(\pm\) 0.10 \\
* Beam Search (beam size 20) & 2.08 \(\pm\) 0.11 & 2.29 \(\pm\) 0.11 & 3.09 \(\pm\) 0.13 & 1.71 \(\pm\) 0.13 & 2.42 \(\pm\) 0.11 & 2.29 \(\pm\) 0.14 & 2.47 \(\pm\) 0.12 & 3.35 \(\pm\) 0.13 \\ \hline
* & & & & & & & & \\ Extrep bigram WD \textgreater{}0.5 & 2.62 \(\pm\) 0.10 & 2.54 \(\pm\) 0.12 & 3.35 \(\pm\) 0.12 & 2.13 \(\pm\) 0.11 & 2.63 \(\pm\) 0.11 & 2.56 \(\pm\) 0.11 & 2.93 \(\pm\) 0.11 & 3.48 \(\pm\) 0.11 \\ Extrep bigram WD \textgreater{}1.25 & 2.78 \(\pm\) 0.09 & 2.82 \(\pm\) 0.13 & 3.40 \(\pm\) 0.12 & 2.27 \(\pm\) 0.12 & 2.54 \(\pm\) 0.09 & 2.76 \(\pm\) 0.10 & 3.05 \(\pm\) 0.11 & 3.53 \(\pm\) 0.14 \\ Extrep bigram WD \textgreater{}3.5 & 2.83 \(\pm\) 0.11 & 2.93 \(\pm\) 0.10 & **3.56 \(\pm\) 0.10*
* & **2.43 \(\pm\) 0.11*
* & 2.47 \(\pm\) 0.11 & 2.83 \(\pm\) 0.10 & 3.14 \(\pm\) 0.10 & 3.62 \(\pm\) 0.12 \\ Extrep Bigram WD \textgreater{}nf & 2.74 \(\pm\) 0.11 & 2.87 \(\pm\) 0.14 & 3.49 \(\pm\) 0.12 & 2.32 \(\pm\) 0.13 & 2.56 \(\pm\) 0.11 & 2.75 \(\pm\) 0.12 & 3.13 \(\pm\) 0.12 & 3.59 \(\pm\) 0.12 \\
* Repetition-controlled baseline & 2.86 \(\pm\) 0.12 & 2.82 \(\pm\) 0.12 & 3.53 \(\pm\) 0.10 & 2.40 \(\pm\) 0.11 & 2.62 \(\pm\) 0.13 & 2.84 \(\pm\) 0.12 & 3.10 \(\pm\) 0.11 & 3.58 \(\pm\) 0.14 \\ \hline
* & & & & & & & & & \\ Question-controlled CT 0 & **2.87 \(\pm\) 0.12*
* & 2.84 \(\pm\) 0.13 & 3.51 \(\pm\) 0.10 & 2.46 \(\pm\) 0.11 & 2.36 \(\pm\) 0.09 & 2.76 \(\pm\) 0.09 & 3.10 \(\pm\) 0.10 & 3.49 \(\pm\) 0.12 \\ Question-controlled CT 1 & 2.82 \(\pm\) 0.11 & 2.88 \(\pm\) 0.11 & 3.42 \(\pm\) 0.10 & 2.46 \(\pm\) 0.12 & 2.47 \(\pm\) 0.11 & 2.79 \(\pm\) 0.13 & 3.14 \(\pm\) 0.11 & 3.55 \(\pm\) 0.10 \\ Question-controlled CT 4 & 2.78 \(\pm\) 0.12 & 2.88 \(\pm\) 0.10 & 3.47 \(\pm\) 0.11 & 2.40 \(\pm\) 0.09 & 2.53 \(\pm\) 0.13 & 2.83 \(\pm\) 0.13 & **3.24 \(\pm\) 0.11*
* & 3.59 \(\pm\) 0.10 \\
* Question-controlled CT 7 & 2.81 \(\pm\) 0.10 & **2.99 \(\pm\) 0.11*
* & 3.54 \(\pm\) 0.09 & 2.35 \(\pm\) 0.11 & 2.66 \(\pm\) 0.12 & 2.92 \(\pm\) 0.12 & 3.11 \(\pm\) 0.10 & 3.47 \(\pm\) 0.10 \\ Question-controlled CT 10 & 2.67 \(\pm\) 0.13 & 2.87 \(\pm\) 0.11 & 3.52 \(\pm\) 0.12 & 2.35 \(\pm\) 0.12 & 2.63 \(\pm\) 0.12 & 2.66 \(\pm\) 0.10 & 2.94 \(\pm\) 0.11 & 3.53 \(\pm\) 0.12 \\ Question-controlled CT 10 (boost) & 2.68 \(\pm\) 0.12 & 2.74 \(\pm\) 0.09 & 3.42 \(\pm\) 0.12 & 2.19 \(\pm\) 0.13 & **2.79 \(\pm\) 0.11*
* & 2.74 \(\pm\) 0.11 & 3.00 \(\pm\) 0.12 & 3.45 \(\pm\) 0.13 \\ \hline
**Specificity control (CT)** & & & & & & & & & \\ Specificity-controlled CT 0 & 2.79 \(\pm\) 0.10 & 2.93 \(\pm\) 0.09 & 3.44 \(\pm\) 0.12 & 2.38 \(\pm\) 0.11 & 2.56 \(\pm\) 0.12 & 2.84 \(\pm\) 0.12 & 3.12 \(\pm\) 0.13 & 3.61 \(\pm\) 0.11 \\ Specificity-controlled CT 2 & 2.78 \(\pm\) 0.12 & 2.74 \(\pm\) 0.11 & 3.39 \(\pm\) 0.13 & 2.31 \(\pm\) 0.13 & 2.56 \(\pm\) 0.13 & 2.74 \(\pm\) 0.12 & 2.99 \(\pm\) 0.11 & 3.47 \(\pm\) 0.10 \\ Specificity-controlled CT 4 & 2.82 \(\pm\) 0.10 & 2.80 \(\pm\) 0.13 & 3.44 \(\pm\) 0.14 & 2.32 \(\pm\) 0.13 & 2.51 \(\pm\) 0.12 & 2.78 \(\pm\) 0.15 & 3.09 \(\pm\) 0.13 & 3.46 \(\pm\) 0.13 \\ Specificity-controlled CT 7 & 2.81 \(\pm\) 0.12 & 2.91 \(\pm\) 0.13 & 3.43 \(\pm\) 0.11 & 2.45 \(\pm\) 0.10 & 2.49 \(\pm\) 0.11 & 2.81 \(\pm\) 0.12 & 3.15 \(\pm\) 0.12 & 3.55 \(\pm\) 0.11 \\ Specificity-controlled CT 9 & 2.80 \(\pm\) 0.13 & 2.78 \(\pm\) 0.10 & 3.41 \(\pm\) 0.12 & 2.35 \(\pm\) 0.13 & 2.28 \(\pm\) 0.11 & 2.79 \(\pm\) 0.11 & 2.91 \(\pm\) 0.11 & 3.51 \(\pm\) 0.12 \\ \hline
**Speciticity control (WD)** & & & & & & & & \\ Specificity-controlled WD -10 & 2.76 \(\pm\) 0.11 & 2.41 \(\pm\) 0.12 & 3.19 \(\pm\ 

Plots of human evaluation results for all configurations

Figure 9: Calibrated human evaluation scores for all models. This is the same data as in Table 8.