Venue: NAACL 2022

# Towards a Progression-Aware Autonomous Dialogue Agent

Abraham Sanders\({}^{1}\), Tomek Strzalkowski\({}^{1}\), Mei Si\({}^{1}\), Albert Chang\({}^{1}\),

Deepanshu Dey\({}^{1}\), Jonas Braasch\({}^{1}\), Dakuo Wang\({}^{2}\)

###### Abstract

Recent advances in large-scale language modeling and generation have enabled the creation of dialogue agents that exhibit human-like responses in a wide range of conversational scenarios spanning a diverse set of tasks, from general chit-chat to focused goal-oriented discourse. While these agents excel at generating high-quality responses that are relevant to prior context, they suffer from a lack of awareness of the overall direction in which the conversation is headed, and the likelihood of task success inherent therein. Thus, we propose a framework in which dialogue agents can evaluate the progression of a conversation toward or away from desired outcomes, and use this signal to inform planning for subsequent responses. Our framework is composed of three key elements: (1) the notion of a "global" dialogue state (GDS) space, (2) a task-specific progression function (PF) computed in terms of a conversation's trajectory through this space, and (3) a planning mechanism based on dialogue rollouts by which an agent may use progression signals to select its next response.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

All human conversation serves some purpose. These may range from negotiating an agreement to explaining a topic to maintaining a social relationship. People are generally capable of forming an assessment, sometimes subconsciously, whether a conversation is going well or not and adjusting their behavior accordingly. Such assessment, which underlies most human conversation, is essential in continuous awareness of the direction where the interaction is heading and whether the parties are in sync or not, e.g., Bernieri and Rosenthal (1991). In a task-oriented interaction, the participants assess if progress towards a successful outcome is being made. In a negotiation, parties assess if an agreement is likely. Even in a casual conversation, people intuitively sense when to continue, when to change the subject, or when to stop. Based on such (subjective) assessment, participants adjust what to say next: whether to push forward, make a concession, soften the tone, digress, or say goodbye. A wide range of research in conversation and discourse analysis is devoted to these and related issues including Beebe and Masterson (2000); Cassell et al. (2007); Friedman (2004); Gremler and Gwinner (2008); Langevin et al. (2003); however, recent efforts in Dialogue State Tracking (DST) have been primarily focused on collecting fine-grained details (e.g., slot-value pairs for

Figure 1: Our framework applied to the charity solicitation task in Persuasion For Good Wang et al. (2019). Given the dialogue history (center left), the system uses rollouts Lewis et al. (2017) to simulate the outcome of two response candidates (bottom, in red). Each rollout is mapped as a path through the Global Dialogue State space (center right) where it can be compared with similar outcomes. The candidates are finally ranked using the Progression Function (top), and the best is selected.



travel booking or restaurant reservation) (Balaraman et al., 2021) without concern for the overall direction and quality of the conversation, even though the latter is critical for achieving human-level dialogue interaction.

As such, we approach dialogue state tracking at a higher level, focusing instead on what we call the **Global Dialogue State (GDS)**. Given a conversational task (e.g., negotiation), the global state of a dialogue reflects the most likely outcome (e.g., a strong agreement or a stalemate) given the history of the dialogue up to the current turn. In contrast to traditional DST, the global state remains invariant to the specific details discussed at each turn (e.g., names, dates, quantities) that are typically the concern of slot-filling models. Rather, global dialogue states are influenced by the contexts in which these details occur (e.g., "I _would love to_ donate SS5 to this charity!" vs. "I _would never_ donate S5 to this charity"). Thus, the global state of a dialogue can be measured in terms of its _semantic similarity_ to other groups of dialogues for the same task, which can be naturally formulated as a cluster-assignment problem in the dialogue embedding space. That is, a dialogue which is assigned at the current turn to a cluster of highly successful outcomes may assume a high likelihood of success, and likewise a dialogue assigned to a cluster of unsuccessful outcomes may assume a low likelihood of success. It follows from this that the path of a dialogue through global state space can be used to derive a **Progression Function (PF)** to give turn-level estimates of task success, which can in turn be used by a dialogue agent to inform its next response.

The remainder of this paper is organized as follows: In Section 2 we review relevant literature pertaining to dialogue state tracking and response planning; in Section 3 we formally define the global dialogue state and progression function, propose **supervised** and **unsupervised** approaches for modeling them, and describe how they can be used to assess and select dialogue response candidates; in Section 4 we experimentally apply our framework to the charity solicitation task in the Persuasion For Good dataset Wang et al. (2019), reporting results from automatic and manual evaluations; and in Sections 5 and 6 we conclude with a discussion of limitations and future directions. Code for our methods and experiments has been released, 1 and a listing of software packages we use can be found in Appendix A.

Footnote 1: Available at [https://github.rpi.edu/LACAI/dialogue-progression](https://github.rpi.edu/LACAI/dialogue-progression)

## 2 Related Work

Our work lies at the intersection of dialogue state tracking and response planning. As previously noted, we approach dialogue state at a much higher level than is typically seen in the DST literature. Our concept of global dialogue state is not mutually exclusive with traditional DST approaches, which we refer to from here on as **local** DST. Rather, an effective dialogue system might integrate local and global DST approaches to enable simultaneous tracking of user intents and slot-value pairs (needed for interfacing with external resources) and the overall likelihood of conversational success.

### Dialogue State Tracking

Local DST approaches are used in task-oriented (also called goal-oriented) dialogue systems. Local DST is responsible for identifying user intent (e.g., search for restaurants) and extracting slot-value pairs (e.g., location, price range). Recent DST systems perform state tracking in a diverse set of domains, including food ordering Lertvittayakumjorn et al. (2021), travel reservations Qin et al. (2021), negotiations He et al. (2018), and many others. Datasets such as MultiWOZ Budzianowski et al. (2018); Eric et al. (2020); Zang et al. (2020) and SGD Rastogi et al. (2020) provide large-scale testbeds for training single DST systems that generalize across many task domains. However, local DST is generally not deployed in open-domain end-to-end dialogue systems that focus on social interaction and user engagement, recent examples including DialoGPT Zhang et al. (2020), Meena Adiwardana et al. (2020), and BlenderBot Roller et al. (2021); Xu et al. (2021). In open-domain models, the task is unconstrained and thus it makes little sense to employ traditional slot-based dialogue state trackers. Instead, these models track state implicitly in their latent representations of dialogue history. Unlike local DST, global state tracking is applicable in both the task-oriented and open-domain settings.

### Dialogue Response Planning

Many approaches exist for planning in dialogue response generation. Planning helps a dialogue agent maintain coherence over multiple turns and stay on track to complete its goal. Lewis et al.

 

(2017) introduce Dialogue Rollouts, allowing a negotiation agent to simulate the remainder of a conversation based on each of multiple candidate responses and select the one which yields the best outcome. Yarats and Lewis (2018) follow up by separating semantic planning and surface realization for response generation by first producing a latent semantic representation of the dialogue plan and then conditioning on it during generation with rollouts. Similarly, Jiang et al. (2019) implement a look-ahead module to implicitly predict multiple future turns in an end-to-end encoder-decoder architecture, experimenting with negotiation and restaurant reservation settings. These works all experiment in task domains where goal achievement is explicitly measurable, which is not true in the general case. Thus we propose to combine such methods with our progression function which provides estimates of goal completion likelihood. Particularly, in this paper we demonstrate the use of rollouts with the PF as a reward signal.

## 3 Methods

The goal of our system is to construct a global dialogue state space for a task-specific dataset and learn a progression function to estimate how well an ongoing dialogue is progressing toward the desired outcome of the task. The quantity output by the progression function is an estimate of a dialogue-level attribute which indicates task success (e.g. satisfaction in a customer service task). In many task domains, the success of a conversation cannot be completely measured by a single attribute. For example, in the charity solicitation task we use in our experiments, donation amount is the primary success attribute. Here, there are cases where the conversation appears to go very well, but ultimately no donation is made for unexpected reasons such as the soliciete not being able to afford to donate. One could reasonably expect such an outcome to be "acceptable" in the context of a solicitation task since the soliciete has engaged with the solicitor and displayed interest, and we cannot reasonably expect the solitictor to force a donation out of someone who cannot afford it. Thus we introduce the **"acceptability score"**, a synthetic attribute that measures success by considering multiple factors (e.g., donation amount and sentiment). For any dialogue dataset, the acceptability score combines multiple dialogue-level attributes in a way sensitive to their covariance with the primary success attribute:

\[\text{ACC}_{D}=\text{prim}_{D}+\sum_{i=1}^{|\mathbf{v}_{D}|}\text{Cov}(\text {prim},\text{attr}_{i})\cdot\mathbf{v}_{Di} \tag{1}\]

where \(\text{prim}_{D}\) is the primary success attribute (e.g. donation amount) value for dialogue \(D\), \(\mathbf{v}_{D}\) is the vector of all other attribute values (e.g., sentiment) for dialogue \(D\), and Cov(prim, attr\({}_{i}\)) is the training set covariance between the primary success indicator and the \(i\)-th other attribute. We define the output of the progression function to be an estimate of the acceptability score.

To learn the progression function, dialogue-level attribute annotations must exist for use in this purpose. However, in many settings such annotations are not available in sufficient quantity to directly learn a progression model with sufficient generalization. Consequently, we propose **supervised** and **unsupervised** approaches for learning the global state and progression models.

### Unsupervised Approach

#### 3.1.1 Global Dialogue State

In the unsupervised approach, the GDS space is a dialogue embedding space where clusters of embeddings represent groups of dialogues with similar semantic content. For each complete dialogue \(D\) in the training set, all utterances are independently embedded and then pooled to create a dialogue-level embedding \(\mathbf{u}_{D}\in\mathbb{R}^{d}\) where \(d\) is the embedding size. The GDS space is thus given as a matrix in \(\mathbb{R}^{N\times d}\) where \(N\) is the number of complete dialogues. To embed utterances we take advantage of pre-trained sentence encoders exposed to large-scale corpora. Specifically, we use a publicly available MPNet Song et al. (2020) model fine-tuned for semantic textual similarity using a contrastive objective on over 1B training pairs from 32 distinct datasets. 2 To combine utterance embeddings into a dialogue-level embedding we use recency-weighted mean pooling. The recency weight \(\beta\) determines how much emphasis is placed on more recent utterances, where \(\beta=0\) means all utterances are weighted evenly and \(\beta>0\) means that more emphasis is placed on more recent utterances. The motivation for recency weighting is to test the hypothesis that more recent developments in a conversation are more relevant for predicting current 

progression toward a goal. For example, a conversation may start out off-task with participants engaging in small talk, and then later re-focus.

The embedding for dialogue \(D\) with \(|D|\) utterances is thus formulated as \({\bf u}_{D}=U^{T}\text{softmax}({\bf r})\) where \(U\) is the matrix of utterance vectors in \(\mathbb{R}^{|D|\times d}\) and \({\bf r}\in\mathbb{R}^{|D|}\) is a vector of evenly spaced real numbers over the interval \([0,\beta]\). The softmax ensures all recency weights sum to 1 and can be interpreted as probabilities as done with attention scores in Bahdanau et al. (2015); Vaswani et al. (2017). As shown in Figure 3, each utterance is thus weighted by a monotonically increasing probability mass where higher values of \(\beta\) cause more mass to be concentrated at the end of the dialogue.

The unsupervised GDS model is a clustering of the dialogues in their embedding space. The dialogue embeddings are either clustered directly or after projection to a lower-dimensional space using Parametric UMAP Sainburg et al. (2020); McInnes et al. (2018a). We experiment with k-means and HDBSCAN McInnes and Healy (2017); Campello et al. (2013) to cluster the embeddings. For k-means, we choose the number of clusters \(k\) and train with 10 random initializations. For HDBSCAN, we choose the minimum cluster size and minimum samples hyperparameters, and the optimal number of clusters are selected automatically. Unlike k-means which simply partitions the embedding space, HDBSCAN classifies some embeddings as noise points. Clustering hyperparameters are selected by cross-validation on several metrics as described later in Section 4. The process of constructing the GDS model is illustrated in Figure 2.

The clusters output by this process can be interpreted as the equivalence classes of final global states possible for the task represented in the dialogue dataset. To estimate the global state of an ongoing dialogue \(D^{\prime}\), it is embedded as \({\bf u}_{D^{\prime}}\in\mathbb{R}^{d}\) in the same manner as the complete training dialogues, followed by optional dimensionality reduction. The trained k-means or HDBSCAN model is then used to assign \(D^{\prime}\) to one of the existing clusters, or possibly as a noise point in the case of HDBSCAN.

Each cluster is assigned an aggregate acceptabil

Figure 3: Recency weight \(\beta\) controls how much emphasis is placed on recent utterances when computing \({\bf u}_{D}\).

Figure 2: Architecture of the supervised and unsupervised GDS and PF models (top). In GDS space (top right), each cluster is characterized by similar dialogue semantics, and is thus interpreted as the class of typical outcomes for dialogues within. GDS and PF can be used with rollouts (bottom) to allow a dialogue agent to plan ahead.

 

ity score by taking an average of acceptability for each dialogue in the cluster. If k-means is used, we aggregate using a 10% trimmed mean across all dialogues in the cluster. If HDBSCAN is used, a probability is returned for each dialogue representing the likelihood that it is a member of its assigned cluster, so we compute the probability-weighted average across all dialogues in the cluster. Dialogues classified as noise points are ignored.

To visualize the GDS model, Parametric UMAP is used again to project the clustered dialogue embeddings into \(\mathbb{R}^{2}\) or \(\mathbb{R}^{3}\). As shown in Figure 1, the GDS model can be mapped as a scatter plot with each cluster labeled by its aggregate values. If k-means is used, the cluster centroids can be displayed as a bold point within each cluster. HDBSCAN clusters do not have centroids, but they do have a number of representative points that are close to the cluster core. We average these points to simulate a centroid for display purposes, and likewise show it as a bold point within each cluster. To show how an ongoing dialogue \(D^{\prime}\) traverses the GDS space over time, its embeddings at each turn \(t\) are projected onto the map and connected with line segments to form a path.

#### 3.1.2 Computing Progression

Since each cluster in the GDS space is intended to represent a class of end-task global states, we compute the progression of an ongoing dialogue \(D^{\prime}\) with respect to the likelihood that its final global state will rest in each individual cluster. Supposing there are \(k\) final clusters after running k-means or HDBSCAN, we compute a probability vector \(\mathbf{p}_{D^{\prime}}\in\mathbb{R}^{k}\) such that \(\mathbf{p}_{D^{\prime}i}=P(\mathbf{u}_{D^{\prime}}\in C_{i})\) for \(i\in\{1,\ldots k\}\) where \(C_{i}\) is cluster \(i\). \(\mathbf{p}_{D^{\prime}}\) is computed differently for k-means and HDBSCAN. K-means does not produce a probabilistic soft clustering, so we define \(\mathbf{p}_{D^{\prime}}\) with respect to the proximity of \(\mathbf{u}_{D^{\prime}}\) to the centroids of each cluster:

\[\mathbf{p}_{D^{\prime}}=\text{softmax}\left(\frac{1}{||\mathbf{ u}_{D^{\prime}}-\mathbf{c}_{i}||_{2}}~{}:~{}i\in\{1,\dots k\}\right) \tag{2}\]

where \(\mathbf{c}_{i}\in\mathbb{R}^{d}\) is the centroid of cluster \(i\). HDBSCAN does produce a probabilistic soft clustering, so in that case \(\mathbf{p}_{D^{\prime}}\) is already computed.

We ultimately want the closest (or most probable) clusters for ongoing dialogue \(D^{\prime}\) to have the most sway in estimating its progression at the current point in time. That is, if \(D^{\prime}\) has moved into a cluster of high-success outcomes, its progression should increase. Likewise if \(D^{\prime}\) has moved away from such a high-success cluster, either into a lower-success cluster or off-task into a noisy or unknown region of the GDS space, its progression should decrease. Thus, once \(\mathbf{u}_{D^{\prime}}\) is computed, we estimate its progression as the probability-weighted average of the aggregate acceptability scores assigned to each cluster. This is formulated as

\[\text{PROG}(\mathbf{u}_{D^{\prime}})=\frac{\mathbf{v}^{T}\mathbf{ p}_{D^{\prime}}}{\sum_{i=1}^{k}\mathbf{p}_{D^{\prime}i}} \tag{3}\]

where \(\mathbf{v}\in\mathbb{R}^{k}\) is a vector of the aggregate acceptability scores assigned to each cluster. The scaling factor in the denominator ensures that ongoing dialogue embeddings classified as noise points by HDBSCAN will not be assigned progression values close to zero as a consequence of not belonging to any cluster, which can cause significant fluctuation in the progression function as the dialogue traverses noisy regions of the GDS space. 3 Figure 2 illustrates how progression of an ongoing dialogue depends on its position in GDS space.

Footnote 3: For HDBSCAN we also experiment with softmax for re-scaling \(\mathbf{v}^{T}\mathbf{p}_{D^{\prime}}\), giving \(\text{PROG}(\mathbf{u}_{D^{\prime}})=\text{softmax}(\mathbf{v}^{T}\mathbf{p}_ {D^{\prime}})\).

### Supervised Approach

For the supervised approach, we simply fine-tune RoBERTa Liu et al. (2019) to directly predict acceptability given the dialogue history text, where all utterances are concatenated into a single sequence. To construct the GDS space we obtain the dialogue level embedding \(\mathbf{u}_{D}\) directly from the CLS (<<>) token for each complete dialogue in the training set, and cluster them as in Section 3.1.1. Unlike the unsupervised approach where recency weighting is used to "attend" to more recent parts of the dialogue, the supervised fine-tuning process causes the CLS embedding to aggregate the parts of the dialogue most relevant to the task objective, which is more optimal than the recency heuristic. Also, unlike the unsupervised approach where progression for an ongoing dialogue is computed with respect to its embedding, here progression is directly predicted by RoBERTa. In our experiments we compare RoBERTa-base, RoBERTa-large, and RoBERTa-large-adapted, the latter receiving additional domain adaptation training for dialogue. Domain adaptation is done via Masked Language Modeling (MLM) on a self-generated version of 

the Gutenberg Dialogue Dataset (Casky and Recski, 2021). Hyperparameters and model weights from domain adaptation training are provided with our code release.

### Response Planning

To allow a dialogue agent to use the progression function as feedback for response planning, we adopt **Dialogue Rollouts**(Lewis et al., 2017) to simulate the outcomes of a set of response candidates. A rollout for a response candidate simulates the next \(N\) turns of the conversation (for both participants) given that candidate is used. At each turn of a negotiation task, Lewis et al. (2017) sample a set of \(c\) response candidates and \(s\) rollouts per candidate. They score each rollout by a deterministic reward (the value of the items "won" by the agent during negotiation), and rank each candidate by the average of its rollout scores. The highest ranking candidate is then selected by the agent. As shown in Figure 2, we generalize this process to any task for which a progression function can be learned, replacing the deterministic reward with the progression function value. To demonstrate this, we fine-tune the 762M parameter DialoGPT (Zhang et al., 2020)4 as a dialogue response generator and use beam sampling to generate response candidates and rollouts. We select DialoGPT for this task as it is pre-trained on a large Reddit dialogue corpus.

Footnote 4: Obtained from [https://huggingface.co/microsoft/DialoPFT-large](https://huggingface.co/microsoft/DialoPFT-large)

## 4 Experiments

### Dataset

We apply our framework to the Persuasion For Good dataset (Wang et al., 2019), which is a crowdsourced dialogue dataset where the task for an individual playing the role of persuader is to convince another individual playing the role of persuadee to make a donation to a well-known children's charity. We selected this dataset since it has a clear task objective (to solicit donations), but a complex relationship between dialogue content and success. The dataset authors identify 10 distinct persuasion strategies used to solicit donations, where different strategies correlate with donation amount at different strengths. Additionally, participants in Persuasion For Good dialogues complete a pre-task psychological survey, yielding 23 attributes based on the Big-Five personality traits (Goldberg, 1992), the Moral Foundations endorsement Graham et al. (2011), the Schwartz Portrait Value (Cieciuch and Davidov, 2012), and the Decision-Making style (Hamilton et al., 2016) questionnaires for each individual. The dataset authors demonstrated varying degrees of correlation between these psychological attributes and the end-task donation amount. The complexity in measuring progression in this context, coupled with it being a relatively small dataset, makes Persuasion For Good an interesting and challenging testbed for our framework. Persuasion For Good contains 1017 dialogues, each with approximately 10 turns per speaker (20 utterances).

### Progression Function Experiments

As the objective of the task is to solicit donations, we consider the end-dialogue persuadee donation amount to be the primary dialogue success indicator. We also augment the dataset by computing average dialogue sentiment. To compute sentiment we use a RoBERTa model5 fine-tuned on the sentiment classification task of the TweetEval benchmark (Barbieri et al., 2020), which was publicly released by the benchmark authors. We score sentiment at the utterance level in the range \([-1,1]\) by multiplying the sentiment class probabilities predicted by RoBERTa for negative, neutral and positive by \(\{-1,0,1\}\) respectively and summing the result. We then average the utterance-level sentiment score for each dialogue.

Footnote 5: Obtained from [https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment)

We filter the dataset to eliminate dialogues with end-task donation amounts outside the allowed task parameters (between S0 and S2 USD), and use a regular expression to filter out dialogues where the persuadee fails to make a donation after promising a non-zero dollar amount in the conversation. After filtration we are left with 751 dialogues for our study. We split the dialogues into a training and test set, leaving 577 dialogues for training and 174 for testing. After splitting, we mean-center the dialogue values in the training set for each attribute and scale them to have unit variance. We apply the same transformation to the test set using the distribution parameters of the training set. Our final pre-processing step is to compute the acceptability score. To do this, we compute the covariance matrix of the dialogue-level attribute values in the training set, which include the donation amount and psychological attributes for both the persuader and persuadee from the original dataset, along with 

our computed sentiment scores. Since the values are all standardized, the covariances are equivalent to Pearson's \(r\). We select the covariances of all attributes with respect to the persuadee donation amount (see Figure 5 in Appendix B) and define the acceptability score of each dialogue \(D\) as defined in Section 3. We use the same covariances obtained from the training set to compute acceptability scores on the test set. After pre-processing, the training set has 52 total attributes. These include the persuadee/persuader donation amounts, psychological variables, sentiment, and the acceptability score.

#### 4.2.1 Progression Model Training

We train four progression models as outlined in Sections 3.1 and 3.2: (1) Unsupervised, (2) RoBERTabase, (3) RoBERTa-large, and (4) RoBERTa-large-adapted. For each model, 10% of the training set is held out as a validation set (58 dialogues). For the unsupervised model, a grid search is run for the hyperparameters (e.g., # clusters, recency \(\beta\), dim. reduction, etc.) over the validation set, and the final model is obtained by re-training over the full training set using the best hyperparameters. The final model uses k-means for clustering with \(k=21\) and recency weight \(\beta=0.3\). A complete hyperparameter listing and details on the grid search can be found in Appendix F. For the supervised RoBERTa models, fine-tuning is done with AdamW Loshchilov and Hutter (2019) and an initial learning rate of \(3\times 10^{-5}\) for a maximum of 30 epochs. Early stopping is used over the validation set with the checkpoint corresponding to the lowest validation loss selected as the final model.

#### 4.2.2 Automatic Evaluation

We evaluate the progression models on the following automatic metrics: (1) Mean Absolute Error (**MAE**) in predicting dialogue acceptability, and (2) Pearson's correlation (**r**) between overall PF slope and dialogue acceptability. With MAE we validate that the progression function is able to estimate success of a complete dialogue, while PF slope correlation validates that during an ongoing dialogue, progression increases over time for high-success dialogues and decreases over time for low-success dialogues. To measure PF slope correlation, we fit a least-squares regression line to the progression curve of each dialogue in the test set, and measure Pearson's \(r\) between the regression slopes and their corresponding acceptability scores. For robustness we repeat this evaluation 33 times with varying initialization seeds for each model type (final hyperparameters stay constant) and report the means and standard deviations across runs in Table 1.

Unsurprisingly, the supervised models outperform the unsupervised model on both metrics, which is expected since they directly optimize a mean squared error objective. Of the supervised models, the RoBERTa-large instances perform the best, with dialogue domain adaptation boosting each metric. Pearson's \(r\) is significant at the \(p<0.01\) level for all runs (the null hypothesis is non-correlation).

#### 4.2.3 Manual Evaluation

To obtain a more precise evaluation, we asked three annotators to estimate sentence-level progression for dialogues in our test set. Two graduate students and one postdoc in our lab served as our annotators. For each of twelve randomly selected test dialogues, each annotator rated all sentences on a scale of {-1, 0, 1} for progression, with -1 indicating regression from the task goal, 0 indicating neutral progression, and +1 indicating progression toward the task goal. Altogether, each annotator provided 431 sentence ratings across 244 utterances. After aggregating at the utterance level, average inter-annotator agreement is 0.57 (Cohen's kappa). For each dialogue, the cumulative sum of the utterance-level manual ratings creates a ground-truth progression curve, as shown in Figure 4.

We evaluate the progression models against the ground-truth curves using Pearson's correlation (\(r\)) since the PF output and cumulative manual ratings are continuous and on different scales. We report the following correlations: (1) between utterance-level PF value and ground-truth value (**uttt**), (2) between utterance-level PF slope and ground-truth slope (**utt-sl**), (3) between dialogue-level PF slope and ground-truth slope (**dlgl-sl**), and (4) between dialogue-level PF slope and the final ground-truth progression value (**dlgl-sl**-f**). We repeat this evaluation for each of the 33 uniquely initialized model in

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **MAE** & **r** \\ \hline Unsurprised* & 1.37 (\(\pm\) 0.01) & 0.40 (\(\pm\) 0.02) \\ RoBERTa-base & 0.99 (\(\pm\) 0.06) & 0.50 (\(\pm\) 0.04) \\ RoBERTa-large & 0.97 (\(\pm\) 0.10) & 0.54 (\(\pm\) 0.06) \\ RoBERTa-large-adapted & **0.95 (\(\pm\) 0.05)** & **0.87 (\(\pm\) 0.03)** \\ \hline \multicolumn{3}{l}{For Pearson’s \(r\), \(p<0.01\) for all runs (two-tailed; \(H_{0}\) is non-correlation).} \\ \multicolumn{3}{l}{* Hyperparameters of the unsupervised model can be found in Appendix G.} \\ \end{tabular}
\end{table}
Table 1: Progression Function Auto Eval Results: Mean (SD) of MAE and Pearson’s \(r\) across runs.

 

stances of each type from Section 4.2.2, averaging each metric across raters and then across runs. We report the aggregate means and standard deviations across runs in Table 2. A complete listing of results for each rater (averaged across runs) is provided in Appendix C, along with detailed explanations of each metric (Appendix D). Additionally, Figure 10 in Appendix E provides examples of disagreement between the PF and ground-truth progression curves which can be challenged despite high interannotator agreement, demonstrating difficulty in establishing ground-truth for this open-ended task.

#### 4.2.4 Benefit of Domain Adaptation

To verify the beneficial effect of domain adaptation we perform two-tailed paired t-tests to confirm the differences in means between RoBERTa-large and RoBERTa-large-adapted on all automatic and manual metrics. For each metric, we pair the results from both models for each run of the same seed, since their regression heads would have received identical initializations. We find that the means of **utt**, **utt-sl**, **dlg-sl**, and **dlg-sl-f** differ at the \(p<0.01\) significance level, and the means of the automatic Pearson's \(r\) metric differ at the \(p<0.05\) level. This confirms our intuition that domain adaptation for dialogue prior to fine-tuning the regression objective aids generalization in this task.

### Rollout Experiments

To demonstrate the ability of the PF to guide a dialogue agent, we use it to score rollouts generated with DialoGPT as described in Section 3.3. Specifically, we design a self-play experiment to automatically evaluate the effect of PF-guided rollouts on the success of the solicitation task in Persuasion For Good. The following summarizes the experimental setup, procedure, and results.

#### 4.3.1 Experimental Setup

First, we fine-tune DialoGPT to generate responses on Persuasion For Good. We add speaker control tokens to the vocabulary so that the model can be conditioned to generate as the persuader or persuadee, respectively. Training is done with AdamW (initial lr=\(5\times 10^{-5}\)) for 6 epochs with early stopping over a 10% validation set using perplexity. The final model checkpoint was selected after 3 epochs, achieving validation perplexity of 8.82.

We then select a progression model to use for all self-play runs. Since the supervised RoBERTa-large-adapted model achieved the best average scores across all automatic and manual evaluations, we randomly select one of the 33 RoBERTa-large-adapted instances from Section 4.2.2 to use for all runs. We use this instance for rollout scoring and to measure the progression of each self-play dialogue.

Finally, we train a binary classifier to identify if the persuadee has stated the intent to donate in a conversation, which we use to detect successful self-play dialogues. We fine-tune a RoBERTa model as a classifier using just the persuadee's utterances as input and use binarized donation labels in Persuasion For Good as targets. Specifically, for each dialogue the label is 0 if the donated amount is $0, otherwise it is 1. We use the manually verified intended donation labels from Persuasion For Good "ANNSET" for our validation and test splits and use the remaining end-task donation labels for training. Training is done with early stopping over the validation split using macro F1. The final model checkpoint achieved test F1 of 0.89 and test accuracy of 0.90. All three trained models used in this experiment are available to download (see our code release for instructions and hyperparameters).

#### 4.3.2 Self-Play Procedure

From our test set of 174 dialogues, we manually filter out 41 in which the persuadee pledges a donation within the first 10 utterances, leaving 133 remaining conversations. For each of these, the response generator is given the first 10 utterances as context and tasked to complete a second set of 10 utterances, playing the role of both the persuader

Figure 4: Left: ground-truth progression curve given by the cumulative sum of utterance-level manual ratings. Right: estimated curve from RoBERta-large-adapted.

 

and persuadee. Since the task is solicitation, we allow the generator to use rollouts only when acting as the persuader. We perform the self-play experiment using three persuader modes: (1) with no rollouts **(No RO)**, (2) with 2 response candidates, 2 rollouts per candidate, and 3 utterances per rollout **(2x2x3)**, and (3) with 3 response candidates, 3 rollouts per candidate, and 5 utterances per rollout **(3x3x5)**. For each utterance in each rollout, we use beam sampling with num_beams=6, top_k=50, top_p=0.95, and temperature=\(1.5+0.002\cdot T\) where \(T\) is the number of tokens in the dialogue history. After generation, we compute the following metrics for each dialogue: (1) progression using the selected RoBERTa-large-adapted instance (**Prog.**), (2) persuader and persuadee sentiment using the sentiment classifier from Section 4.2 (**ER Sent.** & **EE Sent.**), and (3) the percent of test dialogues where the persuadee pledges a donation amount or explicitly states intent to donate, as detected by the binary donation intent classifier (**EE Don.%**).

#### 4.3.3 Self-Play Results

For robustness we repeat this procedure 5 times with varying generation seeds for each persuader mode. In total, 1,995 self-play dialogues are completed (133 dialogues for each of 3 modes for each of 5 seeds). We average each metric across dialogues and then across runs, and report the aggregate means and standard deviations across runs. Additionally, to verify the benefit of rollouts, we perform two-tailed paired t-tests to confirm the differences in means between the rollout-enabled modes **(2x2x3 and 3x3x5)** and the baseline (**No RO**). For each metric, we average the results across runs and pair these averages from both modes for each dialogue. Results are shown in Table 3.

We observe that the mean progression increases significantly when rollouts are used. This is expected since response candidates with the highest average end-rollout progression are selected. We also observe that rollouts lead to higher average sentiment for both the persuader and persuadee, which makes sense due to the correlation between sentiment and the acceptability score (see Figure 5 in Appendix B). Finally, rollouts yield a higher percentage of dialogues with a pledged or intended donation. 6 All of these results are significant at the \(p<0.01\) level except for **EE Don.%** in **2x2x3** mode which is significant at \(p<0.05\).

Footnote 6: We randomly selected 30 of the 1,995 self-play dialogues and manually inspected them for donation intent. We found the classifier labeled 26 of them correctly (87% accuracy).

Although progression is noticeably higher for the **3x3x5** mode than for the **2x2x3** mode (0.95 vs 0.69), all other metrics are close between these modes with a small advantage in **3x3x5** mode. This suggests that scaling rollout simulations can be beneficial, but there may be diminishing returns for simulation size. Example self-play dialogues are provided in Tables 7, 8, and 9 in Appendix H.

## 5 Limitations & Future Direction

We recognize several limitations of our study which warrant follow-up investigation. This study focuses on a single task and dataset, and thus is subject to the assumptions and biases therein. Since we intend our framework to be general, it is prudent to perform additional studies to verify the efficacy of our methods on a variety of datasets spanning multiple dialogue domains and tasks. Also, although we provide automatic evaluation of the ability of rollouts to improve performance on a solicitation task, we cannot assume that humans would respond in the same way as DialoGPT. Thus, human evaluation is needed to further validate this approach.

## 6 Conclusion

In this work we introduced the concept of global dialogue state and proposed a framework with which a dialogue agent can gain awareness of where an ongoing conversation is headed, the likelihood of a successful outcome, and how its own response decisions impact the overall direction of the dialogue. We demonstrated that an unsupervised approach to modeling the GDS space and progression function is feasible, which is useful in sparsely-labeled settings. However, we showed that with domain-adaptation pre-training for dialogue, supervised methods are preferable when labels are available. Finally, we demonstrated how using the PF as a feedback mechanism via dialogue rollouts allows an agent to improve outcomes on a solicitation task.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Mode** & **Prog.** & **ER Sent.** & **EE Sent.** & **EE Don. \%** \\ \hline No RO & 0.01 (\(\pm\) 0.24) & 0.51 (\(\pm\) 0.03) & 0.44 (\(\pm\) 0.06) & 38\% (\(\pm\) 6\%) \\ \hline
2x2x3 & 0.69 (\(\pm\) 0.29) & 0.62 (\(\pm\) 0.05) & 0.49 (\(\pm\) 0.07) & 45\% (\(\pm\) 10\%)† \\
3x3x5 & **0.95 (\(\pm\) 0.16)** & **0.66 (\(\pm\)** 0.02) & **0.52 (\(\pm\) 0.04)** & **47\% (\(\pm\) 11\%)** \\ \hline \hline All 2x2x3 and 3x3x5 means significant at \(p<0.01\) on 0.05 if marked † 



### Ethical Considerations

#### Ethical Dialogue Systems

We acknowledge the potential risks inherent in the deployment of goal-oriented dialogue systems, and especially note that care must be taken to ensure persuasive dialogue systems are designed for beneficial use as discussed by Wang et al. (2019). Concretely, when applying our framework, care must be taken to ensure that the goal of the system (defined by the primary success attribute of the acceptability score) should be generally accepted as beneficial. For example, our basis for dialogue acceptability in this work is with respect to raising money for children's charity. In general, the achievement of the system's goal should not intentionally lead the user or any other party to harm. Additionally, the definition of acceptability, through its primary or any other correlated attributes, should not allow for discriminative responses, purposefully malicious discourse, or other violations of accepted ethical standards. For example, we include sentiment as secondary attributes in the acceptability score, which, when applied via dialogue rollouts, encourages the system to be courteous, polite, and respectful. It is possible with minimal effort to include further secondary attributes that identify bias, hate speech, and other indicators to help the system remain safe to use.

#### Annotator Compensation

All manual annotators were recruited on a voluntary basis in an educational setting and did not receive or expect monetary compensation. Specifically, two graduate students and one postdoc in our lab served as our annotators.

#### Environmental Impact

All training and inference in this work was done with two NVIDIA Quadro RTX 8000 GPUs. The most compute-intensive portion of the work was the additional domain adaptation pre-training for RoBERTa-large-adapted (see Section 3.2), which took approximately two weeks. After that the multi-seed self-play evaluations took approximately four days, and all other operations (e.g., training and evaluating PF models, fine-tuning DialoGPT) took 24 hours or less.

## Acknowledgements

We would like to thank our manual annotators for their valuable contribution and the anonymous reviewers for their helpful feedback. This paper is based upon work supported in part by the United States Air Force under Contract No. FA8750-21-C-0075 and in part by the IBM Corporation under the Artificial Intelligence Research Collaboration Agreement No. W1771793 between IBM and Rensselaer. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of USAF or IBM Corporation.

## References

* Abadi et al. [2015] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
* Adiwardana et al. [2020] Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Roman Thoppilan, Zi Yang, Apoorty Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. _arXiv preprint arXiv:2001.09977_.
* Bahdanau et al. [2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Balaraman et al. [2021] Vexake Balaraman, Seyedmostafa Sheikhalishahi, and Bernardo Magnini. 2021. Recent neural methods on dialogue state tracking for task-oriented dialogue systems: A survey. In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 239-251.
* Barbieri et al. [2020] Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1644-1650, Online. Association for Computational Linguistics.
* Beebe and Masterson [2000] S.A. Beebe and J.T. Masterson. 2000. _Communicating in Small Groups: Principles and Practices_. Longman.



F.J. Bernieri and R. Rosenthal (1991)Interpersonal coordination: behavior matching and interactional synchrony. In R.S. Feldman and B. Rime, editors, FUNDARKDD, pp. 5016-5026. External Links: Link Cited by: SS4.
* a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 5016-5026. External Links: Link Cited by: SS3.
* R. JGB Campello, D. Moulavi, and J. Sander (2013)Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-172. Cited by: SS1.
* J. Cassell, A. Gill, and P. Pepper (2007)Coordination in conversation and rapport. In Proceedings of the Workshop on Embodied Language Processing, Prague, Czech Republic, pp. 41-50. External Links: Link Cited by: SS1.
* J. Cacciuch and E. Davidov (2012)A comparison of the invariance properties of the pvq-40 and the pvq-21 to measure human values across german and polish samples. In Survey Research Methods, Vol. 6, pp. 37-48. Cited by: SS1.
* R. Csaky and G. Recski (2021)The Gutenberg dialogue dataset. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 138-159. External Links: Link Cited by: SS1.
* H. Eric, R. Goel, S. Paul, A. Sethi, S. Agarwal, S. Gao, A. Kumar, A. Goyal, P. Ku, and D. Hakkani-Tur (2020)MultiWOZ 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 422-428. External Links: Link Cited by: SS1.
* W. Falcon and The PyTorch Lightning team (2019)PyTorch lightning. External Links: Link Cited by: SS1.
* R. Friedman (2004)Studying negotiations in context: an ethnographic approach. Internat'l Negotiation. External Links: Link Cited by: SS1.
* L. R Goldberg (1992)The development of markers for the big-five factor structure. Psychological assessment4 (1), pp. 26. Cited by: SS1.
* J. Graham, B. A. Nosek, J. Haidt, R. Iyer, S. Koleva, and P. H. Ditto (2011)Mapping the moral domain. Journal of personality and social psychology101 (2), pp. 366. Cited by: SS1.
* D. Gremler and K. P. Gwinner (2008)Rapport-building behaviors used by retail employees.. Retailing. External Links: Link Cited by: SS1.
* K. Hamilton, S. Shih, and S. Mohammed (2016)The development and validation of the rational and intuitive decision styles scale. Journal of personality assessment98 (5), pp. 523-535. External Links: Document, ISSN 0004-0003, Link Cited by: SS1.
* C. R. Harris, K. Jarrod Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wiesser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Bert, A. Haldane, J. Fernandez del Rio, M. Wiebe, P. Peterson, P. Gerard-Marchant, K. Sheppard, T. Reddy, Warren Weckesger, H. Abbasi, C. Gohlke, and Travis E. Oliphant (2020)Array programming with NumPy. Nature585 (7825), pp. 357-362. External Links: Link Cited by: SS1.
* H. He, D. Chen, A. Balakrishnan, and P. Liang (2018)Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2333-2343. External Links: Link Cited by: SS1.
* J. D. Hunter (2007)Matplotlib: a 2d graphics environment. Computing in Science & Engineering9 (3), pp. 90-95. External Links: Link Cited by: SS1.
* Z. Jiang, X. Mao, Z. Huang, J. Ma, and S. Li (2019)Towards end-to-end learning for efficient dialogue agent by modeling looking-ahead ability. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Stockholm, Sweden, pp. 133-142. External Links: ISBN 978-1-4503-3611-1, Link Cited by: SS1.
* W. Langewitz, M. Nubling, and H. Weber (2003)A theory-based approach to analysing conversation sequences. Epidemologia e Pschiatria Socialae12 (2), pp. 103-108. External Links: Link Cited by: SS1.
* P. Lervittyakumjorn, D. Bonadianm, and Saab Mansour (2021)Knowledge-driven slot constraints for goal-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 3407-3419. External Links: Link Cited by: SS1.
* M. Lewis, D. Yarats, Y. Dauphin, D. Parikh, and D. Batra (2017)DeAL or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 2443-2453. External Links: Link Cited by: SS1.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) 

Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* McInnes and Healy (2017) Leland McInnes and John Healy. 2017. Accelerated hierarchical density based clustering. In _2017 IEEE International Conference on Data Mining Workshops (ICDMW)_, pages 33-42. IEEE.
* McInnes et al. (2017) Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. _The Journal of Open Source Software_, 2(11):205.
* McInnes et al. (2018a) Leland McInnes, John Healy, and James Melville. 2018a. Umap: Uniform manifold approximation and projection for dimension reduction. _arXiv preprint arXiv:1802.03426_.
* McInnes et al. (2018b) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018b. Umap: Uniform manifold approximation and projection. _The Journal of Open Source Software_, 3(29):861.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 8024-8035.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* ten (2015) plotly technologies inc. 2015. Collaborative data science.
* Qin et al. (2021) Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanxiang Che, and Ting Liu. 2021. GL-GIN: Fast and accurate non-autoregressive model for joint multiple intent detection and slot filling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 178-188, Online. Association for Computational Linguistics.
* Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Chaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8689-8696.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 300-325, Online. Association for Computational Linguistics.
* Sainburg et al. (2020) Tim Sainburg, Leland McInnes, and Timothy Q Gentner. 2020. Parametric umap: learning embeddings with deep neural networks for representation and semi-supervised learning.
* Satopaa et al. (2011) Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a "kneedle" in a haystack: Detecting knee points in system behavior. In _2011 31st International Conference on Distributed Computing Systems Workshops_, pages 166-171.
* Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. In _Advances in Neural Information Processing Systems_, volume 33, pages 16857-16867. Curran Associates, Inc.
* Tsukcuoglu et al. (2020) the pandas development team. 2020. pandascdev/pandas: Pandas.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreti, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Virtanen et al. (2017) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stefan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake Vanderglas, Denis Laxalde, JosePerktold, Robert Cimmerman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antonio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. _Nature Methods_, 17:261-272 

Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5635-5649, Florence, Italy. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentinhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xu et al. (2021) Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond goldfish memory: Long-term open-domain conversation. _arXiv preprint arXiv:2107.07567_.
* Yarats and Lewis (2018) Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dialogue. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 5587-5595. PMLR.
* Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghaw Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_, pages 109-117, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 270-278, Online. Association for Computational Linguistics.

 

Software Packages Used

Table 4: Software packages used in obtaining or presenting the results in this work

\begin{tabular}{l c l l} \hline \hline
**Package** & **Version** & **Citation** & **URL** \\ \hline hdbscan & 0.827 (Mclmes et al., 2017) & [https://hdbscan.readthedocs.io/kneed](https://hdbscan.readthedocs.io/kneed) & 0.7.0 (Satopa et al., 2011) & [https://kneed.readthedocs.io/Matplotlib](https://kneed.readthedocs.io/Matplotlib) & 3.34 (Hanr, 2007) & [https://mapliblib.org/](https://mapliblib.org/) \\ NumPy & 1.19.5 (Harris et al., 2020) & [https://numpy.org/](https://numpy.org/) \\ Pandas & 1.2.4 (the pandas development team, 2020) & [https://pandas.pydata.org/](https://pandas.pydata.org/) \\ plotly & 5.10 (goldy technologies inc., 2015) & [https://plotly.com/python/](https://plotly.com/python/) \\ PyTorch & 1.9.0 (Paszke et al., 2019) & [https://pytorch.org/](https://pytorch.org/) \\ PyTorch Lightning & 1.5.8 (Falcon and learn, 2019) & [https://pytorchlightning.ai](https://pytorchlightning.ai) \\ scikit-learn & 0.24.1 (Petdeposia et al., 2011) & [https://scikit-learn.org/](https://scikit-learn.org/) \\ SciPy & 1.6.2 (Virtanen et al., 2020) & [https://scipy.org/sciplipb/index.html](https://scipy.org/sciplipb/index.html) \\ Sentence-Transformers & N/A* (Reimers and Garevych, 2019) & [https://sbert.net/](https://sbert.net/) \\ TensorFlow & 2.5.1 (Abadt et al., 2015) & [https://tensorflow.org/](https://tensorflow.org/) \\ Transformers & 4.11.3 (Wolf et al., 2020) & [https://buuggingface.co/transformers/](https://buuggingface.co/transformers/) \\ tumor-learn & 0.5.1 (Mellens et al., 2018b) & [https://umap-learn.readthedocs.io/](https://umap-learn.readthedocs.io/) \\ \hline \end{tabular}

## Appendix B Training Set Covariances For Acceptability Score

 

Figure 8: **dlg-sl**: Pearson’s \(r\) (right) between dialogue-level PF slopes (center, e.g., see line) and ground-truth slopes (left, e.g., see line), for all 12 dialogues. Dialogue-level slopes are computed by fitting least-squares regression lines to the progression curves. Points shown on the right are from annotator 3. This metric is intended to measure the ability of the overall PF trend to approximate the ground-truth progression curve.

Figure 6: **utt**: Pearson’s \(r\) (right) between utterance-level PF values (center, e.g., circled) and ground-truth values (left, e.g., circled) for all 244 utterances across 12 dialogues. Points shown on the right are from annotator 3. This metric is intended to measure if the PF and ground-truth progression curves assign similar values (relative to their respective scales) at each step of an ongoing dialogue.

Figure 7: **utt-sl**: Pearson’s \(r\) (right) between utterance-level PF slopes (center, e.g., see triangle) and ground-truth slopes (left, e.g., see triangle), for all 244 utterances across 12 dialogues. Utterance-level slopes are computed as the differences in the progression curves between two dialogue steps. Points shown on the right are from annotator 3. This metric is intended to measure if the PF and ground-truth progression curves move in the same direction at each step of an ongoing dialogue.

Figure 9: **dlg-sl**-f**: Pearson’s \(r\) (right) between dialogue-level PF slopes (center, e.g., see line) and the final ground-truth progression value (left, e.g., circled), for all 12 dialogues. Dialogue-level slopes are computed by fitting least-squares regression lines to the progression curves. Points shown on the right are from annotator 3. This metric is intended to measure the ability of the overall PF trend to predict the end-task outcome.

 



## Appendix E PF Disagreement Examples

* 3. ER: rs an international organization that promotes children's rights, provides relief and helps support children in developing countries, digest that sound like the community you get behind?
4. EE: No thank you.
5. ER: May last day? War help could really make a difference in a child's time of a child who didn't have enough to eat, just a small fee of help may have saved them, are you sure your vote not interested?
6. EE: I am already making a difference in many children lives.
7. ER: That’s great, it sounds like rs an issue you really care about men, why not got some of their action as charity thirst doing nearly past work like Save the Children?
8. EE: It must contact 1 years of my life to chanty
9. EE: From not asking you to donate years of your life, Just a couple of comets for possible, is that something you could possibly do
10. EE: No thank you
11. ER: We wrote probably on a computer right now, so you probably have quite a bit or energy yourself; both you think you would ignore the 2.5 comets for somebody children around the world?
12. EE: No thank you!
13. ER: Do you have children yourself What if your children were in a position these kids are, wouldn't you want someone to help them?
14. EE: Where are those children parent?
15. EF: If only they had parental support, but in war-tom counties like Syria there appears being killed in stars or are fighting for their country, they have no help that shirts where they need your object, please find in your heart to just give a little bit [the hands, all mL salmon].
16. EE: My small change word to a thing for those kids. When they need is to be extracted from those countries so that they can have a better life not 0.25.
17. ER: Well with our current president isn't so hot they bring going to be brought up the US, so they need any help they can get, and 25 cents might not seem like let, but if I should people donated 25 cents then the impact will be huge. Unordered fat many set some size at but, but a can really make a difference. You can donate some of all of your payment from this to Sueue the Children, i/or really appreciate it for you did, but understand if you still meet went to what do you say?
18. EE: If it makes you happy, go ahead and take 50
19. RE: On thrust you s much, you have no idea how much that 50 will mean to those children, bank you very much and have a wonderful night you believed so!
20. EE: No problem have a good night

Figure 10: Two examples where the PF (RoBERTa-large-adapted) disagrees with multiple annotators. We note that the PF correctly decreases (**Right**) and stays negative (**Left**) given the turns boxed in red showing poor progression.



Grid Search Details for Unsupervised Model

```
for\(\beta\in\{0.0,0.1,\dots,2.0\}\)do for\(d\in\{2,16,32,64,128,768\}\)do fornormalize_embeddings \(\in\{True,False\}\)do fordistance_metric \(\in\{Cosine,Euclidean\}\)do \(\triangleright\) k-means experiments for\(k\in\{2,3,\ldots,30\}\)do forinverse_distance \(\in\{True,False\}\)do forstandardized_proximity \(\in\{True,False\}\)do  measure_PF_slope_r0; \(\triangleright\) HDBSCAN experiments formin_cluster_size \(\in\{10,20,\dots,100\}\)do forsoft_value_aggregation \(\in\{True,False\}\)do forprob_scaling \(\in\{None,softmax,sum\}\)do for standardized_proximity \(\in\{True,False\}\)do  measure_PF_slope_r0;
```

**Algorithm 1**Grid search for hyperparameter tuning of the unsupervised progression model on the validation set. Descriptions for each hyperparameter are provided in Table 6.

## Appendix G Final Unsupervised Model Hyperparameters

The final unsupervised model uses k-means (\(k=21\)), \(\beta=0.3\), \(d=768\), normalized embeddings, euclidean distance, and inverse distance for centroid proximity.

### Examples from rollout self-play experiment

The following examples in Tables 7, 8, and 9 compare self-play between the baseline **No RO** mode and the two rollout modes **2x2x3** and **3x3x5**. In each example, the utterance where the persuadee (EE) states intent to donate is highlighted in red. The complete dialogues from the self-play experiments are included with our code release.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Hyperparameter** & **Description** \\ \hline \(\beta\) (recency weight) & Controls how much emphasis is placed on recent tokens when computing dialogue embeddings. \\ \(d\) (embedding size) & The dimensionality of dialogue embedding. Values \(<768\) reduced with Parametric UMAP. \\ normalize embeddings & If True, embiddings are normalized to have unit magnitude after dimensionality reduction. \\ distance\_metric & The distance metric used by Parametric UMAP and centroid proximity calculations.* \\ k (number of clusters) & Number of clusters use for k-means. \\ inverse\_distance & If True, Euclidean centroid proximity is computed as the inverse distance instead of negative distance.** \\ standardized\_proximity & If True, centred proximities are converted to z-scores before progress in computed. † \\ min\_dister\_size & Minimum number of points in a HDBSCAN cluster. Clusters with fewer points get merged into larger ones. \\ soft\_value\_aggregation & if True, HDBSCAN cluster attribute aggregations are weighted with cluster membership probabilities. \\ prob\_scaling & The type of scaling applied to progression computed from HDBSCAN cluster membership probabilities. \\ \hline \multicolumn{3}{l}{*: Centroid proximity calculations refer to progression computation. Proximities used during clustering are always Euclidean.} \\ \multicolumn{3}{l}{*: Does not apply to cosine distance.} \\ \multicolumn{3}{l}{†: Applies only if k-means is used, or if HDBSCAN is used with softmax probability scaling.} \\ \end{tabular}
\end{table}
Table 6: Hyperparameter Descriptions 



\begin{table}
\begin{tabular}{l l} \hline \hline
**CONTEXT:** & \\ \hline EE: Hello, Have you ever donated to a charity? \\ EE: A few times, yes. \\ ER: ow was your experience? \\ EE: It was not I didn’t really think much of it. Most of the time its just a dollar or two. \\ EE: Do you donate all tie time? \\ EE: No so much. \\ EE: Have we heard of Save the children? It is pretty well known around the world. \\ EE: I think so. \\ EE: What do you think that they do? \\ EE: Office memory to child. Support efforts in war torn countries. Healthcare. All that stuff. \\ \hline
**NO ROLLOUTS (progression: 2.465; ER sentiment: 0.69; EE sentiment: 0.03)** \\ \hline EE: I It is a worthy cause. You can donate all you want from it. \\ EE: I'm sorry. I’m broke right now. That’s why I’m asking if you do donate all the time. \\ EE: Its okay. I understand and agree. I think six s a good cause. \\ EE: So what is the family but soon take i? \\ EE: I don’t have a specific charity, but I do think they are a decent charity. \\ EE: Are they a non-profit? What did they do? \\ EE: It’s a non profit. They help protect chiffens rights to health and education. \\ EE: Well. Until there may be at great charity. They provide help in other ways besides donating to them? \\ EE: Yes they help with children in the US and other developing countries. \\ EE: So I’m guessing the money goes towards programs to help those children get a proper education? \\ \hline
**2x2x3 Roll LOUTS (progression: 0.15; EB sentiment: 0.67; EE sentiment: 0.63)** \\ \hline  & \\ \end{tabular} 
\begin{tabular}{l l} \hline \hline
**EC: I** & \(I\) \\ \hline EE: I thank Jo. I have donated to the Red Cross and the Salvation Army in the past. \\ EE: I donated a couple of months ago. Do you donate often? I donate when I can. \\ EE: Yes, I donate to theme as well. How much do you like to donate? \\ EE: I donate S.50 each month. Do you do the same? \\ EE: Yeah, that sounds great. Any bit helps though. Do you have an amount in mind? \\ EE: I would like to donate half. Is that a good idea? \\ EE: Yes that would be great. The children really do need it! I donate to them as well. \\ EE: If my best can argue on a generous donation. How much would you like me to donate? \\ EE: I think wall would he fine. I’m not sure, but that is a very generous amount. \\ EE: Welit, it is nice to think that I’m not getting hit by a bot, so that’s a good thing. \\ \hline \hline \end{tabular}
\end{table}
Table 8: Rollout Self-Play Example 2: **No RO vs. 3x3x5** (dialogue #88; generation seed: 756)

* [105] M. C. Bender, J. R. Friedman, and A. K. Sridharan. The role of the social influence on social dynamics in social networks. In _Social Networks: Proceedings of the National Academy of Sciences_, volume 2498 of _Lecture Notes in Social Science_, pages 37-60, 2000.
* [106] D. G. Lowe and T. L. Jordan. On the social influence of social interactions on social dynamics. _Journal of Sociology_, 2(1):105-118, 1999.
* [107] P. H. Lee and Y. N. Kim. Social interaction between social interactions and social interactions. _The Journal of Social Psychology_, 2(1):105-116, 1999.

[MISSING_PAGE_POST]