SlowGuess/NAACL_2022_OCR huggingface dataset:
Abraham Sanders<sup>1</sup>, Tomek Strzalkowski<sup>1</sup>, Mei Si<sup>1</sup>, Albert Chang<sup>1</sup>, Deepanshu Dey<sup>1</sup>, Jonas Braasch<sup>1</sup>, Dakuo Wang<sup>2</sup>
$^{1}$ Rensselaer Polytechnic Institute, Troy, NY, USA,  $^{2}$ IBM Research, USA {sandea5,tomek,sim,changa4,deyd,braasj}  $@$  rpi.edu dakuo.wang@ibm.com
Recent advances in large-scale language modeling and generation have enabled the creation of dialogue agents that exhibit human-like responses in a wide range of conversational scenarios spanning a diverse set of tasks, from general chit-chat to focused goal-oriented discourse. While these agents excel at generating high-quality responses that are relevant to prior context, they suffer from a lack of awareness of the overall direction in which the conversation is headed, and the likelihood of task success inherent therein. Thus, we propose a framework in which dialogue agents can evaluate the progression of a conversation toward or away from desired outcomes, and use this signal to inform planning for subsequent responses. Our framework is composed of three key elements: (1) the notion of a "global" dialogue state (GDS) space, (2) a task-specific progression function (PF) computed in terms of a conversation's trajectory through this space, and (3) a planning mechanism based on dialogue rollouts by which an agent may use progression signals to select its next response.
All human conversation serves some purpose. These may range from negotiating an agreement to explaining a topic to maintaining a social relationship. People are generally capable of forming an assessment, sometimes subconsciously, whether a conversation is going well or not and adjusting their behavior accordingly. Such assessment, which underlies most human conversation, is essential in continuous awareness of the direction where the interaction is heading and whether the parties are in sync or not, e.g., Bernieri and Rosenthal (1991). In a task-oriented interaction, the participants assess if progress towards a successful outcome is being made. In a negotiation, parties assess if an agreement is likely. Even in a casual conversation, people intuitively sense when to continue, when to change the subject, or when to stop. Based on such
[Rollout 1.a] Prob: 0.941; Score: 1.646
ER: You have an opportunity to make a donation today. I will ask your donation and if you agree to it, then you can choose any amount that you like from  $0 to all your payment ($ 2).
EE: Okay, I will try to donate $2. I think it is important to help save the children.  
ER: Thank you very much, it is important to help save the children.
EE: Okay, I will try to donate $2, and thank Candidate: OK.846
[Rollout 2.a] Prob:0.977;Score:-2.249
ER: That's understandable, Do you like to donate to a child's charity, like Save the Children?
EE: I do not think so. I don't think I have any particular reason for doing so.
ER: Well, it's an awesome organization. I have always wanted to help children. Have you ever donated to a charity before?
EE: No, I don't have much money, but I like to do what I can. What does the charity do? Candidate Score: -2.172
Figure 1: Our framework applied to the charity solicitation task in Persuasion For Good (Wang et al., 2019). Given the dialogue history (center left), the system uses rollouts (Lewis et al., 2017) to simulate the outcome of two response candidates (bottom, in red). Each rollout is mapped as a path through the Global Dialogue State space (center right) where it can be compared with similar outcomes. The candidates are finally ranked using the Progression Function (top), and the best is selected.
(subjective) assessment, participants adjust what to say next: whether to push forward, make a concession, soften the tone, digress, or say goodbye. A wide range of research in conversation and discourse analysis is devoted to these and related issues including (Beebe and Masterson, 2000; Cassell et al., 2007; Friedman, 2004; Gremler and Gwinner, 2008; Langewitz et al., 2003); however, recent efforts in Dialogue State Tracking (DST) have been primarily focused on collecting fine-grained details (e.g., slot-value pairs for
Human Language Technologies, pages 1194 - 1212
travel booking or restaurant reservation) (Balaraman et al., 2021) without concern for the overall direction and quality of the conversation, even though the latter is critical for achieving human-level dialogue interaction.
As such, we approach dialogue state tracking at a higher level, focusing instead on what we call the Global Dialogue State (GDS). Given a conversational task (e.g., negotiation), the global state of a dialogue reflects the most likely outcome (e.g., a strong agreement or a stalemate) given the history of the dialogue up to the current turn. In contrast to traditional DST, the global state remains invariant to the specific details discussed at each turn (e.g., names, dates, quantities) that are typically the concern of slot-filling models. Rather, global dialogue states are influenced by the contexts in which these details occur (e.g., "I would love to donate $5 to this charity!" vs. "I would never donate $5 to this charity"). Thus, the global state of a dialogue can be measured in terms of its semantic similarity to other groups of dialogues for the same task, which can be naturally formulated as a cluster-assignment problem in the dialogue embedding space. That is, a dialogue which is assigned at the current turn to a cluster of highly successful outcomes may assume a high likelihood of success, and likewise a dialogue assigned to a cluster of unsuccessful outcomes may assume a low likelihood of success. It follows from this that the path of a dialogue through global state space can be used to derive a Progression Function (PF) to give turn-level estimates of task success, which can in turn be used by a dialogue agent to inform its next response.
The remainder of this paper is organized as follows: In Section 2 we review relevant literature pertaining to dialogue state tracking and response planning; in Section 3 we formally define the global dialogue state and progression function, propose supervised and unsupervised approaches for modeling them, and describe how they can be used to assess and select dialogue response candidates; in Section 4 we experimentally apply our framework to the charity solicitation task in the Persuasion For Good dataset (Wang et al., 2019), reporting results from automatic and manual evaluations; and in Sections 5 and 6 we conclude with a discussion of limitations and future directions. Code for our methods and experiments has been released, $^{1}$  and
a listing of software packages we use can be found in Appendix A.
Our work lies at the intersection of dialogue state tracking and response planning. As previously noted, we approach dialogue state at a much higher level than is typically seen in the DST literature. Our concept of global dialogue state is not mutually exclusive with traditional DST approaches, which we refer to from here on as local DST. Rather, an effective dialogue system might integrate local and global DST approaches to enable simultaneous tracking of user intents and slot-value pairs (needed for interfacing with external resources) and the overall likelihood of conversational success.
2.1 Dialogue State Tracking
Local DST approaches are used in task-oriented (also called goal-oriented) dialogue systems. Local DST is responsible for identifying user intent (e.g., search for restaurants) and extracting slot-value pairs (e.g., location, price range). Recent DST systems perform state tracking in a diverse set of domains, including food ordering (Lertvittayakumjorn et al., 2021), travel reservations (Qin et al., 2021), negotiations (He et al., 2018), and many others. Datasets such as MultiWOZ (Budzianowski et al., 2018; Eric et al., 2020; Zang et al., 2020) and SGD (Rastogi et al., 2020) provide large-scale testbeds for training single DST systems that generalize across many task domains. However, local DST is generally not deployed in open-domain end-to-end dialogue systems that focus on social interaction and user engagement, recent examples including DialogGPT (Zhang et al., 2020), Meena (Adiwardana et al., 2020), and BlenderBot (Roller et al., 2021; Xu et al., 2021). In open-domain models, the task is unconstrained and thus it makes little sense to employ traditional slot-based dialogue state trackers. Instead, these models track state implicitly in their latent representations of dialogue history. Unlike local DST, global state tracking is applicable in both the task-oriented and open-domain settings.
2.2 Dialogue Response Planning
Many approaches exist for planning in dialogue response generation. Planning helps a dialogue agent maintain coherence over multiple turns and stay on track to complete its goal. Lewis et al.
Available at https://github.rpi.edu/LACAI/ dialogue-progression
(2017) introduce Dialogue Rollouts, allowing a negotiation agent to simulate the remainder of a conversation based on each of multiple candidate responses and select the one which yields the best outcome. Yarats and Lewis (2018) follow up by separating semantic planning and surface realization for response generation by first producing a latent semantic representation of the dialogue plan and then conditioning on it during generation with rollouts. Similarly, Jiang et al. (2019) implement a look-ahead module to implicitly predict multiple future turns in an end-to-end encoder-decoder architecture, experimenting with negotiation and restaurant reservation settings. These works all experiment in task domains where goal achievement is explicitly measurable, which is not true in the general case. Thus we propose to combine such methods with our progression function which provides estimates of goal completion likelihood. Particularly, in this paper we demonstrate the use of rollouts with the PF as a reward signal.
The goal of our system is to construct a global dialogue state space for a task-specific dataset and learn a progression function to estimate how well an ongoing dialogue is progressing toward the desired outcome of the task. The quantity output by the progression function is an estimate of a dialogue-level attribute which indicates task success (e.g. satisfaction in a customer service task). In many task domains, the success of a conversation cannot be completely measured by a single attribute. For example, in the charity solicitation task we use in our experiments, donation amount is the primary success attribute. Here, there are cases where the conversation appears to go very well, but ultimately no donation is made for unexpected reasons such as the solicitee not being able to afford to donate. One could reasonably expect such an outcome to be "acceptable" in the context of a solicitation task since the solicitee has engaged with the solicitor and displayed interest, and we cannot reasonably expect the solicitor to force a donation out of someone who cannot afford it. Thus we introduce the "acceptability score", a synthetic attribute that measures success by considering multiple factors (e.g., donation amount and sentiment). For any dialogue dataset, the acceptability score combines multiple dialogue-level attributes in a way sensitive to their covariance with the primary
success attribute:
$$
\mathrm {A C C} _ {D} = \operatorname {p r i m} _ {D} + \sum_ {i = 1} ^ {| \mathbf {v} _ {D} |} \operatorname {C o v} (\operatorname {p r i m}, \operatorname {a t t r} _ {i}) \cdot \mathbf {v} _ {D i} \tag {1}
$$
where  $\mathrm{prim}_D$  is the primary success attribute (e.g. donation amount) value for dialogue  $D$ ,  $\mathbf{v}_D$  is the vector of all other attribute values (e.g., sentiment) for dialogue  $D$ , and  $\mathrm{Cov}(\mathrm{prim}, \mathrm{attr}_i)$  is the training set covariance between the primary success indicator and the  $i$ -th other attribute. We define the output of the progression function to be an estimate of the acceptability score.
To learn the progression function, dialogue-level attribute annotations must exist for use in this purpose. However, in many settings such annotations are not available in sufficient quantity to directly learn a progression model with sufficient generalization. Consequently, we propose supervised and unsupervised approaches for learning the global state and progression models.
3.1 Unsupervised Approach
3.1.1 Global Dialogue State
In the unsupervised approach, the GDS space is a dialogue embedding space where clusters of embeddings represent groups of dialogues with similar semantic content. For each complete dialogue  $D$  in the training set, all utterances are independently embedded and then pooled to create a dialogue-level embedding  $\mathbf{u}_D \in \mathbb{R}^d$  where  $d$  is the embedding size. The GDS space is thus given as a matrix in  $\mathbb{R}^{N \times d}$  where  $N$  is the number of complete dialogues. To embed utterances we take advantage of pre-trained sentence encoders exposed to large-scale corpora. Specifically, we use a publicly available MPNet (Song et al., 2020) model fine-tuned for semantic textual similarity using a contrastive objective on over 1B training pairs from 32 distinct datasets. To combine utterance embeddings into a dialogue-level embedding we use recency-weighted mean pooling. The recency weight  $\beta$  determines how much emphasis is placed on more recent utterances, where  $\beta = 0$  means all utterances are weighted evenly and  $\beta > 0$  means that more emphasis is placed on more recent utterances. The motivation for recency weighting is to test the hypothesis that more recent developments in a conversation are more relevant for predicting current
2Obtained from https://huggingface.co/ sentence-transformers/all-mpnet-base-v2
Figure 2: Architecture of the supervised and unsupervised GDS and PF models (top). In GDS space (top right), each cluster is characterized by similar dialogue semantics, and is thus interpreted as the class of typical outcomes for dialogues within. GDS and PF can be used with rollouts (bottom) to allow a dialogue agent to plan ahead.
progression toward a goal. For example, a conversation may start out off-task with participants engaging in small talk, and then later re-focus.
The embedding for dialogue  $D$  with  $|D|$  utterances is thus formulated as  $\mathbf{u}_D = U^T\mathrm{softmax}(\mathbf{r})$  where  $U$  is the matrix of utterance vectors in  $\mathbb{R}^{|D|\times d}$  and  $\mathbf{r}\in \mathbb{R}^{|D|}$  is a vector of evenly spaced real numbers over the interval  $[0,\beta ]$ . The softmax ensures all recency weights sum to 1 and can be interpreted as probabilities as done with attention scores in (Bahdanau et al., 2015; Vaswani et al., 2017). As shown in Figure 3, each utterance is thus weighted by a monotonically increasing probability mass where higher values of  $\beta$  cause more mass to be concentrated at the end of the dialogue.
Figure 3: Recency weight  $\beta$  controls how much emphasis is placed on recent utterances when computing  $\mathbf{u}_D$ .
The unsupervised GDS model is a clustering of the dialogues in their embedding space. The dialogue embeddings are either clustered directly or after projection to a lower-dimensional space
using Parametric UMAP (Sainburg et al., 2020; McInnes et al., 2018a). We experiment with k-means and HDBSCAN (McInnes and Healy, 2017; Campello et al., 2013) to cluster the embeddings. For k-means, we choose the number of clusters  $k$  and train with 10 random initializations. For HDBSCAN, we choose the minimum cluster size and minimum samples hyperparameters, and the optimal number of clusters are selected automatically. Unlike k-means which simply partitions the embedding space, HDBSCAN classifies some embeddings as noise points. Clustering hyperparameters are selected by cross-validation on several metrics as described later in Section 4. The process of constructing the GDS model is illustrated in Figure 2.
The clusters output by this process can be interpreted as the equivalence classes of final global states possible for the task represented in the dialogue dataset. To estimate the global state of an ongoing dialogue  $D'$ , it is embedded as  $\mathbf{u}_{D'} \in \mathbb{R}^d$  in the same manner as the complete training dialogues, followed by optional dimensionality reduction. The trained k-means or HDBSCAN model is then used to assign  $D'$  to one of the existing clusters, or possibly as a noise point in the case of HDBSCAN.
Each cluster is assigned an aggregate acceptabil
ity score by taking an average of acceptability for each dialogue in the cluster. If k-means is used, we aggregate using a  $10\%$  trimmed mean across all dialogues in the cluster. If HDBSCAN is used, a probability is returned for each dialogue representing the likelihood that it is a member of its assigned cluster, so we compute the probability-weighted average across all dialogues in the cluster. Dialogues classified as noise points are ignored.
To visualize the GDS model, Parametric UMAP is used again to project the clustered dialogue embeddings into  $\mathbb{R}^2$  or  $\mathbb{R}^3$ . As shown in Figure 1, the GDS model can be mapped as a scatter plot with each cluster labeled by its aggregate values. If k-means is used, the cluster centroids can be displayed as a bold point within each cluster. HDBSCAN clusters do not have centroids, but they do have a number of representative points that are close to the cluster core. We average these points to simulate a centroid for display purposes, and likewise show it as a bold point within each cluster. To show how an ongoing dialogue  $D'$  traverses the GDS space over time, its embeddings at each turn  $t$  are projected onto the map and connected with line segments to form a path.
3.1.2 Computing Progression
Since each cluster in the GDS space is intended to represent a class of end-task global states, we compute the progression of an ongoing dialogue  $D'$  with respect to the likelihood that its final global state will rest in each individual cluster. Supposing there are  $k$  final clusters after running  $k$ -means or HDBSCAN, we compute a probability vector  $\mathbf{p}_{D'} \in \mathbb{R}^k$  such that  $\mathbf{p}_{D'i} = P(\mathbf{u}_{D'} \in C_i)$  for  $i \in \{1, \dots, k\}$  where  $C_i$  is cluster  $i$ .  $\mathbf{p}_{D'}$  is computed differently for  $k$ -means and HDBSCAN. K-means does not produce a probabilistic soft clustering, so we define  $\mathbf{p}_{D'}$  with respect to the proximity of  $\mathbf{u}_{D'}$  to the centroids of each cluster:
$$
\mathbf {p} _ {D ^ {\prime}} = \operatorname {s o f t m a x} \left(\frac {1}{\left| \left| \mathbf {u} _ {D ^ {\prime}} - \mathbf {c} _ {i} \right| \right| _ {2}}: i \in \{1, \dots k \}\right) \tag {2}
$$
where  $\mathbf{c}_i\in \mathbb{R}^d$  is the centroid of cluster  $i$ . HDBSCAN does produce a probabilistic soft clustering, so in that case  $\mathbf{p}_{D^{\prime}}$  is already computed.
We ultimately want the closest (or most probable) clusters for ongoing dialogue  $D'$  to have the most sway in estimating its progression at the current point in time. That is, if  $D'$  has moved into
a cluster of high-success outcomes, its progression should increase. Likewise if  $D'$  has moved away from such a high-success cluster, either into a lower-success cluster or off-task into a noisy or unknown region of the GDS space, its progression should decrease. Thus, once  $\mathbf{u}_{D'}$  is computed, we estimate its progression as the probability-weighted average of the aggregate acceptability scores assigned to each cluster. This is formulated as
$$
\operatorname {P R O G} \left(\mathbf {u} _ {D ^ {\prime}}\right) = \frac {\mathbf {v} ^ {T} \mathbf {p} _ {D ^ {\prime}}}{\sum_ {i = 1} ^ {k} \mathbf {p} _ {D ^ {\prime} i}} \tag {3}
$$
where  $\mathbf{v} \in \mathbb{R}^k$  is a vector of the aggregate acceptability scores assigned to each cluster. The scaling factor in the denominator ensures that ongoing dialogue embeddings classified as noise points by HDBSCAN will not be assigned progression values close to zero as a consequence of not belonging to any cluster, which can cause significant fluctuation in the progression function as the dialogue traverses noisy regions of the GDS space. 3 Figure 2 illustrates how progression of an ongoing dialogue depends on its position in GDS space.
3.2 Supervised Approach
For the supervised approach, we simply fine-tune RoBERTa (Liu et al., 2019) to directly predict acceptability given the dialogue history text, where all utterances are concatenated into a single sequence. To construct the GDS space we obtain the dialogue level embedding  $\mathbf{u}_D$  directly from the CLS (
) token for each complete dialogue in the training set, and cluster them as in Section 3.1.1. Unlike the unsupervised approach where recency weighting is used to "attend" to more recent parts of the dialogue, the supervised fine-tuning process causes the CLS embedding to aggregate the parts of the dialogue most relevant to the task objective, which is more optimal than the recency heuristic. Also, unlike the unsupervised approach where progression for an ongoing dialogue is computed with respect to its embedding, here progression is directly predicted by RoBERTa. In our experiments we compare RoBERTa-base, RoBERTa-large, and RoBERTa-large-adapted, the latter receiving additional domain adaptation training for dialogue. Domain adaptation is done via Masked Language Modeling (MLM) on a self-generated version of
3For HDBSCAN we also experiment with softmax for re-scaling  $\mathbf{v}^T\mathbf{p}_{D'}$  , giving PROG  $(\mathbf{u}_{D'}) = \mathrm{softmax}(\mathbf{v}^T\mathbf{p}_{D'})$
the Gutenberg Dialogue Dataset (Csaky and Recski, 2021). Hyperparameters and model weights from domain adaptation training are provided with our code release.
3.3 Response Planning
To allow a dialogue agent to use the progression function as feedback for response planning, we adopt Dialogue Rollouts (Lewis et al., 2017) to simulate the outcomes of a set of response candidates. A rollout for a response candidate simulates the next  $N$  turns of the conversation (for both participants) given that candidate is used. At each turn of a negotiation task, Lewis et al. (2017) sample a set of  $c$  response candidates and  $s$  rollouts per candidate. They score each rollout by a deterministic reward (the value of the items "won" by the agent during negotiation), and rank each candidate by the average of its rollout scores. The highest ranking candidate is then selected by the agent. As shown in Figure 2, we generalize this process to any task for which a progression function can be learned, replacing the deterministic reward with the progression function value. To demonstrate this, we fine-tune the 762M parameter DialoGPT (Zhang et al., 2020)<sup>4</sup> as a dialogue response generator and use beam sampling to generate response candidates and rollouts. We select DialoGPT for this task as it is pre-trained on a large Reddit dialogue corpus.
We apply our framework to the Persuasion For Good dataset (Wang et al., 2019), which is a crowdsourced dialogue dataset where the task for an individual playing the role of persuader is to convince another individual playing the role of persuadee to make a donation to a well-known children's charity. We selected this dataset since it has a clear task objective (to solicit donations), but a complex relationship between dialogue content and success. The dataset authors identify 10 distinct persuasion strategies used to solicit donations, where different strategies correlate with donation amount at different strengths. Additionally, participants in Persuasion For Good dialogues complete a pre-task psychological survey, yielding 23 attributes based on the Big-Five personality traits (Goldberg, 1992), the Moral Foundations endorsement (Graham et al.,