SlowGuess/NAACL_2022_OCR huggingface dataset:
2011), the Schwartz Portrait Value (Cieciuch and Davidov, 2012), and the Decision-Making style (Hamilton et al., 2016) questionnaires for each individual. The dataset authors demonstrated varying degrees of correlation between these psychological attributes and the end-task donation amount. The complexity in measuring progression in this context, coupled with it being a relatively small dataset, makes Persuasion For Good an interesting and challenging testbed for our framework. Persuasion For Good contains 1017 dialogues, each with approximately 10 turns per speaker (20 utterances).
4.2 Progression Function Experiments
As the objective of the task is to solicit donations, we consider the end-dialogue persuadee donation amount to be the primary dialogue success indicator. We also augment the dataset by computing average dialogue sentiment. To compute sentiment we use a RoBERTa model fine-tuned on the sentiment classification task of the TweetEval benchmark (Barbieri et al., 2020), which was publicly released by the benchmark authors. We score sentiment at the utterance level in the range  $[-1,1]$  by multiplying the sentiment class probabilities predicted by RoBERTa for negative, neutral and positive by  $\{-1,0,1\}$  respectively and summing the result. We then average the utterance-level sentiment score for each dialogue.
We filter the dataset to eliminate dialogues with end-task donation amounts outside the allowed task parameters (between  $0 and$ 2 USD), and use a regular expression to filter out dialogues where the persuadee fails to make a donation after promising a non-zero dollar amount in the conversation. After filtration we are left with 751 dialogues for our study. We split the dialogues into a training and test set, leaving 577 dialogues for training and 174 for testing. After splitting, we mean-center the dialogue values in the training set for each attribute and scale them to have unit variance. We apply the same transformation to the test set using the distribution parameters of the training set. Our final pre-processing step is to compute the acceptability score. To do this, we compute the covariance matrix of the dialogue-level attribute values in the training set, which include the donation amount and psychological attributes for both the persuader and persuadee from the original dataset, along with
Obtained
huggingface.co/cardiffnlp/
twitter-roberta-base-sentiment
<sup>4</sup>Obtained from https://huggingface.co/microsoft/DialoGPT-large
our computed sentiment scores. Since the values are all standardized, the covariances are equivalent to Pearson's  $r$ . We select the covariances of all attributes with respect to the persuadee donation amount (see Figure 5 in Appendix B) and define the acceptability score of each dialogue  $D$  as defined in Section 3. We use the same covariances obtained from the training set to compute acceptability scores on the test set. After pre-processing, the training set has 52 total attributes. These include the persuadee/persuader donation amounts, psychological variables, sentiment, and the acceptability score.
4.2.1 Progression Model Training
We train four progression models as outlined in Sections 3.1 and 3.2: (1) Unsupervised, (2) RoBERTa-base, (3) RoBERTa-large, and (4) RoBERTa-large-adapted. For each model,  $10\%$  of the training set is held out as a validation set (58 dialogues). For the unsupervised model, a grid search is run for the hyperparameters (e.g., # clusters, recency  $\beta$ , dim. reduction, etc.) over the validation set, and the final model is obtained by re-training over the full training set using the best hyperparameters. The final model uses k-means for clustering with  $k = 21$  and recency weight  $\beta = 0.3$ . A complete hyperparameter listing and details on the grid search can be found in Appendix F. For the supervised RoBERTa models, fine-tuning is done with AdamW (Loshchilov and Hutter, 2019) and an initial learning rate of  $3 \times 10^{-5}$  for a maximum of 30 epochs. Early stopping is used over the validation set with the checkpoint corresponding to the lowest validation loss selected as the final model.
4.2.2 Automatic Evaluation
We evaluate the progression models on the following automatic metrics: (1) Mean Absolute Error (MAE) in predicting dialogue acceptability, and (2) Pearson's correlation  $(\mathbf{r})$  between overall PF slope and dialogue acceptability. With MAE we validate that the progression function is able to estimate success of a complete dialogue, while PF slope correlation validates that during an ongoing dialogue, progression increases over time for high-success dialogues and decreases over time for low-success dialogues. To measure PF slope correlation, we fit a least-squares regression line to the progression curve of each dialogue in the test set, and measure Pearson's  $r$  between the regression slopes and their corresponding acceptability scores. For robustness
Table 1: Progression Function Auto Eval Results: Mean (SD) of MAE and Pearson's  $r$  across runs.
Model	MAE	r
Unsupervised*	1.37 (± 0.01)	0.40 (± 0.02)
RoBERTa-base	0.99 (± 0.06)	0.50 (± 0.04)
RoBERTa-large	0.97 (± 0.10)	0.54 (± 0.06)
RoBERTa-large-adapted	0.95 (± 0.05)	0.57 (± 0.03)
For Pearson's  $r, p < 0.01$  for all runs (two-tailed;  $H_0$  is non-correlation).  
* Hyperparameters of the unsupervised model can be found in Appendix G.
we repeat this evaluation 33 times with varying initialization seeds for each model type (final hyperparameters stay constant) and report the means and standard deviations across runs in Table 1.
Unsurprisingly, the supervised models outperform the unsupervised model on both metrics, which is expected since they directly optimize a mean squared error objective. Of the supervised models, the RoBERTa-large instances perform the best, with dialogue domain adaptation boosting each metric. Pearson's  $r$  is significant at the  $p < 0.01$  level for all runs (the null hypothesis is non-correlation).
4.2.3 Manual Evaluation
To obtain a more precise evaluation, we asked three annotators to estimate sentence-level progression for dialogues in our test set. Two graduate students and one postdoc in our lab served as our annotators. For each of twelve randomly selected test dialogues, each annotator rated all sentences on a scale of  $\{-1, 0, 1\}$  for progression, with -1 indicating regression from the task goal, 0 indicating neutral progression, and +1 indicating progression toward the task goal. Altogether, each annotator provided 431 sentence ratings across 244 utterances. After aggregating at the utterance level, average inter-annotator agreement is 0.57 (Cohen's kappa). For each dialogue, the cumulative sum of the utterance-level manual ratings creates a ground-truth progression curve, as shown in Figure 4.
We evaluate the progression models against the ground-truth curves using Pearson's correlation  $(r)$  since the PF output and cumulative manual ratings are continuous and on different scales. We report the following correlations: (1) between utterance-level PF value and ground-truth value (utt), (2) between utterance-level PF slope and ground-truth slope (utt-sl), (3) between dialogue-level PF slope and ground-truth slope (dlg-sl), and (4) between dialogue-level PF slope and the final ground-truth progression value (dlg-sl-f). We repeat this evaluation for each of the 33 uniquely initialized model in
Table 2: Progression Function Manual Eval Results: Mean (SD) of  $\mathbf{{utt}},{\mathbf{{utt}} - \mathbf{{sl}}},\mathbf{{dlg}} - \mathbf{{sl}},\& \mathbf{{dlg}} - \mathbf{{sl}} - \mathbf{f}$  across runs.
Model	utt	utt-sl	dlg-sl	dlg-sl-f
Unsuperv.	0.09 (± 0.13)	0.04 (± 0.01)	-0.01 (± 0.07)	-0.07 (± 0.07)
RoBERTa-b	0.39 (± 0.07)	0.18 (± 0.03)	0.26 (± 0.06)	0.26 (± 0.06)
RoBERTa-l	0.39 (± 0.08)	0.18 (± 0.06)	0.34 (± 0.09)	0.35 (± 0.10)
RoBERTa-l-a	0.48 (± 0.05)	0.24 (± 0.03)	0.43 (± 0.06)	0.41 (± 0.07)
$b =$  base;  $l =$  large;  $l - a =$  large-adapted.
1. ER: Hello, how are you today?
20. EE: No I am sorry again. I will only do this in my town 21. ER: This can go to sponsor a child in your town, ...
Figure 4: Left: ground-truth progression curve given by the cumulative sum of utterance-level manual ratings. Right: estimated curve from RoBERTa-large-adapted.
stances of each type from Section 4.2.2, averaging each metric across raters and then across runs. We report the aggregate means and standard deviations across runs in Table 2. A complete listing of results for each rater (averaged across runs) is provided in Appendix C, along with detailed explanations of each metric (Appendix D). Additionally, Figure 10 in Appendix E provides examples of disagreement between the PF and ground-truth progression curves which can be challenged despite high inter-annotator agreement, demonstrating difficulty in establishing ground-truth for this open-ended task.
4.2.4 Benefit of Domain Adaptation
To verify the beneficial effect of domain adaptation we perform two-tailed paired t-tests to confirm the differences in means between RoBERTa-large and RoBERTa-large-adapted on all automatic and manual metrics. For each metric, we pair the results from both models for each run of the same seed, since their regression heads would have received identical initializations. We find that the means of  $\mathbf{utt}$ ,  $\mathbf{utt-sl}$ ,  $\mathbf{dlg-sl}$ , and  $\mathbf{dlg-sl-f}$  differ at the  $p < 0.01$  significance level, and the means of the automatic Pearson's  $r$  metric differ at the  $p < 0.05$  level. This confirms our intuition that domain adaptation for dialogue prior to fine-tuning the regression objective aids generalization in this task.
4.3 Rollout Experiments
To demonstrate the ability of the PF to guide a dialogue agent, we use it to score rollouts generated
with DialogGPT as described in Section 3.3. Specifically, we design a self-play experiment to automatically evaluate the effect of PF-guided rollouts on the success of the solicitation task in Persuasion For Good. The following summarizes the experimental setup, procedure, and results.
4.3.1 Exexperimental Setup
First, we fine-tune DialogGPT to generate responses on Persuasion For Good. We add speaker control tokens to the vocabulary so that the model can be conditioned to generate as the persuader or persuadee, respectively. Training is done with AdamW (initial  $\mathrm{lr} = 5 \times 10^{-5}$ ) for 6 epochs with early stopping over a  $10\%$  validation set using perplexity. The final model checkpoint was selected after 3 epochs, achieving validation perplexity of 8.82.
We then select a progression model to use for all self-play runs. Since the supervised RoBERTa-large-adapted model achieved the best average scores across all automatic and manual evaluations, we randomly select one of the 33 RoBERTa-large-adapted instances from Section 4.2.2 to use for all runs. We use this instance for rollout scoring and to measure the progression of each self-play dialogue.
Finally, we train a binary classifier to identify if the persuadee has stated the intent to donate in a conversation, which we use to detect successful self-play dialogues. We fine-tune a RoBERTa model as a classifier using just the persuadee's utterances as input and use binarized donation labels in Persuasion For Good as targets. Specifically, for each dialogue the label is 0 if the donated amount is \(0, otherwise it is 1. We use the manually verified intended donation labels from Persuasion For Good "ANNSET" for our validation and test splits and use the remaining end-task donation labels for training. Training is done with early stopping over the validation split using macro F1. The final model checkpoint achieved test F1 of 0.89 and test accuracy of 0.90. All three trained models used in this experiment are available to download (see our code release for instructions and hyperparameters).
4.3.2 Self-Play Procedure
From our test set of 174 dialogues, we manually filter out 41 in which the persuadee pledges a donation within the first 10 utterances, leaving 133 remaining conversations. For each of these, the response generator is given the first 10 utterances as context and tasked to complete a second set of 10 utterances, playing the role of both the persuader
Table 3: Rollouts self-play results: Mean (SD) of progression, sentiment, and % donated across runs.
Mode	Prog.	ER Sent.	EE Sent.	EE Don. %
No RO	0.01 (± 0.24)	0.51 (± 0.03)	0.44 (± 0.06)	38% (± 6%)
2x2x3	0.69 (± 0.29)	0.62 (± 0.05)	0.49 (± 0.07)	45% (± 10%)†
3x3x5	0.95 (± 0.16)	0.66 (± 0.02)	0.52 (± 0.04)	47% (± 11%)
All  $2 \times 2 \times 3$  and  $3 \times 3 \times 5$  means significant at  $p < 0.01$  (or 0.05 if marked  $\dagger$ ) when compared to No RO with two-tailed paired t-tests. ER and EE refer to the persuader and persuadee respectively.
and persuadee. Since the task is solicitation, we allow the generator to use rollouts only when acting as the persuader. We perform the self-play experiment using three persuader modes: (1) with no rollouts (No RO), (2) with 2 response candidates, 2 rollouts per candidate, and 3 utterances per rollout  $(2\mathbf{x}2\mathbf{x}3)$ , and (3) with 3 response candidates, 3 rollouts per candidate, and 5 utterances per rollout  $(3\mathbf{x}3\mathbf{x}5)$ . For each utterance in each rollout, we use beam sampling with num_beams  $= 6$ , top_k = 50, top_p = 0.95, and temperature  $= 1.5 + 0.002 \cdot T$  where  $T$  is the number of tokens in the dialogue history. After generation, we compute the following metrics for each dialogue: (1) progression using the selected RoBERTa-large-adapted instance (Prog.), (2) persuader and persuadee sentiment using the sentiment classifier from Section 4.2 (ER Sent. & EE Sent.), and (3) the percent of test dialogues where the persuadee pledges a donation amount or explicitly states intent to donate, as detected by the binary donation intent classifier (EE Don.%).
4.3.3 Self-Play Results
For robustness we repeat this procedure 5 times with varying generation seeds for each persuader mode. In total, 1,995 self-play dialogues are completed (133 dialogues for each of 3 modes for each of 5 seeds). We average each metric across dialogues and then across runs, and report the aggregate means and standard deviations across runs. Additionally, to verify the benefit of rollouts, we perform two-tailed paired t-tests to confirm the differences in means between the rollout-enabled modes (2x2x3 and 3x3x5) and the baseline (No RO). For each metric, we average the results across runs and pair these averages from both modes for each dialogue. Results are shown in Table 3.
We observe that the mean progression increases significantly when rollouts are used. This is expected since response candidates with the highest average end-rollout progression are selected. We also observe that rollouts lead to higher average sentiment for both the persuader and persuadee,
which makes sense due to the correlation between sentiment and the acceptability score (see Figure 5 in Appendix B). Finally, rollouts yield a higher percentage of dialogues with a pledged or intended donation.  ${}^{6}$  All of these results are significant at the  $p < {0.01}$  level except for EE Don.% in  $2 \times  2 \times  3$  mode which is significant at  $p < {0.05}$  .
Although progression is noticeably higher for the  $3 \times 3 \times 5$  mode than for the  $2 \times 2 \times 3$  mode (0.95 vs 0.69), all other metrics are close between these modes with a small advantage in  $3 \times 3 \times 5$  mode. This suggests that scaling rollout simulations can be beneficial, but there may be diminishing returns for simulation size. Example self-play dialogues are provided in Tables 7, 8, and 9 in Appendix H.
5 Limitations & Future Direction
We recognize several limitations of our study which warrant follow-up investigation. This study focuses on a single task and dataset, and thus is subject to the assumptions and biases therein. Since we intend our framework to be general, it is prudent to perform additional studies to verify the efficacy of our methods on a variety of datasets spanning multiple dialogue domains and tasks. Also, although we provide automatic evaluation of the ability of rollouts to improve performance on a solicitation task, we cannot assume that humans would respond in the same way as DialogoGPT. Thus, human evaluation is needed to further validate this approach.
In this work we introduced the concept of global dialogue state and proposed a framework with which a dialogue agent can gain awareness of where an ongoing conversation is headed, the likelihood of a successful outcome, and how its own response decisions impact the overall direction of the dialogue. We demonstrated that an unsupervised approach to modeling the GDS space and progression function is feasible, which is useful in sparsely-labeled settings. However, we showed that with domain-adaptation pre-training for dialogue, supervised methods are preferable when labels are available. Finally, we demonstrated how using the PF as a feedback mechanism via dialogue rollouts allows an agent to improve outcomes on a solicitation task.
<sup>6</sup>We randomly selected 30 of the 1,995 self-play dialogues and manually inspected them for donation intent. We found the classifier labeled 26 of them correctly (87% accuracy).
Ethical Dialogue Systems
We acknowledge the potential risks inherent in the deployment of goal-oriented dialogue systems, and especially note that care must be taken to ensure persuasive dialogue systems are designed for beneficial use as discussed by Wang et al. (2019). Concretely, when applying our framework, care must be taken to ensure that the goal of the system (defined by the primary success attribute of the acceptability score) should be generally accepted as beneficial. For example, our basis for dialogue acceptability in this work is with respect to raising money for children's charity. In general, the achievement of the system's goal should not intentionally lead the user or any other party to harm. Additionally, the definition of acceptability, through its primary or any other correlated attributes, should not allow for discriminative responses, purposefully malicious discourse, or other violations of accepted ethical standards. For example, we include sentiment as secondary attributes in the acceptability score, which, when applied via dialogue rollouts, encourages the system to be courteous, polite, and respectful. It is possible with minimal effort to include further secondary attributes that identify bias, hate speech, and other indicators to help the system remain safe to use.
Annotator Compensation
All manual annotators were recruited on a voluntary basis in an educational setting and did not receive or expect monetary compensation. Specifically, two graduate students and one postdoc in our lab served as our annotators.
All training and inference in this work was done with two NVIDIA Quadro RTX 8000 GPUs. The most compute-intensive portion of the work was the additional domain adaptation pre-training for RoBERTa-large-adapted (see Section 3.2), which took approximately two weeks. After that the multi-seed self-play evaluations took approximately four days, and all other operations (e.g., training and evaluating PF models, fine-tuning DialogGPT) took 24 hours or less.
We would like to thank our manual annotators for their valuable contribution and the anonymous re
viewers for their helpful feedback. This paper is based upon work supported in part by the United States Air Force under Contract No. FA8750-21-C-0075 and in part by the IBM Corporation under the Artificial Intelligence Research Collaboration Agreement No. W1771793 between IBM and Rensselaer. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of USAF or IBM Corporation.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaojiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Vevake Balaraman, Seyedmostafa Sheikhalishahi, and Bernardo Magnini. 2021. Recent neural methods on dialogue state tracking for task-oriented dialogue systems: A survey. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 239-251.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1644–1650, Online. Association for Computational Linguistics.
S.A. Beebe and J.T. Masterson. 2000. Communicating in Small Groups: Principles and Practices. Longman.
F.J. Bernieri and R. Rosenthal. 1991. Interpersonal coordination: Behavior matching and interactional synchrony. In R.S. Feldman and B. Rime, editors, Fundamentals of nonverbal behaviors. Studies in emotion and social interaction. Cambridge University Press.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In *Pacific-Asia conference on knowledge discovery and data mining*, pages 160–172. Springer.
Justine Cassell, Alastair Gill, and Paul Tepper. 2007. Coordination in conversation and rapport. In Proceedings of the Workshop on Embodied Language Processing, pages 41-50, Prague, Czech Republic. Association for Computational Linguistics.
Jan Cieciuch and Eldad Davidov. 2012. A comparison of the invariance properties of the pvq-40 and the pvq-21 to measure human values across german and polish samples. In Survey Research Methods, volume 6, pages 37-48.
Richard Csaky and Gábor Recski. 2021. The Gutenberg dialogue dataset. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 138-159, Online. Association for Computational Linguistics.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association.
William Falcon and The PyTorch Lightning team. 2019. Pytorch lightning.
Ray Friedman. 2004. Studying negotiations in context: an ethnographic approach. *Internat'l Negotiation*.
Lewis R Goldberg. 1992. The development of markers for the big-five factor structure. Psychological assessment, 4(1):26.
Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology, 101(2):366.