SlowGuess/NAACL_2022_OCR huggingface dataset:
Our problem and approach are applicable to any general dialogue control setting. In our experiments, we controlled the customer in task-oriented
dialogue. This is useful for constructing a customer bot that imitates real-life customers. By controlling the customer simulator (for example through control words), we can develop a training environment for coaching customer service agents in a variety of diverse situations. For all datasets, we select control words from the utterances of the customer by selecting the top  $M$  ranked words based on tfidf. For some real-world applications, control words can also be manually selected by a designer.
MultiWoz 2.3: The first dataset we evaluate on is MultiWoz 2.3 (Han et al., 2020), which is widely used in the dialogue community. The dataset has over 10K dialogues and 5 domains.
TaskMaster-3: The second is another commonly used task-oriented dialogue dataset TaskMaster-3 (Byrne et al., 2019). This dataset has 23,757 dialogues in the movie ticketing domain.
Action-Based Conversations Dataset (ABCD): The final dataset (Chen et al., 2021) includes a set of agent-customer conversations focused on solving customer problems. The dataset contains over 10k dialogues and is also focused on one domain.
$\mathcal{W}_{\mathrm{first}}$ : The first baseline is a naive approach that outputs all control words in the first response of the conversation and nothing afterwards, which means words are not appropriately timed.
Fine-tuned: This approach simply generates responses using the fine-tuned language model  $M$ .
Prompt: This method is based on prompting approaches (Li and Liang, 2021; Ribeiro et al., 2018; Jiang et al., 2020; Madotto et al., 2021). Because we focus on the plug-and-play setting, we simply append control words to the beginning of the context and generate using this modified input.
Directed Beam Search (DBS): This is a constrained text generation approach (Pascual et al., 2020), in which keywords are generated using logit modification and beam search. It is not optimized for long-term control and is highly dependent on the ordering of control words.
Constrained Sentence Generation by Metropolis-Hastings Sampling (CGMH):
This method (Miao et al., 2019) is based on stochastic search methods that insert, delete, and replace words in a sentence with the requirement
Methods	LT-SR	f1-score	Human eval	Overall average
Prompt	0.23	0.34	0.87	0.48
DBS	0.42	0.28	0.72	0.47
CGMH	0.90	0.17	0.3	0.46
FOP-retrieval	0.82	0.39	0.82	0.68
FOP-guided	0.74	0.41	0.81	0.67
Table 1: Summary table of results, including long-term success rate (LT-SR) from Figure 4 averaged over datasets for 9 control words, F1-score from the overall F1 column of Table 2 that averages F1 over datasets, and human eval from Table 3 averaged over all metrics and divided by 5 to get a number between 0 and 1.
that control words need to be present. It is neither optimized for long-term generation of control words nor forward generation and is particularly susceptible to aggressively generating all control words in a single response. It was also originally applied to the task of keyword-to-phrase generation so we adapted it to dialogue generation by prompting the language model with the dialogue context and also replaced a bidirectional RNN model with our transformer-based model.
7.1 Aggregated Results
We begin by presenting a top-level overview of our main baselines and methods because each evaluation metric captures a different aspect of performance. Table 1 includes averaged scores across tasks, parameters, and/or metrics for the main results in Tables 2 and 3 and Figure 4. These include results of our two proposed automatic metrics of long-term success rate and control word F1-score (Section 4) as well as human-evaluated quality metrics (Section 7.4). In subsequent sections, we will examine each of these results more closely.
The key insight in these aggregated results is that while FOP-based methods are not always the best-performing system for each metric, they are consistently the most reliable. Specifically, CGMH has high success rate, but lowest F1 and human scores. Prompt, on the other hand has the highest human evaluation scores but the worst success rate. This is not too surprising. It is, after all, an unmodified language model, so it should be fluent and on topic when viewed by a human. However, given its extremely low success rate, it is not viable for long-form controlled generation. In contrast, FOP-based methods are either the top 1 or 2 performing
Figure 4: Long-term success rate computed on simulated roll-outs for MultiWoz, TaskMaster, and ABCD. Details on hyperparameters are in Appendix A.3.
system across all summary statistics.
7.2 Long-term Success Rate
The first analysis involves comparing all methods on long-term success rate, which measures the percentage of control words in generated simulated roll-outs. To do this, we train a separate user model with the training dataset. We perform a roll-out per test example with 10 generated system responses and 10 generated user responses and compute the percentage of control words in the generated system responses. When counting the number of generated words, we compare word stems.
Figure 4 shows the performance of all approaches when varying the number of control words. Both of our approach variants (FOP-retrieval and FOP-guided) have higher success rates than Prompt and DBS. Prompt is the method with the lowest performance because including the control words at the beginning without any re-training doesn't provide the model with sufficient information to generate the control words. DBS does well when there is only a few control words but struggles as the number of control words increases. This is because DBS is not able to filter out words that are irrelevant at the current time step and instead simply tries to generate the words one by one. This method is also unable to handle words when not in the exact order it should appear.
FOP-retrieval, in some cases, has higher performance than FOP-guided because it will get all keywords in the retrieved response correct. FOP-guided can choose to ignore these keywords if the LM overrides it. So, we would expect FOP-retrieval to do better on this metric, compared to
Methods	MultiWoz 2.3	TaskMaster-3	ABCD	Overall avg(f1)
p	r	f1	p	r	f1	p	r	f1
Wfirst	0.25	0.18	0.21	0.22	0.19	0.2	0.29	0.24	0.27	0.23
Fine-tuned	0.64	0.23	0.34	0.82	0.34	0.48	0.68	0.13	0.22	0.35
Prompt	0.45	0.18	0.25	0.81	0.36	0.49	0.69	0.18	0.29	0.34
DBS	0.4	0.2	0.27	0.43	0.27	0.33	0.39	0.17	0.24	0.28
CGMH	0.27	0.18	0.21	0.17	0.03	0.05	0.27	0.22	0.24	0.17
FOP-retrieval	0.38	0.18	0.25	0.68	0.38	0.49	0.65	0.33	0.44	0.39
FOP-guided	0.36	0.18	0.24	0.62	0.48	0.54	0.6	0.36	0.45	0.41
Table 2: Precision, recall, and F1-score for all methods on Multiwoz, TaskMaster, and ABCD. These metrics capture whether the approaches generate control words at the right time by using the control words in the ground truth response as a proxy. The last column is the macro  $f1$ -score average across all datasets.
Methods	FL	CC	RL	CO	DV
DBS	4.60*	3.65†	3.80	2.90	3.10†
CGMH	1.70†	1.24†	1.52†	1.12†	1.82†
FOP-retrieval	4.81	4.77	3.63	2.82	4.35
FOP-guided	4.36†	4.53*	3.77	3.12	4.47
Prompt	4.87	4.98	4.30	4.22	3.42
True	4.88	4.90	4.83	4.92	4.80
Table 3: Human evaluation of simulated roll-outs. FL: fluency; CC: control-consistency; RL: relevance; CO: Coherence; DV: diversity. * and † indicate significant differences from the best result in that column (bolded, excluding True and Prompt) with p-value < 0.05 and < 0.001 respectively, using Welch's t-test. Annotators rated fluency, control-consistency, and relevance per response, while coherence and diversity were annotated per conversation. All metrics are on a scale of 1 to 5.
FOP-guided. We also include an ablation experiment in Appendix A.1.1 to analyze the effect of removing the sliding window in FOP-guided. CGMH seems to do well on long-term success rate, but human evaluation (Section 7.4) results reveal that the generated responses are not very fluent. This method is one that can game previous evaluation metrics, as it tends to condense many or all control words into one utterance. Thus, these approaches are better evaluated through the next set of metrics: precision, recall, and F1-score.
7.3 Control Word P/R/F1
We now measure how well the approaches generate control words at the right time using precision, recall, and F1-score. Table 2 compares these metrics on all datasets. We see that, on average across all datasets, FOP-guided gets higher F1-scores compared with baseline methods. This is because by retrieving similar futures, we are able to guide the language model towards generating control words at
appropriate points in the conversation. FOP-guided does worse on MultiWoz because the dataset contains more domains and has much more variety in the conversations. This diversity makes it hard for retrieval-based methods to successfully find similar conversations to guide generation.
The naive approach  $\mathcal{W}_{\mathrm{first}}$  gets low recall and precision since it only outputs the control words at the first utterance. Similar to  $\mathcal{W}_{\mathrm{first}}$ , CGMH gets low F1-scores because it generates many control words early in the conversation rather than at a natural time. DBS also does not do well on these evaluation metrics as it is highly affected by the order of control words, while our method is able to retrieve similar futures to generate appropriate words at the current time step. Finally, Prompt does well on precision but not on recall as it's not explicitly guided to generate the control words.
7.4 Human Evaluation
Finally, we rate all methods on human evaluation. We follow recent work on good evaluation practices for text generation approaches (Karpinska et al., 2021). Further details are in Appendix A.4.
Fluency: Is the response fluent and grammatical?
Control consistency: When control words appear in the response, are they appropriately used?
Relevance: Is the response a natural reply to the previous utterance in the conversation?
Coherence: Are all of the system responses in the conversation coherent with respect to each other?
Diversity: Is there diversity in the system responses of the conversation?
Two raters annotated each example, and agreement was measured using Krippendorff's alpha for each of the 5 metrics (0.84, 0.74, 0.82, 0.76, 0.67). We present results in Table 3 for all five approaches as well as for the ground truth conversation. We
focus on comparisons between DBS, CGMH, and the FOP methods, as these were the methods that performed comparably on control metrics (at least  $40\%$  on long-term success rate) and thus are reasonable baselines for long-term control.
CGMH consistently gets low scores across all metrics. Compared to DBS, FOP-guided performs similarly on fluency, relevance, and coherence but much better on control-consistency and diversity, which could be because retrieval helps decide naturally what to say throughout the conversation. FOP-guided is at least as good as FOP-retrieval on relevance, coherence, and diversity, while only slightly worse on fluency and control-consistency. This is because FOP-guided uses the context and retrieved sentence to generate a response, while FOP-retrieval selects an already fluent historical response. Overall, human evaluation results highlight that both of our proposed methods generate realistic, coherent text, while also generating a high percentage of control words.
In this paper, we propose the problem of constrained dialogue generation, which involves controlling dialogue responses such that a set of control words appear at some point in the future of the conversation. We propose a new set of metrics as well as a novel method that leverages retrieval of relevant conversations to control future generated responses. We show on three datasets that our method outperforms several constrained text generation baselines on quantitative metrics as well as human evaluation. As far as we are aware, this is the first work to address the problem of long-term control for dialogue generation.
We thank S.R.K Branavan and Derek Chen for their insightful feedback. We thank Tianyi Zhang for his starting code that we built upon in this work. We also want to thank Ethan Elenberg, Felix Wu, Clemens Rosenbaum, Sam Altschul, David Sontag, and the rest of the ASAP research team for all of their feedback in making this work stronger.
Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. 2018. Generating more interesting responses in neural conversation models with distributional constraints. arXiv preprint arXiv:1809.01215.
Pawel Budzianowski and Ivan Vulic. 2019. Hello, it's gpt-2-how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4516-4525, Hong Kong, China. Association for Computational Linguistics.
Hengyi Cai, Hongshen Chen, Yonghao Song, Xiaofang Zhao, and Dawei Yin. 2020. Exemplar guided neural dialogue generation. In International Joint Conference on Artificial Intelligence (IJCAI).
Yu Cao, Wei Bi, Meng Fang, and Dacheng Tao. 2020. Pretrained language models for dialogue generation with multiple input sources. arXiv preprint arXiv:2010.07576.
Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. 2021. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3002-3017, Online. Association for Computational Linguistics.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc'Aurelio Ranzato. 2020. Residual energy-based models for text generation. arXiv preprint arXiv:2004.11714.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2020. Augmenting transformers with knn-based composite memory for dialogue. arXiv preprint arXiv:2004.12744.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833.
Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. arXiv preprint arXiv:1906.09308.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017,
System Demonstrations, pages 43-48, Vancouver, Canada. Association for Computational Linguistics.
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. 2019. Bias correction of learned generative models using likelihood-free importance weighting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jiatao Gu, Kyunghyun Cho, and Victor OK Li. 2017. Trainable greedy decoding for neural machine translation. arXiv preprint arXiv:1702.02429.
Prakhar Gupta, Jeffrey P Bigham, Yulia Tsvetkov, and Amy Pavel. 2020. Controlling dialogue generation with semantic exemplars. arXiv preprint arXiv:2008.09075.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909.
Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using gpt-2. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 583-592.
Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Wei Peng, and Minlie Huang. 2020. Multiwoz 2.3: A multi-domain task-oriented dataset enhanced with annotation corrections and co-reference annotation. arXiv preprint arXiv:2010.05594.
Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. arXiv preprint arXiv:1704.07138.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Jeff Johnson, Matthijs Douze, and Herve Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using mechanical turk to evaluate open-ended text generation. arXiv preprint arXiv:2109.06835.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Phillip Lippe, Pengjie Ren, Honda Haned, Bart Voorn, and Maarten de Rijke. 2020. Diversifying task-oriented dialogue response generation with prototype guided paraphrasing. arXiv preprint arXiv:2008.03391.
Andrea Madotto, Etsuko Ishii, Zhaojiang Lin, Sumanth Dathathri, and Pascale Fung. 2020. Plug-and-play conversational models. arXiv preprint arXiv:2010.04344.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt-based learning for dialogue systems. arXiv preprint arXiv:2110.08118.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. Cgmh: Constrained sentence generation by metropolis-hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6834-6842.
Damian Pascual, Beni Egressy, Florian Bolli, and Roger Wattenhofer. 2020. Directed beam search: Plug-and-play lexically constrained language generation. arXiv preprint arXiv:2012.15416.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pretrained auto-regressive model. arXiv preprint arXiv:2005.05298.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49.
Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. arXiv preprint arXiv:1804.06609.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865.
Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Lei Sha. 2020. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8692-8703.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019. Target-guided open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5624-5634, Florence, Italy. Association for Computational Linguistics.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel-Kedziorski, Jianfeng Gao, Hannaneh Hajishirzi, Mari Ostendorf, et al. 2020. A controllable model of grounded response generation. arXiv preprint arXiv:2005.00613.
Can Xu, Wei Wu, Chongyang Tao, Huang Hu, Matt Schuerman, and Ying Wang. 2019. Neural response generation with meta-words. arXiv preprint arXiv:1906.06050.
Semin Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tur. 2019. Deepcopy: Grounded response generation with hierarchical pointer networks. arXiv preprint arXiv:1908.10731.
A.1 Additional results
A.1.1 Ablation of window in FOP-guided
We ran ablation experiments comparing FOP-guided with a version without the sliding window. Table 4 includes the results for all of the baselines on the most difficult setting for ABCD (9 control words).
Methods	Long-term success rate
Prompt	0.15
DBS	0.38
CGMH	0.91
FOP-retrieval	0.72
FOP-guided	0.69
FOP-guided (no-window)	0.56
Table 4: Ablation experiment for the most difficult setting in ABCD (9 control words). FOP-guided without a sliding window performs worse on long-term success rate.
Our approach FOP-guided gets more than  $10\%$  more control words in simulated rollouts, compared with FOP-guided without the window approach, which highlights the usefulness of the sliding window component. We also compare the two FOP-guided variants when varying the number of control words and see that FOP-guided consistently performs better (Figure 5).
Figure 5: Long-term success rate on ABCD, comparing FOP-guided and FOP-guided without a sliding window.
A.2 Example simulations on ABCD
In Tables 5, 6, 7, 8, and 9, we show some example simulations on the ABCD dataset using a trained agent model for each of the methods.
A.3 Experiment details
We did a hyperparameter search over the following lambda values  $\{0, 5, 10, 15, 20, 25\}$  for all datasets. On both ABCD and MultiWoz, the best hyperparameter for FOP-guided was  $\lambda = 15$  and for DBS, it was  $\lambda = 20$ . For TaskMaster, the best hyperparameter for FOP-guided was  $\lambda = 10$  and for DBS, it was  $\lambda = 15$ . CGMH was run with the recommended hyperparameters from the authors.
For all datasets, we used the number of candidate generations for FOP-guided as  $N = 10$  and the window size for logit modification as  $q = 4$ . The number of examples used for multiple splits of each dataset is as follows: For the ABCD dataset, we used 8034 conversations for training and 1004 conversations each for dev and test splits. In the Multiwoz dataset, we used 8438, 1000, 1000 as train, dev and test splits respectively. Finally, for the Taskmaster-3 dataset, we used 16629, 3564, 3564 as train, dev and test datasets respectively.
We used the GPT2-medium model from the hugging-face repository as the pre-trained language model for all of our experiments. This model contains 345M parameters.