SlowGuess/NAACL_2022_OCR huggingface dataset:
For our language model, we use an off-the-shelf implementation of GPT2-medium (Radford et al., 2018). This language model is finetuned on a transcript of each scenario in the dataset containing the context (title, description) and spoken dialogue. The prices in the dialogue are masked out with a special price token allowing us use GPT to generate templates which we can substitute prices into. The input to the language model is a concatenation of the scenario context and the dialogue history.
The Q-function is parameterized as a feedforward network that maps states and actions into a single scalar representing the Q-value. To process the utterances into the state and action, we separately compute state and action embeddings by taking the average of the masked GPT2 attention embeddings of the entire dialogue history up to the current utterance. These embeddings are then concatenated with the prices (represented as a fraction of the list price) and message types (represented as a one-hot vector) to produce a single vector that is given to the critic as input. The critic is parametrized using a 2-layer feedforward network
with hidden sizes of 256 and ReLU nonlinearities. Additional details about our model architectures, language model sampling method, and how inputs to the Q-function are structured, can be found in Appendix A.1.
Our experimental evaluation aims to compare our proposed goal-directed offline RL dialogue method to both prior dialogue management approaches and language modeling baselines. We conduct two studies: an objective evaluation against other dialogue agents to measure each method's performance in negotiation, and a subjective human study to measure the overall end-to-end performance of the system in a similar manner to prior work (He et al., 2018; Jaques et al., 2019). Qualitative results showing actual dialogue generated from our method can be found in Appendix A.4. We consider 4 baseline approaches. The first is the current state-of-the-art approach for the CraigslistBargain task proposed by He et al. (2018) (referred to as the retrieval-based baseline). This is a hierarchical approach to dialogue generation that separately handles language generation and dialogue management. This method parses utterances into coarse "dialogue acts," which represent high-level categorizations of the utterance such as greetings, offers, or counter-offsers. An RL agent is then trained against a learned model of the environment to select "dialogue acts", and a retrieval-based generator is then used to convert "dialogue acts" back into text. In contrast, our method directly generates text, and does not require any manually designed categorizations of natural dialogue into dialogue acts. Since our method utilizes a modern language model, we also include a pure language modeling baseline, which consists of the same GPT-2 language model Radford et al. (2018) finetuned on the CraigslistBargain dataset using the same method as done in CHAI. This baseline allows us to determine whether any improvement from our method is due to the language model, or to the use of offline RL. Finally, we also evaluate the end-to-end approaches described by Lewis et al. (2017), which include a dialogue agent trained via supervised learning, and an RL agent optimized for the task objective.
5.1 Simulated Evaluation
To evaluate the effectiveness of our offline RL goal-directed dialogue system, we first conducted a sys
Method	vs Rule-based	vs Stingy	vs Utility
Acc%	Revenue	Acc%	Revenue	Acc%	Revenue
CHAI-prop	61.5	0.48 ± 0.39	57.5	0.39 ± 0.35	99.0	0.70 ± 0.17
CHAI-CQL	74.0	0.51 ± 0.33	77.5	0.49 ± 0.30	98.0	0.70 ± 0.19
CHAI-BRAC	62.0	0.52 ± 0.41	47.0	0.38 ± 0.41	99.0	0.71 ± 0.17
Language Model	48.5	0.29 ± 0.32	51.5	0.27 ± 0.28	20.5	0.14 ± 0.28
He et al. (2018) (Utility)	1.0	0.01 ± 0.10	0.0	0.00 ± 0.00	11.0	0.07 ± 0.22
He et al. (2018) (Fairness)	84.0	0.70 ± 0.32	80.0	0.59 ± 0.31	100.0	0.72 ± 0.15
He et al. (2018) (Length)	53.0	0.46 ± 0.43	49.0	0.37 ± 0.38	100.0	0.72 ± 0.16
Lewis et al. (2017) (RL)	83.5	0.17 ± 0.24	83.0	0.19 ± 0.25	64.5	0.46 ± 0.37
Lewis et al. (2017) (SL)	38.5	0.17 ± 0.27	46.5	0.21 ± 0.27	18.0	0.13 ± 0.28
Method	vs Fairness	vs Length	Overall (mean)
Acc%	Revenue	Acc%	Revenue	Acc%	Revenue
CHAI-prop	99.0	0.90 ± 0.15	92.5	0.79 ± 0.27	81.9	0.65 ± 0.34
CHAI-CQL	99.5	0.87 ± 0.14	94.5	0.79 ± 0.24	88.7	0.67 ± 0.29
CHAI-BRAC	100.0	0.85 ± 0.03	91.0	0.76 ± 0.25	79.8	0.65 ± 0.34
Language Model	25.5	0.19 ± 0.35	18.5	0.14 ± 0.32	32.9	0.21 ± 0.31
He et al. (2018) (Utility)	100.0	1.00 ± 0.00	100.0	1.00 ± 0.00	42.4	0.42 ± 0.49
He et al. (2018) (Fairness)	0.0	0.00 ± 0.00	100.0	0.70 ± 0.16	72.8	0.54 ± 0.35
He et al. (2018) (Length)	100.0	1.00 ± 0.00	100.0	0.78 ± 0.18	80.4	0.66 ± 0.36
Lewis et al. (2017) (RL)	88.0	0.26 ± 0.34	71.5	0.31 ± 0.36	78.1	0.28 ± 0.33
Lewis et al. (2017) (SL)	60.0	0.48 ± 0.46	53.0	0.42 ± 0.46	43.2	0.28 ± 0.39
Table 1: Acceptance rates and normalized average revenue generated comparing CHAI using proposal sampling (CHAI-prop), conservative Q-learning (CHAI-CQL), and behavior regularization (CHAI-BRAC). The baselines consist of a retrieval-based agent (He et al., 2018), Lewis et al. (2017) and a fine-tuned language model (higher is better) against 5 different evaluation bots. The mean score across all evaluation bots is reported in the right-most column. Numbers are reported as means and standard deviations over 200 trials.
Metric	Fluency	Coherency	On-Topic	Human-Likeness	Total
CHAI-prop	4.31 ± 0.97	3.91 ± 1.17	4.16 ± 0.99	3.47 ± 1.27	15.84 ± 3.86
He et al. (2018) (Utility)	3.56 ± 1.34	2.47 ± 1.39	3.09 ± 1.40	2.13 ± 1.13	11.25 ± 4.50
Lang. Model	4.06 ± 1.11	2.66 ± 1.36	3.63 ± 1.18	2.50 ± 1.10	12.84 ± 3.66
Table 2: Human evaluation scores comparing CHAI, He et al. (2018), and language model (higher is better). Numbers are reported as means and standard deviations over 32 trials. CHAI scores the highest across all metrics.
tematic study including the 3 variations of CHAI outlined in Sec. 4: the proposal sampling method (CHAI-prop), CQL method (CHAI-CQL), and behavior regularized method (CHAI-BRAC). In order to ensure that the results are not overfit to a single strategy, we run each method against a suite of 5 evaluation buyer agents, based on the retrieval agents presented by He et al. (2018). We choose these agents because they have been evaluated by humans as being human-like and have the strongest performance on the CraigslistBargain benchmark task. Specifically, we use the rule-based and RL agents proposed by He et al. (2018) (trained using "utility", "fairness", and conversation "length" as rewards). To introduce additional variety in negotiation styles, we additionally modify the rule-based agent to offer  $25\%$  of the difference between offers rather than splitting the difference between offers, which we refer to as the "Stingy" rule-based agent. We record the percentage of negotiations that result in an accept and the average normalized revenue generated per negotiation, which totals the average
sale price (rejections have zero revenue) normalized by the listing price of the advertisement. Our results are presented in Table 1.
Overall, we find that among the variations of CHAI, the conservative Q-learning variant performs the best by a small margin, but results are very comparable between all 3 variations. This suggests that the particular choice of offline regularizer is far less important than the CHAI framework of utilizing a pre-trained language model with Q-function scoring. On average, CHAI-CQL performs significantly higher on acceptance rate and similarly on revenue to the next best agent, the retrieval agent (He et al., 2018) using conversation length as reward. Computing statistical significance between these two methods, we find that  $p < 1.96 * 10^{-9}$  using a chi-squared test for acceptance rate, indicating that the difference in acceptance rates is statistically significant. We find that  $p < 0.946$  using a t-test for revenue, indicating that the difference in revenue is not significant. We also note that the performance of CHAI
Title: FULLY REMODELED 4 Bdrm, 1 Bath, Duplex - $3395
Description: 4 bed / 1 bath Type: Duplex City/Neighborhood: Emeryville Showing: By Appointment Only Utilities: Paid by tenant (not included in rent amount) Features: Hardwood Floor, Washer Dryer, newly renovated, stainless steel appliances
Buyer: hi
CHAI: Hi there. Are you interested in my rental?
Buyer: yes, when is the next opening to view the apartment?
CHAI: Today.
Buyer: how much are the utilities per month?
CHAI: Utilities are not included in the rental amount, but I am willing to negotiate the price.
Buyer: since utilities are not included, could you do $3000?
CHAI: I can't go that low. What about $3300?
Buyer: sounds good!
CHAI: Perfect! Thank you!
Buyer: Hi
Retrieval: hi
Buyer: when is the next opening to view the apartment?
Retrieval: its right by apartment merrit
Retrieval: yes they are
Retrieval: deal!
Retrieval: offer $3000
LM: Hi, are you interested in my property?
Buyer: Yes, when is the next opening to view the apartment?
LM: Open until 6pm tomorrow.
Buyer: Sounds good, how much are the utilities per month?
LM: Utilities are included in the monthly rent.
Buyer: Okay, and can you do $3000?
LM:offer $2200
Figure 4: A comparison between similar negotiations talking to CHAI-prop (left), a retrieval-based agent (He et al., 2018) (middle), and language modeling (LM, right) for the same listing. The bot takes the role of the seller, and a human is the buyer. CHAI gives both descriptive responses to questions and reasonable bargaining behavior, whereas the retrieval-based agent only shows good bargaining behavior, and the language model agent only gives descriptive responses.
has significantly less variation across evaluations against different buyer agents than the retrieval-based agents. For example, the retrieval agent with utility reward scores near-zero on 3 evaluations but scores near-perfectly on the other two. This suggests that the CHAI framework produces dialogue agents that are more consistent and less susceptible to exploitation. We also note that implementing the retrieval method (He et al., 2018) requires hand-designing high-level dialogue actions, topic categories, and rules for parsing or labeling these components. These designs are specifically tailored to the CraigslistBargain task, whereas such hand-engineering for CHAI does not exist outside of the interface requirements to the task itself. Thus, CHAI has significantly weaker assumptions, generates dialogue end-to-end via RL, and yet is able to narrowly outperform prior methods. Among prior methods with similar assumptions to CHAI (the language modeling baseline and (Lewis et al., 2017)), CHAI outperforms by a wide margin on both acceptance rates and revenue.
We ran an additional ablation study on the choice of reward in Appendix A.3. We find that this has a significant effect on performance, and we based our reward design on balancing between maximizing acceptance rate (through a rejection penalty) and revenue (through the utility function).
5.2 Human User Study
To evaluate the effectiveness and naturalness of our offline RL goal-directed dialogue system, we conducted a user study with 16 individuals, who were each asked to carry out 2 negotiations with each of three agents. The users were then asked to rate the conversation on fluency, coherency, ontopicness, and human-likeness on a 5-point Likert scale. Fluency specifically refers to the frequency of grammatical and word-choice errors. Coherency measures whether the agent's responses are coherent. On-topic measures how well the agent was aligned with performing the task at hand. Finally, human-likeness measures how similar the agent's responses were to a human. Because of the cost of human evaluations, we were limited in our ability to evaluate as many baselines. Therefore, we chose methods that were the most directly comparable - the simplest variation of CHAI (CHAI-prop) optimized for utility, evaluated against the utility-optimized agent from He et al. (2018), and a language model baseline that shares the same finetuning procedure as CHAI. Additional details of the user study, including the questions posed to the users and statistical significance tests, are included in Appendix A.3.1.
Results are shown in Table 2. We ran a one-way repeated measures ANOVA test, and found that the type of agent used leads to statistically sig
nificant rating differences for all metrics (with at least  $p < 0.01$ ). CHAI outperforms both baselines on almost all metrics, except for fluency, where both CHAI and the language model perform similarly. The fact that fluency is similar between the two models makes sense, since both methods use a GPT-2 model to generate utterances. However, the language modeling baseline lacks an understanding of the task goal, and therefore makes unreasonable offers or responds in illogical ways (see Appendix A.4 for examples). It therefore scores lower on other metrics as compared to CHAI. This result suggests that the ability of language models to execute goal-directed dialogues is limited by a lack of awareness of the task objectives, and that offline RL potentially addresses this issue, producing dialogue that is perceived as more coherent, task-oriented, and human-like.
In Fig. 4 we show a comparative example between CHAI, the retrieval-based agent (He et al., 2018), and the language modeling baseline on the same scenario with human responses. CHAI and the language modeling baseline tend to produce more specific responses to the prompt due to the use of a language model, rather than generating text via the usage of templates. For example, when presented with a questioned about utilities and viewing time, both methods are able to answer the question, whereas the retrieval-based agent gives a nonsequitur answer. However, the language modeling baseline struggles in understanding prices, and offers  $2200 when the buyer requested$ 3000. CHAI is able to demonstrate understanding both language and the flow of the negotiation by offering reasonable counter-offers to the user, such as responding to a low offer with "I can't go that low" and offering a higher counter-offer that the Buyer accepts. Additional examples from our human evaluation can be found in Appendix A.4.
We presented a system for goal-directed dialogue based on combining offline RL with finetuned language models. CHAI learns with RL, but does so from offline datasets of human dialogue. The language model allows CHAI to benefit from large-scale unsupervised pre-training, and the offline RL component enables CHAI to select responses that are more likely to lead to a successful task outcome. Quantitatively, CHAI achieves higher acceptance rate at higher revenue than prior dialogue manage
ment systems designed for this task.
Goal-oriented dialogue agents have many potentially useful applications such as building personal assistants, improving accessibility to technology for the disabled or the elderly, and simply saving time by automating menial tasks. Of course, as with any natural language generation technology, this kind of method can be used both beneficially and maliciously, for example by users who aim to create intentionally deceptive and realistic agents.
While CHAI provides a proof-of-concept that offline RL can successfully learn complex human-interactive tasks such as dialogue, it also has limitations. The goal of RL is to maximize reward, which can lead to unintended responses – for example, without additional objective terms, there is no reason for CHAI to be truthful. Similar issues affect language models more broadly, though we anticipate that it would be easier to address such issues in RL by employing better reward design. Although reward design can itself be a difficult problem, it does provide a more direct lever for influencing the agent's behavior than what is available in standard language models, which must be directed either through the choice of training data or other indirect mechanisms. In CHAI, we only investigated a single task due to architectural constraints, and the exact same architecture presented in this paper (e.g. a price prediction head) may not be directly transferable to other domains. However, using a value-based selection mechanism is more generally applicable to any goal-oriented task.
An exciting direction for future work is to extend offline RL to address a wider range of human-interactive tasks, particularly tasks with longer-range dependencies and delayed rewards, where complex task goals can lead to the emergence of dialogue that is ultimately more useful to human users.
We thank Natasha Jacques, Daniel Fried, Dilek Hakkani-tur, Yang Liu, Alexandros Papangelis, and Wei Wei for insightful discussions. We would also like to thank all of the anonymous participants in the user study. This research was supported by the Office of Naval Research.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. arXiv preprint arXiv:1607.00070.
Chieh-Yang Chen, Pei-Hsin Wang, Shih-Chieh Chang, Da-Cheng Juan, Wei Wei, and Jia-Yu Pan. 2020. Airconcierge: Generating task-oriented dialogue via efficient large-scale knowledge retrieval. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 884-897.
Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019. Semantically conditioned dialog response generation via hierarchical disentangled self-attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696-3709.
Grace Chung. 2004. Developing a flexible spoken dialog system using simulation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 63-70.
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45-51.
Wieland Eckert, Esther Levin, and Roberto Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 80-87. IEEE.
Mihail Eric and Christopher D Manning. 2017. A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 468-473.
Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy networks with two-stage training for dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 101-110.
Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research &
Development in Information Retrieval, pages 1371-1374.
Milica Gašić, Filip Jurčíček, Blaise Thomson, Kai Yu, and Steve Young. 2011. On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 312-317. IEEE.
Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In Ninth International Conference on Spoken Language Processing.
Kailirroi Georgila and David Traum. 2011. Reinforcement learning of argumentation dialogue policies in negotiation. In Twelfth Annual Conference of the International Speech Communication Association.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. 2020. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. arXiv preprint arXiv:2007.11091.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In *Conference on Empirical Methods in Natural Language Processing*, pages 2333-2343. Association for Computational Linguistics.
Peter A Heeman. 2009. Representing the reinforcement learning state in a negotiation dialogue. In 2009 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 450-455. IEEE.
James Henderson, Oliver Lemon, and Kallirroi Georgila. 2008. Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets. Computational Linguistics, 34(4):487-511.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. 2018. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293.