SlowGuess/NAACL_2022_OCR huggingface dataset:
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems.
Esther Levin, Roberto Pieraccini, and Wieland Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on speech and audio processing, 8(1):11-23.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In *Conference on Empirical Methods in Natural Language Processing*, pages 2443-2453.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep reinforcement learning for dialogue generation. In Conference on Empirical Methods in Natural Language Processing.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. International Conference on Learning Representations.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of NAACL-HLT, pages 2060-2069.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2017. Coherent dialogue with attention-based language models. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3252-3258. AAAI Press.
Tim Paek. 2006. Reinforcement learning for spoken dialogue systems: Comparing strengths and weaknesses for practical deployment. In Proc. Dialog-on-Dialog Workshop, Interspeech. CiteSeer.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan-deh, Lars Liden, and Jianfeng Gao. 2020. Soloist:
Few-shot task-oriented dialog with a single pretrained auto-regressive model. arXiv preprint arXiv:2005.05298.
Roberto Pieraccini, David Suendermann, Krishna Dayanidhi, and Jackson Liscombe. 2009. Are we there yet? research in commercial spoken dialog systems. In International Conference on Text, Speech and Dialogue, pages 3-13. Springer.
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Herve Frezza-Buet. 2011. Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):1-21.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. OpenAI Blog.
Abdelrhman Saleh, Natasha Jaques, Asma Ghandehar-ioun, Judy Shen, and Rosalind Picard. 2020. Hierarchical reinforcement learning for open-domain dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8741-8748.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152.
Iulian Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2017. Multiresolution recurrent neural networks: An application to dialogue response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Satinder Singh, Michael Kearns, Diane Litman, and Marilyn Walker. 1999. Reinforcement learning for spoken dialogue systems. Advances in neural information processing systems, 12:956-962.
Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach. Oxford University Press on Demand.
Pei-Hao Su, Paweł Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 147-157.
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arXiv preprint arXiv:1606.02689.
Pei-Hao Su, David Vandyke, Milica Gašić, Dongho Kim, Nikola Mrkšić, Tsung-Hsien Wen, and Steve Young. 2015. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. In Sixteenth Annual Conference of the International Speech Communication Association.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. 2020. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. arXiv preprint arXiv:2006.06814.
Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393-422.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2021. Alternating recurrent dialog model with large-scale pre-trained language models. pages 1292-1301.
Yifan Wu, George Tucker, and Ofir Nachum. 2019. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361.
Denis Yarats and Mike Lewis. 2018. Hierarchical text generation and planning for strategic dialogue. In International Conference on Machine Learning, pages 5591-5599.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky. 2016. Strategy and policy learning for nontask-oriented conversational systems. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 404-412.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1208-1218.
For our language model, we use an off-the-shelf implementation of GPT2-medium (Radford et al., 2018). This language model is finetuned on a transcript of each scenario in the dataset containing the context (title, description) and spoken dialogue. The prices in the dialogue are masked out with a special price token allowing us use GPT to generate templates which we can substitute prices into. To produce language samples, we concatenate the context of the scenario along with the dialogue history and feed it to the language model. The language model then generates the next utterance. This process is repeated in order to get multiple samples.
The Q-function is a feedforward network that maps the states and actions into a single scalar representing the Q-value. To process the utterances into the state and action, we separately compute state and action embeddings by taking the average of the masked GPT2 attention embeddings of the entire dialogue history up to the current utterance. State and action embeddings are then concatenated with the prices (represented as a percentage of the list price) and message types (represented as a one-hot vector) to produce a single vector that is given to the critic as input. The critic is parametrized as a 2-layer feedforward network with hidden sizes of 256 and ReLU nonlinearities.
The equations below describe the inputs of the Q-function. "STRCAT" is a custom string formatting function that concatenates the context and utterances in the dialog and prefixes each utterance with the string "Buyer:" or "Seller:". "EMBED" calculates the masked attention embeddings from GPT2. dialog represents the dialog history up to the current state,  $[s_{u,1},a_{u,1},\ldots ,a_{u,t - 1},s_{u,t}]$ , and candidate denotes a candidate utterance  $a_{u,t}$  generated by the language model that is being considered for scoring.
$$
\begin{array}{l} s _ {\text {e m b e d}} = \text {E M B E D} (\text {S T R C A T} (s _ {\text {c o n t e x t}}, \text {d i a l o g})) \\ a _ {\text {e m b e d}} = \text {E M B E D} (\text {S T R C A T} (s _ {\text {c o n t e x t}}, \text {d i a l o g} + \text {c a n d i d a t e}) \\ s = \left[ s _ {\text {e m b e d}}, s _ {\text {p r i c e}}, s _ {\text {t y p e}} \right] \\ a = \left[ a _ {\text {e m b e d}}, a _ {\text {p r i c e}}, a _ {\text {t y p e}} \right] \\ q = Q (s, a) \\ \end{array}
$$
Hyperparameter Selection. For our Q-learning algorithm, we used default hyperparameters from an SAC implementation and did not vary the parameters. We used a critic learning rate of  $3 * 10^{-4}$ , and a soft target update rate of 0.05. For the language model architecture, we finetuned 2 GPT models architectures (GPT2-small, and GPT2-medium). In order to select which model to use, we manually rated samples from the language generated in their quality across both model types and checkpoints, and selected the best performing model. We selected the GPT2-medium architecture at training epoch 2000.
Compute Resources. We finetuned our language models on TPUs (TPU v3-8, 16GB memory per core with 8 cores) within a GCP instance. We trained our Q-learning models on an internal compute cluster using an Nvidia 1080 GPU (12 GB memory).
A.3 Reward Ablation Study
Because we are limited in the number of evaluations possible in a human study, we use a simulated evaluation against another chatbot to run an ablation study measuring the effect of using different reward functions. Specifically, we instantiate a rule-based dialogue manager proposed by (He et al., 2018) as the "buyer" and negotiate with it on randomly sampled scenarios from the dataset. This is done for both our method and the baselines, and the results are tabulated in Table 3.
We evaluated 5 variations of CHAI. CHAI(final) is the method used in our paper, which uses 2 components to the reward: positive reward for the price the item was sold at, and a penalty for the episode ending in a rejection. The specific reward used was  $10^{*}$  the price sold (normalized by the list price) if the offer was accepted, or a penalty of -20 if the offer was rejected. CHAI(penalty) uses the same reward,
except with an increased rejection penalty. CHAI(accept) is given a positive reward of  $+20$  for episodes ending in an accept and negative reward of  $-20$  for episodes ending in a rejection, without regard to the price. CHAI(utility) is rewarded solely for the price an item is sold at at  $10^{*}$  the price sold. Finally, CHAI(fair) is rewarded for negotiating to a midpoint price between the buyer and seller's target prices.
We report the acceptance rate, average revenue, and average offers made and offers accepted. We see that CHAI(accept), CHAI(fair) and CHAI(penalty) achieve higher acceptance rates, but offer lower prices on average. In contrast, CHAI(utility) and CHAI(final) offer higher prices with lower acceptance rates, with the pure utility optimizing agent CHAI(utility) offering the highest prices with the lowest acceptance rates. Thus, we can see how changing the reward function can significantly affect the behavior of the resulting agent.
Method	Accept Rate	Prices Offered	Prices Accepted	Revenue
CHAI(accept)	0.74	0.80 ± 0.15	0.74 ± 0.13	0.55 ± 0.35
CHAI(fair)	0.76	0.80 ± 0.15	0.75 ± 0.13	0.57 ± 0.34
CHAI(penalty)	0.90	0.77 ± 0.14	0.77 ± 0.12	0.68 ± 0.26
CHAI(utility)	0.34	0.96 ± 0.29	0.76 ± 0.13	0.29 ± 0.42
CHAI(final)	0.66	0.84 ± 0.14	0.87 ± 0.07	0.51 ± 0.38
Table 3: Mean and standard deviation of revenue, acceptance rate, and prices sold & offered, over 50 samples when negotiating against a baseline chatbot. Revenue and prices are reported as a fraction of the list price.
A.3.1 User Study Parameters
Setup. We conducted our user study through a web-interface, where an advertisement from the test set of CraigslistBargain is displayed to the user. This is shown in Fig. 5. Users were instructed to "type any message to speak with the bot and negotiate", and a chatbot agent replied to each message. After the user is satisfied with the negotiation, they are asked to indicate whether they want to accept or reject the current offer on the item.
Figure 5: The interface presented to participants in the user study. We display the title, price, and description of the advertisement, and users are able to chat with the bot through a textbox.
After interacting with the chatbot agent, users were given a survey and asked to rate the bot on a 5-point Likert scale: strongly agree (5), agree (4), neutral (3), disagree (2), strongly disagree (1). The ratings were:
- The bot was fluent (did not make grammatical or word choice errors).
- The flow of the conversation was coherent.
- The bot was on-topic.
- The bot demonstrated human-like behavior.
These questions correspond to the fluency, coherency, on-topicness, and human-likeness scores reported in our paper, respectively.
Manipulated factors. We evaluate 3 chatbot agents. The first is CHAI. The second is a retrieval-based baseline  $(\mathrm{RL}_{\mathrm{utility}}(\mathrm{act})$  as proposed in (He et al., 2018)). The this is a language modeling baseline based on finetuning GPT2-medium on the CraigslistBargain dataset.
Dependent measures. We measure the performance of each chatbot agent according to 4 subjective metric (fluency, coherency, on-topicness, and human-likeness), which correspond to the survey questions given to participants described above. Each metric is rated on a 5-point Likert scale. We also measure the price that was agreed upon, and whether the negotiation resulted in an acceptance or a rejection.
Risks The risks presented to participants were minimal, and participants were never placed in the way of physical harm. There was a small probability the chatbot could generate offensive or otherwise inconsiderate language, as the language generation for CHAI and the language model baseline were unconstrained. However, we did not observe this behavior prior to the study, and did not observe this behavior during the study. We minimized the risk of confidentiality breaches by anonymizing all data stored as user IDs – participant names were not stored on our servers.
Subject allocation. We recruited 10 male and 6 female participants, with an average age of 24. Participation was voluntary, and subjects were not compensated for their participation. Participants were asked for consent before participating, and consent for including examples from their interactions for this paper. All examples contained in this paper are anonymized and contain no personally identifiable information.
Prior to participation in the study, each user was provided with instructions for how to use the interface, and a practice evaluation against the retrieval agent to familiarize them with the user interface and rating system. Each user interacted with all 3 dialogue agents (CHAI, retrieval, and language model) twice, with the order of interaction randomized per user. The advertisement displayed to the user is also uniformly sampled from the test set for each trial, independently from the agent being used.
Analysis. We ran a one-way repeated measures ANOVA test for each metric reported (fluency, coherency, on-topic, human-like) to examine the effect of chatbot agent on the metric.
Results showed that the type of agent used lead to statistically significant differences in the ratings. We found that:
- For fluency,  $f(2,30) = 5.8333$ ,  $p < 0.0072$
- For coherency,  $f(2,30) = 16.9518$ ,  $p < 0.0001$
- For on-topicness,  $f(2,30) = 10.1840, p < 0.0004$
- For human-likeness,  $f(2,30) = 20.1592, p < 0.0001$
A.4 Additional Qualitative Results
In this section, we include additional examples collected from human dialog for CHAI-prop, the retrieval method of (He et al., 2018), and the pure language modeling baseline.
A.4.1 CHAI-prop
Title: Ubiquiti Networks UAP-AC-LITE - $65
Description: Price is firm. Just a few month old. As New. With box and all parts. Great reliable AP. Set up once and forget about it; never need to reboot, restart like consumer AP; especially when you have high demand traffic with lots of concurrent users. Perfect for large house or business.
Buyer: I noticed that your price is firm. Would you be flexible in any way?
CHAI: That is good. I am asking $52 for this excellent product.
Buyer: how old is your product
CHAI: It has been in use for only about 6 months. It has never been restarted.
Buyer: why are you selling it
CHAI: I do not use this product.
Buyer: is it damaged
CHAI: It has a small scratch on the front. It is still new. I would offer $52.
Buyer:Could you do 40
CHAI: I would be willing to let it go for $40.
Buyer: How about $10
CHAI: It is new, in excellent condition. I would not take less than $40.
Buyer: But you just said it was scratched
CHAI: It is not scratched.
Buyer: Sorry I can't do more than 35
CHAI: Well, I can go down to $35.
Buyer:ok
CHAI: Great.
Title: Nokia Batteries - One new one used - $10
Description: I have these two batteries left over from my old phone. Maybe you can use them? Both are in excellent working order and ready to use. Message me if you are interested.
Buyer: I've been looking for batteries, what about 50 cents?
CHAI: Hi. I still have them. They're used, but I have used them and they work perfectly.
Buyer: How much life does the used one have?
CHAI: $9
Buyer: How many times have you used the old battery?
CHAI: 1 to 2 times
Buyer: Did you not use your phone much?
CHAI: Nope, I've been away from it for about 6 months.
Buyer: Ah, I see that makes sense.
CHAI: I can do $6
Buyer: I doubt many people would want to buy a used battery though, how many other offers do you have?
CHAI: I will let them go for $7.
Buyer: But you offered 6, 7 is higher?
CHAI: No. But I'm really looking for around $6 for them.
Buyer: I'll pick it up right now for 4 dollars
CHAI: No deal
Buyer: But you were not being fair earlier when you went from six to 7 dollars
CHAI: I can go to $5
Buyer: I'm holding firm on 4 dollars, especially because the batteries are used
CHAI: I can let them go for $5
Buyer: No one really buys used batteries though, what other offers are you getting?
CHAI: I'm just going to let them go for $4
Title: Couch and love seat - $80
Description: Couch and love seat from a non-smoking pets free house. Upholstery is clean and doesn't have any stain. Some zippers attaching the cushions to the frame are broken. Solid wood frame. pics coming
Buyer: I am interested in purchasing your couch and love seat
CHAI: Great! These were ordered from Crate & Barrel and are in like new condition. I'm willing to negotiate a little, but I'm asking $72 for them.
Buyer: How long have you owned them?
CHAI: I've had them for a few years. They are perfect for my new apartment and they were an absolute steal when I bought them.
Buyer: How much did you purchase them for?
CHAI: They are brand new, I'm asking $66.
Buyer: How much did you originally buy the love seat and couch from from Crate and Barrel?
CHAI: I originally paid $65 for both, but I figured I'd give it a shot since I love the couch.
Buyer: So you are asking for more money then you originally bought them for?
CHAI: Yes. I was hoping to get more for the couch and love seat. How much would you like for the couch?
Buyer: I do, but I can only offer $40
CHAI: I can only go down to $40 if you come pick up and I will throw in two chairs for you to use.
Buyer: Great! That's a deal.
CHAI: Great!
Title: Fit Bike VH2 - $200
Description: 20.75” top tube All in perfect working order. Very good condition - no rust - always stored indoors. Shiny. Few scratches/scrapes on pegs, etc, but very well cared for. Gyro brake. Thanks for interest. If its still here it is still available.
Buyer: Hello, I'm interested in the bike
CHAI: It is in great shape. It's a well-run bike. I bought it new, so I'm making it available.
Buyer: how long have you had it?
CHAI: I bought it new last year. It's in excellent condition.
Buyer: can I see pictures?
CHAI: Of course. How much are you looking to spend on the bike?
Buyer: I'm looking to spend under $150
CHAI: That's not a bad price. Would you be able to pay $180 for it?
Buyer: How about $160?
CHAI: $160 and you have a deal.
Buyer: Done
A.4.2 Retrieval
Title: Media Cabinet (Moving Sale) - $20
Description: Slide out tray for media storage Condition: Structurally sound Good condition Search Steve2017 for my other listings or simply click on the "more ads by this user" link found under the map. Please email me if you are interested. Serious inquiries only. Thank you
Buyer: Hello, how are you?
Retrieval: i am well, how are you?
Buyer: Good. I'm interested in buying the media cabinet.
Retrieval: how can i help you today?
Buyer: Can you provide more pictures?
Retrieval: yes
Buyer: How old is the item?
Retrieval: less than 1 year olf
Buyer: I would like to buy it for $15.
Retrieval: deal
Title: Adams trail-a-bike shifter 7 - $100
Description: This is an awesome bike trailer for your kid. It features 7 shiftable gears so that your kid can learn to shift but more importantly shift down or up to continue to help you pedal the bike. I know that without the shiftable gears my kid would either think the pedaling was too hard at times or going so fast they couldn't feel their pedaling helping. It took some time to teach them but definitely the way to go.
Buyer: Hello, I'm wondering about the availability of this bike trailer
Retrieval: hello
Buyer: Is this bike trailer still available?
Retrieval: i've had it for about 5 years now.
Buyer: great, is it still available?
Retrieval: yes it is
Buyer: When would I be able to pick it up?
Retrieval: yeah, that's fine.
Buyer: Is there a date by which I would be able to pick it up?
Retrieval: sure, i could do that for you.
Buyer: Okay, is there any damage to the bike trailer? Retrieval: yes what questions do you have?
Buyer: I'm wondering if there's any damage to the bike trailer
Retrieval: great! i'm asking $100.00 for it!
Buyer: Would you be able to decrease the price to $90?
Retrieval: sure. i'd be willing to let it go for $90.00
Buyer: Okay great, I'll pick it up today. Does that work for you?
Retrieval: yes absolutely. its a great deal
Title: File Cabinet metal - $70
Description: I have 4 cabinets file metal 4-Drawer for sell. They are good condition. they are black .I sell them for  $70\$ each$ . Thank you for reading!
Buyer: Why are you selling
Retrieval: yes it is still for sale.
Buyer: How much is it
Retrieval: it's new
Buyer: Sorry, can you repeat that
Retrieval: where are you. how far from alameda(ca)?
Buyer: I am close to [removed]. It's pretty far
Retrieval: how about $70.00?
Buyer: I don't know the condition. Could you please provide me with some details
Retrieval: it looks extremely heavy and in great condition! i could do $70.00...
Buyer: Is there any damage
Retrieval: nope. overall condition is extremely well
Buyer: How heavy are they
Retrieval: yes, very good condition.
Buyer: How about 25
Title: Samsung Gear S2 - $175
Description: Available in Black or white Flawless! Gear, Samsung, S3, S2, smartwatch, smart, watch, fitbit, pedometer, fitband, fit, band, bit, Samsung Gear Fit 2, classic, iwatch
Buyer: Is this an S3 or S2?
Retrieval: yes. are you interested? it's in great condition
Buyer: I'm interested.
Retrieval: do you have any questions?
Buyer: Is it a Gear S2 or Gear S3?
Retrieval: yes its a full tablet so it does come with the charger