SlowGuess/NAACL_2021_OCR huggingface dataset:
automation via conversational assistant. CoRR, abs/2001.02284.
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal-oriented dialogues (multidogo): Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4518-4528.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5370-5381. Association for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020a. Schema-guided dialogue state tracking task at dstc8. arXiv preprint arXiv:2002.01359.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020b. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In AAAI, pages 8689-8696. AAAI Press.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012, pages 5149-5152. IEEE.
Pararth Shah, Dilek Hakkani-Tür, Gökhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry P. Heck. 2018. Building a conversational agent overnight with dialogue self-play. CoRR, abs/1801.04871.
Alane Suhr, Claudia Yan, Jacob Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2119-2130. Association for Computational Linguistics.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Jason D. Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient
end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 665-677. Association for Computational Linguistics.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 808-819. Association for Computational Linguistics.
Chien-Sheng Wu and Caiming Xiong. 2020. Probing task-oriented dialogue representation from language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Virtual Online, November 16 - November 20, 2020, pages 5036-5051, Online. Association for Computational Linguistics.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2204-2213. Association for Computational Linguistics.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020, pages 270-278. Association for Computational Linguistics.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu 0001. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4623-4629. ijcai.org.
A Agent Training Details
Optimizing agents performance can be split into preparation before the HIT (Human Intelligence Task), improving HIT itself, and ongoing training afterwards. Starting with the pre-HIT phase, the major steps largely center around multiple rounds of qualifications to filter for the highest quality workers available. During the post-HIT phase, effort shifts to ensuring that each worker becomes increasingly comfortable with the task.
Pre-HIT Phase Qualifications take the form of online quizzes which serve the purpose of training motivated workers in addition to simply removing unqualified ones. When designing the qualification, the number and style of questions were iterated on to limit the feeling of a tight time constraint, while still remaining quite difficult. In fact, some agents who had previously had actual customer service jobs mentioned they felt like they were right back at the office. This difficulty resulted in a high rejection rate, which was costly because we paid Turkers $2 regardless of passing the exam (with a larger $8 bonus for passing). Although, the cost was well worth the trade-off since having high quality agents would pay dividends down the road.
To move efficiently, we leaned heavily on multiple choice questions and MTurk APIs to help automate grading and assignment of qualifications. Finally, we learned that including screenshots of the Agent Dashboard in the quizzes was a great way to familiarize the agents with the platform before performing the actual task.
During-HIT Phase The HIT itself was priced at  $1.50 for completing the conversation with an extra$ 1.00 bonus for identifying the correct customer intent at the end-of-chat survey. Since agents are naturally focused on getting done as quickly as possible, they would often only take the customer's requests into account, bypassing a key characteristic of what makes ABCD unique. However, by encouraging agents to focus on the customer intent, they were forced to peruse the Agent Guidelines for the associated subflow. Thus, we found this incentive critical for aligning agent behaviors with optimal outcomes.
Post-HIT Phase For ongoing training, we began producing small lists of bulletpoints to the agents on areas they could improve on. Fur
Figure 3: Crowdworker feedback in chat platform after the completion of the final batch of data collection.
thermore, we would highlight examples of good and bad decision-making and appropriate behavior when representing the fictitious "AcmeBrands" retail company. Finally, we also recorded videos which gave agents a view of how an "ideal" agent would behave at every step of the chat. We found that by engaging with the Turkers through the group chat and respecting their feedback, they were very willing to work on improving despite not having extra monetary incentive to do so.
In total, the agents were quite wonderful to work with and their end-of-task feedback strongly suggests they enjoyed the process as well. (See Figure 3) We credit this to the training details mentioned in this section and the development of the Expert Live Chat procedure.
B Optimizing Available Workers
In a regular Mechanical Turk (MTurk) setup, HITs are made available to a large audience who can pick up as many or as few as they want. Expert Live Chat dictates a dialogue between two speakers, so we need two types of workers: agents and customers. Let us consider the number of agents available as  $A$  and the number of customers available as  $C$ . Given budget constraints, we can only pay some maximum number of workers  $M$ . Simultaneously, given time constraints, we need a minimum number of conversations collected per week, which is a function of the number of available workers  $N = f(A, C)$ . This leads to three issues that must be considered in conjunction:
$\mathbf{N} < \mathbf{A} + \mathbf{C} < \mathbf{M}$  Operating the Agent Dashboard requires a highly skilled worker, so efficient data collection is limited by the number of available agents. Although the customer side of ABCD is a simpler task, there is still a minimum bar to be met to prevent (a) customers who spam with random text (b) customers who fake scenarios or (c) customers who hoard HITs and never show up to the chat. Thus, there needs to be a sufficient amount of both agent  $A$  and customers  $C$  qualified and available in order to surpass the minimum threshold set by  $N$ . However, simply paying more per HIT bumps up against the limits set by  $M$ .
$\mathbf{C} >> \mathbf{A}$  Since training agents is more resource intensive than training customers, it makes sense to simply have more of the latter. Yet by doing so leads to an issue where customers wait around for agents when they arrive in the waitroom. In a typical scenario, a customer might leave the tab open to work on other tasks, but when they are eventually paired, the customer is often busy doing something else, leaving the chat to flounder. In the worst case, the customer starts to verbally abuse the agent about the long wait time when they are finally paired.
A  $>>$  C Finding as many agents as possible is not the solution either because now the agents will end up waiting around for customers. If the waiting periods are too long, agents will abandon the task and disparage your reputation on various forms of social media. Since the task is difficult, the pool of workers who may eventually qualify as agents is finite, so too many poor interactions can halt the data collection process completely.
To resolve this situation, we begin with the maximum number of workers  $M$  as the starting constraint given a fixed budget. If we qualify too many workers, then we will not have enough budget left for the actual conversations, so instead we qualify workers in mini-batches. Since the pool of potential workers who may meet the strict requirements for agents  $A$  is more limited than customer candidates  $C$ , we start on the agent side. Given some amount of qualified agents, only a percentage of them will show up at the desired time slot to perform the task. Thus, we increment the batch size until the number of available workers passes the minimum  $A > N / 2$ .
To limit the number of customers who show up, we filter for users by location, number of completed HITS, and sufficient rating. We also establish an exam that is purposely very easy (to minimize costs), but just hard enough to deter bots and spammers. To raise the likelihood that the customer will show up, we include a question in the quiz which simply asks when the customer is available to perform the HIT. We really emphasize this question and make it required, so workers are aware of its significance. This allows us to tune the customer count such that  $C \approx A$ .
Note that due to the higher pay rate, agents are more likely to show up than customers. Therefore, there needs to be a higher ratio of customers to account for this imbalance. For some intuition on where to start, we found that a good rule of thumb was to consider the appearance ratio as inversely proportional to the ratio of pay. One final insight is to make the HITs heavily dependent on bonus pay, with base pay very low. This will keep spammers away since they will end up with a pittance when attempting to game the system.
To improve the waitroom experience, we added a feature where a user's place in the queue would be updated live, along with a timer indicating the expected wait. For Turkers willing to wait around, helpful and encouraging messages would also be displayed to keep them occupied. Alternatively, for Turkers who were multi-tasking, visual and audio notifications were added to signify the start of a chat, allowing them to attend to other tasks in the meantime. We believe our modifications have only scratched the surface and that improving the user experience for data collection offers an interesting line of HCI research to explore.
C Cascading Evaluation
To motivate cascading dialogue success (CDS) over typical other accuracy metrics, consider the scenario where a model gets  $80\%$  of turns correct, while still achieving  $0\%$  accuracy on the conversation level because it always messes up somewhere right at the end of the dialogue. A turn-based metric would over-estimate performance since such a metric fails to capture the model's consistent shortcomings in closing conversations. On the other hand, conversation-based metrics under-estimate the model's performance because such measures fail to account for the fact that the system is mostly successful. Moreover, each evaluation would be limited to occurring only once per conversation, which makes inefficient use of scarce data as a resource.
Instead, cascading dialogue success creates an evaluation example for the remainder of each conversation starting from each turn. For example, suppose a chat contained 4 turns: [A, B, C, and D], training instances can be created with this data that include: [A, B, C, D], [B, C, D], [C, D] and [D] by itself. Now imagine the model consistently predicted turn  $C$  incorrectly, and everything else correct. Then its scores would be  $2/4$ ,  $1/3$ , 0 and 1, respectively. Averaging across all turns would yield a final cascading success rate of  $45.8\%$ . A turn-based metric would yield  $75\%$  while a conversation-based metric would yield  $0\%$ . Thus, CDS allows a model to earn partial credit on what it has learned without severe penalties in either direction.
D Model Training Hyperparameters
When training the best model for Action State Tracking, we ended up with a learning rate of 3e-5, hidden dimension of 1024, weight decay of 0.05 and a batch size of 10 examples. Training lasted for 14 epochs, where we early stopped if overall accuracy failed to improve for three epochs in a row. The RAdam optimizer had a linear warm-up for three epochs, with hyperparameters kept at their defaults of 0.9 and 0.999. We also add the lexicalized slots into the vocabulary of the tokenizer.
For Cascading Dialogue Success, our best model had a 1e-5 learning rate, 1024 hidden dimension and no weight decay. The batch size was shrunk to 3 examples, but this was due purely to memory rather than performance reasons. Train
ing was set to 21 epochs, and again we early stopped if overall accuracy failed to improve for three epochs in a row. Finally, the optimizer again had a linear warm-up for three epochs with hyperparameters kept at their defaults.
E Intent Info and Guidelines
We augment the model with access to intent information in two ways. First, the subflow is translated into an index which is concatenated to all input contexts so the model can leverage this information. Second, the intent classifier is directly fed the solution, which is what allows it to trivially reach perfect accuracy.
We leverage the Agent Guidelines by using it to mask invalid action predictions. More specifically, given a predicted subflow, the guidelines outline all possible actions and values within that subflow. With this information, a mask is created before training and applied during evaluation to only allow valid actions.
F Conversation Examples
Since ABCD was collected using Expert Live Chat rather than templates, we observe various linguistic diversity in the chats. These phenomena limit the ability of models to memorize artificial patterns when making predictions.
Co-reference
CUS: I'd like to return something
AGT: OK
AGT: Can I get your full name
AGT: Also user name, email address, order id
AGT: Membership level and reason for return
CUS: Alessandro Phoenix, aphoenix872@email.com, order ID is 4024067912
CUS: I'm at the Gold level. I'm returning it because it's the wrong size
Chit-Chat
AGT: Do you need any more help?
CUS: a break, I need a coffee break.
CUS: but no, nothing from you
CUS: thanks for the save
AGT: Haha have a good break! And have an even better day.
AGT: Ok, there was a mistake made. Do you have the Shipping Status?
CUS: It's in transit
AGT: Ok, that means it's already out for shipment.
CUS: so two are being sent?
AGT: Yes. Unfortunately that means when you get the item you will need to call back and make a return
CUS: oh you gotta be kidding me!
Table 5: Examples of linguistic phenomenon.
G Agent Guidelines
(screenshot on following page)
H Customer Panel
Manage Account Flow
Status - Service Added
The customer believes that an unknown service was added to their account.
- [Interaction] Ask the customer for their Full name or Account ID with [Pull up Account]. This loads information in the background with information related to this user.
- [Interaction] Find out how the customer heard about this issue: Options are one of the three ["email", "spouse", "news"]
- Enter this into the input box and [Record Reason]
- [KB Query] To decide whether or not this is a valid error:
Tell the customer you will check your system about this problem.
- You do not have to enter anything into the field.
- Then [Ask the Oracle] if the customer made an error, which will return Yes or No.
- <Communication> If the customer is mistaken (Oracle says No), then explain:
If the customer was sent an email
- Let them know this was a scam email, and if they check the sender, it isn't from the official AcmeCorp marketing team.
If the customer was told this from their spouse
let them know they must have misheard. They have nothing to worry about and there are no extra charges.
If the customer saw this in the news
then let them know nicely that they must have misunderstood. They have nothing to worry about and there are no extra charges.
- [Interaction] If the customer is correct (Oracle says Yes), then remove the extra service
Enter in "remove service" and use the [Update Account] button.
Tell the customer that the extra service has been removed.
- [Interaction] If it is a legitimate error, then you can offer a refund to the customer
The amount should be for $40
Enter "40" in the field and then click on [Offer Refund]
If nothing works, just apologize and end the conversation.
Customer Service Site
Figure 4: A example subflow Status - Service Added under the Mange Account Flow in the Agent Guidelines
Leave Early
If asked about your preferences, pick anything you like. You want to get a refund because you just bought the item described below, but changed your mind. Explain your problem to the agent, provide any information that is requested and attempt to get your refund before the item ships. Your refund amount is the price of the product below.
Name: Albert Sanders
Phone Number: (099) 109-0570
Member Level: Silver
Email Address: as205981@email.com
Account ID: WCUSTYQERI
Username: as205981
Order Statement
Purchase Date: 2020-01-02
Order ID: 6449343909
Payment Method: Credit Card
Address: 9990 Kennedy St
La Fayette, CA 58913
In Original Packaging: Yes
Product: Jeans
Brand: Guess
Amount: $49
Figure 5: The customer chat interface (left) shows an on-going conversation with customer messages in grey, agent messages in blue, and actions in green. The customer prompt (right top) grounds the customer to a specific issue and backstory. The info sections (right middle and bottom) contains values that the customer has to provide in the conversation as well as other meta-data such as product information.