SlowGuess/NAACL_2021_OCR huggingface dataset:
ing slots. The reverse is also true - a given slot can be associated with multiple domains or buttons. Lastly, both contain values that can be enumerable (i.e. payment types or shipping statuses) or non-enumerable (phone numbers or email addresses). Following the pattern set by Rastogi et al. (2020b), enumerable values are given in the ontology to be accessible by a model, whereas the non-enumerable items are not.
Despite the similar structure, AST deviates from DST since predicting the right action requires not only parsing the customer utterance, but also adhering to Agent Guidelines. Suppose a customer is entitled to a discount which will be offered by issuing a [Promo Code]. The customer might request  $30\%$  off, but the guidelines stipulate only  $15\%$  is permitted, which would make "30" a reasonable, but ultimately flawed slot-value. To measure a model's ability to comprehend such nuanced situations, we adopt overall accuracy as the evaluation metric for AST.
6.2 Cascading Dialogue Success
Since the appropriate action often depends on the situation, we propose the Cascading Dialogue Success (CDS) task to measure a model's ability to understand actions in context. Whereas AST assumes an action occurs in the current turn, CDS gives an agent the additional options of responding with an utterance or ending the conversation. Moreover, proficiency is no longer measured as success over isolated turns but rather as success over sequences of consecutive turns.
Formally, given  $C_t = [x_1, x_2, \ldots, x_t]$  as a context composed of utterances  $x^c, x^a \in \mathcal{U}$  and actions  $x^b \in \mathcal{A}$ , a model should predict all remaining steps  $x_{>t}$  along with their realized forms. Pos-
sible next steps are to take an action, respond with text or end the task. When the next step is an action  $x_{t+1}^b$ , the model should predict the button with its slots and values as in AST. If the agent speaks in the next step  $x_{t+1}^a$ , the model should rank the true utterance highest, as measured by recall metrics. Finally, the model should recognize when to end the conversation.
Rewarding the model only when it predicts every step correctly is counter-productive because minor variations in sentence order do not alter overall customer satisfaction. Therefore, CDS is scored using a variation on Cascading Evaluation (Suhr et al., 2019). Rather than receiving a single score for each conversation, cascaded evaluation allows the model to receive "partial credit" whenever it successfully predicts each successive step in the chat. This score is calculated on every turn, and the model is evaluated based on the percent of remaining steps correctly predicted, averaged across all available turns. (See Appendix C for more details.)
6.3 Baseline Models
We also run several baselines on these new tasks. The backbone of all our baseline systems is a pre-trained Transformer-based model acting as a context encoder. More specifically, given the dialogue history as a series of utterances, we first join the utterances together with a [SEP] token and then tokenize the entire input using WordPiece (Schuster and Nakajima, 2012). Next, we feed the entire input into a BERT model and perform a learned pooling on the hidden states in the final layer, which results in a fixed-length latent vector  $h_{enc} \in \mathbb{R}^{128}$  (Wolf et al., 2019). Afterwards, we attach a variety of prediction heads conditioned on the  $h_{enc}$  vector to generate the final output. Details of the prediction heads for the two proposed tasks are described next.
We break down Action State Tracking (AST) into two sub-problems, button-slot prediction and value-filling. Given the ontology, button prediction is a straightforward classification task over 231 known options, so the prediction head is just a linear classifier with a softmax activation for normalization:  $P_{b\cdot slot} = \mathrm{Softmax}(W_a h_{enc}^\top + b_a)$ .
To handle value-filling, we further decompose
the task into predicting enumerable and non-enumerable values. The ontology lists out all  $|E|$  enumerable values, so the prediction head  $p^{enum}$  simply maps the hidden state  $h_{enc}$  into the appropriate dimensions. To handle non-enumerable values, we follow the insight from (Ma et al., 2019) which notes that practically all such values are stated by the customer in conversation, so a model can copy these values from the tokenized context. During pre-processing, we extract up to  $|N|$  unique tokens from the natural language customer utterances, where  $p^{copy}$  then represents the distribution over these possible options.
We imitate the TRADE architecture from (Wu et al., 2019), where conditioned on the action, the model chooses to either copy from the context  $p_{copy}$  or select from the enumerable entities  $p_{enum}$  based on a gating mechanism. The gate is conditioned on the hidden state  $h_{enc}$  as well as a learned context vector  $c_i$ . Concretely,
$$
p ^ {e n u m} = \operatorname {S o f t m a x} \left(W _ {e} h _ {e n c} ^ {\top} + b _ {e}\right) \in \mathbb {R} ^ {| E |}
$$
$$
p ^ {c o p y} = \operatorname {S o f t m a x} \left(W _ {c} h _ {e n c} ^ {\top} + b _ {c}\right) \in \mathbb {R} ^ {| N |}
$$
$$
c _ {i} = W _ {c} ^ {\top} \cdot p ^ {c o p y} \in \mathbb {R} ^ {h i d}
$$
$$
p ^ {g a t e} = \sigma \left(W _ {g} \cdot \left[ h _ {e n c}; c _ {i} \right]\right) \in \mathbb {R} ^ {1}
$$
$$
P _ {v a l} = [ p ^ {g a t e} \times p ^ {c o p y}; (1 - p ^ {g a t e}) \times p ^ {e n u m} ] \in \mathbb {R} ^ {| E + N |}
$$
where  $\sigma$  represents the Sigmoid function and  $[\cdot ;\cdot ]$  is the concatenation operation. The final value predictions are the argmax of  $P_{val}$  which merge the probabilities of  $p^{enum}$  and  $p^{\text{copy}}$  together.
For Cascading Dialogue Success (CDS), we also tackle next step selection, utterance ranking, and intent classification. Next step selection is a choice between retrieve utterance, take action and end conversation. Intent classification consists of choosing from the 55 available subflows. Given this basic setting, both tasks use the same setup of a linear layer followed by a softmax, albeit with their own respective weights  $W_{NS} \in \mathbb{R}^{3 \times hid}$  and  $W_{IC} \in \mathbb{R}^{55 \times hid}$ . When the next step is to take action, the AST model is reused to determine the button-slot and value. When end conversation is selected, all future predictions are ignored, much like an <EOS> symbol signifies stopping.
This leaves us with utterance ranking, which is only evaluated when retrieve utterance is chosen as the next step. Our ranker reproduces the design
$^2$  Choosing larger  $|N|$  leads to higher recall, but lower precision. We found  $N = 100$  to work well in practice.
Sentences with similar semantics may be formulated in several ways, so we opt for response retrieval over text generation since common metrics (i.e. BLEU score) tend to become unreliable in these situations (Liu et al., 2016).
from (Guu et al., 2020), where the encoded context  $h_{ctx}$  is compared against each encoded candidate response  $h_{cand}$  to produce a ranking score. To embed each  $j^{th}$  candidate  $d_j$  we first create its input  $d_j^{input}$ . Following standard practice, we prepend the candidate text  $d_j$  with [CLS], separate the individual utterances  $u_i$  within the candidate response using a [SEP] token, and append a final [SEP] token afterwards. (Devlin et al., 2019). This input  $d_j^{input}$  is then fed into a static pretrained BERT model to get an initial hidden state, which is finally projected using a learned weight  $W_{d_j} \in \mathbb{R}^{128 \times hid}$  to produce  $h_{cand}$ . To obtain  $h_{ctx}$  we start with the hidden state  $h_{enc}$  from before and apply a projection matrix  $W_{UR} \in \mathbb{R}^{128 \times hid}$  to reach the desired dimensionality.
$$
d _ {j} ^ {\text {i n p u t}} = [ \mathrm {C L S} ] u _ {1} [ \mathrm {S E P} ] u _ {2} [ \mathrm {S E P} ] \dots [ \mathrm {S E P} ] u _ {n} [ \mathrm {S E P} ]
$$
$$
h _ {c a n d} = W _ {d _ {j}} \mathrm {B E R T} _ {\mathrm {b a s e}} \left(d _ {j} ^ {\text {i n p u t}}\right) ^ {\top} \in \mathbb {R} ^ {1 2 8}
$$
$$
h _ {c t x} = W _ {U R} h _ {e n c} ^ {\top} \in \mathbb {R} ^ {1 2 8}
$$
$$
f (x _ {i}, d _ {j}) = h _ {c t x} ^ {\top} h _ {\mathrm {c a n d}}
$$
$$
P _ {j} ^ {\text {r a n k}} = \frac {\exp (f (x _ {i} , d _ {j}))}{\Sigma_ {d _ {j} ^ {\prime}} \exp f (x _ {i} , d _ {j} ^ {\prime})}
$$
The final rank is given by normalizing each  $j^{th}$  score against all other candidate scores. We use the training objective from (Henderson et al., 2019) to calculate the loss:
$$
\mathbb {J} = \sum_ {j = 1} ^ {M = 1 0 0} P (x _ {i}, d _ {j}) - \sum_ {i = 1} ^ {M} \log \sum_ {j = 1} ^ {M} \exp^ {f (x _ {i}, d _ {j})}
$$
where  $M$  is the size of the total candidate set.
6.4 Experiments
We performed experiments on the two newly proposed tasks, AST and CDS. AST consists of two subtasks, button-slot prediction and value-filling, while CDS builds on this with three additional subtasks of next step selection, utterance ranking, and intent classification. For both tasks, we experimented with two types of frameworks, a pipeline version and an end-to-end version. The pipeline version trains each subtask separately while the end-to-end optimizes all tasks jointly (Liang et al., 2020; Rastogi et al., 2020a; Ham et al., 2020).
The pipeline model uses a BERT model trained with the RAdam optimizer (Liu et al., 2020). To test the performance of different pretrained models under the end-to-end framework, we
Metric	Pipeline	BERT	Albert	RoBERTa
B-Slot	86.7%	89.9%	90.9%	93.6%
Value	42.1%	61.6%	61.0%	67.2%
Action	32.3%	59.5%	59.2%	65.8%
Table 3: Metrics for Action-State Tracking. Pipeline values come from models trained on individual subtasks, other models are trained jointly end-to-end.
experiment with three additional encoders, AlBERT (Lan et al., 2020), RoBERTa (Liu et al., 2019) and RoBERTa-Large. AlBERT model has an inter-sentence coherence task and a lighter memory footprint compared to BERT, while RoBERTa model has substantially more data and hyper-parameter tuning in pretraining than BERT. In the future, we also plan to include GPT-based models, such as DialogGPT (Zhang et al., 2020) in our comparison.
6.5 Results
For both tasks, moving from the pipeline architecture to a jointly trained method displayed noticeable improvement in accuracy. As hinted at in prior works (Liang et al., 2020), we suspect the group effort gives each subtask extra supervision from other subtasks for more data efficient training. In the AST task, we found steady improvements as we move from the older to the newer models with vanilla BERT at  $59.5\%$  accuracy and RoBERTa doing the best at  $65.8\%$ . For the CDS task, we found a similar trend where RoBERTa-Large outperforms BERT, but only by a mere  $0.6\%$ . We hypothesize this small gap between models is due to the fact that none were particularly trained on dialogue data which impacts their ability to produce a useful encoding (Wu and Xiong, 2020).
Separately, we evaluate CDS subtask difficulty by asking human volunteers to select the correct label from a list of possible options. As an example, workers would be presented with 55 different classes for Intent Classification and asked to choose the right one. Since humans typically struggle when choosing from large collections of items, fine-tuned models performed roughly on par or better compared to humans in this unnatural setting. On the other hand, human evaluation for the overall CDS task was judged by measuring the success rate in a standard conversational scenarios where behavioral instincts are activated, so humans were able to excel on this environment.
Model	Intent	Nextstep	B-Slot	Value	Recall@1/5/10	Cascading Eval
Human	85.5%	84.0%	79.0%	77.5%	N/A	82.7%
Pipeline	90.4%	83.8%	86.7%	42.1%	26.2/51.7/63.1	18.2%
BERT-base	89.3%	87.6%	85.9%	73.1%	21.7/46.6/58.7	31.3%
AlBERT	88.5%	87.2%	86.1%	70.4%	22.1/47.4/58.9	31.2%
RoBERTa	89.7%	87.8%	87.6%	73.1%	21.6/46.7/58.6	31.5%
RoBERTa-Large	90.5%	87.5%	88.5%	73.3%	22.0/47.8/59.1	31.9%
BERT-base w/o Action Info	88.4%	76.8%	83.7%	63.4%	18.6/43.0/57.9	29.2%
BERT-base w/ Guidelines	83.2%	87.5%	85.6%	72.4%	21.8/46.9/58.5	30.6%
BERT-base w/ Intent Info	100%	88.6%	88.9%	73.8%	22.2/47.6/59.1	32.3%
BERT-base w/ Intent + Guide	100%	89.2%	89.3%	74.0%	22.6/48.1/59.4	32.7%
Table 4: Cascading dialogue success task performance with breakdown of all five subtasks. Numbers displayed are the average of three seeds. Human evaluation conducted with size of 100 samples per person.
We perform an ablation study to test the significance of the key features in ABCD. Recall, actions are characterized by their dual nature of requiring signals from both the customer and the company guidelines. To that end, we provided the ground truth intent to measure the impact of the customer side. Conversely, we also test the company side by masking out invalid buttons based on the insight that the Agent Guidelines are useful for narrowing down the range of possible actions. In both situations, we would expect that providing such oracle guidance would boost performance. Lastly, note that the appropriate action depends on the outcomes of prior actions, so for a final experiment we removed prior actions and their explanations from the context to test their impact on task success. (See Appendix E for details.)
We observe that supplying the intent information to the BERT model causes a noticeable boost in dialog success, bringing the score to  $32.3\%$ . However, augmenting the model with knowledge of the guidelines unexpectedly dropped performance down to  $30.6\%$ . Further analysis revealed the imperfect intent classifier would occasionally mask out valid buttons, leaving only incorrect ones to choose from. As a result, the downstream action predictor would be prevented from doing its job, causing errors to accumulate. To test this hypothesis, we ran another model (Intent+Guide) which had access to guidelines along with an oracle intent classifier. This model reached the peak observed performance of  $32.7\%$ , highlighting the importance of both components. As a final result, removing action information away from action-based conversations unsurprisingly causes a major performance drop (Table 4).
In conclusion, we have presented ABCD which includes over 10K dialogues that incorporate procedural, dual-constrained actions. Additionally, we established a scalable method for collecting live human conversations with unequal partners. We found that pre-trained models perform decent on Action State Tracking, but there is a large gap between humans agents and the top systems for Cascading Dialogue Success.
We plan to incorporate GPT-related models (Hosseini-Asl et al., 2020), as alternate forms of preprocessing have shown promise in other NLP tasks. Other techniques could also be used to incorporate speaker info, action semantics and other meta-data. Wholly new systems that attend to the Agent Guidelines in a fully differentiable manner are also worth exploring. By grounding dialogues to in-depth scenarios with explicit policies, we hope to have pushed towards a better understanding of dialogue success.
The authors would like to thank Tao Lei, Felix Wu and Anmol Kabra for their feedback and support. We would also like to thank the anonymous NAACL 2021 reviewers for pointing out specific areas of confusion in our submission, which we have tried our best to clarify.
This paper presents a new dataset which was collected through the use of crowdworkers. All agent workers were compensated a fair wage based on their local standard of living, where their location was determined during the vetting process. (Please refer to Appendix A for more details.)
Amazon. 2019. Alexa Skills Kit.
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In ICLR. OpenReview.net.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 5016-5026. Association for Computational Linguistics.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4515-4524. Association for Computational Linguistics.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4619-4629. Association for Computational Linguistics.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326-335.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017b. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2951-2960.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster 0001, Angela Fan, Michael Auli, and Jason Weston. 2019.
Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR. OpenReview.net.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In SIGDIAL Conference, pages 37-49. Association for Computational Linguistics.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Google. 2019. Actions on Google Assistant.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. CoRR, abs/2002.08909.
Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue system using gpt-2. In Proc. of the 34th AAAI Conference on Artificial Intelligence. ACL.
He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1766-1776. Association for Computational Linguistics.
He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2333-2343. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Pawel Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrksic, and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5392-5404. Association for Computational Linguistics.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796.
John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS), 2(1):26-41.
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. Codraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495-6513.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR. OpenReview.net.
Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2443-2453. Association for Computational Linguistics.
Weixin Liang, Youzhi Tian, Chengcai Chen, and Zhou Yu. 2020. Moss: End-to-end dialog system framework with modular supervision. In AAAI, pages 8327-8335. AAAI Press.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2122-2132. The Association for Computational Linguistics.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. CoRR, abs/1907.11692.
Yue Ma, Zengfeng Zeng, Dawei Zhu, Xuan Li, Yiying Yang, Xiaoyuan Yao, Kaijie Zhou, and Jianping Shen. 2019. An end-to-end dialogue state tracking system with machine reading comprehension and wide & deep classification. CoRR, abs/1912.09297.
Alena Moiseeva, Dietrich Trautmann, and Hinrich Schütze. 2020. Multipurpose intelligent process