SlowGuess/NAACL_2022_OCR huggingface dataset:
The Topv2 Dataset. The Topv2 dataset (Chen et al., 2020) is a collection of queries produced by crowdsourced workers and intended for smart voice assistants. Topv2 has compositional queries with hierarchical meaning representations and extends the original TOP dataset (Gupta et al., 2018) with six additional domains. We present statistics for each domain in Table 1. The eight domains vary widely, including the number of samples (ranging from 13k to 31k), the number of slots (from 5 to 33), and the portion of flat utterances. Accordingly, when we convert Topv2 instances to multi-turn QA instances, the average number of questions per instance varies a lot across the domains. We test our QA approach on Topv2 under the full-data setting for all domains, and select four domains to study in a few-shot setting, with only 10 samples per intent and slot (10SPIS).
The Pizza Dataset. Pizza is a new TOP dataset (Arkoudas et al., 2021) consisting of complex utterances that order pizzas and drinks. Pizza consists of 2.4M training examples that are synthetically generated from a CFG, along with 348 dev examples and 1357 test examples generated and annotated by MTurk workers. Although the training set is large, we focus on few-shot settings with 30, 50, and 100 examples randomly drawn from the dev set. The low-resource setting is indeed challenging—the dev set has only 107 unique slot values, whereas the test set has 180, requiring models to generalize well.
We use a standard metric for evaluating TOP performance: unordered exact match accuracy (abbreviated as EM), which does a node-to-node comparison between the generated parse tree and the golden parse tree, modulo sibling order. EM doesn't take partially correct parses into account, so given a reference and a hypothesis, EM is either 0 or 1.
We use the T5-large model (Raffel et al., 2020) as the backbone of our QA framework. We choose a learning rate between 5e-6 and 5e-4, a batch size from \{32, 64, 96, 128\} for the full-data setting, and a batch size from \{8, 16, 24\} for the few-shot setting. We search for the best set of hyperparameters with 16 random trials for each configuration. With full data, we train single-turn models for 10 epochs and multi-turn models for 30 epochs to select the best-performing checkpoint on validation. For few-shot learning, we train for 3000 steps and make a checkpoint every 100 steps. We used 8 Tesla V100 GPUs with 32 GB memory for all our training.
We consider two baselines. The first applies a seq2seq model trained on logical forms (LFs) expressed in the "TOP-decoupled format," which removes from the parse tree all text that does not appear in a leaf slot (Aghajanyan et al., 2020). For this baseline, we use the BART-Large model described by Lewis et al. (2020), because this model is commonly used for TOP (Rongali et al., 2020; Aghajanyan et al., 2020; Shrivastava et al., 2021), and we also use T5-large, because that is the backbone of our QA implementation. This allows us to eliminate any benefit that may be derived from the architecture itself. The second baseline method we consider generates canonical paraphrases of the original utterances (Shin et al., 2021), and we use T5-large to allow for a direct comparison.
Note that for Topv2 we only compare our method against the first baseline, since that dataset doesn't have a canonical representation. Pizza does, so for that dataset we compare our method against both baselines (LFs in Top-decoupled notation and canonical paraphrases). Since we're not aware of any existing results on the baseline methods for individual Topv2 domains and for Pizza, we implemented both with HuggingFace (Wolf et al., 2020) and performed hyperparameter tuning with
domain	#inst	#int	#slt	flat %	depth	#q/inst
alarm	30488	8	9	84%	2.16	3.28
event	13160	11	17	80%	2.37	6.55
message	14602	12	27	84%	2.23	3.97
music	17320	15	9	100%	1.98	2.81
navigation	30044	17	33	57%	2.68	6.18
reminder	26133	19	32	79%	2.45	7.67
timer	17392	11	5	96%	2.00	2.39
weather	31403	7	11	100%	1.93	3.16
Table 1: Domain statistics for Topv2. #q/inst is the average number of questions per utterance in multi-turn QA. Navigation and reminder are two most nested domains. Music and weather have completely flat utterances.
the same computation budget given to our method.
We first present Topv2 results for both full-data and few-shot settings. The baseline methods, labeled as BART/T5 LF, are trained on linearized Top-decoupled trees. We include four variants of the QA approach: ST QA and MT QA, as well as MSP ST QA and MSP MT QA (the same as ST and MT QA, except that the models are trained on the the masked span prediction objective). We report the full-data EM scores in Table 2 and the 10SPIS EM scores in Table 3. The relative gain is computed between the best QA approach and the best baseline approach in each domain.
In the full-data setting, all four QA variants outperform the baselines, with MSP ST QA having a slight overall edge, exceeding the best baseline method by 3.07 absolute points. We have the largest relative gain  $(6.19\%)$  in reminder. In general, we see smaller improvements in the flatter domains: music, weather, timer, alarm are the four domains with the smallest semantic-tree depths, and relative gains for these domains are below  $3\%$ .
We see that different QA approaches are close to each other when we have enough data; ST QA performs only marginally better than MT QA. However, ST QA has shorter latency and may therefore be practically preferable.
We choose four representative domains to perform low-resource experiments. For 10SPIS, MSP MT QA is a clear winner over the other approaches, improving the best baseline method by 17.16 absolute points. ST QA has the worst performance. Our explanation is that ST QA requires the generation of long texts (longer than the Top-Decoupled LFs), and it is too challenging to learn a complex new task with only 10 instances per intent and slot.
Therefore, in a few-shot setting, the benefit of having short answers is much clearer. Additionally, it is worth noting that having an objective that is well-aligned with the pretraining stage provides a significant benefit. It improves MT QA by nearly 5 absolute points on average, and it is a game changer for ST QA, in that with this one modification ST QA achieves competitive performance in three out of four domains.
We also test whether the inclusion of previous answers and metadata into the context helps in few-shot scenarios (the difference may be negligible with full data). We remove answers to previous questions ("w/o state") and metadata ("w/o metadata") from MT QA and report the scores obtained from both changes. The results suggest that both state and metadata make strong contributions to performance. Excluding prior answers has a more negative impact on average.
We note that our QA approach is very competitive against state-of-the-art results in the literature. For instance, the best reported result for 10SPIS reminder is 61.47 (Chen et al., 2020)<sup>2</sup>, which was achieved by first pretraining the model with six source domains. We improved that result by 9 absolute points while training only with the target domain's data.
We next report results on Pizza, where we compare our approach against both LF-trained baselines (with LFs expressed in TOP-decoupled notation) and canonical paraphrases. Table 4 summarizes results for three low-resource settings. We obtain the greatest improvement with 30 training examples, for a relative gain exceeding  $8\%$ . Consistently with the Topv2 results, as more training examples become available the gap between the QA approaches and the baselines shrinks. Even though T5 Canonical and ST QA are similar, the results show that including context information about an order is beneficial, especially when there are fewer training examples. MT QA has a slight edge over ST QA here, as asking more questions mitigates the burden of generating longer and more complex sequences.
4.6 Additional Analysis: Depth vs. Accuracy
We investigate how our approach performs on fulldata Topv2 as a function of semantic depth. We chose to perform the analysis on navigation and
2We only mention reminder because Chen et al. (2020) don't report 10SPIS results for the other domains.
Alarm	Event	Message	Music	Navigation	Reminder	Timer	Weather	Average
BART LF	88.68	85.66	92.73	80.35	83.08	82.66	77.91	89.81	85.11
T5 LF	88.58	85.12	88.85	82.60	82.77	78.39	83.46	89.95	84.97
ST QA	90.38	88.58	95.44	82.83	85.35	84.98	83.61	91.94	87.89
MT QA	90.05	88.43	96.81	83.60	82.07	87.72	81.84	92.22	87.84
MSP ST QA	90.27	88.88	95.07	82.95	86.25	86.35	84.05	91.62	88.18
MSP MT QA	90.17	88.43	96.81	84.97	82.78	87.78	81.91	92.34	88.15
Relative Gain	1.79%	3.76%	4.40%	2.87%	3.82%	6.19%	0.71%	2.82%	3.29%
Table 2: Topv2 results with full data. MSP refers to the model trained on masked span prediction. The relative gain for each domain is computed between the best QA variant and the best baseline method. The average is computed across eight domains. The best numbers are in bold.
Alarm	Navigation	Reminder	Weather	Average
BART LF	45.82	44.89	51.46	62.75	51.23
T5 LF	59.34	37.04	46.81	57.47	50.17
ST QA	52.54	28.39	48.58	52.25	45.44
MT QA	60.57	55.56	69.72	81.89	66.94
w/o state	50.22	48.18	69.65	78.95	61.75
w/o metadata	52.19	52.87	68.81	79.53	63.35
MSP ST QA	65.91	30.46	58.74	74.35	57.37
MSP MT QA	62.13	60.16	70.48	80.78	68.39
Relative Gain	11.1%	34.0%	37.0%	30.5%	28.1%
Table 3: Topv2 results in the 10SPIS setting. For MT QA, we include results obtained by removing previous answers from the context ("w/o state") and by removing domain metadata.
30	50	100
BART LF	65.67	72.00	83.51
T5 LF	74.06	77.82	86.07
T5 Canonical	71.17	77.67	87.47
ST QA	77.30	81.58	89.75
MT QA	66.96	79.35	85.98
MSP ST QA	77.45	83.14	88.34
MSP MT QA	80.01	83.99	91.74
Relative gain	8.03%	7.93%	4.96%
Table 4: Pizza results in three few-shot settings (training with 30, 50, and 100 examples).
D	L	#inst	LF EM	QA EM	Rel. Gain
Navigation	1	5.35	790	84.56	82.27	-2.71%
2	8.60	2689	86.28	91.56	6.12%
3	9.58	569	88.40	90.15	1.98%
4	12.09	1905	77.80	81.67	4.97%
6	12.46	115	46.96	50.43	7.39%
Timer	1	5.48	190	76.84	75.26	-2.06%
2	5.70	3746	86.28	85.69	-0.68%
4	7.82	310	53.87	71.29	32.34%
Table 5: Topv2 full-data result analysis per depth level for navigation and timer. The average utterance length (L) and the number of test instances (#inst) are shown for each depth (D). LF EM is from the best baseline method and QA EM is from the best QA method.
timer because navigation has the greatest average tree depth, and we want to explain why our method gave the lowest improvements on timer.
We present the breakdown in Table 5, where
LF EM shows EM scores from the best baseline method generating TOP-decoupled trees, and QA EM shows EM scores from the best QA variant. For utterances with shallow semantics, generating TOP-decoupled trees outperforms QA; but as we move towards deeper semantic structures, the benefit of using a more naturalized representation becomes evident. Indeed, for both navigation and timer, QA performs best at the deepest level. Thus, for practical purposes, when the goal is to build a system that achieves high accuracy across all utterance groups, it is worth considering a combination of conventional LF generation for shallow utterances and a QA model for more complex utterances. We define the length of an utterance to be the number of words the utterance has. Table 5 lists the average utterance length for each depth, and shows that input size is a good proxy for semantic complexity (and could thus be used to quickly decide between the two approaches).
We have presented a reduction of compositional task-oriented parsing to abstractive QA, whereby parse tree nodes are recovered by posing queries to a QA model. We have also proposed to train QA models with the MSP (masked span prediction) objective, to better leverage the massive amount of linguistic knowledge gained during pretraining. We experimentally evaluated single-turn QA and multi-turn QA on two public datasets, both in full-data and in few-shot settings, with and without MSP, and showed that they consistently outperform a number of powerful baseline techniques, including canonical paraphrasing, in both settings. The MSP variants perform best on average, with particularly dramatic improvements obtained in the few-shot setting.
Armen Aghajanyan, Jean Maillard, Akshit Shrivastava, Keith Diedrick, Michael Haeger, Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, et al. 2020. Conversational semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5026-5035.
Konstantine Arkoudas, Nicolas Guenon des Mesnards, Melanie Rubino, Sandesh Swamy, Saarthak Khanna, and Weiqi Sun. 2021. Pizza: a task-oriented semantic parsing dataset.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Rakesh Chada and Pradeep Natarajan. 2021. Fewshotqa: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6081-6090.
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5090-5100.
Shrey Desai, Akshit Shrivastava, Alexander Zotov, and Ahmed Aly. 2021. Low-resource task-oriented semantic parsing via intrinsic modeling. arXiv preprint arXiv:2104.07224.
Xinya Du, Luheng He, Qi Li, Dian Yu, Panupong Pasupat, and Yuan Zhang. 2021. Qa-driven zero-shot slot filling with weak supervision pretraining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 654-664.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 264-273.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2018. Long text generation via adversarial training with leaked information. In Proceed
ings of the AAAI Conference on Artificial Intelligence, volume 32.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified mrc framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859.
Bing Liu and Ian Lane. 2016. Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling. In Proc. Interspeech 2016, pages 685-689.
Elman Mansimov and Yi Zhang. 2021. Semantic parsing in task-oriented dialog with recursive insertion-based encoder. arXiv preprint arXiv:2109.04500.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Mahdi Namazifar, Alexandros Papangelis, Gokhan Tur, and Dilek Hakkani-Tur. 2021. Language model is all you need: Natural language understanding as question answering. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7803-7807. IEEE.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Subendhu Rongali, Konstantine Arkoudas, Rubino Melanie, and Hamza Wael. 2022. Training naturalized semantic parsers with very little data. In Proceedings of IJCAI 2022.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don't parse, generate! a sequence to sequence architecture for task-oriented semantic parsing. In Proceedings of The Web Conference 2020, pages 2962-2968.
Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana.
Akshat Shrivastava, Pierce Chuang, Arun Babu, Shrey Desai, Abhinav Arora, Alexander Zotov, and Ahmed Aly. 2021. Span pointer networks for non-autoregressive task-oriented semantic parsing. arXiv preprint arXiv:2104.07275.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Jiawei Zhou, Tahira Naseem, Ramón Fernandez As-tudillo, and Radu Florian. 2021. Amr parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585-5598.