SlowGuess/NAACL_2022_OCR huggingface dataset:
Wenting Zhao $^{1}$  * and Konstantine Arkoudas $^{2}$  and Weiqi Sun $^{2}$  and Claire Cardie $^{1}$ $^{1}$ Cornell University  $^{2}$ Amazon Alexa AI
{wzhao,cardie}@cs.cornell.edu,{arkoudk,weiqisun}  $@$  amazon.com
Task-oriented parsing (TOP) aims to convert natural language into machine-readable representations of specific tasks, such as setting an alarm. A popular approach to TOP is to apply seq2seq models to generate linearized parse trees. A more recent line of work argues that pretrained seq2seq models are better at generating outputs that are themselves natural language, so they replace linearized parse trees with canonical natural-language paraphrases that can then be easily translated into parse trees, resulting in so-called naturalized parsers. In this work we continue to explore naturalized semantic parsing by presenting a general reduction of TOP to abstractive question answering that overcomes some limitations of canonical paraphrasing. Experimental results show that our QA-based technique outperforms state-of-the-art methods in full-data settings while achieving dramatic improvements in few-shot settings.
Task-oriented parsing (TOP) takes an utterance as input and generates an unambiguous specification of a task that can be executed by machine (Gupta et al., 2018). Traditional approaches to TOP treat the task as an instance of slot filling (Liu and Lane, 2016), first classifying the intent of the utterance as a whole and then tagging tokens with slot labels that identify relevant entities (such as numbers, persons, dates or times, organizations, and so on). However, this approach only works for simple utterances that have flat rather than compositional semantics. That is, slot-filling approaches cannot produce nested (or "hierarchical") meaning representations, such as the one shown in Fig 1, where slots and intents can be composite and contain other slots (or intents) as proper parts.
A straightforward way to handle compositional semantics is to formulate TOP as a seq2seq prob-
lem, where the input sequence is the utterance and the output sequence is a linearized representation of a semantic tree (shown in the bottom part of Fig 1). The recent development of high-performing transformer-based pretrained language models (PLMs) (Lewis et al., 2020; Raffel et al., 2020; Brown et al., 2020) that can be fine-tuned on specific tasks (such as a particular TOP dataset) has made this formulation feasible.
This approach has achieved state-of-the-art performance on a variety of TOP datasets (Rongali et al., 2020; Aghajanyan et al., 2020; Chen et al., 2020). However, the output sequences are typically not expressed in natural language but rather in a mixture of natural language and special symbols (such as [SL:]) that were not seen during pretraining and have no meaning to the PLMs. Depending on the task, the output sequences may hardly contain any natural language at all (for example, if the outputs are SQL queries). It would seem reasonable to conjecture that performance would improve if we could reformulate TOP as a more conventional NLP task in which both the inputs and the outputs are expressed in natural language, as such a reformulation might be better able to leverage the PLM's pretraining.
To bridge this gap, Shin et al. (2021) reduced TOP to a canonical-paraphrasing task. They first fine-tune a PLM to map a natural utterance  $u$  to another canonical utterance  $u'$ , where canonical utterances belong to a controlled fragment of the relevant natural language. Then  $u'$  can be translated into the desired meaning representation (semantic tree) via a context-free grammar that can parse all and only canonical utterances. However, this approach has a major limitation: canonical fragments can be defined and parsed by hand-written grammars only in closed-world domains where the set of underlying entities is fixed and known in advance (e.g., in a domain where people ask questions about basketball players, all of whom are
*Work done during an internship at Amazon Alexa AI.
Human Language Technologies, pages 4418 - 4427
Utterance: Look up directions to the nearest parking near S Beritania Street.
Semantic parse tree of the utterance:
Linarized parse tree:
Figure 1: A sample utterance and its semantic parse tree in the Topv2 dataset, where nodes starting with "IN:" are intents and nodes starting with "SL:" are slots. A parse tree always has an intent node at the root. An intent typically corresponds to a verb and can be viewed as an action, with a sequence of slots as its arguments. A slot may have additional intents nested in it, recursively. A linearized parse tree that can be processed by seq2seq models is shown at the bottom.
statically known). That condition rarely holds in task-oriented domains, and therefore this method is not applicable to datasets like Topv2. Moreover, as input utterances grow more structurally complex, the associated canonical utterances become much longer, and generating long sequences is known to be challenging for PLMs (Guo et al., 2018).
In this paper we focus on scenarios where it is not viable to specify a canonical grammar with a fixed set of rules. We instead propose to formulate TOP as abstractive question answering, in such a way that answering all questions for a given utterance allows us to reconstruct the parse tree of that utterance. Specifically, we introduce single-turn QA (ST QA for short), which poses one single query for a given utterance, and multi-turn QA (MT QA), which dynamically constructs multiple questions for each utterance, depending on previous answers. Because single-turn QA asks only one question, it has lower latency; however, the model must generate a longer text representing the entire parse tree. By contrast, multi-turn QA generates shorter answers that are more straightforward to parse; however, all questions need to be answered correctly, and if there are dependencies between
questions, they can't be run within the same batch. We study these two approaches and their tradeoffs in both full-data and low-resource settings.
1. We propose a general reduction of compositional TOP to abstractive QA and introduce two specific variants: single-turn QA and multi-turn QA, each with unique benefits.
2. We train the abstractive QA models with a masked span prediction (MSP) objective, one of the pretraining objectives of the seq2seq model, which is shown to yield very substantial improvements in few-shot scenarios.
3. We evaluate ST QA and MT QA on two public datasets, Topv2 (Chen et al., 2020) and Pizza (Arkoudas et al., 2021), and show that our results improve on the state of the art by  $3\%$  on full-data Topv2,  $28\%$  for few-shot Topv2, and  $7\%$  on few-shot Pizza.
Task-oriented parsing has been extensively studied in the literature. The most prevalent approach is seq2seq modeling, which maps utterances to their meaning representations, typically expressed as a mixture of natural language and tokens such as brackets and artificial intent and slot identifiers (Rongali et al., 2020; Zhou et al., 2021; Aghajanyan et al., 2020; Shrivastava et al., 2021; Mansimov and Zhang, 2021); we take this approach as our main baseline.
Few-shot semantic parsing has also attracted wide interest. Chen et al. (2020) applied a different training paradigm; they assumed there are several source domains with labeled data and adopted a first-order meta-learning algorithm, Reptile, to train their model.
Shin et al. (2021) argued that PLMs are better suited for directly generating natural language rather than task-specific meaning representations, and thus they fine-tune PLMs to generate canonical paraphrases, which can then be parsed by a context-free grammar to produce the corresponding semantic trees. They further improve performance by augmenting input sequences with similar examples as prompts. Rongali et al. (2022) push that direction further by leveraging small amounts of unannotated data. We use canonical paraphrasing as one of our
<sup>1</sup>The QA datasets converted from the TOP format are available at https://github.com/amazon-research/semantic-parsing-as-abstractive-qa.
belines. As we already noted in the introduction, canonical paraphrasing is not widely applicable in open-world task-oriented parsing; our QA-based approach overcomes that limitation. Desai et al. (2021) applied modifications to the linearized trees to make them more natural.
Inspired by the recent success of QA-driven approaches to a wide range of NLP tasks, such as dialogue state tracking (Gao et al., 2019), named entity recognition (Li et al., 2020), and multi-task learning (McCann et al., 2018; Du et al., 2021), Namazifar et al. (2021) framed semantic parsing as an extractive QA task. This limits its scope to non-compositional semantic structures. Our work is the first to use QA for semantic parsing of arbitrarily nested and complex meaning representations. Moreover, in contrast to previous approaches, our formulation usually results in fewer questions.
3 Reducing TOP to Abstractive QA
We now present a general method for reducing compositional TOP to abstractive QA. Given an utterance, our goal is to recover its semantic parse tree by asking questions and parsing the answers returned by a QA model. For this to succeed, all questions associated with an input utterance need to be answered correctly. At a high level, we formulate questions so as to build the required parse tree in a top-down, left-to-right fashion: We first ask a question to determine the root node, and then we recursively proceed towards the leaves. We start by describing multi-turn QA; the single-turn case is discussed in Section 3.3.
Each QA instance is a triple consisting of a context, a question, and an answer. We use the parse tree in Fig. 1 to illustrate the corresponding (multiturn) sequence of QA triples shown in Fig. 2. The context is the utterance provided by the user plus general information about the domain and/or state from previous turns; the question corresponds to a particular node of the parse tree; and the answer provides the content of that node. We first extract the top-level intent (get directions), then the corresponding slots (destination), then the value of each slot (destination is the nearest parking near S Beritania Street), and then we start to recursively repeat this process on the phrase the nearest parking near S Beritania Street (by asking what is the intent included in that utterance segment). We note that our system is able to handle negative answers: If there isn't a nested intent in an utterance segment, the
Multi-turn QA:
Q: A user may intend to get directions, get distance, get estimated arrival time, get estimated departure time, get estimated duration, get road condition information, get route's information, get traffic information, get location, make unsupported navigation, or update directions. A user said, "Look up directions to the nearest parking near S Beritania Street." What did the user intend to do?
A: get directions
Q: The slots for get directions may be locations, arrival datetimes, road conditions to avoid, waypoints, amounts, paths, sources, travel methods, road conditions, waypoints to avoid, departure datetimes, paths to avoid, obstructions to avoid, and destinations. A user said "Look up directions to the nearest parking near S Beritania Street." The user's intent is to get directions. What are the slots?
A:destination
Q: A user said "Look up directions to the nearest parking near S Beritania Street." The user's intent is to get directions, and the slot is destination. What is the destination?
A: the nearest parking near S Beritania Street
Q: The nested intent in destination may be get school's location, get home's location, get location, get event, and get workplace's location. A user said "Look up directions to the nearest parking near S Beritania Street." The user's intent is to get directions, and the destination is the nearest parking near S Beritania Street. Is there an intent included in "the nearest parking near S Beritania Street?"
A: get location
Single-turn QA:
Q: All possible intents from a user are [...], and slots could be [...]. A user said, "Look up directions to the nearest parking near S Beritania Street." What did the user intend to do?
A: The user intended to get directions, where destination is nearest parking near S Beritania Street. The intent for "nearest parking near S Beritania Street" is to get location, where location's category is parking and location modifiers are near S Beritania Street; nearest. The intent for "near S Beritania Street" is get location, where location is S Beritania Street and search radius is near.
Figure 2: Examples of multi-turn QA (top) and single-turn QA (bottom) for the utterance Look up directions to the nearest parking near S Beritania Street.
system returns none.
3.1 State Tracking
The triples we have described are processed independently from one another. To provide richer information to the QA model, we include previous answers as additional context (shown in italicized red font in Fig. 2). We represent a previous question-answer pair by combining them into a declarative sentence, and we stack all previous states together. This essentially encodes all parse-tree ancestors in natural language, which can potentially help the QA system resolve ambiguities. For example, the nested intent in the nearest parking near S Beritania Street could be different depending on whether
this was a source or a destination.
3.2 Incorporating Domain Metadata
A given domain has predefined semantics, i.e., a fixed number of intents, each of which has an associated set of slots that describe important aspects of user requests. We incorporate metadata relevant to the current node as additional context (shown in blue bold font in Fig. 2). This effectively reduces the space of possible answers. For instance, while a domain may have a large number of intents, the intents that can appear at a particular node (e.g., at the root level) may be substantially fewer. And because we explicitly list all possible intents, the QA system can simply copy and paste the appropriate tokens, which is easier than searching over the entire vocabulary.
3.3 Single-turn QA vs Multi-turn QA
We note that there is no need to limit ourselves to one question for every node. For the example of Fig. 2, for instance, we don't ask one question for each child node of IN:GET_LOCATION individually (e.g., what is the first/second/third slot); instead, we simply ask "what are the slots?" and the answer should be "location modifier, location category." Another example is when we ask "what is the location modifier?", the answer being "nearest; near S Beritania Street."
On the extreme side, we could ask one question that would return an answer representing the entire parse tree, and this becomes somewhat similar to canonical paraphrasing (Shin et al., 2021). However, canonical paraphrasing assumes there is concrete grammar that specifies a controlled fragment of natural language (all and only the canonical utterances), which can be used to map sentences from that fragment into parse trees. That assumption often fails in open-world TOP domains; for example, when someone asks for directions, the destination could be expressed by an unbounded number of phrases (my parent's house, a restaurant that satisfies an arbitrary set of constraints, etc.) that cannot be specified a priori by a closed-form grammar.
In the single-turn part of Fig. 2, we show how we compress an entire parse tree into a single QA triple. The bold blue context is again encoding the domain's metadata. From left to right, the answer explicitly lists the relevant intents, their associated slots, and the tokens corresponding to the slots in a top-down direction. We deal with nested intents by
recursively adding new sentences, which start with the tokens under which the intent is nested.
While our approach is more general than canonical paraphrasing, we still prefer canonicalization when possible, as the corresponding fragments tend to be more easily learnable. In the case of Pizza, for instance, a canonical grammar can be defined fairly straightforwardly. We illustrate the use of such canonical utterances in combination with our QA approach in Fig. 3. For multi-turn QA, we ask one question for each order in the utterance, and the answer is the order's canonical paraphrase. We also include the previous answers when asking about the next order, to prevent the QA system from repeating the same orders. For single-turn QA, we ask only one question for all orders. We also include the canonical paraphrasing formulation for comparison.
Thus, when canonical representations exist, single-turn QA and paraphrasing are similar; the main difference is that our formulation always includes a context. However, we reiterate that canonicalization is often not viable in TOP.
In summary, the general principle guiding the design of multi-turn interactions is that we first ask questions about intents, then we ask questions about their slots and slot values, and then repeat the process if we detect a nested intent in a slot. As for single-turn QA, we only ask one question and ensure that the answer encodes the entire tree. When canonical grammars exist in a given domain, they can be used to train single-turn QA systems in a straightforward way.
We experimentally evaluate the two QA variants and show how one may be preferred over the other under different settings.
3.4 Using MSP Objectives
Chada and Natarajan (2021) have shown that fine-tuning pretrained seq2seq models to perform QA tasks with too few examples leads to much degraded performance, while training a QA model with a loss function directly aligned with the pretraining objective performs better. Inspired by this observation, we explore the following change to our QA formulation: instead of making QA a separate downstream task, we treat it as one of the pretraining tasks—masked span prediction, for which the models are trained to generate the entire masked span given one unique mask token (Raffel et al., 2020). Accordingly, instead of asking the
ORDER
PIZZAORDER	PIZZAORDER	DRINKORDER
NUMBER	SIZE	STYLE	NUMBER	SIZE	TOPPING	COMPLEX	TOPPING	NUMBER	SIZE	DRINKTYPE
1	1	1	1	1	1	1	1	1	1	1
two	large	everything	two	large	mushrooms	QUANTITY	TOPPING	six	large	coke
extra	cheese
Canonical paraphrasing:
Grammar for drink orders:
I want [number] [size]/[column] [drinktype] [container]
Grammar for pizza orders:
I want [number] [size] pizza in the [style] style with [topping1; topping2; ...] with [complex_topping] without [not_topping1; not_topping2; ...]
Input: "[utterance]"
Output: I want two large pizza in the everything style; two large pizza with mushrooms with extra cheese; six large-sized cokes
Q: A user said: "[utterance]" What order did the user place?
Q:/ A user said: "[utterance]" The user ordered [MASK].
A: two large pizza in the everything style
Q: A user said: "[utterance]" What order did the user place in addition to two large pizza in the everything style?
Q/: A user said: "[utterance]" The user ordered [MASK] in addition to two large pizza in the everything style.
A: two large pizza with mushrooms with extra cheese
Q: A user said: "[utterance]" What order did the user place in addition to two large pizza in the everything style and two large pizza with mushrooms with extra cheese?
Q/: A user said: "[utterance]" The user ordered [MASK] in addition to two large pizza in the everything style and two large pizza with mushrooms with extra cheese.
A: six large-sized cokes
Q: A user said: "[utterance]" What order did the user place in addition to two large pizza in the everything style and two large pizza with mushrooms with extra cheese and six large-sized cokes?
Q/: A user said: "[utterance]" The user ordered [MASK] in addition to two large pizza in the everything style and two large pizza with mushrooms with extra cheese and six large-sized cokes.
A:none
Q: A user said: "... What orders did the user place?
Q/: A user said: “...” The user ordered [MASK].
A: The user ordered two large pizza in the everything style; two large pizza with mushrooms with extra cheese; six large-sized cokes
Figure 3: Parse tree, canonical paraphrasing formulation, multi-turn, and single-turn QA formulations for the utterance "i would like two large pizzas with everything and two large pizzas with mushrooms and extra cheese and four large cokes" in the Pizza dataset. The  $\mathbf{Q}^{\prime}$  questions are used for training with the MSP (masked span prediction) objective.
model questions and having it generate arbitrary answers, we rephrase the question-answer pair as a declarative sentence where the answer is masked. Thus, the model now has to denoise and recover the masked segment. We show an example of this approach in Fig. 3. The Q's are the original questions, whereas the primed Q's in red are the declarative
sentences with the answers masked out. The answers to both Q's and primed Q's are the same, as mask tokens cover the exact answers.
3.5 Converting Answers to Parse Trees
Single-turn QA. In single-turn QA, each sentence of the answer concerns up to a fixed number of levels in the parse tree. Take the Topv2 instance in the bottom part of Fig.2 as a running example, where each sentence contains three levels: The first level is an intent node, the second level is its slots, and the third level is the value of each slot. In the first sentence, which always corresponds to the root intent, we take what follows the phrase "The user wants to" as the intent, in the part before the first comma; the rest of the sentence is of the form "where  $S$  is  $V$ ", where  $S$  is the slot and  $V$  is the value. We note that one slot could have multiple values separated by semicolons, and for each value we create one slot node. In the case where an intent node doesn't have slot children, the sentence simply stops after the first part. To expand subsequent sentences into tree nodes, we make these sentences start with "The intent in [subutterance] is", so we can traverse the parse tree to find the "[subutterance]" node and expand the tree from there.
Multi-turn QA. In the multi-turn model, reconstructing the parse tree is more straightforward. We categorize all questions into three groups: the first asking the intent for a (sub-)utterance, the second asking the slots that appear in an intent, and the third asking for the value given to a slot. We parse the questions for an utterance sequentially and identify the group to which each question belongs. When the question asks for the top-level intent, we build the root node. When it asks for the intent of a sub-utterance, we traverse the parse tree to look for the leaf node containing the exact text and add an intent child node (with the text as a node attribute) from there. For a question from the second group, we find the intent node whose attribute has the same sub-utterance and append the slots as its children. When the question asks for a slot value, we traverse the parse tree again, find which slot node they belong to, and add it as a child node.
A final note for both single- and multi-turn QA: When we detect invalid entities generated by the QA models, we stop parsing and simply count such an instance as an incorrect output.
We evaluate our method against two state-of-the-art seq2seq techniques, one generating linearized parse trees and the other generating canonical paraphrases. We chose Topv2 and Pizza because they are the only two task-oriented datasets we are aware of with nested meaning representations that cannot be produced by slot-filling approaches. We investigate both multi- and single-turn QA. In the former, a question aims to recover one or several nodes in the parse tree; in single-turn QA, we only ask one question to reconstruct the entire tree. Additionally, we perform ablation analysis to evaluate the contribution of each component, as described in Sec. 3. We show that our method achieves superior performance, particularly on those TOPv2 utterances that have more nested semantics.