SlowGuess/NAACL_2021_OCR huggingface dataset:
SMBOP: Semi-autoregressive Bottom-up Semantic Parsing
Ohad Rubin  
Tel Aviv University  
ohadr@mail.tau.ac.il
Jonathan Berant  
Tel Aviv University  
Allen Institute for AI  
joberant@cs.tau.ac.il
The de-facto standard decoding method for semantic parsing in recent years has been to autoregressively decode the abstract syntax tree of the target program using a top-down depth-first traversal. In this work, we propose an alternative approach: a Semi-autoregressive Bottom-up Parser (SMBOP) that constructs at decoding step  $t$  the top-  $K$  sub-trees of height  $\leq t$ . Our parser enjoys several benefits compared to top-down autoregressive parsing. From an efficiency perspective, bottom-up parsing allows to decode all sub-trees of a certain height in parallel, leading to logarithmic runtime complexity rather than linear. From a modeling perspective, a bottom-up parser learns representations for meaningful semantic sub-programs at each step, rather than for semantically-vacuous partial trees. We apply SMBOP on SPIDER, a challenging zero-shot semantic parsing benchmark, and show that SMBOP leads to a 2.2x speed-up in decoding time and a  $\sim 5\mathrm{x}$  speed-up in training time, compared to a semantic parser that uses autoregressive decoding. SMBOP obtains 71.1 denotation accuracy on SPIDER, establishing a new state-of-the-art, and 69.5 exact match, comparable to the 69.6 exact match of the autoregressive RAT-SQL+GRAPPA.
Semantic parsing, the task of mapping natural language utterances into programs (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Clarke et al.; Liang et al., 2011), has converged in recent years on a standard encoder-decoder architecture. Recently, meaningful advances emerged on the encoder side, including developments in Transformer-based architectures (Wang et al., 2020a) and new pretraining techniques (Yin et al., 2020; Herzig et al., 2020; Yu et al., 2020; Deng et al., 2020; Shi et al., 2021). Conversely, the decoder has remained roughly constant for years, where the abstract syntax tree of the target program is autoregressively decoded in a
top-down manner (Yin and Neubig, 2017; Krishnamurthy et al., 2017; Rabinovich et al., 2017).
Bottom-up decoding in semantic parsing has received little attention (Cheng et al., 2019; Odena et al., 2020). In this work, we propose a bottom-up semantic parser, and demonstrate that equipped with recent developments in Transformer-based (Vaswani et al., 2017) architectures, it offers several advantages. From an efficiency perspective, bottom-up parsing can naturally be done semiautoregressively: at each decoding step  $t$ , the parser generates in parallel the top-  $K$  program sub-trees of depth  $\leq t$  (akin to beam search). This leads to runtime complexity that is logarithmic in the tree size, rather than linear, contributing to the rocketing interest in efficient and greener artificial intelligence technologies (Schwartz et al., 2020). From a modeling perspective, neural bottom-up parsing provides learned representations for meaningful (and executable) sub-programs, which are sub-trees computed during the search procedure, in contrast to top-down parsing, where hidden states represent partial trees without clear semantics.
Figure 1 illustrates a single decoding step of our parser. Given a beam  $Z_{t}$  with  $K = 4$  trees of height  $t$  (blue vectors), we use cross-attention to contextualize the trees with information from the input question (orange). Then, we score the frontier, that is, the set of all trees of height  $t + 1$  that can be constructed using a grammar from the current beam, and the top-  $K$  trees are kept (purple). Last, a representation for each of the new  $K$  trees is generated and placed in the new beam  $Z_{t + 1}$ . After  $T$  decoding steps, the parser returns the highest-scoring tree in  $Z_{T}$  that corresponds to a full program. Because we have gold trees at training time, the entire model is trained jointly using maximum likelihood.
We evaluate our model,  $\mathrm{SMBOP^1}$  (SeMi-autoregressive Bottom-up semantic Parser), on SPI
'Rhymes with 'MMMBop'.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: An overview of the decoding procedure of SMBOP.  $Z_{t}$  is the beam at step  $t$ ,  $Z_{t}^{\prime}$  is the contextualized beam after cross-attention,  $F_{t + 1}$  is the frontier  $(\kappa, \sigma, \geq$  are logical operations applied on trees, as explained below),  $F_{t + 1}^{\prime}$  is the pruned frontier, and  $Z_{t + 1}$  is the new beam. At the top we see the new trees created in this step. For  $t = 0$  (depicted here), the beam contains the predicted schema constants and DB values.
DER (Yu et al., 2018), a challenging zero-shot text-to-SQL dataset. We implement the RAT-SQL+GRAPPA encoder (Yu et al., 2020), currently the best model on SPIDER, and replace the autoregressive decoder with the semi-autoregressive SMBOP. SMBOP obtains an exact match accuracy of 69.5, comparable to the autoregressive RAT-SQL+GRAPPA at 69.6 exact match, and to current state-of-the-art at 69.8 exact match (Zhao et al., 2021), which applies additional pretraining. Moreover, SMBOP substantially improves state-of-the-art in denotation accuracy, improving performance from  $68.3 \rightarrow 71.1$ . Importantly, compared to autoregressive semantic parsing, we observe an average speed-up of  $2.2\mathrm{x}$  in decoding time, where for long SQL queries, speed-up is between  $5\mathrm{x}-6\mathrm{x}$ , and a training speed-up of  $\sim 5\mathrm{x}$ .
Problem definition We focus in this work on text-to-SQL semantic parsing. Given a training set  $\{(x^{(i)},y^{(i)},S^{(i)})\}_{i = 1}^{N}$ , where  $x^{(i)}$  is an utterance,  $y^{(i)}$  is its translation to a SQL query, and  $S^{(i)}$  is the schema of the target database (DB), our goal is to learn a model that maps new question-schema pairs
$(x, S)$  to the correct SQL query  $y$ . A DB schema  $S$  includes: (a) a set of tables, (b) a set of columns for each table, and (c) a set of foreign key-primary key column pairs describing relations between table columns. Schema tables and columns are termed schema constants, and denoted by  $S$ .
RAT-SQL encoder This work is focused on decoding, and thus we implement the state-of-the-art RAT-SQL encoder (Wang et al., 2020b), on top of GRAPPA (Yu et al., 2020), a pre-trained encoder for semantic parsing. We now briefly review this encoder for completeness.
The RAT-SQL encoder is based on two main ideas. First, it provides a joint contextualized representation of the utterance and schema. Specifically, the utterance  $x$  is concatenated to a linearized form of the schema  $S$ , and they are passed through a stack of Transformer (Vaswani et al., 2017) layers. Then, tokens that correspond to a single schema constant are aggregated, which results in a final contextualized representation  $(\mathbf{x},\mathbf{s}) = (\mathbf{x}_1,\dots ,\mathbf{x}_{|x|},\mathbf{s}_1,\dots ,\mathbf{s}_{|\mathbf{s}|})$ , where  $\mathbf{s}_i$  is a vector representing a single schema constant. This contextualization of  $x$  and  $S$  leads to better representation and alignment between the utterance and schema.
Second, RAT-SQL uses relational-aware self
2Our code is available at https://github.com/OhadRubin/SmBop
attention (Shaw et al., 2018) to encode the structure of the schema and other prior knowledge on relations between encoded tokens. Specifically, given a sequence of token representations  $(\mathbf{u}_1,\dots ,\mathbf{u}_{|\mathbf{u}|})$  relational-aware self-attention computes a scalar similarity score between pairs of token representations  $e_{ij}\propto \mathbf{u}_iW_Q(\mathbf{u}_jW_K + \mathbf{r}_{ij}^K)$ . This is identical to standard self-attention ( $W_{Q}$  and  $W_{K}$  are the query and key parameter matrices), except for the term  $\mathbf{r}_{ij}^{K}$ , which is an embedding that represents a relation between  $\mathbf{u}_i$  and  $\mathbf{u}_j$  from a closed set of possible relations. For example, if both tokens correspond to schema tables, an embedding will represent whether there is a primary foreign key relation between the tables. If one of the tokens is an utterance word and another is a table column, a relation will denote if there is a string match between them. The same principle is also applied for representing the self-attention values, where another relation embedding matrix is used. We refer the reader to the RAT-SQL paper for details.
Overall, RAT-SQL jointly encodes the utterance, schema, the structure of the schema and alignments between the utterance and schema, and leads to state-of-the-art results in text-to-SQL parsing.
RAT-SQL layers are typically stacked on top of a pre-trained language model, such as BERT (Devlin et al., 2019). In this work, we use GRAPPA (Yu et al., 2020), a recent pre-trained model that has obtained state-of-the-art results in text-to-SQL parsing. GRAPPA is based on ROBERTA (Liu et al., 2019), but is further fine-tuned on synthetically generated utterance-query pairs using an objective for aligning the utterance and query.
Autoregressive top-down decoding The prevailing method for decoding in semantic parsing has been grammar-based autoregressive top-down decoding (Yin and Neubig, 2017; Krishnamurthy et al., 2017; Rabinovich et al., 2017), which guarantees decoding of syntactically valid programs. Specifically, the target program is represented as an abstract syntax tree under the grammar of the formal language, and linearized to a sequence of rules (or actions) using a top-down depth-first traversal. Once the program is represented as a sequence, it can be decoded using a standard sequence-to-sequence model with encoder attention (Dong and Lapata, 2016), often combined with beam search. We refer the reader to the aforementioned papers for further details on grammar-based decoding.
We now turn to describe our method, which pro
Algorithm 1: SMBOP
1 input: utterance  $x$ , schema  $S$
2  $\mathbf{x},\mathbf{s}\gets \mathrm{Encode}_{\mathrm{RAT}}(x,S)$
3  $Z_{0}\gets$  Top-  $K$  schema constants and DB values
4 for  $t\gets 0\dots T - 1$  do
5  $Z_{t}^{\prime}\gets \mathrm{Attention}(Z_{t},\mathbf{x},\mathbf{x})$  6  $F_{t + 1}\gets \mathrm{Score - frontier}(Z_t')$
7  $F_{t + 1}^{\prime}\gets \arg \max_{K}(F_{t + 1})$
8  $\begin{array}{rl} & {\big{\lfloor}Z_{t + 1}\leftarrow \mathrm{Re}\mathrm{present - beam}(Z_t,F_{t + 1}^{\prime})} \end{array}$
9 return arg maxz(ZT)
vides a radically different approach for decoding in semantic parsing.
3 The SMBoP parser
We first provide a high-level overview of SMBoP (see Algorithm 1 and Figure 1). As explained in §2, we encode the utterance and schema with a RAT-SQL encoder. We initialize the beam (line 3) with the  $K$  highest scoring trees of height 0, which include either schema constants or DB values. All trees are scored independently and in parallel, in a procedure formally defined in §3.3.
Next, we start the search procedure. At every step  $t$ , attention is used to contextualize the trees with information from input question representation (line 5). This representation is used to score every tree on the frontier: the set of sub-trees of depth  $\leq t + 1$  that can be constructed from subtrees on the beam with depth  $\leq t$  (lines 6-7). After choosing the top-  $K$  trees for step  $t + 1$ , we compute a new representation for them (line 8). Finally, we return the top-scoring tree from the final decoding step,  $T$ . Steps in our model operate on tree representations independently, and thus each step is efficiently parallelized.
SMBOP resembles beam search as in each step it holds the top-  $K$  trees of a fixed height. It is also related to (pruned) chart parsing, since trees at step  $t + 1$  are computed from trees that were found at step  $t$ . This is unlike sequence-to-sequence models where items on the beam are competing hypotheses without any interaction.
We now provide the details of our parser. First, we describe the formal language ( $\S 3.1$ ), then we provide precise details of our model architecture ( $\S 3.2$ ) including beam initialization ( $\S 3.3$ ), we describe the training procedure ( $\S 3.4$ ), and last, we discuss the properties of SMBOP compared to prior work ( $\S 3.5$ ).
Operation	Notation	Input → Output
Set Union	∪	R × R → R
Set Intersection	∩	R × R → R
Set difference	\	R × R → R
Selection	σ	P × R → R
Cartesian product	×	R × R → R
Projection	Π	C' × R → R
And	∧	P × P → P
Or	∨	P × P → P
Comparison	{≤, ≥, =, ≠}	C × C → P
Constant Union	□	C' × C' → C'
Order by	τasc/dsc	C × R → R
Group by	γ	C × R → R
Limit	λ	C × R → R
In/Not In	∈,∉	C × R → P
Like/Not Like	~,≠	C × C → P
Aggregation	Gagg	C → C
Distinct	δ	C → C
Keep	κ	Any → Any
Table 1: Our relational algebra grammar, along with the input and output semantic types of each operation.  $P$ : Predicate,  $R$ : Relation,  $C$ : schema constant or DB value,  $C'$ : A set of constants/values, and  $\text{agg} \in \{\text{sum}, \max, \min, \text{count}, \text{avg}\}$ .
3.1 Representation of Query Trees
Relational algebra Guo et al. (2019) have shown recently that the mismatch between natural language and SQL leads to parsing difficulties. Therefore, they proposed SemQL, a formal query language with better alignment to natural language.
In this work, we follow their intuition, but instead of SemQL, we use the standard query language relational algebra (Codd, 1970). Relational algebra describes queries as trees, where leaves (terminals) are schema constants or DB values, and inner nodes (non-terminals) are operations (see Table 1). Similar to SemQL, its alignment with natural language is better than SQL. However, unlike SemQL, it is an existing query language, commonly used by SQL execution engines for query planning.
We write a grammar for relational algebra, augmented with SQL operators that are missing from relational algebra. We then implement a transpiler that converts SQL queries to relational algebra for parsing, and then back from relational algebra to SQL for evaluation. Table 1 shows the full grammar, including the input and output semantic types of all operations. A relation  $(R)$  is a tuple (or tuples), a predicate  $(P)$  is a Boolean condition (evaluating to True or False), a constant  $(C)$  is a schema constant or DB value, and  $(C')$  is a set of constants/values. Figure 2 shows an example re
(a) Unbalanced tree
(b) Balanced tree
Figure 2: An unbalanced and balanced relational algebra tree (with the unary KEEP operation) for the utterance "What are the names of actors older than 60?", where the corresponding SQL query is SELECT name FROM actor WHERE age  $\geq 60$ .
lational algebra tree with the corresponding SQL query. More examples illustrating the correspondence between SQL and relational algebra (e.g., for the SQL JOIN operation) are in Appendix B. While our relational algebra grammar can also be adapted for standard top-down autoregressive parsing, we leave this endeavour for future work.
Tree balancing Conceptually, at each step SMBoP should generate new trees of height  $\leq t + 1$  and keep the top-  $K$  trees computed so far. In practice, it is convenient to assume that trees are balanced. Thus, we want the beam at step  $t$  to only have trees that are of height exactly  $t$  (t-high trees).
To achieve this, we introduce a unary KEEP operation that does not change the semantics of the subtree it is applied on. Hence, we can always grow the height of trees in the beam without changing the formal query. For training (which we elaborate on in §3.4), we balance all relational algebra trees in the training set using the KEEP operation, such that the distance from the root to all leaves is equal. For example, in Figure 2, two KEEP operations are used to balance the column actor.name. After tree balancing, all constants and values are at height 0, and the goal of the parser at step  $t$  is to generate the gold set of  $t$ -high trees.
To fully specify Alg. 1, we need to define the following components: (a) scoring of trees on the frontier (lines 5-6), (b) representation of trees (line 8), and (c) representing and scoring of constants and DB values during beam initialization (leaves). We now describe these components. Figure 3 illustrates the scoring and representation of a binary operation.
Scoring with contextualized beams SMBOP maintains at each decoding step a beam  $Z_{t} =$
$((z_1^{(t)},\mathbf{z}_1^{(t)}),\ldots ,(z_K^{(t)},\mathbf{z}_K^{(t)}))$  , where  $z_{i}^{(t)}$  is a symbolic representation of the query tree, and  $\mathbf{z}_i^{(t)}$  is its corresponding vector representation. Unlike standard beam search, trees on our beams do not only compete with one another, but also compose with each other (similar to chart parsing). For example, in Fig. 1, the beam  $Z_{0}$  contains the column age and the value 60, which compose using the  $\geq$  operator to form the age  $\geq 60$  tree.
We contextualize tree representations on the beam using cross-attention. Specifically, we use standard attention (Vaswani et al., 2017) to give tree representations access to the input question:  $Z_{t}^{\prime}\gets \mathrm{Attention}(Z_{t},\mathbf{x},\mathbf{x})$ , where the tree representations  $(\mathbf{z}_1^{(t)},\dots ,\mathbf{z}_K^{(t)})$  are the queries, and the input tokens  $(\mathbf{x}_1,\dots ,\mathbf{x}_{|x|})$  are the keys and values.
Next, we compute scores for all  $(t + 1)$ -high trees on the frontier. Trees can be generated by applying either a unary (including KEEP) operation  $u \in \mathcal{U}$  or binary operation  $b \in \mathcal{B}$  on beam trees. Let  $\mathbf{w}_u$  be a scoring vector for a unary operation (such as  $\mathbf{w}_{\kappa}, \mathbf{w}_{\delta}$ , etc.), let  $\mathbf{w}_b$  be a scoring vector for a binary operation (such as  $\mathbf{w}_{\sigma}, \mathbf{w}_{\Pi}$ , etc.), and let  $\mathbf{z}_i', \mathbf{z}_j'$  be contextualized tree representations on the beam. We define a scoring function for frontier trees, where the score for a new tree  $z_{\mathrm{new}}$  generated by applying a unary rule  $u$  on a tree  $z_i$  is defined as follows:
$$
s (z _ {\mathrm {n e w}}) = \mathbf {w} _ {u} ^ {\top} F F _ {U} ([ \mathbf {z} _ {i}; \mathbf {z} _ {i} ^ {\prime} ]),
$$
where  $FF_{U}$  is a 2-hidden layer feed-forward layer with relu activations, and  $[\cdot ;\cdot ]$  denotes concatenation. Similarly the score for a tree generated by applying a binary rule  $b$  on the trees  $z_{i},z_{j}$  is:
$$
s (z _ {\mathrm {n e w}}) = \mathbf {w} _ {b} ^ {\top} F F _ {B} ([ \mathbf {z} _ {i}; \mathbf {z} _ {i} ^ {\prime}; \mathbf {z} _ {j}; \mathbf {z} _ {j} ^ {\prime} ]),
$$
where  $FF_{B}$  is another 2-hidden layer feed-forward layer with relu activations.
We use semantic types to detect invalid rule applications and fix their score to  $s(z_{\mathrm{new}}) = -\infty$ . This guarantees that the trees SMBOP generates are well-formed, and the resulting SQL is executable. Overall, the total number of trees on the frontier is  $\leq K|\mathcal{U}| + K^2 |\mathcal{B}|$ . Because scores of different trees on the frontier are independent, they are efficiently computed in parallel. Note that we score new trees from the frontier before creating a representation for them, which we describe next.
Recursive tree representation after scoring the frontier, we generate a recursive vector representation for the top-  $K$  trees. While scoring is done with
Figure 3: Illustration of our tree scoring and representation mechanisms.  $z$  is the symbolic tree,  $\mathbf{z}$  is its vector representation, and  $\mathbf{z}'$  its contextualized representation.
contextualized trees, representations are not contextualized. We empirically found that contextualized tree representations slightly reduce performance, possibly due to optimization issues.
We represent trees with another standard Transformer layer. Let  $\mathbf{z}_{\mathrm{new}}$  be the representation for a new tree, let  $e_{\ell}$  be an embedding for a unary or binary operation, and let  $\mathbf{z}_i,\mathbf{z}_j$  be non-contextualized tree representations from the beam we are extending. We compute a new representation as follows:
$$
\mathbf {z} _ {\text {n e w}} = \left\{ \begin{array}{c c} \operatorname {T r a n s f o r m e r} (\mathbf {e} _ {\ell}, \mathbf {z} _ {i}) & \text {u n a r y} \ell \\ \operatorname {T r a n s f o r m e r} (\mathbf {e} _ {\ell}, \mathbf {z} _ {i}, \mathbf {z} _ {j}) & \text {b i n a r y} \ell \\ \mathbf {z} _ {i} & \ell = \text {K E E P} \end{array} \right.
$$
where for the unary KEEP operation, we simply copy the representation from the previous step.
Return value As mentioned, the parser returns the highest-scoring tree in  $Z_{T}$ . More precisely, we return the highest-scoring returnable tree, where a returnable tree is a tree that has a valid semantic type, that is, Relation (R).
3.3 Beam initialization
As described in Line 3 of Alg. 1, the beam  $Z_{0}$  is initialized with  $K$  schema constants (e.g., actor, age) and DB values (e.g., 60, "France"). In particular, we independently score schema constants and choose the top- $\frac{K}{2}$ , and similarly score DB values and choose the top- $\frac{K}{2}$ , resulting in a total beam of size  $K$ .
Schema constants We use a simple scoring function  $f_{\mathrm{const}}(\cdot)$ . Recall that  $\mathbf{s}_i$  is a representation of a
constant, contextualized by the rest of the schema and the utterance. The function  $f_{\mathrm{const}}(\cdot)$  is a feedforward network that scores each schema constant independently:  $f_{\mathrm{const}}(\mathbf{s}_i) = \mathbf{w}_{\mathrm{const}} \tanh(W_{\mathrm{const}} \mathbf{s}_i)$ , and the top- $\frac{K}{2}$  constants are placed in  $Z_0$ .
DB values Because the number of values in the DB is potentially huge, we do not score all DB values. Instead, we learn to detect spans in the question that correspond to DB values. This leads to a setup that is similar to extractive question answering (Rajpurkar et al., 2016), where the model outputs a distribution over input spans, and thus we adopt the architecture commonly used in extractive question answering. Concretely, we compute the probability that a token is the start token of a DB value,  $P_{\mathrm{start}}(x_i) \propto \exp(\mathbf{w}_{\mathrm{start}}^\top \mathbf{x}_i)$ , and similarly the probability that a token is the end token of a DB value,  $P_{\mathrm{end}}(x_i) \propto \exp(\mathbf{w}_{\mathrm{end}}^\top \mathbf{x}_i)$ , where  $\mathbf{w}_{\mathrm{start}}$  and  $\mathbf{w}_{\mathrm{end}}$  are parameter vectors. We define the probability of a span  $(x_i, \ldots, x_j)$  to be  $P_{\mathrm{start}}(x_i) \cdot P_{\mathrm{end}}(x_j)$ , and place in the beam  $Z_0$  the top-  $\frac{K}{2}$  input spans, where the representation of a span  $(x_i, x_j)$  is the average of  $\mathbf{x}_i$  and  $\mathbf{x}_j$ .
A current limitation of SMBOP is that it cannot generate DB values that do not appear in the input question. This would require adding a mechanism such as "BRIDGE" proposed by Lin et al. (2020).
To specify the loss function, we need to define the supervision signal. Recall that given the gold SQL program, we convert it into a gold balanced relational algebra tree  $z^{\mathrm{gold}}$ , as explained in §3.1 and Figure 2. This lets us define for every decoding step the set of  $t$ -high gold sub-trees  $\mathcal{Z}_t^{\mathrm{gold}}$ . For example  $\mathcal{Z}_0^{\mathrm{gold}}$  includes all gold schema constants and input spans that match a gold DB value,  $^3\mathcal{Z}_1^{\mathrm{gold}}$  includes all 1-high gold trees, etc.
During training, we apply "bottom-up Teacher Forcing" (Williams and Zipser, 1989), that is, we populate<sup>4</sup> the beam  $Z_{t}$  with all trees from  $\mathcal{Z}_{t}^{\mathrm{gold}}$  and then fill the rest of the beam (of size  $K$ ) with the top-scoring non-gold predicted trees. This guarantees that we will be able to compute a loss at each decoding step, as described below.
Loss function During search, our goal is to give high scores to the possibly multiple sub-trees of