SlowGuess/NAACL_2021_OCR huggingface dataset:
Bailin Wang, Mirella Lapata and Ivan Titov  
Institute for Language, Cognition and Computation  
School of Informatics, University of Edinburgh
bailin.wang@ed.ac.uk, {mlap, ititov}@inf.ed.ac.uk
Semantic parsing aims at translating natural language (NL) utterances onto machine-interpretable programs, which can be executed against a real-world environment. The expensive annotation of utterance-program pairs has long been acknowledged as a major bottleneck for the deployment of contemporary neural models to real-life applications. In this work, we focus on the task of semi-supervised learning where a limited amount of annotated data is available together with many unlabeled NL utterances. Based on the observation that programs which correspond to NL utterances must be always executable, we propose to encourage a parser to generate executable programs for unlabeled utterances. Due to the large search space of executable programs, conventional methods that use approximations based on beam-search such as self-training and top-k marginal likelihood training, do not perform as well. Instead, we view the problem of learning from executions from the perspective of posterior regularization and propose a set of new training objectives. Experimental results on OVERNIGHT and GEOQUERY show that our new objectives outperform conventional methods, bridging the gap between semi-supervised and supervised learning.
Semantic parsing is the task of mapping natural language (NL) utterances to meaning representations (aka programs) that can be executed against a real-world environment such as a knowledge base or a relational database. While neural sequence-to-sequence models (Dong and Lapata, 2016; Jia and Liang, 2016a) have achieved much success in this task in recent years, they usually require a large amount of labeled data (i.e., utterance-program pairs) for training. However, annotating utterances with programs is expensive as it requires expert knowledge of meaning representations (e.g., lambda calculus, SQLs) and the environ-
list all 3 star rated thai restaurants
Program Candidates	Gold	Exe
select restaurant where starrating = thai	X	X
select restaurant where cuisine > 3	X	X
select restaurant where starRating = 3	X	✓
select restaurant where starRating = 3 and cuisine = thai
✓	✓
Figure 1: Candidate programs for an utterance can be classified by executability (Exe); note that the gold program is always in the set of executable programs. We propose to utilize the weak yet freely available signal of executability for learning.
ronment against which they are executed (e.g., a knowledge base, a relational database). An alternative to annotation is to collect answers (or denotations) of programs, rather than programs themselves (Liang et al., 2013; Berant et al., 2013). In this work, we focus on the more extreme setting where there are no annotations available for a large number of utterances. This setting resembles a common real-life scenario where massive numbers of user utterances can be collected when deploying a semantic parser (Iyer et al., 2017). Effectively utilizing the unlabeled data makes it possible for a semantic parser to improve over time without human involvement.
Our key observation is that not all candidate programs for an utterance will be semantically valid. This implies that only some candidate programs can be executed and obtain non-empty execution results.<sup>1</sup> As illustrated in Figure 1, executability is a weak signal that can differentiate between semantically valid and invalid programs. On unlabeled utterances, we can encourage a parser to only focus on executable programs ignoring non-executable ones. Moreover, the executability of a program
<sup>1</sup>In the rest of this paper, we extend the meaning of 'executability', and use it to refer to the case where a program is executable and obtains non-empty results.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2747-2759 June 6-11, 2021. ©2021 Association for Computational Linguistics
can be obtained from an executor for free without requiring human effort. Executability has previously been used to guide the decoding of a semantic parser (Wang et al., 2018). We take a step further to directly use this weak signal for learning from unlabeled utterances.
To learn from executability, we resort to marginal likelihood training, i.e., maximizing the marginal likelihood of all executable programs for an unlabeled NL utterance. However, the space of all possible programs is exponentially large, as well as the space of executable ones. Hence, simply marginalizing over all executable programs is intractable. Typical approximations use beam search to retrieve a handful of ('seen') programs, which are used to approximate the entire space. Using such approximations can lead to optimization getting trapped in undesirable local minima. For example, we observe that encouraging a model to exploit seen executable programs hinders exploration and reinforces the preference for shorter programs, as discussed in Section 6.3. This happens because shorter programs are both more likely to be among 'seen' programs (probably due to using locally-normalized autoregressive modeling) and more likely to be executable. To alleviate these issues, we derive three new alternative objectives, relying on a new interpretation of marginal likelihood training from the perspective of posterior regularization. Our proposed objectives encode two kinds of inductive biases: 1) discouraging seen non-executable programs, which plays a similar role to encouraging seen executable ones but does not share its drawback of hindering exploration; 2) encouraging sparsity among executable programs, which encourages a parser to only focus on a subset of executable programs by softly injecting a sparsity constraint. This is desirable, as there are only one or few correct programs for each utterance (see Figure 1), and an accurate parser should assign probability mass only to this subset. We collectively call these objectives X-PR, as a shorthand for Execution-guided Posterior Regularization.
We conduct experiments on two representative semantic parsing tasks: text-to-LF (logical form) parsing over a knowledge base and text-to-SQL (Zelle and Mooney, 1996) parsing over a relational database. Concretely, we evaluate our methods on the OVERNIGHT (Wang et al., 2015a) and GEO-QUERY datasets. We simulate the semi-supervised learning setting by treating  $70\%$  of the training
data as unlabeled. Empirical results show that our method can substantially boost the performance of a parser, trained only on labeled data, by utilizing a large amount of unlabeled data.
- We show how to exploit unlabeled utterances by taking advantage of their executability.
- To better learn from executability, we propose a set of new objectives based on posterior regularization.
- Our method can help a base parser achieve substantially better performance by utilizing unlabeled data.
Our code, datasets, and splits are publicly available at https://github.com/berlino/tensor2struct-public.
Semi-Supervised Semantic Parsing In the context of semantic parsing, semi-supervised models using limited amounts of parallel data and large amounts of unlabeled data treat either utterances or programs as discrete latent variables and induce them in the framework of generative models (Kočisky et al., 2016; Yin et al., 2018). A challenge with these methods is that (combinatorially) complex discrete variables make optimization very hard, even with the help of variational inference. In this work, we seek to directly constrain the discriminative parser with signals obtained from executions. Our method can potentially be integrated into these generative models to regularize discrete variables.
(Underspecified) Sequence-Level Rewards There have been attempts in recent years to integrate sequence-level rewards into sequence-to-sequence training as a way of accommodating task-specific objectives. For example, BLEU can be optimized for coherent text generation (Bosselut et al., 2018) and machine translation (Wu et al., 2018) via reinforcement learning or beam-search (Wiseman and Rush, 2016). In this work, we resort to marginal likelihood training to exploit binary executability rewards for semantic parsing (i.e., whether a program is executable or not), which has been shown to be more effective than REINFORCE (Guu et al., 2017).
More importantly, our binary reward is underspecified, i.e., there exist many spurious programs that enjoy the same reward as the gold program.
This issue of learning from underspecified rewards underlies many weakly-supervised tasks, e.g., learning from denotations (Liang et al., 2013; Berant et al., 2013), weakly supervised question answering (Min et al., 2019). Previous work tried to model latent alignments (Wang et al., 2019) between NL and programs to alleviate this issue. In this work, we take an orthogonal direction and propose several training objectives that alleviate the impact of spurious programs.
Execution for Semantic Parsing Execution has been utilized in semantic parsing (Wang et al., 2018) and the related area of program synthesis (Chen et al., 2019). These approaches exploit the execution of partial programs to guide the search for plausible complete programs. Although partial execution is feasible for SQL-style programs, it cannot be trivially extended to general meaning representation (e.g., logical forms). In this work, we explore a more general setting where execution can be only obtained from complete programs.
3 Executability as Learning Signal
In this section, we formally define our semi-supervised learning setting and show how to incorporate executability into the training objective whilst relying on the marginal likelihood training framework. We also present two conventional approaches to optimizing marginal likelihood.
Given a set of labeled NL-program pairs  $\{(x_i^l,y_i^l)\}_{i = 1}^N$  and a set of unlabeled NL utterances  $\{x_j\}_{j = 1}^M$ , where  $N$  and  $M$  denote the sizes of the respective datasets, we would like to learn a neural parser  $p(y|x,\theta)$ , parameterized by  $\theta$ , that maps utterances to programs. The objective to minimize consists of two parts:
$$
\mathcal {J} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {\sup } \left(x _ {i} ^ {l}, y _ {i} ^ {l}\right) + \lambda \frac {1}{M} \sum_ {j = 1} ^ {M} \mathcal {L} _ {\operatorname {u n s u p}} \left(x _ {i}\right) \tag {1}
$$
where  $\mathcal{L}_{sup}$  and  $\mathcal{L}_{unsup}$  denote the supervised and unsupervised loss, respectively. For labeled data, we use the negative log-likelihood of gold programs; for unlabeled data, we instead use the log marginal likelihood (MML) of all executable pro
grams. Specifically, they are defined as follows:
$$
\mathcal {L} _ {\sup } (x, y) = - \log p (y | x, \boldsymbol {\theta}) \tag {2}
$$
$$
\mathcal {L} _ {\text {u n s u p}} (x) = - \log \sum_ {y} R (y) p (y | x, \boldsymbol {\theta}) \tag {3}
$$
where  $R(y)$  is a binary reward function that returns 1 if  $y$  is executable and 0 otherwise. In practice, this function is implemented by running a task-specific executor, e.g., a SQL executor.
Another alternative to unsupervised loss is REINFORCE (Sutton et al., 1999), i.e., maximize the expected  $R(y)$  with respect to  $p(y|x,\theta)$ . However, as presented in Guu et al. (2017), this objective usually underperforms MML, which is consistent with our initial experiments.
3.2 Self-Training and Top-K MML
MML in Equation (3) requires marginalizing over all executable programs which is intractable. Conventionally, we resort to beam search to explore the space of programs and collect executable ones. To illustrate, we can divide the space of programs into four parts based on whether they are executable and observed, as shown in Figure 2a. For example, programs in  $P_{\mathrm{SE}} \cup P_{\mathrm{SN}}$  are seen in the sense that they are retrieved by beam search. Programs in  $P_{\mathrm{SE}} \cup P_{\mathrm{UE}}$  are all executable, though only programs in  $P_{\mathrm{SE}}$  can be directly observed.
Two common approximations of Equation (3) are Self-Training (ST) and Top-K MML, and they are defined as follows:
$$
\mathcal {L} _ {\mathrm {S T}} (x, \boldsymbol {\theta}) = - \log p \left(y ^ {*} \mid x, \boldsymbol {\theta}\right) \tag {4}
$$
$$
\mathcal {L} _ {\text {t o p - k}} (x, \boldsymbol {\theta}) = - \log \sum_ {y \in P _ {\mathrm {S E}}} p (y | x, \boldsymbol {\theta}) \tag {5}
$$
where  $y^{*}$  denotes the most probable program, and it is approximated by the most probable one from beam search.
It is obvious that both methods only exploit programs in  $P_{\mathrm{SE}}$ , i.e., executable programs retrieved by beam search. In cases where a parser successfully includes the correct programs in  $P_{\mathrm{SE}}$ , both approximations should work reasonably well. However, if a parser is uncertain and  $P_{\mathrm{SE}}$  does not contain the gold program, it would then mistakenly exploit incorrect programs in learning, which is problematic.
A naive solution to improve Self-Training or Top-K MML is to explore a larger space, e.g., increase the beam size to retrieve more executable
2We review the comparison between REINFORCE and MML in the appendix.
Seen Programs	Unseen Programs
* PSE	PUE
PSN	PUN
(a) Partitioned program space. Red asterisk denotes the most probable executable program  $y^{*}$ . P stands for program; subscript S stands for seen, U for unseen, E for executable, and N for non-executable.
$$
\mathcal {L} _ {\mathrm {S T}} (x, \boldsymbol {\theta}) = - \log p (y ^ {*} | x, \boldsymbol {\theta})
$$
$$
\mathcal {L} _ {\text {t o p - k}} (x, \boldsymbol {\theta}) = - \log \sum_ {y \in P _ {\mathrm {S E}}} p (y | x, \boldsymbol {\theta})
$$
$$
\mathcal {L} _ {\text {r e p u l s i o n}} (x, \boldsymbol {\theta}) = - \log \left(1 - \sum_ {y \in P _ {\mathrm {S N}}} p (y | x, \boldsymbol {\theta})\right)
$$
$$
\begin{array}{l} \mathcal {L} _ {\text {g e n t l e}} (x, \boldsymbol {\theta}) = - p \left(P _ {\mathrm {S E}} \cup P _ {\mathrm {S N}}\right) \log \sum_ {y \in P _ {\mathrm {S E}}} p (y | x, \boldsymbol {\theta}) \\ - p \left(P _ {\mathrm {U E}} \cup P _ {\mathrm {U N}}\right) \log \sum_ {y \in P _ {\mathrm {U E}} \cup P _ {\mathrm {U N}}} p (y | x, \boldsymbol {\theta}) \\ \end{array}
$$
$$
\mathcal {L} _ {\text {s p a r s e}} (x, \boldsymbol {\theta}) = - \sum_ {y \in P _ {\mathrm {S E}}} q _ {\text {s p a r s e}} (y) \log p (y | x, \boldsymbol {\theta})
$$
(b) Five objectives to approximate MML.
Figure 2: In (a) the program space is partitioned along two dimensions: executability and observability. In (b) we show two commonly used objectives (Self-Training and Top-K MML) and the three objectives proposed in this work.
programs. However, this would inevitably increase the computation cost of learning. We also show in the appendix that increasing beam size, after it exceeds a certain threshold, is no longer beneficial for learning. In this work, we instead propose better approximations without increasing beam size.
We first present a view of MML in Equation (3) from the perspective of posterior regularization. This new perspective helps us derive three alternative approximations of MML: Repulsion MML, Gentle MML, and Sparse MML.
4.1 Posterior Regularization
Posterior regularization (PR) allows to inject linear constraints into posterior distributions of generative models, and it can be extended to discriminative models (Ganchev et al., 2010). In our case, we try to constrain the parser  $p(y|x,\theta)$  to only assign probability mass to executable programs. Instead of imposing hard constraints, we softly penalize the parser if it is far away from a desired distribution  $q(y)$ , which is defined as  $\mathbb{E}_q[R(y)] = 1$ . Since  $R$  is a binary reward function,  $q(y)$  is constrained to only place mass on executable programs whose rewards are 1. We denote all such desired distributions as the family  $\mathcal{Q}$ .
Specifically, the objective of PR is to penalize
the KL-divergence between  $\mathcal{Q}$  and  $p$ , which is:
$$
\begin{array}{l} \mathcal {J} _ {\mathcal {Q}} (\boldsymbol {\theta}) = D _ {\mathrm {K L}} [ \mathcal {Q} | | p (y | x, \boldsymbol {\theta}) ] \\ = \min  _ {q \in \mathcal {Q}} D _ {\mathrm {K L}} [ q (y) | | p (y | x, \boldsymbol {\theta}) ] \tag {6} \\ \end{array}
$$
By definition, the objective has the following upper bound:
$$
\begin{array}{l} \mathcal {J} (\boldsymbol {\theta}, q) = D _ {\mathrm {K L}} [ q (y) | | p (y | x, \boldsymbol {\theta}) ] \\ = - \sum_ {y} q (y) \log p (y | x, \boldsymbol {\theta}) - \mathcal {H} (q) \tag {7} \\ \end{array}
$$
where  $q\in \mathcal{Q}$ $\mathcal{H}$  denotes the entropy. We can use block-coordinate descent, an EM iterative algorithm to optimize it.
$$
\mathrm{E}:q^{t + 1} = \operatorname *{arg  min}_{q\in \mathcal{Q}}D_{\mathrm{KL}}[q(y)||p(y|x,\boldsymbol{\theta}^{t})]
$$
$$
\mathbf {M}: \boldsymbol {\theta} ^ {t + 1} = \underset {\boldsymbol {\theta}} {\arg \min } - \sum_ {y} q ^ {t + 1} (y) [ \log p (y | x, \boldsymbol {\theta}) ]
$$
During the E-step, we try to find a distribution  $q$  from the constrained set  $\mathcal{Q}$  that is closest to the current parser  $p$  in terms of KL-divergence. We then use  $q$  as a 'soft label' and minimize the cross-entropy between  $q$  and  $p$  during the M-step. Note that  $q$  is a constant vector and has no gradient wrt.  $\theta$  during the M-step.
The E-step has a closed-form solution:
$$
q ^ {t + 1} (y) = \left\{ \begin{array}{c c} \frac {p \left(y \mid x , \boldsymbol {\theta} ^ {t}\right)}{p \left(P _ {\mathrm {S E}} \cup P _ {\mathrm {U E}}\right)} & y \in P _ {\mathrm {S E}} \cup P _ {\mathrm {U E}} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
where  $p(P_{\mathrm{SE}} \cup P_{\mathrm{UE}}) = \sum_{y' \in P_{\mathrm{SE}} \cup P_{\mathrm{UE}}} p(y'|x, \boldsymbol{\theta}^t)$ .  $q^{t+1}(y)$  is essentially a re-normalized version of  $p$  over executable programs. Interestingly, if we use the solution in the M-step, the gradient wrt.  $\boldsymbol{\theta}$  is equivalent to the gradient of MML in Equation (3). That is, optimizing PR with the EM algorithm is equivalent to optimizing MML. The connection between EM and MML is not new, and it has been well-studied for classification problems (Amini and Gallinari, 2002; Grandvalet and Bengio, 2004). In our problem, we additionally introduce PR to accommodate the executability constraint, and instantiate the general EM algorithm.
Although the E-step has a closed-form solution, computing  $q$  is still intractable due to the large search space of executable programs. However, this PR view provides new insight on what it means to approximate MML. In essence, conventional methods can be viewed as computing an approximate solution of  $q$ . Specifically, Self-Training corresponds to a delta distribution that only focuses on the most probable  $y^{*}$ .
$$
q _ {\mathrm {S T}} ^ {t + 1} (y) = \left\{ \begin{array}{l l} 1 & y = y ^ {*} \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Top-K MML corresponds to a re-normarized distribution over  $P_{\mathrm{SE}}$
$$
q _ {\mathrm {t o p - k}} ^ {t + 1} (y) = \left\{ \begin{array}{c c} \frac {p (y | x , \boldsymbol {\theta} ^ {t})}{p (P _ {\mathrm {s E}})} & y \in P _ {\mathrm {s E}} \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Most importantly, this perspective leads us to deriving three new approximations of MML, which we collectively call X-PR.
4.2 Repulsion MML and Gentle MML
As mentioned previously, Self-Training and Top-K MML should be reasonable approximations in cases where gold programs are retrieved, i.e., they are in the seen executable subset  $(P_{\mathrm{SE}}$  in Figure 2a). However, if a parser is uncertain, i.e., beam search cannot retrieve the gold programs, exclusively exploiting  $P_{\mathrm{SE}}$  programs is undesirable. Hence, we consider ways of taking unseen executable programs  $(P_{\mathrm{UE}}$  in Figure 2a) into account. Since we never directly observe unseen programs  $(P_{\mathrm{UE}}$  or  $P_{\mathrm{UN}})$ , our heuristics do not discriminate between executable and non-executable programs  $(P_{\mathrm{UE}} \cup P_{\mathrm{UN}})$ . In other words, upweighting  $P_{\mathrm{UE}}$  programs will inevitably upweight  $P_{\mathrm{UN}}$ .
Based on the intuition that the correct program is included in either seen executable programs  $(P_{\mathrm{SE}})$  or unseen programs  $(P_{\mathrm{UE}}$  and  $P_{\mathrm{UN}})$ , we can simply push a parser away from seen non-executable programs  $(P_{\mathrm{SN}})$ . Hence, we call such method Repulsion MML. Specifically, the first heuristic approximates Equation (8) as follows:
$$
q _ {\text {r e p u l s i o n}} ^ {t + 1} (y) = \left\{ \begin{array}{c c} \frac {p (y | x , \boldsymbol {\theta} ^ {t})}{1 - p (P _ {\text {s N}})} & y \notin P _ {\text {s N}} \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Another way to view this heuristic is that we distribute the probability mass from seen nonexecutable programs  $(P_{\mathrm{SN}})$  to other programs. In contrast, the second heuristic is more 'conservative' about unseen programs as it tends to trust seen executable  $P_{\mathrm{SN}}$  programs more. Specifically, the second heuristic uses the following approximations to solve the E-step.
$$
q _ {\text {g e n t l e}} ^ {t + 1} (y) = \left\{ \begin{array}{c c} \frac {p (P _ {\text {S E} \cup \text {S N}})}{p (P _ {\text {S E}})} p (y | x, \boldsymbol {\theta} ^ {t}) & y \in P _ {\text {S E}} \\ p (y | x, \boldsymbol {\theta} ^ {t}) & y \in P _ {\text {U E}} \cup P _ {\text {U N}} \\ 0 & y \in P _ {\text {S N}} \end{array} \right.
$$
Intuitively, it shifts the probability mass of seen non-executable programs  $(P_{\mathrm{SN}})$  directly to seen executable programs  $(P_{\mathrm{SE}})$ . Meanwhile, it neither upweights nor downweights unseen programs. We call this heuristic Gentle MML. Compared with Self-Training and Top-K MML, Repulsion MML and Gentle MML lead to better exploration of the program space, as only seen non-executable  $(P_{\mathrm{SN}})$  programs are discouraged.
4.3 Sparse MML
Sparse MML is based on the intuition that in most cases there is only one or few correct programs among all executable programs. As mentioned in Section 2, spurious programs that are executable, but do not reflect the semantics of an utterance are harmful. One empirical evidence from previous work (Min et al., 2019) is that Self-Training outperforms Top-K MML for weakly-supervised question answering. Hence, exploiting all seen executable programs can be sub-optimal. Following recent work on sparse distributions (Martins and Astudillo, 2016; Niculae et al., 2018), we propose to encourage sparsity of the 'soft label'  $q$ . Encouraging sparsity is also related to the minimum entropy and low-density separation principles which are commonly used in semi-supervised learning (Grandvalet and Bengio, 2004; Chapelle and Zien, 2005).
3Please see the appendix for the proof and analysis of PR.
To achieve this, we first interpret the entropy term  $\mathcal{H}$  in Equation (7) as a regularization of  $q$ . It is known that entropy regularization always results in a dense  $q$ , i.e., all executable programs are assigned non-zero probability. Inspired by SparseMax (Martins and Astudillo, 2016), we instead use L2 norm for regularization. Specifically, we replace our PR objective in Equation (7) with the following one:
$$
\mathcal {J} _ {\text {s p a r s e}} (\boldsymbol {\theta}, q) = - \sum_ {y} q (y) \log p (y | s, \boldsymbol {\theta}) + \frac {1}{2} \| q \| _ {2} ^ {2}
$$
where  $q\in \mathcal{Q}$  . Similarly, it can be optimized by the EM algorithm:
$$
\mathrm {E}: q ^ {t + 1} = \operatorname {S p a r s e M a x} _ {\mathcal {Q}} (\log p (y | x, \boldsymbol {\theta} ^ {t}))
$$