SlowGuess/NAACL_2021_OCR huggingface dataset:
where  $q$  can be viewed as a soft-label that is induced from  $p$ . For RL,  $q_{\mathrm{RL}}(y) = p_{\theta}(y|x,\theta)R(y)$ ; for MML,  $q_{\mathrm{MML}}(y) = \frac{R(y)p_{\theta}(y|x)}{\sum_{y'}R(y')p_{\theta}(y'|x)} = p_{\theta}(y|x,R(y) = 1)$ . Compared with RL, MML additionally renormalize the  $p$  over executable programs to obtain  $q$ . As a result, MML outputs a 'stronger' gradient than RL due to the renormalization. In our initial experiments, we found that RL is hard to converge whereas MML is not.
Proof of the E-Step Solution Denote the set of executable programs as  $\mathcal{V}$ . Since  $q(y)$  is 0 for non-executable programs, we only need to compute it for executable programs in  $\mathcal{V}$ . We need to solve the following optimization problem:
$$
\min  _ {q} - \sum_ {y \in \mathcal {V}} q (y) \log p (y | x, \boldsymbol {\theta}) + \sum_ {y \in \mathcal {V}} q (y) \log q (y)
$$
s.t.  $\sum_{y\in \mathcal{V}}q(y) = 1$
$$
q (y) \geq 0 \tag {14}
$$
By solving it with KKT conditions, we can see that  $q(y) \propto p(y|x,\pmb{\theta})$ . Since  $q(y)$  needs to sum up to 1, it is easy to obtain that  $q(y) = \frac{p(y|x,\pmb{\theta})}{\sum_{y \in \nu} p(y|x,\pmb{\theta})}$ .
Relation-Aware Encoding of Semantic Parsers Relation-aware encoding was originally introduced for text-to-SQL parsing in Wang et al. (2020) to accommodate discrete relations among schema items (e.g., tables and columns) and linking between an
Type of x	Type of y	Edge label	Description
Entity	Entity	RELATED-F	there exists a property p s.t. (x,p,y) ∈ K
RELATED-R	there exists a property p s.t. (y,p,x) ∈ K
Entity	Property	HAS-PROPERTY-F	there exists an entity e s.t. (x,y,e) ∈ K
HAS-PROPERTY-R	there exists an entity e s.t. (e,y,x) ∈ K
Property	Entity	PROP-TO-ENT-F	there exists an entity e s.t. (y,x,e) ∈ K
PROP-TO-ENT-R	there exists an entity e s.t. (e,x,y) ∈ K
Utterance Token	Entity	EXACT-MATCH	x and y are the same word
PARTIAL-MATCH	token x is contained in entity y
Entity	Utterance Token	EXACT-MATCH-R	y and x are the same word
PARTIAL-MATCH-R	token y is contained in entity x
Utterance Token	Property	P-EXACT-MATCH	x and y are the same word
P-PARTIAL-MATCH	token x is contained in property y
Property	Utterance Token	P-EXACT-MATCH-R	y and x are the same word
P-PARTIAL-MATCH-R	token y is contained in property y
Table 2: Relation types used for text-to-LF parsing.
utterance and schema items. Let  $\{\pmb{x}_i^{(0)}\}_{i = 0}^{N - 1}$  denote the input to the parser consisting of NL tokens and the (linearized version of) environment; relation-aware encoding changes the multi-head self-attention (with  $H$  heads and hidden size  $d_{x}$ ) as follows:
$$
e _ {i j} ^ {(n, h)} = \frac {\boldsymbol {x} _ {i} ^ {(n)} W _ {Q} ^ {(n , h)} \left(\boldsymbol {x} _ {j} ^ {(n)} W _ {K} ^ {(n , h)} + \boldsymbol {r} _ {i j} ^ {K}\right) ^ {\top}}{\sqrt {d _ {z} / H}}
$$
$$
\alpha_ {i j} ^ {(n, h)} = \operatorname {s o f t m a x} _ {j} \left\{e _ {i j} ^ {(n, h)} \right\} \tag {15}
$$
$$
\pmb {z} _ {i} ^ {(n, h)} = \sum_ {j = 1} ^ {n} \alpha_ {i j} ^ {(n, h)} (\pmb {x} _ {j} ^ {(n)} W _ {V} ^ {(n, h)} + \pmb {r} _ {i j} ^ {V}).
$$
where  $\alpha_{ij}^{(n,h)}$  denotes the attention weights of head  $h$  at layer  $n$ ,  $0 \leq h < H$ ,  $0 \leq n < N$ , and
$W_{Q}^{(h)}, W_{K}^{(h)}, W_{V}^{(h)} \in \mathbb{R}^{d_{x} \times (d_{x} / H)}$ . Most importantly,  $r_{ij}^{K}$  and  $r_{ij}^{V}$  are key and value embeddings of the discrete relation  $r_{ij}$  between items  $i$  and  $j$ . They are incorporated to bias attention towards discrete relations.
In this work, we re-use the relations from Wang et al. (2020) for our text-to-SQL parsing task on the GEOQUERY dataset. For text-to-LF parsing on the OVERNIGHT dataset, we elaborate on how we define discrete relations. The input of text-toLF parsing is an utterance and a fixed knowledge base  $\kappa$  which is represented as a set of triples in the form of (entity1, property, entity2). To feed the input into a transformer, we first linearize it into an ordered sequence which contains a list of utterance tokens, followed by a sequence of entities and properties. To model the relations among the
items of this sequence, we define relations between each pair of items, denoted by  $(x,y)$ , in Table 2.
Statistics of Datasets The numbers of examples in each domain are shown in Table 3. Four domains of OVERNIGHT and GEOQUERY contain only around 1000 examples.
Results of Supervised Models The results of supervised models (upper bounds) are shown in Table 4. Our parser achieves the best performance among models without cross-domain training. This confirms that we use a strong base parser for our experiments on semi-supervised learning.
Beam Size We try the beam size from  $\{4,8,16,32,64\}$  and finally picks 16 which performs best in the OVERNIGHT dataset. We also try a larger beam size of 128 during preliminary experiments. However, the model is extremely slow to train and does not outperform the one with beam size 16.
Semi-Supervised Learning with  $10\%$  Data We also investigate a semi-supervised setting where only  $10\%$  of the original training data are treated as labeled data. This is more challenging as four domains of OVERNIGHT and GEOQUERY only have around 100 labeled utterance-program pairs in this setting. The results are shown in Table 5. In terms of average accuracy, Sparse MML achieves the best performance. The margin of improvement, compared to the lower bound, is relatively smaller than using  $30\%$  data ( $3\%$  vs  $4.2\%$ ). As all objectives rely on beam search, a weak base parser, as indicated by the lower bound, is probably
BASKETBALL	BLOCKS	CALENDAR	HOUSING	PUBLICATIONS	RECIPES	RESTAURANTS	SOCIAL	GEO
all	1952	1995	837	941	801	1080	1657	4419	880
Table 3: Data sizes of the OVERNIGHT and GEOQUERY dataset.
Model	BASKETBALL	BLOCKS	CALENDAR	HOUSING	PUBLICATIONS	RECIPES	RESTAURANTS	SOCIAL	Avg.
Wang et al. (2015b)	46.3	41.9	74.4	54.5	59.0	70.8	75.9	48.2	58.8
Jia and Liang (2016b)	85.2	58.1	78.0	71.4	76.4	79.6	76.2	81.4	75.8
Herzig and Berant (2017)	85.2	61.2	77.4	67.7	74.5	79.2	79.5	80.2	75.6
Su and Yan (2017)	86.2	60.2	79.8	71.4	78.9	84.7	81.6	82.9	78.2
Ours	87.7	62.9	82.1	71.4	78.9	82.4	82.8	80.8	78.6
Herzig and Berant (2017)*	86.2	62.7	82.1	78.3	80.7	82.9	82.2	81.7	79.6
Su and Yan (2017)*	88.2	62.7	82.7	78.8	80.7	86.1	83.7	83.1	80.8
Table 4: Test accuracy of supervised models on all domains for OVERNIGHT. Models with * are augmented with cross-domain training.
Model	OVERNIGHT	GEO
BASKETBALL	BLOCKS	CALENDAR	HOUSING	PUBLICATIONS	RECIPES	RESTAURANTS	SOCIAL	Avg.
Lower Bound	68.0	35.3	40.5	34.4	39.1	45.8	59.3	63.7	48.3	42.7
Self-Training	66.5	37.8	48.2	39.2	41.0	41.2	63.0	63.8	50.1	44.1
Top-K MML	70.3	36.8	42.3	30.7	37.3	44.4	61.1	62.2	48.1	40.1
Repulsion MML	70.8	37.3	44.6	37.8	39.8	44.4	60.2	67.8	50.3	44.8
Gentle MML	68.3	38.1	45.8	39.2	43.5	44.9	59.9	64.0	50.5	46.6
Sparse MML	73.4	42.1	46.4	38.1	42.2	43.1	63.0	64.9	51.3	45.2
Upper Bound	87.7	62.9	82.1	71.4	78.9	82.4	82.8	80.8	78.6	74.2
Table 5: Execution accuracy of supervised and semi-supervised models on all domains of OVERNIGHT and GEO-QUERY. In semi-supervised learning,  $10\%$  of the original training examples are treated as labeled and the remaining  $90\%$  as unlabeled. Lower bound refers to supervised models that only use labeled examples and discard unlabeled ones whereas upper bound refers to supervised models that have access to gold programs of unlabeled examples. Avg. refers to the average accuracy of the eight OVERNIGHT domains.
harder to improve. By taking account of unseen programs, Repulsion MML and Gentle MML tend to be more useful in domains such as HOUSING, PUBLICATIONS, RECIPIES where the base parsers are very weak (accuracy is around  $40\%$ ). Moreover, Gentle MML performs best in GEOQUERY.