SlowGuess/NAACL_2021_OCR huggingface dataset:
where the top-E-step can be solved by the SparseMax operator, which denotes the Euclidean projection from the vector of logits  $\log p(y|x,\pmb{\theta}^t)$  to the simplex  $\mathcal{Q}$ . Again, we solve the E-step approximately. One of the approximations is to use top-k SparseMax which constrains the number of non-zeros of  $q$  to be less than  $k$ . It can be solved by using a top-k operator and followed by SparseMax (Correia et al., 2020). In our case, we use beam search to approximate the top-k operator and the resulting approximation for the E-step is defined as follows:
$$
q _ {\mathrm {s p a r s e}} ^ {t + 1} = \operatorname {S p a r s e M a x} _ {y \in P _ {\mathrm {S E}}} \left(\log p (y | x, \boldsymbol {\theta} ^ {t})\right)
$$
Intuitively,  $q_{\mathrm{sparse}}^{t + 1}$  occupies the middle ground between Self-Training (uses  $y^*$  only) and Top-K MML (uses all  $P_{\mathrm{SE}}$  programs). With the help of sparsity of  $q$  introduced by SparseMax, the M-step will only promote a subset of  $P_{\mathrm{SE}}$  programs.
Summary We propose three new approximations of MML for learning from executions. They are designed to complement Self-Training and Top-K MML via discouraging seen non-executable programs and introducing sparsity. In the following sections, we will empirically show that they are superior to Self-Training and Top-K MML for semi-supervised semantic parsing. The approximations we proposed may also be beneficial for learning from denotations (Liang et al., 2013; Berant et al., 2013) and weakly supervised question answering (Min et al., 2019), but we leave this to future work.
5 Semantic Parsers
In principle, our X-PR framework is model-agnostic, i.e., it can be coupled with any semantic parser for semi-supervised learning. In this work, we use a neural parser that achieves state-of-the-art performance across semantic parsing tasks. Specifically, we use RAT-SQL (Wang et al., 2020) which features a relation-aware encoder and a grammar-based decoder. The parser was originally developed for text-to-SQL parsing, and we adapt it to text-to-LF parsing. In this section, we briefly review the encoder and decoder of this parser. For more details, please refer to Wang et al. (2020).
5.1 Relation-Aware Encoding
Relation-aware encoding is originally designed to handle schema encoding and schema linking for text-to-SQL parsing. We generalize these two notions for both text-to-LF and text-to-SQL parsing as follows:
- enviroment encoding: encoding enviroments, i.e., a knowledge base consisting of a set of triples; a relational database represented by its schema
- **enviroment linking:** linking mentions to intended elements of environments, i.e., mentions of entities and properties of knowledge bases; mentions of tables and columns of relational databases
Relation-aware attention is introduced to inject discrete relations between environment items, and between the utterance and environments into the self-attention mechanism of Transformer (Devlin et al., 2019). The details of relation-aware encoding can be found in the appendix.
5.2 Grammar-Based Decoding
Typical sequence-to-sequence models (Dong and Lapata, 2016; Jia and Liang, 2016a) treat programs as sequences, ignoring their internal structure. As a result, the well-formedness of generated programs cannot be guaranteed. Grammar-based decoders aim to remedy this issue. For text-to-LF parsing, we use the type-constrained decoder proposed by Krishnamurthy et al. (2017); for text-to-SQL parsing, we use an AST (abstract syntax tree) based decoder following Yin and Neubig (2018). Note that grammar-based decoding can only ensure the syntactic correctness of generated programs. Executable programs are additionally semantically correct. For example, all programs in Figure 1 are
Model	OVERNIGHT	GEO
BASKETBALL	BLOCKS	CALENDAR	HOUSING	PUBLICATIONS	RECIPES	RESTAURANTS	SOCIAL	Avg.
Lower bound	82.2	54.1	64.9	61.4	64.3	72.7	71.7	76.7	68.5	60.6
Self-Training	84.7	52.6	67.9	59.8	68.6	80.1	71.1	77.4	70.3	64.2
Top-K MML	83.1	55.3	68.5	56.9	67.7	73.7	69.9	76.2	68.9	61.3
Repulsion MML	84.9	56.3	70.8	60.9	70.3	79.8	72.0	78.3	71.7	64.9
Gentle MML	84.1	58.1	70.2	63.0	71.5	78.7	72.3	76.4	71.8	65.6
Sparse MML	83.9	58.6	72.6	60.3	75.2	80.6	72.6	77.8	72.7	67.4
Upper Bound	87.7	62.9	82.1	71.4	78.9	82.4	82.8	80.8	78.6	74.2
Table 1: Execution accuracy of supervised and semi-supervised models on all domains of OVERNIGHT and GEO-QUERY. In semi-supervised learning,  $30\%$  of the original training examples are treated as labeled and the remaining  $70\%$  as unlabeled. Lower bound refers to supervised models that only use labeled examples and discard unlabeled ones whereas upper bound refers to supervised models that have access to gold programs of unlabeled examples. Avg. refers to the average accuracy of the eight OVERNIGHT domains. We average runs over three random splits of the original training data for semi-supervised learning.
well-formed, but the first two programs are semantically incorrect.
To evaluate X-PR, we present experiments on semi-supervised semantic parsing. We also analyze how the objectives affect the training process.
6.1 Semi-Supervised Learning
We simulate the setting of semi-supervised learning on standard text-to-LF and text-to-SQL parsing benchmarks. Specifically, we randomly sample  $30\%$  of the original training data as the labeled data, and use the rest  $70\%$  as the unlabeled data. For text-to-LF parsing, we use the OVERNIGHT dataset (Wang et al., 2015a), which has eight different domains, each with a different size ranging between 801 and 4,419; for text-to-SQL parsing, we use GEOQUERY (Zelle and Mooney, 1996) which contains 880 utterance-SQL pairs. The semi-supervised setting is very challenging as leveraging only  $30\%$  of the original training data would result in only around 300 labeled examples in four domains of OVERNIGHT and also in GEOQUERY.
Supervised Lower and Upper Bounds As baselines, we train two supervised models. The first one only uses the labeled data (30% of the original training data) and discards the unlabeled data in the semi-supervised setting. We view this baseline as a lower bound in the sense that any semi-supervised method is expected to surpass this. The second one has extra access to gold programs for the unlabeled data in the semi-supervised setting, which means it uses the full original training data. We view this baseline as an upper bound for semi-supervised learning; we cannot expect to approach it as the
executability signal is much weaker than direct supervision. By comparing the performance of the second baseline (upper bound) with previous methods (Jia and Liang, 2016b; Herzig and Berant, 2017; Su and Yan, 2017), we can verify that our semantic parsers are state-of-the-art. Please refer to the Appendix for detailed comparisons. Our main experiments aim to show how the proposed objectives can mitigate the gap between the lower- and upper-bound baselines by utilizing  $70\%$  unlabeled data.
Semi-Supervised Training and Tuning We use stochastic gradient descent to optimize Equation (1). At each training step, we sample two batches from the labeled and unlabeled data, respectively. In preliminary experiments, we found that it is crucial to pre-train a parser on supervised data alone; this is not surprising as all of the objectives for learning from execution rely on beam search which would only introduce noise with an untrained parser. That is,  $\lambda$  in Equation (1) is set to 0 during initial updates, and is switched to a normal value afterwards.
We leave out 100 labeled examples for tuning the hyperparameters. The hyperparameters of the semantic parser are only tuned for the development of the supervised baselines, and are fixed for semi-supervised learning. The only hyperparameter we tune in the semi-supervised setting is the  $\lambda$  in Equation (1), which controls how much the unsupervised objective influences learning. After tuning, we use all the labeled examples for supervised training and use the last checkpoints for evaluation on the test set.
(a) Average Ratios.
(b) Coverage of gold programs.
Figure 3: Effect of different learning objectives in terms of average ratios and coverage (view in color).
Our experiments evaluate the objectives presented in Figure 2 under a semi-supervised learning setting. Our results are shown in Table 1.
Self-Training and Top-K MML First, Top-K MML, which exploits more executable programs than Self-Training, does not yield better performance in six domains of OVERNIGHT and GEOQUERY. This observation is consistent with Min et al. (2019) where Top-K MML underperforms Self-Training for weakly-supervised question answering. Self-Training outperforms the lower bound in five domains of OVERNIGHT, and on average. In contrast, Top-K MML obtains a similar performance to the lower bound in terms of average accuracy.
X-PR Objectives In each domain of OVERNIGHT and GEOQUERY, the objective that achieves the best performance is always within X-PR. In terms of average accuracy in OVERNIGHT, all our objectives perform better than Self-Training and Top-K MML. Among X-PR, Sparse MML performs best in five domains of OVERNIGHT, leading to a margin of  $4.2\%$  compared with the lower bound in terms of average accuracy. In GEOQUERY, Sparse MML also obtain best performance.
Based on the same intuition of discouraging seen non-executable programs, Repulsion MML achieves a similar average accuracy to Gentle MML in OVERNIGHT. In contrast, Gentle MML tends to perform better in domains whose parsers are weak (such as HOUSING, BLOCKS) indicated by their lower bounds. In GEOQUERY, Gentle MML performs slightly better than Repulsion MML. Although it does not
perform better than Repulsion MML, it retrieves more accurate programs and also generates longer programs (see next section for details).
To see how much labeled data would be needed for a supervised model to reach the same accuracy as our semi-supervised models, we conduct experiments using  $40\%$  of the original training examples as the labeled data. The supervised model achieves  $72.6\%$  on average in OVERNIGHT, implying that 'labeling'  $33.3\%$  more examples would yield the same accuracy as our best-performing objective (Sparse MML).
To better understand the effect of different objectives, we conduct analysis on the training process of semi-supervised learning. For the sake of brevity, we focus our analysis on the CALENDAR domain but have drawn similar conclusions for the other domains.
Length Ratio During preliminary experiments, we found that all training objectives tend to favor short executable programs for unlabeled utterances. To quantify this, we define the metric of average ratio as follows:
$$
\text {r a t i o} = \frac {\sum_ {i} \sum_ {y \in P _ {\mathrm {S E}} \left(x _ {i}\right)} | y |}{\sum_ {i} \left| x _ {i} \right| \left| P _ {\mathrm {S E}} \left(x _ {i}\right) \right|} \tag {9}
$$
where  $P_{\mathrm{SE}}(x_i)$  denotes seen executable programs of  $x_i$ ,  $|x|, |y|$  denotes the length of an utterance and a program, respectively, and  $|P_{\mathrm{SE}}(x_i)|$  denotes the number of seen executable programs. Intuitively, average ratio reveals the range of programs that an objective is exploiting in terms of length. This metric is computed in an online manner, and  $x_i$  is a sequence of data fed to the training process.
As shown in Figure 3a, Top-K MML favors shorter programs, especially during the initial steps. In contrast, Repulsion MML and Gentle MML prefer longer programs. For reference, we can compute the gold ratio by assuming  $P_{\mathrm{SE}}(x_i)$  only contains the gold program. The gold ratio for Cal-ENDER is 2.01, indicating that all objectives are still preferring programs that are shorter than gold programs. However, by not directly exploiting seen executable programs, Repulsion MML and Gentle MML alleviate this issue compared with Top-K MML.
Coverage Next, we analyze how much an objective can help a parser retrieve gold programs for unlabeled data. Since the original data contains the gold programs for the unlabeled data, we utilize them to define the metric of coverage as follows:
$$
\text {c o v e r a g e} = \frac {\sum_ {i} I \left[ \hat {y} _ {i} \in P _ {\mathrm {S E}} \left(x _ {i}\right) \right]}{\sum_ {i} \left| x _ {i} \right|} \tag {10}
$$
where  $I$  is an indicator function,  $\hat{y}_i$  denotes the gold program of an utterance  $x_i$ . Intuitively, this metric measures how often a gold program is captured in  $P_{\mathrm{SE}}$ . As shown in Figure 3b, Self-Training, which only exploits one program at a time, is relatively weak in terms of retrieving more gold programs. In contrast, Repulsion MML retrieves more gold programs than the others.
As mentioned in Section 4.3, SparseMax can be viewed as an interpolation between Self-Training and Top-K MML. This is also reflected in both metrics: Sparse MML always occupies the middle-ground performance between ST and Top-K MML. Interestingly, although Sparse MML is not best in terms of both diagnostic metrics, it still achieves the best accuracy in this domain.
In this work, we propose to learn a semi-supervised semantic parser from the weak yet freely available executability signals. Due to the large search space of executable programs, conventional approximations of MML training, i.e., Self-Training and Top-K MML, are often sub-optimal. We propose a set of alternative objectives, namely X-PR, through the lens of posterior regularization. Empirical results on semi-supervised learning show that X-PR can help a parser achieve substantially better performance than conventional methods, further bridging the gap between semi-supervised learning and supervised learning. In the future, we
would like to extend X-PR to related tasks such as learning from denotations and weakly supervised question answering.
Acknowledgements We would like to thank the anonymous reviewers for their valuable comments. We gratefully acknowledge the support of the European Research Council (Titov: ERC StG Broad-Sem 678254; Lapata: ERC CoG TransModal 681760) and the Dutch National Science Foundation (NWO VIDI 639.022.518).
Massih-Reza Amini and Patrick Gallinari. 2002. Semi-supervised logistic regression. In ECAI, pages 390-394.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. 2018. Discourse-aware neural rewards for coherent text generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 173-184, New Orleans, Louisiana. Association for Computational Linguistics.
Olivier Chapelle and Alexander Zien. 2005. Semisupervised classification by low density separation. In AISTATS, volume 2005, pages 57-64. CiteSeer.
Xinyun Chen, Chang Liu, and Dawn Song. 2019. Execution-guided neural program synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Gonçalo M. Correia, Vlad Niculae, Wilker Aziz, and André F. T. Martins. 2020. Efficient marginalization of discrete and structured latent variables via sparsity. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany. Association for Computational Linguistics.
Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11:2001-2049.
Yves Grandvalet and Yoshua Bengio. 2004. Semisupervised learning by entropy minimization. In Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pages 529-536.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Jonathan Herzig and Jonathan Berant. 2017. Neural Semantic Parsing over Multiple Knowledge-bases. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 623-628, Stroudsburg, PA, USA.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016a. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016b. Data Recombination for Neural Semantic Parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Stroudsburg, PA, USA.
Tomáš Kocisky, Gábor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. 2016. Semantic parsing with semi-supervised sequential autoencoders. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1078-1087, Austin, Texas. Association for Computational Linguistics.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516–1526, Copenhagen, Denmark. Association for Computational Linguistics.
Percy Liang, Michael I. Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389-446.
Andre F. T. Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1614-1623. JMLR.org.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2851-2864, Hong Kong, China. Association for Computational Linguistics.
Vlad Niculae, André F. T. Martins, Mathieu Blondel, and Claire Cardie. 2018. Sparsemap: Differentiable sparse structured inference. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3796-3805. PMLR.
Yu Su and Xifeng Yan. 2017. Cross-domain semantic parsing via paraphrasing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1235-1246, Copenhagen, Denmark.
Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pages 1057-1063. CiteSeer.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3774-3785, Hong Kong, China. Association for Computational Linguistics.
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. arXiv preprint arXiv:1807.03100.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015a. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015b. Building a Semantic Parser Overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1332-1342, Stroudsburg, PA, USA.
Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296-1306, Austin, Texas. Association for Computational Linguistics.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7-12, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018. StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 754–765, Melbourne, Australia. Association for Computational Linguistics.
MML vs. RL Another choice for learning from executions is to maximize the expected reward:
$$
\mathcal {J} _ {\mathrm {R L}} (x, \boldsymbol {\theta}) = \mathbb {E} _ {p (y | x, \boldsymbol {\theta})} [ R (y) ] \tag {11}
$$
Recall that our MML objective is defined as:
$$
\mathcal {J} _ {\mathrm {M M L}} (x, \boldsymbol {\theta}) = \log \sum_ {y} R (y) p (y | x, \boldsymbol {\theta}) \tag {12}
$$
As shown in previous work (Guu et al., 2017), the gradients (wrt.  $\theta$ ) of RL and MML have the same form:
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {J} (x, \boldsymbol {\theta}) = \sum_ {y} q (y) \nabla_ {\boldsymbol {\theta}} \log p (y | x, \boldsymbol {\theta}) \tag {13}
$$