SlowGuess/ACL_2020_OCR huggingface dataset:
Test Against Unseen Problems
Constraint	B=1	B=10	B=102	B=103
None	0.0%	3.0%	11.5%	21.8%
Previous	17.8%	28.4%	34.2%	38.3%
Syntactic	27.5 %	35.4%	42.1%	47.8%
SymTable	31.0%	39.2	46.0%	49.3%
Backoff	31.2%	39.4%	46.1%	49.6%
Table 1: Comparison of the fraction of program passed when  $B = 10^{0,1,2,3}$  under different constraints; constraint satisfied by hierarchical beam search with the default hyper-parameters mentioned in Section 5. "Previous" refers to the previous state of the art model.
Figure 5(d), the lead of SymTable on Syntactic grows linearly: the more these two algorithms search, the more budget is needed by Syntactic to reach the same level as SymTable. Syntactic needs nearly 600 more budget to have comparable performance with SymTable that uses 400 budget.
We notice that all of our constrained search methods outperform the previous state-of-the-art. Averaged across all test examples, Backoff can solve  $55.1\%$  of the problems within 100 budget, which is  $\approx 10\%$  higher than the previous work. On unseen workers (problems), the top 11 (top 52) candidates of Backoff solve the same fraction of problems as the top 3000 candidates of the best performing algorithm in Kulal et al. (2019).
6.3 Regular vs. Hierarchical Beam Search
We use regular beam search with beam width  $W = 200$  to generate  $B = 100$  valid candidate full programs. We did not experiment with  $B = 1000$  because beam search with  $W \geq B \geq 1000$  is computationally intractable. For hierarchical beam search we experiment with  $W = 10, 25, 50$  for scaffold search and keep the top  $K = \min(W, 20)$  scaffolds for subsequent searches.
Table 2 compares the performance of hierarchical beam search against regular beam search with different beam sizes under Syntactic and SymTable constraints. We find that if hierarchical beam search is used, even dropping the beam width
Test Against Unseen Workers, Syntactic
Method, Width	B=1	B=10	B=102
H, W=10	42.8%	51.7%	59.1%
H, W=25	42.8%	51.8%	59.3%
H, W = 50	42.8%	51.9%	59.3%
R, W=200	42.4%	51.3%	58.2%
Test Against Unseen Workers, SymTable
Method, Width	B=1	B=10	B=102
H, W=10	45.4%	54.3%	61.0%
H, W=25	45.6%	54.7%	61.9%
H, W = 50	45.8%	55.1%	62.6%
R, W=200	45.6%	54.9%	61.9%
Table 2: Comparison of different beam size with Syntactic and SymTable constraint when tested against unseen workers. H/R refers to hierarchical/regular beam search and  $W$  is the beam width. The same results on unseen problems can be seen in appendix.
from 50 to 10 leads to negligible change in performance. In contrast, even with a large beam width  $W = 200$ , regular beam search method cannot efficiently search for the solution and leads to a noticeable drop in performance.
We observe a similar trend for SymTable: regular beam search with beam width  $W = 200$  underperforms hierarchical search with beam width  $W = 25$ . However, if we further decrease the hierarchical beam search width from 25 to 10 in this setting, we observe a significant drop in performance, possibly because there are more variable usage variations than syntactic variations.
6.4 Scaffold Search vs. Brute Force Method
We now compare scaffold search to the brute force algorithm as described in section 4.3. We make  $B = 50,000$  attempts for the brute force method so that its performance can match at least the top 10 candidates of our constrained approach and make the lead metrics meaningful. To save computation and avoid compiling all 50,000 programs, we early reject every candidate that does not fulfill our constraints.
The lead of our approaches against the brute force algorithm is shown in Figure 6. After being adjusted for the constraint checking quota used, the lead of our approach is tens of thousands ahead of the unconstrained approach. Scaffold search saves lot of computation by inducing a little overhead earlier in the search process.
Figure 6: Lead of SymTable and Syntactic constraints on non-constrained approach with equal quota on test set with unseen (a) workers and (b) problems.
7.1 Program Candidate Variations
Beam search has the problem of producing fewer variations at the beginning of the search. Such a weakness might be tolerable if we only care about the top 1 candidate, but becomes disastrous in a search setting where we want the top  $B$  candidates, whose variation is typically spread across the entire program.
We describe the following procedure to formally define this intuition. We first aggregate code piece choices for each line for all the top  $B$  programs. As shown in Figure 8(a), we construct a matrix such that each column corresponds to a full program candidate; the number  $r$  in the  $i^{th}$  row and  $j^{th}$  column means that on line  $i$ , the  $j^{th}$  full program candidate chooses the  $r^{th}$  code piece candidate (i.e.  $y_{ic_i} = y_{ir}$ ). Then we can build a prefix tree (Figure 8(b)) by treating each column as a string, where each traversal from the root to a leaf is a complete candidate program  $y$ . We define the representative branch/program as a traversal from the root to a leaf that always chooses the child that contains the most leaves (with ties being broken randomly). For each of the remaining  $B - 1$  programs/traversals, we find the smallest line number where it starts to diverge from the representative branch. Among these  $B - 1$  programs, we count the fraction of divergences that take place in the first/second half of the lines. For example, in Figure 8(b),  $0\%$  of the divergences occur in the first half.
We compare hierarchical vs. regular beam search under syntactic constraints with different beam widths  $W$ : hierarchical  $W = 10,50$  and regular  $W = 50,200$ . We group the programs by length  $L$ , consider the top  $B = 25$  attempted programs for each problem and report the fraction of divergences that occur in the first half of the program length for each group.
Length L	H 10	H 50	R 50	R 200
(0, 10]	45.4%	45.5%	43.6%	45.5%
(10, 20]	63.2%	63.4%	58.2%	63.4%
(20, 30]	63.6%	63.6%	56.8%	63.6%
(30, 40]	67.2%	67.3%	58.2%	67.3%
(40,∞)	69.4%	68.8%	56.8%	68.8%
Table 3: Fraction of divergence in the first half of the program, grouped by program length  $L$ . In the column headers, H/R represents Hierarchical/Regular beam search under Syntactic constraint, and the number represents beam width  $W$ . The column with the lowest fraction is underlined.
The results can be seen in Table 3. For regular beam search, a moderate beam width  $W = 50$  consistently brings fewer variations in the first half of the program, and it needs a larger  $W = 200$  to fix this problem. In contrast, a small  $W$  for hierarchical beam search produces the same amount of variations in the first half of the program. The same statistics under SymTable constraints can be seen in the appendix (Table 5) and the conclusion holds similarly.
7.2 Rejection by Constraints
In this section we give representative examples on what program candidates are rejected by our syntactic and symbol table constraints.
Syntactic Constraints As mentioned in Section 5, about  $26\%$  of the lines do not have pseudocode. They may correspond to “}”, “int main() {”, “{”, “return 0”, “}”; or “;”. These lines need contextual information to select valid code pieces and naively combining the top 1 candidate from each line independently will always produce grammatically invalid programs. Syntactic constraints also rule out stylistic ambiguities. For example, when there is only one statement within an if statement, the programmer can optionally include a curly brace. However, the pseudocode does not contain such detailed information about style. Both “if (...) {” and “if (...)” might be valid, but only one of them can be correct given the context of a program. Our syntactic constraints, which contain a curly brace constraint, can help us select the right code piece.
Symbol Table (SymTable) Constraints Pseudocode annotations are sometimes implicit about variable declarations. Given the instruction “set N to 222222”, both code pieces (1) “int N =
Reason (percentage)	Pseudocode	Gold Solution	Model Generation
(a) Generation wrong (47.5%)	let value1, value2, val, a, b be integers with val = 0	int value1, value2, val, a, b = 0;	int value1, value2, val = 0, a, b;
(b) Needs type disambiguation (12.5%)	s[occur[i][j] + k] = letter - a + A	s[occur[i][j] + k] = letter - 'a' + 'A';	s[occur[i][j] + k] = 'letter' - a + A;
(c) Needs syntax disambiguation (0.5%)	else if dB is less than dW	} else if (dB < dW)	else if (dB < dW)
(d) Variable name typos (15.0%)	if lfg = 1	if (flg == 1) {	if (lfg == 1) {
(e) Pseudocode wrong (23.5%)	set ans = 25*length of s	ans += (25 * s.length());	int ans = 25 * s.length();
Figure 7: Categorized error analysis for lines that no generated code piece is functionally equivalent to the gold. The percentage in the parentheses refers to the fraction of this category out of the 200 samples.
Figure 8: (a) A matrix that represents each candidate's choices of code pieces for each line. (b) A prefix tree constructed by treating each column as a string; the representative branch is the second column and marked with red color.
222222;” and (2)  $\mathrm{N} = 222222$  ;” are potentially valid. We might disambiguate this case with a SymTable constraint: if the variable is declared before in the same scope, then we know this code piece should not contain a repeated declaration and hence we should choose candidate (2); otherwise we should choose (1) to avoid using undeclared variables. SymTable constraints are also helpful when the pseudocode does not put quotation marks around string/character literals. Consider the instruction "if lucky is A then do the following" with the ground truth code piece "if (lucky  $= =$  'A') {"}. The model might misunderstand A as a variable name and generate "if (lucky  $= =$  A) {"}. This error can be ruled out by SymTable constraint if variable A is undeclared.
However, SymTable constraints do not preclude all errors related to declarations. Consider the following generation where the last line is wrong:
A programmer will usually not declare new variables in the last line of a variable scope. However, technically this is not an invalid statement and the SymTable constraint fails to reject this wrong candi
date. Extra modelling is needed to take into account programming conventions and common sense.
7.3 Code Piece Error Analysis
So far we have focused on combining independent candidates from each line together to search for the target program. This heavily depends on the underlying model to generate potentially correct code pieces. However, in  $32\%$  of the programs at least one "hard" line has no generated code piece that is functionally equivalent to the solution, thus indicating plenty of room for improvement. To help the readers understand the bottleneck for code piece generation and point out important future directions, we randomly sampled 200 "hard" lines and manually analyzed why the generation fails by looking at the top 1 candidate of the model. The error analysis is available on our GitHub.
We group the failures into the following categories, giving a detailed breakdown and examples in Figure 7. (a) The model generation is wrong despite clear pseudocode; this typically happens when the gold code piece is long or highly compositional. (b, c) The pseudocode contains ambiguity; the model generation is reasonable but either needs (b) variable type clarification or (c) syntactic context. This requires incorporating contextual information of the program into the code piece generation process. (d, e) The pseudocode either (d) consists of variable name typos or (e) is completely wrong.
Alfred V Aho, Ravi Sethi, and Jeffrey D Ullman. 1986. Compilers, principles, techniques. Addison wesley, 7(8):9.
Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In Proceedings
of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pages 2123-2132. JMLR.org.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Xinyun Chen, Chang Liu, Richard Shin, Dawn Song, and Mingcheng Chen. 2016. Latent attention for if-then program synthesis. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pages 4581-4589, USA. Curran Associates Inc.
Keith Ellul, Bryan Krawetz, Jeffrey Shallit, and Mingwei Wang. 2005. Regular expressions: New results and open problems.
Carlo Ghezzi and Dino Mandrioli. 1979. Incremental parsing. ACM Transactions on Programming Languages and Systems (TOPLAS), 1(1):58-70.
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1-119.
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig. 2018. Retrieval-based neural code generation. arXiv preprint arXiv:1808.10025.
Srinivasan Iyer, Alvin Cheung, and Luke Zettlemoyer. 2019. Learning programmatic idioms for scalable semantic parsing. arXiv preprint arXiv:1904.09086.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588.
G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. Spoc: Search-based pseudocode to code. arXiv preprint arXiv:1906.04908.
Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomás Kočisky, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017a. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07535.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017b. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the
55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139-1149, Vancouver, Canada. Association for Computational Linguistics.
Armando Solar-Lezama. 2009. The sketching approach to program synthesis. In Proceedings of the 7th Asian Symposium on Programming Languages and Systems, APLAS '09, pages 4-13, Berlin, Heidelberg. Springer-Verlag.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Chenglong Wang, Po-Sen Huang, Alex Polozov, Marc Brockschmidt, and Rishabh Singh. 2018. Execution-guided neural program decoding. CoRR, abs/1807.03100.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In The 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada.
Maksym Zavershynskyi, Alex Skidanov, and Illia Polosukhin. 2018. Naps: Natural program synthesis dataset. arXiv preprint arXiv:1807.03168.
A.1 Primary Expressions
Table 6 contains the grammar we use for the syntactic constraint and Table 7 defines the generation of terminal symbols.
A.2 CFG Description Size of SymTable
We show that we cannot specify the SymTable constraint in a context free grammar without exponential description complexity w.r.t. the number of variables declared. The intuition is that, since repeated declarations of a variable are not allowed, we need to keep track of all the variables that have been declared every time when verifying whether the next line is valid; however, a CFG, when transformed into a pushdown automata, is only allowed to peek at the top of the stack to decide the state transition. This means the symbol on the top of the stack, the state, or the transition rule need to have full information of about whether each variable has been declared, which contains exponentially many possibilities w.r.t. the number of variables.
Our proof is an adaptation of Ellul et al. (2005), which proves this property for the language that accepts all the permutations of a fixed number of variables. We refer the readers to this paper if more details of the proof are needed. To formalize, we consider a simple grammar of  $K$  characters  $\{v_{1},\ldots ,v_{K}\}$ , where  $v_{i}$  means, semantically, declaring the variable  $v_{i}$ , and the language  $L$  consists of all the possible sequences of declarations that have no repetition.
$$
L = \left\{\operatorname {c o n c a t} _ {j = 1} ^ {k} v _ {a _ {j}} \mid a _ {j _ {1}} \neq a _ {j _ {2}} \text {i f} j _ {1} \neq j _ {2}, k \leq K \right\} \tag {5}
$$
We prove that
Theorem 1  $L$  has at least  $\tilde{\Omega}(1.37^K)$  description complexity $^4$  as a context-free grammar.
Intuitively, it means if we want to use a CFG to specify  $L$ , we need the sum of total length of the production rules and number of symbols to be at least exponential.
Proof: Since we can convert any CFG with size  $B$  to Chomsky Normal Form (CNF) with size  $O(B^{2})$ , the above statement would be implied if we prove that  $L$  needs  $\tilde{\Omega}(1.37^{2K}) = \tilde{\Omega}(1.89^{K})$  description size in Chomsky Normal Form.
We use Lemma 31 from Ellul et al. (2005):
Lemma 2 Let  $S$  be the start symbol of the CFG. Then for all  $w \in L$ , there exists a symbol  $A$  with
$$
S \Longrightarrow^ {*} \alpha A \beta \Longrightarrow^ {*} w \tag {6}
$$
such that if  $A$  yields  $y$  in  $w$  (i.e.  $w = \alpha y\beta$ ),  $\frac{1}{3}|w| \leq |y| \leq \frac{2}{3}|w|$ .
In other words, for any member of the language, we can find a symbol in the derivation responsible for between 1/3 and 2/3 of the final yield.
Let  $P_K$  be all sequences of permutations of the  $K$  variables and thus  $P_K \subset L$ . Then by Lemma 2, for every permutation  $\pi \in P_K$  we can find yield  $y_{\pi}$  that is yielded by a single symbol such that  $\frac{1}{3} K \leq |y_{\pi}| \leq \frac{2}{3} K$ . Now we consider two permutations  $\pi_1$  and  $\pi_2$ . If  $y_{\pi_1}$  and  $y_{\pi_2}$  are yielded by the same symbol, then they must have the same length (this is the part where the proof is slightly different from Ellul et al. (2005)): suppose the contrary, w.l.o.g., let  $|y_{\pi_1}| > |y_{\pi_2}|$ . By the definition of a context free grammar, we can replace the sub-string  $y_{\pi_2}$  in  $\pi_2$  by  $y_{\pi_1}$  to create a new string  $y_{\pi_2}'$  which is still a member of  $L$ . We have  $|y_{\pi_2}'| = K - |y_{\pi_2}| + |y_{\pi_1}| > K$  by assumption. However, there are in total  $K$  variables; by the pigeonhole principle there must be a variable that is declared twice, and hence  $y_{\pi_2}' \notin L$  and we obtain a contradiction.
Then all the assumption needed by Theorem 30 in Ellul et al. (2005) hold and  $L$  has description complexity  $\tilde{\Omega}(1.89^{K})$  in CNF and hence  $L$  has description complexity  $\tilde{\Omega}(1.89^{K/2}) = \tilde{\Omega}(1.37^{K})$ .
A.3 Hardness of Satisfying SymTable
We show that combining code pieces from each line under the SymTable constraint is NP-Hard in general. We first remind the readers of the set packing problem:
Definition 3 Assume the universe to be  $\mathcal{V}$ , and suppose we are given a family of subsets  $\mathcal{S}$  from the power set of  $\mathcal{V}$ , i.e.  $P(\mathcal{V}) = \{S \mid S \subseteq \mathcal{V}\}$  and  $\mathcal{S} \subseteq P(\mathcal{V})$ . We want to determine whether we can find a packing  $\mathcal{K} \subseteq \mathcal{S}$  for which all sets in  $\mathcal{K}$  are pairwise disjoint and with size  $|\mathcal{K}| \geq L$  for some fixed  $L > 0$ . This problem is called the set packing problem, and is known to be NP-complete.
Following the notation in section A.2, for each line  $l \in [L]$ , we construct the  $C = |\mathcal{S}|$  code piece candidates  $y_{lS}$  for  $S \in \mathcal{S}$  as
$$
y _ {l S} = \operatorname {c o n c a t} _ {v \in S} v. \tag {7}
$$
$\tilde{\Omega}$  ignores all the poly  $(K)$  multiplicative factors.
Test Against Unseen Problems, Syntactic
Method, Width	B=1	B=10	B=102
H, W=10	27.4%	35.3%	42.0%
H, W=25	27.5%	35.4%	42.1%
H, W=50	27.5%	35.4%	42.1%
R, W=200	27.1%	34.7%	41.0%
Test Against Unseen Problems, SymTable
Method, Width	B=1	B=10	B=102
H, W=10	30.3%	38.1%	43.1%
H, W=25	30.9%	39.2%	45.7%
H, W=50	31.0%	39.2%	45.9%
R, W=200	30.7%	38.9%	45.4%
Table 4: Comparison of different beam size with Symtactic and SymTable constraint when tested against unseen problems. H/R refers to hierarchical/regular beam search and  $W$  is the beam width. This table is structured similarly as 2 .
Length L	H 25	H 50	R 50	R 200
(0, 10]	40.7%	41.5%	39.4%	41.5%
(10, 20]	60.9%	59.8%	54.3%	61.3%
(20, 30]	62.2%	61.3%	54.2%	61.3%
(30, 40]	66.0%	66.1%	56.8%	66.1%
(40,∞)	69.0%	68.7%	57.9%	68.7%
Table 5: Fraction of divergence in the first half of the program, grouped by program length  $L$ . In the column headers, H/R represents Hierarchical/Regular beam search under SymTable constraint, and the number represents beam width  $W$ .
We easily see that there is a set packing of size  $L$  if and only if there is a valid code piece combination under SymTable constraint (declarations need to be disjoint for each line). Hence we finish our reduction proof.  $\square$
A.4 Beam Search on Unseen Problems
Table 4 contains similar information as in Table 2, except that the results are obtained on testing with unseen problems. The exact same conclusion holds: for regular beam search, small beam size hurts performance, but hierarchical beam search can solve this problem.
A.5 Variation under SymTable Constraints
Table 5 contains similar information as Table 3, but for SymTable constraints. The same trend holds: regular beam search with small beam size have fewer variations in the first half of the program.
Symbol	Production Rule
program	stmt program
function program
stmt	for stmt
if stmt
while stmt
dowhile stmt
terminal stmt;
X*	X*X
X
< EMPTY >
function	return_type function_name (args) {start stmt*}end
return_type function_name (type*);
args	< EMPTY >
arg, args
arg	type arg_name
forstmt	forstart terminal_parentheses terminalstmtend;
forstart terminal_parentheses {stmt*}end
whilestmt	whilestart terminal_parentheses terminal stmtend;
whilestart terminal_parentheses {stmt*}end
dowhilestmt	do start {stmt*} while terminal_parenthesesend;
do start terminal stmt while terminal_parenthesesend;
ifstmt	single if stmt elif stmt* else stmt
single if stmt elif stmt*
single if stmt	if start terminal_parentheses terminal stmtend;
if start terminal_parentheses {stmt*}end
elif stmt	elif start terminal_parentheses terminal stmtend;
elif start terminal_parentheses {stmt*}end
else stmt	else start terminal stmtend;
else start {stmt*}end
Table 6: The full primary expression grammar we are using. Each line is a production rule.  $X$  is a generic symbol.
Terminal	Implementation
terminal_parentheses	a string that has matching parentheses and starts with parentheses
terminalstmt	a string that does not contain “;”, “for”, “if”, “else”, “while”, “do”
for, if, else, while, do	reserved key words
function_name, arg_name	function name and function argument name
return_type, type	type in C++
Table 7: The definition of the terminals appearing in Table 6