SlowGuess/NAACL_2022_OCR huggingface dataset:
Parameteric Model For the CSL parametric model, we selected a learning rate from [0.01, 0.05, 0.1]. We selected a number of context states  $|S|$  from [32, 64], except for SCAN where we analyzed a larger number of context states on the MCD splits, as discussed in Appendix C.2. For SCAN we selected  $|S| = 2$  for all splits, except for MCD1 and MCD3, where we found  $|S| = 4$  to give more consistent performance on the dev sets. This was the only case where we used different hyperparameters for different splits of the same dataset.
Sampling We sampled 100,000 synthetic examples for all datasets. We provide some analysis of the effect of this in Appendix C.7. For COGS, we biased sampling to increase the number of longer examples (as discussed in § 3.2) by setting the bias  $\delta = 6$ , and otherwise used  $\delta = 0$ . We limited the maximum recursion depth to 5 for SCAN, 10 for SMCalFlow, and 20 for GeoQuery and COGS.
T5 Fine-Tuning We started with the same configuration for fine-tuning T5 as Shaw et al. (2021). We similarly selected a learning rate from  $[1e^{-3}, 1e^{-4}, 1e^{-5}]$  for each dataset. We use learning rate of  $1e^{-3}$  for SCAN and SMCalFlow and  $1e^{-4}$  for GeoQuery and COGS.
B.5 Training Details
We train the CSL model on 8 V100 GPUs, which takes less than 1.5 hours for all splits. We fine-tune
T5 on 32 Cloud TPU v3 cores $^{21}$  for 10,000 steps, which takes less than 6 hours for all splits.
C.1 Performance Breakdown
We extend the performance breakdown analysis of §4.4 to all splits, with results reported in Table 7.
C.2 Varying Context Sensitivity
Varying the number of context states  $|S|$  can vary the degree of context sensitivity in the CSL model. This can be important because we want our model to be able to accurately model  $p(y|x)$ , which we assume is shared between the source and target distributions, but we also want to sample new inputs  $x$  that may have low probability under the source distribution due to the novel compositions they contain.
As a step towards understanding the trade-offs related to context sensitivity, we compute the average  $\log p(x,y)$  and  $\log p(y|x)$  according to CSL models with different number of context clusters  $|\mathcal{S}|$ . The results are reported in Table 8. The results also let us compare  $\log p(x) = \log p(x,y) - \log p(y|x)$ .
A constraint on the number of context states  $|S|$  is in some ways similar to a constraint on the number of nonterminal symbols in a conventional SCFG. Notably, for SCAN, writing a SCFG that unambiguously maps inputs to outputs requires 2 unique nonterminal symbols, and we observe that, similarly,  $|S| \geq 2$  is required to reach  $100\%$  accuracy on the dev set. We also observe that while the models with larger  $|S|$  fit the training set better, the log likelihood of the dev sets is highest with  $|S|$  in the range between 2 and 4, indicating that the optimal place on the tradeoff curve is not at the extremes. We also note that there is some variance across the different splits for the optimal number of types, with some values leading to less than optimal modeling of  $p(y|x)$ .
It is also worth noting that, regardless of the number of context states, the structural conditional independence assumptions in our model can be too strong, harming the accuracy of modeling  $p(y|x)$ . For example, consider a rule for coordination,  $\mathrm{NT} \rightarrow \langle \mathrm{NT}_{[1]} \text{ and } \mathrm{NT}_{[2]}, \mathrm{NT}_{[1]} \wedge \mathrm{NT}_{[2]} \rangle$ . In our model, we cannot condition the expansion of  $\mathrm{NT}_{[2]}$  on the corresponding expansion of  $\mathrm{NT}_{[1]}$  or the parent context in which the coordination rule
$^{21}$ https://cloud.google.com/tpu/
Dataset	%xCSL	x ∈ xCSL	x∉xCSL	All
T5	CSL	Aug.	T5	CSL	Aug.	T5	CSL	Ens.	Aug.
SCAN Jump	100.0	99.5	100.0	99.7	—	—	—	99.5	100.0	100.0	99.7
SCAN Left	100.0	62.0	100.0	100.0	—	—	—	62.0	100.0	100.0	100.0
SCAN Length	100.0	14.4	100.0	99.2	—	—	—	14.4	100.0	100.0	99.2
SCAN MCD	100.0	15.4	100.0	99.4	—	—	—	15.4	100.0	100.0	99.4
COGS Gen.	99.9	89.8	99.6	99.5	40.9	0.0	100.0	89.8	99.5	99.5	99.5
GeoQuery Std.	76.3	97.2	97.3	98.1	78.9	0.0	77.9	92.9	74.3	93.0	93.3
GeoQuery Templ.	61.0	93.1	96.6	97.1	71.6	0.0	76.9	84.8	58.9	86.9	89.3
GeoQuery TMCD	44.3	88.4	90.3	93.9	53.8	0.0	59.9	69.2	39.9	70.0	74.9
GeoQuery Length	29.0	51.2	91.6	83.6	35.4	0.0	61.3	40.0	26.6	51.7	67.8
SMCalFlow-CS 8-S	30.5	96.0	85.6	95.5	79.8	0.0	78.3	84.7	26.1	81.6	83.5
SMCalFlow-CS 8-C	6.6	52.3	79.6	72.7	33.4	0.0	50.1	34.7	5.3	36.5	51.6
SMCalFlow-CS 16-S	29.5	95.9	85.6	95.9	80.1	0.0	78.2	84.7	25.2	81.7	83.4
SMCalFlow-CS 16-C	11.6	59.7	84.4	84.4	42.7	0.0	58.4	44.7	9.8	47.5	61.4
SMCalFlow-CS 32-S	30.4	96.0	88.1	95.5	80.5	0.0	79.0	85.2	26.7	82.8	84.0
SMCalFlow-CS 32-C	13.0	74.4	87.2	88.4	56.7	0.0	67.8	59.0	11.3	60.6	70.4
Table 7: Performance breakdown for all splits, including non-compositional and synthetic splits, in addition to those already presented in Table 2.
Split	Log prob.	Context States, |S|
1	2	3	4	8	32
mcd1 (train)	p(x,y)	-15.14	-12.55	-10.96	-9.60	-9.21	-9.09
mcd1 (train)	p(y|x)	-0.72	0.00	0.00	0.00	0.00	0.00
mcd1 (dev)	p(x,y)	-16.49	-15.04	-17.47	-13.32	-18.85	-20.61
mcd1 (dev)	p(y|x)	-0.76	0.00	0.00	0.00	0.00	0.00
mcd2 (train)	p(x,y)	-14.38	-11.72	-10.62	-10.26	-9.26	-9.13
mcd2 (train)	p(y|x)	-0.58	0.00	0.00	0.00	0.00	0.00
mcd2 (dev)	p(x,y)	-17.60	-14.10	-19.84	-17.58	-19.45	-21.82
mcd2 (dev)	p(y|x)	-0.97	0.00	-1.99	-0.36	0.00	-0.04
mcd3 (train)	p(x,y)	-15.29	-12.56	-10.86	-11.04	-9.11	-9.09
mcd3 (train)	p(y|x)	-0.75	0.00	0.00	0.00	0.00	0.00
mcd3 (dev)	p(x,y)	-16.33	-14.96	-11.25	-19.09	-19.83	-20.71
mcd3 (dev)	p(y|x)	-0.75	0.00	0.00	-0.10	0.00	0.00
Table 8: We compare the average log probability CSL assigns to examples from the SCAN MCD train and dev sets, for different numbers of context states,  $|S|$ .
is applied, in order to capture notions of type agreement. Such limitations are similar to the limitations of conventional PCFGs to sufficiently model structural dependencies for syntactic parsing (Klein and Manning, 2003).
In general, better understanding and optimizing the trade-offs related to context sensitivity for compositional generalization is an important direction for future work.
C.3 Comparing CSL and NQG
CSL and NQG of Shaw et al. (2021) vary across several dimensions, as the two systems use different grammar induction algorithms and different model parameterizations. Here we compare the two
approaches across both dimensions independently.
Grammar Induction As discussed in §3.1.4, the largest set of changes to the CSL algorithm from that of NQG were to improve the scalability of the induction algorithm, as both algorithms scale superlinearly in both dataset size and the length of input and output strings. The runtime of grammar induction on the GeoQuery standard split on a standard workstation CPU is around 15 minutes for NQG, and  $< 1$  minute for CSL. More importantly, we did not find it feasible to run NQG for SMCalFlow-CS, while CSL enables grammar induction to be completed within 10 hours with parallelization.
CSL also supports QCFG rules with  $>2$  nonter
Parsing Model	GEOQUERY	COGS
Templ.	TMCD	Len.	Gen.
BERT + SpanLabel	58.8	41.0	23.3	99.2
CSL Gen. Model	58.9	39.9	26.6	99.4
minals while NQG does not. We found allowing up to 4 nonterminals can improve the coverage of induced grammars for COGS and SMCalFlow-CS, with some example rules shown in Table 5 in Appendix A. Notably, for COGS, the induction algorithm of NQG induced a grammar that can only derive  $64.9\%$  of the test set.
Model Parameterization We compare the performance of CSL's simple generative model with that of the span-based model of NQG which uses a BERT-Base encoder in Table 9. Both models use the same grammar induced via the CSL algorithm. Overall, the models perform comparably, despite CSL having far fewer parameters (e.g. for GeoQuery, CSL has only 51,200 parameters $^{22}$  while NQG has over 110M parameters as it includes a BERT-Base encoder), not leveraging pre-trained neural networks, and being a generative model to support sampling (in contrast to NQG which is a discriminative model). Incorporating pre-trained neural components into a model such as CSL could be a promising future direction.
For SMCalFlow-CS, given a grammar induced by CSL, we found that it can be computationally infeasible to train a discriminative NQG model, due to the need to compute the partition function which sums over all possible derivations of the input. As a generative model, CSL avoids the need to compute a partition function during training.
C.4 Comparison with GECA
Hyperparameters For SCAN, the reported results in Table 1 use a window size of 1. Using a window size of 2 improves performance on the MCD split (24.9% vs. 22.8%) but hurts performance on the other splits. For GeoQuery, we used
Table 9: We compare the accuracy of the span-based BERT-Base model of NQG (BERT + SpanLabel) with that of the CSL generative model (when used as a parsing model) for the same induced grammar (induced using CSL), on compositional splits of GeoQuery and COGS.
TRAINING EXAMPLES
<jump, JUMP>
<walk, WALK>
<jump and walk, JUMP WALK>
SYNTHETIC EXAMPLES (GECA)
<walk and walk, WALK WALK>
<jump and jump, JUMP JUMP>
SYNTHETIC EXAMPLES (CSL)
<walk and walk, WALK WALK>
<jump and jump, JUMP JUMP>
<walk and jump, WALK JUMP>
<walk and walk and jump, WALK WALK JUMP>
<walk and jump and walk and walk, WALK JUMP WALK WALK>
...
Figure 5: We show the set of derivable synthetic examples for GECA (with window size  $= 2$ ) and CSL, given an illustrative example of training examples. CSL can derive a significantly larger set of synthetic examples than GECA, and also assigns a probability to each derivable example.
the default window size of 4 for the GeoQuery experiments. We attempted to run GECA on COGS and SMCalFlow-CS also using the default hyperparameters and did not find it to be computationally tractable. The algorithm's iteration over templates and fragments can become prohibitive for larger-scale datasets.
Analysis From Table 1 we see that augmenting the training data using CSL outperforms GECA across both synthetic and non-synthetic evaluations. GECA relies on the simple assumption that fragments are interchangeable if they appear in the same context. It is restricted by a pre-defined window size for fragments, and does not support recursion. Figure 5 compares differences in the sets of derivable synthetic examples for a notional set of training examples. In this case, CSL can derive a much larger set of recombinations by inducing the rule  $\mathrm{NT} \rightarrow \langle \mathrm{NT}_{[1]} \text{ and } \mathrm{NT}_{[2]}, \mathrm{NT}_{[1]} \mathrm{NT}_{[2]} \rangle$ , which can be applied recursively.
C.5 Limitations of QCFGs
The mapping from inputs to outputs in SCAN, COGS, and GeoQuery are all well supported by QCFGs. However, grammars were used to generate the data for SCAN and COGS, so this is perhaps not surprising. While GeoQuery inputs were written by humans, the distribution of queries in the dataset is influenced by the capabilities of the underlying execution engine based on logic programming; the dataset has a large number of nested noun phrases in inputs that map directly to nested
22For each rule and non-terminal token in the induced grammar, CSL has a number of parameters equal to the number of context states.
FunQL clauses in outputs.
SMCalFlow The induced grammars have relatively low coverage on SMCalFlow, as shown by  $\%_{CSL}^{x}$  in Table 7, although they are still sufficient to improve the performance of T5. One reason for the low coverage is that inputs in SMCalFlow often reference specific names, locations, and meeting subjects, such as "setup up a sales meeting with Sam and his manager" where "sales meeting" and "Sam" must be copied to the output program as string literals. Sequence-to-sequence models with copy mechanisms or shared input-output vocabularies can handle such copying, but the QCFGs induced by our method do not support generalization to such novel tokens. Extending the method to support such string copying could significantly improve coverage.
Another reason for the low coverage is that the mismatch between the nesting of prepositional phrases in the input (e.g., "at NT" and "with NT") and the corresponding clauses in the output program tree makes it difficult to induce QCFG rules that enable recombination of different prepositional phrases in different contexts.
The induced QCFGs are also limited in other cases, such as their inability to "distribute" over groupings correctly. Since the training data only contains example such as \(\langle\) Jennifer and her boss, (person = "Jennifer") (FindManager (person = "Jennifer"))\rangle, the induced rule NT \(\rightarrow\) \(\langle\) NT and her boss, (person = "NT") (FindManager (person = "NT"))\rangle, the induced grammar cannot correctly generate test examples like \(\langle\) Jennifer and Eli and their bosses, (person = "Jennifer") (person = "Elli") (FindManager (person = "Jennifer"))\rangle (FindManager (person = "Elli"))\rangle.\)
CFQ We also evaluated the feasibility of our approach to improve T5 performance on CFQ (Key-sers et al., 2020), a popular synthetic dataset for evaluating compositional generalization. We found it was challenging to induce QCFGs with reasonable coverage for CFQ. First, the SPARQL queries in CFQ contain variables, which are not well supported by QCFGs (Wong and Mooney, 2007). Additionally, the mapping from queries to SPARQL in CFQ requires notions of commutativity (both "M0 edited and directed M1" and "M0 directed and edited M1" will be mapped to "M0 ns:filmdirector.film M1 . M0 ns:film.editor.film M1") and distributivity (edited in "edited M1 and
M2" will appear twice in "\(?x0 ns:film.editor.film M1 . ?x0 ns:film.editor.film M2") that are also not well supported by QCFGs. Such limitations can potentially be partially overcome by designing intermediate representations for CFQ (Furrer et al., 2020; Herzig et al., 2021), but a complete solution likely requires an extension to the class of allowable rules in \(\mathcal{G}\) beyond those a QCFG formalism supports, such as better support for variables (Wong and Mooney, 2007) and the ability to apply rewriting rules to generated output strings.
C.6 Sampling Temperature
In this section we study the impact of the parametric model on data augmentation. To do this, we consider varying the sampling temperature, applied to  $\theta_{t,r}$  prior to normalization. We compare the accuracy of T5+CSL-Aug. for different temperatures for SMCalFlow-CS, and also with sampling from a uniform distribution rather than using the CSL parametric model for all splits, which can be viewed as using temperature  $= \infty$ .
Table 10 shows that using the parametric model outperforms uniform sampling by a large margin on most splits. However, for SMCalFlow-CS, increasing the sampling temperature can lead to improved performance. To help understand why increasing temperature improves performance on the SMCalFlow-CS cross-domain splits, we computed the number of single-domain examples and cross-domain examples in the 100,000 sampled synthetic examples. Sampling from uniform distribution generates on average 17,764 cross-domain examples comparing with sampling from CSL which generates on average 1,114 cross-domain examples. The significant larger number of synthetic cross-domain examples might explain the improvement on cross-domain performance when increasing sampling temperature, especially on the 8-shot split, given the small number of cross-domain examples in the original training data.
C.7 Semi-Supervised Learning
If we have unlabeled data from our target distribution, consisting of inputs only, we can incorporate this data when training our generative model in a straightforward way. Here we propose a new experiment to evaluate a method for semi-supervised learning that leverages such unlabeled examples. We assume that the unlabeled inputs from the development set are available during training.
Temp.	SCAN	COGS	GEOQUERY	SMCALFLOW-CS
Jump	Left	Len.	MCD	Gen.	Std.	Templ.	TMCD	Len.	8-S	8-C	16-S	16-C	32-S	32-C
∞	97.2	98.0	42.0	43.8	91.4	92.9	88.0	72.9	51.5	83.4	63.1	84.4	62.6	83.8	71.5
10	—	—	—	—	—	—	—	—	—	82.8	62.4	83.1	63.8	82.5	71.0
1	99.7	100.0	99.2	99.4	99.5	93.3	89.3	74.9	67.8	83.5	51.6	83.4	61.4	84.0	70.4
Table 10: We compare sampling from CSL and uniform distribution (temperature  $= \infty$ ) for the same induced grammar.
Semi.	Example #	GEOQUERY	SCAN
Std.	Templ.	TMCD	Len.	MCD
✓	1K	92.9	89.3	72.5	66.4	63.8
1K	93.2	89.3	74.4	67.9	88.9
100K	93.3	89.3	74.9	67.8	99.4
✓	100K	93.2	89.4	75.6	67.3	99.4
Experiment Setting In this setting, conceptually, when optimizing  $\theta$ , we want to maximize both the likelihood of  $p(x,y)$  for  $\mathcal{D}$  and the marginal likelihood  $p(x)$  for the unlabeled data. As the latter requires marginalizing over derivations for  $x$  and all possible outputs, and this can be a large set to sum over, we approximate this by first labeling the unlabeled data following the inference procedure described in §3.1.5 using the generative model trained only on  $\mathcal{D}$ , which can be interpreted as a hard-EM approach. During this process, we discard any unlabeled that cannot be derived given  $\mathcal{G}$ . We then re-train the generative model on both sets of data following the standard procedure, duplicating the "unlabeled" data a configurable number of times to achieve the desired ratio to the original labeled data.
We evaluate our CSL + T5-Aug. in this setting using unlabeled data from the development set. We use the SCAN MCD splits as they have dev sets available. We also evaluate performance on the GeoQuery splits. We compare the performance with and without unlabeled data using 1,000 and 100,000 synthetic examples.
Results Results are reported in Table 11. Incorporating unlabeled examples leads to improvements when sampling only 1,000 examples, but leads to minimal improvements when sampling 100,000 examples. We did not find positive results based on initial experiments for SMCalFlow-CS, likely due to the low coverage of the induced grammars on the target examples (see Table 7), as the
Table 11: We compare the accuracy of T5 + CSL using data augmentation with and without unlabeled data.
Split	Mean	Stdev.
Std.	93.3	0.2
Templ.1	92.5	0.3
Templ.2	88.1	0.8
Templ.3	87.2	0.6
TMCD1	77.2	1.1
TMCD2	71.3	0.6
TMCD3	76.3	0.2
Len.	67.8	0.3
Table 12: The mean and standard deviation of 3 runs for T5+CSL-Aug. on GeoQuery dataset.
method we evaluated cannot leverage unlabeled examples that are not covered by the induced grammar.
C.8 GeoQuery Variance
The variance of T5+CSL-Aug. for GeoQuery is reported in Table 12.
C.9 SMCalFlow-CS Error Analysis
We sampled 20 prediction errors for T5+CSL-Aug. from single-domain and cross-domain development sets respectively. We found a large number of errors are due to ambiguous and inconsistent annotations. Table 13 shows some examples of such errors. First, the subject string is determined inconsistently for training and testing examples. Second, the same source can be mapped to different targets which express the same meaning. Third, some examples require additional context to generate the correct output. Among the errors we sample, around  $60\%$  of single-domain errors and around  $35\%$  of cross-domain errors fall into these three types. In addition, for the cross-domain examples, T5+CSL-Aug. sometimes struggles with nesting programs in a correct way when examples require querying an org chart for more than one people as discussed in Appendix C.5.
Train Source: Can you create an Meeting for Saturday  $1:00\mathrm{pm}$
Train Target: ( Yield :output (CreateCommitEventWrapper) :event (CreatePreflightEventWrapper):constraint (Constraint[Event] :start (?= (DateTimeWithDefaults) :date (NextDOW :dow # (DayOfWeek "SATURDAY")) :time (NumberPM :number # (Number 1))) :subject (?= (# (String "Meeting)))))
Dev Source: Schedule a meeting on Thursday at 8 : 30 AM .
Dev Target: ( Yield :output ( CreateCommitEventWrapper :event ( CreatePreflightEventWrapper :constraint ( Constraint[Event] :start (?= ( DateTimeWithDefaults :date ( NextDOW :dow # ( DayOfWeek " THURSDAY " ) ) ) :time ( HourMinuteAm :hours # ( Number 8 ) :minutes # ( Number 30.0 ))))))
Dev Prediction: ( Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :start (?= (DateTimeWithDefaults :date (NextDOW :dow # (DayOfWeek "THURSDAY"))):time (HourMinuteAm :hours # (Number 8) :minutes # (Number 30.0)))) :subject (?= (# (String "meeting"))))
Train Source: Schedule 3 pm tentative shareholders huddle
Train Target: ( Yield :output ( CreateCommitEventWrapper :event ( CreatePreflightEventWrapper :constraint ( Constraint[Event] :start (?= ( NextTime :time ( NumberPM :number # ( Number 3 )) ) ) ) :subject (?= # ( String " tentative shareholders huddle " ) ) )))
Dev Source: Schedule 3 pm tentative shareholders huddle
Dev Target: (Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :showAs (?= # (ShowAsStatus "Tentative" ) ) :start (?= (NextTime :time (NumberPM :number # (Number 3)))) :subject (?= # (String " shareholders huddle" ))))
Dev Prediction: ( Yield :output ( CreateCommitEventWrapper :event ( CreatePreflightEventWrapper :constraint ( Constraint[Event] :start (?= ( NextTime :time ( NumberPM :number # ( Number 3 )) ) ) ) :subject (?= # ( String " tentative shareholders huddle " )) ))))
Dev Source: create football game on tuesday at 8
Dev Target: (Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :start (?= (DateTimeWithDefaults :date (NextDOW :dow (# DayOfWeek "TUESDAY")) ) :time (NumberPM :number (# (Number 8)))) :subject (?= (# (String "football game")))))
Dev Prediction: ( Yield :output (CreateCommitEventWrapper :event (CreatePreflightEventWrapper :constraint (Constraint[Event] :start (?= (DateTimeWithDefaults :date (NextDOW :dow # (DayOfWeek "TUESDAY")) ) :time (NumberAM :number # (Number 8)))) :subject (?= # (String "football game"))))
Table 13: Example prediction errors for T5+CSL-Aug. and their closest training example if any for the SMCalFlow-CS dataset.