SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We find that constrained decoding under performs unconstrained decoding for some settings. We attribute this to the insufficient coverage of our grammars. Our grammars were induced from the training data, and thereby fail to cover novel components or combinations at test time. They are also constrained to copy quoted strings from the input utterance. However this is not strictly followed in some datasets. Table 6 shows the fraction of test outputs covered by our grammars. We could relax the grammar constraints to ensure full coverage; we leave such exploration to future work. In realistic situations, we expect grammars to be provided by domain developers ensuring full coverage.
5.3 Impact of Context
The datasets in BenchCLAMP require a model to use a variety of contexts. SMCalFlow, TreeDST and CoSQL datasets all have conversational context. Spider and CoSQL have database schema context which informs the target SQL prediction. BenchCLAMP allows us to perform a controlled investigation of the effect of context. Table 7 shows that while using the last agent and user utterance is helpful for all settings, the low-data regime does even better when using only the last agent utterance; without more data, training struggles to learn how to utilize (or ignore) the additional
Table 6: Grammar coverage (%) of the gold outputs in the test set of BenchCLAMP datasets.
Dataset	Test Set Coverage %
SMCalFlow	99.6
TreeDST	96.3
MTOP (en)	96.6
Overnight (blocks)	100.0
Spider	98.8
CoSQL	99.4
AMR	99.7
PTB 3	100.0
UD-EWT	99.9
context. We find similar results for CoSQL in Table 8. Also, SQL prediction always benefits from including database values in the context along with the database schema information. We use the best settings for context for each data regime for benchmarking results in tables 2, 3 and 4.
Table 7: Lispress match accuracy of unconstrained fine-tuned T5-large with varying conversational context on SMCalFlow and TreeDST. We find more context hurts in low resource settings but helps in medium and high resource settings.
Conv. Context	SMCalFlow	TreeDST
Low	Med	High	Low	Med	High
No context	37.0	63.8	72.9	42.4	68.4	76.0
Last agent Utt.	42.6	70.7	80.6	59.6	82.7	87.9
Last user & agent Utt.	40.0	71.5	81.3	58.8	86.2	90.0
Table 8: Test suite execution accuracy of unconstrained BART-large on CoSQL with varying context. More context hurts in low resource settings but helps in medium and high resource settings.
Conversational
Context	DB values?	Low	Med	High
No context	no	21.3	35.9	34.4
yes	25.3	38.9	40.3
Last interaction	no	24.1	40.4	39.1
yes	28.2	44.2	44.4
All interactions	no	24.3	36.8	43.0
yes	24.9	44.9	48.8
5.4 Few-Shot Prompting
In few-shot prompting scenario, we manipulate the context choice and ordering of examples in our prompt to Codex. The results in Table 9 show that ordering the most similar example at the end closest to the generation heads is helpful in the low-data regime, indicating that GPT-3 and Codex pay more attention to the recent past. In higher data regime, all prompt examples are almost equally relevant, hence the order does not matter as much. We find that context does not help; one of the reasons being that we can fit fewer examples in the prompt if we include context for each example.
We experiment with BM25 and similarity models for prompt retrieval for few-shot prompting. For similarity models, we pick top models from each of the three categories in Sentence Transformers leaderboard [29]: all-mpnet-base-v2, multi-qa-mpnet-base-dot-v1 and sentence-t5-xxl [25]. We find that SentenceT5-xxl surpasses other similarity models for prompt retrieval. SentenceT5 outperforms BM25 in low resource settings, but performs relatively worse when more data is available.
Table 9: (Left): Lispress match accuracy of the few-shot prompted Codex DaVinci model on SMCalFlow with different prompt order and conversational context. More context hurts few-shot prompted models. Ordering the most relevant examples closer to the generation heads improve performance. (Right): Lispress match accuracy of Codex on SMCalFlow with different prompt retrieval methods. We used code-davinci-001 with best last prompt order and no context.
Prompt Order	Conv. Context	Low	Med
Random	No context	35.7	52.0
Best First	No context	34.3	53.0
Best Last	No context	36.7	52.0
Best Last	Last agent Utt.	34.0	41.0
Best Last	Last user & agent Utt.	26.0	31.0
Prompt Retrieval Method	Low	Med
BM25	36.7	55.0
all-mpnet-base-v2	38.0	49.0
multi-qa-mpnet-base-dot-v1	38.3	50.0
sentence-t5-xxl	39.3	52.0
Table 10: Effect of different grammar induction data on the Lispress match constrained decoding accuracy of fine-tuned T5-large.
Constraint Grammar	SMCalFlow	TreeDST
Low	Med	Low	Med
Unconstrained	42.6	71.5	59.6	86.2
Induced from train split	45.6	73.1	62.3	87.2
Induced from full train	46.3	73.1	64.2	87.2
5.5 Grammars induced from Less Data
The grammars for SMCalFlow, TreeDST and MTOP were induced using the full train dataset. This grammar is then used even with low and medium resource train splits. We expect the grammar will be provided by a developer of the domain and hence will cover all valid representations. For completeness, we report here the impact of using grammar induced from the corresponding train sets. Table 10 shows the results of decoding with train split induced grammar, and compares the performance with unconstrained decoding and decoding with grammar induced from full train set. The gains from constraints drop by  $1 - 2\%$  for low resource splits when using train split induced grammar instead of full train induced grammar. It does not affect results for medium resource splits.
We introduce a benchmark comprising nine parsing datasets with varying target representations. We support few-shot prompting, fine-tuning and constrained decoding for all autoregressive language models and sequence-to-sequence models on these datasets. We hope that this work will encourage language model developers to consider parsing tasks as a test-bed in future work.
Our benchmark includes data in multiple languages (all languages included in the MTOP dataset) but we only evaluate on English datasets due to compute constraints. Few-shot prompted experiments were evaluated on relatively small test sets on three datasets due to API cost limitations. As a result, we noticed high variance in the results (see Appendix section A for variance results).
Thanks to Yunmo Chen who helped set up Llama-2 models for our benchmark experiments.
[1] Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crime, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin
1962-1979, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1204. URL https://aclanthology.org/D19-1204.
[52] Daniel Zeman et al. Universal dependencies 2.11, 2022. URL http://hdl.handle.net/11234/1-4923. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
[53] Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. Stack-based multi-layer attention for transition-based dependency parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1677-1682, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1175. URL https://aclanthology.org/D17-1175.
[54] Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-sql with distilled test suite. In The 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2020.
A Variance of Results
All low-resource results are a mean of three training data splits. Table 11 reports the average standard deviation for each model over the three low resource splits. We find a high standard deviation of GPT-3 and Codex; one of the factors being the small size of the test set (100 examples). Fine-tuned models show relatively low variance, consistently having standard deviation lower than  $2\%$ .
Table 11: Standard deviation of the scores for each language model over the three low resource splits.
Model	Avg. Standard Deviation
GPT-3	4.7
Codex	3.2
T5-base	1.2
CodeT5-base	1.1
BART-large	1.4
T5-large	2.0
T5-3B	1.5
B Compute Details
We used Microsoft Azure to run all our experiments. For experiments with T5-base, CodeT5-base and BART-large, we used a single V100 GPU with 32 GB memory. We used 2 V100 GPUs for fine-tuning T5-large and 4 GPUs for fine-tuning T5-3B. Training time ranged from 2 to 6 hours depending on the size of the model.
C Dataset and Model License and Privacy Implications
The licenses of the datasets included in BenchCLAMP are listed in Table C. We could not find a license for MTOP, but the dataset was made freely available by the creators. Our benchmark is available under the MIT licence at https://github.com/microsoft/semanticParsing_with_constrained_lm.
The datasets in BenchCLAMP either use sentences from news articles or utterances generated by crowd-workers following instructions to simulate user interactions. No real user interaction data was used to generate these datasets. As a result, there is no privacy risk in using these datasets.
For fine-tuning experiments, we use pre-trained models from Huggingface. All models were released under the Apache 2.0 license.
Table 12: Licenses of datasets covered by BenchCLAMP.
Dataset	License
SMCalFlow [1]	CC BY-SA 4.0
TreeDST [10]	CC BY-SA 3.0
MTOP [18]	-
Overnight [45]	CC BY-SA 4.0
Spider [50]	CC BY-SA 4.0
CoSQL [51]	CC BY-SA 4.0
AMR 2.0 [4]	LDC License
PTB 3 [23] (Constituency parsing)	LDC License
UD-EWT [52] (Dependency parsing)	CC BY-SA 4.0 International