BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing
Authors: Subhro Roy
Sam Thomson
Tongfei Chen
Richard Shin
Adam Pauls
Jason Eisner
Benjamin Van Durme
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Syntactic parsing
semantic parsing
benchmark
constrained decoding
GPT-3
TL;DR: Benchmark to evaluate any LLM on syntactic and semantic parsing

Abstract: Recent work has shown that generation from a prompted or fine-tuned language model can perform well at semantic parsing when the output is constrained to be a valid semantic representation. We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output meaning representations, as well as a constrained decoding interface to generate only valid outputs covered by these grammars. We provide low, medium, and high resource splits for each dataset, allowing accurate comparison of various language models under different data regimes. Our benchmark supports evaluation of language models using prompt-based learning as well as fine-tuning. We benchmark seven language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or even surpass state-of-the-art methods for both syntactic and semantic parsing when the model output is constrained to be valid.

Bench A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing
Subhro Roy! Sam Thomson'
Tongfei Chen'
Richard Shin !
Adam Pauls
Jason Eisner Benjamin Van Durme!
Microsoft Semantic Machines Scaled Cognition
jusn.jeiineer microsoft.com Padpauls scaledcognition..cm
Abstract
Recent work has shown that generation from prompted or fine-tuned language model can perform well at semantic parsing when the output is constrained to be valid semantic representation. We introduce BenchCLAMP, Benchmark to eval- uate Constrained LAnguage Model Parsing, that includes context-free grammars for seven semantic parsing datasets and two syntactic parsing datasets with varied output representations, as well constrained decoding interface to generate only valid outputs covered by these grammars. We provide low, medium, and high resource splits for each dataset, allowing accurate comparison of various language models under different data regimes. Our benchmark supports evaluation of lan- guage models using prompt-base learning well fine-tuning. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models achieve similar performance or surpass state-of-the-an methods for syntactic and semantic parsing when the model output is constrained to be valid.
1 Introduction
Large pretrained language models can achieve state-of-the-art performance on host of NLP tasks when fine-tuned on target data [22, 28, 46, 13]. Models like GPT-3 [5], Codex [9] and TO [31] have also shown impressive zero- and few-shot performance when prompted only with task descriptions and examples. Research large language models is typically validated by performance downstream NLP tasks. Past work has evaluated new pretrained language models on classification, extraction, and generation, among others [22, 13, 20]. However, parsing tasks are generally not considered testbed for such evaluation. The outputs of parsing tasks are structured objects such as parse trees or code. State-of-tt ooftthe-ar art systems thus involve task- or -ataset-specific model architectures and target representation constraints. Evaluating language models on parsing tasks test capabilities not captured by commonly used evaluation tasks.
Recently, Shin et al. [36] and Scholak et al. [32] have shown that generation from a fine-tuned or few-shot prompted language model can perform competitively in semantic parsing tasks, when the output of the language model is constrained to produce valid meaning representa- tions. However, it is still challenging to set up constrained generation for new dataset and language model due to the variation in output formalisms and model-specific tokenization. In this paper, introduce new benchmark called BenchCLAMP (Benchmark for Constrained Language Model Parsing) that covers nine parsing datasets with seven different output for- malisms. We release context-free free grammars for each dataset and provide toolkit perform
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.
* Work done while at Microsoft Semantic Machines.

---

efficient constrained decoding to generate valid output representations. Our benchmark reduces the barrier for language model developers to evaluate on parsing, The benchmark is available at https: / / thub. com/m / grosoft semant ie ic_parsing__ sa_inn__ wi wwithhhoe th_constra ned__mm
We benchmark eight pretrained language models using BenchCLAMP. We find that fine-tuning encoder-decoder pretrained language models can come close to or surpass the performance of state- of-the-art methods on all parsing datasets. Constrained generation via domain-specifie grammar provides performance gains for most fine-tuned language models. The improvement is high in low- resource settings but the relative improvement reduces when more training data becomes available. We find constrained decoding to be essential for few-shot prompted models and for tasks with complex constraints like constituency and dependency parsing. both these cases, we find language models struggle to generate valid representations without constrained decoding. In addition, we ablate different ways to encode context in the input, prompt structure, and retrieval methods for few-shot prompted language models. We present comprehensive study establishing concrete techniques to reliably use language models for syntactic and semantic parsing tasks.
2 Related Work
Language Models for Semantic Parsing: Recent work has shown that one can generate an analysis of natural language sentence, such semantic parse, by asking large language model to continue a prompt that includes the sentence [9, 18, 18.33.. We refer to this as "language model parsing." To avoid ill-formed analyses, it possible to constrain the generation that the generated output satisfies hard task-specific constraints. Shin et al. [36] showed that constrained generation from few-shot prompted GPT-3 and fine-tuned BART models outperformed task-specific semantic parsing architectures in low-resource settings. Scholak et al. [32] were able to achieve state-of-the-art performance in SQL prediction by fine-tuning T5-3B model [28] and using constrained decoding. Recent work on AMR parsing has also shown positive results with sequence-to-sequeet model parameters [6, 3]. As the above works used different evaluation settings, it hard to see which techniques work best under different data regimes.
training with pretrained language
Syntactic parsing tasks like constituency and dependency parsing requires the outputs to be well aligned with the input. All input tokens need to be covered by the output constituency or dependency parse. As a result, most solutions for syntactic parsing has involved custom decoders or inference algorithms [14, 49, 53, 21J. However there has been some work on generating linearized representations of the output [15, 47, 19, 12).
Language Models for Syntactie Parsing:
Multiple benehmarks have been introduced to track progress on specific NLP tasks and to encourage multi-task learning with diverse datasets. The GLUE [42]. SuperGLUE [43], BIG-bench [37] and HELM [20] are widely used by language model developers. However, these benchmarks focus classification, span extraction and generation, and do not include structured prediction tasks like parsing. More recently, the UnifiedSKG G [48] benchmark has been introduced that converts a suite of tasks requiring structured knowledge into text-to-text format. In contrast to their work, we cover a wide range of parsing tasks covering AMR, SMCalFlow, constituency and dependency parsing. While they focus on unconstrained generation, develop grammars and constrained decoding interface to support valid representation generation.
NLP Benchmarks:
Benchmark Details
3.1 Data Setup
BenchCLAMP includes nine popular parsing datasets with a varied set of meaning representation formalisms (details in Table 1). We create several splits for each dataset. For datasets that do not release public test sets, like SMCalFlow, Spider, and CoSQL, we treat the development set as the test set, and sample 10% of the training set and treat it as the development set for splits creation.
1. We create three low-resource train splits of 500 examples, each uniformly sampled from the training portion of the dataset. We create a single low-resource development set of 50 examples sampled from the development portion of the dataset. We report mean of these splits.
We similarly create a medium-resouree train split of 5000 examples paired with development set of 500 examples.

---

Table 1: List of datasets covered by BenchCLAMP, along with evaluation metric and an example representation. All representations are linearized into sequence that can be produced by language model. We use task and dataset specific metrics to evaluate performance.
Dataset
Metric
Example Representation
SMCalFlow [1]
(Yield (Event, start (FindNunNextEEvnd
Lispress Match
TreeDST [10]
(Bvent.subjecct, (?-- "meeting")) 11)))
MTOP [18]
Exact Match
IIN:Get_Mesaaee [SL:Type_Content video] [SL:Sender Atlas]]
Overnight [45]
Denotation Match
(call listValue (call getPzopezty en en.block.block (string color)))
Spider [50]
Test suite
SELECT born__sst_______________________ FROM head GROUP BY born stan_ttate HAVING count (+) >=
CoSQL [51]
Execution
AMR 2.0 [4]
Smatch
(e/establish-01 :ARG1 (m/model :mod (1/innovate-01 :ARG1 (i2/industzy))))
PTB 3 [23] (Con- EVALB
(S (NP (NNP Ms.) (NNP Haag)) (VP (VBZ plays) (NP (NNP Elianti))) ( .))
stituency parsing)
UD-EWT [52] (De- LAS pendency parsing)
(1-Read, reat, 0) (2-some, obj, 1) (3-of, case, 6) (4-the, det, 6) (5-following,
amod, 6) (6-1inks, nmod, 2)
3. We consider a high-resource split with the entire training set of the dataset, paired with the medium-resource development set.
To make feasible for researchers to evaluate large pretrained models on BenchCLAMP. we ran- domly sample smaller test for datasets with large released test sets. Specifically, we sample 2000 examples from the test sets of SMCalFlow, TreeDST and MTOP datasets and evaluate test performance on this smaller set. We use the full test set for all other datasets. We also release smaller sundomly-sampled 100-example test set for each dataset to evaluate models accessed through costly API calls like GPT-3 and Codex. Results on 100-example test set will have wide error bars and should be used with caution. See the Appendix for a discussion on result variance. We allow for evaluation on full test sets of the datasets to compare with state the art results.
For datasets that include dialogue interactions, we ensure that all turns of a dialogue belong to the same split. The Overiight train set already small (< 5k examples), medium split for it. We release data splits for all domains of Overnight and all MTOP languages, but for brevity we benchmark on single Overnight domain (blocks) and single MTOP language (English) in this paper.
do not have separate
Linearizing Representations: We the dataset representations is for the MTOP, PTB-3 and the SQL datasets. For AMR, we use the setup provided by [40, 41] to linearize the representations into sequences for training, and to convert output model predictions to AMRs for evaluation. For SMCalFlow and TreeDST, we use the Lispress format (I (LiSP-like serialization for programs) of the data released by [27). We linearize dependency parses into sequence of dependency triples. For the example in Table 1, our representation has the following form:
(Read, root, root) (some, obj. Read) (of, case, links) (the, det, links) (following, amod, links) (links, mmod, some)
To convert such a representation back to a dependency parse, we find head token indices for each token based on string match with the predicted head token. In case there are multiple mentions of the head token, we select the one that closest to the token being considered. We can correctly roundtrip 95.7% of parses in the test set using this approach. For completion, we also release a loss less linearization of dependency parses which is similar to the one above but with the head token replaced by the corresponding token index in the input sequence.
3

---

3.2 Grammars
We release context-free grammars for all datasets constrain generation to valid meaning representa- tions. The grammar creation process is specific to each dataset.
1. For SMCalFlow and TreeDST, we use the Lispress-format datasets released by Platanios et al. [27]. We create a non-terminal corresponding to each type present in the training data. For each (sub-jexpression with type I, we add a production rule with the non-terminal for / generating the non-terminals of its component (sub-Jexpression types, component terminal plan fragments.
2. For MTOP, we add a non-terminal corresponding to each intent and slot. Each intent non-terminal generates an expression comprising slot -on-terminals. Similarly each slot non-terminal can generate an expression with nested intent non-terminals. Slot non-terminals can also generate terminal strings copied from the input utterance.
3. We use publicly available SQL grammar [2] for Spider and CoSQL. For each example, we add schema-specific constraints to the grammar to generate consistent table and column names. This is similar to "parsing without guards" in Scholak et al. [32].
4. For constituency parsing, we define non-terminal for an expression and constituent label. We add production rules where the label non-terminal can produce any of the constituent labels seen in the training data. The expression non-terminal can either produce terminal tokens generate a constituent label coupled with a expression non-terminal. This context free grammar covers constituent parse tree representation shown in Table 1. To additionally ensure that all tokens in the input utterance are covered by the generated parse tree, we additionally maintain a state in our parsing algorithm during decoding, allowing tokens to appear in the order seen in the utterance, and allowing the generation to end only when all input tokens have been generated.
For dependency parsing, we extract the set of dependency relations from the training set and define non-terminal that can produce any of them. We then define non-terminal that can generate a sequence of triples each comprising two tokens from the utterance and a dependency relation. Similar to our approach with constituency parsing, maintain a parse state during decoding to ensure all tokens from the utterance are covered in order in the generated output.
For all data splits, scenarios, the grammar will be provided by a domain developer, and hence will have complete coverage of the domain (even when some plan fragments might not have appeared in the low-resource dataset). We also add results with grammars induced from low-resource splits in section 5.5.
the full training data to derive the grammar. We envision that in realistic
4 Experimental Setup
4.1 Language Models Evaluated
We use BenchCLAMP to fine-tune and evaluate five language models with varying number of parameters: T5-base (220M), T5-large (770M), T5-3B (3B) [28], BART-large (406M) [17] and CodeT5-base (220M) [44]. The input to our model is the utterance concatenated with the string representation of the context (conversation context, database schema, etc.), and the output is the target parse. We evaluate three large language models: GPT-3 [5]. Codex [9] and Llama-2 [39]', using few-shot prompting on the 100 example test sets. Unless otherwise state, we use the OpenAl API for GPT-3 and code podd-davinci-O00
for Codex. For each input (utterance concatenated with context) we select a set of 20 relevant examples from the training set using BM25 [30] or a sentence transformer [29] based similarity model. We create a prompt using these examples, following the template in Shin al. [36] and limiting the total length of the prompt to be 1500 tokens. This leaves room in GPT-3's buffer to generate an output of up to 548 tokens.
text-davinci-o00
We release data splits for all domains of Overnight and all languages in MTOP. But for brevity, we benchmark on single domain of Overnight (blocks) and single language from MTOP (English). All other datasets used in the benchmark are in English. We evaluate few-shot prompted GPT-3, Codex, and Llama-2 on three BenchCLAMP datasets. All other models are evaluated on the complete evaluation suite.
We use 1lama-22 7b in our evaluation. Llama-2 is served using vLLM [16] to create the same OpenAl API as GPT-based models.

---

We use the code released by Shin et al. [36] to support incremental constrained generation of semantic representations. This code maintains a chart according to Earley's s algorithm [11] that can be used to determine the set of legal next tokens and can also be efficiently updated after particular token is selected. We extend their method to support all autoregressive language models and sequence-to- sequence models. Unless otherwise mentioned, we always use constrained decoding.
Table 2: Performance of language models on SMCalFlow, TreeDST and MTOP. indicates few-shot prompted and evaluated on the 100 example test set; numbers above the horizontal bar are thus comparable to those below. Prompt exemplars retrieved by BM25; 2 by dense vector retrieval using SextenceTS-xxl. Remaining LMs are finetuned and evaluated BenchCLAMP test sets. We report results with both constrained and unconstrained decoding to illustrate the contribution of constraints. Metrics are datasct-specific (see Table 1). The best score in each column is boldfaced.
Grammar
SMCalflow
TreeDST
MTOP (en)
LM
Constraints
Low
Med
High
Low
Med
High
Low
Med
High
GPT-3f1
Yes
26.0
48.0
49.0
35.7
54.0
53.0
46.3
56.0
57.0
Codex 11
Yes
36.7
55.0
56.0
46.3
61.0
62.0
53.3
68.0
72.0
Llama-2-7b'!
No
25.0
35.0
44.0
31.0
50.0
59.0
43.0
60.0
62.0
GPT-312
Yes
33.0
45.0
52.0
38.7
56.0
59.0
50.0
62.0
64.0
Codex
Yes
39.3
52.0
62.0
47.7
58.0
64.0
55.7
67.0
70.0
Llama-2-76b
No
29.0
39.0
42.0
28.0
46.0
58.0
44.0
61.0
62.0
No
38.2
67.5
77.6
57.2
84.4
89.3
54.7
79.3
84.5
T5-base
Yes
41.6
69.7
78.6
62.0
85.8
89.4
57.5
80.1
84.3
No
33.3
65.6
80.8
50.3
83.5
90.3
44.1
75.6
80.6
CodeT5-base
Yes
37.3
67.5
81.1
56.8
84.4
90.0
47.1
75.8
81.1
No
36.1
68.1
82.2
52.0
84.0
90.2
57.8
81.6
85.8
BART-large
Yes
42.5
71.4
83.0
61.1
86.4
89.8
61.7
82.1
85.3
No
42.6
71.5
81.3
59.6
86.2
90.0
55.4
82.1
85.6
TS-large
Yes
46.3
73.1
82.1
64.2
87.2
90.1
59.3
82.5
85.3
No
43.5
73.8
82.6
58.5
85.9
90.7
60.9
83.2
86.3
T5-3B
Yes
48.7
75.9
83.0
64.1
87.2
90.3
64.1
83.4
85.6
Table 3: Performance of fine-tuned language models on Spider, CoSQL and Overnight datasets. We report test suite execution accuracy [54] for the SQL datasets and denotation accuracy for Overnight.
Grammar
Spider
CoSQL
Overnight (blocks)
LM
Constraints?
Low
Med
High
Low
Med
High
Low
High
No
30.8
54.1
55.6
24.0
39.9
40.8
63.9
63.7
T5-base
Yes
33.8
56.8
58.9
26.8
42.7
43.4
64.4
63.9
No
36.6
57.4
61.9
26.1
45.9
48.1
60.0
64.7
CodeT5-bass
Yes
37.6
58.0
62.2
27.3
46.2
48.2
60.4
65.2
No
41.8
59.1
63.9
30.9
49.9
52.8
60.7
63.4
BART-large
Yes
42.5
62.7
63.9
29.1
48.8
51.5
61.2
63.7
No
42.4
64.6
65.7
30.7
50.0
53.8
62.1
68.7
TS-large
Yes
44.1
65.5
66.5
32.1
52.4
55.5
62.8
68.9
No
46.4
68.4
70.9
32.6
$4.7
53.4
62.8
66.2
T5-3B
Yes
48.6
70.3
72.3
34.7
56.4
56.2
63.2
66.2
4.2
Format for Model Inputs
For experiments related to fine-tuned language models with SMCalflow and TreeDST with last user and agent utterance as context, the input to model has the format u, where u is the input natural language utterance, / is the last user utterance, is the last agent utterance and is separator

---

Table 4: Performance of fine-tuned language models on constituency (PTB-3), dependency (UD- EWT) and AMR paring. We report bracketing F1 using EVALB [34] for PTB-3, labeled attachment score (LAS) for UD-EWT, and Smatch [7] for AMR parsing.
Grammar
PTB-3
UD-EWT
AMR 2.0
LM
Constraints?
Low
Med
High
Low
Med
High
Low
Med
High
52.7
75.0
T5-base
Yes
83.1
93.1
94.6
80.2
88.2
89.4
51.3
72.0
75.0
48.0
66.0
74.0
CodeT5-hase
Yes
70.9
86.7
92.1
73.2
84.0
85.8
46.7
66.0
74.0
57.0
74.0
81.0
BART-large
Yes
78.0
93.9
95.7
84.1
89.7
90.6
55.0
75.0
81.0
No
57.3
76.0
81.0
TS-large
Yes
84.5
94.4
95.7
80.6
90.1
91.0
57.0
76.0
81.0
No
60.0
77.0
82.0
TS-3B
Yes
77.6
93.9
96.2
83.1
90.4
91.3
59.0
77.0
83.0
symbol. When using only last agent utterance as context, the input a and for using no context, the input to the model is simply u.
We use the the following format for Spider and CoSQL: c, u, where any conversational context applicable, d is a rendering of the database schema with or without values and u is the user utterance. We use the database schema representation used in Scholak al. [32] for d. For concatenate the past utterances in the conversational context with the separator symbol J.
Our shot prompting experiments the prompt template of Shin et al. [36]. Given language input, we retrieve relevant prompt examples and create prompt with the following format:
Let's 8
translate what. a human user says into what computer might say.
Euman: (Prompt Example 1 Input)
Computer: (Prompt Example ] Euman:
(Prompt Example Input
Computer: (Prompt Example Output)
Human:
(Current Input))
Computer:
4.3 Training Details
For fine-tuning experiments, we train the language models with batch size 32 for 000 steps using AdaFactor [35], saving a checkpoint every 5000 steps. We use 1000 linear warmup steps and then linear decay the learning rate T5-3B for which we only used 10-4 to save compute. The best performing checkpoint on the dev set used report scores on the test set.
We tune all models with learning rates 10-3 and 10 except for
Results
5.1
Benchmarking Language Models
We show the performance of language models on BenchCLAMP datasets in tables 2, 3 and 4. We find that performance increases with model size for most fine-tuned language models. Few-shot prompting of GPT-3 and Codex are still not par with fine-tuned models. For non-SQL datasets, even smaller language models reach close to the best performance
high resource setting. However, for Spider and CoSQL, model size seems important in all data settings. This is likely because the model has to reason about the database schema to generate SQL queries, making a harder learning problem. See sections 5.3 and 5.4 for details on how use context in model inputs for these experiments. We skip

---

unconstrained generation results for constituency and dependency parsing since models often fail to generate valid parses for these tasks, and evaluating a valid substring not supported by released evaluation tools. Entirely rejecting invalid parses leads to very low performance.
Table 5 compares our constrained T5-3B model with the best-performing models in the literature. We outperform state-of-the-at models on the SMCalFlow, TreeDST and Overnight-loock datasets. For Spider and CoSQL, our scores are lower than the state-of-the- despite using similar constrained language model approach. This is likely because we use general SQL grammar to constrain decoding, whereas the SQL in these datasets covers only small fraction of the SQL grammar. We believe using more constrained grammar will improve performance. The state-of-the-an method for AMR also uses a CLAMP model but with additional task specific pretraining. The best models for PTB-3 and UD-EWT are designed specific to the task. Our language model line-tuning paired with constrained decoding achieves similar performance without any task specific modifications.
Table Comparison of our fine-tuned T5-3B model with current state of the art models on full test sets. We report exact match accuracy for Spider and CoSQL to match the settings of previous work. The best score in each row boldfaced.
Dataset
Current State the Art
Our T5-3B
SMCalFlow
80.4 [27]
83.7
Tre:DST
88.1 [27]
91.5
MTOP (en)
86.4 [26]
85.7
Overnight
65.2 [8]
66.2
(blocks)
Spider
75.5 [32]
72.2
CoSQL
56.9 [32]
52.3
AMR
85.4 [3]
83.0
PTB 3
96.4 [38]
96.2
UD-EWT
91.5 [24]
91.3
5.2 Effect of Constraints
Decoding constrained by grammar is essential for few-shot prompted models like GPT-3 and Codex that are accessed via API calls. Without a grammar, we noticed that these models explore a large number of invalid paths leading to high latency and API cost. We also found constrained decoding essential for source side prediction tasks like constituency and dependency parsing. They require each token in the input to be covered in the output. We found even fine-tuned language models struggle to learn these constraints leading to very low performance with unconstrained decoding.
Tables 2, 3 and show the effect of constraints while generating from fine-tuned large language models. We find that constrained decoding is most beneficial in low-data regimes, giving on average 2.7% gain over unconstrained decoding. In the high-resoune setting, the average gain is less than 1%, suggesting that the full data nearly sufficient learn the constraint system.
We find that constrained decoding under performs unconstrained decoding for some settings. We attribute this to the insufficient coverage of grammars. Our grammars were induced from the training data, and thereby fail to cover novel components or combinations at test time. They are also constrained copy quoted strings from the input utterance. However this is not strictly followed in some datasets. Table 6 shows the fraction of test outputs covered by our grammars. We could relax the grammar constraints to ensure full coverage; we leave such exploration future work. In realistic situations, we expect grammars to be provided by domain developers ensuring full coverage.
5.3 Impact of Context
The datasets in BenchCLAMP require a model to use a variety of contexts. SMCalFlow, TreeDST and CoSQL datasets all have conversational context. Spider and CoSQL have database schema context which informs the target SQL prediction. BenchCLAMP allows us to perform a controlled investigation of the effect of context. Table 7 shows that while using the last agent and user utterance helpful for all settings, the low-data regime does even better when using only the last agent utterance; without more data, training struggles to learn how to utilize (or ignore) the additional
7

---

Table 6: Grammar coverage (%) of the gold outputs in the test set of BenchCLAMP datasets.
Dataset
Test Set Coverage %
SMCalFlow
99.6
TreeDST
96.3
MTOP (en)
96.6
Overnight (blocks)
100.0
Spider
CoSQL
99.4
AMR
99.7
PTB
100.0
UD-EWT
99.9
context. We find similar results for CoSQL in Table 8. Also. SQL prediction always benefits from including database values in the context along with the database schema information. We use the best settings for context for each data regime for benchmarking results in tables 2, and 4.
Table Lispress match accuracy of unconstrained fine-tuned Ta-large with varying conversational context on SMCalFlow and TTe We find more context hurts in low resource settings but helps medium and high resource settings.
SMCalFlow
TreeDST
Conv, Context
Low
Med
High
Low
Med
High
No context
37.0
63.8
72.9
42.4
68.4
76,0
Last agent
42.6
70.7
80.6
59.6
82.7
87.9
user agent utt.
81.3
58.8
86.2
90.0
Table 8: Test suite execution accuracy of unconstrained BART-large on CoSQL with varying context.
More context hurts in low resource settings but helps in medium and high resource settings.
Conversational Context
DB values?
Low
Med
High
21.3
35.9
34.4
No context
yes
25.3
38.9
40.3
24.1
40.4
39.1
Last interaction
yes
28.2
44.2
44.4
24.3
36.8
43.0
All interactions
yes
24.9
44.9
48.8
5,4 Few-Shot Prompting
In few-shot prompting scenario, we manipulate the context choice and ordering of examples in our prompt to Codex. The results in Table show that ordering the most similar example at the end closest to the generation heads is helpful in the low-data regime, indicating that GPT-3 and Codex pay more attention to the recent past. In higher data regime, all prompt examples are almost equally relevant, hence the order does not matter much. We find that context does not help; reasons being that we can fit fewer examples in the prompt if we include context for each example.
of the
We experiment with BM25 and similarity models for prompt retrieval for few-shot prompting. For similarity models, we pick top models from each of the three categories in CentenceTrrnsffommer leaderboard [29]: that SentenceTS-xx surpasses other similarity models for prompt retrieval. SentenceT5 outperforms BM25 in low resource settings, but performs relatively worse when more data is available.
battaa--mmnee--b--dttt--- and sentence-15-xxl [25]. We find

---

Table 9: (Left): Lispress match accuracy of the few-shot prompted Codex DaVinci model on SMCalFlow with different prompt order and conversational context. More context hurts few-shot prompted models. Ordering the most relevant examples closer to the generation heads improve performance. (Right): Lispress match accuracy of Codex on SMCalFlow with different prompt retrieval methods. We used code-davinci--e with best last prompt order and no context.
Prompt Order
Cony, Context
Low
Med
Prompt Retrieval
Low
Med
Method
Random
No context
35.7
52.0
Best First
No context
34.3
53.0
BM25
36.7
55.0
Best Last
No context
36.7
52.0
all all-mppec-base-v2
38.0
49.0
Best Last
Last agent
34.0
41.0
multi-qa-mpnet--
38.3
50.0
Last
Last user & agent
26.0
31.0
base-dot-v1
utt.
sentence-15-xxl
39.3
52.0
Table 10: Effect of different grammar induction data on the Lispress match constrained decoding accuracy of fine-tuned T5-large.
SMCallFlww
TreeDST
Constraint Grammar
Low
Med
Low
Med
Unconstrained
42.6
71.5
59.6
86.2
Induced from train split
45.6
73.1
62.3
87.2
Induced from full train
46.3
73.1
64.2
87.2
5.5 Grammars induced from Less Data
The grammars for SMCalFlow, TreeDST and MTOP were induced using the full train dataset. This grammar is then used even with low and medium resource train splits. We expect the grammar will be provided by developer completeness, report here the impact of using grammar induced from the corresponding train sets. Table shows the results of decoding with train split induced grammar, and compares the performance with unconstrained decoding and decoding with grammar induced from full train set. The gains from constraints drop by 2% for low resource splits when using train split induced grammar instead of full train induced grammar. It does not affect results for medium resource splits.
the domain and hence will cover all valid representations. For
6 Conclusion
We introduce a benchmark comprising nine parsing datasets with varying target representations. We support few-shot prompting, fine-tuning and constrained decoding for all autoregressive language models and sequence-to-see to-sequence models on these datasets. We hope that this work will encourage language model developers consider parsing tasks test-bed in future work.
7 Limitations
Our benchmark includes data multiple languages (all languages included the MTOP dataset) but only evaluate English datasets due to compute constraints. Few-shot prompted experiments were evaluated on relatively small test sets on three datasets due to API cost limitations. As result, we noticed high variance in the results (see Appendix section A for variance results).
Acknowledgements
Thanks to Yunmo Chen who helped set up Llama-2 models for our benchmark experiments.
References
[1] Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dornrr, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin

---

Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Πlya Lintsbakh, Andy McGovem, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy. Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for doi:
Computational Linguistics, 8:556-571, 2020.
1a16///aalaa0033333 URL https: / //www.aclwelt htw.:/.......aoooooo///22222222 wttp::........l1.... 1.36.
[2] antlr. grammars-v4, v4. httpp:://gtiaeni
ant /et///graaarrrra - v4, 2022.
[3] Xuefeng Bai, Yulong Chen, and Yue Zhang. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Anmual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 6001-6015, Dublin, Ireland, May 2022.
Association for Computational Linguistics. doi: 10.
URL
htteps://cclaanhooooyyoooo
https: /acp:://acaaaaa
ac httpaaoooooo...////222...... wwtp////22002
aal.2accllonnng .4111.15..... 4.
[4] Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperabillity with Discourse, pages 178-186, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: http://aaaaaatttttaaeeeeaa org //Wl W1 2322.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Computing Research Repository, arXiv:2005. 14165, 2020.
[6] Deng Cai and Wai Lam. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290-1301, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/2853///22200aclllll 119. URL https: : / aclanthologyyy org/ 2020. acl-main. 119.
[7] Shu Cai and Kevin Knight. Smatch: an evaluation metric for semantic feature structures. In Pro- ceedings of the 51 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https::://// /www./ww....... ac web. org/ att.tnttohoooyyoo htt.o/l/ooyp1111111111
[8] Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. Semantic parsing with dual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 51-64, Florence, Italy, July 2019. Association for Computational Linguistics. doi: (0.18653//1/P19-1100 URL ht https::p/// /www. aclweb.whh...... org/ http://nthooooooooooooooooo (ttpllloyyyyyy/laeeeegee 1007.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.033. 003344, 2021. URL https: /ttp///aavvvvvv. org/ /
03374.
[10] Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li. Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid Ó Séaghdha, and Anders Johannsen. Conversational semantic
10

---

parsing for dialog state tracking. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 8107-8117 8117, Online, November 2020. Association for Computational Linguistics. doi: 10.116633/1/20000.o 100m https: accl anthologyy www. 2020. emnlp-ma 651.
h0tp:/00m00mmmmmam......... URL
[11] Jay Earley. An efficient context-free parsing algorithm. Communications of the ACM, 13 (2):94- 10294-10,,00 1970. doi: 10.1145/362000.3322355. URL https: http::::/ddi..rg/1...o
362007.362035. 362035.
[12] Daniel Femández-Gonzálezz and Carlos Gómez-Rodríguee Enriched in-order linearization for faster sequence-to-see -o-uuuence constituent parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4092-4099, Online, July 2020. Association for Computational Linguisties. doi: 18853/11/2200 a00020..cllmman main. URI. https: / ac antho org 2020 acl-ma .tm.ann........... 376.
[13] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with mesdiiinttdisettanglle embedding sharing. CoRR, URL http:: /tttpp::///aiiiii //tp///aavvvv//aa//aa//eeaeaawea org/ abs/ 2111.111 09543.
2021.
[14] Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2676-2686, Melbourne, Australia, July 2018, Association for Computational Lin- guistics. doi: 10.18653/v1/118 1249. URL https: https////laaaatoooaao.ee org/Pl Pat//18---0 1 1249.
[15] Petrov Koo, S Petrov, Sutskever, GE Hinton, O Vinyals, and L Kaiser. Grammar foreign language. In Advances in Neural Information Processing Systems, 2015.
[16] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace. editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611-626.... ACM, 2023. doi: 10.114573600006636113333 361 3165.
httt:p::/aod..egg/00111155533 URL https: org http:/1..11111///000
[17] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.1166/11/2200..0 acl- mainm 703. URL https: / www. acl web. org/ anthology /200 acl -main. 703.
[18] Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta. Sonal Gupta, and Yashar Mehdad. MТOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2950-2963 Online, April 2021. Association for Computa- tional Linguisties. doi: 10. hatp:///mmmm....mmmmmmma org/ 22t/2221...2 eacl-main 257.
URL https: http:///aaaaaaloca//////
[19] Zuchao Li. Jiaxun Cai, Shexia He, and Hai Zhao. Seq2seq dependency parsing. In Proceedings of the 27th International Conference Computational Linguistics, pages 2203-3214, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https:
/ //aclanthology org/ c18-1178111
[20] Perey Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen
11

---

Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, 2022.
URL https:://///... /atpivaaaavv../.a///ai http://wwiiiaaaaaaaaa////a/o /a////b////// 2211. 0911c.
[21] Jiangming Liu and Yue Zhang. In-Order Transition- Constituent Parsing. Transactions of 11 2017. ISSN 2307-387X. doi: 10.1162/tacl 00070. URL https: htt.//iioooorg// 1 2al111//tcc// 0000000
the Association for Computational Linguisties,
[22] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /692, 2019.
[23] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewiez. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)313-330, 1993. URL https: http//aaaaaaoooooo...oo.o.o
org/393 2004.
[24] Alireza Mohammadshahi and James Henderson. Recursive non-autoregressivv graph-to-graph transformer for dependency parsing with iterative refinement. Transactions of the Association for Computational Linguistics, 9:120-138, 2021. doi: 10.1162/tacl a 00358. URL https: // //aclanthology wttp//2...aall1...... tacl-1.8.
[25] Jianmo Ni, Gustavo Hernandez. Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/.18//221/2022. mudings-aall114 URL https: / wwc.antthooog...org//aaaaa wtn.://......o222........nnnon-. acl 146.
[26] Panupong Pasupat, Yuan Zhang. and Kelvin Guu. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 7683-7698, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 00.18653/v1/022 wttp///eeml2mmmaaiii..i www.lppmmiinn600 URL https:ttp///// /acpaaaaaantooyy..aaaroo org/ emm. e
607.
[27] Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. Value-agnostic conversational semantic parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 3666-3681, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/200 lcl- URL https: / /acl anthology org/ orgg/021110.0
2021 acl- 284.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL html.
[29] Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: //axxiv.. org/ abs /198 10084.
[30] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. CoRR,
2021. URL https: / aaari.iiivv...... /bb /1///1111..0 08633.
[31] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2021.
12

---

[32] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lin- guisties. doi: 10.1 1221663/11/2221111 2 mannpp-min..79. URL https: atc/ccaaattttt http:///aaaaaahlaaaeceee org/
2021. , wwn main. 779.
[33] Nathan Schucher, Siva Reddy, and Harm de Vries. The power of prompt tuning for low-resource semantic parsing. CoRR, abs/2110.08522 2021. URL https: www.a.xiivvviio.a.a/aaaaael httpaaaiiii.g.aaaaaa//. 2110. 08525.
[34] Satoshi Sekine and Michael Collins. Evalb bracket scoring program. URL: http://www: cs. nyu. eeluees/pnjecs//pooeeeseeeedd 1997.
[35] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596 4604. PMLR, 10-15 Jul 2018. URL http::: httphhtss/sprcceediinogo http:///ooceeeeeeg.eee.. ngs. ml press/
v80/shazeed 8a. html.
[36] Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Con- strained language models yield few-shot semantic parsers. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lin- guistics. doi: 10. 10.18653/v120211 wttp:////222...mmmaa....... 2021. emn 1ommnimmnnn
URL https: ac anthology....... .rrg/g//
608.
[37] Aarohi Srivastava, Abhinav Rastogi, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv, abs/2206.046155. 2022.
[38] Yuanhe Tian, Yan Song. Fei Xia, and Tong Zhang. Improving constituency parsing with span attention. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1691-1703, Online, November 2020. Association for Computational Linguis- ties. doi: 10. 8653/v1/2020 1t00:///222...nnnggeeeemmm findings-emmlp.155.. URL https: https:/t///att https///aaaaaahhlloagoe org/ 2020 findiiiiiinnn nmnnnnnlsseepnnmmm 153.
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CORR, abs/2307. A00222307.09288, 2023. doi: URL ht htt /ooi/ooooo/oo/ooooo org/ .4854488500 arxiv. ..t.xiivvv000 2307.09288.
[40] Rik van Noord and Johan Bos. Dealing with co-reference neural semantic parsing. In Proceedings of the 2nd Workshop on Semantic Deep Learning (SemDeep-2), pages 41-49, Montpellier, France, September 2017. Association for Computational Linguistics. URL https: //aclanthology org/.17-7306..
[41] Rik van Noord and Johan Bos. Neural semantic parsing by character-based translation: Experi- ments with abstract meaning representations. Computational Linguistics the Netherlands Journal, 7:93-108, 2017.
13

---

[42] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 18653//1/W188-4446 URL https: /aa//alaantttt /aclantlooooooy
org /118- 6.
[43] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for peuproe- language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::tp... : http:///ooceeeedeecec .pf. 201
neurips cc/ cc/paper////eeeeeceeeceeeeeee
[44] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeTT5:
Identifier-awar unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 11/8663//1/22211 http:/...n.21...p...m.// mapp-maii....... URL https: http::///aaaaaahoo/o org 2/22/0211111 emnlp-main 685.
[45] Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics Natural Language Processing (Volume 1: Long
and the 7th International Joint Conference Pupers), pages 1332-1342, Beijing, China, July 2015. doi: 10.3115/:1//5--122.. URL
https: / www a www.bw...... org/ http://nnhhlogyyooooo patlloggy//1112333
[46] Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. CoRR, abs/2109.09193, 2021. URL https: //ap::::////xvva/////
arxiv. wwtpsiiiiorr//aaaaaa abs/ 2109.0.0. 09193.
[47] Sam Wiseman and Alexander M. Rush. Sequence-to-sequence learning as beam-search optimiza- tion. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing, pages 1296-1306, Austin, Texas, November 2016. Association for Computational Lin- guistics. doi: 10. 10.18633/v1/1166 137.. URL https : / / aclanthology. org/D16-111 1137.
[48] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga. Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. EMNLP. 2022.
[49] Songlin Yang and Kewei Tu. Bottom-up constituency parsing and nested named entity recogni- tion with pointer networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Lingnistics (Volume 1: Long Papers), pages 2403-2416. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 00.18653.w1/2022.acl-loooo 171. URL https: / ac http:///laaaaattc....../.o /rg/20222 acl- acl- 171.
[50] Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 391 1-3921, Brussels, Belgium, October-November 2018. doi: 10.18653/v/DDI1--4222. URL https::///// www.
aclweb. org www./wwwwww.bb...////ww/w//ww...
[51] Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang. Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga. Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li. Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: conversational text-to-SQL. challenge towards cross-domaie natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE) pages
14

---

1962-1979, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19 12244 URL attts::/aclanthhen www.tthoooooooooo/11111 2aag/DD1-2200 1204.
[52] Daniel Zeman et al. Universal dependencies 2.11, 2022. URL http: / /hdl. handle. pet//1234/1-11 1923.. LINDAT/CLARIAIICC/ digital library at the Institute of Formal and Applied Linguistics (ÚFAL.), Faculty of Mathematics and Physics, Charles University.
[53] Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen. Stack-based multi-layer attention for transition maased dependency parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1677-1682, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18663//1/DI7-11755 URL
https: /cacl///ccccc anthol ogy org D1 117155
[54] Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-sql with distilled test suite. In The 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2020.
15

---

A Variance of Results
All low-resource results are mean of three training data splits. Table 1I reports the average standard deviation for each model over the three low resource splits. We find a high standard deviation of GPT-3 and Codex; one of the factors being the small size of the set (100 examples). Fine-tumed models show relatively low variance, consistently having standard deviation lower than 2%.
Table 11: Standard deviation of the scores for each language model over the three low resource splits.
Model
Avg. Standard Deviation
GPT-3
Codex
3.2
T5-base
CodeT5-base
1.1
BART-large
1.4
T5-large
2.0
T5-3B
1.5
B
Compute Details
We used Microsoft Azure to run all our experiments. For experiments with T5-base, CodeT5-base and BART-large, we used single VI00 GPU with 32 GB memory. We used V100 GPUs for fine-tuning T5-large and 4 GPUs for fine-tuning T5-3B. Training time ranged from 2 to 6 hours depending on the size of the model.
C Dataset and Model License and Privacy Implications
The licenses of the datasets included in BenchCLAMP are listed in Table C. We could not find a license for MTOP, but the dataset was made freely available by the creators. Our benchmark is available under the MIT licence at https:://gttie wwt.////ccoooooot///aaaaaaaaaaaa
rineeerrrindddde In.
The datasets in BenchCLAMP either use sentences from news articles or utterances generated by crowd-workers following instructions to simulate user interactions. No real user interaction data was used to generate these datasets. As result, there is no privacy risk in using these datasets.
For fine-tuning experiments, use pre-trained models from Huggingface. All models were released under the Apache 2.0 license.
Table 12: Licenses of datasets covered by BenchCLAMP
Dataset
License
SMCalFlow [1]
CC BY-SA CC 3.0
TreeDST [10]
MTOP [18]
Overnight [45]
CC BY-SA 4.0
Spider [50]
CC BY-SA
CoSQL
CC BY-SA 4.0
AMR 2.0 [4]
LDC License
PTB 3 [23] (Constituency parsing)
LDC License
UD-EWT [52] (Dependency parsing)
CC BY-SA 4.0 International
16


Reviews:
[{"review_id": "wdTKi2mHo0", "reviewer": "Reviewer_T3oy", "summary": "", "questions": "", "limitations": "Yes, the authors have discussed limitations in Section 7.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Similar to UnifiedSKG, this paper unifies several syntactic and semantic parsing benchmarks as a seq2seq task, so that LMs can be evaluated on such tasks directly. Such unification is desirable since current LMs behave like general problem solvers that output sequences. BenchCLAMP provides helpful resources for evaluating LMs on parsing tasks.  \n2. The authors release context-free grammars for the proposed benchmark, which sets up a standard way to use constrained decoding during evaluation.  \n3. BenchCLAMP features different eval settings with different number of training samples.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TSq6e3NdW8", "reviewer": "Reviewer_1bZU", "summary": "", "questions": "", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper is well-written and easy to understand. For example, the preprocessing for each dataset is described clearly.\n\n2. This paper conducts a comprehensive model comparison, including five encoder-decoder models that require fine-tuning and two prompt-based models. In addition, it is shown that there is a significant improvement when grammar constraints are added to the models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Xe5gQIl26z", "reviewer": "Reviewer_pzWP", "summary": "", "questions": "", "limitations": "(everything is fine)", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Constraining LMs, in general and in particular for parsing, is a very promising research direction. I believe that this benchmark will be useful to the NLP community.\nIn addition, the experiments are interesting and well presented.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0pxmPKaT9s", "reviewer": "Reviewer_ZUCA", "summary": "", "questions": "", "limitations": "The limitations section of the paper is fair and well-written", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The collection of tasks included in BenchCLAMP gives a comprehensive picture of the performance of a language model for constrained generation. \n* The methodology for constructing the datasets and fine-tuning and evaluating is well-motivated.\n* The paper shows that fine-tuned encoder-decoder language models remain the best performing models for semantic and syntactic parsing, ahead of pretrained large models like GPT-3. This suggests an interesting target for improving language models. These models also perform close to or better than state-of-the-art parsing methods on the datasets included in BenchCLAMP.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper assembles a set of 7 semantic parsing and 2 syntactic parsing datasets, together with CFGs for parsing their meaning representations, and constraint decoding code to generate valid outputs from these grammars. The linearize the meaning representations for use with autoregressive sequence models. They provide evaluation code for LMs that are fine-tuned or are prompted, and provide different splits for training/evaluation for the purpose of standardization. They test 7 LMs on this benchmark - including GPT-3 and Codex. 

Strengths:
1. Well written, easy to read, well motivated, and good limitations section.
2. Comprehensive i.e. Includes both seq2seq and decoder-only models, and includes models that are fine-tuned, as well as models that are prompted. 
3. Has a strong takeaway - i.e. fine-tuned seq2seq still outperform pretrained models + prompting on semantic/syntactic parsing tasks. 

Weaknesses:
1. Newer models such as Llama, ChatGPT, GPT-4 are not included. The authors have promised to include Llama in their camera-ready, and are unable to present results from the OpenAI models since the returned output is insufficient to perform constrained decoding. This is a good resolution of this concern by the authors. 
2. An analysis of how performance changes with model scale would improve the impact of this paper. 
3. Models can perform end-tasks today without being aware of syntactic/semantic parses, so this might limit the utility of this benchmark. Nevertheless, these classic tasks remain of interest to the NLP community, so this may not be too big of a concern.
Decision: Accept (Poster)