Title: BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing
Summary: BenchCLAMP evaluates Constrained LAnguage Model Parsing on syntactic and semantic parsing tasks. It provides context-free grammars using both prompt-based learning and fine-tuning across different data resource settings.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: syntactic parsing, semantic parsing
Phenomenon defined by authors: Yes
Phenomenon definition: "Parsing tasks are generally not considered a testbed for such evaluation. The outputs of parsing tasks are structured objects such as parse trees or code. State-of-the-art systems thus involve task- or dataset-specific model architectures and target representation constraints. Evaluating language models on parsing tasks test capabilities not captured by commonly used evaluation tasks."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Generating syntactic or semantic representations from natural language inputs, which includes context-free grammars for semantic and syntactic parsing datasets, as well as a constrained decoding interface to generate only valid outputs covered by these grammars.
Task item definition: Natural language utterance; structured output representation (constituency parse tree, dependency relations, formal meaning representation)
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: While not all examples are derived from real user interactions, they are designed to represent realistic use cases for parsing.
Dataset size: train: 5500
Dataset size notes: train:5500, dev: 550
Dataset metadata available: Yes
Dataset metadata detail: dataset type, output formalism, evaluation metric
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean scores across different data splits and standard deviation for low-resource settings
Metric subscores: Yes
Metric subscores detail: Data resource settings; parsing datasets; constraint settings 
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding