Title: XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations
Benchmark: XSEMPLR
Summary: The paper introduces XSEMPLR, a unified benchmark for cross-lingual semantic parsing featuring 22 natural languages and 8 meaning representations by examining and selecting 9 existing datasets to cover 5 tasks and 164 domains. They use XSEMPLR to conduct a benchmark study on a wide range of multilingual language models, including encoder-based models (mBERT, XLM-R), encoder-decoder models (mBART, mT5), and
decoder-based models (Codex, BLOOM). The findings show that large multilingual
language models are still inadequate for performing CLSP tasks. They also find that the performance gap between monolingual training and cross-lingual transfer learning is still significant for multilingual models, though it can be mitigated by cross-lingual few-shot training.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon:  cross-lingual semantic parsing
Phenomenon defined by authors: Yes
Phenomenon definition: Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs).
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task is to train a model to convert a sentence in natural language into a meaning representation (e.g., SQL, programming code, Prolog, Functional Query Language, etc.).
Task item definition: A pair of input and output where input is a text in natural language and output is a text of input's meaning representation
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Train set: ~42k, test set: ~7500, Dev set: ~5500
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality