Title: BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer
Benchmark: BUFFET
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Few-shot Cross-lingual Transfer
Phenomenon defined by authors: Yes
Phenomenon definition: Few-shot cross-lingual transfer is defined as the ability to adapt models to a task in a new language using a limited number of training data in the target language. 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Converting all tasks into a unified text-to-text format, where models must generate appropriate outputs given inputs with k-shot examples in target languages.
Task item definition: Instruction, k-shot training and validation examples, test examples (input text and expected output)
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: Yes
Dataset metadata detail: language, task type, data curation method, output format, languages per task
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: hierarchical averaging
Metric statistics: Dataset score is calculated as a macro-average of the per-language score.
Metric subscores: Yes
Metric subscores detail: language, dataset, task type, resource level, fine-tuning/ICL
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / In-context Learning
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / In-context Learning