Title: A Benchmark for Learning to Translate a New Language from One Grammar Book
Benchmark: MTOB
Summary: This paper introduces MTOB, a benchmark for learning to translate between English and
Kalamang—a language with less than 200 speakers and therefore virtually no presence on the web—using several hundred pages of grammatical reference materials. This task framing is novel in that it asks a model to learn a language from a single human-readable book of grammar explanations, rather than a large mined corpus of in-domain data. While LLM baselines do not yet match human performance, their experiments show a clear trend that increasing LLM quality and context window size improves translation quality.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: translation between English and Kalamang
Phenomenon defined by authors: No
Phenomenon definition: The ability to translate between English and Kalamang
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: LLMs are asked to translate a sentence from/to English to/from  Kalamang. The experiments are done in zero-shot and few-shot settings.
Task item definition: A pair of sentences 
Task source: The examples are created by a linguist 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: test set: 100
Dataset size notes: train set: 400 
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Unknown 
Metric statistics: Unknown
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Language Modelling / In-context Learning
New taxonomy: General Capability / In-context Learning