Title: Understanding and Mitigating Language Confusion in LLMs
Benchmark: LCB
Summary: The paper introduces a benchmark to measure language confusion in LLMs. They investigate language confusion on the line and word level in two practical settings: a) Monolingual generation, where a user queries the LLM in a given language, implicitly requesting an answer in the same language; and b) cross-lingual generation, where a user explicitly instructs a model to generate text in a different language.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Language Confusion
Phenomenon defined by authors: Yes
Phenomenon definition: LLMs are often unable to consistently generate text in the user’s desired language, or the
appropriate language given the context. They call this category of error “language confusion”.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: They investigate language confusion on the line and word level in two practical settings: a) Monolingual generation, where a user queries the LLM in a given language, implicitly requesting an answer in the same language; and b) cross-lingual generation, where a user explicitly instructs a model to generate text in a different language.
Task item definition: A sentence (prompt)
Task source: Modified from another benchmark (e.g. translation into another language), For some part of the data they include human generated prompts 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 7100
Dataset metadata available: Yes
Dataset metadata detail: Language of the prompt and the original data source 
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Metric definition: The paper introduces 2 new metrics for language confusion. Line-level pass rate (LPR) and Word-level pass rate (WPR).
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality