Title: BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Benchmark: BLEnD
Summary: The paper introduces BLEND, a novel benchmark comprising hand-crafted question-answer pairs designed to evaluate LLMs on everyday cultural knowledge across 16 countries/regions and 13 languages, including low-resource ones. It demonstrates significant performance disparities among models, showing cultural and linguistic biases, especially in underrepresented regions.
Contribution: answer format: short-answer and MCQ, 52.6k question-answer pairs, BLEND includes 500 question templates that reflect daily life aspects across six socio-cultural categories: food, sports, family, education, holidays/celebrations/leisure, and work-life.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon:  cultural knowledge and multilingual cultural commonsense understanding
Phenomenon defined by authors: Yes
Phenomenon definition: knowledge of everyday cultural practices that are specific to different countries and regions. This includes understanding what people commonly do, eat, or experience in their daily lives within a specific cultural and linguistic context. Specifically, dimensions such as food, sports, celebrations, education, family, and work-life are considered.

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is to evaluate large language models on their ability to correctly answer short-answer and multiple-choice questions about everyday cultural practices from various countries and regions, using either local languages and English. Human evaluation is conducted on short-answer questions with annotators coming from the tested regions.

Task item definition: "Al-en-06": {
"question": "대한민국 학교 급식에서 흔히 볼 수 있는 음식은 무엇인가요?",
"en_question": "What is a common school cafeteria food in your country?",
"annotations": [
{
"answers": [
"김치"
],
"en_answers": [
"kimchi"
],
"count": 4
},
{
"answers": [
"밥",
"쌀밥",
"쌀"
],
"en_answers": [
"rice"
],
"count": 3
},
...
],
"idks": {
"idk": 0,
"no-answer": 0,
"not-applicable": 0,
"others": []
}
},
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 52.6k
Dataset metadata available: Yes
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean, Anova for p-values, Tukey-HSD
Metric subscores: Yes
Metric subscores detail: by language (native and English)/country (region)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: for short-answer questions, there is a human evaluation, which to some extent can represent the validity of the questions
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge