Title: SafeWorld: Geo-Diverse Safety Alignment
Benchmark: SAFEWORLD
Summary: The paper introduces a benchmark to evaluate LLMs' ability to generate culturally sensitive and legally compliant responses across diverse global contexts. It also proposes a multi-dimensional automatic safety evaluation framework for assessing the contextual appropriateness, accuracy, and comprehensiveness of responses. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: alignment with geo-diverse safety standards
Phenomenon defined by authors: Yes
Phenomenon definition: "ability to respond appropriately, precisely, and helpfully to queries involving a culturally or legally sensitive content" (page 4)

with cultural and legal safety defined as: 

"Cultural safety defines an environment that is spiritually, socially, emotionally, and physically safe for people [44]. It is about adhering to cultural and social norms, which dictate appropriate scenario within a society." (page 4)
"Legal safety refers to abiding the policies enacted by governments, with each country having its own set of regulations designed to maintain social order and stability. These rules establish standards for acceptable scenario, resolve conflicts, and protect the rights and well-being of individuals and communities." (page 4)

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are four types of tasks, each consist of a scenario -- illustrating a culturally or legally sensitive or insensitive -- and a question. Depending on the task, the LLM is expected to answer or not answer, identify the insensitive scenario / guideline violation, etc.
Task item definition: Query (scenario + question), type of query, ground-truth norm/policy
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 2,342
Dataset size notes: Train: 45,746
Dataset metadata available: Yes
Dataset metadata detail: query type, ground truth norms/policies, country/region
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Alignment / Safety
New taxonomy: General Capability / Safety