reviews: 
Comments: This paper introduces CULTURALBENCH, a new benchmark for evaluating large language models' (LLMs) cultural knowledge across 45 global regions, including underrepresented areas like Bangladesh, Zimbabwe, and Peru. The benchmark consists of 1,227 human-written, human-verified questions spanning 17 cultural topics, such as food preferences and greeting etiquette. The authors propose two evaluation setups: CULTURALBENCH-Easy and CULTURALBENCH-Hard, which feature the same questions phrased differently to assess the models' robustness to variations in question formulation.

The paper introduces a comprehensive, diverse benchmark to assess LLMs’ cultural understanding across regions that are often neglected in existing datasets. By using two different question setups, the authors highlight that LLMs' performance is sensitive to how questions are framed, with differences as high as 27.3% observed in GPT-4o. The authors compare LLMs, including GPT-4o and Llama3-8b, against human performance. While human annotators achieve 92.6% accuracy, the best LLM (GPT-4o) reaches only 61.5% accuracy on the more challenging setup, revealing significant gaps in cultural knowledge. The paper identifies specific shortcomings in LLMs, particularly in regions like South America and the Middle East, as well as difficulties in handling questions with multiple valid answers.
Strengths: The authors collect a culture-related question-answering dataset by using a pipeline consisting of three steps: 1) Red-teaming data collection; 2) Human Quality Chek 3) Filtering. The proposed datset is verified by five annotators. There are several merits of this dataset:

1. The dataset is divided into CulturalBench-easy and CulturalBench-hard, which are more differentiable than current culture-related datasets, covering more diverse topics and more challenging questions.

2. The model's performance on CULTURALBENCH-Hard is far behind human performance, where GPT-4o achieves the best performance.

3. The authors propose some interesting analysis on different providers' LLMs on questions from different regions. For example, they find that models perform better in questions relating to North America, South Asia, and West/South Europe than South America, East Europe, and the Middle East. Meanwhile, Qwen-2-72B and Mistral Large do not perform well in answering their own region's cultural knowledge.
Weaknesses: My major concern with this work is the form of the multiple choice question. As many researchers agree and argue before, LLMs usually show unstable and unpredictable performance when are required to answer multiple choice questions. Why not design some generation tasks to evaluate LLM's capability towards culture-related questions?

While culture-related question-answering problem is not novel at the moment given the previous similar work, including CultureLLM, CultureBank, and SeaLLM. I am wondering whether the authors could propose more diverse data types, not limited to the multiple-choice format. How do your multiple-choice formats compare to or improve upon the formats used in these works?

In addition, the experimental results are not counterintuitive enough. For instance, GPT-4o leads in most regions, Models score lower in questions relating to South America, East Europe, and the Middle East, etc. More in-depth analyses are expected to be done.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3