reviews: 
Comments: The paper describes work on developing a high-quality benchmark called CulturalBench for measuring cultural knowledge of LLMs. The benchmark contains a total of 1,227 high-quality and human-verified English prompts covering culture-based questions from 45 global regions and split into two versions of the benchmark: CulturalBench-Easy and CulturalBench-Hard. The paper introduces CulturalTeaming which is the process of constructing high-quality generation questions from cultures and validation based on majority voting. The paper highlights differences of CulturalBench from existing culture-evaluating benchmarks including quality checks for all 1,227 questions, discovery-based topics, and higher topic coverage. One particularly finding that stood out for me is that model providers' flagship models were not able to obtain higher performance in cultural knowledge relating to their region.
Strengths: The paper is well-written and easy to read. I particularly appreciate the level of completeness of the paper and how it packages its contributions in terms of benchmarking cultural knowledge of LLMs. I find the CulturalTeaming process used to ensure quality of questions crafted by humans towards specific cultural practices as part of the benchmark to be one of the paper's strengths. The author/s highlight this difference compared to existing benchmarks where instances are not verified iteratively and strongly through majority voting. In terms of results, the paper has interesting findings such as learning that model providers from specific regions (ex. Qwen from Asia/China) seem not to meet the expectation that it will emerge as the best performing model for East Asian culture assessment.
Weaknesses: Some of the issues I find from the paper are as follows:

1. **Small test instance count which has certain disadvantages.** 1,227 test instances may be a good initial overview of performance but not enough to tell something about an LLM’s true/reliable understanding of geographic-specific culture (ex. South Europe or Oceania) as it only contains 20-100 test instances (even less if you revert it back to its original question-type form). The quality control in developing the benchmark is an advantage (and I think this should be a common practice that other benchmarks can replicate) but I’m concerned about the practical usability of said benchmark as the trend in evaluating culture understanding is more favorable towards depth on a specific global region than a catch-all/one-size-fits-all evaluation.

2. **Selective results using commercial models.** The authors did perform in-depth analysis of the question type and versions of models in Section 5.2 but only for commercial models. This does not make sense to me as there is very little to know about these models due to their closed nature. It would make the paper stronger if the same analysis is done on more transparent/open-weight models used like Llama, Qwen and even models not included in the experiments but are made with culture in mind such as Aya models by Cohere. 

3. **Lack of discussion on how to grow and develop the benchmark.** For culture knowledge benchmarks like this one, it might strengthen the paper if the authors provide sufficient and convincing discussion on how the community or any plans to grow and develop the benchmark, given its small number of test instances. This can be in the form of improving the framework in Figure 1. Without this, the benchmark will only cover a snapshot of cultures of different geographical areas which, again, makes its real-life usage questionable.

4. **Wrong accusation on poor diversity of previous benchmarks**. I oppose the author/s statement to label topic-specific benchmarks focusing specifically on food, social etiquette, etc as "narrow" and of  "poor diversity" in the related work section. As a reader, the tone of the paragraph comes as negative. While these benchmarks cover more specific aspects of culture, they have their own application as to why it is important to benchmark LLMs on different specific cultural aspects like food. They are considered semantic proxies of culture by a survey paper (Adilazuarda et al., 2024), which is a good starting point for understanding how LLMs capture cultures. This is not a negative connotation per se. Moreover, I would argue that it is better to have a benchmark that prioritizes depth on a specific aspect of culture but provides solid, empirical insights on that direction than a benchmark that tries to "catch-all" cultures but obtains very limited unrepresentative findings from small test instances.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3