Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yu Ying Chiu
authors: Liwei Jiang
authors: Bill Yuchen Lin
authors: Chan Young Park
authors: Shuyue Stella Li
authors: Sahithya Ravi
authors: Mehar Bhatia
authors: Maria Antoniak
authors: Yulia Tsvetkov
authors: Vered Shwartz
reviews: 
Comments: This paper proposes CulturalBench, an AI-assisted red-teaming collected dataset. The dataset aims to help improve the cultural understanding ability of LLMs, which is so meaningful. Besides, the datasets is collected by local people, covering lots of areas and topics. The experiments in the paper also improve the effectiveness of the dataset.
Strengths: 1. The paper proposes a pipeline for collect cultural data leveraging AI-assisted red-teaming.

2. The human collecting process is professional and considerable.
Weaknesses: 1. I am confusing about the experiment in the last part of sec.5.1. You supposed that models perform better on the easy version because models can guess the answer of some questions without seeing the questions. Your experiment is to compute the semantic similarity of the cultural name with each answer. I think it may only prove the cultural relativity of each pair. A better exploration may be to control the variables. For example, disturb the choices of each question to change their cultural relativity, and see the new results. For example, you could conduct this additional experiment of randomizing answer choices and re-evaluating model performance, to more conclusively demonstrate whether models are relying on cultural relativity heuristics.

2. Can CulturalBench helps to enhance models' cultural understanding? More experiments on prompting or fine-tuning may be more interesting. For example, you can fine-tune models using your data and conduct in-context learning and other prompt engineerings.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The authors have created a cultural knowledge benchmark, CulturalBench, through AI assisted interactive red-teaming.
The major contribution I would consider to be relevant to HCI aspects, of how they setup and conducted the benchmark curation using human annotators and with AI assistance.

I believe such work fits better in HCI focused conferences.

Some other recent work (such as CultureBank) leveraged social networks (reddit, twitter etc.) to curate similar datasets and benchmarks. One can argue that collecting such data indirectly provides wider representation, especially in multicultural regions, which the authors of this paper briefly mention but dont go into much details. I.e., when thinking about culture knowledge of a country/region that is largely multi-cultural, which aspects play a large role or how those multiple identities (county of origin, ethnicity, religion, current place of residence) interplay to define what is the 'culture' of this region?
Strengths: Devised an effective AI assisted framework for engaging human annotators in curation of cultural knowledge benchmarks.
The paper is well written and the AI assisted framework holds promise to increase annotators involvement/engagement.
Weaknesses: The main focus of this paper is in describing the human annotators setup with AI assisted implementation.

It lacks some of the comparison with other cultural knowledge benchmarks (such as CultureBank) that were generated using other approaches and as such, lacks the discussion of how this approach is better and what are the limitations comparing to others.

As alluded by the authors, no fine-grained (and I would add, no consideration for multiplicity of identities) cultural classification - which is mentioned to be a limitation of the annotation platform used -- that may have non-neglected impact on the resulted benchmark.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3