The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities
Authors: Harsh Nishant Lalai
Raj Sanjay Shah
Jiaxin Pei
Sashank Varma
Yi-Chia Wang
Ali Emami
Venue: COLM 2025 (2025)
Keywords: geographic representation
LLM evaluation
fairness and bias
reasoning capabilities
cross-cultural NLP
interactive question answering
TL;DR: LLMs demonstrate geographic bias in entity deduction games, with notably better performance on entities from the Global North and West despite controlling for language, popularity and frequency factors.

Abstract: Large Language Models (LLMs) have been extensively tuned to mitigate explicit biases, yet they often exhibit subtle implicit biases rooted in their pre-training data. Rather than directly probing LLMs with human-crafted questions that may trigger guardrails, we propose studying how models behave when they proactively ask questions themselves. The 20 Questions game, a multi-turn deduction task, serves as an ideal testbed for this purpose. We systematically evaluate geographic performance disparities in entity deduction using a new dataset, Geo20Q+, consisting of both notable people and culturally significant objects (e.g., foods, landmarks, animals) from diverse regions. We test popular LLMs across two gameplay configurations (canonical 20-question and unlimited turns) and in seven languages (English, Hindi, Mandarin, Japanese, French, Spanish, and Turkish). Our results reveal geographic disparities: LLMs are substantially more successful at deducing entities from the _Global North_ than the _Global South_, and the _Global West_ than the _Global East_. While Wikipedia pageviews and pre-training corpus frequency correlate mildly with performance, they fail to fully explain these disparities. Notably, the language in which the game is played has minimal impact on performance gaps. These findings demonstrate the value of creative, _free-form_ evaluation frameworks for uncovering subtle biases in LLMs that remain hidden in standard prompting setups. By analyzing how models initiate and pursue reasoning goals over multiple turns, we find geographic and cultural disparities embedded in their reasoning processes. 
We release the dataset (Geo20Q+) and code at https://sites.google.com/view/llmbias20q/home.

Published as a conference paper at COLM 2025
The World According to LLMs: How Geographic Origin Influ- ences LLMs' Entity Deduction Capabilities
Harsh Nishant Lalai* BITS, Pilani
Raj Sanjay Shah"
Jiaxin Pei Stanford University
Georgia Institute of Technology
Sashank Varma Georgia Institute of Technology
Yi-Chia Wang Stanford University
Ali Emami
Emory University
Abstract
Large Language Models (LLMs) have been extensively tuned to mitigate explicit biases, yet they often exhibit subtle implicit biases rooted in their pre-training data. Rather than directly probing LLMs with human-crafted questions that may trigger guardrails, we propose studying how models behave when they proactively ask questions themselves. The 20 Ques- tions game, multi-turn deduction task, serves
ideal testbed for this purpose. We systematically evaluate geographic performance dispari- ties in entity deduction using new dataset, Geo20Q", consisting of both notable people and culturally significant objects (e.g., foods, landmarks, animals) from diverse regions. We test popular LLMs across two game- play configurations (canonical 20-question and unlimited turns) and in seven languages (English, Hindi, Mandarin, Japanese, French, Spanish, and Turkish). Our results reveal geographic disparities: LLMs are substantially more successful at deducing entities from the Global North than the Global South, and the Global West than the Global East. While Wikipedia pageviews and pre-training corpus frequency correlate mildly with performance, they fail to fully explain these disparities. Notably, the language in which the exp game is played has minimal impact on performance gaps. These findings demonstrate the value of creative, free-form evaluation frameworks for un- covering subtle biases in LLMs that remain hidden in standard prompting setups. By analyzing how models initiate and pursue reasoning goals over multiple turns, we find geographic and cultural disparities embedded in their reasoning processes. We release the dataset (Geo20Qt) and code https:/ sites. gowgle.coogoooieieeeeo /view hete//viee//liliaaeee /llmbmmm00///aaaaa hooe.
1 Introduction
In response to growing concerns around bias and fairness, Large Language Models (LLMs) like Google's Gemini (Team, 2023), Meta's LLaMA (Touvron et al., 2023), and OpenAl's GPT-4 (OpenAl, 2023) have undergone extensive alignment to minimize explicit biases (Bai et al., 2024). Paradoxically, these alignment efforts have made subtle biases harder to detect: models now avoid overtly problematic responses but may still encode prejudices in their reasoning processes and knowledge hierarchies (Borah Mihalcea, 2024; Kumar et al., 2024; Lim & Pérez-Ortiz, 2024). As result, standard prompting-ban that only examine model outputs can give a false impression of fairness.
evaluations
To study such behaviors, we must move beyond surface-level probing with human- formulated questions, which often trigger guardrails and only reveal what models say, not how they reason. Instead, we propose a complementary approach: allow LLMs to proactively ask sequence of questions to deduce the answer and analyze their inquiry chains. By examining which questions models prioritize, how they navigate ambiguity, and what
"Equal contribution. (arrta/@gaechheddu (pedropei, yichiaw) yichiaw)@stanford..o @stanfordd.ddd diieemam@@eooyyedu
Emails: f2021 F20212665a 16665@gaa.bit--iiaaiii
(rajsanjayshah,

---

Published as a conference paper at COLM 2025
Effel
Success
Jack Ma
Fallre
Taj Mehrel
Figure 1: Country-level success rates of Gemini 2.0 + on entity deduction across 20 regionally specific entities (10 each for Things and Notable people ). Darker shades indicate higher deduction success rates. We also show illustrative examples highlighting performance gaps: the model successfully deduces entities like the Eiffel Tower and LeBron James while struggling with counterparts like the Taj Mahal and Jack Ma
hypotheses they generate when left to their own devices, we observe biases in their reasoning that remain hidden in standard evaluations.
The 20 Questions game provides compelling testbed for this form of evaluation (Zhang al., 2023). In this interactive task, different instances of an LLM play both roles, the guesser: attempting to identify an unknown entity by asking a sequence of yes/ res/no/ no/maybe questions, and the judge: responding to these queries based on know ledge of the target entity (Figure shows an example of a game). To succeed coherence, systematically narrow the hypothesis space, and ask strategically informative questions (Mazzaccara et al., 2024). This process reveals which features and associations the model prioritizes when reasoning about entities from different regions, with unique insights into its implicit geographic knowledge hierarchies and cultural assumptions.
guesser, a model must maintain multi-turn
In particular, we focus on principal axis for bias, ge- ographic origin, and ask: Do models demonstrate systematic differences in their ability to deduce entities based on geographic origin? As illustrated in Figure 1, our findings show sub- stantial regional variation in deduction success rates, with models performing notably better on Western entities like the Eiffel Tower compared to their Eastern counterparts like the Taj Mahal.
lving entty?
No
it hater monument?
Yes
located Bumpe?
To support this investigation, we introduce Geo20Q*, a new dataset of geographically diverse entities curated from Wikipedia. Unlike traditional 20 Questions datasets, which focus generic entities (e.g,, popcorn, cars, etc.), Geo20Q* consists of popular, geographically specific entities from around the world, including both culturally salient things (Things); e.g., foods, landmarks, animals) and notable indi-
Elftel Tower?
Bingel
viduals Notable people; e.g. politicians, academics, sport players, etc.). To our knowledge, this is the first geographi- cally balanced entity deduction dataset specifically designed to evaluate implicit bias in LLM reasoning.
An example 20 Questions game played by Gemini 2.0 for the entity "Eif- fel Tower."
Figure 2:
Note: (1) We use Al-generated cartoon representations of entities and are not intended to be accurate or factual depictions. (2) The map shown using standard projection; results hold regardless of projection choice, though equal-area projections may represent country sizes differently.
2

---

Published as a conference paper at COLM 2025
In addition to the standard 20-question limit (the canonical setting), we introduce unlimited-turn configuration that removes constraints on reasoning depth. While the 20-question setting naturalistic and faithful to the original game, it may prematurely truncate valid reasoning paths. The unlimited-tum setting allows models to follow deeper, exploratory inquiry chains, providing richer insights into deduction behavior and failure modes.
We organize our investigation around the following research questions:
RQ1: What is the top-line deduction performance of popular LLMs on our Geo20Q' dataset? How do they vary across entity types (Things vs. Notable people) and gameplay settings (canonical vs. unlimited turns)?
RQ2: Can entity popularity (Wikipedia pageviews) and pre-training corpus-base based frequency of entity explain the observed disparities?
RQ3: To what extent does the language in which the game is played influence the model's ability to deduce entities from different geographic origins?
RQ4: Do LLMs exhibit geographic performance disparities in their ability deduce entities, particularly between the Global North and Global South, and the Global West and Global East?
To answer these questions, we evaluate three widely used LLMs across multiple gameplay configurations and seven languages. Our findings reveal significant geographic disparities in model performance. LLMs consistently demonstrate superior ability to deduce entities from the Global North and West compared to those from the Global South and East. These disparities persist even when controlling for entity popularity and appear largely unaffected by the language of gameplay. Our analysis reveals geographic biases in model-initiated reasoning that standard evaluations miss, showing the value of creative benchmarking approaches for assessing fairness in LLMs.
2 Related Work
Recent years have seen growing attention fairness and bias in LLMs (Gallegos et al., 2024b), with research addressing various dimensions of social bias, including gender (de Vassimon Manela al., 2021; Dong et al., 2024), race (Nadeem et al., 2021; Yang et al., 2024), ethnicity (Ahn & Oh, 2021; Hanna et al., 2025), age (Nangia et al., 2020), and sexual orientation (Cao & Daumé III, 2020). While techniques like data augmentation (Gallegos et al., 2024a) and model alignment (Ouyang et al., 2022) have reduced overt biases, they've ve created a new challenge: models whose surface-level outputs appear unbiased but whose reasoning processes still contain implicit biases (Bai al., 2025; Zhao et al., 2025).
Bias and Fairness:
These subtle biases are particularly difficult to detect with traditional evaluation methods that rely on direct questioning or classification-based probes. Our work advances this literature by introducing novel evaluation paradigm that studies bias through model- initiated inquiry chains rather than responses to human-crafted prompts, allowing us to uncover biases that remain hidden in standard evaluations.
Entity Deduction and Interactive Evaluation: Deductive tasks involving ambiguous entities have been used to evaluate reasoning capabilities in various contexts, from dialogue systems (Aliannejadi et al., 2019) to vision-language models (Cho et al., 2021). Early interactive frameworks like ESP (von Ahn & Dabbish, 2004) and Peekaboom (von Ahn et al., 2006) demonstrated the effectiveness of deduction-based gameplay for collaborative tasks, while visual reasoning datasets like GuessWhat?! (De Vries et al., 2017) extended this paradigm to object identification through interactive dialogues.
In the context of LLMs, the 20 Questions framework has recently emerged as valuable probe for studying deductive reasoning (Zhang et al., 2023), decision-making (Bertolazzi et al., 2023), and question generation (Mazzaccara et al., 2024). However, prior work has primarily focused on using this framework to evaluate general reasoning capabilities rather than to detect biases. Our research represents the first application of the 20 Questions
3

---

Published as a conference paper at COLM 2025
paradigm specifically for uncovering geographic bias in LLMs, with new methodological approach to fairness evaluation through multi-turn, model-initiated reasoning.
Geographic and Cultural Representation in NLP: While research fairness has increas- ingly addressed demographic biases, geographic performance disparities remain underex- plored, with most studies focusing on Western contexts, particularly the U.S. (Yogarajan et al., 2023; Besse et al., 2022; Yin et al., 2022). The limited work in this area has typically measured surface-level representations, such entity coverage language support, rather than evaluating how models reason about entities from different regions (Gupta et al., 2023; Bhagat al., 2024; Li al., 2023).
Recent studies have emphasized the importance of geographic diversity in NLP evaluations (Manvi et al., 2024; Mirza al., 2024; Shafayat al., 2024; Schwöbel et al., 2023), but few have operationalized this concern through multi-turn reasoning tasks. Our work fills this gap by systematically evaluating how geographic origin shapes both success rates and reasoning efficiency in entity deduction.
Task Setup
This section describes the implementation of the 20 Questions deduction game, the eval- uation metrics for LLM performance, and the construction of our geographically diverse entity dataset designed to probe performance disparities.
3.1 Game Design
Our 20 Questions framework, adapted from Zhang et al. (2023), simulates the classic deduction game where two instances of the same base LLM play different roles: one as the "judge," (aware of the target entity) and one the "guesser," (attempting to identify the entity through strategic questioning). The game proceeds through two key configurations:
Canonical setting: The guesser has exactly 20 turns to correctly identify the entity, mirroring the traditional game format.
Unlimited-turn setting: The guesser can continue questioning until it either makes a correct guess, gives up, or reaches 150 turns an empirical upper bound based on performance gains plateau (see Appendix Figure 4).
Role Definitions:
Judge: Receives the target entity and the guesser's questions, responding only with "Yes," " "No," " or "Maybe". The judge confirms correct guesses with "Bingo!" and acknowledges when the guesser gives up. In the canonical setting, the judge prompts for a final guess on the 20th turn.
Guesser: Has no knowledge of the entity and must rely solely on the dialogue history to formulate informative questions and ultimately identify the target. The guesser must maintain coherence across turns while systematically narrowing the hypothesis space.
Since different LLMs may have varying training corpora and temporal knowledge cutoffs, we use two instances of the same model for the judge and guesser roles to ensure consistent knowledge access and eliminate cross-model mismatches. For example, an LLM might associate Lionel Messi with FC Barcelona, while another, trained on newer data, might link him to Inter Miami, leading to inconsistent game dynamics used together. All entities are based on Wikipedia articles, a source that is extensively used in the pre-training corpora of all LLMs under consideration. This design choice mitigates the risk that the judge (or guesser) completely lacks knowledge about a given entity, as these entities are highly likely to be present in the models' knowledge base. Next, all prompts follow each model's recommended formatting and include appropriate instructions to guide the guesser's 's inquiry strategy (full prompts in Appendix A.4.1). Our evaluation focuses on model performance in the guesser role, this captures the abilities we aim to assess.
Implementation Details:
4

---

Published as a conference paper at COLM 2025
3.2 Evaluation Metrics
We evaluate model performance using two primary metrics:
Success Rate (Accuracy): The percentage of games in which the guesser correctly identifies the target entity. This serves as our primary metric for evaluating more efficient reasoning.
The average number of turns required for the guesser to deduce the correct entity in successful games. Lower values indicate faster and more efficient reasoning.
Number of Turns to Answer:
We focus on these performance indicators rather than other potential metrics, for example, the number of "Yes" responses (Zhang et al., 2023), which we find to be unreliable. For instance, entities with broadly applicable attributes (e.g,, celebrities or landmarks) may elicit more "Yes" answers, simply because broader range of questions apply to them. In the Appendix A.6, we also examine early termination behavior, where models abandon the game after multiple turns with outputs like "I give up."
3.3 Geo20Q+: A Curated Dataset of Geographically Specific Entities
To evaluate geographic variation in deduction perfor- mance, we introduce Geo20Q", curated dataset of re- gionally specific entities spanning diverse locations. En- tities fall into two broad categories: culturally significant objects (reference: Things; ; e.g., foods, landmarks, ani- mals, and other culturally significant objects) and notable individuals (reference: Notable people; e.g. public fig- ures, artists, politicians). Figure shows the distribu- tion of categories for Things and professions amongst Notable people.
Other
Jitarsty
Landnar
16.3%
(a) Things categories
Falliin
2.4%
Entity Sourcing. For Things, we collect pages from the English Wikipedia across thematic categories such as tourist attractions, mountains, rivers, universities, an- imals, foods, plants, and cultural artifacts. Using tom scraper, we extract region-tagged pages within each category, rank them by pageviews, and collect up to 50 entities per country, yielding N = 8375 Things. For
Soot
(b) Notable people- professions
Notable people, we source data from the global notabil- ity dataset by Laouenan et al. (2022), which contains 2.1 million notable individuals from 172 countries. We rank individuals based on their English Wikipedia pageviews and choose the most popular individuals until every coun- try has at least 50 representatives, resulting in a total of N 24049 Notable people.
Figure 3: Distribution of en- tity types across (a) Things and (b) Notable people. Detailed breakdown in Appendix Fig- ure
Disambiguation and Filtering, Many entities have associations with multiple countries; for instance, Elon Musk has ties to South Africa, Canada, and the U.S., while saffron is cultivated in several regions. To assign a single country of origin, we query each LLM to identify the country with which the entity is most strongly associated, constrained to options explicitly mentioned on the entity's 's Wikipedia page. We retain only those entities for which all models agree on a primary country (approximately 88% of candidates), ensuring consistent geographic attribution. This filtering step reduces ambiguity and ensures that any differences in model performance reflect reasoning about clearly attributed entities, not inconsistencies in geographic labeling background knowledge in model training data.
5

---

Published as a conference paper at COLM 2025
Avoiding Contamination. Rather than using generic, high-frequency entities (e.g. "car," "fruit") that are likely to be memorized during pretraining, we focus on geographically specific and fine-grained instances (e.g., "Audi," "Alphonso mango"). This reduces the risk of models succeeding through rote recall and encourages evaluation of genuine deductive reasoning entity-specific knowledge.
Experimental Setup
4.1 Models evaluated
In our experiments, we evaluate the performance of widely used LLMs with strong scores on public benchmarks (Chiang et al., 2024). These include proprietary models such as 4P-----mmini (OpenAl, 2023), and 2.0 Gemini-2.0-Flash (Team, 2023), well as one leading open-source model, 0 Alama-3.--70-IIntttee (Touvron et al., 2023). All models claim strong multilingual capabilities and represent different architectural approaches and training methodologies. By including both proprietary and open-source models from different developers, we ensure our findings reflect broader patterns in state-of-the-art LLMs rather than idiosyncrasies of particular model family.
4.2 Multilingual Evaluation
To assess whether language influences geographic performance disparities (RQ3), we con- duct experiments in seven high-resource languages spanning multiple language families and writing systems: English, French, Hindi, Spanish, Japanese, Mandarin, and Turkish (Robinson et al., 2023). While some LLMs support multilingual capabilities more robustly than others, all models evaluated in this work explicitly claim support for these high- resource languages. In particular, we focus on high-resource languages because they are substantially better represented in pretraining corpora, helping to ensure minimum level of output quality across languages.
To play a game in language, first automatically localize the game setup, including using an LLM (GPT-40) to translate the judge and guesser prompts as well as the entity names to match the expected usage of the target language. Next, for each language, we ask at least one in-house native/ fluent speaker to review and, needed, modify the prompt templates, observe a few games, and verify the consistency of LLM behavior in gameplay (see appendix A.9). Finally, all native /fuent speakers confirmed the quality of translation and the functional integrity of the deduction game across all seven languages.
4.3 Geographic Granularity
To efficiently test our research questions while maintaining comprehensive coverage, we organize entities at two levels of geographic granularity:
Continent-Level Analysis (RQ1-3): For each of the six inhabited continents (Africa, Asia, Europe, North America, South America, and Australia), we select the 100 most-viewed entities from Wikipedia in each category (TTings and Notable people) This yields 1, 1,200 entities evaluated per model, with a total of 50,400 games across all models, languages, and configurations (3 models 1,200 entities 7 languages configurations).
Country-Level Analysis (RQ4): To examine geopolitical and cultural divides more precisely, we collect the 10 most popular entities of each type from 172 countries. We then organize countries into four key groupings:
Global North (23 countries): Economically developed regions, including North Amer- ica, Europe, and parts of East Asia (Odeh, 2010)
Global South (149 countries): Less economically dominant regions across Africa, Latin America, South and Southeast Asia (Odeh, 2010)

---

Published as a conference paper at COLM 2025
Global West (32 countries): Regions with predominantly Western cultural traditions,
primarily North America and Western Europe (Turner & Khondker, 2010)
Global East (142 countries): Regions with distinct non-Western cultural traditions,
including much of Asia and the Middle East (Turner Khondker, 2010)
These groupings allow us examine how both development-related disparities /North/- South) and cultural representation biases (West/Eas) might influence model performance. For more details regarding country grouping, refer to Appendix A.10.
4.4 Measuring Entity Prevalence
To determine whether performance disparities can be explained by entity prominence in training data (RQ2), we employ two complementary measures:
Wikipedia Popularity: We use Wikipedia pageview statistics as proxy for an entity's overall cultural prominence and information availability.
Corpus Frequency: We estimate how frequently models may have encountered entities during pretraining using the Dolma dataset (Soldaini et al., 2024) through the WiMBD project (Elazar et al., 2023). This 3-trillion token corpus drawn from diverse internet sources allows us to calculate frequencies that approximate representation data. Since the actual training corpora for our evaluated models are not publicly available, these frequencies serve as reasonable proxy for training exposure.
typical LLM training
Together, these measurements help us assess whether geographic performance disparities simply reflect data availability or suggest more fundamental biases in how models represent and reason about entities from different regions.
Findings
RQ1: Top-line model performance across entity types and gameplay settings
To establish baseline performance characteristics, we first examine deduction success rates and efficiency across models, entity types, and gameplay configurations.
Entity Type
Canonical gameplay
Unlimited turn
+20
Things
9.2 (16.6)
(25
10.7 (15.1)
(44.84
43.3 (43.1)
44.7
Notable people
24.8 (15.1)
35.7 (15.)
(16.1)
66.7 (34.5)
78.0
85.1
Average
17.0 (15.9)
24.1 (1.1)
23.4 (15.6)
(39.)
60.7 (36.6)
64.9 (40.8)
Table 1: Average Success Rates (in %) and Number of Turns to Answer (in parentheses) across different models. For example, in the Canonical setting and for Notable people, Gemini 2.0 has Success rate of 85,1 % and takes an average of 31.1 Turns to answer. Results are based on continent-level evaluation (1,200 entities per model). Success rates are shown as percentages, and lower numbers of turns indicate more efficient deduction.
Key performance patterns (Table 1): Across both settings, models demonstrate consis- tently higher success with Notable people than Things (canonical: 25-36% vs. 9-13%; unlimited: 67-85% vs. 43-45%), suggesting stronger knowledge representation for notable in- dividuals than regionally specific objects. Allowing unlimited turns improves performance across all conditions by 2.5-3x (e.g., GPT-4o-mini's Things success increases from 9% to 43%), though at the cost of requiring 2-3x more turns for successful deductions. Among the models, Gemini 2.0 achieves the highest overall performance in the unlimited setting (65% average, 85% for Notable people) ), greater efficiency in some conditions, indicating differences in reasoning paths across model architectures.
while LLaMA 3.3 shows competitive results with
7

---

Published as a conference paper at COLM 2025
Setting
Entity Type
Asia
Africa
Australia
Europe
N. America
S. America
Notable people
26.5 (16.3)
21.3 (16.8)
20.8 (15.4)
43.8 (14.3)
45.0 (13.6)
24.8 (16.3)
Can.
Things
8.0 (14.3)
15.5 (15.5)
7.5 (15.4)
12.5 (16.0)
9.0 (14.2)
9.5 (16.2)
Notable people
72.8 (32.2)
77.8 (33.5)
62.0 (35.4)
76.8 (25.6)
78.8 (27.5)
73.3 (35.3)
Unl.
Things
43.0 (45.1)
46.5 (38.3)
37.8 (46.5)
45.0 (45.6)
34.3 (41.8)
39.5 (45.5)
Table 2: Average Success Rates (in %) and Number of Turns to Answer (in parentheses, averaged over all models) for the 100 most popular entities per continent. Results are shown separately for Notable people and Things in both the canonical (20 turns) and unlimited (150 turns settings. The highest values in each row are highlighted in bold. More detailed results are available in Appendix Table
Continent-wise variation in performance (Table 2): We observe sigmificant geographic performance disparities that vary by setting. In the canonical 20-turn game play with Notable people, models show a clear western advantage, with nearly double the success rates for European and North American celebrities (43.8% and 45.0%) compared to those from Asia (26.5%), South America (24.8%), Africa (21.3%), and Australia (20.8%). When allowed unlimited turns, performance gaps narrow considerably as success rates increase across all regions (Notable people 62-79%, Things: 34-47%), with Africa showing partic- ularly strong gains. However, efficiency disparities persist even with convergent success rates, Western entities are deduced in significantly fewer turns ( (Notable people: Europe: 25.6, North America: 27.5) compared to other regions (32-35 turns). This pattern suggests that while models possess information about entities from diverse regions, they require more reasoning steps to access it for non-Western entities, indicating differences in how this knowledge is structured and retrieved. contrast, gameplay with Things reveals different pattern, with models achieving the highest success rates for African entities rather than Western ones.
RQ2: Can entity popularity and pre-training corpus frequency explain geographic performance disparities?
To investigate whether the performance disparities observed in RQ1 simply reflect differ- ences in entity prominence, we analyze two explanatory variables: (1) popularity and (2)
corpus frequency. If performance differences primarily stem from data availability, we would expect these metrics to strongly predict accuracy and the number of turns required for deduction.
Entity Type
Variable
Turns
09
O.R
R²
O.R
R²
O.R
R²
Popularity
Can.
0.000
1.000"
-0.119-*
1.000
-0.011
Things
Unl.
1.000*
-0.006
1
0.000
1.000
-0.68
Frequency
Can.
1.001
0.000
L00G
-0.199
1001
-0.40
Unl.
1002
-0.009
L.000
0.000
1.001
-0.018*
Popularity
Can.
L002
-0.288
1:03
-0.13
1.03
-0.90
Notable people
Unl. 4
-0.023"
T
-0.039
1.000
-0.048
Can.
LO02
-0.133
1003
-1.03"
1.02
4.005
Frequency
Unl.
1001
-0.012*
1.002
-0.024 1.001
*.
Table 3: Regression analyses of the relationship between entity characteristics and model performance. For logistic regressions predicting Success, we report Odds Ratios (O.R.) where values greater than 1.0 indicate positive effects (higher frequency/ / popularity + higher success probability). For linear regressions predicting Number of Turns, we report R² values indicating variance explained. Significance: #** p 0.001, *pp < 0.01, *p < 0.05, Ap < 0.1.
We conducted two types of regression analyses: (1) logistic regression with Success as the dependent variable, reporting odds ratios as x01x to represent the effect of a 1% increase in the predictor; and (2) linear regression with Number of Turns the dependent variable,
8

---

Published as a conference paper at COLM 2025
reporting R² to measure explanatory power. Given the heavy-tailed distribution of entity frequency (Appendix Figure 5), we used log-transformed values to capture the relationship.
Table 3 presents the results of our regression analyses examining these relationships. Despite statistically significant associations in many cases, the predictors show surprisingly limited explanatory power. For success prediction, odds ratios hover near 1.0, indicating that even large popularity or frequency increases correspond to minimal success probability changes. For predicting the number of turns, R² values, while significant, are consistently low (typically below 0.1), showing that these factors explain less than 10% of the variance in deduction efficiency. These patterns hold across all models, entity types, and gameplay settings, suggesting that while data availability contributes to model performance, it cannot fully explain the substantial geographic performance disparities observed in RQ1.
RQ3:
To what extent does the language in which the game played influence the model's ability to deduce entities from different geographic origins?
Entily Type
Continent
English
Hindi
Mandarin
Spanish
Japanese
Turkish
French
Average
Asia
(49.8)
15 (40.0)
(51.9)
35 (52.0)
22 (41.5)
(73.8)
(42.9)
26 (50.2)
Africa
(12.3)
(31.8)
42
25 (33.0)
(36.2)
(35.9)
(36.8
Things
Australia
(60.7)
17 (48.3)
14 (62.1)
(42.5)
18 (34.2)
11 (35.5)
49 (46.0)
27 (47.1)
Europe
(37.8)
25 (33.5)
23 (49.1)
(39.3)
26 (32.7
(61.99 (52.4)
34 (40.1)
North Am
(54.8)
(46.9)
(341)
(30.6)
(50.))
(46.1)
South Am
42 (56.6)
(40.3)
(51.3)
(35.2)
13 (47.8)
(56.4)
(39.1)
(46.6)
Asia
95 (28.5)
89 (28.8)
72 (24.3)
84 (27.3)
68 (23.1)
60 (0(2999
(28.)))
78 (27.)
Africa
(35.4)
(33.9)
()
(345)
Notable people
Australia
(32.4)
(35.8)
(32.5)
69 (31.9)
(35.5)
53 (26.4)
(37
(33.1)
Europe
87 (26.8)
83 (25.6)
81 (22.7)
83 (32.5)
76 (23.0)
48 (25.2)
81 (29.8)
77 (26.5)
N. America
(25.2)
(27.4)
(34.1)
68 (23.5)
51
(35.6
(28.5)
S. America
(39.5)
(28.8)
(26.2)
(33.5)
64 (29.4)
41 (31.0)
84 (31.))
(31.3)
Table 4: Success Rates (in %) and Number of Turns to Answer (in parentheses) for Gemini- 2.0-Flash in the unlimited-tun setting. Detailed results are available in Appendix Table 8.
We use the best-performing model (Geminiz.) to evaluate whether the language of gameplay affects deduction performance across geographic regions. Table 4 presents success rates and efficiency metrics for seven languages (English, Hindi, Mandarin, Spanish, Japanese, Turkish, and French) across entities from all six inhabited continents.
The results show that English consistently has the highest success rates for most regions, while the language of gameplay has unreliable impact on performance. For Notable people, performance remains stable across languages, with success often exceeding 70% in all regions for most languages (except Turkish), while, for Things, success rates vary noticeably by language. Importantly, we find no evidence that regionally associated languages provide systematic advantage for entities from those regions (Yin et al., 2022). For example, Hindi (mainly spoken in Asia) achieves only 15% accuracy on Asian Things compared to 53% in English, and Mandarin shows no advantage for Asian entities either.
RQ4: Geopolitical Disparities in deduction - Global North vs South and West vs East.
Notable people
Things
Global North va South
Global West vs East
Global North vs South
Global West vs East
Turns
Model
Accasen
Acc.con
Sig.
Acc.yee
Acc.pe
Sig.
Acch
Sig.
Accyeet
Aoc
Sig.
@
17.0
7.0
/
2.7
00
21.7
47
/
14.4
5.3
/
8.5
2.9
35.0
17.7
20
30.9
23.4
10.6
3.6
63.9
53.8
62.0
53.0
38.3
31,0
x
36.0
31,0
x
Unl
00
63.9
39.6
52.6
40.6
35,0
17.7
32.1
71.6
83.9
77.8
70.2
47.1
35.7
2
45.8
353
Table 5: Comparison of Success Rates (in %) for entities from the Global North vs. South and Global West vs. East across different models, entity types, and gameplay settings. We report the Average Success Rate for each group, along with the statistical significance of their differences, using the Mann-Whitney U test. Significance levels: **'p < 0.001, "tp < 0.01, pp < 0.05, Ap < 0.1. Results based on country-level aggregation

---

Published as a conference paper at COLM 2025
Based on Table 5, we observe consistent deduction performance disparities across the Global North vs. South and Global West vs. East divide.
Our results show that models are significantly more successful at deducing entities from the Global North than from the Global South across all models, most settings, and entity types, particularly for Notable people. For example, Gemini 2.0 achieves 30.9% success on Northern Notable People vs. 10.2% on Southern ones in the canonical setting (p < 0.001). These gaps persist even in the unlimited-turn setting, though they narrow slightly.
Global North vs. South (Development-Orientel Divide):
Global West vs. East (Cultural Representation Divide): We find similar patterns here, LLMs are consistently more successful deducing entities from the Global West, especially for people-based entities. In the canonical setting, Gemini 2.0 succeeds on 23.4% of Western Notable People vs. only 10.6% of Eastern ones (p < 0.001).
Together, these findings suggest that development-rreated and cultural biases are implic- itly embedded in how LLMs conduct inquiry, with models favoring more economically and epistemically dominant regions, even when not explicitly asked to do so.
6 Conclusion
We present a novel, inquiry-based framework using the 20 Questions game to surface im- plicit geographic biases in LLMs. Evaluating three models on our constructed geographically diverse dataset, Geo20Q', we find that: (1) LLMs perform better on Notable people than
Things, with Gemini 2.0 achieving the highest accuracy. (2) Entity popularity and corpus frequency weakly correlate with performance but do not explain observed disparities. (3) The language of gameplay has minimal impact on performance.
(4) Most importantly, models show geopolitical bias (Global North/West > South/East), by favoring more economically and culturally dominant regions. Our results argue for creative, free-form evaluation frameworks that shift the focus from outputs to reasoning processes, enabling the discovery of subtle and often hidden biases in LLM behavior.
7 Discussions and Limitations
Our primary goal was to propose a novel evaluation strateey that effectively surfaces implicit geographic biases in LLMs. Our analyses tested specific hypothesis, whether pretraining data coverage could explain geographic performance disparities, and demonstrated that such biases are only partially explained by entity popularity and frequency, indicating these behaviors arise from a complex interplay of factors. While our framework provides a unique lens into LLM reasoning, it is ultimately a tool for surfacing potential biases, not proving their absence. A lack of bias observed within our game setup does not imply that model is unbiased overall; it may simply reflect the limitations of the task structure.
In this work, we do not explicitly analyze misidentiifiattin behavior, where models guess entities from more dominant regions (e.g., the Global North) in place of less-represented ones. Next, we do not investigate specific failure modes in deduction, such as repetitive questioning, planning inefficiencies, or inconsistent reasoning. Our findings are embedded in single interactive framework; other evaluation paradigms may reveal different forms of bias. Moreover, due to academic resource constraints, our set of models used is limited, and our language coverage focuses high-resource languages. Finally, we do not perform profession-wise and category-wise breakdown of Things and Notable people, which can help describe which types of entities are more susceptible to bias.
Ethics Statement
This research investigates implicit geographic biases in LLMs using a model-initiated 20 Questions framework. Rather than designing probes, which may themselves encode as-
10

---

Published as a conference paper at COLM 2025
sumptions, we adopt free-form deduction setup and curate a geographically diverse dataset spanning cultural and economic axes (e.g,, Global North vs. Global South, Global West vs. East). Our goal is not to rank hierarchize regions, but to surface systemic dispar- ities in how LLMs reason about entities from different parts of the world. We acknowledge that these geographic groupings broad and imperfect proxies for cultural and political complexity. While they help quantify disparities, they do not fully capture the nuance of global representation.
Data access and model use: No personal or sensitive data was used in this study. All models were accessed via official APIs or publicly released weights. Results are reported in aggregate to avoid singling out individuals or groups. We validated multilingual prompts in-house and did not employ external annotators.
Intended Impact: We hope this work contributes to more inclusive, robust evaluation prac- tices and motivates broader inquiry into multilingual and region-specific model assessment, particularly in underrepresented linguistic and cultural contexts.
References
Jaimeen Ahn and Alice Oh. Mitigating language-dependent ethnic bias in BERT. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 533-549, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653 1tt/:///1122222ee1mmmme/o www.aanttooooy.oog/2//882 .mn.l-mmiin..... 42/.
URL https:
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. Asking clarifying questions in open-domain information-seekingg conversations. In Proceedings of the 42nd international aem sigir conference on research and development in information retrieval, Pp. 475-484, 2019.
Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint #rXiv:2402.04105, 2024.
Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 12211))2411222812,, 2025. doi: 2..1073/pna..24162222 https://www. pnas. wwwpo//////////////////////o 100./.....0//// 1073/00a5 2416228122.
URL
Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. Chat- GPT's information seeking strategy: Insights from the 20-questions game. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natu- ral Language Generation Conference, pp. 153-162, Prague, Czechia, September 2023. As- sociation for Computational Linguistics. doi: 00.18633/v1/2022 org.:ogg....og//ee wrg/ httpss///acaaathooogyooo///////
11. URL
https://aclanthologgy
11/.
Philippe Besse, Eustasio del Barrio, Paula Gordaliza, Jean-Michel Loubes, and Laurent Risser. A survey of bias in machine learning through the prism of statistical parity. The American Statistician, 76(2):188-198, 2022.
Kirti Bhagat, Kinshuk Vasisht, and Danish Pruthi. Richer output for richer countries: Uncovering geographical disparities in generated stories and travel recommendations. arXiv preprint ar Xxxoo2241..03320 2024.
Angana Borah and Rada Mihalcea. Towards implicit bias detection and mitigation in multi-agent Iim interactions. arXiv preprint arXiv:2410.02584, 2024.
Yang Trista Cao and Hal Daumé III. Toward gender-inclusive coreference resolution. In
Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. -595, Online, July Senguissics,,ppaa
2020. Association for Computational Linguistics. doi: 10.18653 /v1 / acl-main.411.. URL https: https://allaaathhacc
/ 2020 ael-mainn.48/.
11

---

Published as a conference paper at COLM 2025
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong, Jianfeng Gao, Mengdi Wang, and Bill Dolan. Contrastive multi-document question generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 12-30, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v /2011. eacl-main.2. URL https: htt.h:tt////tctteae hatpps:/aaaaaatiie https/////aaatooog....o.oo ort.orgg//222 httpg//22..eeeleemmeeee -mai.......
Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Paola Merlo, Jorg Tiedemann, and Reut Tsar- faty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Pp. 2232-2242, Online, April 2021. Asso- ciation for Computational Linguistics. doi: 10.18653/v11 v1/2222212222 /t21//111111111/mmmm eacl-mmiin.190 URL wet.hholog..org/2211..... eacl-main. 190/.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5503-5512, 2017.
Xiangjue Dong, Yibo Wang, Philip S Yu, and James Caverlee. Disclosure and mitigation of gender bias in lims. arviv preprint ar Xrx::2202..1199, 2024.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? ar Xie preprint ar xriv:22310.00007 2023.
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey, Computational Linguistics, 5017 09 2024a. ISSN 0891-20177 doi: 10.1162/col 00524. URL https://doi.org//// 1162/coli 00524.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. Bias and fairness in large language models: A survey. Compulational Linguistics, 50(3):1097-1179, 2024b.
Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca J Passonneau. So- ciodemographic bias in language models: A survey and forward path. arXiv preprint arXiv:2306.08155 2023.
John J Hanna, Abdi Wakene, Andrew O Johnson, Christoph U Lehmann, and Richard J Medford. Assessing racial and ethnic bias in text generation by large language models for health care-related tasks: Cross-sectional study. Journal of Medical Internet Research, 27: e57257, 2025.
Divyanshu Kumar, Umang Jain, Sahil Agarwal, and Prashanth Harshangi. Investigating implicit bias in large language models: A large-scale study of over 50 llms. arviv preprint arXiv:2410.1 128644 2024.
Morgane Laouenan, Palaash Bhargava, Jean-Benoft Eyméoud, Olivier Gergaud, Guillaume Plique, and Etienne Wasmer. cross-verified database of notable people, 3500bc 2018ad. Scientific Data, (1):2990 2022.
Bryan Li, Samar Haider, and Chris Callison-Burch. This land is ( (Your, My} land: Evaluating geopolitical biases in language models. ar Xiv preprint arXi:22305.14611 2023.
Serene Lim and María Pérez-Ortiz. The african woman is rhythmic and soulful: An investi- gation of implicit biases Ilm open-ended text generation. Xiv preprint arXiv:2407.012770 2024.
12

---

Published as a conference paper at COLM 2025
Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. Large language models are geographically biased. arXio preprint (fXic:2402.02669, 2024.
Davide Mazzaccara, Alberto Testoni, and Raffaella Bernardi. Learning to ask informative questions: Enhancing 1lms with preference optimization and expected information gain. Xrv preprint ar Xiv::44006. 17453, 2024.
Shujaat Mirza, Bruno Coelho, Yuyuan Cui, Christina Pöpper, and Damon McCoy, Global- liar: Factuality of llms over time and geographic regions. arXiv preprint arXiv:2401.17839
2024.
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguisfics and the 11th International Joint Conference Natural Language Processing (Volume Long Papers), Pp. 5356-5371, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 21/2222.1.cclooogg 416. URL https: /atp:/aallllaaaa wwwlanhoooooy....ooooooooooooooe wrg/ acl- long. 416/.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), PP. 1953-1967, Online, November 2020. Association for Computational Linguistics. doi: emnlp-main.154. URL https:://claatthlllee org/2020. 1er22220eemlpmmall emnlp-main. 154/.
Lemuel Ekedegwa Odeh. A comparative analysis of global north and global south economies. 2010.
OpenAI. GPT-4 technical report. Technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Nathaniel R Robinson, Perez Ogayo, David R Mortensen, and Graham Neubig. Chatgpt mt: Competitive for high-(but not low-) resource languages. arXiv preprint ar arXio:2309.07423, 2023.
Pola Schwöbel, Jacek Golebiowski, Michele Donini, Cédric Archambeau, and Danish Pruthi. Geographical erasure in language generation. arXiv preprint ar arXto:2310.1477, 2023.
Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. Multi-fact: Assessing factuality of multilingual Iims using factscore. arXiv preprint arXiv:2402.18045, 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXin:2402.00155 2024.
Gemini Team. Gemini: A family of highly capable multimodal models, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Bryan Turner and Habibul Haque Khondker. Globalization east and west. 2010.
Luis von Ahn and Laura Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '04, pp. 319-326, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581137028. doi: URL /ttps://doi... org/10 /0. 10////...155/98888 1145/985692. 985733.
13

---

Published as a conference paper at COLM 2025
Luis von Ahn, Ruoran Liu, and Manuel Blum. Peekaboom: game for locating objects in images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '06, pp. 55-64, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933727. doi: 10.145/1124472.1114777 1124782.
URL https::/doi..rrgeeeeee www.ooo...gg/.... /ttp:///0000.r///11.1141//117171 11411111////2
Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, and Zhiyong Lu. Unmasking and quanti- fying racial bias of large language models in medical report generation. Communications Medicine, 4(1):176, 2024.
Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. Geomlama: Geo-diverse commonsense probing multilingual pre-trained language models. Xr preprint arXiv:2205.12247 2022.
Vithya Yogarajan, Gillian Dobbie, Te Taka Keegan, and Rostam J Neuwirth. Tackling bias in pre-trained language models: Current trends and under-representte societies. Xr preprint ar arXi::231.00150) 2023.
Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. Probing the multi-turn planning capabilities of Ilms via 20 question games. Xio preprint arXio:2310.01463 2023.
Yachao Zhao, Bo Wang, and Yan Wang. Explicit vs. implicit: Investigating social bias in large language models through self-reflection. arXiv preprint arXiv:2501.02299: 2025.
A Appendix
A.1 Empirical upper bound for unlimited-turn setting
To examine the impact of increasing the number of allowable turns in the deduction game, identify the optimal cutoff point to be around 150 turns, beyond which the Success Rate no longer improves significantly (Figure 4). Hence, we can use 150 turns as a proxy for the unlimited-turn setting.
1.0
0.8
Average Success Rate
0.6
0.4
GPT-4o Mini
Notable People
GPT-4o Mini Things
Gemini 2.0 Notable People
0.2
Gemini 2.0 Things
0.1
LLaMA 3.3 Notable People
LLaMA 3.3 Things
0.0
100
125
150
200
Number of Turns
Figure 4: Average Success Rates for the varying maximum Number of Turns across different models using 100 most popular entities from each continent. The Success Rates are almost horizontal for the model after 150 turns.
14

---

Published as a conference paper at COLM 2025
A.2 Continent-wise variation in performance for each model
We present a detailed continent-level analysis for each model, revealing geographic dis- parities in entity deduction across different regions. LLMs have superior performance
Things from Africa and Europe, whereas Notable people from Europe and North America (Table 6).
Turns
Entity
Model
Asia
Africa
Australia
Europe
North Am
South Am
2 (16.5)
16 (15.4)
(16.1)
10 (18.5)
(16.1)
11 (17.1)
00
15 (13.4)
17 (15.7)
(16.5)
13 (15.2)
15 (15.0)
8(15.5)
Things
1.5
7 (13.3)
14 (14.9)
(14.8)
7 (16.4)
(13.0)
(16.2)
8 (14.0)
15 (16.0)
6 (16.1)
20 (15.9)
(12.8)
10(16.3)
20
Avg.
8 (14.3)
15(13.5)
(15.4)
12 (16.0)
(14.2)
(16.2)
14 (15.1)
23 (16.5)
11 (16.3)
38 (14.6)
37 (12.5)
(16.0)
8
29 (16.2)
26 (17.3)
23 (14.5)
53 (13.8)
54 (13.5)
29 (15.9)
Notable people
1.5
23 (16.7)
16 (16.4)
(14.7)
(13.6)
(12.4)
(17.1)
40 (17.1)
20 (17.2)
30 (16.0)
49 (15.3)
45 (14.1)
30 (16.9)
Avg.
26 (16.3)
21 (16.8)
20 (15.4)
43 (14.3)
45 (13.6)
24 (16.5)
?
48 (48.5)
56 (40.1)
37 (47.0)
38 (51.0)
38 (40.7)
39 (41.8)
09
40 (41.2)
42 (36.4)
38 (42.4)
54 (49.0)
43 (40.4)
43(48.7)
Things
1.5
31 (41.9)
42 (34.5)
28 (36.1)
39 (36.4)
22 (27.3)
34 (35.1)
150
2.0
53 (49.8)
46(42.2)
44 (60.7)
49 (37.8)
34 (54.7)
42 (56.6)
Avg.
43 (45.1)
46 (38.3)
(46.5)
(43.6)
(41.8)
(45.5)
?
58 (38.0)
78 (36.0)
51 (41.0)
64 (27.6)
78 (31.5)
71 (33.1)
0
88 (37.2)
79 (31.6)
56 (34.1)
83 (19.7)
86 (28.8)
76 (29.3)
Notable people
1.5
50 (28.2)
63(31.0)
60 (34.0)
73 (28.4)
71 (24.7)
67(04)
2.0
95(25.5)
89 (35.3)
81 (32.4)
87 (26.8)
80 (25.1)
79 (39.5)
Avg.
72 (32.2)
77 (33.5)
(35.4)
76 (25.6)
78 (27.5)
(35.3)
Table 6: Average Success Rates (in %) and Number of Turns to Answer (in parentheses) across different models and continents, taking 100 most popular entities from each continent
A.3 Verifying model's entity knowledge via prompt-based evaluation
Our experiments are conducted with single run per entity for each model-language- configuration combination, which results in total of 50,400+ games (3 models x 1,200 entities 7 languages x 2 configurations for RQ1-3). Therefore, having multiple runs conducted per entity would incur non-trivial computation costs.
To better account for the stochasticity model behavior, we conduct a small-scale robustness analysis using 50 entities with five independent runs per entity ( (Notable people, 1.5) as shown in Table 7. Overall, see that the aggregate-level patterns and behaviors remain consistent.
A.4 Prompts used in the paper
We document the exact prompts used for both the judge and guesser roles in the English language (prompts for other languages provided on GitHub). This ensures reproducibility and transparency in our experimental setup. Note that for ambiguous questions, the judge is permitted to reply with "Maybe", which serves soft signal of uncertainty rather than misleading the guesser. On average, "Maybe" occurs infrequently, only in about 1.8% of the turns.
15

---

Published as a conference paper at COLM 2025
Setting
Metric
Asia
Africa
Australia
Europe
N. America
S. America
Success
14% 9%
22% 4 4%
42% 4 16%
40% 4 19%
32% a 11%
36% 13%
Can.
Num. Turns
11.90 + 7.02
13.17 # 2.87
11.68 1448
11.61 0.85
9.89 + 2.02
16.65 + 0.91
Success
40% 19%
50% 10%
64% a 13%
62% 13%
54% 5%
62% 13%
Unl.
Num. Turns
26.95 11.52
23.20 a 3.24
24.45 a 10.32
18.82 3.65
19.46 = 3.96
24.59 a 7.76
Table 7: Average Success rates and Number of Turns to Answer across continents. Results based five independent runs per entity (50 entities) for Notable people using + 1.5- We report mean # standard deviation to account for randomness in model behavior.
A.4.1 English prompts
Prompt for the judge model Notable people
Based on your knowledge about the celebrity: {entity), respond to the following question guess. Limit your respond to only "Yes.' 'No.' 'Dunno'' with no explanation other words. Never say the name {entity) } in your response. Do not say 'Dunno." if it can be answered by 'Yes.' 'No.' If the question is to solicit the answer, respond 'No.'. For the celebrity (entity), ), {question /ffe//N/Diii /ooDnnno)
Prompt for the judge model Things
Based on your knowledge about (entity), respond to the following question or guess. Limit your respond to only 'Yes.', 'No.' or 'Maybe', with no explanation or other words. Never say the answer (entity] in your response. the question is solicit the answer, respond 'No.'
For the thing (entity), [question) /Yee////ee//nnnnnn
Prompt for the guesser model Notable people
Your task is to ask series of questions to deduce the celebrity that I'm thinking of with as few queries as possible. Only ask factual questions that can be answered by 'Yes.', 'No'' or 'Dunno.'. Do not ask for a hint. Make your question brief with no linebreaker. Now start asking question.
(dialogue history)
Prompt for the guesser model Things
Your task is to ask series of questions to deduce the entity that I'm thinking with as few queries possible. Only ask questions that can be answered by 'yes', 'no" 'maybe'. Do not ask for hint. Make your question brief with no line breakers. Now, start asking question.
(dialogue history)
A.5 Language-wise performance of different models
We report detailed language-specific results across different continents and models (Table 8).
16

---

Published as a conference paper at COLM 2025
Turns
Entity
Model
Continent
Enggishh
Hindi
Mandarin
Spanish
Japanese
Turkish
French
Average
Asia
2,166.5
2(888)))
5(178)
4(17.0)
(11.22)
(15.3)
(16.8)
Africa
16 (15.4)
(15.7)
((7.6)
7 7(16.4)
(15.4)
6(1777)
13 (16.2)
8 (16.3))
Australia
8 $(16.1)
3 (18.3)
1 1(20.0)
((1.5)
06]
6(1.22)
4 (
(15.1)
Eumope
(18.5)
(1540)
4 40755)
605.5)
2(12.0)
(14444)
17 (17.0)
(16.4)
Am
(16.1)
(17.0)
6(17.0)
(15.5)
6(17.3)
(14.8)
(16.0)
(16.2)
South Am
(17.))
(16.0)
((17.))
7(17.1)
(18.0)
(17.55)
(17.8)
(17.22
Asia
703.37
6(14.7)
S(11.)]
6(132))
2(10.0)
7(15.1)
51332)
Africa
(14.9)
(16.5)
4 (16.2)
((15.3)
(16.3)
(15.9)
(15.1)
Things
Australia
(14.8)
(18.0)
((4..))
(14.9)
(16.8)
0.6-1
(17.0)
(13.6)
Parope
(16.4)
(164)
2 (0600
(1220)
(1800)
(18.0)
9 (14.7)
(16.5)
North Am
((3.00)
(12.0)
1 (12.0)
5 (15.6)
(1105))
1 (12.0)
7 (13.4)
4 (12.6)
South Am
(36-2)
(1540)
6033))
(162))
(138)
Asia
S1142007
[1688)
(155)
71147)
7(151)
(15.5)
6(130)
Africa
15 (16.1)
(16.0)
(1722)
11 (15.4)
8 (15.9)
(13.5)
17 (17.5)
(15.9)
Australia
(16.0)
(20.0)
3 (19.0)
9(68))
9 (15.9)
(13.3)
7 (17.6)
(16.9)
20
Europe
(15.9)
(16.2)
3 (1633)
(1577.7)
(15.0))
(17.0)
18 (16.0)
(16.0)
Am
(12.8)
(17.3)
(19.0)
((162)
(18.2)
(14.0)
(14.7)
(16.0)
20
South Am
10 (16.63)
(14.0)
(
12 (16-9)
4 (13.8)
1 (20.0)
12 (16.5)
6 (13.9)
Asia
14 (4(15.1)
17 (15.5)
27 (1566
23 (15.3)
22 (15.9)
28 (16.9)
23 [15.9]
22 (15.7)
Africa
(16.5)
(17777
22 (17.4)
(16.4)
(17.0)
(17.8)
(17.3)
(17.1)
Australia
(16.4)
(13.8)
16 (1559)
7 (16.9)
14 (16.1)
(16.1)
(16.5)
(15.9)
Earope
38 (14.6)
28 (13.5)
43 (13.5)
28 (15.1)
46 (14.7)
37 (14.3)
32 (13.9)
36 (14.2)
North Am
37 (12.5)
24 (14.6)
39 (13.5)
31 (14.1)
28 (13.3)
48 (13.0)
42 (14.0)
36 (13.5)
South Am
(16.0)
36 (15-6)
24 (16.1)
(16.2)
(16.0)
(163)
Asia
(3666)7
19((5577
(1559
(15.4)
(9(15.5)
(14.))
(1561
(15.6)
Africa
16 (16.4)
11 (16.4)
11 (17.8)
13 (16.8)
9 (15.6))
(13.9)
11 (15.5)
11 (16.0)
Notable people
15
Europe
Australia
(5377)
19 (14.7)
13 (15.9)
(13.7)
14 (15.0)
(1344
16 (15.2)
(14.2)
13 (14.)
(12.7)
7 (13.3)
(11.3)
8 (15.9)
(14.1)
13 (15.0)
(13.2)
North Am
44 (12.5)
17 (13.6)
28 (12.9)
36 (13.3)
33 (12.0)
19 (10.2)
38 (14.4)
31 (12.7)
South Am
14 (17.1)
12 (14.9)
21 (17.0)
18 (16-1)
16 (15.8)
9 (15.1)
13 (15.3)
15 (15.9)
Asia
40117.)
(1616.1)
1951)8)
MZ (15.3)
36 (15.3)
30 (14.)
32 (16.9)
35 (15.8)
Africa
(0
26 (154)
28 (15.3)
(16.3)
(15.9)
22 (16.2)
Australia
(1(.6))
(15.7)
20 (16.1)
28 (15.2)
22 (14.9)
(13.2)
21 (16.2)
(15.3)
49 (15.3)
47 (14.7)
55 (15.1)
46 (14.0)
49 (13.7)
27 (12.4)
46 (14.1)
46 (14.1)
North Am
45 (14.2)
37 (14.5)
35 (12.7)
35 (13.5)
44 (12.8)
30 (11.6)
27 (14.7)
36 (13.4)
South Am
(16.9)
(16.9)
34 (15.2)
(15.9)
(16.6)
(14.9)
(15.4)
29 (16.0)
Asia
48 (48.5)
18 (51.7)
14 (35.9)
30 (12.3)
15 (37.9)
20 (53.5)
35 (36.5)
25 (43.7)
Africa
56 (40.2)
15 (38.9)
28 (30.0)
37 (34.9)
16 (41.7)
28 (47.2)
36 (33.2)
31 (38.0)
Australia
37 (47.0)
15 (48.9)
20 (42.0)
19 (39 5)
13 (4K
20 (38.9)
27 (42.1)
22 (43.7)
Europe
(51.0)
(36.2)
26 (33.5)
(36.0)
(36.7)
(55.0)
(30.4)
25 (38.3)
North Am
(40.7)
(11.7)
25 (30.0)
21 (43.6)
24 (36.8)
18 (33.4)
33 (35.2)
25 (37.3)
South Am
39 (41.8)
7 (50.7)
24 (31.9)
31 (40.4)
12 (53.4)
18 (60.1)
38 (43.1)
24 (45.9)
Asia
(111199
21 (58.9)
125
21 (38.6)
13 (40.9)
(38.0)
17 (25.5)
16 (13.4)
Africa
(34.5)
(45.7)
(33.3)
(37.3)
(26.5)
(34.9)
Things
Australia
28 (36.1)
10 (51.)
10 (35.5)
15 (30.))
11 (43.1)
(97.0)
20 (27.7)
14 (45.8)
15
Europe
39 (36.4)
18 (42.4)
6 (36.7)
(518)
11 (34.0)
(58.5)
28 (33.0)
17 (41.)
North Am
22 (27.4)
13 (45.5)
8 (58.5)
13 (35
5 (43.4)
(12.0)
14 (27
11 (35.6)
South Am
(35.1)
(45.4)
(18.1)
(38.9)
(19.5)
(27.1)
(60.3)
Asia
(49.8)
(5000.0)
(5(999)
35 (52.0)
22 (41.5)
6 (73.8)
22 (82.9)
26 (50.2)
Africa
46 (42.3)
10 (31.8)
25 (34.4)
42 (44.3)
25 (33.0)
11 (36.2)
55 (35.9)
31 (36.8)
Australia
(60.7)
(48.3)
14 (62.1)
36 (12.5)
18 (34.2)
11 (35.5)
29 (66.0)
27 (47.1)
20
Europe
(37.8)
(33.5)
23 (49.1)
(39.3)
26 (32.7)
(52.4)
(41.))
(40.1)
North Am
34 (54.8)
18 (46.9)
10 (58.6)
22 (474)
16 (34.1)
5 (30.6)
36 (30.8)
20 (46.1)
130
South Am
Asia
42 (56.6)
(33.0)
46 (36.5)
13 (40.3)
69 (352)
16 (513)
65 (30.5)
39 (35
13 (17.8)
67 (31.1)
62 (31.1)
8 (56.4)
6 (26.2)
48 (39.1)
61 (326)
26 (66.6)
Africa
(36.0)
(40.3)
64 (32.2)
(38.4)
67 (35.1)
(30.9)
(27.)
65 (34.3)
Australia
31 (41.0)
23 (34.3)
41 (32.3)
40 (13.6)
41 (10.0)
37 (36.1)
47 (35.2)
40 (37.8)
Europe
64 (27.6)
56 (29.4)
75 (25.6)
54 (31.)
84 (26.7)
71 (32.4)
65 (25.5)
67 (28.3)
North Am
78 (31.5)
(31.1)
75 (274)
60 (29.6)
63 (M47)
71 (25.7)
74 (26.2)
68 (29.4)
South Am
(38.2)
(28.6)
78 (30.2)
(29.8)
76 (30.3)
(33.0)
(27.2)
(30.3)
Asia
30 (28.2
455 (27.5)
49 (243))
56 (2
33 (23.
23 (20.5)
51 (1.4)
44 (26.1)
Africa
63 (31.0)
47 (31.8)
38 (27.)
53 (3
37 (33.4)
15 (23.9)
51 (30.8)
43 (30.3)
Notable people
15
Europe
Australia
(34.0)
(28.4)
28 (27.1)
(25.9)
32 (26.7)
(22.3)
52 (33.5)
(22.8)
64 (19.7)
31 (27.3)
18 (27.2)
(14.2)
40 (32.2)
(28.5)
37 (29.6)
(23.1)
North Am
71 (24.7)
46 (28.0)
40 (18.1)
72 (24.0)
47 (18.8)
28 (17:4)
68 (24.5)
53 (22.2)
South Am
67 (40.4)
54 (34.3)
52 (26.2)
69 (34.7)
41 (27.9)
23 (22.4)
62 (36.9)
53 (31.8)
95[28.5)
(28.8)
72 (243)
84 (27.3)
68
60 (25.9)
50 (25.1)
75 (27.1)
Africa
(35.4)
(33.9)
71 (32.0)
(40.6)
(27.))
(34.5)
(33.1)
73 (33.8)
Australia
81 (52.4)
66 (35.8)
63 (32.5)
69 (31 9)
51 (35.5)
53 (26.4)
20 (37.3)
65 (33.1)
20
Earope
87 (26.8)
83 (25.6)
81 (22.)
83 (32.5)
76 (23.0)
48 (25.2)
81 (29.8)
77 (26.5)
North Am
(25.2)
77 (51.4)
73 (274)
75 (3L
68 (23.5)
51 (23.0)
73 (35.6)
71 (28.5)
South Am
79 (39.5)
(28.8)
68 (26.2)
80 (33.5)
64 (29.4)
41 (31.0)
84 (31.1)
72 (31.3)
Table 8: Average Success Rates (in %) and Number of Turns to Answer (in parentheses) across different models, continents, and languages, taking the 100 most popular entities from each continent
17

---

Published as a conference paper at COLM 2025
A.6 Qualitative analysis: Model give up behavior
In qualitative analyses, we observed that LLMs often abandon the game after many turns with outputs like "I give up." We include this metric to measure the average number of turns before such early termination. To detect when the guesser has given up, we use GPT-4o-mini to verify at each turn. Lower values indicate quick abandonment, while higher values suggest greater persistence. GPT-4o-miní showed the highest persistence while deducing entities, while Gemini-1.5 fl5-slshh showed the least (Table 9).
Model
Turns to Give Up
108.9
15
67.2
84.8
Table 9: Average Turns at which different mod- els Give Up on deduc- ing an entity
Here is an example of a game with the target entity University of Otago (a public research collegiate university based Dunedin, Otago, New Zealand) in which the model eventually abandons the game after being unable to guess the target entity.
Turn 81, [Guesser ]: Is it a university located in the United Kingdom?
Turn 81, , [Judge No.
Turn 82, [Guesser] ]: Is it university located in Canada?
Turn 82, [Judge]: No.
Turn 83, [Guesser]: Is it university located in Europe?
Turn 83, [Judge]: No.
Turn 84, [Guesser]: Is it university located in Australia?
Turn 84, [Judge]: No.
Turn 85, [uuesser]] Is it university located in Asia?
Turn 85,
[Judge]: No.
Turn 86,
[Guesser] ]: Is it university located in the United States?
Turn 86, [Judge]: No.
Turn 87,
[Guesser]: I unable to guess the entity you thinking of. give up.
A.7 Example of geographic bias deduction path:
The following example shows the deduction path for the target entity A. P. Abdul Kalam (an Indian president, poet, and scientist). Despite the answer being an Indian figure, the model's reasoning chain systematically eliminates Western options before considering Asian and Indian possibilities:
Turn 9,
[Judge]: ]: Yes.
Turn 10,
|Guesser |: Is the poet American?
Turn 10,
[Judge No.
Turn 11,
[Guesser]: Is the poet British? [Judge]: No.
Turn 11,
Turn 12,
[Guesser]] ]: Is the poet from France?
Turn 12,
[Judge]: No.
Turn 13,
[Guesser]: is the poet from Latin America?
Turn 13, ]Jddgel: No.
Turn 14, Guesser |: Is the poet from Asia?
Turn 14, [Judge]: Yes.
Turn 15, [Guesser]: Is the poet Japanese?
Turn 15, [Judgel: No.
Turn 16, [uuesser] |: Is the poet from India?
Turn 16, [Judgel: I: Yes.
Turn 17,
[Guesser]: ]: Is the poet Rabindranath Tagore?
Turn 17,
[Judge No.
Turn 18,
[Guesser]] Is the poet Rumi?
Turn 18, [Jugge]: No.
18

---

Published as a conference paper at COLM 2025
A.8 Heavy-tail distribution of entity frequency in the Dolma dataset
The frequency vs. rank distribution of the combined entities follows a heavy-tailed, power- law-like behavior, consistent with Herd-Immunity Power Scale Invariant (HiPSI) laws observed in natural language (Figure 5). This motivated us to use log transformations of frequency while doing the regression analysis.
5.0
4.3
4.0
3.5
0.5
240
480
730
960
1201
Rank
Figure 5: Heavy-tail distribution of entity frequency values in the Dolma dataset
19

---

Published as a conference paper at COLM 2025
A.9 Multilingual prompt validation process
For each language, at least one native or fluent speaker, recruited from our academic networks, reviewed all translations, edited prompts needed, and observed multiple games to check for accuracy and consistency. Their review protocol included:
1. Verifying and, if needed, editing the prompt templates and translations to ensure naturalness and accuracy in each language.
2. Observing several sample games played in the target language to check the con- sistency of the gameplay: We suggested that every speaker review around three games; however, they were free to review many games as they liked.
3. Confirming the final quality of translations and reporting any issues or corrections (e.g,, overly literal translations, culturally inappropriate terminology, or ambiguous instructions).
Any issues (such as awkward phrasing or ambiguous instructions) were corrected based on their feedback.
A.10 Division of countries
Global West United States, Canada, United Kingdom, France, Germany, Italy, Spain, Portu- gal, Netherlands, Belgium, Sweden, Norway, Finland, Denmark, Iceland, Austria, Switzer- land, Luxembourg, Ireland, Australia, New Zealand, Estonia, Latvia, Lithuania, Poland, Czech Republic, Slovakia, Hungary, Slovenia, Malta, Greece, Croatia
Global North United States, Canada, United Kingdom, France, Germany, Ital,, Spain, Portu- gal, Netherlands, Belgium, Sweden, Norway, Finland, Demmark, Iceland, Austria, Switzerland, Luxembourg, Ireland, Australia, New Zealand, Japan, South Korea
The countries in the Global North marked in italics are also part of the Global West.
20

---

Published as a conference paper at COLM 2025
A.11 Detailed breakdown of entity types
We categorize Things into groups such landmarks, food, animals, etc., and Notable people into professions such as sports, politics, academia, etc. (Figure 6).
Mahico Instrument River
Mountain
2.0% 3.7%
13.0%
5.0%
University
Object
9.0%
Animel
42.3%
Landmark
10.9%
(a) Top categories of culturally significant things.
Academia
Religion
Adminstration Millarr
Corporate
Nobility
2.8%
17.2%
.3%%
Culture
Sport
(b) Top professions among notable individuals.
Figure 6: Distribution of entity types across Things and Notable people.
21


Reviews:
[{"review_id": "EKQPwi1YJj", "reviewer": "Reviewer_g6ic", "summary": "This paper investigates the potential systematic geographic disparities or biases encoded in large language models (LLMs). To do so, the authors introduce a new dataset named Geo20Q+, which includes two broad categories: Things and Notable People. They then evaluate several popular LLMs under two gameplay configurations: the canonical 20-question format and an unlimited-turns variant. Experiments are conducted in multiple high-resource languages to eliminate the confounding variable of input/output language. The results show that LLMs are generally better at guessing entities from the Global North than the Global South, and from the Global West than the Global East. Additionally, the authors examine whether such disparities are correlated with the popularity of the entity (measured via Wikipedia pageviews and entity frequency), which appears to have minimal impact.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is generally well-written and enjoyable to read.\n\n- The newly introduced dataset seems useful and could serve as a complementary evaluation format to standard prompt-based assessments of LLM knowledge.\n\n- The evaluation is extensive and interesting, though I have some concerns (see below).", "reasons_to_reject": "- I have concerns regarding the two-role game design. There may be instances where the judge provides an incorrect answer during the conversation -- either due to insufficient knowledge of the entity or because the guesser's question is ambiguous, yet the judge is still required to respond with a \"yes\" or \"no.\" In such cases, the guesser might never arrive at the correct answer.\n\n- Although the authors conduct extensive experiments, they do not provide explanations for why LLMs display such disparities or biases. This is a crucial aspect of the paper. While the results suggest that entity popularity plays a minimal role, the authors are encouraged to propose some hypotheses, perhaps in the limitations or future work sections.", "questions_to_authors": "- In lines 121–122, the authors mention that they retain only those entities for which all models agree on a primary country. How many entities were filtered out at this stage?\n\n- To address the first weakness, would it be possible to first investigate whether the model actually possesses knowledge of the entity through a separate prompt-based evaluation?\n\n- In lines 173–175, the phrase \"span multiple language families and writing systems\" appears twice -- it’s redundant.\n\n- There is also inherent randomness in the game -- the guesser may not always use the same strategy, potentially leading to different success rates or numbers of turns. Perhaps I missed it, but I didn’t see how the authors account for this randomness. Are multiple runs conducted for each entity?\n\n- The paper doesn’t clearly describe how frequency is computed. I assume the authors use WiMBD to calculate how many documents contain the string representing the entity -- could this be clarified?", "extra_scores": {}}, {"review_id": "3hfy7Fsn8J", "reviewer": "Reviewer_vWma", "summary": "The paper investigates implicit geographic bias in large language models (LLMs) using a 20 Questions-style game. One instance of a model plays the role of the judge (who knows the target entity), while another instance acts as the guesser, asking yes/no/maybe questions to identify the entity. The entities span culturally significant people and objects from different global regions. The authors introduce a new dataset, Geo20Q+, which is highly diverse and geographically balanced, and evaluate model performance across several axes: entity popularity, pretraining corpus frequency, language, and geopolitical groupings (Global North vs. South and West vs. East). The study compares mainly \"western\" LLMs in both canonical (20-turn) and unlimited-turn (max 150 turns) settings across seven languages.\nA key contribution lies in shifting the fairness evaluation paradigm from static prompt-response bias probing to dynamic multi-turn, model-initiated reasoning. This approach enables identification of subtle geographic disparities that may not surface in traditional single Q&A evaluations (not probed).\nThe paper is clearly written and provides sufficient methodological detail to follow the experimental design and replicate key aspects, though some elements such as regression quality and reproducibility require clarification.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Presents a novel and well-executed study that extends fairness research in LLMs to an underexplored axis: geographic origin. \n- Introduces Geo20Q+, a large, diverse and highly curated dataset of notable individuals and culturally significant objects balanced across 172 countries and multiple entity types.\n- Evaluates three major LLMs (GPT-4o-mini, Gemini 2.0, and LLaMA 3) across two gameplay formats and seven languages.\n- Uses a creative and rigorous multi-turn game framework that captures model-initiated reasoning paths, offering deeper insight than static prompts.\n- Provides systematic evidence of geographic performance disparities that persist even after controlling for Wikipedia pageviews and corpus frequency.\n- The results broadly confirm and extend prior work on geographic underrepresentation, especially with respect to Global South and East entities.\n- Demonstrates that geographic disparities persist even after controlling for popularity/frequency and across languages.\n- Paper is clearly written, easy to follow, and methodologically mostly sound.", "reasons_to_reject": "- Although the evaluation strategy is interesting, one could argue it is not strictly needed; standard bias evaluations may still suffice for some questions. The added value over simpler probing techniques should be better justified. \n- The paper does not mention any plans to release the Geo20Q+ dataset (maybe I missed it), which limits reproducibility and broader community utility.\n- Reproducibility concerns due to unclear temperature used during completion generation.While this may be less critical in the absence of a public dataset, it still undermines methodological clarity.\n- While the paper is promising and well written, a major issue lies in the core regression analysis. Table 3 reports multiple negative $R^{2}$ values which are not explained. Maybe these are adjusted $R^{2}$ values? The authors rely on these results to argue that entity popularity and frequency do not explain model performance disparities, yet the low or negative variance explained weakens this claim, especially without raw scatter plots or visualizations to support the interpretation.\n- The use of odds ratios is statistically valid, but not very intuitive for readers. Including raw correlations or binned visual summaries would improve clarity and interpretability.\n\nI would adjust my rating accordingly if these issues are addressed or clarified.", "questions_to_authors": "- Could you show a clear example of a deduction path that illustrates a geographic bias (e.g., Western vs. Eastern entity), to better understand how the method surfaces such disparities?\n- Was initial data collection done only in English?\n- Could you share or visualize success rate or number of turns as a function of entity popularity or frequency (e.g., scatter plots or binned averages)? This would greatly help interpret the regression analyses.\n- Could you clarify whether the reported $R^{2}$ values in Table 3 are adjusted?\n- Do you control for number of guesses allowed per game as well? similar to number of turns up to \"Bingo\" or max turns.\n- Do early exits or refusals to guess count as deduction failures in your success rate metric? (Apologies if this was mentioned and I missed it.).\n- What motivated the use of odds ratios to assess popularity/frequency effects? Would raw correlations or binned plots be more interpretable and transparent?\n- What’s the ratio of English to non-English data? Could this correlate better with performance than popularity?\n\n- Table 1: Please add units (e.g., \"% success\", \"# turns\").\n- Table 5: Please define symbols (✓, ✗, *, ***) explicitly in the caption or text.\n- Appendix A.3: it appears the prompt labels for the guesser and judge roles may be reversed.", "extra_scores": {}}, {"review_id": "3YXuuUqTDO", "reviewer": "Reviewer_2qWz", "summary": "This work proposes a 20-questions dataset on geographically specified entities, and evaluates how several LLMs perform on it along several dimensions of geographic biases, from country, to entity type, to language (7 total). The questions are selected to be balanced across countries and were verified by native speakers. They also performed a correlation study of performance to the pretraining corpora and Wikipedia pageviews.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea of using 20 questions for geographic entities to evaluate LLM's geographic biases is simple yet well-motivated.\n* The curation of the dataset is carefully done and balanced across countries.\n* The correlation study between performance and Wikipedia views or the pretraining corpora is well thought-out and interesting.\n* The experiments were done in 7 high-resource languages. The finding that language has a moderate impact on performance is interesting.\n* The results are presented in various granularities, such as by country, by continent, by Global North/South, by people vs. things, etc.", "reasons_to_reject": "* I am not sure the claim in lines 123-126 is warranted, that this dataset avoids contamination better than other 20Q datasets. For example, LeBron James or any other famous sports star (47.9% of the people entities) are definitely commonly studied.\n* The Global North/South West/East divisions are overly simplified and  inaccurate. For example, richer countries like UAE, Singapore, and Saudi Arabia are not in Global North. And why are all Latin American countries considered in the Global East, while they are literally in the West, and most follow Catholicism, a decidedly western religion? Perhaps for each division you should omit certain countries which do not neatly fall into a category. Even if you cite 2 works, these are from 2010 and you can fix their mistakes in your work.\n* The corpus frequency analysis only considers the English-only Dolma, while your entities are multilingual. This makes that analysis less robust. You should have used a corpora like Fineweb-2 instead.\n* RQ2 does not account for variants of the same entity. For example, \"Jack Ma\" could be \"Ma Yun\", \"马云\", \"CEO Ma\", etc.", "questions_to_authors": "* I don't like the title \"The World According to LLMs\", this is overly generic\n* 35-36: avoid anthropomorphizing LLMs, use \"models output\" and \"models encode\" instead\n* 29: paradoxically is too strong, use \"however\" or \"interestingly\"\n* Figure 1: could you also have a version of this figure which has the multilingual entities?\n* what does the + stand for in Geo20Q+? It's confusing and could suggests there are more than 20 questions\n* 49-50: should say \"we evaluate an unlimited turn configuration\" instead of \"we introduce\", since this is part of the evaluation not of the dataset\n* Figure 1: consider using an equal-area map projection, as the current one makes countries in the Global North appear larger which goes against the spirit of your findings\n* Consider adding citations: https://arxiv.org/abs/2305.14610 https://arxiv.org/abs/2205.12247\n* 174: remove this citation, seems like padding\n* 4.2: can you add more details on the processes in which native speakers confirmed the quality of translation and observed some games, in an appendix section? like how they were recruited, any observations or corrections they made\n* Table 1: add % to the success rates to make more clear.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Geo20Q+, a novel curated dataset of geographically specific entities in the 20-questions game format, to evaluate LLMs for the potential of systematic geographic disparities. Geo20Q+ comes in seven high-resource languages (English, Hindi, Mandarin, Japanese, French, Spanish, and Turkish) and allows for free-form evaluation, as LLMs are instructed to proactively ask questions and guess the target entity (thing or notable people).

The authors provided convincing additional information during the author rebuttal. The reviewers unanimously agree that this is a good paper which should be accepted. I second this. This is a good paper, which introduces a new evaluation protocol and interesting dataset to test LLMs for systematic geographic biases.

I urge the authors to check the font/template in future submissions, and fix it for the camera-ready as it deviates from the COLM template. Because this paper was flagged and saved/went for review.
Decision: Accept