Geographical Erasure in Language Generation
Authors: Pola Schwöbel
Jacek Golebiowski
Michele Donini
Cedric Archambeau
Danish Pruthi
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
fairness
language generation
bias
world knowledge
TL;DR: Large language models underpredict certain countries, erasing them from dialogue. We measure and mitigate this effect.

Abstract: Large language models (LLMs) encode vast amounts of world knowledge. However, since these models are trained on large swaths of internet data, they are at risk of inordinately capturing information about dominant groups. This imbalance can propagate into generated language. In this work, we study and operationalise a form of geographical erasure wherein language models underpredict certain countries. We demonstrate consistent instances of erasure across a range of LLMs. We discover that erasure strongly correlates with low frequencies of country mentions in the training corpus. Lastly, we mitigate erasure by finetuning using a custom objective.

Geographical Erasure in Language Generation
Pola Schwöbel
Jacek Golebiowski
Michele Donini
Amazon
Amazon
Amazon Berlin, Germany
Berlin, Germany
Berlin, Germany
schwobel@emazon.
jacekgo@ammazon.co
donini aariiiamazo.cccom de
Cédric Archambeau*
Danish Pruthi'
Helsing
Indian Institute of Science (IISc)
Berlin, Germany
Bangalore, India
sedrric.archambeeueeeellsiigggg. ai
dani danishpeiisc.ac. in
Abstract
Prediction Probabilities
Large language models (LLMs) encode vast amounts of world knowledge. However, since these models are trained on large swaths of internet data, they at risk of inordinately capturing information about dominant groups. This imbalance can propagate into generated language. In this work, study and op- crationalise a form of geographical erasure, wherein language models underpredict certain countries. We demonstrate consistent instances of erasure across a range of LLMs.. We dis- cover that erasure strongly correlates with low frequencies of country mentions in the training corpus. Lastly, we mitigate erasure by finetun- ing using custom objective.'
0.10
English Speaking Population (Mio.)
300
200
100
Extent of Underprediction
0.1
1 Introduction
0.0
Large pretrained models serve as base models for many downstream NLP applications, including question-answerrigg, dialogue, common-sense rea- soning, classification, tagging, translation, marisation, and generation (Devlin et al., 2018; Brown et al., 2020; Chowdhery et al., 2022). De- spite their increasing utility, there are concerns about how they reflect and amplify biases in the training data. For instance, unfiltered data originat- ing from the internet is known to be rife with toxic, misogynistie, and stereotyping content. Many stud- ies highlight biases in model outputs, primarily concerning representational harms (Barocas et al., 2017), where section society (e.g., women, LGBTQ+ communities) are represented in poor light, or are ignored by the system (Bolukbasi et al., 2016; Guo and Caliskan, 2021; May et al., 2019;
Figure I: Some countries are vastly underpredicted compared to their English speaking populations. Top: Country probabilities assigned by GPT-NeoX when prompted with "I live in". Middle: English speaking populations per country. Bottom: Countries experienc- ing erasure, i.e. underprediction compared to their pop- ulation by at least a factor (see $3). Data missing for grey countries (see $6).
Tan and Celis, 2019). While important, such stud- ies predominantly examine biases related to race, gender, occupation and sexual orientation.
An important- and often overlooked aspect of inclusive model development is geographical inclusion. This is particularly important time when most large-scale model training efforts come from a small set of regions. Further, these mod- els are trained using internet data, whose access, in the first place, unequally distributed (Blank, 2017; Center, 2021). Minimising cultural and geo-
'Work done while Amazon.
'Code available at https:// htt.. com/amazon-scht ce/ ce/geographice wwchhicallearssse -erasure- in- language gengggtionnnriionn.

---

graphic identities is referred to erasure (Roche, 2019) and is studied by linguists and social scien- tists the context of imperialism and colonialism, where "people are silenced in the historical record [..],, their contemporary presence rendered invisi- ble, and their existence written out of the future" (Roche, 2019). Automated systems and their devel- opers exclude certain groups unintentionally, but the risk of being "written out of the future" remains pressing: the produced content fed back into the internet. In lifelong learning setups, the generated content becomes the training data of tomorrow's models, closing the vicious circle of reinforcing social hierarchies (see $4.5 of (Sheng et al., 2021)).
the unfair distribution of resources, e.g. when a group is denied bank loans disproportionally by an automated system. Allocational harms tend to be more easily measured through standard fair- ness metrics like demographic parity (Dwork et al., 2012) and equality of opportunity (Hardt et al., 2016). Those do not directly apply to open-ended generation tasks, where we instead study represen- tational harms, which arise when system "repre- sents some social groups less favourable light than others, demeans them, or fails recognise their existence altogether" (Blodgett et al., 2020); the last case being the focus of our work.
Fairness measures for language generation usu- ally define bias as differences between demo- graphic groups (Sheng et al., 2021). For example, Dhamala et al. (2021) find that female pronouns are more likely to elicit positive text from an LLM than male pronouns. Similarly, Huang al. (2019) compare different occupations, names and coun- tries on produced sentiments. Nangia et al. (2020) and Nadeem et al. (2020) compare the probabil- ity of stereotypical and non-stereotypical sentences under a model in order to measure whether it en- codes stereotypes against different demographic groups Along the same lines, the WinoGender test (Rudinger et al., 2018) measures gender biases in co-reference resolution tasks. Taking distribu- tional view similar to our work, Rac et al. (2021) investigate biases in the context of occupation, how- ever, they again compare predictions for different genders with each other. In general, such compara- tive bias tests are well-adapted by authors propos- ing new models (Touvron et al., 2023; Rae et al., 2021; Hoffmann et al., 2022; Scao al., 2022). In- stead of comparing model predictions against each other, we compare model predictions to real world ground truth distributions in order to quantify bias.
In this paper, we reveal instances of geograph- ical erasure, wherein models underpredict geo- graphical regions (see Fig. 1). For instance, GPT2 assigns nine times higher likelihood to live in Canada" than "I live in Pakistan", whereas Pak- istan's 's English-speaking population is almost four times to that of Canada. By comparing model out- puts with population statistics, we operationalise geographical erasure ($3). Using this measure, first demonstrate the existence of erasure for sev- eral countries across different prompt formulations ($4.1). Studying the consistency across a range of language models from the GPT and LLaMA model families, we find that several countries - Nigeria, Pakistan, Eswatini, Uganda and Madagascar - are affected by erasure under all models ($4.2)) Fol- lowing related work (Lin et al., 2021; Rae et al., 2021; Nadeem et al., 2020), we study the impact of model size on the extent of erasure, and find it to be small - erasure occurs across all sizes ($4.3). To identify the causes of erasure, we compute the unigram frequencies of countries in the training corpus ($4.4). They closely match our model pre- dictions indicating that the composition of training data is main source of erasure. Lastly, we alle- viate erasure via supervised finetuning. We study the impact of mitigation on generation quality as measured in perplexity on Wikitext-2-v Our fine- tuning strategy proves to be an effective mitigation mechanism which generalises and has small impact on generation quality ($4.5).
Ground mush- measures are not commonly used as metric for fairness but important when evaluating a model's truthfulness. Petroni et al. (2019) and Lin et al. (2021) provide datasets of real world facts against which to benchmark LLMs' knowledge. Similar to our work, Zhou et al. (2022) measure the frequency of country predictions, and how underprediction correlates with country's GDP. Contrary to their count-based approach we propose more fine-grained metric for erasure and extend the analysis to auto-regressive models. Un- like theirs, our crasure metric can be employed as a loss function for finetuning, to specifically miti-
2 Related Work
The literature on fairness in machine learning dis- tinguishes between representational and alloca- tional harms (Barocas et al., 2017; Crawford, 2017; Blodgett et al., 2020). Allocational harms concern

---

gates crasure. Liang al. (2022) propose a similar metric for crasure in the domains of gender and race. Like us, they compare model to ground truth distributions, though they measure a total variation distance where we use a Kl--divergence based ap- proach (sce $3.3). The authors assume uniform ground truth whereas we construct domain spe- cific distribution (see $3.2). Lastly, unlike ours, their analysis does not cover any mitigation efforts. Similar in spirit, geographical representativeness has been studied for text-to-image generation mod- els (Rojas et al., 2022; Basu et al., 2023).
e.g.. 27 "Uganda" is tokenised into x/ - 2 ="g", 2 ='anda.. As consequence, p(r, tends to be smaller for multi-token country names. Concerningly, Zhou et al. (2022) show that this issue predominately impacts low GDP-countries. Some countries in X' are referred to by more than one name, e.g.. "UK" and "United King- dom". We disambiguate the countries using list of alternative names? to obtain the final p(a;/c) A ( ((ffle) for all alternative names xç.
In the following sections, we sometimes write p(z;/c) = Pi omitting the dependency on the prompt unless ambiguous. Note that we work di- rectly on the model probabilities and discuss the impact on generated language in $6.
3 Method
Our goal is to measure, and later mitigate, the ex- tent to which large pretrained models underpredict some countries when generating language. We for- malise this notion here. Note that while we are studying autoregressive models in this work, the methodology extends straightforwardly to masked interested in measuring
3.2 Obtaining Ground Truth
To measure erasure, compare the generation distribution (given by equation 2) a ground truth distribution p'T over the candidate countries, writ- ing (x;) = p!re as before. The ground truth is given by real world data, i.e., we compare our predictions to the actual population of country Tú. We adjust for the fact that our models are trained on English texts only by considering English speaking populations ground truth (see $6 for limitations of this approach). The number of English speak- ers per country is obtained from a Wikipedia list containing data for M = 127 countries at the time of writing-vy -e use these 127 countries for our analysis. Unlike the model predictions p( (c). the ground truth (xx)) is prompp-independent. We will generalise model predictions to be prompt- independent well by marginalising prompts in $3.5. See Figure (left) for example of model predictions and ground truth.
models. Similarly, and reducing geographical erasure, but the analy- sis can be applied other attributes where ground truth is available. For example, one could measure erasure with respect to age, ethnicity, religion or gender using the same formalism.
3.1 Obtaining Model Predictions
Let be our language model over vocabulary Ω. We consider open-ended generation tasks for au- toregressive models. Such models predict the next token given previous ones, i.e. for sequence of T, Ω the probabilities factorise
tokens I:
1-1
1. = II 1l 1tk).
(1)
1=1
We use pretrained models and condition on short prompt, or context, of variable length L: c I.L Given this prompt, we compute the predictive distribution over a set of M candidates { = X C Ωi see $3.2 for how these M coun- tries are chosen. For candidate country x; E X' we compute p(z;(e)
3.3 Measuring Erasure
With these prerequisites in place, we can now for- malise erasure using the relationship predictive distribution and ground truth.
Definition 1 (Erasure Set). For a ratio threshold > we define the erasure under model p, ground truth rue and prompt as
p(z,lc) p(11,c) ((Zi,c) p(c) Exexp(e,c)
(2)
true
S -
(3)
i.c., compute p("I live in zz²") for all candi- date countries x; and normalise. If country is tokenised into multiple tokens, x; = = !! we mul- tiply the probability of the J subtokens accord- ing to (I). As before, superscript indicates po- sition and subscript indicates the country name,
p(zzlc)
LList alternative country names https://en. wik ipedia. wrg/wwki/listtttt..okkk s, retrieved on Sept. 26th, 2023.
'From https: / //////n.uu wwtppppeia..rgg/wiii//////////ee wttpw/kii/iss...o.eooooooooo untries_by Bes______gggg Sept. 26th, 2023.
retrieved on

---

the United States
India
125
SS
Kl(p'orlp)) ER'
Pakistan
100
y=
0.8
Nigeria
Philippines
0.6
United Kingdom
S.
Germany
Uganda
Canada
Egypt
France
Australia
Ground Truth
0
0%
12% p(x,cc)
25%
0
2
6
8 10
0
2 6
8 10
Figure 2: Understanding erasure. Left: OpenI.LaMA, 7B vastly underpredicts the occurrence of Pakistan, Nigeria and Uganda. We plot country predictions given prompts p(z;le) for different re- re-phrasings of the prompt live in" (boxplots) and ground truth (barplot, grey). Country names experiencing erasure (2, E Sg. see $3.3) are in red. We show the 12 countries with the largest English speaking populations (in decrasing order). Middle: Erasure set size [S.] as a function shaded area) over different rephrasings (see $3.5) of the same prompt. The dashed line marks = 3, the threshold we use in the experiments. This choice further motivated in $3.4, Right: Comparing ER (blue) for different r to the KT.-divergence (red). We pick 3, the integer value for which KT. and ER the most similar.
for LaeALLaMA, 7B. We plot the median (solid line) and 25 75 percentiles (blue
In the example in Figure (left), we prompt the LaMA model with different versions of the prompt "I live in" and aggregate the predictions (see $3.5 for rephrasing and aggregation). We then compute the erasure set for r = 3, i.c. countries that are three times more prevalent in the ground truth than in our predictions. We obtain S3 = (Pakistan, Nigeria, Uganda, Bangladesh, Iraq, Madagascar and Eswatini). A simple metric of erasure is the size of the erasure set [S,] for a user-specified r, here, |Sa] 7
with small populations are also taken into account. Hence we report (log-)ratios in the definition of ER" (4). On the other hand, while we believe this sensitivity to dess-populated countries is im- portant, we do acknowledge that underpredicting big ground truth populations is particularly harmful as impacts lot of users. Thus, we weight the log-ratios by the ground truth probabilities p! A third factor in our choice metric is the close relation of (4) and the KL-divergence (L ER" is an additive component of the KL-divergence:
|S,|| measures how many countries are "erased" (underrepresented by at least factor r). To obtain more fine grained numerical evaluation measure by how much they are underrepresented compared ground truth by reporting the following metric. Definition (Erasure). Erasure w.r.l. ground truth yue at threshold r is defined as
true
(L(ppuul(p)
Σ true log
Pi
Ps Pi
+
(5)
ieS,
true
P.
true
log
Pi
S
This close relation to well-defined divergence measure allows for theoretical analysis and helps practitioners build existing intuitions.
p!ue
ER" ER"(p (prree p) = -E tue log
(4)
Pi
SES,
The choice of r is crucial hyperparamter, [8,] and ER" are defined in terms of r. We discuss the impact here and visualise it in Figure (middle and right). For small values of r, we include all terms in (5), i.e.,
3.4 Properties ER"
A careful conceptualisation of any proposed fair- ness metric is crucial (Schwöbel and Remmers, 2022; Blodgett et al., 2020). We motivate our definition of ER" here. Firstly, if p t then EB" (i p) for all i.e., erasure occurs when the distributions match. Secondly, unlike the total variation distance suggested Liang et al. (2022), we want our metric to sensitive to rel- ative rather than absolute errors, so that countries
lim S, - X and lim ER" Ep" ,p) (p [p) r-0
r00
For larger values of r, we instead have
lim S, = 0 and lim ER" (p = 0.

---

See Figure 2 (right) for this relationship. Since we want to measure crasure or underprediction, we study cases where ptuee > p, i.e., for values r > 1.4 We pick r to be an integer such that ER" (p"e, PP) = (L(p"mel1 [p), that is r = 3 in the experiment in Fig. 2 (right). We find that this value is the same across all our models (see Appendix A), so we choose is based a mathematical heuristic. An alterna- tive way of choosing this parameter might be im- plied by legal or ethical constraints. For example, a guideline on adverse impact by the US Equal Em- ployment Opportunity Commission (1979) defines "a substantially different rate of selection" at 80%. In this labour market use case, r = 1 1/0.8 1.25 would be the corresponding hyperparameter. Differentiability an important property of our metric since we want to use it for finetuning LLMs in $4.5. For fixed r, ER" is differentiable almost everywhere (with respect to the network weights). Singularities occur at those points that add new countries to the erasure set S, in (3), i.c., weights such that Pic P& for any country k.
of recent advances in LLMs, another way to au- tomatically rephrase prompts is by using a model that has been finetuned for paraphrasing (Niu et al., 2020). Even simpler, we use an off the--sheef model by prompting ChatGPT to rephrase the ē ="I am from" seed prompt. After manually removing ir- relevant prompts we obtain 16 base formulations. We further expand the set of prompts by replacing sentence subjects. For example, expand "I live in" into ("You live in", "He lives in", "She lives in", ....), producing total [D] 955 prompts. Details and a list of all prompts can be found Appendix B. We use the dataset of 955 prompts to approximate the marginal in (6) assuming different prior probabilities p(c[M) as follows:
3 globally. This choice of
(1) Uniform prompt distribution:
1 [D]
p(clM)
then
(7)
p(alM)=
0e uni M).
(8)
D
DED
()) Model-induced prompt distribution:
3.5 Prompt Rephrasing
p(c)
p(cM)
(9)
ECED P(c)
The erasure set definition in (3), and consequently the notion of erasure in (4) are prompt-dependent. However, we are interested more generally in the model's world knowledge rather than its comple- tion of a specific prompt. Hence, we would like aggregate the effect over all prompts encoding the meaning M ='home country"", by using the following marginal distribution:
where p(c) is the probability given by the autore- gressive language model (1). In this case,
p(alM) Σ p(alc)p(c(M)
88 model
( (xlM).
(10)
4 Experiments
P(zlM) = / placc)pldMMdd
In this section, we show the existence of geograph- ical erasure across different LLMs and different prompt wordings ($4.1). We highlight the con- sistency of erased countries across models ($4.2) and investigate the impact of model size on era- sure ($4.3). We identify possible causes of erasure ($4.4) and explore mitigation strategy ($4.5). The models under consideration (Radford et al., 2019), 117M, 315M, 774M and 1.6B weight versions, GPT-Neo (Black et al., 2021), 125M, 1.3B and 2.7B weight versions, GPT- NeoX, 20B weights (Black et al., 2022) and open source reproductions of the LLaMA model (Tou- vron et al., 2023; Geng and Liu, 2023; Computer, 2023), 3B and 7B weights. We obtain all imple- mentations from HuggingFacc.
(6)
The relationship between prompt e and its meaning M is complex, hence computing (6) is in- tractable. Here, we will rely on simple, pragmatic techniques to cemi-automatically construct a set of sample prompts D ~ Pcc.M) from a seed prompt č. We rephrase é while preserving its meaning to generate additional prompts. This is common prac- tice: Jiang et al. (2020) use mining- and translation- based paraphrasing methods while Romano al. (2006) rely on templates for paraphrasing. In light
GPT2
'There is a degree of symmetry in our measurement: be- ing probability distributions, n and p sum to one. Thus, when "r > for S,, there are other countries for which the opposite is true, i.c. that are overpredicted. In general, we helieve underprediction to be more likely to cause harms than overprediction (sce $ 11), hence we focus on measuring crasure.
2 "Accessed via https:tts////// openaii.pppniicooomm 'Via httph: https://huggingffac co///ocs/transformerrs

---

Procxpts
ER yo many ER p
Brased by how many models
10
1.5
оoоu:: Avg. ER?
8
Brasure TRR*
1.0
6
4
0.5
2
0.0
He comes from
His father bails from
My nephew is citizen of
Your rools are in
Northern Mariana Isl.
Saint Kitts and Nevis
Antigua and Barbuda Malaysia Saint Lucia
St Vincent
Micronesia Iroq
Surimame
Marshall Islands
Eswatini
the Phillipiin Sri Lanka
Popua New Guinea
Ban Aingladesh
Solomon Islands
Nigeria
Cameroon
Pakistan
Uganda
Madaguecar
Sierra Lenne
Our unelo is cilizen of
Figure 3: Geographical erasure occurs all prompt rephrasings, and many countries experience erasure consistently under all models. Left: OpenLLaMA, 7B results for 955 individual prompts (blue dots) along the x-axis, with some example prompts as axis labels. We also plot ER" in aggregate: The blue line is the average over individual prompts Σ. (p"uee p(-(c)). green is the uniform aggregate ER putiisssy (8) and red is the model-induced aggregate ( (iundee_ee)) y respective prompt under the model. The gap between blue and red/green aggregations is explained in $3.5. Right: Of the .M 127 countries, 105 do experience erasure plot model counts here. Bars are coloured according to counts and sorted by GDP per capita (decreasing from left to right). We use aggregated predictions according to Equation 10.
(10). Size of dots corresponds to the probability assigned to the
3 for any of the models. For the remaining 22, we
4.1 Impact of Prompt Wording
4.2
Who is Experiencing Erasure?
We evaluate whether the same countries experi- ence erasure under all the examined 10 models, and what characterises these countries. Out of the M = 127 countries under analysis, 105 do not ex- perience erasure at r = 3 for any of the models. For the remaining 22 nations, Figure 3 (right) shows the number of models by which they are erased. Worryingly, Eswatini, Nigeria, Pakistan, Uganda and Madagasear experience erasure under all 10 analysed models. The x-axis in Figure (right) is ordered by GDP per capita, in decrcasing order from left to right.?
We start by investigating how dependent erasure is on the exact phrasing of the prompt. We prompt the models with rephrased versions of "1 live in" (see $3.5) and compute erasure (p'T (p"""e,p((( -p-lc)) for each prompt c. In Figure 3 (left), we plot the erasure for individual prompts (dots); (ii) the av- erage erasure 2 Ee E (p p(-cc) denoted by the blue dotted line; (iii) crasure for the uniform marginal distribution from (8) using a green dash- dotted line; and (iv) erasure for the modeel-nduced marginal distribution from (10) red dashed line. The size of the blue dots indicates p(cl.M)
4.3 Impact of Model Size
The magnitude of erasure 3 (c)) dif- fers across the phrasings c, however, erasure ex- ists in all versions (that is, ER3 > O with p-value < 0.01). We note that erasure under the aggre- gate distribution is smaller than the average era- sure (p pogg uni < A/ E. (p (p-"ee,p(-cc) in Figure 3 (left)). This follows from Jensen's in- equality (see Appendix C for details). Throughout the remainder of the paper, we will report the ag- gregates from (8) and (10) along with boxplots of ER3 to account for the variance due to rephrasings.
Related work (Lin et al., 2021; Rae et al., 2021: Nadeem et al., 2020) reports mixed results on the relationship between model size and bias. On the one hand, Lin et al. (2021) report that on the Truth- fulQA benchmark, "[]aarger models are less truth- ful". This is because large models surface the com- mon human misconceptions that the questions are designed elicit. Such misconceptions are likely present the training data which the larger mod-
2 Data from https: //en. wiki likkiediia.oe wwwp/www///////o//o wed.i..rg///kkiiii......... wwwii//i/ttoooo _ccuntries_by GDP (nominal) on Sept. 26th, 2023.

---

ER (p "pp p)
ER ' mrain .p)
Frequencies
2.0
GPT2
GPT-NeoX
2,0
GPT-Nco
Pakistan
NooX
Nigeria
Open
prs moki
the Philippines
1.5
1.5
Erasure ER²
Uganda
Bangladesh
1.0
1.0
Sri Lanka
Cameroon
0.5
0.5
plae
Madagascar
(Nex))
Papua New Guinea
0.0
0.0
117M
125M
345M
774M
1315M
1558M
27000
3000M
7000M
20000M
125M
1315M
2700M
200000
0%
5%
10%
3
Parameters
Parameters
p(country)
Figure 4: Erasure in models closely matches distribution country mentions in training data. Left: Geo-
graphical erasure GPT-type models of different sizes. Size is the x-axis (axis scale). Blue models are GPT2, red models are Gl',-Neo, yellow are OpenLLaMA and the green model is GPT-NeoXX crosses below each plot are aggregated results over different prompts (see Eq. 10).
dashed line indicates (y" y!rain) Middle: Geographical erasure for GPT-type models of different sizes, assuming training frequency the ground truth (instead of world population). Right: Ground truth (red) and Pile training data (green) distributions compared to GPT-NeoX predictions (blue) on countries the erasure set S3 (of GPT-NeoX w.r.t. ground truth).
els match more faithfully, Similarly, Nadeem et al. (2020) find that the larger models exhibit more stereotyping, again this is probably because they match stereotypes in the training data more closely. On the other hand, Rae et al. (2021) "do not see consistent correlation between model size and bias" in their tests for gender-occupation bias.
rence in the training data yraain (x)), we compute the number of times each country x is mentioned
in the dataset, i.e., p ε # mentions of x. These mention counts are weighted by the num- ber of training epochs this document was included while training (dataset weights wa from Gao et al. (2021)). We account for alternative country names as described in $3.1. Thus, the final formula be- comes
We visualise the extent geographical erasure with varying model sizes in Figure 4 (left). Like Rac et al. (2021), we do not find model size to have big impact. We hypothesise that even the smaller models closely mimic the frequency distribution (of country mentions) in the training corpus, similar to Rae et (2021)'s experiment. We believe that this is the case in the test by Lin al. (2021) and Nadeem et al. (2020), because their tests go much beyond unigram frequencies, and smaller models do not exhibit such subtle biases. We explore the relationship of data bias and model bias below.
train x x
Wi Σ #I E Ed,
(11)
dEdalasets
RE.A
where A represents a set of alternative country names given country and represents counts. Once the counts are gathered, the results are normalised to determine the final values of tran (x). which we compare to the outputs of LLMs.
Specifically, compare ground truth y! (x), training data prain (x) and GPT-NcoX predictions p(z) for countries x S3 (Figure 4, right). We see that countries experiencing erasure are indeed underrepresented in the training data, and the pre- diction probabilities of these countries are similar to their frequency distribution in the training corpus (pr)) > p(x) R train (x:))
4.4 Impact of Training Data
We hypothesise that training data is an important factor for erasure: models underpredict countries which appear in the data infrequently compared to their population. To study the relationship be- tween training data bias and model bias, we extract the distribution of country mentions in the training data. We consider the Pile dataset (Gao al., 2021) used to preitrain the GPT-Nco LLMs analysed in this study. To determine the probability of occur-
We then compute erasure against the training data, ER ty .p), iee. considering the ground truth to be plain (Figure 4, middle). We find that cra- sure values in this case are considerably lower. For

---

Random Splits
Pronoun Splits
Verb Splits
0.8
28
0
0.8
28
Triining
Popllccly
Training
Perplesit
Tnining
Perpleaily
0.6
26
0.6
0.6
26
ER loss
Perplexity
0.4
24
0.4
24
0.4
24
0.2
22
0.2
0.2
22
0.0
20
0.0
0.0
20
0
4
Epoch
Epoch
Epoch
Figure 5: Finetuning effectively alleviates erasure. We plot average ER" on training (blue) and test (red) set prompts during 5 epochs of finetuning of the GPT2-small model. Error bars indicate minima/maxima over folds.
instance ER'(pliin mo = 0.08 for GPT-
generated distribution for all our prompts to match the ground truth), while not impacting the general language modelling abilities. We evaluate whether the language modelling abilities deteriorate by mea- suring perplexity on Wikitext-2-v (Merity et al., 2016) before and after every epoch of finetuning.
NeoX compared to ER" (y , _="e__odel) 0.46, i.e., erasure using the world population (Figure 4,
left). This indicates that the GPTNeoX family of LLMs mimic the training distribution (of coun- try mentions). Furthermore, sure score of the training data compared to ground truth, ER (ptree prrain), is itself 0.16, which closely matches the erasure for models trained on this data. The high correlation between data bias and model bias suggests the composition of training data is key source of erasure in the investigated LLMs.
find that the era-
To measure how well our finctuning strategy gen- eralises, we compare three different ways of per- forming train-test splits of our 955 prompts. These include Random partitioning: we randomly split the prompts into 75% training and 25% test data; Pronouns: we split the prompts based on the pro- nouns they contain, e.g. all prompts containing "she", "you", "we" and "they" are in the training set, "T" and "'he' in the test set; Verbs: we divide along verb groups, e.g. prompts containing "to live in" and be a citizen" of are in the training set, "to reside in" is in the test set. These three setups require increasing levels of generalisation.
4.5 Mitigation
In this section, we explore finetuning as strategy mitigate erasure. We perform gradient updates on a pretrained GPT2 model to minimise the era- sure loss E (p p) on the training data given by prompt data set D. We note that our finetuning strat- egy differs from the related approach from $6.2 of Zhou et al. (2022) in that we have formulated a loss function which allows to perform supervised finetuning. Zhou al. (2022) instead continue training the model using the standard masked lan- guage modelling loss with augmented data related to underpredicted countries.
For all three setups, we repeat the experiment on 5 different folds and plot the results in Figure 5. We find that our finetuning strategy is effective: the av- erage erasure [D] EeED ER ((t""',,(((()))))))___) p(-lc)) is small after 5 epochs of finetuning, both on training (blue) and test data (red). The model generalises well in the random case (Figure 5, left) and to new pro- nouns ( (Figure 5, middle). As expected, verb splits are the most challenging for our model, where we see that the erasure values decrease but not as much as we see in other splits (Figure 5, right). In all cases, we only small deterioration in language modelling performance, as indicated by an approxi- mate 5% increase in the perplexity (The green lines in the plots of Figure 5 correspond to perplexity). We compare this successful mitigation strategy to alternatives in Appendix D.
We use the AdamW (Loshchilov and Hutter, 2017) optimiser with learning rate 3e and train for an additional 5 epochs (including one epoch of warmup under a linear schedule). We find that due to our loss function's 's direct dependency on the logits and the re-normalisation of probabilities over X' (2) our finetuning strategy works best for deter- ministic models, hence set dropout rates to 0 for embeddings, encoder, pooling and attention lay- ers. For finetuning, we update the bias terms only, following Zaken al. (2021). This a memory ef- ficient strategy that is expected to work particularly well in settings with constant outputs (we want the

---

5 Conclusion
Knowledge encoding vs. language generation. Our erasure metric is based on country probabili- ties given a prompt p(aafc). These probabilities can be interpreted as knowledge encoded by the model. When generating text, the model probabilities are used to sample next tokens. Sampling (or decod- ing) can be performed using different strategies, e.g. greedy or beam search (Klein et al., 2017) to max- imise probabilities, top-k (Fan al., 2018 8) and top-p (Holtzman et al., 2019) sampling strategies to generate more diverse outputs.
We motivated the need for large language models to be more geographically inclusive-wiie which remains be an overlooked aspect of inclusive model de- velopment. Specifically, we studied and formalised a notion of geographical erasure, which captures the countries that are underpredicted and the extent to which they are underpredicted. We discussed how our formulation captures many desirable prop- erties. In our experiments, we found clear instances of geographical erasure, which were consistently observed across 10 different language models. Per- haps unsurprisingly, the output probabilities of lan- guage models closely follow the frequencies of country mentions in the training corpora, a likely cause of crasure. We examined a finetuning- based mitigation strategy and found it to be effective in alleviating erasure.
Our work does not analyse the effect of the decoding mechanism since we work directly on p(z;(e) instead of generated text. This is not un- common in prior work, likelihood-based methods such as perplexity or cross-entropy are a customary way to evaluate language modelling abilities, also in modern LLMs (Radford et al., 2019).
Compared to evaluating the full generation pipeline, erasure (as defined in Equation (4)) can be thought of as a lower bound to erasure under sampling: instead of considering the full predic- tive distribution, the above sampling mechanisms only consider high-probability candidates, erasing low-probability countries to even larger degrees.
6 Limitations
Languages considered. We limit our analysis to models trained on English texts, and hence we prompt them in English only. Our methodology extends to other languages straightforwardly. For example, to replicate the geographical experiment with a Spanish language model, one would auto- generate Spanish prompts (or translate the English ones from Appendix B).
Causes of erasure. Our analysis covers two po- tential sources for crasure: training data and model size. Model bias is commonly explained by data bias (e.g. Bender et al. (2021); Schwöbel (2022) and Buolamwini and Gebru (2018)). In our work, we have not experimentally established the cause, our experiments instead indicate high correlation of model biases and data biases in $4.4, suggest- ing that data is a likely source of erasure. Data, however, is not the only biasing factor. Model ar- chitecture and training paradigm determine how the data is used by the model. Hence, they deter- mine whether data bias is mitigated or exacerbated (Hooker, 2021). We examine the impact of model size and find that it has little to impact on geo- graphical erasure $4.3)). Examining the impact of other factors on erasure is left to future work.
The language (of prompts) used to analyze era- sure should be accounted for while collecting ground truth data: for instance, English speaking countries are expected to have higher probability conditioned on "I live in", and similarly Spanish speaking countries conditioned on "Vivo en" are likely have higher probabilities. In our work, factor this by considering English speaking popula- tions as ground truth in $4 (and one would proceed accordingly for a model in a different language).
Difficulty in obtaining ground truth. Language specific ground truth data is less reliable and harder to obtain than raw population counts. Such statis- tics are often self-reported and the level of profi- ciency differs dramatically across regions, espe- cially since the numbers include second language speakers.? Since we only measure erasure for coun- tries where y!tue is available, the availability of lan- guage specific ground truth data is itself a biasing factor. This is evident from Figure 1, which depicts how the lack of ground truth data predominantly affects central African regions.
A Acknowledgements
This work greatly benefited from discussions with Mansi Gupta, Luca Franceschi, Bilal Zafar, Gian- luca Detommaso, Martin Wistuba and Prabhu Teja Sivaprasad. We also appreciate the feedback on our work by Navreet Kaur and anonymous EMNLP re- viewers. DP acknowledges Adobe Inc. and Kotak IISe AI-ML Centre (KIAC) for their support.

---

References
language modeling with pathways, arXiv preprint arXiv:2204.02311.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Al- locative versus representational harms in machine learning. In 9th Annual conference of the special in- terest group for computing, information and society.
Together Computer. 2023. Redpajama-data: An open source recipe to reproduce llama training dataset.
K Crawford. 2017. The trouble with bias-nips nips 2017 keynote-kat kate crawford# nips2017. The Ar- rificial Intelligence Channel. https://www. youtube. com/watch.
Abhipsa Basu, R Venkatesh Babu, and Danish Pruthi. 2023. Inspecting the geographical representativeness of images from text-to- to-a models. Conference on Computer Vision and Pattern Recognition (CVPR).
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova, 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.0440
Emily M Bender, Timmit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM confer- ence on fairness, accountability, and transparency. pages 610-623.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference fairness, accountability, and transparency, pages 862-872.
Sid Black, Stella Biderman. Eric Hallahan, Quentin Authony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-ncox-20b: An open-souree autoregressive language model. arXiv preprint arXiv:2204.06745
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd inno- vations in theoretical computer science conference, pages 214-226622222222
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hicrarchical neural story generation. arXiv preprint arXiv:1805.044 04833.
Grant Blank. 2017. The digital divide among twitter users and its implications for social research. Social Science Computer Review, 35(6):679-69)
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, hts/21001.00277.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of" bias" in nip. arXiv preprint 200.:24005.44000000
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai, 2016. Man computer programmer as woman is to home- maker? debiasing word embeddings. Advances in neural information processing systems, 29.
Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama.
Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embed- dings contain distribution of human- like biases. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122-133.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equal- ity of opportunity in supervised learning. Advances in neural information processing systems, 29.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Conference on fair- ness, accountability and transparency, pages 77-91. PMLR.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint a003v: 155566
Pew Research Center. 2021. Internet/broadband fact sheet.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling
Sara Hooker. 2021. Moving beyond "algorithmic bias is a data problem". Patterns, 2(4).

---

Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stan- forth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019. Reducing sen- timent bias language models via counterfactual evaluation. arXiv preprint arXiv: 1ate1199111......
Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1
Gerald Roche. 2019, Articulating language oppression: colonialism, coloniality and the erasure of tibet's minority languages. Patterns of prejudice, 53(5):487- 514.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-453
William A Gaviria Rojas, Sudnya Diamos, Keertan Ran- jan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The dollar street dataset: Im- ages representing the geographic and socioeconomic diversity of the world. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel- lart, and Alexander M Rush. 2017. Opennmt: Open- source toolkit for neural machine translation. arXiv preprint (ex.:i171100225
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110
Lorenza Romano, Milen Ognianov Kouylekov, Idan Szpektor, Ido Kalman Dagan, and Alberto Lavelli. 2006. Investigating a generic paraphrase-bas proach for relation extraction. In 11th Conference of the European Chapter of the Association for Compu- tational Linguistics (EACL 2006), pages 409-416.
ap-
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Mcasuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint 17Xiv:1804.0930/
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access. multilingual language model. arXiv preprint arXiv:2211.051/
Chandler May, Alex Wang. Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:I903. 10561.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Pola Schwöbel. 2022. Learned data augmentation for bias conrection. PhD Thesis.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.00 09456..
Pola Schwöbel and Peter Remmers. 2022. The long are of faimess: Formalisations and ethical discourse. In 2022 ACM Conference on Fairness, Accountability. and Transparency, pages 2179-2188.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman, 2020. Crows-pairs: chal- lenge dataset for measuring social biases in masked language models. arXiv preprint 2010000000
Emily Sheng, Kai-Wei Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. arXiv preprint wwXiv22005.0455.
Tong Niu, Semih Yavuz, Yingbo Zhou, Nitish Shirish Keskar, Huan Wang, and Caiming Xiong. 2020. Un- supervised paraphrasing with pretrained language models. arXiv preprint arXiv:2010.1228.
Yi Cher Tan and L Blisa Celis. 2019. Assessing so- cial and intersectional biases in contextualized word representations. Advances in neural information pro- cessing systems, 32.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander H Miller, and Scbastian Riedel. 2019. Language models knowl- edge bases? arXiv preprint 1ww.:i::90.000000
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 230.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188)9
US Equal Employment Opportunity Commission. 1979. Questions and answers to clarify and provide a com- mon interpretation of the uniform guidelines employee selection procedures. Federal Register, 40(43).
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John

---

Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bittit: Simple parameter-efficien fine-tuning for transformer-bueed masked language- models. arXiv preprint 106.:2206610199
Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. 2022. Richer countries and richer representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2074-2085.

---

A Choosing additional Models
Secondly, we apply template-based rephrasing similar Romano al. (2006). We replace the subject of the sentence from "I" to alternative pro- nouns ("She", "He", "We", "They", "You") ad- justing verb forms accordingly as well as com- binations of ("My", "Her", "His", "Our", "Their") x "uncle", "aunt", "brother", "sister". "niece". "nephew", "mother", "father", "mom", "daughter", "son", "cousin", "friend", "relative"). This second step extends the 16 base formulations into our final dataset D containing 955 prompts.
Section 3.4 compares ER" for different values of the Kl-divergence. We pick r = 3 in this exper- iment such that ER(p" .p" = ure 6 contains the same experiment for all models under consideration. The optimal choice according to this heuristic r 3 for all of them.
KL(p (p [pp) Fig-
B Details on Prompt Rephrasing
As described in $3.5, we formulate different ver- sions of a seed prompt ="I am from "' in order to produce a set of diverse prompt wordings D. We do this in two steps. First, we prompt Chat- GPT to rephrase the seed prompt "I am from". We manually remove irrelevant prompts, e.g.. those containing country adjectives rather than nouns. This first step extends the seed prompt into to the following 16 base formulations.
C Variability across Prompts
Recall from (8) that
SEg = P4
M( = Ep(c/M) [p(z]e)]]
under uniform p(clM). By Jensen's inequality,
ER' (p (nro P pEg
B.1 List of prompts
puue Eptc/M) [p(xi|c)]
Σ true log
Pi
1. am from
icS,
pruec p(as(c)
2. live in
I M)
Σ true log
LiES
3. I hail from
1
ER' (P
the p(aa(c)).
4. My homeland is
Thus, erasure under the aggregate distribution ER" ER ppee_unn) is a lower bound to the average erasure.
5. I come from
6. was born and raised in
D Alternative Mitigation Strategies
7. am citizen of
Finetuning for other values of r: In $4.5 we miti- gate erasure by finetuning, employing ER3 as loss function (Figure 5). This choice corresponds to a minimal intervention where we only modify the distributions for affected countries at rate above 3; we do not address any underprediction by smaller degree.
8. I originate from
9. My roots are in
10. grew up
11. was brought up in
Finetuning with r = 0 (Fig. 7) is a stronger inter- vention, matching the distributions for all countries
12. was raised
(since (y ,p) = (L())e))))) see $3.4). As before, we can match the full distributions and achieve (n .p) 2 0 after only 5 epochs of finetuning, However, due to the more drastic inter- vention into the model distribution ptuue the drop in language modelling performance larger. Perplex- ity increases by almost 20% compared to 5% in Figure 5. Note the different y- y-axis scales between Figures 5 and
13. I was born in
14. My place of origin is
15. reside
16. My home country is
"Accessed via https:/cchttoopena..cmm/

---

gpt2
gp12-medium
gpl2-large
1.0
1.0
1.0
KLIp"M(lp)
KL
0.8
ER
0.8
ER
0.8
ER (p .p)
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
gpt2 x1
gpt-neo- 125M
gpt gpt-neo-
apt-neo-1.33 1.3B
1.0
1.0
1.0
KL(p)))))) (R
KLIp"Hilp) ER' (R
KL Up)
0.8
0.8
0.8
ER ER .p)
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
6
8
10
0
2
6
8
10
2
6
10
gpt-neo-2.78
gpt-neox-20b
open llama 3b
1.0
1.0
1.0
KLip (R .p)
(l)
KLip Ip)
0.8
0.8
ER (p
0.8
ER ER tr .p)
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
2
2
6
Figure 6: The relationship between ER" and KI.-divergence is similar for all models. We compare ER (blue) for different the KL-divergence (red), repeating the experiment from $3.4 for all models. The median is plotted as a solid line, 25th t5th percentiles as shaded areas (over different versions of the "I live in" -prompt, see $3.5).
We pick r 3, the integer value for which and ER" are most similar for all models (dashed line, black).
Random Splits
Pronoun Splits
Verb Splits
Training
Poplextyy
Training
Perplaxity
Training
Perplexary
0.8
28
0.8
Test
28
0.8
28
ER loss
Parplexity
0.6
26
26
0.4
24
0.4
24
0.4
24
0.2
22
0.2
0.2
22
0.0
20
0.0
0.0
20
2
4
Epoch
Epoch
Epoch
Figure 7: Finetuning at r = also mitigates erasure, though at a higher perplexity cost. Like in Figure we plot average ER" training (blue) and test (red) set prompts during 5 epochs of finetuning of the GPT2-small model. Error bars indicate minima/mmxima over 5 folds. Note the different y-axis scale compared to Figure 7.
No mitigation
ER3
ER
Training loss
0.5722 + 0.0160
0.0054 + 0.0003
0.0211 + 0.0003
0.5641
Test loss
0.6065 0.0543
0.0068 0.0014
0.0566 0.0020
Perplexity
24.6716
25.8694 0.0913
29.5201 0.0297
24.5357
Table 1: Summary of mitigation experiments. Finetuning with ER3, ERd and mitigation via optimising 7. We report the training and test loss well as perplexity after the 5th finetuning epoch. Best values bolded.

---

0.8
0.6
35
Perplexity
ER
0.4-
0.948 Perplexity
0.2
25
0.8
1.0
1.2
1.4
Softmax temperature
Figure 8: Mitigating ER" using the temperature pa- rameter r is less successful than full finetuning, ER" and perplexity are plotted as a function of The op- timal value (minimising ER" w.r.t r) is 0.948, dashed line.
Mitigation via Temperature Softmax: A sim- ple way to modify the model distribution is via the softmax temperature parameter of the model. We have used Here, we experiment with modifying to mitigate ER' (p""e, .p) such that
all previous experiments.
ER" min ER (y p+).
(12)
Figure shows ER" and perplexity as a function r. The optimal value (minimising ER" w.r.t r) is 0.948, dashed line. This mitigation method is compared to fine-tuning of the neural network pa- rameters from earlier experiments in Table 1. The two middle columns correspond to the finetuning results from Figure 5 and Figure 7, the rightmost column contains the results for varying temperature parameter
Perhaps unsurprisingly, mitigation attempts with a single parameter т are much less successful than using full finetuning (small drop in ER" only, see first row of Table l). Perplexity, however, improves slightly over the original model.


Reviews:
[{"review_id": "Gr98Y73Ad2", "reviewer": "Reviewer_MJ7R", "summary": "", "questions": "Q1. On line 177, what does the subscript c stand for? Similar in 186. It's confusing that you define the variable and then use it a subscript.\n\nQ2. From lines 206-207: The ground-truth probabilities are computed from a wikipedia list but the link only lists 12 countries. Why is that? The population statistics is also quite suspect: for example, the size of the English speaking population in the US is the same as the population of the country. Since this is a critical piece in the measurement of erasure, the authors should be more transparent of how they calculated the groundtruth probabilities.\n\n* A stylistic comment: lines 212-216 aren't related to ground truth and it would improve the readability if it is folded in section 3.1\n\nQ3. The summands in Eq.4 can be written as \\sum_i p_i^{true}\\logp_i^{true} - \\sum_i p_i^{true}\\log p_i. Unless I'm mistaken, the left part (the entropy) is model independent and will remain the same for any model. Why not just compute the right part (the cross entropy) as the erasure metric?      \n\nQ4. I wouldn't characterize the analysis in 4.4 as \"causes for erasure\". The paper shows quite clearly that there is correlation between training frequency and the measure of bias that is proposed. Other research has also linked training frequency with bias or accuracy in generation. However, the evidence is correlation and not causation, isn't it?  Training frequency is an artifact of the data and the training protocol and not the (sole) reason for the bias; conversely, if the frequency of country mentions was controlled during training to be equal, it's still likely that you would see erasure. Could the authors provide more explanation? \n\nQ5. The model probabilities could be wildly different if, say for example, the temperature parameter is varied. How should the measure be adjusted to account for the stochasticity due to such hyper parameters? Is that a problem that the authors have encountered?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their in-depth review and are glad to learn that they found our study sound and illuminating, and appreciated the desirable properties of the erasure measure. We will gladly incorporate all suggestions, including clarification on correlation vs. causation as well as presentation and stylistic improvements. Please find detailed comments below.\n\n\n__Subscript of x [Q1]__\n\nThis is a slight abuse of notation which was meant to improve readability. The subscript $c$ of $x$ indicates that $x$ is a token appearing in the prompt $c$. The superscript indicates the position. E.g., for prompt c=”I live in” we write $x_c^1 = $“I”, $x_c^2=$“live”, and so forth. We will clarify this in our draft. \n\n\n__Wikipedia list used for ground truth [Reasons to reject & Q2]__\n\n> Q2 (i): From lines 206-207: The ground-truth probabilities are computed from a Wikipedia list but the link only lists 12 countries. Why is that?\n\nThe discussed [Wikipedia list](https://en.wikipedia.org/wiki/List_of_countries_by_English-speaking_population) contains $127$ countries (at the time of writing the paper and as on August 26, 2023). Those are the $M=127$ countries we include in our analysis (see Sec. 3.1; will be moved to 3.2 as per stylistic comment). We believe the confusion might arise from the fact that we only “show the 12 countries with the largest English speaking populations” (see Figure 2, caption). We will update the section to clarify this. \n\n> Q2 (ii): The population statistics is also quite suspect: for example, the size of the English speaking population in the US is the same as the population of the country. \n\nFor most countries, the data comes from official census reports, which have been collated in the Wikipedia page on [List of countries by English-speaking population](https://en.wikipedia.org/wiki/List_of_countries_by_English-speaking_population). We directly use the data summarised in the Wikipedia page. For the US, figures come from the 2011–2015 American Community Survey 5-year estimates by the U.S. Census Bureau for persons age 5 and older. Total English speakers are those who either spoke English at home (i.e. as first language), or reported speaking another language at home but could speak English \"very well\" or \"well\" (i.e. as an additional language). The total population as per the survey was about 331 million, of which 316 million people were English speakers (about 95% of the population). \n\n\n__Cross-entropy vs. KL-divergence [Q3]__\n\nYes, one can write  \n$ \\text{KL}(p^{true} || p) = \\sum_i p_i^{true} \\log p_i^{true} - \\sum_i p_i^{true}\\log p_i = - H(p^{true}) + H(p^{true}, p), $\n\nwhere the entropy term $H(p^{true})$ is constant. \n\nWe include the constant term in our definition of $\\text{ER}^r$ for the sake of interpretability: Consider the case $r=0$. If $p^{true}=p$ we would like to obtain that $\\text{ER}^0=0$, because no erasure occurs when the distributions match. If we instead defined erasure as the cross-entropy $\\text{ER}^r := H(p^{true}, p)$, we would obtain non-zero erasure for matching distributions (we would then have $\\text{ER}^0(p^{true}, p^{true})=H(p^{true})$). \n\n\n__Training data: causation vs. correlation [Q4]__\n\nThanks for pointing this out, it is true that we have only shown correlation and not causation between training data bias and model bias. We will update the paper to make sure that any hypothesis we propose is clearly marked as such and cannot be confused with verified claims.\n\nWe much appreciate your experiment suggestion of controlling “the frequency of country mentions [...] during training to be equal”. Together with the related suggestions by reviewer u9cg, we will consider this in future work.\n\n\n__Temperature parameter and stochasticity during generation [Reasons to reject & Q5]__\n\nWe analyse the probability distribution that a model assigns to the candidate tokens. Specifically, we apply softmax to the logits returned by the language modelling head to obtain $p(x_k | c)$. Applying a temperature parameter $\\tau$ to this softmax directly modifies $p_{\\tau}(x_k | c)$. Hence, it indeed has a very direct effect on $\\text{ER}^r$. In our work, we have treated this parameter as fixed ($\\tau=1$), but the measure could be adjusted to account for a free temperature parameter $p_{\\tau}(x_k | c)$ by defining $\\text{ER}^r$ as the minimum over $\\tau$:\n\n$\\text{ER}^r = \\min_{\\tau} \\text{ER}^r( p^{true}, p_{\\tau})$.\n\nPicking the optimal tau in this way can be viewed as a single-parameter erasure mitigation strategy: we modify $p_{\\tau}(x_k | c)$, until we find the parameter that minimises erasure. We hypothesise that this strategy will be somewhat successful at mitigating erasure, but will come at the cost of a large perplexity increase. Smaller temperatures lead to predicting the low probability countries more often but will likely present rather drastic interventions in the model distribution (similar, but even more extreme than the $r=0$ experiment we added under review u9cg). We will add this experiment to the camera ready version.\n\nFor sampling strategies (beam search, top-k, top-p, etc.) and their impact on erasure see §6: Knowledge encoding vs. language generation.\n \t\t\t\t\t\t\t\t\n___\n\nWe believe that we address the reviewer’s concerns and have clarified “minor points [that] may need extra support or details” and sincerely hope that the reviewer will consider increasing their soundness rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed measure for erasure has many desirable properties such as interpretability, ease in computation, and generalizability beyond geographical erasure to other types of erasure. The application of this measure for other attributes could help us understand the range of the erasure biases LLMs hold and take steps to mitigate them. The proposed measure also lends itself nicely to accounting for erasure in a loss function and finetuning the model specifically to alleviate erasure.", "reasons_to_reject": "Even though I think the analyses are generally sound and very illuminating, I found some inconsistency in how the groundtruth probabilities were constructed (please see Q2) . Since it's a crucial piece in the measure proposed, it can crucially affect the downstream analysis and mitigation. It is also unclear if and how the measure should be adjusted to account for stochasticity in generation, such as the use of temperature parameter to control generation.", "questions_to_authors": "Q1. On line 177, what does the subscript c stand for? Similar in 186. It's confusing that you define the variable and then use it a subscript.\n\nQ2. From lines 206-207: The ground-truth probabilities are computed from a wikipedia list but the link only lists 12 countries. Why is that? The population statistics is also quite suspect: for example, the size of the English speaking population in the US is the same as the population of the country. Since this is a critical piece in the measurement of erasure, the authors should be more transparent of how they calculated the groundtruth probabilities.\n\n* A stylistic comment: lines 212-216 aren't related to ground truth and it would improve the readability if it is folded in section 3.1\n\nQ3. The summands in Eq.4 can be written as \\sum_i p_i^{true}\\logp_i^{true} - \\sum_i p_i^{true}\\log p_i. Unless I'm mistaken, the left part (the entropy) is model independent and will remain the same for any model. Why not just compute the right part (the cross entropy) as the erasure metric?      \n\nQ4. I wouldn't characterize the analysis in 4.4 as \"causes for erasure\". The paper shows quite clearly that there is correlation between training frequency and the measure of bias that is proposed. Other research has also linked training frequency with bias or accuracy in generation. However, the evidence is correlation and not causation, isn't it?  Training frequency is an artifact of the data and the training protocol and not the (sole) reason for the bias; conversely, if the frequency of country mentions was controlled during training to be equal, it's still likely that you would see erasure. Could the authors provide more explanation? \n\nQ5. The model probabilities could be wildly different if, say for example, the temperature parameter is varied. How should the measure be adjusted to account for the stochasticity due to such hyper parameters? Is that a problem that the authors have encountered?", "paper_topic_and_main_contributions": "The paper quantitatively explores the underprediction of certain countries by large language models in language generation. The authors term this as \"geographical erasure\" and propose a measure to calculate the extent to which some LLMs can be affected by geographical erasure. The key analytical findings of the paper include that many autoregressive LLMs suffer from geographical erasure; model size is uncorrelated with the measure proposed, indicating all models, big and small, may suffer from geographical erasure; and erasure of countries is directly proportional to the frequency of the country mentions in the training data. Finally, supervised finetuning is shown to mitigate erasure without adversely impacting LLMs generative performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* On line 55, need a space at the start of the sentence\n* On line 355, plots --> prompts\n* On line 392, one --> on"}, "extra_text": {}}, {"review_id": "1qIZLHGlQI", "reviewer": "Reviewer_U4e2", "summary": "", "questions": "Why do we need to make the probabilities of ‘I am from ___.' the same as the portion of people who speak English? Moreover, someone who can speak English does not necessarily mean that he/she will only use English on the internet. For example, numerous Japanese people can speak English but that does not mean that they will type English on the Internet.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We’d like to clarify the following points:\n\n**Novelty of our work [as mentioned in reasons to reject]:**\n\nThe literature on fairness and biases has predominantly focused on concerns related to gender, race, occupation, and (in some cases) sexual orientation. For perspective, there are at least 304 published papers that analyse gender bias in NLP (as per Stanczak & Augenstein, 2021). Comparatively, there are few studies that analyse models for geographical biases, and as the reviewer notes, Zhou et al. (2022) is an important exception. The work from Zhou et al. is related to ours in spirit, as it also aims to study biases from a geographical lens. Their work primarily concerns the quality of embeddings for different nations, and they briefly hint at how certain richer countries might be over-predicted. Our work attempts to understand the degree to which certain countries may be underpredicted in generation: to do so, we formalize a notion of geographical erasure, measure erasure for different models (with varying sizes), identify the potential sources of erasure, and evaluate techniques to mitigate it. \n\n> The loss objective is not novel.\n\nThis is _not_ true: Unlike us, Zhou et al. do not propose a custom metric that can be used as a loss for fine-tuning. Their analysis is based on empirical distributions (i.e., the frequency with which country names appear in training data and model generations). In §6.2 they finetune on modified training data using the cross entropy loss. We measure erasure directly using the distributions parameterised by the model which gives rise to a novel, effective finetuning objective.\n\n**Reference Distributions**\n\n> “why models need to treat all countries the same as the objective of pretraining” and “Why do we need to make the probabilities of ‘I am from ___.' the same as the portion of people who speak English?”\n\nOur proposed measure of erasure, and subsequent mitigation, requires a reference distribution (ground truth). This reference should capture some notion of ideal (or aspired) model behaviour. Measuring or mitigating bias is inherently a normative process. We chose a reference distribution of the English speaking population, since the investigated models are monolingual (trained and used in English language). This reference distribution takes inspiration from principles of proportional representation (a notion common in electoral systems), wherein the benefits are divided in the same proportions as the number of people in a given demography. Having said this, one could directly use our formulation for a different reference distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focus on identifying and reduce the problem of underrepresented countries by models, which seems to be novel.\n2. This paper further utilizes the loss function proposed by Zhou et al. (2022) to mitigate the underrepresented phenomenon.", "reasons_to_reject": "1.  The loss objective is not novel.\n2. This problem is not novel.\n3. The prompts seem to be not reasonable as it does not have any other contexts. Evaluating the relationship between English Speaking Population and Prediction Probabilities is not reasonable as not everyone uses the internet and not everyone has access to the internet. \n4. Moreover, it is not sure whether English Speaking Population contains international students or workers.\n5. It is still unknown why models need to treat all countries the same as the objective of pretraining or finetuning is to make the model predict test data. And in test data, that kind of unbiased thing is natural.", "questions_to_authors": "Why do we need to make the probabilities of ‘I am from ___.' the same as the portion of people who speak English? Moreover, someone who can speak English does not necessarily mean that he/she will only use English on the internet. For example, numerous Japanese people can speak English but that does not mean that they will type English on the Internet.", "paper_topic_and_main_contributions": "This paper focuses on measuring and removing the models’ bias towards different countries. They first propose a metric and then use it to measure the importance of different countries to the models. Finally, they reduce the models’ bias by fintuning.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ml2H746GcO", "reviewer": "Reviewer_YS6F", "summary": "", "questions": "Although a series of figures with description could illustrate the experimental settings and design, a clear table with some case study could further help readers to understand the paper, like how the prompt sentences work. Besides, a precise results of experiments listed in the table (like add in the appendix) could bring more inspiration.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and endorsement of our work. \n\n__Interpretability of $\\text{ER}^r$:__\n\nSection 3.4 and Fig. 2 aim to give some theoretical and empirical guidance on how to interpret $\\text{ER}^r$. We appreciate the callout for more discussion on the interpretability of the ER metric and will include additional details in the revision. If the reviewer has any concrete questions or experimental suggestions we will make sure to integrate them.\n\n__Table with experimental detail and reproducibility:__\n\nThanks for the suggestion, we will happily add a table containing the results of our experiments numerically, along with additional details of the experimental setup. We will also release the code upon publication (see footnote 1). In combination, we hope that these two points will address the difficulty the reviewer sees in terms of reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper focuses on a relative unpopular objective, geographical erasure, rather than common discussion on representational harms such as biases related to race, gender, occupation and sexual orientation. However, a complete reading of the problem is given and a series of related experiments are designed on many common large language models to prove the conclusions.\n\n2. This paper proposes a measurement ER to measure the erasure. And base on that the analyses for the models' ability on geographical representation are discussed. Beyond that, the author further proposes a fine-tuning strategy by optimizing the ER to mitigate the erasure problem. The whole framework of this study is a full closure of the loop.\n\n3. Several figures in the paper are well designed and clearly demonstrate the relationship between the pre-training corpora and the model's erasure capability.", "reasons_to_reject": "The presentation of ER measurement is mostly based on frequency and probability, which is intuitive however less interpretability. More discussion regarding ER could be elaborated.", "questions_to_authors": "Although a series of figures with description could illustrate the experimental settings and design, a clear table with some case study could further help readers to understand the paper, like how the prompt sentences work. Besides, a precise results of experiments listed in the table (like add in the appendix) could bring more inspiration.", "paper_topic_and_main_contributions": "This paper aims to address the unbalance and bias happened in pre-training of large language models, specifically for the geographical erasure situation, which is lack of focus. The paper demonstrates consistent instances of erasure across a range of LLMs, of which results can be largely explained by the frequencies of country mentions in training corpora. Although, this paper propose a custom objective for fune-tuning to mitigate the geographical erasure.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CXiLqoaQE3", "reviewer": "Reviewer_u9cg", "summary": "", "questions": "* Could you elaborate on why one could not simply try to mitigate the bias by fine-tuning the model to express the true distribution? \n\n* Source of erasure: A start would be to replace/shuffle country mentions and re-train models (possibly on a smaller scale) and check if they display similar behaviors. An alternative could be methods that use influence functions [2].  (Note: This is not a request for additional experiments, but a suggestion for future works)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful analysis and insightful feedback. We are glad to learn that they found our analysis to be interesting, methods to be sound, and our proposal of measuring and reducing erasure to be useful. We are also grateful for the references and suggestions to improve our work.\n\n__Ground truth and transferability [Reasons to reject]__\n\nOur proposed measure of erasure, and subsequent mitigation, requires a reference distribution (ground truth). This reference should capture some notion of ideal or aspired model behaviour. We acknowledge the difficulty of obtaining ground truth for the geography case in §6: Languages Considered. Our methodology could be applied to other domains and other types of biases. For new domains it may be more difficult to obtain ground truth. On the other hand, there might be domains for which it is easier (e.g., cases where a uniform distribution is appropriate as is assumed in other works, e.g. Liang et al., ‘22). We will expand our discussion on this in the limitation section.\n\nMore generally, it is difficult to effectively measure and tackle biases for domains where ground truth data or a notion of ideal model behaviour is not available. For example, even simple fairness notions like demographic parity can only be computed if the demographic group label is available (gender, race, etc.).  This is a shortcoming of supervised bias mitigation strategies rather than our method specifically. We will include this discussion in the limitations of our method.\n\n\n__Sources of erasure: causation vs. correlation [Reasons to reject, Q2]__\n\nOur claim of causation might indeed be overly strong. We will update the text to state high correlation rather than causation. \n\nWe appreciate your suggestions to experimentally support claims of causation. We hypothesise that after performing the intervention you propose (replacing/shuffling country mentions to obtain uniform rates during training), we would see little-to-no erasure.\nSimilarly, the suggestion about using influence functions to verify causes is also an interesting one. As you mention, retraining a model and computing influence scores to establish causation is outside the scope of the current submission. We are looking forward to exploring these directions in future work.\n\n__Fine-tuning the model to express the true distribution [Q1]__\n\nThis is a great idea and our methodology supports it. Specifically, if we set $r=0$ we include all the terms in the KL-divergence (see §3.4: The choice of $r$). Hence, we minimise the KL-divergence between $p^{true}$ and $p$ on the full domain, i.e, all countries. \n\nThe reason why we did not include this in the original draft is as follows: If one indeed assumes that erasure up to a factor $r$ is acceptable (a user choice), then finetuning with respect to the chosen $\\text{ER}^r$ is a minimal intervention in the sense that it will only modify the distributions for the affected countries at rate $r$. As such, it has a relatively small impact on model perplexity (see Fig. 6 for $r=3$). Since more countries are affected, we would expect finetuning on the full distribution ($r=0$) to have a much bigger effect on perplexity. \n\nWe ran the experiment and indeed found that the impact on perplexity is larger when we train with $r=0$ than with $r=3$; perplexity increases by almost 20% compared to 5%. See this [figure](https://www.dropbox.com/scl/fi/ac9l3xak9kp8r6dfegiu9/finetuning_r-0.png?rlkey=4v6q1fdbasxh3ue75tboufzcp&dl=0), and note the different y-axis label compared to Fig. 6 in the draft.\n\n___\n\nWe believe that the additional experiment and explanations provide “extra support or details” to address the reviewer’s soundness concerns. We hope the reviewer will consider increasing their soundness score, or engage with us in further discussion to point out points that might remain missing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work presents a very interesting study on geographic bias in LLMs. The authors propose a sound method, an interesting analysis, and show the usefulness of their approach across different models (and sizes). Also, they promise to release their code upon publication.", "reasons_to_reject": "Overall, the presented work is interesting and presents various analysis across several models. Some points that could be addressed to improve the paper are:\n\n* While the proposed method is easy to apply directly during fine-tuning (as a loss), one limitation is the transferability to other types of biases where it is more difficult to obtain ground-truth distributions or an exhaustive list of targets to mitigate. (should be included in the limitations section)\n\n* The claim that the training data is a key source for geographical erasure is very strong and should be hedged. While the analysis that reveals a high correlation indicates that this is very likely, additional experiments would be necessary to deduce causation. (See suggestions and also [1])", "questions_to_authors": "* Could you elaborate on why one could not simply try to mitigate the bias by fine-tuning the model to express the true distribution? \n\n* Source of erasure: A start would be to replace/shuffle country mentions and re-train models (possibly on a smaller scale) and check if they display similar behaviors. An alternative could be methods that use influence functions [2].  (Note: This is not a request for additional experiments, but a suggestion for future works)", "paper_topic_and_main_contributions": "This work presents a novel bias mitigation method focused on mitigating geographic biases of LLM generated texts. To do so, the authors define erasure as the 'KL divergence' between true and predicted distributions for 'erased' sets that are constructed using a user-defined threshold r. They then continue to construct 16 different prompts for probing LLMs. In their experiments, the authors probe four different LLMs (with overall ten different model sizes). While they find that a majority of countries (~82.7%) do not suffer from erasure for r=3, they also identify 10 countries that are 'erased' across all models. Their additional analysis then identifies the training data (the Pile) a likely source of such erasure. Finally, the authors show that utilizing their metric for fine-tuning can mitigate existing geographic biases with small deterioration in model performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Hooker, Sara. \"Moving beyond “algorithmic bias is a data problem”.\" Patterns 2.4 (2021).\n\n[2] Zylberajch, Hugo, Piyawat Lertvittayakumjorn, and Francesca Toni. \"HILDIF: Interactive debugging of NLI models using influence functions.\" Proceedings of the First Workshop on Interactive Learning for Natural Language Processing. 2021.\n\nSome more references on model size and biases (so far there seems to be no clear conclusion wrt. this matter):\n\n* Tal, Yarden, Inbal Magar, and Roy Schwartz. \"Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias.\" Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP). 2022.\n* Hooker, S., Moorosi, N., Clark, G., Bengio, S., & Denton, E. (2020). Characterising bias in compressed models. arXiv preprint arXiv:2010.03058.\n* Silva, Andrew, Pradyumna Tambwekar, and Matthew Gombolay. \"Towards a comprehensive understanding and accurate evaluation of societal biases in pre-trained transformers.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n* Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big?🦜. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623)."}}]

Author rebuttal:

Meta-review: The paper "Geographical Erasure in Language Generation" presents work on the imbalance of generated data from LLMs, more concrete about the lack of representation of certain languages in generated data. The paper also presents a method to counter this phenomenon. 

Arguments for rejecting the paper mentioned by the reviewers are related to the novelty of the task and details on basic assumptions.
Arguments for accepting the paper mentioned by the reviewers are related to the aspect studied (geographic erasure) and its potential to extend this to other similar phenomena.
Decision: Accept-Findings