DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity
Authors: Melissa Hall
Candace Ross
Adina Williams
Nicolas Carion
Michal Drozdzal
Adriana Romero-Soriano
Venue: Accepted by TMLR ()

Abstract: The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, we introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparities, an important step towards building responsible visual content creation systems. We use our proposed indicators to analyze potential geographic biases in state-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at a cost to prompt-consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone. Code is available at https://github.com/facebookresearch/DIG-In/.

Published in Transactions on Machine Learning Research (12/2023)
DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity
Melissa Hall", Candace Ross Adina Williams", Nicolas Carion', Michal Drozdzal Adriana Romero 1.2.3
'FAIR, Meta, Milaa Quebec AT Institute, 8 MeGill University, + CCanada CIPAR AT Chair
Reviewed on OpenReview: https: // opwreevew..........m net/ fornn? id- FDt2UGH1N2
Abstract
The unprecedented photorealistic results achieved by recent text-lo-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparitics, an important step towards building responsible visual content creation systems. We use our proposed indicalors to analyze potential geographic biases in saate-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at cost to prompt- consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone. Code is available hhtpps //gtthbb.cc/accccboreeeea//////a
1 Introduction
Over the last year, research progress on generative models for visual content creation has experienced a large acceleration. Recent text-to-image generative systems such as Stable Diffusion (Rombach al. 2021
DALL-E 2 (Ramesh et al. 2022). Imagen (Saharia et al. 2022), and Make-a-Scene (Gafni et al.) 2022
have shown unprecedented quality. These systems enable widespread use of plug-and-play content creation solutions, making it crucial to understand their potential biases and out-of tnrooff-raiinng---e training-distribution distribution behavior.
Prior works in the text and vision domains advocate for detailed documentation of model behavior Mitchel et al. 2018), , introduce benchmarks (Schumann et al., 2021 Hazirbas et al., 2021] , and study the potential fairness risks and biases of vision (Buolamwini Gebru 2018: DeVries et al. (019a)), language (Bianchi
et al., 2022 Smith al. 2022) and vision-language (Zhao al.
2017 Hall et al. 2023b) systems trained
large scale crawled data. However, only handful have focused better understanding the potential
risks and biases of text-to-image generative systems (Cho et al. 2022 Luccioni et al. 2023 Bansal et al.
2022 These works either propose qualitative evaluations to highlight biases in generated images or provide narrow quantitative methods focusing on demographic traits of people represented in the images (e.g. gender and skintone) and do not take into consideration metrics commonly used in the image generation literature.
Beyond human-centric representations, stereotypical biases may occur for representations of objects across the world. Disparities have been observed between continents like Furope, Africa, and the A Americas within

---

Published in Transactions on Machine Learning Research (12/2023)
Region Incicator
Region- Object Indicator
Object Consistency Indicator
Mosorzee with Peecizion & Conncage
Meazured with Dewciaion Coveeage
Missured with CLIFSoore
Law
Blgh
Divecalty
Realism Diveratt
Regloa=Objeen
Begonn-Obeett
Genacradd
Genicaadd:
Los Realiam Diversity
High Devissity
Adica
kanmue
Regron
Eonstetency
Africs
Europe
Modell
High Modii
Precisin
50
40
30
Africe
to Sanope?
20
40
20
40 -
Consletency Modil
Coverage -ice
Coverage
Cooking pat
Dita poiata cxnete differens geonyyt ling kites and proccutiv avsiat.
High
Model
We introduce three quantitative Indicators for measuring gaps in performance between geographic regions in text-to-image models. These Indicators allow for the identification of regions and objects for which state-of-the-art models perform poorly, as shown in the examples here.
Figure 1:
third-party object classification systems DeVrrees et al. 2019a), weakly- or self-supervised models (Singh let al. 2022: Goyal et al., 2022). and large scale languge models (Gustalson et al. 2023 Yet the evaluation of geographic representation of objects and their surroundings in text-to-image systems is under-explored, and mainly utilizes qualitative studies among human annotators to highlight goographic biases in generative text-lo-image models.
In this work, we build upon these studies by introducing quantitative benchmarks of geographic disparities in generative models that complement human studics. We introduce three indicators for evaluating the realism, diversity, and consistency of text-lo-image generative systems across geographic regions: the Region Indicator probes for disparities in realism and diversity of generated images, the Object-Region Indicator measures object-specific realism and diversity of generated images, and the Object Consistency Indicator evaluates prompt-generation consistency (sce Figure D The Region and Object-Region indicators are grounded with reference datasets of real images and adaptable to any real-world dataset of reference images and object prompts.
We then use the indicators to evaluate widely used state-of-the-art text-to-image generative systems. These include two versions of latent diffusion models (Rombach et al. 2021) from free, publicly accessible APTs, denoted as "LDM 1.5 (Open)" and "LDM 2.1 (Open)," " and one from a closod, paid API, denoted as "LDM 2.1 (Closed)." " We also evaluate a diffusion model utilizing CLIP image embeddings (Ramesh et al.] 2022), denoted as "DM w/ CLIP Latents," " well as GLIDE (Nichol et al., 2021). We focus on evaluating whether these systems create content representative of objects' geographical diversity around world regions. We study more than 40 objects using prompts of increasing geographic specificity. For each indicator, we also perform qualitative analyses that demonstrate its efficacy in identifying relevant disparities in generations.
The findings of our analysis are summarized as follows:
Text-to-image systems tend to generate less realistic and diverse images when prompting with Africa and West Asia than Europe as compared to real, representative images in these regions.
Prompting with geographic information can reduce the scores of our indicators, highlighting poor object consistency, limited representation diversity, and stereolyped generations.
Text-to-image systems tend exhibit region-level disparitics more for some objects than others. In some cases, this is due to variations in how objects are represented in the reference dataset of real images, but more often is the result of stereotypes embedded in text-to-image systems.
2

---

Published in Transactions on Machine Learning Research (12/2023)
Across regions and indicators, the older, publicly accessible latent diffusion model and the newer, closed version tend to be the best and worst performing models, respectively. The DM w/ CLIP Latents model has strong consistency and realism, while GLIDE has strong realism, but suffers from poor diversity and consistency.
Although these systems have improved in image quality and consistency on standard benchmarks over time (Rombach et al.) 2021 Gafni ct al.) 2022 Ramesh ct al.) 2022 Saharia et al.; 2022), our indicators suggest that this may have been at the expense of werl-world geographic representation. For example, the more recent latent diffusion model ranks lower in all three indicators than an earlier version trained with less data.
We hope that our indicators and their use in performing a comprehensive evaluation of sources of geographic bias in generative image models constitute an essential first step towards responsible visual content creation.
2 Related work
Bias and diversity in vision and vision-language systems. Prior work shows that many societal biases are learned by vision models: race and gender biases are contained in the embedding space of models (Steed & Caliskan 2021 Wolfe & Caliskan 2022b), and disparities for different demographic groups occur in lasks such image classification (Goyal et al. 2022 Hall et al. 2023a and object detection (DeVries el. al. Gustafson et al. 2023). Similar to vision models, vision- language models, e.g. CLIP
2019a;
RRafordd et al.
2021 Srinivasan & Bisk 2021 Wolfe & 2021), contain harmful blases in their embedding spaces (Ross et al.
Caliskan 2022a). Models also show biases on rownstream lasks incl including image captioning (Hendricks el al. 2018) and video understanding (Hazzibas ef al. 2021 Porgali et. al. 2023). In addition, these analyses extend
to the large-scale datasets used for training and benchmarking (Birhane et al., 2021 Hazirbas et al.) 2021 Porgali et al.] 2023 Ramaswamy et al.) (023). These works tend to focus on demographic biases related to visual person-related attributes, with less attention on geographic biases.
Bias and diversity in generative text-to-image systems. Recent papers analyzing biases in generative
models (Luccioni et al. 2023 Friedrich et al. 2023 Cho et al. 2022 Bansal et al. 2022) utilize automated
metrics to identify biases in image generative systems but primarily focus on disparities across gender and ethnicity of groups of people rather than geographically diverse representations of objects. Bianchi et al. [2022) study both demographic biases and geographic diversity in text-to-image systems qualitatively, showcasing
the presence of bias in text-to-image systems. Concurrent work (Basu et al. 2023 measures the geographic representation of objects in text-to-imann to-image systems by performing user studies on a subset of 10 objects. None of these works performs a systematic quantitative analysis of object geodiversity of text-to- tonttto--mmgeeee systems, grounds the analysis on a reference dataset, or considers a large scale number of objects and their associated gencrations.
Generative models metrics. The evaluation of conditional image generative models often involves different metrics to assess desirable properties of the models, such as image quality/realle generation diversity, and conditioning-general
consistency (DeVries et al. 2019b Fréchet Inception Distance (FID; Heusel et al. 2017bb and Inception Score (Salimans et al.) 2016 have been extensively used to measure image generation quality and diversity. These metrics yield distribution-specffi scores grouping quality and diversity aspects of conditional image generation, therefore hindering the analyses of trade-offs and individual failures cases. To overcome these limitations, precision and recall-based metrics (Sajjadi et al., 2018
Kynkäänniemi et al. 2019 Naeem et al., 2020 Shmelkov et al. 2018 Ravuri & Vinyals 2019) disentangle visual sample quality/ rrallssm and diversity into two different metrics. The diversity of generated samples has also been measured by computing the perceptual similarity between generated image patches (Zhang et al. 2018) or, more recently, the Vendi score (Friedman & Dieng 2022). In addition, consistency of generated images with respect to the input conditioning has been assessed by leveraging modality specific metrics and pro-trained models. For example, the CLIPscore (Hessel et al., [002]] computes the cosine similarity between text prompts used in text-to-image generative models and their corresponding generations, and TIFA (Hu) et al. 2023) assesses consistency through visual question answering. In this work, we introduce indicators
3

---

Published in Transactions on Machine Learning Research (12/2023)
Prompt {object) in (region)
(object) in
Model
Dataset
{object)
(country)
Object-region Balanced (29K)
Object-region Balanced (29K)
Object-region Balanced (29K)
LDM 1.5 (Open)
GeoDE
LDM 2.1 (Open)
LDM 2.1 (Closed)
DollarStreet
Full (22K)
Full (22K)
Full (22K)
GLIDE
Object-region Balanced (29K)
Object-region Balanced (29K)
GeoDE
x
DM w/ CLIP Latents
DollarStreet
x
x
x
Table 1: Number of samples generated per dataset, for each model and prompt. Full indicates the original distribution of the corresponding dataset. Object-region Balanced denotes fixed number of images for each object and region. Using balanced dataset controls correlations between regions and objects.
that leverage precision, coverage, and CLIPscore report measurements disoggregaled by geographic region, allowing for gramular understanding of potential disparities of text-to-image generative models.
3 Experimental Set-up
The proposed indicators require access to reference datasets of real images, a pre-trained image feature extractor, and pro-trained text-to-image systems which are queried to gather datasets of generated images. We introduce the datasets, models, and prompts used in this Section and describe the Indicators in Section
3.1 Reference, real world datasets
We use GeoDE (Ramaswany et al. 2023) and DollarStreet (Rojas et al. 2022 reference datasets to compute realism and diversity of the generated images. Example images are shown in Appendix A.1
GeoDE contains images objects classes across the geographic regions of Africa, the Americas, West. Asia, Southeast Asia, East Asia, and Europe. Images were taken by residents of each region and are subject to strict quality requirements ensuring objects fill at least 25% of the image and not occluded or blurred.
DollarStreet contains images of approximately 200 concept classes across Africa, the Americas, Asia, and Eu- rope. Professional and volunteer photographers especially targeted "poor and remote environments" (Gapmin der 2021) to take photos of relevant objects and scenes. We filter out images containing multiple object labels.
We filter each dataset include only classes that are objects and have images for all regions. We exclude subjective classes (e.g. nácest shoes or favorite home decorations). While GeoDE is roughly balanced across regions and objects by design, we sample to ensure the same number of images for each object and region to control for any correlations. This yields 27 objects with 180 images per object-region combination. DollarStreet is quite imbalanced, so use the original distribution to maintain a sufficient number of samples for each region. This yields 95 objects with approximately 3,500 images from Africa, 3,500 images from the Americas, 9,000 images from Asia, and 2,700 images from Europe. In order to use the image feature extractor required to compute the proposed geodiversity indicators, we center crop all images.
3.2 Models
We evaluate generated images across five different models: two from commercial APIs and three from open source APIs. To maintain consistency, all image resolutions are 512 512. Where possible, we use the safety filter associated with each system. We find that the filter affects approximately 0.3% of generated images, roughly balanced across region and prompt. All models were accessed in May and June of 2023.
4

---

Published in Transactions on Machine Learning Research (12/2023)
We first evaluate several versions of latent diffusion models (Rombach et al. 2021 term "LDM 1.5 (Open)," is trained a public dataset of approximately 2 billion images followed by further steps on higher resolution images and fine- tuning on aesthetic images. We access the model via an API containing open-sourced model weights. The second, "LDM 2.1 (Open)," is initially trained on a public dataset approximately 5 billion images excluding explicit material, further trained for multiple iterations on increasingly high-resolution samples, then fine-tuned. We also access this model via an API containing open-sourced model weights. The third, "LDM 2.1 (Closed))" is similar to the aforementioned model but accessed via a paid API intended for artistic creativity. We follow the recommended parameters for each system, and sample with 30 diffusion steps for LDM 2.1 (Closed) and with 50 steps for the open sourced API. We use classifier-fres free guidance scale of 7
The first, which we
We also evaluate GLIDE (Nichol et al. 2021). an open-sourced diffusion model using classifier free guidance trained on a dataset filtered to remove images of people and violent objects. We follow the suggested parameters and sample with 100 diffusion steps. Finally, we evaluate a model that uses a multi-modal implementation of generative pre-trained transformer leveraging CLIP image embeddings (Ramesh et. al.) 2022) and has approximately 3.5 billion parameters. We call this model "DM w/ CLIP Latents," and access it through a paid API.
3.3 Querying text-to-image generative systems to obtain datasets of generated images
We query text-to-image systems with prompts of varying geographical specificity. First, we prompt with [object]; referring to any object class name in the reference dataset. Second, we prompt with [object) in (region); using the region defined in the real world datasct. Third, to further understand how the specificity of the geographic prompting afTects the image generation, we use the prompt {object) {country). For each region and object combination in GeoDE, we balance the prompt across the top three countries for the respec- tive region included in the reference datasct. For DollarStroet, we use the most represented countrics per region.
In Table we show the quantity of images generated for each prompt. In our generations, we match the balanced version of GeoDE, consisting of 27 objects each represented by 180 images per region. For DollarStrect, we follow the original distribution of images of the reference dataset.
4 Geodiversity indicators
In this section, we first discuss relevant metrics used in the image generation literature and how they map to realism (also referred to as quality), diversity, and consistency generated images. Then, we introduce geodiversity indicators that build on these metrics to quantitatively evaluate disparities across regions.
4.1 Realism, diversity and consistency metrics
Rather than using single metric like FID (Heusel et al., 2017b) to evaluate disparities, we disaggregate across several metrics in order to better understand possible trade-offs between realism, diversity, and consistency.
Realism (Precision): We measure realism by quantilying how similar generated images are to a reference dataset of real images using precision. We follow the definition of precision introduced in (Kynkäänniemi et al., 2019), which determines the proportion of generated images that lie within manifold of real images. The manifold of real images is estimated by building hyperspheres around data points of real images, where the radii of hyperspheres is controlled by the distance to the k-th nearest neighbor in pre-defined feature space, e.g., the feature space of the Inceptionv3 (Szegedy et al. 2015). Formally, precision is computed as:
D
P(D,, Da)
Σ 1=1
1 hg Emanifold(P.))
(1)
[D,]
We that increasing the number of steps does not appear to alter the trends observed this analysis. Similarly, increasing the classifier guiddnnc does not result any significant improvements of the observed consistency shortcomings.
5

---

Published in Transactions on Machine Learning Research (12/2023)
where D, ( is dataset of real image features, and Dg (1g') is dataset of generated image features.
It is worth noting that, in the case of geodiversity indicators, we aim to account for variations in the distribution of reference images across geographic region. For example, regions with more economic diversity may have larger manifolds of images due to more variation in objects related to economic status, such as houses and cars. As result, improved metrics that mcasure realism in terms of density may not be robust. these variations, density more strongly rewards generated images whose features occur in regions densely packed with real samples.
Diversity (Coverage): To measure diversity of generated images, As in the case of precision, coverage requires estimating the manifold of real images from reference dataset. Coverage is mcasured as the proportion of real-data hyperspheres that contain generated images and is more robust to outliers than recall the metric that is usually used to complement precision. Because it measures the extent to which real data is represented by the generated data, coverage indicates the generated images' diversity. Formally, coverage is computed as:
coverage (Naeem et al. 2020
D. Σ
1
C(Dr, D
1
Bist Ai chypersphere (1.)))
(2)
It worth noting that some text-to-image systems generate unrealistic - e.g. low quality or inconsistent yet diverse samples, thus metrics that rely on estimating the manifold of generated images, such as recall (Kynkaänniemi al. 2019), may lead to manifold over-estimations which in turn may result in unreliably high measured diversities.
Consistency (CLIPScore): We measure the input-output consistency of the generative model by comput- ing the CLIPscore (Hessll et al. 2021 i.e., cosine similarity between two embeddings. The first embedding corresponds to the text prompt CLIP embedding used to condition the text-to-image generation process, whereas the second embedding corresponds to the CLIP embedding of the generated image. Intuitively, the higher the CLIPscore, the higher the input-output consistency. Formally, CLIPscore defined as:
fo
Igg
fe (p(9)
CLIPscore 2ig 2
(..0
(3)
llfo
2gg
(2) I1 (p())
where ag an image generated with the prompt p(0) fo is the visual CLIP embedding function, and fr is the textual CLIP embedding function.
(4)
4.2 Region Indicator
Goal. The Region Indicator seeks to probe Lexl-to-image generative systems for disparities in the realism and diversity generated images across different regions.
Requirements. The Region Indicator requires a reference dataset of real images containing images from around the world and associated regional labels, a generated dataset of images from a pre-trained text-to-image generative system, and pre-trained image feature extractor we use the InceptionV3 (Szegedy et al. 2015
given its ubiquity in the literature to compute metrics such as FID (Heusel et al. 2017a Casanova al.
2021 Ramesh et al. 2022 Rombach et al. 2021
For reference datasets, we split GeoDE into 6 regional datasets each containing images of objects from non-overlapping regions in the original dataset. We split DollarStreet into 4 regional datasets each containing images of objects from non-overlapping regions in the original dataset.
For generated datasets, we consider three sets of generated images obtained using the [object), {object) in {region}, and {object) in {country) prompts. Following the reference datasets, we split each of the

---

Published in Transactions on Machine Learning Research (12/2023)
generated datasets into regional datasets containing generated region-specific images. For the {object) in (region) prompt, we consider all object images generated for given region to constitute each regional dataset. For the {object} in {country) prompt, we compose regional datasets by merging generations for all countrics within the same region, where the country list is defined by the GeoDE and DollarStreet datasets. For the {object) prompt, we sample the image generations such that the object distribution matches the one in the reference region-specific datasets.
Metrics. We compute precision and coverage per region, where each region is represented by the set of objects in the reference datasets. In particular, we compute P(D,, D(',, where D.d is the dataset of features of real images from region R, and Df is the generated dataset of features of generated images associated with region R. In the same spirit, we compute C(D D.). This coverage definition measures the overall regional diversity by grouping object-specific conditionings according region.
4.3 Object-Region Indicator
Goal. The Object-Region Indicator seeks to probe text-lo-image generalive system by measuring object- specific realism and diversity of generated images of objects across different regions, complementing the Region Indicator by providing detailed information at object level.
Requirements. The Object-Region Indicator requires a reference dataset real images containing images of objects from around the world and their associated object class and regional labels, a generated dataset of images from pre-trained text-to-image generative system, and pre-trained image feature extractor once again, we use the InceptionV3.
For reference datasets, we split GeoDE into every possible object-region combination present in the original GeoDE dataset. We do not use the DollarStreet dataset as it is very imbalanced and the number of samples for certain object-region combinations too small.
For generated datasets, we consider three datasets of generated images, obtained using the {object), [object] in {region}; and {object} in {country) prompts. Following the reference datasets, we split the generated datasets into as many datasets object-region combinations present in the GooDE datasct. As in Section 4.2 we merge {object} in {country) generations for all countries within the same region and sample the [object] generations to follow the reference distributions of object-region images.
Metrics. We compute precision and coverage per object-region combination. In particular, we compute P(DO.R g DO."), where DO.1t is the dataset of features of real images of object O in region R, and DO. is the dataset of features of generated images corresponding to object O and region R. Similarly, we compute C(DO.R, DO.R). Note that, contrary to the region indicator, the object-region coverage measures object-region conditional diversity í.e. given an object-region prompt, how diverse the generations are.
4.4 Object Consistency Indicator
Goal. The Object Consistency Indicator seeks to probe text-to-image generative systems for disparities in consistency across generations of specific objects from around the world.
Requirements. By contrast other indicators, the Object Consistency Indicator does not rely on a reference dataset of images. Instead, it requires dataset of bbject-prompte generated dataset of images from a pre-trained text-to-image generative system, and pre-trained image and text feature extractors (CLIP).
For object prompts, we build a dataset of prompts {object} featuring all objects in the GeoDE dataset.
For generated datasets, we consider three sets of generated images, obtained using the {object), {object) (region}, and [object} datasets as object-region combinations present in the GeoDE dataset. We follow the procedure described in sections 4.2 and 4.3 to compose region-specific datasets of generated images.
{country} prompts. We split cach of the generated datasets into as many
7

---

Published in Transactions on Machine Learning Research (12/2023)
Metrics. For each generation, we compute the CLIPScore between the generated images and the {object) prompt as follows: CCIPscore
(z( p!()) where (0) is the generated image obtained by conditioning the
text-to-image system with prompt p(.) Next, stratify the scores by object. For each object class, we We report the 10th-percentile CI.IPSoore across all objects as the
compute the 10th percentile CLIPScord
consistency indicator.
5 Results and observations
Model
LDM 1.5 (Open): [ooject)
LDM 2.1 (Ogen): {object)) {country3
LDM 2.1 (Closed): (object)
LDM 1.5 (Open): [ooject)
(region) (country)
GUDE: [object)
LDM 2.1 (Closed): (object)
[country]
LDM 1.5 (Open): (ooject)
GUDE: [object) [region)
DM w/ CLIP Latents: [object)
LDM 2.1 (Open): cooject)
GUDE: [object) [country]
DM w/ CUP Lahents: (object) (region)
LDM 2.1 (Open): (ooject)
(region)
LDM 2.1 (Closed): (object)
DM w/ CLIP Latents: (object) [country)
Africa
Americas
EastAsia
50
Precision
Precision
Precision
40
40
40
30
30
30
20
20
20
20
40
20
40
20 Coverage
40
Coverage
->>
Coverage
Europe
SouthEastAsia
WestAsia
Precision
Precision
Precision
40
30
30
20
20
20
20
40
20
40
20
40 Coverage -->
Coverage -->
Coverage -.>
Figure Precision (quality) and coverage (diversity) measurements evaluated with the GeoDE dataset.
We present quantitative results for the three indicators and perform qualitative analyses for exemplar regions and objects. Additional qualitative examples are included in the Appendix. For each indicator, we first discuss region-level disparities, then trends that occur for all regions.
5.1 Region Indicator
Figure 2 show results for the Region Indicator using GeoDE the reference dataset, and results with DollarStreet are shown in Appendix A.5.
Disparities when prompting without geographic information. For the {object} prompt, precision and coverage are quite similar across regions. As an example, the first column of Figures 3 and 4 show images for car and slove. Indeed we see that, for a given model, the generated images are not especially similar to a specific region as compared to the others.
BBy looking at the distribution of CLIPscores, observe that the spread of the distribution, and in particular, the tail of the distribution corresponding to the lowest CLIPscores, tend to be the most informative capture inconsistencics in the generations. This perhaps unsurprising given how CLIPscore has been used filtering tool to display the best image generations system Hee et 2022
8

---

Published in Transactions on Machine Learning Research (12/2023)
Africa
DM w/ CLIP Latents
DM w/ CLIP Latents
LDM 1.5 (Open)
LDM 2.1 (Closed)
Africa
Europe
1.5 (Open)
Europa
West Asia
LDM 2.1 (Closed)
West
(a) GeoDE.
(b) {object}.
(c) {object] in fregion).
Figure 3: Random examples of cars. Sub-figure (a) Real images from Africa, Europe and West Asia. Sub-figure (b) Generations obtained with {object] setup using three models. Sub-figure (c) Generations obtained with [object] in {region] for three models and three regions.
Africa
DM w/ CLIP Latents
DM w/ CLIP Latents
LDM (Open)
LDM 2.1 (Closed)
Africa
Europe
LDM 1.5 (Open)
Europe
West Asia
LDM 2.1 (Closed)
West Asia
(a) GeoDE.
(b) {object}.
(c) {object} in {region]:
Figure 4: Random examples of sloves. Sub-figure (a) Real images from Africa, Europe and West Asia. Sub-figure (b) Generations obtained with [object) setup using three models. Sub-figure (c) Generations obtained with {object} in {region] for three models and three regions.
9

---

Published in Transactions on Machine Learning Research (12/2023)
DM w/ CLIP Latents LDM 1.5 (Open)
DM w/ CLIP Latents
LDM 1.5 (Open)
Africa
Africa
Europe
Europe
West Asia
West Asia
(a) Car
(b) Stove
Figure 5: Random examples of generations of cars and stoves obtained with {object} in {country) setup.
Disparities when prompting with region. When prompting with region, patterns of disparities appear. When evaluating with GeoDE the reference dataset, generations corresponding Africa, East Asia, and West Asia tend to have poorer coverage and lower precision than Europe and the Americas. Upon visual inspection, generations using region prompting tend to be homogeneous and stereotyped for lower performing regions and do not capture the diversity of the objects in the real dataset. For example, generated images of cars prompted for Africa (Figure 3) are almost exclusively boxy SUVs in rural or desert-like backgrounds, which are rare in the real dataset. Generated images of sloves in West Asia (Figure 4 are circular basins filled with coal (for DM w/ CLIP Latents) or stove-oven combinations outdoor settings (for LDM 1.5 (Open)), neither of which appear often the real datasot.
Disparities when prompting with country. We find that the coverage increases for all models except GLIDE, and most regions, when prompting with country names rather than regions. This generally comes at little cost to, and in some cases helps, precision. Visual inspection confirms that prompting with country rather than region tends to yield more diverse generations. For example, the images of stoves generated by DM w/ CLIP Latents and LDM 1.5 (Open) for the region prompt (Figure 4 are more homogeneous for a given region than when prompted with {object} in {country) (Figure 5). Similarly, the background of images showing cleaning equipment show both indoor and outdoor scenes across all regions when prompting with country, while generally depicting only with country information has mixed effect, increasing coverage for Africa and East Asia but decreasing coverage and precision for all other regions.
the other for the region prompt. For GLIDE, prompting
Observations consistent across regions. The more recent LDM 2.1 (Closed) and LDM 2.1 (Open) tend to have lower precision and coverage than the older LDM 1.5 (Open). This could be due to extra rounds of aesthetic training in latter versions of the models, causing greater deviation from real-world images. In addition, all models except DM w/ CLIP Latents have a degradation in precision and coverage when including region information in prompts. We suspect this is because prompting with geography causes stereotypical
10

---

Published in Transactions on Machine Learning Research (12/2023)
bag
cooking_po
dog
100
100
100
80
80
80
80
Precision
60
Precision
60
Precision
60
Precision
60
40
20
20
20
20
0
25
50
100
0
50
75
100
0
25
75
100
25 Coverage
50
100
Coverage
Coverage ..>
Coverage
100
hand soap
100
medicine
100
stove
100
toothbrush
80
80
80
80
Precision
Precision
60
Precisoon
Precision
40
40
40
40
20
20
20
20
%
25
50 Coverage -->
100
0
25
50
75
100
0
25
50
75
100
25
50
100
Coverage -->
Coverage
->
Coverage -->
Model and region
Africa. LDM 1.5 (Open)
EastAsia, LDM 1.5 (Open)
SouthEastAsia, LDM 1.5 SouthEastAsia,
Africa, LDM 2.1 (Closed)
EastAsia, LDM 2.1 (Closed)
2.1 (Closec)
Alrica, DM w/ CLIP Latents
EastAsia, DM w/ CLIP Latents
SouthEastAsia, DM w/ CLIP Latents
Americas, LDM 1.5 (Open)
Europe, LDM 1.5 (Open)
WestAsia, LDM 1.5 (Open)
Americas, LDM 2.1 (Clased)
Europe, LDM 2.1 (Closed)
WestAsia, LDM 2.1 (Closed)
Americas, DM w/ CLIP Latents
Europe, DM w/ CLI Latents
WestAsia, DM w/ CLIP Latents
Figure 6: Region-Objed Indicator for {object> prompt.
backgrounds and objects that are generally not represented in the real, reference dataset. Our DollarStrect results provide additional insights in this regard: while our previously observed patterns across models and prompts are generally consistent when evaluating with DollarStreet, we find that prompting with regions shows an improvement for Africa (shown in Appendix A.5). This reversal in trend for Africa between GeoDE and Dollar Street may be due to Dollar Street's focus on lower-income, less photographed locations, well its lack of enforemmnnt of a minimum size for pictured objects. Tts images likely contain lower-income settings and pronounced backgrounds more similar to the generated images depicting stercotypes of the region.
5.2 Region-Object Indicator
show precision and coverage results for the Object-Region Indicator using eight objects found in GeoDE for DM w/ CLIP Latents, LDM 2.1 (Closed) and LDM 1.5 (Open). We focus on these objects in particular they highlight notable patterns of disparities that can be identified with this Indicator. Additional results covering all models and objects can be found in the Appendix.
Figures U
Disparities when prompting without geographic information. When interpreting the {object} prompt results in Figure D we observe that precision and coverage patterns vary across objects. First, objects such as bag exhibit rather consistent precision and coverage across regions, but vary across models. The low variance the inter-region measurements suggest that given the generated images remain constant across comparisons bag is similarly represented across regions in the reference dataset. Figure g shows that this is true for GeoDE. While generations of DM w/ CLIP Latents and LDM 1.5 (Open) depict bags, those of LDM 2.1 (Closed) align with the model's lower precision and coverage in rarely showing bags.
Second, objects have more variation in coverage across regions than in precision for all models. For example, the concept dog tends to have lower coverage in West Asia than other regions, whereas coverage for Europe tends to be higher. This suggests there are differences representation of the concept dog the reference dataset between these regions. In Figure I0 we observe that dogs lying by grass and photographed from above appear more frequently for West Asia GeoDE and not common in the generations.
11

---

Published in Transactions on Machine Learning Research (12/2023)
bag
cooking_poi
dog
100
100
80
80
80
80
Precision
60
Precision
60
Precision
60
Precision
60
40
20
20
20
20
0
25
50
100
0
50
75
100
0
25
75
100
25 Coverage
50
100
Coverage
Coverage ..>
Coverage
100
hand soap
100
medicine
100
stove
100
toothbrush
80
80
Precision
Precision
Precision
Precision
40
40
40
40
20
20
20
20
0
25
50 Coverage -->
100
0
50
75
100
0
25
75
100
25 Coverage -->
50
100
Coverage -->
Coverage
->
Model and region
Africa. LDM 1.5 (Open)
EastÁsia, LDM 1.5 (Open)
SouthEastAsia, LDM 1.5 (Open) SouthEastAsia.
Africa, LDM 2.1 (Closed)
EastAsia, LDM 2.1 (Closed)
2.1 (Closec)
Alrica, DM w/ CLIP Latents
EastAsia, w/ CLIP Latents
SouthEastAsia, DM w/ CLIP Latents
Americas, LDM 1.5 (Open)
Europe, LDM 1.5 (Open)
WestAsia, LDM 1.5 (Open)
Americas, LDM 2.1 (Clased)
Europe, LDM 2.1 (Closed)
WestAsia, LDM 2.1 (Closed)
Americas, DM w/ CLIP Latents
Europe, DM w/ CLI Latents
WestAsia, DM w/ CLIP Latents
Figure 7: Region-Object Indicator for {object} in {region} prompt.
bag
cooking_pes
dog
100
100
80
80
80
80
Precision -.>
Precision
60
60
Precision
60
60
40
20
20
20
20
0
25
50
100
0
50
75
100
0
25
75
100
25 Coverage
50
100
Coverage
Coverage ..>
Coverage
100
hand, soap
100
medicine
100
stove
100
toothbrush
80
80
Precision
Precision
Precision
Precision
40
40
40
40
20
20
20
20
0
25
50
100
50
75
100
25
75
100
25 Coverage -->
50
100
Coverage -->
Coverage -.>>
Coverage
->>
Model and region
Africa. LDM 1.5 (Open)
EastÁsia, LDM 1.5 (Open)
SouthEastAsia, LDM 1.5 (Open)
Africa, LDM 2.1 (Closed)
EastAsia, LDM 2. (Closed)
SouthEastAsia 2.1 (Closed)
Atrica, DM w/ CLIP Latents
EastAsia, w/ CLJP Latents
SouthFastAsia, DM w/ CLIP Latents
Americas, LDM 1.5 (Open)
Europe, LDM 1.5 (Open)
WestAsia. LDM 1.5 (Open)
LDM 2.1 (Closed)
Europe, LDM 2.1 (Closed)
WestAsia, LDM 2.1 (Closed)
Americas, DM w/ CLIP Latents
Europe, DM w/ CLP Latents
WestAsia, DM w/ CLIP Latents
Figure 8: Region-Object Indicator for {object} in {country) prompt.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Africa
DM w/ CLIP Latents
DM w/ CLIP Latents
LDM 1.5 (Open)
LDM 2.1 (Closed)
Africa
Europe
LDM (Open)
Europe
West Asia
LDM 2.1 (Closed)
West
(a) GeoDE.
(b) {object}.
(c) {object] in {region}.
Figure 9: Random examples of bags. Sub-figure (a) Real images from Africa, Europe and West Asia. Sub-figure (b) Generations obtained with {object} setup using three models. Sub-figure (c) Generations obtained with [object] in {region} for three models and three regions.
Africa
DM w/ CLIP Latents
DM w/ CLIP Latents
LDM (Open)
LDM 2.1 (Closed)
Africa
Europe
LDM 1.5 (Open)
Europe
West Asia
LDM 2.1 (Closed)
West Asia
(a) GeoDE.
(b) {object}.
(c) {object} in {region].
Figure 10: Random examples of dogs. Sub-figure (a) Real images from Africa, Europe and West Asia. Sub-figure (b) Generations obtained with [object) setup using three models. Sub-figure (c) Generations obtained with {object} {region) for three models and three regions.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Third, objects such cooking pot, hand soop, and stove have notable disparities in precision and coverage across regions for at least one model. For example, cooking pol shows stronger precision in East Asia than Africa and the Americas, despite the Americas and Africa showing higher coverage for LDM 1.5 (Open). For hand soop, Africa achioves among the top precision and coverage for DM w/ CLIP Latents and Southcast Asia the least. The disparities observed for the concept slove may be explained by the mismatch between representation of the concept in the generations versus different regions of the GeoDE datasct. Figure [ shows that LDM 1.5 (Open)'s generations of stoves focus primarily on frontal perspectives of freestanding stoves/ fireplaces. Yet, these generations do not include portable stoves nor rectangular gas cooktops found in Geo West Asia, aligning with the lower coverage values. DM w/ CLIP Latents's generations of the concept stove depict close-ups of single burners with fire lit up in many cases, which do not appear frequently in GeoDE. LDM 2.1 (Closed) does not always display stoves when prompted for this object.
Disparities when prompting with region. When considering the [object) in {region} prompt results in Figure 7 some objects like medicine continue to show disparities similar to those for the {object) prompt while other trends change. This is unsurprising given that the generative systems now leverage specific region information. For example, objects such bag and toothbrush now present accentuated performance disparities more notably in precision, but also in coverage for DM w/ CLIP Latents and LDM 1.5 (Open). These new trends are captured in Figure 9 which shows that these two models most often generate bags outdoors surrounded by rather stereotyped backgrounds. For example, backgrounds in generations of Europe include pavements made of setts or cobblestones, whereas those in Africa more often depict rural areas. For stove, coverage becomes low for most models and regions, and precision displays disparities of up to 70% within a single model. Figure 4 shows that DM w/ CLIP Latents and LDM 1.5 (Open) generate interior freestanding stoves/ /freellcisseeess often with lit. fires, for Europe and predominantly outdoor rudimentary stoves for West Asia and Africa. Few of these representations are common in GeoDE.
As seen for the Region Indicator, including region information when prompting harms the performance across regions and models for many object classes. This includes dog, which previously achieved among the highest precision and coverage scores. Figure [0] highlights that dogs tend to appear outdoors when region information is included, and, similar to bog generations, the image backgrounds appear rather stereotyped.
We find that the prompt specificity affects the LDM 2.1 (Closed) model particularly, rosulting in lower precision and coverage than other systems for many objects including medicine, bug, cooking pol, or loothbrush. Figures 4 and g show that LDM 2.1 (Closed) appears to disregard the object to be generated and produces images of outdoor scenes that likely represent the model's perception of each region. We investigate this more with the Object Consistency Indicator in the following section.
Disparities when prompting with country. When focusing on {object} in (country)>prompter generations in Figure g we continue to observe disparities in performance across regions for all models. Intuitively, given the broad regions considered, providing country-specifi information may guide the model towards different generations than when providing region information. Prompting with country appears to improve the performance for some object classes, but hurt it for others. For example, adding country-speeific information appears to benefit the generations of car in many regions. Figure 5 shows that DM w/ CLIP Latents and LDM 1.5 (Open) now exhibit more diversity of car types in Africa including both SUVs and sedans and car surroundings still dominated by rural areas but now with more infrastructure. Car generations of countries in West Asia show increased diversity in car types as well as an increased presonce of cars in urban areas. Similarly, in the case of slove, prompting LDM 1.5 (Open) with countries within West. Asia leads to generations which include freestanding electric and gas stoves. These observations are echoed in Figure 8 which highlights an increase in precision and coverage for those models, objects, and regions.
When considering all 27 object classes, we observe that making the prompt more specific (from regions to countries) helps improve precision and coverage on average for Africa, West Asia and East Asia for most models (see Figure 16 in Appendix).
Observations consistent across regions. For (object), LDM 1.5 (Open) achieves the overall highest object-region performance, with only few objects below 50% performance threshold for both precision
14

---

Published in Transactions on Machine Learning Research (12/2023)
Europe
Southeest
Americes Europe
Southesst
025
0.22
0.2
022
0.20
0.20
024
0.4
0.24
032
022
022
620
LCW (Cowe))
(:ppm)
100000
020
LOW 21 (Cowec)
LDM 21 (00m)))
OM DMWIC Letees
LOW (Kisee)
1 10pmn))
WI Litie
LOW S1Q00)
LOM 51
2 100m
LOW 2. (GBse))
LOW 1 (0000)
LOW2, 10M21100000 10gen)
10 2.1 (CTA)))
LON Sup2
L0M2.1 (00mn)
DM CI Leter
920
10M 23 (Cose)))
LOME S0M22110000 (0000))
LEW (clsee))
LOK 9lts
LOM2, 100m))
ra WPG Letort
Low 21 (clope))
LOM 9002
LDW 2.1 L108ch)
DK W/ Liab
Lnw 2. (Gisee)
QUY
LOW 2 Li0pen)
Lww (clsse))
GLDS
(00m)
GLD-
LOW 51000)
21
LOW L3
WM 2.
DM wicLe
(a) {o.}
(b) {object) in {region}
(c) {object] in {country)
Figure 11: Geode dataset prompts, bar plots representing average of per class lowest 10% Clipscores.
and coverage. Although for some objects such as beg, DM w/ CLIP Latents shows better precision-coverage results, the higher precision values often come at the expense of lower coverage see e.g. condle or dog. For {object} in {region), LDM 2.1 (Closed) exhibits low performance (below 30% in both precision and coverage) for large number of object-region combinations. The poor performance of LDM 2.1 (Closed) may highlight the overall lack of consistency in the system's 's generations perceived upon visual inspection. Yet, observe that for {object} in {country), the low performance previously observed for some objects slightly recovers. Overall, LDM 1.5 (Open) appears to benefit more from making the prompts more specific than its paid-API counterpart LDM 2.1 (Closed).
5.3 Object Consistency Indicator
LDM 2.1 (Closed):
LDM 1.5 (Open):
DM w/ CLIP Latents:
(a) bag
(b) car
(c) hat
(d) stove
Figure 12: Examples of images with CLIPscore in 10 percentile for different objects, regions and models. Images obtained by prompting with [object]. The CLIPscore is computed between the generated image and [object).
We now report the consistency indicator using the prompts coming from the GcoDE dataset. Figure 11 displays bar plots representing the average across objects of the 10-percentile of CcorScore for three model prompting setups. In the Appendix we report the full distríbution of CL.Pscores for each model/region set
15

---

Published in Transactions on Machine Learning Research (12/2023)
Observations when prompting without geographic information. Figure 1la depicts CLIPScores for models prompted with {object). We observe that LDM 2.1 (Closed) reaches the lowest consistency indicator suggesting that this model has some object-image consistency issues. This is confirmed via visual inspection: LDM 2.1 (Closed) frequently generates images that do not show the object named in the prompt (sce first row in Figure 12 In addition, DM w/ CLIP Latents and LDM 1.5 (Open) achieve the highest. consistency scores, with the latter reaching top scores. Our visual inspection of low-scored images for the two models suggests that both exhibit good object-generation consistency. However, the scenes generated by both models are very different, with LDM 1.5 (Open) generating objects in diverse backgrounds and DM w/ CLIP Latents producing object closeups with very simple, uniform backgrounds (see bottom two rows in Figure 12
depicts consistency scores for models prompted with [object] in (region]. First, we note that moving from {object} to {object} in {region) prompts reduces the consistency scores for all models. Images displayed in Figure 13 reveal that models rely less on the [object] part of the prompt and start to generate region specific content not necessarily related the object, such as with bag for LDM 1.5 (Open) and LDM 2.1 (Closed) and car in Africa for DM w/ CLIP Latents. Next, we observe that DM w/ CLIP Latents achieves the highest consistency indicator. This is supported by visual inspection of Figure our consistency indicator, DM w/ CLIP Latents performs best for Europe and worst for Africa. LDM 2.1 (Closed) has the worst indicator score, achieving worst results for Europe and the best for the Americas. Indeed, the generations displayed in Figure 13] show low object consistency for this model for all regions. LDM 1.5 (Open) is sccond to the best performing model: for some objects and regions this model produces inconsistent generations, such as for bag in Africa and hat in Europe.
Disparities when prompting with region. Figure 11b
13 where all its generations are consistent with the object. According to
Disparities when prompting with country. Models prompted with {object} in {country) are depicted in Figure 11 In general, we observe that the consistency indicator is in line with our observations for the {object} in {region} setup some model/region combinations benefit from more fine-grained geographic information while others are affected negatively. For example, we observe the largest positive change for generations of Africa by LDM 2.1 (Open) and the largest negative change for generations of the Americas by LDM 2.1 (Closed). Visual analysis of the images with low CLIPscore (Appendix Figure 22 reveals country specific content of the generations where objects such as bag frequently depict country's 's flag.
6 Limitations and Mitigation Directions
In this section we discuss limitations of our work and suggest directions for mitigations of observed disparities. We hope that this will inform future work focused on responsible text-to-image generative systems.
6.1
Limitations
While the indicators we introduce are useful in understanding and benchmarking performance disparities between goographic regions in text-to-image generative models, they have some limitations.
Restrictions of the reference dataset First, the precision- and coverage- based indicators are dependent on the particular reference dataset of real images used for evaluation and are susceptible to their representation of groups. Thus, they may lead to skewed conclusions about quality and diversity the real reference datasot is unreliable. In our work, we aim to address this by selecting two datasets collected independently across geographic regions and observe that findings are sensible when accounting for the representation of groups in the real dataset. For example, DollarStreet was collected with special focus on rarely-photographed or low-income households and tends to show more variation across region than GeoDE. In Appendix A.5 we demonstrrte that, unsurprisingly, results of our Indicators show that the biased generations are more representative of the images of Africa in DollarStreet than in GeoDE.
Dataset size also an important consideration when using these metrics. A larger reference dataset can make it to challenging to cover all modes of variation while a smaller reference dataset may make it hard to estimate
16

---

Published in Transactions on Machine Learning Research (12/2023)
LDM 2.1 (Closed):
LDM 1.5 (Open):
DM w/ CLIP Latents:
LDM 2.1 (Closed):
LDM 1.5 (Open):
DM w/ CLIP Latents:
(a) bag
(b) car
(c) hat
(d) stove
Figure 13: Examples of images with CLiPscore in 10th percentile for different objects, regions and models.
Images obtained by prompting with [object) in (region]. Rows 1-3 depict Africa and rows 4-6 depict Europe. The CL.PPscore is computed between the generated image and {object].
the manifold of real images. We also urge careful considcration of the portrayal groups, potential biases in the data collection process, and reliability of labels in any reference dataset utilized for these indicators.
Potential biases of feature extractors In addition, all three indicators are dependent on the feature extractor used. While the use of Inception V3 well-ostablished when assessing performance of image generative models (Heusel et al.] 2017aa] Casanova et al.) 2021 Ramesh et al. 2022 Rombach et al.] 2021). future work may investigate the effect of different extractors observed disparities. Prior work has shown that the InceptionV33 model can be susceptible to "fringe features" appearing in ImageNet (Kynkäänniemi ct al. 2023) and have bias towards texture over shape (Geirhos et al.) 2022). In addition, the lack of geographically diverse images in ImageNet (Shankar et al.) 2017) may mean the InceptionV3 model is better suited for identifying similarities of Western Western-centric centric representations. More modern models trained on larger scale data, such as DINOv2 (Oquab et al. (023). may be stronger candidates as feature extractors (Stein)
ct al. 2023 even as their relative recency mcans that their biases are less explored.
For the object consistency indicator aim to reduce possible impact of group-level biases in CLIPscore by using only the {object} term when measuring similarity. However, the indicator would be impacted if CLIP finds certain variations of given object (such as large stove top accompanied by an oven) more representative the concept than others (like a camp stove or a portable electric stove top). A future study analyzing how various feature extractors relate to human preference quality, diversity, and consistency across geographics would provide additional guidance regarding the operationalization of these metrics.
Not able to replace qualitative evaluations While these mcasurements allow the identification and benchmarking of trends in performance across regions without relying on costly and time-consuming human evaluations, they should not replace thorough qualitative analyses that allow understanding of why and
17

---

Published in Transactions on Machine Learning Research (12/2023)
how observed disparities occur. Furthermore, these assessments not sufficient for understanding more fine-grained patterns such as imbalances within regions, e.g. by income-level, patterns of co-occurrences between multiple object classes in a single image.
Possible extensibility to other demographic groups We hope that future work investigates the application of this method to different types of demographic groups with the use of reference dataset containing demographic information like gender, age, and skin-tone and target classes.
6.2 Mitigation Directions
Our analysis highlights the need for future work focused on developing methods for reducing gaps in quality, diversity, and consistency across geographic groups. For the models trained with publicly accessible data, it would be helpful to understand if similarly stereotypical and homogeneous images are present in the training data. For example, perhaps generated images reflect training data in which image-text pairs containing region information feature colorful and vibrant geographies common in travel websites or stock photos. In addition, mitigation efforts may focus on the role of the text-encoder in perpetuating geographic biases, as suggested by previous work (Gustafson et al. 2023 Richards et al. 2023). Finally, the variations in prompting may also be used to improve generation performance between groups. For example, markedness in language (Blodgett et al. p021 of geographic information may affect model performance. Markedness refers to the fact that people mention certain kinds of information only when it is unusual, relevant, or remarkable. For instance, people from places with yellow bananas mention the color of banana more often when it is green (Sedivy) 2003). It is possible that our prompts mark geographical information that speakers generally wouldn't, i.e. the default assumption would probably be that an object is wherever you are and thus gcography may not be worth mentioning. In addition, prompting with languages beyond English may increase representation for regions where other languages are common, especially as multi-lingun generation capabilities become more prevalent. We discuss preliminary invostigations in several of these areas in Appendix A.6
7 Conclusion
In this work, we introduce three indicators used for evaluating disparities in realism, diversity, and consistency of generated images across geographic regions. These indicators complement existing time- and cost-intensive qualitative evaluation methods by allowing for the automatic benchmarking of models grounded in real world, representative datasets. They provide understanding of disparitics in generations at increasingly granular levels of evaluation, aggregated first across objects for each region, then split across objects and regions. Furthermore, they allow the disentangling of cases when models struggle to generate images that are consistent to a given prompt from those in which generations lack realism and divorsity while also accounting for variations in density of representations between regions.
We use these indicators to evaluate images of objects generated by widely used state-of-the-art text-to-image models across regions, with prompts of increasing geographic specificity. Through the Region Indicator, we find that existing models often have less realism and diversity of generations of objects when prompting with Africa and West Asia than Europe. This is reflected in qualitative analyses, where, e.g. we see the prompt car in Africa yields homogenous, boxy SUVs in consistently desert-like backgrounds. With the Object-Region Indicator, we see that region-level disparities occur more for some objects than others. For example, we find that generations of stove tend to have higher realism for Europe than Africa and West Asia. In some cases, disparities are due to differences in reprosentation of the object in the real world reference datasct, but more often they are due to stereotypes embedded in lext-to-image systems. With the Object Consistency Indicator, find that prompting with geographic information can come at cost to object consistency. Specifically, models like LDM 2.1 (Closed) struggle to generate images of a given object when prompts include geographic information. Of notable concern, we see from all three indicators that progress in image quality has come at the cost of real-world geographic representation. Altogether, these analyses demonstrate the indicators' efficacy in measuring limitations in the representation diversity of generated images.
We hope that this work allows for a more thorough understanding of geographic disparities in existing text- to-image generative models and encourages frequent benchmarking when iterating on model development.
18

---

Published in Transactions on Machine Learning Research (12/2023)
References
Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai-Wei Chang. How well can text-to-image generative models understand ethical natural language interventions? In EMNLP (Short), 2022.
Abhipsa Basu, R Venkatesh Babu, and Danish Pruthi. Inspecting the geographical representativeness of imagos from text-to-image models. arviv preprint 2Xi:22305.10800 2023.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale, 2022.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. CoRR, abs/21 201000190, 2021. URL
01963
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereolyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Lènguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004 1015, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/vv //1l/v1vvv// 201//2211aaa Lacl- long.81. URL https://aclaattiooggg www.lloggg.rgg///// 2021
acl-long.80) 81
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance- conditioned gan. Advances in Neural Information Processing Systems, 34:27517-27529 2021.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-
image generative transformers. CoRR, abs/2202.04000 2022. URL
Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recognition work for everyone? CoRR, abs/1906.022000 19066.0269,, 2019a. URL
Terrance DeVries, Adriana Romero, Luis Pineda, Graham W. Taylor, and Michal Drozdzal. On the evaluation of conditional gans. CoRR, abs/1907.08175, 2019b. URL http://aaxivv.rrg/bb///91....
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning, 2022.
Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusson: Instructing text-to-image generation models on fairness, 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Mako-a-scene: Sceno-basod text-to-image generation with human priors. In Computer Vision-ECCY 2022: 17th European Conference, Tel Avir, Isruel, Oclober 23-27, 27, 2022, Proceedings, Parl XV, pp. 19-106. Springer, 2022.
Gapminder. Dollar street dataset. 24.
2021. Accessod: 2023-01-
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trainee cnns are biased towards texture; increasing shape bias improves accuracy and robustness, 2022.
Priya Goyal, Adriana Romero Soriano, Caner Hazirbas, Levent Sagun, and Nicolas Usunier. Fairness indicators for systematic assossments of visual feature extractors. In 2022 ACM Conference on Fairness, Accountabillly, and Transparency, FAceT '22, pp. 70 88, New York, NY, USA, 2022. Association for Computing Machinery.
Laura Gustafson, Megan Richards, Melissa Hall, Caner Hazirbas, Diane Bouchacourt, and Mark Ibrahim. Pinpointing why object recognition performance degrades across income levels and geographies, 2023.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Melissa Hall, Bobbic Chern, Laura Gustafson, Denisse Ventura, Harshad Kulkarni, Candace Ross, and Nicolas Usunier. Towards reliable assessments of demographic disparities in multi-label image classifiers, 2023a.
Melissa Hall, Laura Gustafson, Aaron Adcock, Ishan Misra, and Candace Ross. Vision-language models performing zero-shot tasks exhibit gender-based based disparities, 2023b. URL
11100
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, and Cristian Canton Ferrer. Towards measuring fairness in ai: the casual conversations dataset. IEEE Transactions on Biometrics, Behavior, and Identity Science, 4(3):324-332, 2021.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXic preprint arXiv:2210.0751 2022.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Port III, volume 11207 of Lecture Notes in Computer Science, pp. 793-811. Springer, 2018
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08711 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in nearra information processing systems, 30, 2017a.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günner Klambauer, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500, 2017b. URL
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering, arXiv preprint arXio:2303. 1 11897, 2023.
Tuomas Kynkäännicmi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Improved precision and recall metric for assessing generative models, 2019.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance, 2023.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models, 2023.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. CoRR, abs/1810.03993,, 2018. URL http::/aaxiv.orrr///// org/ aaw.aa1../1111111111111111
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. CoRR, abs/2002.099977, 2020. URL https://arxiv.. wwwp:orgggbb.o.... 2002.09797
Alex Nichol, Prafulla. Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishknnn Bob McGrew, llya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. CoRR, (bbs/2112.1001 2021. URL bettps://arxis org/ herg/abs/21111 10741
Maxime Oquab, Témothée Darcel, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabricl Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Bilal Porgali, Vitor Albiero, Jordan Ryda, Cristian Canton Ferrer, and Caner Hazirbas. The casual conversations v2 dataset. arXiv preprinl or orXiv22003048888 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition. In Xivi preprint, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Lext-conditional image generation with clip latents, 2022.
Suman V. Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative modcls. CoRR, 0tss/1955118881 2019. URL http://aazzi..rr//bbb// 1abb1995..1888
Megan Richards, Polina Kirichenko, Diane Bouchacourt, and Mark Ibrahim. Does progress on object recognition benchmarks improve real-world generalization?, 2023.
William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In Thirly-sinth Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synth with latent diffusion models. CoRR, 222.2222..1055,, 2021. URL bttps://arxiv.orgg abs/2112.10755
Candace Ross, Boris Katz, and Andrei Barbu. Mcasuring social biases in grounded vision and language embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Associalion for Computational Linguistics: Human Longuage Technologies, pp. 998- 198-1008, Online, June 2021, Association for Computational Linguistics.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
Mchdi S. M. Sajjadi, Olivier Bachem, Mario Lučié, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Candice Schumann, Susanna Riceo, Utsav Prabhu, Vittorio Ferrari, and Caroline Rebecca Pantofaru. A step toward more inclusive people annotations for fairness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (ATES), 2021.
Julie Sedivy. Pragmatic versus form-based accounts of referential contrast: Evidence for effects of informativity expectations. Journal of psycholinguistic research, 32:3 23, 2003.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world, 2017.
Konstantin Shmelkov, Cordelia Schmid, and Kartock Alahari. How good is my gan? CoRR, abs/1807. 099499 2018. URL. httt:://axxiv.org/bbbaa.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateck Kosaraju, Dhruv Mahajan, Ross B. Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. CoRR, abs/2201.08371,, 2022. URL https;
Eric Michael Smith, Mclissa Hall, Mclanie Kambadur, Elconora Presani, and Adina Williams. "i'm sorry to hear that": Finding new biases in language models with holistic descriptor dataset, 2022.
Tejas Srinivasan and Yonatan Bisk. Worst both worlds: Biases compound pro-trained vision-and and-language models. Xiv preprint arXiv:2104. 1en:2144.06666 2021.
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency,
pp. 701-713, 2021.
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, J. Eric T. Taylor, and Gabriel Geniza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment diffusion models, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CoRR, abs/ 1512..067, 2015. URL http:///rxiv... org/abg/ /bs/
1512.00567
Robert Wolfe and Aylin Caliskan. Markedness in visual semantic ai. In 2022 ACM Conference on Fairness, Accountability, and Trunsparency, FAccT '22, pp. 1269 1279, New York, NY, USA, 2022a. Association for URL
Computing Machinery. ISBN 9781450393522. doi: 10.1145/3532 111383331111133333
1145/3531146 3533183
Robert Wolfe and Aylin Caliskan. American == white in multimodal language-anddimage ai. In Proceedings of the 2022 AAAI/ACM Conference AI, Ethics, and Society, AIES '22, pp. 800 812, New York, NY, USA, 2022b. Association for Computing Machinery. ISBN 9781450392471. doi:
URL https://doii org/10. 11t...1111/1114..501 3534136
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision ond pattern recognition, pp. 586-595, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-We Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2979 2989, Copenhagen, Denmark, September ortpo/oogg..rg//7113322
2017. Association for Computational Linguistics. URL
22

---

Published in Transactions on Machine Learning Research (12/2023)
A
Appendix
A.1 Reference Datasets
Figure 14 shows example images from each of the reference datasets (GeoDE SollarStreet Rojas et al. (0022) used for our Indicators.
Ramaswary et al. (0233 and
While DollarStrect was formally recognized as computer vision dataset in 2022 RRojas et al., 2022 it has
been used extensively for evaluations of geographic biases since 2019 (Derries et al. 2019a Singh
2022
Goyal et al. 2022 Hall et al. 2023a
Gustafson et al. 2023 Richards et al. 2023
A.2 Additional Details: Region Indicator
15 shows the precision and coverage measurements for models evaluated with DollarStreet the reference dataset. As discussed in the main text, prompting with regions shows an improvement for Africa. We believe this is due to Dollar Street's s focus low-income,, remote locations, as well as its lack of enforcement of minimum size for pictured objects.
Figure
A.3 Additional Details: Region-Object Object Indicator
Figures 16 the average object-region indicator for {object) in {region} and {object} in {country). As shown in the figure, most models benefit from leveraging country information for West Asia and Africa. These bonefits are especially pronounced for LDM 1.5 (Open) and LDM 2.1 (Open), whereas GLIDE only benefits modestly for Africa and East Asia.
A.4 Additional Details: Object Consistency Indicator
To accompany our discussion of CLIPScore measurements in the main text, we show the box plots representing the CLIPScore across regions and objects Figure 17
A.5 Additional Example Images
In this section, we show additional example images.
Figure 18 shows generated images corresponding to the prompt [object} for DM w/ CLIP Latents and LDM 1.5 (Open), as well as the GeoDE and DollarStreet reference images. Figure 19 shows images generated with the prompt {object] in {region}, and Figure 20 shows images generated with {object) in {country).
To supplement the CLIPScore examples in the main lext, include examples of images with CLIPScore in the 10th percentile for East Asia using the prompt strueture {object} in {region} in Figure 21 In Figure 22 we show example images in the 10th percentile across Africa, East Asia, and Europe using the prompt structure {object} in {country).
A.6 Preliminary Mitigation Analyses
We perform initial investigations into whether prompting with non- non-English languages prevalent in under- performing regions may help reducing disparities. Tn this work we observed that LDM 1.5 (Open) and LDM 2.1 (Open) have stercotypical representations for the prompt "car in Africa" and "stove in Africa." In a small experiment, we prompt using Arabic, Hausa, and Zulu, which correspond to the most spoken non- EnglEnglish languages in the three African countries best represented in the GeoDE dataset (Egypt, Nigeria, and South Africa, respectively). We find that images generated with non-English languages tend to struggle with prompt consistency and continue to show stereolyped representations as shown in Figure 23 Despite these initial observations, we believe this would be an interesting area to explore in more depth, especially as lingullingual generation is increasingly supported.
We also perform preliminary analysis of possible biases in LAION-2B-an which is used in the training of LDM 1.5 (Open) and LDM 2.1 (Open). We perform search of approximately 14.8 million image-text.
23

---

Published in Transactions on Machine Learning Research (12/2023)
(a) Africa, GeoDE
b East Asia, GeoDE
(c) Europe, GeoDE
(d) Africa, DollarStreet
(e) Asia, DollarStreet
(f) Europe, DollarStreet
Figure 14: Examples of images from GeoDE (a, b and c) and DollarStreet (d, e, and f) datasets. For each datasct display three regions and four random images of the following objects: toothbrush, car, stove and cleaning equipment.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Model and prompt:
LDM [Open): [object)
LDM 2.1 (Open): (object) in (region)
GLIDE: [ob in (country)
LDM
[Open): (object) (region)
LDM 2.1 (Open): (object) (country)
LDM 2.1 (Closed): (abject)
IDMM
[Open): (object) in (country)
GUIDE: (abject)
LDM 2.1 [Closed]: (abject)
(region) (country)
LDM 2 1 (Open): [object)
GUDE: (object) in (region)
LDM 2.1 (Closed): (object)
Africa
Americas
Asia
Europe
60
60
60
60
55
55
55
55
Precision
Precision
Precision
30
30
30
Coverage -->
Coverage -->
Coverage --
Coverage -->
Figure 15: Precision and coverage measurements evaluated with the DollarStreet dataset.
(Closed)
1.5 (Open)
60
60
60
50
50
50
Precision -
Precision
Pracisin
40
40
40
30
30
30
20
2.1 (Open)
60
80
Prompt region
{object) in {region
Africa
{object) in rregion Amencas
{object) in (region; EastAsia
{object) {region), Euroge
Precision
Precision
[object] in [region] SouthEastAsia
40
40
{objec:] in [region), WestAsia
[country). Africa
{object) in [country] J. Americas
{object) in {country EastAsia
{objec: (country Europe
{object) (country),
(countr) SouthEastAsia
{object) in [country). WestAsia
Coverage
Coverage
Figure 16: Average Region-Object Indicator for {object} in {region} and {object) in {country] prompt.
25

---

Published in Transactions on Machine Learning Research (12/2023)
0/10
Saype
Fest
646
dbice
The
Suagee
Sass AAL
ANNLAND Southeed
133
130
833
022
B32
un (0
COP 24 (pen
Lar 11000
De WWw.h.u CuPuueee
LDV cant)
320e
LOP 13 13
LDM Con
UP 1a
U20 20 12e0
(e Pirke
2006
eesh
Loe 13 1eee
2006
10M 2 Coen
LOV 24 Open
130 15500
DRI Dew
(00 1.
IDM 10011 1
U2
104
UIM
(a) }o.}
(b) {object) in fregion}
{object? {country)
Figure 17: Average object CLIPScore in for images generated with GeoDE object prompts across various prompt structures, regions, and models. The box covers the range between Q1 and Q3 quartile values of the data. The whiskers extend up to 1.5 IQR (IQR Q3 Q1) from the edges of the box. Outliers are plotted separate dots. Box plots are sorted by avgerage score, indicated by the triangle.
(b) front door
(c) stove
(d) loothbrush
(c) cleaning equipment
(I) plate of food
Figure 18: Examples of generated images when prompting with {object}. Columns in each object-square correspond to GcoDE, DollarStrect, DM w/ CLIP Latents, and LDM 1.5 (Open).
26

---

Published in Transactions on Machine Learning Research (12/2023)
(a) Africa, car
(b) East Asia, car
Europe, car
(d) Africa, front door
East Asia, front door
(f) Europe, front door
(H) Africa, slove
(h) East Asia, stove
Europe, slove
(i) Africa, toothbrush
(k) East Asia, toothbrush
Europe, toothbrush
Figure 19: Examples of generated images when prompting with {object) in {region). Columns in each object-square correspond to GeoDE, DollarStreet, DM w/ CLIP Latents, and LDM 1.5 (Open).
27

---

Published in Transactions on Machine Learning Research (12/2023)
Africa, car
(b) East Asia, car
Europe, car
(d) Africa, front door
East Asia, front door
(f) Europe, front door
(x) Africa, stove
(h) East Asia, stove
Europe, stove
(i) Africa, toothbrush
(k) East Asia, toothbrush
Europe, toothbrush
Figure 20: Examples of generated images when prompting with {object} in {country). Columns in each object-square correspond to GeoDE, DollarStreet, DM w/ CLIP Latents, and LDM 1.5 (Open).
28

---

Published in Transactions on Machine Learning Research (12/2023)
LDM 2.1 (Closed):
LDM 1.5 (Open):
DM w/ CLIP Latents:
(a) bag
(b) car
(c) hat
(d) stove
Figure 21: Examples of images with CLIPscore in 10th percentile for different objects and models for East Asia. Images obtained by prompting with {object} in (region]. The CLIPscore is computed between the generated image and {object).
LDM 1.5 (Open):
DM w/ CLIP Latents:
LDM 1.5 (Open):
DM w/ CLIP Latents:
LDM 1.5 (Open):
DM w/ CLIP Latents:
(a) bag
(b) car
(c) hat
(d) stove
Figure 22: Examples of images with clip score in 10 percentile for different objects, regions and models.
Images obtained by prompting with {object} in {country). Rows 1-2 depict. Africa, rows 3-4 depict East. Asia, and rows 5-6 depict Europe. The CLIP score computed between the generated image and [object).
29

---

Published in Transactions on Machine Learning Research (12/2023)
"car in Africa"
"stove in Africa"
LDM 1.5 (Open):
LDM 1.5 (Open):
LDM 2.1 (Open):
LDM (Open):
ADDA AFRRA
IA CICESC
English
Arabic
Hausa
Zulu
English
Arabic
Hausa
Zulu
Figure 23: We prompt using Arabic, Hausa, and Zulu, which correspond to the most spoken non- non-English languages in the three African countries best represented in the GcoDE dataset (Egypt, Nigeria, and South Africa, respectively).
pairs identify images whose captions contain either "car" or "bag" in conjunction with either "Europe" "Africa". While fewer than 50 captions contain these terms, their associated images do provide some initial insights. For example, it seems that colorful and vibrant geographies images matching our keywords, similar to the generated images with the respective prompts. We also see that types of bags in LAION between the two regions differ, where images pertaining to Africa have more totes and images pertaining to Europe have more luxur-llooiigg bags. In addition, there are fewer cars and bags depicted in the images of Africa that contain captions with those objects than in images pertaining Europe. We include examples in Figure 24. We recommend more thorough analyses of the role that the training data plays in the disparities observed in our paper important of future study.
frequently conveyed in the
30

---

Published in Transactions on Machine Learning Research (12/2023)
LDM 1.5 (Open) prompt: "Yer [region).
LDM (Open) prompo "oar in fregion!"
LAION-2B 26-ee-2------- (sample) Saption contains "tai" anx) "Iregoon"
Captions (clocksise from top-loff) (1) "Comanon ion of Adhsa car Magnit 10 3
TRUCES
"Farmmeutt Animatints Nanaer Trucko' Puatted
Spring
2015
Morate Truck Outrerr Ptayows Commenc
Test For South Afica Car Magaddie Raily Racing fot Download Free Game For Piass infitable Souncs House Steoollogal
Afric
image
Mercedes Aroce Gustom Cotruction on Sinwe Form
Uxbrecom Trucks Stock Photos
"Sout Afice Car Rentall: sightses the beaches of South
Captions (clockoise
"Sumpe Simu MOD Unlimitd Money andreio"
(2) "val walgon Launch india Bound New T Crosa In Euroe
"WISS CAMPING EUROPE 2010"
Europe
BENTALLS WORLDWIDE
LDM 1.5 (Open) promot: Tbag in (region1"
LDM 2.1 (Open)
LAION-2B-en (sample)
prompt: "080 (renion!"
caption contains "bag" and "Iregion""
shopping
(2) "Wili Zeten Socaeliing in Affics Bag Tiea"
(3) "Tnendy Dag Shopping theg Amoon Style Tate Bag
Camos
Africa
ecc
Affica Map Fashion
Captions (from left):
(1) "E.ropo and the United States burst leather fur fox fur handiseg
ceg waler rk cresss section
rippr bag wumpe
(2) "Woricn Bucket Handbog Europe And The Ur iodd Stanes New Fusition tilld Ciosal Turn paramen Shoulde
Europe
Messsrger Messanger
women
handbags leather bag Europe paskage"
Figure 24: We perform a search of approximately 14.8 million image-text pairs to identify images whose captions contain either "car" or "bag" in conjunction with either "Europe" or " "Africa".
31


Reviews:
[{"review_id": "DR6XIZ8pcK", "reviewer": "Reviewer_XYe4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nThe work is timely, with much attention given to image-synthesis models recently, yet little formal academic study of their biases.\n\nThe analysis is detailed and comprehensive. The different metrics are motivated, justified and described clearly; they capture various factors of interest. Relevant and diverse models are chosen for analysis. There is plenty of discussion and analysis to highlight interesting aspects of the results.\n\nThe paper is very clearly written and pleasant to read.\n\n### Weaknesses:\n\nUse of a single feature extractor (InceptionV3) in defining the metrics, without a proper analysis of that feature extractor's own bias, confounds almost all the quantitative results. Although the authors mention this in the conclusions, it is not sufficiently addressed. At very least, it should be possible to add a discussion of what possible effects biases in the feature-extractor (e.g. lack of African cars in its training data) would have on the different metrics.\n\nThere is no discussion of the impact of language, e.g. whether asking for an African car (or just a car) in an African language changes the pattern of results.\n\nThere is no discussion of how the relevant object classes are chosen for analysis -- e.g. are they those that exhibit most/least bias?\n\nThere is no attempt to measure training dataset bias -- e.g. for the LDMs trained on LAION, the data is public, and it should be possible to assess whether the model is more/less/equally biased as the data itself. This would increase the insight available from the paper, since currently it is unclear if these models are simply propagating training-set bias, or introducing bias of their own -- which presumably is important if we want to fix such biases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to measure geographic biases in generative models of images. It proposes a varity of metrics to characterise such biases, and uses these to perform a detailed empirical analysis of five text-to-image diffusion models, leveraging two existing geographically-unbiased datasets as references. It is found that overall, these models exhibit significant bias, with e.g. loss of realism and diversity when models are prompted to generate images depicting non-European regions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- not critical: unless there is some good reason, refer to models by the names people actually use (Stable Diffusion) etc. -- as-is it just seems to unnecessarily make life harder for the reader\n\n- not critical: rearrange fig.3 (and other similar ones), since currently layout has models=rows at left but models=cols at top, which is rather confusing\n\nSee also the points above under 'weaknesses'.", "broader_impact_concerns": "The paper does not raise any concerns in the traditional sense; however its purpose is to draw attention to such concerns arising with existing models, hence it should have a positive impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iKIH50Jbu4", "reviewer": "Reviewer_2Aoq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ The topic considered is extremely interesting and as far as I know not explored extensively in the literature before.\n\n+ The authors have been extremely detailed in detailing the experimental protocol and the choice made to compare models against each other. Replicating the findings should be straightforward. \n\n+ I liked the effort of the authors to develop metrics that do not require human judgment to be able to scale the evaluation to many data and many models. \n\n## Weaknesses\n\n### Major\n\n1. One point missing from the paper that I would have liked to see discussed more in details is what are the biases present in the training data used by these models and how much the final differences in the model performance depends from the training data used and how much by the architecture used (both in terms of number of parameters and objectives used to train them). This should be possible to assess at least for open source models trained on public data. \nMost of these models are trained on web crawled image-caption pairs. By the nature of the way the data were collected I would expect geographic locations to appear in the caption only when elements of  the picture clearly refer to that country. Therefore in the training data geographic locations might appear only together with their “stereotypical” appearance and more powerful models might be more able to pick up on these nuances in the training data. I found that without discussing the training data perspective the analysis is missing the main source of the problem.\n\n2. No solution or mitigation for the problem is proposed. This is not a weakness of the paper but it limits the contributions also in light of the not complete analysis (according to me) highlighted in weakness 1. In particular the paper highlights the existence of the problem and quantitatively measures it but it does not propose any mitigation strategy.\n\n\n### Minor\n\n3. The proposed metrics makes sense but have two flaws highlighted also by the authors: the dependance on a real reference dataset (that should be very large to guarantee stable metrics) and the dependance on a visual scoring model (Inception v3) that might have its own geographical bias. It would have been interesting to conduct a small user study to verify if the ranking of models emerging from the automatic metrics corresponds to the preferences expressed by human raters. This would have validated the results independently from the possible two flaws highlighted above and could have been limited to one or few prompt/models pairs.\n\n4. I found the layout of the figures often confusing. For example in Fig. 3 in the first column “text-to-image models” corresponds to row, while for column 3-4-5 “models” corresponds to columns. I would suggest using fewer images at higher resolution and a consistent layout. For example in this specific case you could use “models” as a row, adding a row for the real images “GeoDE” and  add “no location/africa/europe/west Asia” as columns. The same comment can be applied to several images throughout the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to study disparities in performance of text-to-image generative models when prompted to generate objects belonging to particular geographic locations. \nIn particular the authors with their experiments highlight that:\n\n* Performance of a model prompted to generate a certain object class in a specific geographic location degrades significantly in terms of precision and coverage/diversity compared to the same model prompted to generate using just the object class.\n\n* Among geographic locations some suffer way more from performance degradation and stereotypical representation than others. In particular “Africa” and “west/east Asia” suffer the most.\n\n* Among the tested models newer and more powerful models seem to suffer more from this kind of disparities in performance depending on the geographic area considered. \n\nThe paper contributions are the analysis of the problem and the definition of the experimental methodology to quantify relevant metrics against a reference dataset of real images of objects in certain geographical locations. Datasets and metrics were already available in the literature, the contribution of the authors is the specific way of combining them together and the analysis of the results. Previous work studying this subject focused mostly on human evaluation that besides being costly and not scalable is in itself exposed to bias.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please have a look at the weaknesses listed above and in particular either consider extending the analysis to the training data (weakness 1) or propose/explore some mitigation strategies (weakness 2). Finally I would suggest improving the layout of the figures as detailed in weakness d.", "broader_impact_concerns": "No additional concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aQbBnzJ7Ne", "reviewer": "Reviewer_5gg4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenths**\n\nThis is one of the first systematic quantitative studies of geographic biases in latest text-to-image generation models.\nDespite the impressive visual quality of these T2I models, they are also notoriously known to be heavily biased towards certain modes.\nA quantitative analysis of such biases based on geographics is pressing as these models become part of our everyday life, and this one arrives timely.\n\nThe studies are carefully conducted, including curating the datasets, designing the metrics and evaluating all models.\nThe findings are interesting and reveal severe geographic biases in all tested models:\n- precision and coverage of the generated images are significantly worse in certain regions than others\n- prompting with specific regions results in deteriorated performance, in terms of precision, coverage and consistency with prompt\n- newer models trained with more data tend to perform worse in terms of geographic mitigating biases\n\n\n\n**Weaknesses**\n\n- The biggest limitation, as the paper also points out, is that the proposed metrics rely heavily on pre-trained feature extractors and well-balanced reference datasets, neither of which has proven sufficiently reliable. However, this is an extremely difficult problem, and these results are still extremely interesting and valuable at this point.\n- In the Region-Object Indicator and Object Consistency Indicator, how was the particular list of objects selected for the study? They seem a bit arbitrary to me. What is the rationale behind this selection?\n- The different indicator names are a bit confusing, especially compounded with various metrics. I wonder if there is a better way of structuring them. Fig. 1 also needs a better caption.\n- It would be great if the authors can also briefly point to potential directions of addressing these biases for future research in the field of image generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies potential geographic biases in prevalent diffusion-based text-to-image generation models. In particular, it proposes three indicators, Region Indicator, Region-Object Indicator and Object Consistency Indicator, and assesses five text-to-image models based on two existing geographically-balanced reference datasets. Several key findings are summarized from the results, highlighting severe geographic biases in all tested models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the manuscript is worth acceptance, but adding some explanations addressing the weaknesses will improve its quality.\n\n**Minor typos**\n- Fig. 1: right subplot title: \"Object-Region\" -> \"Region\"", "broader_impact_concerns": "This paper is explicitly attempting to analyze biases and ethical issues in prevalent image generation models. It will have immense impact on the society.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BvR1EW5vVe", "reviewer": "Reviewer_uMRB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I found the manuscript to be well written and polished.  There are some interesting and timely findings (particularly about SD 1.5 vs 2.0).  And technically everything seems to be correct.  On the other hand, I would rate the technical novelty of this paper as rather incremental.  And the reference datasets used in this paper are not that well known (GeoDE, DollarStreet, both of which are relatively new and unestablished) so comparing against them as the reference itself raises some questions (more details below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a methodology for evaluating image generation models focusing on the biases that can arise across geographic regions/countries. The authors propose 3 metrics; the first two are computed with respect to a reference “groundtruth” set of real images, measuring precision, coverage and text-image similarity.  These metrics can be computed at the per-region level which allows the authors to compare performance disparity between two models on, e.g. Africa vs Europe.\n\nUsing these metrics, the paper studies several recent models including Dall-E 2, Stable Diffusion versions 1.5 and 2.0 and GLIDE.  Among other findings, the authors show that the more recent (closed/proprietary) SD 2.0 model, despite generating higher quality images, struggles to achieve good geographic coverage relative to the older SD 1.5 model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The proposed precision and coverage measures are somewhat related to the more commonly used FID metric.  I would encourage the authors to add discussion comparing against FID.\n* Precision and coverage also have some potential problems: For precision, if the reference dataset is too small, then there could easily be images generated that are realistic but not realistic by this measure. For coverage the tradeoff goes the other way --- if the reference dataset is too large, then it gets harder and harder to cover all the modes of variation --- would be good to add a discussion of this as well\n* The reference datasets used in this paper are not that well known so comparing against them as the reference itself raises some questions.  Given that any interesting dataset that you could use would have some perspective/bias, it’s worth discussing the potential biases within these reference datasets (and provide summary statistics).\n* (Optional but would be nice to have): Add Midjourney to the analysis  \n* Some nits:\n  * I would recommend referring to models by their common names, which would improve readability\n  * Section 4.4: the phrase “Next, we object-stratify the scores and compute the per class percentile 10 of CLIPscores” — this is hard to parse.", "broader_impact_concerns": "N/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended acceptance. This is an interesting and different paper, reviewers believe it is a timely study and also are convinced about the results.
Decision: Accept (poster)