GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
Authors: Vikram V. Ramaswamy
Sing Yu Lin
Dora Zhao
Aaron Bryan Adcock
Laurens van der Maaten
Deepti Ghadiyaram
Olga Russakovsky
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Crowdsourcing
geodiversity

Abstract: Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce stereotypical biases, can contain personally identifiable information, and typically originates from Europe and North America. In this work, we rethink the dataset collection paradigm and introduce GeoDE, a geographically diverse dataset with 61,940 images from 40 classes and 6 world regions, and no personally identifiable information, collected by soliciting images from people across the world. We analyse GeoDE to understand differences in images collected in this manner compared to web-scraping. Despite the smaller size of this dataset, we demonstrate its use as both an evaluation and training dataset, allowing us to highlight shortcomings in current models, as well as demonstrate improved performance even when training on this small dataset. We release the full dataset and code at https://geodiverse-data-collection.cs.princeton.edu/

GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
Vikram V. Ramaswamy', Sing Yu Lin', Dora Zhao
Aaron B. Adcock
Laurens van der Maaten Deepti Ghadiyaram'
Olga Russakovsky'
Priicceton University Stanford University *Work done student Prinecton University
Meta Al
*Runway
Abstract
Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce stereotypical biases, can contain personally identifiable information, and typically originates from Europe and North America. this work, we rethink the dataset collection paradigm and introduce GeoDE, geographically diverse dataset with 01,940 images from 40 classes and 6 world regions, with personally identifiable information, collected by soliciting images from people around the world. We analyse GeoDE to understand differences in images collected in this manner compared web- reb-scraping. We demonstrate its as both evaluation and training dataset, allowing us to highlight and begin to mitigate the shortcomings in current models. despite GeoDE's 's relatively small size. We release the full dataset and code wtthtts::/
htttts://geediiveesss getp//egeovvveeeeeeee data- caata-coolections
princeton. edu/
Introduction
The creation of large-scale image datasets has enabled advances in the performance of computer vision models. Previously limited by internal manual collection efforts [13 15 12), in the past 15 years the size of these datasets has rapidly grown. This growth has been empowered by new data collection framework: scraping web images at scale. These images are either human- (e.g..
ImageNet 8 26)), use tags (c.g., CLIP-400M (24]) or use self-supervision (c.g., PASS 22).
However, these web-scraped datasets come with their downsides. One of these downsides is that these datasets can often contain pernicious social and cultural biases. For example, gender and racial biases can manifest through underrepresentation and/or through stereotypical depictions of certain demographic groups [21 40 34 3. There is also geographic bias: works of e.g., Shankar et al. [27] and de Vries et al. 7 show that web-scraped datasets consist of images mostly from North America and Western Europe.
The other common downsides are copyright, consent and compensation. Dataset creators frequently do not obtain full permission of the content creators and of the people featured in the content K While annotators are compensated, content creators and image subjects rarely are B.. Though there have been efforts to balance datasets [10, clean datasets [36], and protect privacy of depicted subjects by blurring [39, methods that rely on web-scraping cannot fully eliminate these issues [3] 18).
To tackle these issues, an exciting new dataset DollanStreet 25 was recently introduced (licence: CC). Instead of web-scraping, DollarStreet sources data from the Gapminder foundation. It comprises of images taken by volunteer and professional photographers in different countries to illustrate households with different income statuses. This results in 38,479 images from 63 different countries, tagged with 289 labels. DollarStreet overcomes issues of consent, and in many ways the first truly geographically diverse dataset (Tab. [.
While images used are sometimes under the most permissive Creative Commons license, it is unclear creators know the full impacts of their images being used in the training of large scale models.
37th Conference Neural Information Processing Systems (NeurIPS 2023) Track Datasets and Benchmarks.

---

GeoYFCC distribution 10
GeoDE distribution (ours)
Figure 1: We construct a geographically diverse dataset GeoDE that is approximately balanced across world regions. We visualize the images per region, and compare our distribution (right) to that of a previously created diverse dataset Geo [10 (left).
However, DollarStreet curates computer vision dataset from images that have previously been collected and released on the web. In contrast, we present an alternative geographically-divee diveese data collection process (more details in Sec. 3 which allows us to explicitly target a different object-centric image distribution.
Further, with the advent of multimodal foundation models 22 22
revising the idea of 23 32
manual collection of test datasets may be order. With these models, direct access to the training data is frequently impossible, although we know they were trained on trillions of web-scraped examples. Thus, any dataset comprising of previously available images may actually have been included in the model training, violating the core machine learning tenet of separate train/test splits.
We collected geographically diverse dataset of common objects which overcomes many issues described above. Concretely, we commisssion photos of different objects from people across the world from Appen (www. www.aappar aww.a.cppooooooo
Contributions.
global workforce. This naturally resolves consent concerns, similar to DollarStreet [25] and ensures that the data is (at least temporarily) unseen test set. We own the copyright to all the images in GeoDE, have explicit permission from creators to use these images for computer vision, verified that the images do not show identifiable people or other personally identifiable information (PII), and ensured compensation to the content creators.
The resulting Geographically Diverse Evaluation (GeoDE) datase contains 61,940 images roughly balanced across 40 object categories and 6 geographic regions. We find that the object recognition problem becomes surprisingly challenging since the GeoDE images represent the diverse appearance of common objects across six global regions: Africa, the Americas, East Asia, Europe, Southeast Asia, and West Asia. Similar to de Vries et al. [7]. we show that modern object recognition models perform poorly on recognizing objects from Africa, East Asia, and SE Asia. Augmenting current training datasets (like ImageNet [ 26]) with training images from GeoDE yields an improvement of 9% on DollarStrect (25) and 21 % on the test split of GeoDE.
Further, requesting images with specific content mitigated selection bias: web-scraped images are typically uploaded by creators with different incentives, e.g. to generate engagement with Dociting/unique content (28]. and disincentives for mundane everyday content. We show that the distribution of images in GeoDE is different to that in other datasets, when controlling for world region and object class.
Unfortunately, a key drawback of our method is the cost, which is partially the result of aiming for fair compensation to the content creators and curators. The method can also lead other biases (e.g., lack of economic diversity, since workers are required have a smart phone). However, we demonstrate that even small amounts of data collected in this way can be beneficial in remedying some of the concerns with large-scale web-scraped datasets. Our work challenges the current paradigm for dataset collection and illustrates the process of manually curating image datasets. In addition, we introduce the GeoDE dataset which offers a more geodiverse and ethically created alternative for object recognition. Data and code can be found at /tttps://geodiveet weweeedivvvee-dttaa---eae-
wttp:riinnnss...innneeeeeee pwwncetonn edu/
2 Related Work
There are three key research directions that inspired this work. The first is the call to increase geographic diversity in visual datasets 77. In response, there have been attempts to construct such datasets [0 summarized in Tab. [ However, these datasets are still geographically concentrated (in Europe for YCC [10] or India for Openlmages Extended []). The dataset most similar to
"GeoDE is owned and maintained by Princeton; Meta Al involved with the data collection.

---

Table 1: We compare approaches to dataset collection, along with the distribution and sizes of each. Although GcoDE is smaller than standard datasets, we ensure that images are sourced with permission, contain no identifiable people, and are balanced across both regions and object classes.
Dataset
Size; distribution
Collection process: annotation process
Geographic coverage
Personally Identifiahle Info (PII)
ImageNet
14.2M: mostly halanced across classes
Scraped images from the web based on the class label; crowd-sourced annotations
Mostly North
Contains people, subset with faces blurred [38]
26
America & Western Europe (2
PASS
1:4M; N/A (no labels)
Random images from Flickr; no annotations
Flickr, thus mostly North America
No people
Western Europe
OpenImages
9.1M: : long-tailed class distribution
Flickr images with CC-BY licenses; automatic labels with some human verification
Mostl North
Contains people
19
America Western
Europe 27
OpenImages
478K: aaigttiiled class distribution
Crowd- sourced gamified app to collect images; automatic labels and manual descriptions
More than 80% of images are from India
People are blurred
Extended
GeoYFCC
330K; long-tailed class distribution
Flickr images subsampled to be geodiverse; noisy tags
62 countries, hut concentrated in Europe (Fig.
Contains people
10
DollarStreet
38,479; mostly balanced across topics
Images by professional and volunteer photographers; manual labels including houschold income
63 countries Africa, America, Asia & Europe
Yes, with permission
25
GcoDE
61,940; halanced across classes and regions
Crowd-soureed collection using paid workers; manual annotation
Even coverage over six geographical regions (Tab. 3
No identifiable people and other PII
(ours)
ours is DollarStreet 25 also mentioned in the introduction. Both DollarStreet and our GeoDe were collected to improve geographic diversity of image datasets, are relatively small scale datasets (62k for GeoDE, 39k for DollarStreet), and are collected through crowd-sourcing, However, DollarStreet was repurposed as a computer vision dataset by curating the images through GapMinder, a non-pre non-profit organization that collected these images to showcase differences in how people live around the world. Thus, the images in DollarStreet were collected through more social science perspective: for example, images are collected to showcase different everyday actions such as "washing hands" as opposed to objects such as "hand soap". On the other hand, GeoDE is more focused on understanding how objects across the world are visually different.
Second, in using participants to generate visual content, we follow video datasets Charades (28). Epic Kitchens [6] and Ego4d [4]. We similarly leverage paid workers to provide examples that fall outside the common web-scraped distribution. However, we differ in that our key goal is to ensure geographic diversity. This poses unique challenges in recruitment and dataset scope (more in Sec. 3)
Finally, in our data collection efforts we take into account the extensive literature around selection bias in computer vision datasets 431 40 34 29 I1 37 3) and ensure that our dataset is collected responsibly, with attention privacy, consent, copyright and worker compensation 3
3 Collecting GeoDE
We describe our data collection process, including our selection of object classes and world regions.
Selecting the object classes. We focus on object classes that are likely to be visually distinct in different parts of the world. Selecting such objects is chicken-and-egg problem: without geographically diverse dataset at our disposal, it is unclear which objects are diverse. We adopt number of heuristics using existing datasets to find a plausible set. We use simple computer vision techniques (linear models and visual clustering, using features extracted from self-supervised PASS AASS-pretrained models (2) along with manual examination to identify a set of candidate tags from DollarStreet [25] and Geo FCCC [10] (e.g., "chili," "footstool," "stove"). To prune these tags, we remove those that are not objects (c.g., "aretic". "descent"), remove wild animals not found in all regions (e.g.. "gnu", "camel") and group variants of objects (e.g., "stupa", "temple", "church" and "mosque"). The final set of objects is in Tab. 2 and the full process is in the supp. mat.
3

---

Table 2: GeoDE consists of 40 object classes, loosely organized into groups.
Indoor common
Indoor rare
Outdoor common
Outdoor rare
hag. chair, dusthin, hairbrush/comb, hand soap, hat, light fixture. light switch. toothbrush, loohhpaste/ioolppowerr
candle, cleaning
backyard, car, fence, front door. house, road sign, streetligh/llnterm, tree. truck, waste container
bicycle, boat, bus, dog. flag. monument, religious building, stall, storefront, wheelbarrow
equipment, cooking pot, jug, lighter, medicine, plate of food, spices. stove, tyy
Table 3: GeoDE consists of images from six world regions. Within each, there are 3-4 countries contributing to most of the images. Participants from other countries within the region were accepted.
West Asia: Saudi Arabia, UAE, Turkey
Africa:
Egypt, Nigeria, South Africa Indonesia, Philippines, Thailand Italy, Romania, Spain, United Kingdom
East Asia: China, Japan, South Korea
SE Asia:
Americas: Argentina, Colombia, Mexico
Europe:
Selecting diverse geographic regions. We chose six regions: Africa, Central and South America ("Americas"", " East Asia, Europe, SouthEast (""EE" Asia and West Asia. Within each, we targeted 3-4 countries (Tab. 3). The regions were chosen due to the lack of available images from them in most public datasets 27 35]; the countries were chosen based on the presence of participants within Appen's 's database We obtain a roughly even distribution of images across each class and region pair.
Image collection and worker demographics. Workers were asked to upload images for given object class (Fig2). There were 4,500+ workers, representing a range of genders, ages and races (see supp. mat.). All images submitted were manually checked by Appen's 's quality assurance (QA) team.
4 Lessons learned from collecting GeoDE
A key part of this study was to understand if manually taking photographs is a viable alternative to web-scraping. In this section we detail the lessons learned in constructing the GeoDE dataset.
Getting sufficient images of all object classes. While some object classes expectedly proved more difficult than others (e.g.. "monument" or "flag" were simply hard for workers to find), others surprised us. For example, "stove" was originally underrepresented until the definition was clarified to "any cooking surface either electric, gas, induction." Workers' perception of their cooking appliance as "stove" varied, highlighting a vocabulary challenge unique geographically diverse data collection.
Multiple copies of images. Two most common types of error were incorrect images (i.e, not belonging to the class selected) and multiple copies of image. The QA team found that participants often submitted multiple copies of the same not to do so. Workers also sometimes submitted very similar objects (examples in supp. mat.), for example, three hats by the same worker with slightly varying colors). We filtered out such images.
instance from different angles despite instructions
Multiple objects per image. For some of the (especially larger and outdoor) object categories, it was difficult to ensure that other objects were not present, particularly "trees". For example, we found that images of "fences" often have "trees" present, and it was not always possible to discern between objects in the foreground and background (cxamples in supp. mat.). Thus, we requested that images that had significant portion of the image covered by trees be explicitly tagged. These additional annotations can be used to filter and remove such images and/or to analyze errors made by model.
Other. Beyond these, the rest of the data collection went smoothly. Following instructions, only 0.78% images contained identifiable information. Some images contain non-identifiable incidental people in the background (especially for larger object classes, like "monument"). All such images are tagged in GeoDE. We were also able to ensure that the number of images per region is roughly equal, although it was harder to obtain an even number of images per country within each region.
Cost. Collecting images in this way was expensive: each image cost roughly $0.87 for a total cost of $54,000, not including researcher time. This allowed us to compensate photographers as well as the management and QA teams for their labour.
WWiile we do not currently find this to be the case empirically, we acknowledge that the regions themselves quite broad, and certain objects might look different within the region.

---

General Instructions
In this task, you will submit to 3 photos of the same type of object (c.g., upload 3 photos of 3 different bags, please do not upload photos of the same bag from different angles).
1. Please make sure the location function is enahled for the camera.
2. The photo resolution should he at least 640 480.
3. All images should be new photos captured with Appen Mobilc.
4. Please make sure it's single object per image.
5. Please make sure it's well-lit environment and the object clearly visible in the photos.
6. Please make the object occupy least 25% of the image.
Objects captured are foregrounded and not occluded.
Objects should not blurred, e-g., motion blur.
No effects or filters added (cropping is acceptable).
10. Please try to avoid capturing people the images (it's OK people are blurry the background and far from the camera).
11. Please try to avoid capturing vehicle license plates images.
Figure 2: Image collection instructions given to workers.
Stove
House
Asla
Arica
East Asia
Southeast Asia
Americaa
ImageNet
Figure 3: Sample images of object classes in different regions within GeoDE (and ImageNet in the bottom row, for comparison). Note the variety of stoves and houses across geographic regions in GeoDE and also the fact that the stoves are more used (thus arguably more realistic) than in ImageNet. Product labels images in the figure have been blurred.
5 Comparing GeoDE to current datasets
We compare GeoDE with three datasets: the canonical ImageNet 8 the more geographically diverse (but still webscraped) GeoYFCC [10] and finally, the recently curated DollarStreet dataset [25]
Qualitative. Fig. we show subset of 60 GeoDE images of "stoves" and "houses" (more the
appendix). Compared to images from ImageNet, we see a larger variety of stoves: e.g., induction coils, single and two burner stoves. The stoves also appear more used than those in ImageNet. Similarly, for "house," we see a larger range in terms of materials used and size. In the Secs. and we examine the impact of this diversity on visual recognition models.
Statistics. GeoYFCC [10] (license: CC) is webscraped dataset subsampled from YFCC100M 30
to be geographically diverse; thus, by raw counts it much larger dataset compared to both DollarStrect and GeoDE, with over 1M images from 62 countries. However, looking at the regions
5

---

West Asia
Africa
East Asia
Southeast Asia
Americas
Europe
Plate of food
Store- front
Figure 4: We visualize the TSNE plots for several of object classes per region for GeoYFCC (light
blue) and GeoDE (dark blue). While the features do overlap slightly, on the whole, they are very different for dataset distributions, even within each (region, object) tuple. (See Fig. [ DollarStreet)
Africa
Americas
Europe
Africa
Americas
Europe
Plate of food
Stove
Figure 5: We visualize the TSNE plots for several of object classes per region for DollarStreet (cyan) and GeoDE (dark blue). We see that these features overlap significantly more than that of GeoYFCC, however, there are still objects with different distributions (e.g. "Stove" in Americas).
(Fig. reveals imbalances, with most countries from Europe. Moreover, this dataset does not have curated labels, just tags, and the distribution among tags is also long-tailed (the top 20 of 1,197 tags comprise 34% of the dataset). Comparatively, GeoDE is balanced across both regions and classes. DollarStreet [25] is much smaller dataset with only 38,479 photos, comparable 61,940 photos. DollarStreet has more classes but with fewer images per class compared to GeoDE: DollarStreet averages only 133 images per each of its 289 classes (with 382 images on average for its 40 most common classes), while GeoDE average 1,548 images per each of its 40 classes.
GeoDE with
Object appearance. Finally, we attempt to quantify the differences in the appearance of images collected through different methods, by comparing GeoDE with GeoYFCC [10] and DollarStreet (25). We extract features for each dataset using ResNet50 model [77 trained with self-supervised learning SWAV (5) on the PASS dataset (2] (license: CC-BY). We train linear classifiers to predict the dataset given an image: the classifier achieves accuracy of 96.3% when trying to distinguish between YeC and GeoDE, and an accuracy of 96.1 % when trying to distinguish between DollarStreet and GeoDE. However, this could be the result of having different distributions of regions (in the case of Geo YFCC) and different objects (for both). To understand how the dataset distributions different beyond just the class/region frequencies we obtain low-dimensional TSNE embeddings (33 with images that restricted to certain (region, object) pair (Figs. 4 and (5). We see a much more pronounced difference between GeoDE and YeOYFCC, likely due to effects of web-scraping.
6 GeoDE as an evaluation dataset
We now analyze the use of GeoDE as an evaluation dataset, by using it to evaluate two canonical models: the recent CLIP 24 and an ImageNet [ -trained model.
Implementation details. For the CLIP model, we use the weights provided for the ViT-B/32 model. We use text prompts for all 40 object categories as described in the zero-shot recognition setup of [24). To train a model on ImageNet [8], first match the classes of GeoDE and ImageNet. We find the relevant synsets for each GeoDE class in WordNet (20), and include all images of that synset. For two object categories ("backyard" and "toothpaste/toothoowdde
we do not find any matching categories, and we ignore these categories in the quantitative analysis. We split dataset into train (38,353 images), validation (12,794 images), and test (12,795 images) datasets. As in Sec. 5 we extract features using a ResNet50 model [17] trained with self-supervised learning PASS (2) and retrain the final layer.
filtered ImageNet [8]
SWAV
Results. Fig. (left) shows the accuracy across different regions on these two models. Both models perform the best on images from Europe and the worst on images from Africa (difference of more than 7% in both cases). Tab. 4 further breaks out the per-object accuracy for CLIP. While the average accuracy is 82.8%, classes like "dustbin"" (37.3%). "medicine" (54.1 (54.1)), "cleaning equipment" (59.0%) and "spices" (63.2%) perform poorly. Fig. shows example errors.

---

Table Per-class accuracy (in percentage; increasing order) of CLIP 24 on GeoDE. Objects like
"d "dustbin", "medicine" and "cleaning equipment" are poorly recognized, with accuracy as low as 37%.
medicine
clean. equip. spices house
waste comt.
toy backyyard candle
fonce streetlight
Fromt door
hand soup
plate food
light fixture wheelbarrow
storefront toothpaste
toothbruhh
flag religious bld bicyele road sign
cooking pot
brott bairbrush
car momument
lightswitch
dusdbin
stall lighter
stove
truck
tree
jug
bat
dog
bag
37 54 59
63
63 68 69 71
73
74 75 76 76
77
78
85
85 86 88 88
88
89
89 90 90 91 92 92 93 93 95 95 95 96 96 96 96 97 97 98
West Asin
Africa
East Asia
Southessd Asin
Americas
Europe
Teothbruch
Tecthbruc
Toothoruth
Haibrush
Harbbrash
Hand S0Up
Tosthansh
Dusibre
Handssopp
Teuthkrush
Stall
Hand Soop
MUSO
Belgeas
borrow
goas
Bachyrr
Belgous
Sturbort
Saggoos
Storblunt
Mediche
sandssap
Tooth seste/
Light future
Hand seap
Touthpoiter bood toodrponrdr
Taotbausoo
Hand
toothpowder
tooth
toot plastrr
Carda
contatee
Hand snop
Stove
Candl
contaier
todthpowder
Medichee
Eeg
Jkg
Cooling
Figure 6: Example errors that the CLIP 24 model makes on GeoDE images (the ground truth label on the left, CLIP prediction at the bottom). There are some systematic errors, e-g., classifying "house" as a "religious building", particularly on images from Asia. (product labels in figure are blurred).
In seeking to understand the accuracy variation across geographic regions, we compute the minimum and maximum per-region accuracy for each object (Fig. [ right). We also compute the confidence interval for the expected distribution of per-region accuracy for each object based on the object's s overall accuracy We find that 31 of 40 objects have minimum and/or maximum region accuracy that falls outside the corresponding 95% confidence interval, suggesting that these objects (including for example "house" and "dustbin"") exhibit significant geographic variation (at least with respect to the visual distribution learned by CLIP) We further note some differences on individual classes. "Fence" is 88% accurate for images from Europe, but only 60% and 59% for images from Africa and SE Asia respectively. Similarly, "stove" 95% accurate in the Americas but only 67% in East Asia. Visualizing classes using TSNE plots of the features (Fig. (7). we see that these objects are region specific, e.g, "religious buildings" from East and SE Asia uniquely include buildings like monasteries and temples; similarly, single- and two-burner "stoves" are primarily from Africa and SE Asia.
7 Impact of training with GeoDE
Finally, we attempt to answer how training with GeoDE can improve the performance of models trained on web-scraped data. Concretely, we investigate training a model on jointly GeoDE and subsets of ImageNet [8] and demonstrate the combination improves results across geographie regions.
7.1 Training a model with GeoDE
We would like to understand how training model with data from GeoDE affects object recognition. We train linear model using pre-trained features on a dataset comprised entirely of ImageNet images and a dataset comprised of both ImageNet and GeoDE images with the same number of images. The feature extractor a ResNet50 [6] model trained on PASS [ using SwAV 5
Implementation details. We split GeoDE into train (4,970 images per region), validation (between 1,657 and 2,188 images per region), and test (between 1,657 and 2,189 images per region). We use
"We draw 500 random partitions into 6 regions, and compute the resulting per-region accuracies. 5 'Considering the problem of multiple hypothesis testing, we can apply the Bonferonni correction for 40 objects, with a 0.05/40- = 0.001, i.e, 99.9% confidence intervals. Still, of 40 objects fall outside their intervals, confirming that GeoDE as a whole does exhibit statistically significant geographic variation.
We also try model trained from scratch and with finetuning, with similar conclusions (supp. mat.).
7

---

spices
stove
religious building
Figure 7: We show the TSNE plots of objects which have large regional disparities in accuracy in the CLIP trained model, with images embedded. We see differences based region, e.g., "religious buildings" contains a cluster of monasteries and temples, mostly from East and Southeast Asia.
100
Model
WAsia
Africa
EAsia
SEAsia
Americas
Europe
Accuracy
ImageNet
69.4
62.7
63.3
67.3
68.6
69.9
max. accuracy
overal accuracy
CLIP
84.0
78.7
79.9
81.9
84.4
85.8
40
min acouraty
Figure 8: (left) Accuracies (in %) on GeoDE of model trained on subset of ImageNet g (details in Sec. [6) and of CLIP [24). The models perform best on images from Europe, and worst on images from Africaa (right) We compute the maximum and minimum accuracy CLIP [24) for each object across the 6 regions in GeoDE(sorted by overall accuracy). 31 of 40 objects have at least one region whose accuracy falls outside the 95% CI, suggesting significant differences across regions.
the validation dataset to select training hyperparameters. The training set for our ImageNet only model is the same 38.353 image training set constructed in Sec. 6 To construct the training set of our ImageNet and all regions in GeoDE model, we add in the training sets for all 6 regions in GeoDE while removing the same number of images per class from ImageNet. This procedure gives a training set of 29,820 GeoDE images and 8,533 ImageNet [8] images. The models are trained using an SGD optimizer (Ir 0.1, momentum 0.9) for 500 epochs with cross entropy loss. Models were trained using a single GPU (RTX1080 or 2080) and took less than hour. Results are reported on the test set.
Results. We first report results on the GeoDE test set, and notice a significant improvement in accuracy across all regions, as a result of training with both GeoDE and ImageNet (Tab. 55. However, this improvement could come from the ImageNet + GeoDE dataset matching the domain of the GeoDE evaluation set and may not generalize to other datasets. Thus, we also test these models different dataset: the DollarStreet dataset 25 This dataset has been used before as an evaluation benchmark (7), to understand if current object recognition models can perform well on objects from a diverse set of regions. Tab. 5 lists the accuracy for the 4 different regions in DollarStreet, along with the per class accuracies for the object categories that overlap between GeoDE and DollarStreet. We see an increase in performance across most categories, suggesting that GeoDE is more geo-diverse than ImageNet and that there is an advantage using geo-diverse data in the training set.
7.2 Cost-vs-Diversice tradeoffs
The main drawback of GeoDE is the cost of this dataset: images collected in this way cost more than the standard pipeline of web-scraping and crowd-soureing annotations. Thus, it important to identify which classes and regions contribute most to the overall model. To investigate this, we start with the filtered ImageNet dataset described above, vary the amount of GeoDE data from particular region, and analyze the change in overall regional performance.
Implementation Details. We start with the 38,353 filtered ImageNet images and add a region of GeoDE's data back into the dataset and remove the same number of ImageNet images keep the number images and class balance the same. Other training details remain the same as Sec. 7.1
Evaluation. As we are evaluating the GeoDE test set, there are two possible sources of perfor- mance gain: (1) the model able to take advantage of the additional regional information from the GeoDE data; and (2) the GeoDE images were collected using the same collection method as the test set and from Sec. 5 we saw that there is difference in the feature space that can be attributed

---

Table 5: We compare the performance of a model trained on ImageNet g versus one that is trained on both ImageNet and GeoDE. We report results on the test set of GeoDE as well as the DollarStreet [25] images. We see improvement across all regions for both test datasets. We also report the per-class accuracies for the DollarStreet dataset and see improvement across most objects as well.
Tested on GeoDE
Tested on DollarStreet (25
plate of food
clean. equn.equip
cooking pot
light fixture
light switch
West Asia
East Asia
SE Asia Americas 90
America
hand soap
medicine
Africa ImageNet
Europe
Overall
Africa
Asia
Europe bicycle
chair
dustbin
house
stove
Overall
toy
69 63 63 67 69 70,67
45 64
58 75,92
86
19
49
76
49
88
36 77 80 84 89
50 60
+GeoDE 88 87
86
87
89
88
55
74
68
80
995
88
36
61
68
65
91
63
79
78
96
85
58
69
Classes with largest % inc. in AP
Africa (A
East Asa (V)
Europe
Afrrica E. Asia
waste cont. spices, dustbin, clean, equip. relig. big., spices, dusthin, waste cont. dustbin, hand soap. clean, equip., spices, dustbin, spices, clean. equip. medicine waste cont., spices. medicine, clean, equip.
6E
Asia
Americas
SE. Asia
Figure 9: (left) We measure the relative improvement in AP per object when GeoDE images from that region are included in training. Each vertical line represents object (sorted by region of max. improvement). Africa and East Asia see the largest improvement for the most classes. (right) We highlight the classes with the largest increases in the AP when adding in training images from GeoDE.
to the collection method itself (deliberately taking photos rather than web-scraping). In order to distinguish between these two sources, we measure the accuracy on both the region in the train set and accuracy on the images from Europd We also measure the increase in AP for specific objects to better understand which objects benefit most from GeoDE data.
Results. We find that the performance within each specific region and in Europe increase with the additional GeoDE data. The relative increase in performance for the specific region is larger than the increase for Europe, showing the value of data for each region, moreover, the improvements do not saturate, suggesting that more data could lead to further gains. Full results are presented in the supp. mat. We examine the classes that have the largest increase in average precision (AP) as the regional GeoDE images added to the dataset in Fig. 9 (left). We also present the object classes that the (right). In general, we see that specific objects such as "spices", "waste container" and "cleaning equipment" benefit most from regional GeoDE data.
most improvement in Fig. 9
8 Conclusion
We introduced a new dataset GeoDE which uses crowd-sourcing for image collection, a significant departure from the popular computer vision dataset collection paradigm of reb-scraping for image collection. Through this collection method, we ensured that this dataset does not contain personally identifiable information, we own the rights to the images, the image creators were compensated for their work, and were able to control for geographic diversity and object distribution in the dataset. We show that GeoDE a useful dataset for highlighting shortcomings in common models (e.g., CLIP) and can improve performance when added to the training dataset. Also, GeoDE shows that crowd-soureing is a viable image collection method for creating diverse and responsible datasets.
Acknowledgements. This material is based upon work partially supported by the National Science Foundation under Grant No. 2145198 198. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We also acknowledge support from Meta AI and the Princeton SEAS Howard B. Wentz, Jr. Junior Faculty Award to OR. We thank Dhruv Mahajan for his valuable insights during the project development phase. We also thank Jihoon Chung, Nicole Meister, Angelina Wang and the Princeton Visual AI Lab for their helpful comments and feedback during the writing process.
We use Europe as this region had the hest performance when using model trained on just ImageNet.

---

References
[1] Open images extended.
https: hww.eeacchh..ggeeeeoooooa/aaaaa. open- images-extended crowdsourcedd Accessed: 2022-10-16.
[2] Yuki M. Asano. Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi. Pass: An imagenet replacement for self-supervised pretraining without humans. NeurlPS Truck on Datasets and Benchmarks,
2021.
[3] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In WACV, 2021.
[4] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT, 2018.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper-
vised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
[6] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.
[7] Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In CVPR Workshops, 2019.
[8] J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li. and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
[9] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.088
2023.
[10]
Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. In CVPR, 2021.
[11] Chris Dulhanty and Alexander Wong. Auditing ImageNet:
Towards model-driven framework for annotating demographic attributes of large-scale image datasets, arXiv preprint arXiv: 1a0p:::00000344 2019.
[12] M. Everingham. L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object
classes (voc) challenge. IJCV, 2010.
[13] Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, 2004.
[14] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu. Xingyu Liu. Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray. Mengmeng Xu, Eric Zhong- cong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli. Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu. Takuma Yagi. Yunyi Zhu, Pablo Arbelaez. David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfci Yan, and Jitendra Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. arXiv, 2021. [15] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks.
In ECCV, 2016.
[18] Fun Seo Jo and Timnit Gebru. Lessons from archives: strategies for collecting socioculural data in machine learning. In FAceT, 2020.
[19] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images datasct v4; Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[20]
George A Miller. Wordnet: a lexical database for english. Comm. ACM, 1995.
[21] Safyy Umoja Noble. Algoritimss of Oppression: How Search Engines Reinforce Racism. NYU Press,
2018.
[22] OpenAL. DALL-E
[23] OpenAL GPT4, https///oponall.mm//reeeccaaa wttp:////eeoeeeeeeeeaaaaaaaa
g
[24]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
10

---

[25] William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter. Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world. In NeurlPS Datasets Benchmarks Track, 2022.
[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, Imagenet large scale visual recognition challenge. IJCV, 2015.
[27] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. NewrlPS Workshop on Machine Learning for the Developing World 2017.
[28] Gunnar A. Sigurdsson. Gül Varol. Wang, Ali Farhadi, Ivan Laptev, and Abhinav Kumar Gupta.
Hollywood in homes: Crowdsourcing data collection for activity understanding. ECCV, 2016.
[29] Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In ECCV, 2018.
[30] Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YfccI00m: The new data in multimedia research. 2016.
[31]
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. In arXiv preprint arXiv:2302. 1,3971, 2023.
[33]
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
[34]
Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Troha Shirai, Arvind Narayanan, and Olga Russakovsky. REVISE: A tool for measuring and mitigating bias in visual datasets.
IJCV, 2022.
[35] Angelina Wang, Arvind Narayanan, and Olga Russakovsky, REVISE: A tool for measuring and mitigating bias in visual datasets. In ECCV, 2020.
[36] Kaiyu Yang. Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering
and balancing the distribution of the people subtree in the imagenet hierarchy. FAT 2020.
[37] Kaiyu Yang. Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In FAT, 2020.
[38] Kaiyu Yang. Jacqueline Yau, Li Fei, Jia Deng, and Olga Russakovsky, A study of face obfuscation in
imagenet. CoRR, abs/2 1ot.//203.069111 2021.
[39] Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky, study of face obfuscation in imagenet. In ICML,, 2022.
[40] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, 2017.


Reviews:
[{"review_id": "uAthjM0evJ", "reviewer": "Reviewer_TgPt", "summary": "", "questions": "", "limitations": "I think this paper still needs to supplement the comparison of datasets.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors collected a geographically diverse dataset of common objects to provide the diversified model evaluation. \n\nGeoDE enables control of the image distribution to reduce dataset biases and the data is an unseen test set at least temporarily.\n\nGeoDE dataset benefits in remedying some of the concerns with large-scale web-scraped datasets.\n\nThe dataset is selected in six regions, which has more universal.\n\nWritten is clear.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "53QtYoIrNr", "reviewer": "Reviewer_1Lpg", "summary": "", "questions": "", "limitations": "It's a modest improvement over existing approaches. But it's also expensive and not really scalable.\n\nI did not review the images for general quality characteristics.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Great work on building a dataset. Careful attention to licenses. No PII. Good training results. Generally I like it.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JikKrxdkb6", "reviewer": "Reviewer_8Bhq", "summary": "", "questions": "", "limitations": "The author adequately addresses the negative societal impact.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper showcases several strengths, outlined as follows:\n* The paper presents a meticulous approach to collecting images from diverse regions, thoroughly addressing privacy, consent, and copyright considerations. \n* The paper introduced the GeoDE dataset, a novel labeled image dataset specifically tailored for geographically diverse object recognition. This dataset surpasses the DollarStreet dataset regarding the number of included images, enhancing the potential for more comprehensive and accurate model training and evaluation.\n* The paper uncovers regional variations in the appearance of objects, demonstrating how these differences directly impact the accuracy of current object recognition methods.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All the reviewers appreciated the proposed dataset of geographically diverse objects, which may reduce dataset biases in object recognition and remedy generalization issues with large-scale web-scraped datasets. The authors' rebuttal also addressed most of the main concerns of the reviewers and AC find no critical issue, thus recommending acceptance.
Decision: Accept (Poster)