Abstract

Introduction
Data collection and processing are crucial to the machine learning (ML) pipeline and are the source of many biases in AI systems, which have been shown to largely stem from a lack of diverse representation in training datasets (Buo- lamwini and Gebru 2018). Currently, most large-scale com- puter vision datasets are collected via webscraping and sub- sequent data cleaning. For example, the ImageNet database ((Deng et al. 2009); 42607 citations per Google Scholar, accessed Sept. 14, 2022) is comprised of images sourced from search engines like Google and Flickr, while the COCO dataset ((Lin et al. 2014); 26751 citations per Google Scholar, accessed Sept. 14, 2022) is comprised of images sourced entirely from Flickr. Thus, biases inherent to Flickr influence the performance of models for visual tasks as di- verse as object classification, pose estimation, instance seg- mentation, image captioning, and beyond. Some of these dataset biases have been explored in detail: for ImageNet and the Flickr-sourced Open Images dataset (Kuznetsova et al. 2018) it has been shown that data from India, China, and African and South-East Asian countries is vastly under- represented despite their large populations (DeVries et al. 2019); while for COCO, data has been shown to be heavily skewed towards lighter-skinned and male individuals (Zhao, Wang, and Russakovsky 2021). In particular, such biases im- pact the applicability of models in a global context. For in- stance, DeVries et al. (2019) manually sourced image data from 264 globally-distributed households and demonstrated how object recognition model performance drops when ap- plied in lower-income nations. Motivated by the popularity of datasets sourced using Flickr data, we here analyze 1.5M geotagged images in the Flickr database to deeply explore its representation of African people (see Figure 1). In this paper, we aim to highlight the limitations of web- scraping generic and human-centric image data from Africa for ML training purposes. We analyze image data for ev- ery African nation with direct comparisons to population- matched higher-GDP European nations and show that there is far less data available from Africa. We report the distribu- tion of African geotagged image data as a function of fine- grained, intra-national wealth estimates (Chi et al. 2022) and assess data with respect to license restrictions, population size, nominal GDP, Internet usage, and official languages. Additionally, we collect crowdsourced annotations to ex- plore image content, and provide evidence for an "other- ing" phenomenon as the majority of African geotagged im- ages we analyzed were taken by foreigners, while the oppo- site trend is shown for select European nations. Such results highlight the importance of considering geodiversity met- rics beyond ancestry/ethnicity of individuals within images and, moreover, how the mechanisms by which images are obtained can quantitatively and qualitatively affect how the image corpus represents the world ( e.g. imposing a "Western gaze"). Overall, we find that Flickr provides a very limited and skewed representation of African countries which likely contributes to many of the biases in models trained on pop- ular, large-scale image datasets.
Methodology
Data Collection: Flickr Africa    For each nation in Africa, we utilized Flickr queries to construct a dataset of images and associated metadata. Using the FlickrAPI, we scraped images and associated metadata from Flickr between dates 2004-02-10 and 2022-02-10 (18 years) by querying by coun- try name ( e.g. "Togo") and the country name + "people" ( e.g. “Togo people”) for all 54 countries in Africa. Images without valid geotags were excluded, and city/country information were determined using open-source reverse geocode (Pen- 1 Figure 1: A collection of maps displaying relative wealth index (RWI) and geolocation of Flickr Africa images via country name query. Tolerance distance from geotag to nearest RWI-labeled point are: ((a, b) dist: < 300km; (c, d) < 10km). (b) Nations are colored according to total number of geotagged images and the percentages (rounded to one decimal place) is the percentage of geotagged images. South Africa had the highest number of geotagged images and Sao Tome and Principe had the smallest number of geotagged images while Cape Verde had the highest percentage of geotagged images and Rwanda had the man 2020). All data is available at https://doi.org/10.5281/ zenodo.7133542 Population-matched European countries   The data col- lection process was repeated for four European nations. In the interest of comparing data availability and con- tent to higher-GDP European nations, we chose the fol- lowing countries as a function of similar population size ((Wikipedia.org 2022b,d,a)): Switzerland and Sierra Leone (GDP: 841.97k vs. 4.27k); Cyprus and Djibouti (GDP: 27.73k vs. 3.84k); Finland and Central African Repub- lic (CAF) (GDP: 297.62k vs. 2.65k); and Slovenia and Lesotho (GDP: 63.65k vs. 2.56k). For all 58 countries we collected data pertaining to percentage of internet users (Wikipedia org 2022c), nominal GDP (Wikipedia org 2022b), population size (Wikipedia.org 2022d,a) and official languages (Wikipedia.org 2022e). Relative  Wealth  Estimates Fine-grained relative wealth estimates were associated with each geotagged image. We utilize the relative wealth index (RWI) data collected from Low- and Middle-Income Countries (LMICs) by Face- book's Data for Good project ((Chi et al. 2022)); however, this dataset excludes: { Somalia, Seychelles, Sao Tome and Principe, Sudan, and South Sudan}. RWI scores are normal- ized by nation, so the data should only be utilized for intra- national wealth comparisons. Using k-nearest neighbour, we computed the nearest RWI-labeled geographic location of each image and excluded geotagged images beyond  300km to the nearest RWI-labelled location. Manual Content Annotation    Crowdsourced annotations were collected for six additional image features. We used Amazon Mechanical Turk (AMT) to collect annotations de- scribing image contents. Each task (HIT) had 21 images, with six binary questions per image which required the an- notator label the image according to: indoor vs. outdoor set- ting, public vs. private setting, nature vs. manmade setting, the presence of people, real vs. synthetic image type, and of- fensive vs. inoffensive content. We compensated workers at a rate of $15/hour and utilized gold standard images (1 per 20 images) to assess annotator performance. Limitations of Our Approach  We acknowledge three no- table limitations of our method. First, we recognize that ge- olocation data (longitude, latitude) is inherently unreliable. Values may be modified or removed by the user or otherwise not reflect the location of capture, and reverse geolocation methods are computationally expensive and often fail, par- ticularly with geographic locations close to region borders. This motivates our use of both geotags and country name tags for cross-validation of location, though this restricts us to fewer data samples overall. Additionally, some forms of geodiversity are difficult or impossible to determine from visual inspection alone, such as an individual's gender, eth- nicity, or religion. Finally, we were limited to obtaining data using only two queries, namely, by country name or coun- try name + "people". We anticipate future work exploring a wider variety of query terms, both in English and local lan- guages; here, no correlation was determined between domi- nant national languages and geotagged image availability. Ethical Considerations   We note that although the Flickr images analyzed here are all publicly viewable, we show that most have the Flickr default license of "All Rights Re- served". Thus, we have opted to provide image URLs in lieu of images for direct download to avoid duplication of pro- tected content, particularly in the event that a Flickr user chooses to remove or modify the permissions of an image. We acknowledge the weaknesses of this method in terms of consent, as public Flickr images are typically not taken by those in the images (as pointed out in (Birhane and Prabhu 2021)) and Flickr users may wish to avoid the utilization of their images for research purposes. Given that our objective is to critique large-scale image dataset curation strategies which do not respect image licenses ( e.g. the methodology for generating the COCO dataset), we deemed it justifiable to perform basic analyses on protected images and to build awareness regarding widespread license violations in stan- dard AI training pipelines.
Results and Discussion
Data Availability and Geographic Distribution There are very few geotagged images from Africa (see Figure 1b) with drastically fewer than European nations of comparable size (see Table 1); e.g. Switzerland has 18× as many geo- tagged images as Sierra Leone. The number of geotagged images positively increased with population size (corre- lation: 0.412 & 0.538), internet usage (0.474 & 0.385), and GDP (0.599 & 0.748) respective of country-name and country-name+people query; this was found to be statisti- cally significant. Official language was not found to have a meaningful correlation to the number of geotagged im- ages (p-value = 0.2021 & p-value = 0.846) after coding by: 1- (English is the only official language), 2- English is among the two official languages, 3- English among greater than three official languages, 4- English not among less than three official languages, and 5- English not among greater than three official languages. Tags and Licenses   Analysis of the tags found on the im- ages revealed that the most frequent tags were the name of the place where the image was taken, including "Africa", in addition to image contents. The least frequent tags were usually those in foreign languages or tags containing typos or consisting of multiple concatenated words. Such tags are difficult to interpret, let alone utilize for Flickr image queries in the interest of constructing diverse datasets. Additionally, the vast majority of images with query by “[country name]” and “[country name] people” respectively are licensed as "All Rights Reserved" (80.46%, 81.99%), indicating the Flickr default setting when images are up- loaded to the platform. Thus, those constructing datasets us- ing Flickr Africa data must be aware that most images are unavailable for open public use. This further limits ethical access to geographically diverse data. Geodiversity by RWI   To assess the impact of wealth on the availability of geotagged image data, we examine image counts by RWI values binned into 10 percentile groups ( G1- G10). For most nations, the majority of image data comes from the middle RWI regions ( G4, G5, G6 and G7 ) and the least from low RWI regions ( G 1 , G 2 and G 3 ). However, this is not always the case, e.g. Madagascar (Figure 1c) and Algeria (Figure 1d) from which data is commonly sourced from low-income areas (along main roads close to national parks) or high-income areas (in major cities), respectively. RWI has potential as a mechanism for constructing geo- diverse datasets in future work. Local vs. Non-Local Representation  For the African geotagged images we randomly selected for the crowdsourc- ing task, images were far more likely to be taken by for- eigners whereas the opposite trend was observed for high- GDP European nations, according to comparisons of geo- tags and user-reported location. For Sierra Leone, +169% of images were captured by foreigners, compared to Switzer- land (-31%). The same trend applies to Djibouti and Cyprus (+335% and -49%) and CAF and Finland (+272% and - 49%). Thus, images sourced from Africa may be more pre- disposed to bias resulting from an "othering" phenomenon, and less representative of African cultural viewpoints. Image  Content By utilizing crowdsourced annotations, we examine image content data from each matched African/European nation pair. The AMT results revealed that query-by-name images from both African countries and European countries were predominantly "real" (93.47% and 91.94%), “inoffensive” (88.51% and 89.41%), “outdoor” (77.89% and 79.28%), and “nature” (63.68% and 62.96%) images. Nations varied more in terms of total number of images containing people (Sierra Leone 68.25%, Lesotho 59.76%, CAF 52.13%, Djibouti 54.86%, Switzerland 68.45%, Finland 60.76%, Slovenia 71.81%, and Cyprus 67.28%). Given that image content was fairly similar across most attributes annotated, and there ex- ist far fewer geotagged images from Africa, we anticipate
insufficient African data availability for certain computer vi- sion tasks. For example, the lower prevalence of images cap- tured in "private" and "indoor" settings indicates e.g. house- hold object image data inaccessibility, which thereby im- pacts downstream object recognition system models.
Conclusion and Future Work
Geographical context shapes data, and data shapes the per- formance of models trained using such data. The key find- ings from our Flickr Africa data analysis have the potential to be highly impactful in both (1) exposing new limitations of current large-scale image data collection methodologies, and (2) exposing unique data challenges to Africa, includ- ing the lack of data crucial to specific domains ( e.g. a re- searcher cannot source sufficient, representative household object data if very few images are taken within indoor/pri- vate scenes). Notably, we reported on the extreme lack of data availability when compared to wealthy European na- tions; for instance when querying by country name, Switzer- land had 18x the geotagged image data as Sierra Leone, an African nation of similar population size (8.75M vs. 8.30M, respectively), while Sao Tome and Principle only had (776, 116) geotagged images in total (depending on query). More- over, data may be even less accessible according to use case, given that most of the Flickr Africa data has a restrictive use license, and certain image content attributes were found to appear less frequently ( e.g. private and indoor settings). Nationally, higher quantities of geotagged image data was found to positively correlate with population size, GDP, and Internet usage, but no significant correlation was discovered based on dominant national languages. Additionally, we in- terrogate where African image data comes from: generally from middle-wealth regions as measured intra-nationally by RWI, though this differs by nation; and with images mainly taken by foreigners, though the opposite trend is identified in wealthy European nations. Looking forward, we encourage new scholarship center- ing novel methods for sourcing geodiverse datasets and mea- suring new forms of geodiversity specific to Africa, such as analyses of tribal diversity as opposed to the more com- monly studied diversity by race/ethnicity. We openly provide our large-scale dataset to enable future researchers to utilize and augment Flickr Africa for model evaluations across a wide domain of computer vision tasks; likewise, more rig- orous bias identification methods ( e.g. (Wang et al. 2022)) may uncover still more limitations. Finally, we would be in- terested to explore the extent to which privacy and consent are respected in Africa.
Acknowledgments
We wish to thank Jerone Andrews and Dora Zhao in partic- ular for their expertise and assistance with our work pertain- ing to crowdsourcing. We also express gratitude to the whole SONY AI Ethics team for great discussions and informative feedback on this research.