SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Redact privacy leaking image regions and metadata. The European Data Protection Board emphasizes that anonymization of personal data must guard against re-identification risks such as singling out, linkability, and inference [76]. Re-identification remains possible even when nonconsensual subjects' faces are obfuscated, through other body parts or contextual information [242]. One solution is to redact all privacy-leaking regions related to nonconsensual subjects (including their entire bodies, clothing, and accessories) and text (excluding copyright owner information). However, anonymization approaches should be validated empirically, especially when using methods without formal privacy guarantees. Moreover, to mitigate algorithmic failures or biases, human annotators should be involved in creating region proposals, as well as verifying automatically generated proposals, for image regions with identifying or private information [367]. For nonconsensual individuals residing in certain jurisdictions (e.g., Illinois, California, Washington, Texas), automated region proposals requiring biometric identifiers should be avoided. Instead, human annotators should take the responsibility of generating these proposals.
Notwithstanding, to further protect privacy, dataset creators should take steps to ensure that image metadata does not reveal identifying information that data subjects did not consent to sharing. This may involve replacing exact geolocation data with a more general representation, such as city and country, and excluding user-contributed details from metatags containing personally identifiable information, except when this action would violate copyright. However, we do not advise blanket redaction of all metadata, as it contains valuable image capture information that can be useful for assessing model bias and robustness related to instrument factors.
5 Diversity
HCCV dataset creators widely acknowledge the significance of dataset diversity [13, 64, 78, 170, 171, 173, 196, 283, 361, 368], realism [110, 153, 164, 173, 196, 368], and difficulty [13, 16, 64, 73, 78, 91, 110, 173, 196, 198, 361, 368] to enhance fairness and robustness in real-world applications. Previous research has emphasized diversity across image subjects, environments, and instruments [43, 139, 222, 287], but there are many ethical complexities involved in specifying diversity criteria [14, 15]. This section examines taxonomy challenges and offers recommendations.
5.1 Ethical Considerations
Representational and historical biases. The Council of Europe have expressed concerns about the threat posed by AI systems to equality and non-discrimination principles [67]. Many dataset creators often prioritize protected attributes, i.e., gender, race, and age, as key factors of dataset diversity [287]. Nevertheless, most HCCV datasets exhibit historical and representational biases [35, 166, 172, 312, 366]. These biases can be pernicious, particularly when models learn and amplify them. For instance, image captioning models may rely on contextual cues related to activities like shopping [377]
and laundry [376] to generate gendered captions. Spurious correlations are detrimental, as they are not causally related and perpetuate harmful associations [112, 281]. In addition, prominent examples in HCCV research demonstrate disparate algorithmic performance based on race and skin color [42, 43, 54, 123, 142-144, 148, 250, 271, 299, 318, 334, 375]. Most recently, autonomous robots have displayed racist, sexist, and physiognomic stereotypes [158]. Furthermore, face detection models have shown lower accuracy when processing images of older individuals compared to younger individuals [369]. While not endorsing these applications, discrepancies have also been observed in facial emotion recognition services for children in both commercial and research systems [152, 363], as well as age estimation [58, 115, 200].
Despite concerns regarding privacy, liability, and public relations, the collection of special and sensitive category data is crucial for bias assessments [15]. GDPR guidance from the UK ICO confirms that sensitive attributes can be collected for fairness purposes [324]. However, obtaining this information presents challenges, such as historical mistrust in clinical research among African-Americans [92, 191] or the social stigma of being photographed that some women face [166]. Nonetheless, marginalized communities may require explicit explanations and assurances about data usage to address concerns related to service provision, security, allocation, and representation [359]. This is particularly important as remaining unseen does not protect against being mis-seen [359].
The digital divide and accessibility. Healthcare datasets often lack representation of minority populations, compromising the reliability of automated decisions [356]. The World Health Organization (WHO) emphasizes the need for data accuracy, completeness, and diversity, particularly regarding age, in order to address ageism in AI [355]. ML systems may prioritize younger populations for resource allocation, assuming they would benefit the most in terms of life expectancy [355]. The digital divide further exacerbates the underrepresentation of vulnerable groups, including older generations, low-income school-aged children, and children in East Asia and the South Pacific who lack access to digital technology [326, 354]. Insufficient access to digital technology hampers the representation of vulnerable persons in datasets [290], leading to outcome homogenization—i.e., the systematic failing of the same individuals or groups [39].
Confused taxonomies. Sex and gender are often used interchangeably, treating gender as a consequence of one's assigned sex at birth [95]. However, this approach erases intersex individuals who possess non-binary physiological sex characteristics [95]. Treating sex and gender as interchangeable perpetuates normative views by casting gender as binary, immutable, and solely based on biological sex [179]. This perspective disregards transgender and gender nonconforming individuals. Moreover, sex, like gender, is a social construct, as sexed bodies do not exist outside of their social context [45].
Similar to sex and gender, race and ethnicity are often used synonymously [332]. Nations employ diverse census questions to ascertain ethnic group composition, encompassing factors such as nationality, race, color, language, religion, customs, and tribe [328]. However, these categories and their definitions lack consistency over time and geography, often influenced by political agendas and socio-cultural shifts [286]. This variability makes it challenging to collect globally representative and meaningful data on ethnic groups. Consequently, several HCCV datasets have incorporated inconsistent and arbitrary racial categorization systems [7, 267, 343, 374]. For instance, the FairFace dataset [170] creators reference the US Census Bureau's racial categories without considering the social definition of race they represent [240]. The US Census Bureau explicitly states that their categories reflect a social definition rather than a biological, anthropological, or genetic one. Consequently, labeling the "physical race" of image subjects based on nonphysiological categories is contradictory. Furthermore, the FairFace creators do not disclose the demographics or cultural compatibility of their annotators.
Own-anchor bias. HCCV approaches for encoding age in datasets vary, using either integer labels [53, 102, 125, 226, 236, 264, 277, 278, 374] or group labels [84, 105, 193, 302]. Age groupings are often preferred when collecting unconstrained images from the web, as human annotators must infer subjects' ages, which is challenging [48]. This is evident in crowdsourced annotations, where  $40.2\%$  of individuals in the OpenImages MIAP dataset [290] could not be categorized into an age group. Factors unrelated to age, such as facial expression [106, 237, 345] and makeup [83, 237, 313], influence age perception. Furthermore, annotators have exhibited lower accuracy when labeling people outside of their own demographic group [8, 9, 113, 279, 303, 335, 339].
Post hoc rationalization of the use of physiological markers. Gender information about data subjects is obtained through inference [53, 170, 187, 198, 236, 264, 277, 278, 290, 343, 374, 374]
or self-identification [136, 190, 203, 204, 375]. Inference raises concerns as it assumes that gender can be determined solely from imagery without consent or consultation with the subject, which is noninclusive and harmful [87, 127, 179]. Even when combined with non-image-based information, inferred gender fails to account for the fluidity of identity, potentially mislabeling subjects at the time of image capture [277, 278]. Moreover, physical traits are just one of many dimensions, including posture, clothing, and vocal cues, used to infer not only gender but also race [100, 177, 179].
Erasure of nonstereotypical individuals. HCCV datasets frequently adopt a US-based racial schema [170, 190, 203, 204, 264, 343], which may oversimplify and essentialize groups [316]. This approach may not align with other more nuanced models, e.g., the continuum-based color system used in Brazil, which considers a wide range of physical characteristics. Nonconsensual image datasets rely on annotators to assign semantic categories, perpetuating stereotypes and disseminating them beyond their cultural context [180]. Notably, images without label consensus are often discarded [170, 267, 343], potentially excluding individuals who defy stereotypes, such as multi-ethnic individuals [276].
Phenotypic attributes. Protected attributes may not be the most appropriate criteria for evaluating HCCV models [43]. Social constructs like race and gender lack clear delineations for subgroup membership based on visible or invisible characteristics. These labels capture invisible aspects of identity that are not solely determined by visible appearance. Moreover, the phenotypic characteristics within and across subgroups exhibit significant variability [25, 48, 96, 138, 180, 347].
Environment and instrument. The image capture device and environmental conditions significantly influence model performance, and their impact should be considered [222]. Factors such as camera software, hardware, and environmental conditions affect HCCV model robustness in various settings [4, 140, 197, 221, 229, 353, 360, 364, 371]. Understanding performance differences is crucial from ethical and scientific perspectives. For example, sensitivity to illumination or white balance may be linked to sensitive attributes, e.g., skin tone [62, 184, 185, 378], while available instruments or environmental co-occurrences may correlate with demographic attributes [139, 295, 376].
Annotator positionality. Psychological research highlights the influence of annotators' sociocultural background on their visual perception [19, 107, 147, 263, 275, 291]. However, recent empirical studies have evidenced a lack of regard for the impact an annotator's social identity has on data [79, 111]. Only a handful of HCCV datasets provide annotator demographic details [12, 56, 287, 375].
Recruitment and compensation. Data collected without consent patently lacks compensation. Balancing between excessive and deficient payment is crucial to avoid coercion and exploitation [231, 268]. An additional concern is the employment of remote workers from disadvantaged regions [248], often with low wages and fast-paced work conditions [71, 135, 162, 206]. This can lead to arbitrary denial of payment based on opaque quality criteria [98] and prevents union formation [206], creating a sense of invisibility and uncertainty for workers [321].
5.2 Practical Recommendations
Obtain self-reported annotations. Practitioners are cautious about inferring labels about people to avoid biases [15]. Moreover, data access request rights, e.g., as offered by GDPR, California Consumer Privacy Act, and PIPL, may require data holders to disclose inferred information. To avoid stereotypical annotations and minimize harm from misclassification [275], labels should be collected directly from image subjects, who inherently possess contextual knowledge of their environment and awareness of their own attributes.
Provide open-ended response options. Closed-ended questions, such as those on census forms, may lead to incongruous responses and inadequate options for self-identification [156, 179, 274]. Open-ended questions provide more accurate answers but can be taxing, require extensive coding, and are harder to analyze [40, 109, 178, 298]. To balance this, closed-ended questions should be augmented with an open-ended response option, avoiding the term "other", which implies othering norms [285]. This gives subjects a voice [234, 296] and allows for future question design improvement.
Acknowledge the mutability and multiplicity of identity. Identity shift—the intentional self-transformation in mediated contexts [49]—is often overlooked. To address this, we propose collecting self-identified information on a per-image basis, acknowledging that identity is temporal and nonstatic. Specifically, for sensitive attributes, allowing the selection of multiple identity categories without limitations is preferable [304, 309]. This prevents oversimplification and marginalization. While we
acknowledge the potential burden of self-identification on fluid and dynamic identities, an image captures a single moment. Thus, evolving identity may not require metadata updates; however, we recommend providing subjects with mechanisms for updates when needed.
Collect age, pronouns, and ancestry. First, to capture accurate age information, dataset curators should collect the exact biological age in years from image subjects, corresponding to their age at the time of image capture. This approach offers flexibility, insofar as permitting the appropriate aggregation of the collected data. This is particularly important given the lack of consistent age groupings in the literature.
Second, dataset curators should consider opting to collect self-identified pronouns. This promotes mutual respect and common courtesy, reducing the likelihood of causing harm through misgendering [157]. Self-identified pronouns are particularly important for sexual and gender minority communities as they "convey and affirm gender identity" [232]. Significantly, pronoun use is increasingly prevalent in social media platforms [86, 165, 167], workplaces [55], and education settings [20, 212], fostering gender inclusivity [21]. However, subjects should always have the option of not disclosing this information.
Finally, to address issues with ethnic and racial classification systems [180, 286], dataset creators should consider collecting ancestry information instead. Ancestry is defined by historically shaped borders and has been shown to offer a more stable and less confusing concept [17]. The United Nations' M49 geoscheme can be used to operationalize ancestry [329], where subjects select regions that best describe their ancestry. To situate responses, subjects could be asked, e.g., "Where do your ancestors (e.g., great-grandparents) come from?" This avoids reliance on proxies, e.g., skin tone, that risk normalizing their inadequacies without reflecting their limitations [15].
Collect aggregate data for commonly ignored groups. Additional sensitive attributes should also be collected, such as disability and pregnancy status, when voluntarily disclosed by subjects. These attributes should be reported in aggregate data to reduce the safety concerns of subjects [309, 351]. Given that definitions of these attributes may be inconsistent and tied to culture, identity, and histories of oppression [37, 41], navigating tensions between benefits and risks is necessary. Despite potential reluctance, sourcing data from underrepresented communities contributes to dataset inclusivity [37, 168]. Regarding disability, the American Community Survey [330] covers categories related to hearing, vision, cognitive, ambulatory, self-care, and independent living difficulties.
Collect phenotypic and neutral performative features. Collecting phenotypic characteristics can serve as objective measures of diversity, i.e., attributes which, in evolutionary terms, contribute to individual-level recognition [57], e.g., skin color, eye color, hair type, hair color, height, and weight [19]. These attributes have enabled finer-grained analysis of model performance and biases [43, 75, 294, 318, 349, 372]. Additionally, considering a multiplicity of neutral performative features, e.g., facial hair, hairstyle, cosmetics, clothing, and accessories, is important to surface the perpetuation of social stereotypes and spurious relationships in trained models [6, 18, 166, 284, 340].
Record environment and instrument information. Data should capture variations in environmental conditions and imaging devices, including factors such as image capture time, season, weather, ambient lighting, scene, geography, camera position, distance, lens, sensor, stabilization, use of flash, and post-processing software. Instrument-related factors may be easily captured, by restricting data collection to images with exchangeable image file format (Exif) metadata. The remaining factors, e.g., season and weather can be self-reported or coarsely estimated utilizing information such as image capture time and location.
Recontextualize annotators as contributors. Dataset creators should document the identities of annotators and their contributions to the dataset [12, 79], rather than treating them as anonymous entities responsible for data labeling alone [52, 206]. While many datasets [78, 137, 196] neglect to report annotator demographics, assuming objectivity in annotation for visual categories is flawed [23, 169, 219]. Furthermore, using majority voting to reach the assumed ground truth, disregards minority opinions, treating them as noise [169]. Annotator characteristics, including pronouns, age, and ancestry, should be recorded and reported to quantify and address annotator perspectives and bias in datasets [12, 118]. Additionally, allowing annotators freedom in labeling helps to avoid replicating socially dominant viewpoints [219].
Fair treatment and compensation for contributors. In accordance with Australia's National Health and Medical Research Council [231] and the WHO [66], dataset contributors should not only be
guaranteed compensation above the minimum hourly wage of their country of residence [280], but also according to the complexity of tasks to be performed. However, alternative payment models, for example, based on the average hourly wage, may offer benefits in terms of promoting diversity by increasing the likelihood of higher socio-economic status contributors [251].
Besides payment, the implementation of direct communication channels and feedback mechanisms, such as anonymized feedback forms [246], can help to address issues faced by annotators while providing a level of protection from retribution. Furthermore, the creation of plain language guides can ease task completion and reduce quality control overheads. Ideally, recruitment and compensation processes should be well-documented and undergo ethics review, which can help to further reduce the number of "glaring ethical lapses" [293].
Supplementary to established ethical review protocols, we have provided proactive, domain-specific recommendations for curating HCCV evaluation datasets for fairness and robustness evaluations. However, encouraging change in ethical practice could encounter resistance or slow adoption due to established norms [218], inertia [33], diffusion of responsibility [151], and liability concerns [15]. To garner greater acceptance, platforms such as NeurIPS could adopt a model similar to the registered reports format, embraced by over 300 journals [51, 247]. This entails pre-acceptance of dataset proposals before curation, alleviating financial uncertainties associated with more ethical practices.
Nevertheless, seeking consent from all depicted individuals might give rise to logistical challenges. Resource requirements tied to the implementation and maintenance of consent management systems could emerge, potentially necessitating significant investment in technical infrastructure and dedicated personnel. Particularly for smaller organizations and academic research groups, these limitations could present considerable hurdles. A potential solution is forming data consortia [121, 166], which helps address operational challenges by pooling resources and knowledge.
Extending our recommendations to the curation of "democratizing" foundation model-sized training datasets [104, 119, 288, 289] poses an economic challenge. To put this into perspective, the GeoDE dataset of 62K crowdsourced object-centric images [262], without personally identifiable information, incurred a cost of $1.08 per image. While our recommendations may not seamlessly scale to the curation of fairness-aware, billion-sized image datasets, it is worth considering that "solutions which resist scale thinking are necessary to undo the social structures which lie at the heart of social inequality" [130]. Large-scale, nonconsensual datasets driven by scale thinking have included harmful and distressing content, including rape [35, 36], racist stereotypes [131], vulnerable persons [128], and derogatory taxonomies [34, 69, 129, 183]. Such content may further generate legal concerns [2]. We contend that these issues can be mitigated through the implementation of our recommendations.
Nonetheless, balancing resources between model development and data curation is value-laden, shaped by "social, political, and ethical values" [38]. While organizations readily invest significantly in model training [227, 336], compensation for data contributors often appears neglected [337, 338], disregarding that "most data represent or impact people" [380]. Remedial actions could be envisioned to bridge the gap between models developed with ethically curated data and those benefiting from expansive, nonconsensually crawled data. Reallocating research funds away from dominant data-hungry methods [38] would help to strike a balance between technological advancement and ethical imperatives.
However, the granularity and comprehensiveness of our diversity recommendations could be adapted beyond evaluation contexts, particularly when employing "fairness without demographics" [50, 134, 189, 210] training approaches, reducing financial costs. Nevertheless, the applicability of any proposed recommendation is intrinsically linked to the specific context [243]. Decisions should be guided by the social framework of a given application to ensure ethical and equitable data curation.
Just as the concepts of identity evolve, our recommendations must also evolve to ensure their ongoing relevance and sensitivity. Thus, we encourage dataset creators to tailor our recommendations to their context, fostering further discussions on responsible data curation.
This work was funded by Sony Research.