SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jerone T. A. Andrews*
Sony AI, Tokyo
Dora Zhao†
Sony AI, New York
William Thong†
Sony AI, Zurich
Apostolos Modas†
Orestis Papakyriakopoulos†
Alice Xiang
Sony AI, Seattle
Human-centric computer vision (HCCV) data curation practices often neglect privacy and bias concerns, leading to dataset retractions and unfair models. HCCV datasets constructed through nonconsensual web scraping lack crucial metadata for comprehensive fairness and robustness evaluations. Current remedies are post hoc, lack persuasive justification for adoption, or fail to provide proper contextualization for appropriate application. Our research focuses on proactive, domain-specific recommendations, covering purpose, privacy and consent, and diversity, for curating HCCV evaluation datasets, addressing privacy and bias concerns. We adopt an ante hoc reflective perspective, drawing from current practices, guidelines, dataset withdrawals, and audits, to inform our considerations and recommendations.
Contemporary human-centric computer vision (HCCV) data curation practices, which prioritize dataset size and utility, have pushed issues related to privacy and bias to the periphery, resulting in dataset retractions and modifications [78, 126, 175, 216, 244, 320], as well as models that are unfair or rely on spurious correlations [22, 26, 112, 139, 146, 215, 272, 281]. HCCV datasets primarily rely on nonconsensual web scraping [99, 122, 124, 228, 260, 266, 310]. These datasets not only regard image subjects as free raw material [32], but also lack the ground-truth metadata required for fairness and robustness evaluations [91, 171, 196, 216]. This makes it challenging to obtain a comprehensive understanding of model blindspots and cascading harms [30, 85] across dimensions, such as data subjects, instruments, and environments, which are known to influence performance [222]. While, for example, image subject attributes can be inferred [7, 43, 170, 188, 198, 241, 267, 290, 343, 375], this is controversial for social constructs, notably race and gender [28, 132, 179, 180]. Inference introduces further biases [19, 107, 147, 263, 291] and can induce psychological harm when incorrect [47, 275].
Recent efforts in machine learning (ML) to address these issues often rely on post hoc reflective processes. Dataset documentation focuses on interrogating and describing datasets after data collection [5, 27, 44, 94, 108, 149, 243, 258, 273, 307]. Similarly, initiatives by NeurIPS and ICML ask authors to consider the ethical and societal implications of their research after completion [257]. Further, dataset audits [247, 292] and bias detection tools [29, 340] expose dataset management issues and representational biases without offering guidance on responsible data collection. Although there are existing proposals for artificial intelligence (AI) and data design guidelines [35, 79, 116, 160, 202, 247], as well as calls to adopt methodologies from more established fields [154, 159, 166], general-purpose guidelines lack domain specificity and task-oriented guidance [307]. For example, remedies may prioritize privacy and governance [35] but overlook data composition and image content. Other recommended practices lack persuasive justification for adoption [116, 160] or fail to provide proper
*Correspondence to jerone.andrews@sony.com.
$\dagger$  Equal contribution; authors are listed in random order.
contextualization for appropriate application [252, 323, 367]. For instance, the People + AI Guidebook [116] suggests creating dataset specifications without explaining the rationale, and privacy methodologies are advocated without cognizant of privacy and data protection laws [252, 323, 367]. These efforts, which hold significance in promoting responsible practices, would benefit from being supplemented by proactive, domain-specific recommendations aimed at tackling privacy and bias concerns starting from the inception of a dataset.
Our research directly addresses these critical concerns by examining purpose (Section 3), consent and privacy (Section 4), and diversity (Section 5). Compared to recent scholarship, we adopt an ante hoc reflective perspective, offering considerations and recommendations for curating HCCV datasets for fairness and robustness evaluations. Our work, therefore, resonates with the call for domain-specific resources to operationalize fairness [68, 150, 305]. We draw insights from current practices [42, 170, 375], guidelines [31, 222, 231], dataset withdrawals [78, 126, 216], and audits [35, 36, 247], to motivate each recommendation, focusing on HCCV evaluation datasets that present unique challenges (e.g., visual leakage of personally identifiable information) and opportunities (e.g., leveraging image metadata for analysis). To guide curators towards more ethical yet resource-intensive curation, we provide a checklist in Appendix A. $^{3}$  This translates our considerations and recommendations into pre-curation questions, functioning as a catalyst for discussion and reflection.
While several of our recommendations can also be applied retroactively such measures cannot undo incurred harm, e.g., resulting from inappropriate uses, privacy violations, and unfair representation [129]. It is important to make clear that our proposals are not intended for the evaluation of HCCV systems that detect, predict, or label sensitive or objectionable attributes such as race, gender, sexual orientation, or disability.
2 Development Process
HCCV should adhere to the most stringent ethical standards to address privacy and bias concerns. As stated in the NeurIPS Code of Ethics [1], it is essential to abide by established institutional research protocols, ensuring the safeguarding of human subjects. These protocols, initially designed for biomedical research, have, however, been met with confusion, resulting in inconsistencies when applied in the context of data-centric research [218]. For example, HCCV research often amasses millions of "public" images without obtaining informed consent or participation, disregarding serious privacy and ethical concerns [3, 35, 133, 245, 301]. This exemption from research ethics regulation is grounded in the limited definition of human-subjects research, which categorizes extant, publicly available data as minimal risk [218, 256]. Thus, numerous ethically-dubious HCCV datasets would not fall under Institutional Review Board (IRB) oversight [247]. What's more, the NeurIPS Code of Ethics only mandates following existing protocols when research involves "direct" interaction between human participants and researchers or technical systems. Even when research is subjected to supervision, IRBs are restricted from considering broader societal consequences beyond the immediate study context [217]. Compounding matters, CV-centric conferences are still to adopt ethics review practices [306].
These limitations are concerning, especially considering the potential for predictive privacy harms when seemingly non-identifiable data is combined [35, 70, 218] or when data is used for harmful downstream applications such as predicting sexual orientation [192, 344], crime propensity [358, 365], or emotion [10, 224]. Acknowledging this, our research study employed the same principles underpinning established guidelines [24, 331] for protecting human subjects in research to identify ethical issues in HCCV dataset design, namely autonomy, justice, beneficence, and non-maleficence. Autonomy respects individuals' self-determination—e.g., through informed consent and assent for HCCV datasets. Justice promotes the fair distribution of risks, costs, and benefits, guiding decisions on compensation, data accessibility, and diversity. Beneficence entails the proactive promotion of positive outcomes and well-being, e.g., by soliciting individuals' to self-identify, while non-maleficence centers on minimizing harm and risks during dataset design, e.g., by redacting privacy-leaking image regions and metadata.
To ensure comprehensive consideration, we harnessed diverse expertise, following contemporary, interdisciplinary practices [261, 270, 307]. Our team comprises researchers, practitioners, and
3The checklist can also be found at: https://github.com/SonyResearch/responsible_data_curation.
lawyers with backgrounds in ML, CV, algorithmic fairness, philosophy, and social science. With a range of ethnic, cultural, and gender backgrounds, we bring extensive experience in designing CV datasets, training models, and developing ethical guidelines. To align our expertise with the principles, we collectively discussed them, considering each author's background. After identifying key ethical issues in HCCV data curation practices, we iteratively refined them into an initial draft of ethical considerations. We extensively collected, analyzed, and discussed papers spanning a range of themes such as HCAI, HCCV datasets, data and model documentation, bias detection and mitigation, AI and data design, fairness, and critical AI. Our comprehensive literature review incorporated pertinent studies and datasets, resulting in refined considerations with detailed explanations and recommendations for responsible data curation. Additional details are provided in Appendix B.
3 Purpose
In ML, significant emphasis has been placed on the acquisition and utilization of "general-purpose" datasets [259]. Nevertheless, without a clearly defined task pre-data collection, it becomes challenging to effectively handle issues related to data composition, labeling, data collection methodologies, informed consent, and assessments related to data protection. This section addresses conflicting dataset motivations and provides recommendations.
3.1 Ethical Considerations
Fairness-unaware datasets are inadequate for measuring fairness. Datasets lacking explicit fairness considerations are inadequate for mitigating or studying bias, as they often lack the necessary labels for assessing fairness. For instance, the COCO dataset [196], focused on scene understanding, lacks subject information, making fairness assessments challenging. Researchers, consequently, resort to human annotators to infer, e.g., subject characteristics, limiting bias measurement to visually "inferable" attributes. This introduces annotation bias [56] and the potential for harmful inferences [47, 275].
Fairness-aware datasets are incompatible with common HCCV tasks. Industry practitioners stress the importance of carefully designed and collected "fairness-aware" datasets to detect bias issues [150]. Fabris et al. [93] found that out of 28 CV datasets used in fairness research between 2014 and 2021, only eight were specifically created with fairness in mind. Among these, seven were HCCV datasets (scraped from the web) [43, 170, 216, 308, 319, 342, 343], including five focused on facial analysis. Due to the limited availability and delimited task focus of fairness-aware datasets, researchers repurpose "fairness-unaware" datasets [120, 139, 196, 198, 208, 346, 373]. Fairness-aware datasets fall short in addressing the original tasks associated with well-known HCCV datasets, which encompass a range of tasks, such as segmentation [64, 209], pose estimation [13, 196], localization and detection [73, 91, 110], identity verification [153], action recognition [173], as well as reconstruction, synthesis and manipulation [114, 171]. The absence of fairness-aware datasets with task-specific labels hampers the practical evaluation of HCCV systems, despite their importance in domains such as healthcare [155, 220], autonomous vehicles [163], and sports [317]. Additionally, fairness-aware datasets lack self-identified annotations from image subjects, relying on inferred attributes, e.g., from online resources [43, 308, 319].
3.2 Practical Recommendations
Refrain from repurposing datasets. Existing datasets, repurposable but optimized for specific functions, can inadvertently perpetuate biases and undermine fairness [183]. Repurposing fairness-unaware data for fairness evaluations can result in dirty data, characterized by missing or incorrect information and distorted by individual and societal biases [181, 265]. Dirty data, including inferred data, can have significant downstream consequences, compromising the validity of research, policy, and decision-making [14, 63, 265, 341]. ML practitioners widely agree that a proactive approach to fairness is preferable, involving the direct collection of demographic information from the outset [150]. To mitigate epistemic risk, curated datasets should capture key dimensions influencing fairness and robustness evaluation of HCCV models, i.e., data subjects, instruments, and environments. Model Cards explicitly highlight the significance of these dimensions in fairness and robustness assessments [222].
Create purpose statements. Pre-data collection, dataset creators should establish purpose statements, focusing on motivation rather than cause [129]. Purpose statements address, e.g., data collection motivation, desired composition, permissible uses, and intended consumers. While dataset documentation [108, 258] covers similar questions, it is a reflective process and can be manipulated to fit the narrative of the collected data, as opposed to directing the narrative of the data to be collected. Purpose statements can play a crucial role in preventing both hindsight bias [51, 97, 176] and purpose creep, ensuring alignment with stakeholders' consent and intentions [186]. To enhance transparency and accountability, as recommended by Peng et al. [247], purpose statements can undergo peer review, similar to registered reports [238]. Registered reports, recognized by the UK 2021 Research Excellence Framework, incentivize rigorous research practices and can lead to increased institutional funding [51].
4 Consent and Privacy
Informed consent is crucial in research ethics involving humans [230, 235], ensuring participant safety, protection, and research integrity [59, 253]. Shaping data collection practices in various fields [35, 235], informed consent consists of three elements: information (i.e., the participant should have sufficient knowledge about the study to make their decision), comprehension (i.e., the information about the study should be conveyed in an understandable manner), and voluntariness (i.e., consent must be given free of coercion or undue influence). While consent is not the only legal basis for data processing, it is globally preferred for its legitimacy and ability to foster trust [82, 253]. We address concerns related to consent and privacy, and provide recommendations.
4.1 Ethical Considerations
Human-subjects research. As aforementioned in Section 2, HCCV datasets are frequently collected without informed consent or participation, primarily due to the classification of publicly available data as "minimal risk" within human-subjects research. However, beyond possible predictive privacy harms and unethical downstream uses, collecting data without informed consent hinders researchers and practitioners from fully understanding and addressing potential harms to data subjects [218, 333]. Some argue that consent is pivotal as it provides individuals with a last line of defense against the misuse of their personal information, particularly when it contradicts their interests or wellbeing [77, 223, 245, 253].
Creative Commons loophole. Some datasets have been created based on the misconception that the "unlocking [of] restrictive copyright" [35] through Creative Commons licenses implies data subject consent. However, the Illinois Biometric Information Privacy Act (BIPA) [161] mandates data subject consent, even for publicly available images [370]. In the UK and EU General Data Protection Regulation (GDPR) [88] Article 4(11), images containing faces are considered biometric data, requiring "freely given, specific, informed, and unambiguous" consent from data subjects for data processing. Similarly, in China, the Personal Information Protection Law (PIPL) [233] Article 29 mandates obtaining individual consent for processing sensitive personal information, including biometric data (Article 28). While a Creative Commons license may release copyright restrictions on specific artistic expressions within images [370], it does not apply to image regions containing biometric data such as faces, which are protected by privacy and data protection laws [300].
Vulnerable persons. Nonconsensual data collection methods can result in the inclusion of vulnerable individuals unable to consent or oppose data processing due to power imbalances, limited capacity, or increased risks of harm [89, 207]. While scraping vulnerable individuals' biometric data may be incidental, some researchers actively target them, jeopardizing their sensitive information without guardian consent [128, 260].
Paradoxically, attempts to address racial bias in data have involved soliciting homeless persons of color, further compromising their vulnerability [103]. When participation is due to economic or situational vulnerability, as opposed to one's best interests, monetary offerings may be perceived as inducement [117]. Further ethical concerns manifest when it is unclear whether participants were adequately informed about a research study. For instance, in ethnicity recognition research [72], despite obtaining informed consent, criticism arose due to training a model that discriminates between Chinese Uyghur, Korean, and Tibetan faces. Although the study's focus is on the technology
itself [315], its potential use in enhancing surveillance on Chinese Uyghurs raises ethical questions due to the human rights violations against them [333].
Consent revocation. Dataset creators sometimes view autonomy as a challenge to collecting biometric data for HCCV, especially when data subjects prioritize privacy [214, 287, 297]. Nonetheless, informed consent emphasizes voluntariness, encompassing both the ability to give consent and the right to withdraw it at any time [74]. GDPR grants explicit revocation rights (Article 7) and the right to request erasure of personal data (Article 17) [350]. However, image subjects whose data is collected without consent are denied these rights. The nonconsensual FFHQ face dataset [171] offers an opt-out mechanism, but since inclusion was involuntary, subjects may be unaware of their inclusion, rendering the revocation option hollow. Moreover, this burdens data subjects with tracking the usage of their data in datasets, primarily accessible by approved researchers [81].
Image- and metadata-level privacy attributes. Researchers have focused on obfuscation techniques, e.g., blurring, inpainting, and overlaying, to reduce private information leakage of nonconsensual individuals [46, 101, 194, 195, 213, 252, 311, 323, 362, 367]. Nonetheless, face detection algorithms used in obfuscation may raise legal concerns, particularly if they involve predicting facial landmarks, potentially violating BIPA [61, 370]. BIPA focuses on collecting and using face geometry scans regardless of identification capability, while GDPR protects any identifiable person, requiring data holders to safeguard the privacy of nonconsenting individuals. Moreover, reliance on automated face detection methods raises ethical concerns, as demonstrated by the higher precision of pedestrian detection models on lighter skin types compared to darker skin types [352]. This predictive inequity leads to allocative harm, denying certain groups opportunities and resources, including the rights to safety [322] and privacy [80].
It is important to note that face obfuscation may not guarantee privacy [145, 367]. The Visual Reductions dataset [242] includes 68 image-level privacy attributes, covering biometrics, sensitive attributes, tattoos, national identifiers, signatures, and contact information. Training faceless person recognition systems using full-body cues reveals higher than chance re-identification rates for face blurring and overlaying [239], indicating that solely obfuscating face regions might be insufficient under GDPR. Furthermore, image metadata can also disclose sensitive details, e.g., date, time, and location, as well as copyright information that may include names [11, 239]. This is worrisome for users of commonly targeted platforms like Flickr, which retain metadata by default.
4.2 Practical Recommendations
Obtain voluntary informed consent. Similar to recent consent-driven HCCV datasets [136, 254, 268], explicit informed consent should be obtained from each person depicted in, or otherwise identifiable, in a dataset, allowing the sharing of their facial, body, and biometric information for evaluating the fairness and robustness of HCCV technologies. Datasets collected with consent reduce the risk of being fractured, however, data subjects may later revoke their consent over, e.g., privacy concerns they may not have been aware of at the time of providing consent or language nuances [65, 379]. Following GDPR (Article 7), plain language consent and notice forms are recommended to address the lack of public understanding of AI technologies [199].
When collecting images of individuals under the age of majority or those whose ability to protect themselves is significantly impaired on account of disability, illness, or otherwise, guardian consent is necessary [182]. However, relying solely on guardian consent overlooks the views and dignity of the vulnerable person [141]. To address this, in addition to guardian consent, voluntary informed assent can be sought from a vulnerable person, in accordance with UNICEF's principlism-guided data collection procedures [31, 327]. When employing appropriate language and tools, assent establishes the vulnerable person understands the use of their data and willingly participates [31]. If a vulnerable person expresses dissent or unwillingness to participate, their data should not be collected, regardless of guardian wishes.
Informed by the U.S. National Bioethics Advisory Commission's contextual vulnerability framework [60], dataset creators should assess vulnerability on a continuous scale. That is, the circumstances of participation should be considered, which may require, e.g., a participatory design approach, assurances over compensation, supplementary educational materials, and insulation from hierarchical or authoritative systems [117].
Adopt techniques for consent revocation. To permit consent revocation, dataset creators should implement an appropriate mechanism. One option is dynamic consent, where personalized communication interfaces enable participants to engage more actively in research activities [174, 348]. This approach has been implemented successfully through online platforms, offering options for blanket consent, case-by-case selection, or opt-in depending on the data's use [174, 211, 314]. Alternatively, another recommended approach is to establish a steering board or charitable trust composed of representative dataset participants to make decisions regarding data use [255]. The feasibility of these proposals may vary based on a dataset's scale. Nonetheless, at a minimum, data subjects should be provided a simple and easily accessible method to revoke consent [136, 254, 268]. This aligns with guidance provided by the UK Information Commission's Office (ICO), emphasizing the need to provide alternatives to online-based revocation processes to accommodate varying levels of technology competency and internet access among data subjects [325].
Collect country of residence information. Anonymizing nonconsensual human subjects through face obfuscation, as done in datasets such as ImageNet [367], may not respect the privacy laws specific to the subjects' country of residence. To comply with relevant data protection laws, dataset curators should collect the country of residence from each data subject to determine their legal obligations, helping to ensure that data subjects' rights are protected and future legislative changes are addressed [249, 268]. For instance, GDPR Article 7(3) grants data subjects the right to withdraw consent at any time, which was not explicitly addressed in its predecessor [253].