SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- How do you plan to document the identities of data annotators, including capturing demographic details such as pronouns, age, and ancestry? Can you provide insights into your strategy for gathering and preserving this information while respecting privacy and ensuring transparency?
- Can you elaborate on your approach to highlighting the contributions of annotators beyond data labeling in the dataset documentation after the curation process? How do you intend to accurately represent the multifaceted roles and contributions of annotators?
- How do you plan to report the demographic information of annotators to analyze potential sources of bias in dataset annotations? Can you provide more details about your proposed approach for conducting this analysis while ensuring privacy and ethical considerations?
Fair Treatment and Compensation
- How do you plan to ensure that all contributors receive compensation that exceeds the minimum hourly wage of their respective country or jurisdiction of residence? Can you provide insights into your compensation strategy to ensure fair and ethical remuneration?
- Can you elaborate on your approach to exploring alternative payment models, such as compensation based on the average hourly wage? How do you intend to determine a compensation structure that is both fair and reflective of contributors' efforts?
- How will you establish direct communication channels between dataset creators and contributors? Can you provide more details about the methods you intend to implement for effective and transparent communication?
- What communication methods do you plan to explore that maintain the anonymity of contributors? Can you provide insights into your approach to balancing communication and privacy needs, such as using anonymous feedback forms?
- Can you provide information about your strategy for developing clear and accessible plain language guides to facilitate various tasks, such as image submission and data annotation? How do you plan to ensure that these guides effectively assist contributors?
- How do you intend to ensure that contributors from diverse backgrounds can easily understand and follow any instructions provided? Can you elaborate on your approach to promoting inclusivity and accessibility in your communication and guidelines?
- Can you provide details about how you plan to subject your recruitment and compensation procedures to ethics review? What steps will you take to ensure that your procedures align with ethical considerations and best practices?
B Literature Review
Through a thematic search strategy, we identified relevant research studies and datasets, revealing deficiencies in current image data curation practices or proposing potential solutions. By utilizing Semantic Scholar and Google Scholar, we curated relevant papers covering a wide spectrum of themes, including:
HCAI
Guidelines and best practices
Human-subjects research
- Values in design
HCCV datasets
- Diversity and inclusion
Dataset curation
- Representation
Ethical frameworks and considerations
- Robustness and reliability
Data and model documentation
Benchmarking and evaluation
Legal and regulatory considerations
Bias detection and mitigation
- Privacy and data protection
Critical AI
Social implications
- Responsible AI
Auditing and verification
The themes were chosen based on our expertise and experience in designing CV datasets, training models, and developing ethical guidelines. To ensure a focused approach, we manually selected papers aligned with the scope of our study based on the relevance of a paper's title and abstract. This informed our initial categorization scheme, shown in Table 1, detailing key ethical considerations related to HCCV.
Initially broad, we further refined the categories to address the most prominent ethical issues pertaining to HCCV dataset curation, particularly for fairness and robustness evaluations. Consent and privacy categories were combined due to their interrelated nature and the influence of shared legal frameworks.
Additionally, we integrated acquisition-related considerations into the categories of diversity, consent and privacy, as well as purpose, recognizing their interconnectedness in ethical image data collection, labeling, and usage. Maintenance-related matters were intentionally excluded from our scope, as these primarily pertain to post-dataset creation activities, while technical and organizational security measures are typically covered through consent forms. Ownership concerns, often intertwined with privacy issues, were incorporated into the consent and privacy category.
To establish a comprehensive view, we expanded our corpus as necessary. This encompassed examining cited works within our initial corpus, studies referencing our primary sources, and additional contributions by authors from our initial corpus. Our review was supplemented by incorporating publicly available resources from reputable sources, such as government bodies, private institutions, and reliable news organizations. In total, our analysis covered 500 research studies and online resources.