SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? Third party websites, with appropriate licences allowing us to use the images.
Were the individuals in question notified about the data collection? No, as we are collating images collated by third parties. However it is assumed, based on the associated licences, that the individuals depicted were made aware that their images were taken for public access.
Did the individuals in question consent to the collection and use of their data? Please see the previous question.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? Any images will be removed immediately if there are any objections. To facilitate this process, we have a Google Form which can filled out and the authors are notified by email.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? No.
Any other comments? The authors confirm that, to the best of their knowledge, they are using all intellectual property in accordance with their licences, and the use of the data does not violate any rights. The authors do not take responsibility for any licences that change with time.
L.4 Preprocessing/Cleaning/Labelling
Was any preprocessing/cleaning/labelling of the data done (e.g., discretisation or bucketing, tokenisation, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Manual labelling of the data was conducted at the time of the data collection. Images were also sorted into an occupational taxonomy we created.
Was the "raw" data saved in addition to the preprocessed/cleaned/labelled data (e.g., to support unanticipated future uses)? The only data that was collected is present in the open version of VISOGENDER. The data only includes URIs and the VISOGENDER metadata, but no images are downloaded.
Is the software that was used to preprocess/clean/label the data available? No specific software was used during data collection. However, there is code to benchmark models over the dataset via the URLs and metadata files.
L.5 Uses
Has the dataset been used for any tasks already? The dataset is completely novel and, as of October 2023, has only been used in the original VISOGENDER paper.
Is there a repository that links to any or all papers or systems that use the dataset? Yes. Please see our GitHub repository.
What (other) tasks could the dataset be used for? The dataset should only be used as an evaluation dataset for binary gender bias vision-language models. It is not advised to use this dataset for training purposes, as the binary nature of the labels risks erasure of non-binary people represented in these professional roles.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labelled that might impact future uses? We only collect URLs which link to images hosted on third party sites. There is the potential that these links become deprecated, or image licences change over time. The authors do not take responsibility for any changes to image links, or their associated licences, but will immediately remove any problematic images in the event these are identified.
Are there tasks for which the dataset should not be used? This should not be used for the training of models. It is solely intended as an evaluation dataset.
L.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organisation) on behalf of which the dataset was created? This dataset is publicly available and it is encouraged that developers of VLMs use it to assess their models' propensities for gender bias.
How will the dataset be distributed (e.g., tarball on website, API, GitHub)? GitHub.
When will the dataset be distributed? At the time of the paper being published in 2023.
Will the dataset be distributed under a copyright or other intellectual property (IP) licence, and/or under applicable terms of use (ToU)? Yes. We have included a Data Clause which includes the licence and terms of use in the Supplementary Materialsand in our GitHub repository under "LICENCE": https://github.com/oxai/visogender/blob/main/LICENCE. These URLs are distributed based on their royalty free/ Creative Commons licences that the images occupy at the time of curation. The dataset is open source, but we request that the dataset is cited in any subsequent work. The citation can be found alongside the data on our GitHub repository: https://github. com/oxai/visogender.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No, not at the time of data curation (March - May 2023).
L.7 Maintenance
Who will be supporting/hosting/maintaining the dataset? The authors of the paper and the Oxford Artificial Intelligence Society.
How can the owner/curator/manager of the dataset be contacted (e.g., email address)? The first author (Siobhan Mackenzie Hall) can be contacted by email (siobhan.hall@nds.ox.ac.uk), or via a GitHub Issue: https://github.com/oxai/visogender/issues. Alternative, any issues can be logged on the Google Form which is also linked in the README.
Is there an erratum? N/A at the time of publishing.
Will the dataset be updated (e.g., to correct labelling errors, add new instances, delete instances)? Yes. We encourage anyone that finds fault to contact the authors to amend or remove any problematic URLs. Alternatively, please complete this Google Form to report problematic images and/or labels and the authors will be notified by email immediately. The authors undertake to do the following:
1. The authors will proactively investigate the dataset for broken links, with randomised checks of the images themselves to ensure URLs are not redirecting every 6 months
2. We have uploaded code and instructions to GitHub for easy command line running of a script which checks for URL integrity and that the images can be utilised by models. This will be run by the authors every 6 months
3. Further, we have included a Google Form which can be used to identify broken and/or inappropriate links
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? N/A as the images are under Creative Commons and/or royalty-free licences that don't prohibit the use of images for machine-learning purposes.
Will older versions of the dataset continue to be supported/hosted/maintained? Yes, with the exception of any URLs that change with time and potentially link to problematic images.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes. We encourage anyone looking to expand the dataset to contact the authors to discuss this development.