Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace
Authors: Xinyu Yang
Weixin Liang
James Zou
Venue: ICLR 2024 poster (2024)
Keywords: dataset documentation
data-centric AI
large-scale analysis
Primary area: datasets and benchmarks

Abstract: Advances in machine learning are closely tied to the creation of datasets. While data documentation is widely recognized as essential to the reliability, reproducibility, and transparency of ML, we lack a systematic empirical understanding of current dataset documentation practices. To shed light on this question, here we take Hugging Face - one of the largest platforms for sharing and collaborating on ML models and datasets -  as a prominent case study. By analyzing all 7,433 dataset documentation on Hugging Face, our investigation provides an overview of the Hugging Face dataset ecosystem and insights into dataset documentation practices, yielding 5 main findings: (1) The dataset card completion rate shows marked heterogeneity correlated with dataset popularity: While 86.0\% of the top 100 downloaded dataset cards fill out all sections suggested by Hugging Face community, only 7.9\% of dataset cards with no downloads complete all these sections. (2) A granular examination of each section within the dataset card reveals that the practitioners seem to prioritize Dataset Description and Dataset Structure sections, accounting for 36.2\% and 33.6\% of the total card length, respectively, for the most downloaded datasets. In contrast, the Considerations for Using the Data section receives the lowest proportion of content, accounting for just 2.1\% of the text. (3) By analyzing the subsections within each section and utilizing topic modeling to identify key topics, we uncover what is discussed in each section, and underscore significant themes encompassing both technical and social impacts, as well as limitations within the Considerations for Using the Data section. (4) Our findings also highlight the need for improved accessibility and reproducibility of datasets in the Usage sections. (5) In addition, our human annotation evaluation emphasizes the pivotal role of comprehensive dataset content in shaping individuals' perceptions of a dataset card's overall quality. Overall, our study offers a unique perspective on analyzing dataset documentation through large-scale data science analysis and underlines the need for more thorough dataset documentation in machine learning research.

Published as a conference paper at ICLR 2024
NAVIGATING DATASET DOCUMENTATIONS IN AI: A LARGE-SCALE ANALYSIS OF DATASET CARDS ON HUGGING FACE
Xinyu Yang
Weixin Liang" Stanford University
James Zou Stanford University
Cornell University xy4680corne11.c edu
wxlianggestanfoxdd.
j jmesz anford. edu
ABSTRACT
Advances in machine learning closely tied to the creation of datasets. While data documentation is widely recognized as essential to the reliability, reproducibility, and transparency of MI., lack systematic empirical understanding of current dataset documentation practices. To shed light on this question, here we take Hugging Face one of the largest platforms for sharing and collaborating on ML models and datasets - as prominent case study. By analyzing all 7,433 dataset documentation on Hugging Face, our investigation provides an overview of the Hugging Face dataset ecosystem and insights into dataset documentation practices, yielding 5 main findings: (1) The dataset card completion rate shows marked heterogeneity correlated with dataset popularity: While 86.0% of the top 100 downloaded dataset cards fill out all sections suggested by Hugging Face community, only 7.9% of dataset cards with downloads complete all these sections. (2) A granular examination each section within the dataset card reveals that the practitioners
prioritize Dataset Description and Dataset Structure sections, accounting for 36.2% and 33.6% of the total card length, respectively, for the most downloaded datasets. In contrast, the Considerations for Using the Data section receives the lowest proportion of content, accounting for just 2.1 % of the text. (3) By analyzing the subsections within each section and utilizing topic modeling to identify key topics, we uncover what is discussed in each section, and underscore significant themes encompassing both technical and social impacts, as well as limitations within the Considerations for Using the Data section. (4) Our findings also highlight the need for improved accessibility and reproducibility of datasets in the Usage sections. (5) In addition, our human annotation evaluation emphasizes the pivotal role of comprehensive dataset content in shaping individuals' perceptions of dataset card's 's overall quality. Overall, our study offers unique perspective on analyzing dataset documentation through large-scale data science analysis and underlines the need for more thorough dataset documentation in machine learning research.
INTRODUCTION
Datasets form the backbone of machine learning research (Koch et al. 2021). The proliferation of machine learning research has spurred rapid advancements in machine learning dataset development, validation, and real-world deployment across academia and industry. Such growing availability of MI. datasets underscores the crucial role of proper documentation in ensuring transparency, reproducibility, and data quality in research (Haibe- Kains et al.) [2200] (Sodden et al. 2018 Hutson 2018). Documentation provides details about the dataset, including sources of data, methods used to collect it, and preprocessing or cleaning that was performed. This information holds significant value for dataset users, it facilitates quick understanding of the dataset's motivation and its overall scope. These insights are also crucial for fostering responsible data sharing and promoting interdisciplinary collaborations.
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2024
Despite numerous studies exploring the structure and content of dataset cards across various research domains (Afzal et al. 2020; Gebru et al.] [021] Papakyriakopoulos et al.] 2023 Barman et al.) 2023; Costa-jussa et al. 2020), there remains notable gap in empirical analyses of community norms and practices for dataset documentation. This knowledge gap is significant because adherence to community norms and the quality of dataset documentation directly impact the transparency, reliability, and reproducibility in the field of data-driven research. For instance, inadequate dataset descriptions, structural details, or limitations can hinder users from utilizing the dataset appropriately, potentially resulting in misuse or unintended consequences; the absence of information on data cleaning and readiness assessment practices in data documentation limits dataset reusability and productivity gains. Furthermore, without systematic analysis of current dataset documentation practices, we risk perpetuating insufficient documentation standards, which can impede efforts to ensure fairness, accountability, and equitable use of Al technologies.
To address this question, we conducted a comprehensive empirical analysis of dataset cards hosted on Hugging Face, one of the largest platforms for sharing and collaborating ML models and datasets, as a prominent case study. Dataset cards on the Hugging Face platform are Markdown files that serve as the README for dataset repository. While several open-source platforms also facilitate the sharing of ML datasets, such as Kaggle, Papers with Code, and GitHub, we chose Hugging Face for two primary reasons. Firstly, it stands out as of the most popular platforms for developers to publish, share, and reuse ML-bbased projects, offering a vast repository of MI datasets for study, Secondly, Hugging Face is one of the few open-source platforms that offer an official dataset card template. This feature not only enhances the accessibility and user-frien secrffrenndlinss of the dataset card community but also makes the analysis process more efficient and informative.
By analyzing all 7,433 dataset documentation hosted on Hugging Face, our investigation provides an overview of the Hugging Face dataset ecosystem and insights into dataset documentation prac- tices. Based on our research findings, we emphasize the importance of comprehensive dataset documentation and offer suggestions to practitioners on how to write documentation that promotes reproducibility, transparency, and accessibility of their datasets, which can help to improve the overall quality and usability of the dataset community. Our study aims to bridge the notable gap in the community concerning data documentation norms, taking the first step toward identifying deficiencies in current practices and offering guidelines for enhancing dataset documentation.
Total of Datasens SD
repostories Expenential Ft:
Detaser wi dessset card Pire
Namber of Datssets
1000
10,30
20k
Delumat wel dataner card
1000
15c
Dowrloats
10a
5002
30*
Debnnt
Deseel
10k
80% of
Dowe load Tralife
10.0%
70%
50%
Wenkly growch rall:: 3.07%
29%
60%
2022 2023
16P
0%
Non
Dataset
Grneel Eard Doeese
Figure 1: Systematic Analysis of 24,065 Datasets Hosted on Hugging Face. (a) Exponential Growth of Datasets: The Hugging Face platform has seen a remarkable surge in the number of datasets, with the count doubling approximately every 18 weeks. (b) Power Law in Dataset Usage: Dataset downloads on Hugging Face follow a power-law distribution, as indicated by the linear relationship on the log-log plot. The top 82 datasets account for 80% of the total downloads; datasets with documentation dominate the top downloaded datasets. (c) Documentation Associated with Usage: Despite only 30.9% of dataset repositories (7,433 out of 24,065) featuring non-empty dataset cards, these datasets account for an overwhelming 95.0% of total download traffic on the platform.
2
OVERVIEW
Finding
Exponential Growth of Datasets: The number of datasets on Hugging Face doubles every
18 weeks.
Documentation Associated with Usage: 95.0% of download traffic comes from the 30.9% of datasets with documentation.
2

---

Published as a conference paper at ICLR 2024
Exponential Growth of Datasets Our analysis encompasses 24,065 dataset repositories Hug- ging Face uploaded by 7,811 distinct user accounts as of March 16th, 2023 (see Table. S3 for varying documentation practices by creators). The number of datasets exhibits exponential growth, with weekly growth rate of 3.97% and a doubling time of 18 weeks (Fig. Th). As sanity check, the number of dataset repositories reached 35,973 by May 23rd, 2023, confirming the exponential trend.
Power Law in Dataset Usage Although Hugging Face has seen a significant increase in the number of dataset repositories, our analysis reveals sigmificant imbalance in dataset downloads, which follows power law distribution. This means that a small proportion of the most popular datasets receive the majority of the downloads, while the vast majority of datasets receive very few downloads. In fact, our analysis shows that just the 82 datasets with the most downloads account for 80% of total downloads (Fig. [p). Fig, [S4 further demonstrates that the power law distribution persists across various task domains, even with the varied number of datasets within each domain.
Documentation Associated with Usage Despite the importance of dataset cards, only 58.2% (14,011 out of 24,065 dataset repositories contributed by 4,782 distinct user accounts) include dataset cards as Markdown README..d files within their dataset repositories. Among these, 6,578 dataset cards empty, resulting in only 30.9% (7,433 of 24,065 dataset repositories contributed by 1,982 distinct user accounts) featuring non-empty dataset cards (Fig. 1 As illustrated in Fig. T.. dataset cards are prevalent among the most downloaded datasets. Notably, datasets with non-empty dataset cards account for 95.0% of total download traffic, underscoring a potential positive correlation between dataset cards and dataset popularity. For the rest of the paper, we focus our analyses on these 7,433 non-empty dataset cards. We sort these non-empty dataset cards based on the number of downloads for the corresponding datasets. So top k dataset cards (e.g. 100) refer the dataset cards corresponding to the k most downloaded datasets.
3 STRUCTURE OF DATASET DOCUMENTATIONS
Finding
The dataset card completion rate shows marked heterogeneity correlated with dataset popularity: While 86.0% the top 100 downloaded datasets out sections suggested by the Hugging Face community, only 7.9% of dataset cards with downloads complete all these sections.
Section Title
Subsection Title
Description
Dataset Summary
A brief summary of the dalasel, including its intended use, supported taeks, an overview of how and why the datasel was created, etc.
Supported Tasks
Description
Leaderboards
description
tag,
suggested
Languages
The languages represented in the dataset.
Data Instances
ISON-formet example and descrition of typical instance in the datasel.
Dataset Structure
they are Criteria for spliting the data, descriptive statisiles for the features, such as size, average longth, etc.
Mention used as impul output any of tacks dalasel currently supports.
fields present
lype,
Data Spiits
Curation Rationale
Motivation for the creation of the dalaset.
Dataset Creation
Souros
including data collection process, data producerr
source (e.g. news
headlines,
poets,
seritences, elc.).
Annotations
Annotation process, annotation tools. annotators,
Personal and Sensitive
Stalement of whether the datssel contains other
might be considered senellive (e.g., data
Information
that reveals racial or othnic origins. financial health data, alc.).
Considerations for Using the Data
Social Impect
ways the
impact society.
Discussion of Blases
Descriations of specific bleses that
be reflected in data
Other Known Limittions
Other limiztions of the dataset, like annotation artfacts.
Dataset Curators
The people involved in collecting the dotaset and their atfilationss)
Additional Information
Licansing
The license to
webpage available.
Citation Information
BibTes-formatien
for dataset.
Contributions
Thanks to Bgithbb-seemmeee for adding this dalasel.'
Table Community-Fndorse -Endorsed Dataset Card Structure. This table shows the sections and their suggested subsections provided by the Hugging Face community, along with their descriptions. For more information, please refer to http:::///ittuuu wtt../..oo///oggggniccccece http://uuggingaaaaaaatttttttstts datasets/ bllob/main/tenti att.ep/Aaatt///Eee
README, README_ggiieeeeeeeee .md
3

---

Published as a conference paper at ICLR 2024
Community-Endorsee Dataset Card Structure Grounded in academic literature (Mitchell et al. 2019) and official guidelines from Hugging Face (HuggingFace) [021], the Hugging Face community provides suggestions for what to write in each section. This provides a standardized structure for conveying key information about datasets. It generally contains 5 sections: Dataset Description, Dataset Structure, Dataset Creation, Considerations for Using the Data, and Additional Information (Table. [ To examine the structure of dataset cards, we used pipeline that detects exact word matches for each section title. We then identified the section titles and checked whether they had contents (Appendix B.I). If dataset card had all five sections completed, we considered be following the community-enooree dataset card.
dataset card
100%
Adherence to Gommunity-Endorrse Guidelines Correlates with Popularity Our evaluation found that popular datasets have better adherence to the dataset card community-endorsed dataset card structure. As illustrated in Fig. 2 compliance with
Frantion Dataner Cards
Contaiiing Al Soctioss
80%
40%
20%
the template varies significantly among datasets with different download counts. Among the 7,433 dataset cards analyzed, 86.0% of the top 100 downloaded dataset cards have completed all five sections of the Community-endorse
Top 100
Top 500
Top
1900 Top
2500 Top 5000
All DoweloadenO
Figure 2: Highly downloaded datasets consis- tently show better compliance with the community- endorsed documentation structure.
dataset card, while only 7.9% of dataset eards with no downloads follow it. Fig. $5 further reveals that popular dataset cards achieve higher completion in all Hugging Face-recommedded sections. This implies potential correlation between adherence to community-endoosee guidelines and dataset popularity.
PRACTITIONERS EMPHASIZE DESCRIPTION AND STRUCTURE OVER SOCIAL IMPACT AND LIMITATIONS
Finding
Practitioners seem to prioritize on Dataset Description and Dataset Structure sections, which account for 36.2% and 33.6% of the total card length, respectively, on the top 100 most downloaded datasets.
In contrast, the Considerations for Using the Data section receives the lowest proportion content, just 2.1 %. The Considerations for Using the Data section covers the social impact of datasets, discussions of biases, and limitations of datasets.
Social Impact, Dataset Limitations and Biases are Lacking in Most Documentations Following the community-endorss dataset card, we conducted an analysis to determine the level of emphasis placed each section. Fig. (3b shows the word count distribution among the top 100 downloaded dataset cards, revealing their high level of comprehensiveness: 91.0% of them have a word count exceeding 200. We step further into these dataset cards examine the emphasis placed each section. We calculated the word count of each section and its proportion to the entire dataset card. As shown in Fig. p. the Dataset Description and Dataset Structure sections received the most attention, accounting for 36.2% and 33.6% of the dataset card length, respectively. On the other hand, the Considerations for Using the Data section received a notably low proportion of only 2.1 1.1%
Section Length Reflects Practitioner Attention The length of sections within dataset cards is reflective of practitioner attention, and it varies significantly based on the popularity of the dataset. Highly downloaded datasets tend to have more comprehensive and longer dataset cards (Fig. (3)) with an emphasis on the Dataset Description and Dataset Structure sections (Fig. 3 Conversely, less popular datasets have shorter cards (Fig. 31) with a greater emphasis the Additional Information section (Fig. 3d. Despite this, sections such as Dataser Creation and Considerations for Using the Data consistently receive lower attention, regardless of download rates (Fig. 3d. This suggests need to promote more comprehensive documentation, particularly in critical sections, to enhance dataset usage and facilitate ethical considerations.
4

---

Published as a conference paper at ICLR 2024
a
40.0%
700
Percentege of Daleset Cards Word
800
30%
Word Count
500
25%
400
20%
300
10%
13.0%
200
10%
7.0%
100
<200
200-400
400-600
600-800
800-1000
+1000
500
1900
Cords
Word Count
Dataset Descriation Considerations
Dataset Structure
Danzset Creation
Additiona Information
Proportion of Section Length
40%s
Dataset
Dotaset Descripiin
30%
Daraseet Card
Constderations
20%
Ussing
Driae
Top 1000
Top 2000
Top 3000
Top 4000
Top 5000
Top 6000
Top 7000
Figure 3: Section Length Reflects Practitioner Attention. (a) Popularity Correlates with Docu- mentation Length: The top downloaded dataset cards are longer, indicating that they contain more comprehensive information. (b) Distribution of Word Count Among Top 100 Downloaded Dataset Cards (c) Section Length Proportions in Top 100 Downloaded Dataset Cards: The Dataset De- scription and Dataset Structure sections dominate in the top 100 downloaded dataset cards, with proportions of 36.2% and 33.6%, respectively. In contrast, the Considerations for Using the Data section receives the least attention, with a proportion only 2.1%. (d) Section Length Proportion Changes over Downloads: The section length proportion changes over downloads, with Dataset Description and Dataset Structure decreasing in length, and Additional Information and Other increas- ing. Notably, there is consistently low emphasis placed on the Dataset Creation and Considerations for Using the Data sections across all dataset cards with different downloads.
UNDERSTANDING CONTENT DYNAMICS IN DATASET DOCUMENTATION
Finding
Practitioners contributing to the Hugging Face community exhibit high compliance with standards, filling 14 of the 17 recommended subsections across five main sections at a rate exceeding 50%.
Strong Community Adherence to Subsection Guidelines:
Emergence of the Usage Section Beyond the Community Template: Surprisingly. 33.2% dataset cards includes Usage section. The community template does not include such Usage section in its current form and should include one in the future.
Section Content Detection Pipeline To gain a deeper understanding the topics discussed in each section, we conducted content analysis within each section of the community-endorsed dataset card structure, which includes suggested subsections within the five main sections. We used exact keyword matching to identify the corresponding subsections and calculate their filled-out rates. Fig. 4 shows that 14 out of 17 subsections have filled-out rates above 50% indicating adherence to the community-endorsee dataset cards.
Limitation Section is Rare, but Long if it Exists The Considerations for Using the Data section (i.e., limitation section), despite being frequently overlooked and often left empty by practitioners, holds particular significance. When this section is included, it tends to adhere well to community guidelines, with subsections having a completion rate exceeding 50% and a reasonably substantial word count (98.2 words). This suggests that this section has the potential to provide valuable insights and guidance. This motivates our use of topic modeling to identify key discussion topics within this section, potentially aiding practitioners in crafting meaningful content.
5

---

Published as a conference paper at ICLR 2024
Delioer Cards
Section: Uataset Description
Section: Uataset Structure
Section: Ualaset Creation
Dateser:
90.0%
8%
Partonate
51
0%
S Suboesion
Sulvection
Sourcs Data
70.0%
70.0%
Lnaderbocrdds
ledie and
n0 at%
Dasa Flo ds
50 83
Annodatooss
66
6%
Spiits
77.5%
Personal and
Intormatiin
Sersilive
41. 1%%
3%
20%
40%
60%
80%
100%
20%
80%
100%
Section: Considerations for Using the Data
Section: Additional Information
Seation: Other
18.1%
Darasot
Curatoos
19.0%
AbouR
18.8%
Lioureinn Informrtion
50 50.9%
Dacaaooon
67.0%
73.3%
Chaekon
64.3%
Livisaiion
Contibutions
54 m
15%
16.7%
Figure 4: Highlighting the Hugging Face Community's Compliance with Subsection Guidelines. This figure shows subsection filled-out rates within different sections, stratified by download counts. Each section has multiple subsections, with bars representing the filled-out rate of each subsection. Green texts indicate filled-out rates above 50%, while red texts indicate rates below 50%. Of the 17 subsections within the five sections of the community-endorse dataset, 14 have filled-out rates above 50%.
Social Impact Dataset
Topic
may Representative Sentences
educational activities. The creation the dataset contributes to expanding the scope NLP research to under-explored languages across the world.
Adding Spanish
improve their
Technical or Research Scope
This dataset be used gain ghts into social, cultural, and political views of people African countries,
Social cope Background
this matter isn't tackled with enough urgency, we might see the nse of new dark era in Latin America politics many people.
many unscrupulous parties and people manage gain power
Discussion of Blases
Topic
Representative Sentences
Subpapulation Blases
Gender speakers distribution imbalanced, percentage female speakers mostly lower than 50% across languages.
The social blases of the time terms of race, sece, gender, etr. might be encountered this dataset.
With respect to the potential risks, we note that the subjectivity of human annotation would impact on the quality of
Blases from Collection Procedure
terms data collection, by using keywords and user mentions, we are introducing some bias the dats. the list keywords and
restricting
created.
Other Known Limitations
Topic
Representative Sentences
The nature of task introduce variabilty in quality the target translations. number of errors, omissions incorsistencies are expected be found within corpus.
Quality
Processing
augmentation pracess sometimes create nonexistent versions people. atellite a annotation is not as accurate for pixel-level representation due to single point annoaaioos.
Limitaaion
Figure 5: Key Topics in Considerations for Using the Data through Topic Modeling Analysis.
This figure displays the outcomes the topic modeling assessment on the contents of the (a) Social Impact of Datasel Subsection, (b) Discussion of Biases Subsection, and (c) Other Known Limitations Subsection. Fach panel illustrates the human-assigned topic label and representative sentences for each section. Topics generated by Latent Dirichlet Allocation (LDA).
Limitation Section Covers Diverse and Crucial Topics The Considerations for Using the Data section (i.e., limitation section) encompasses diverse and erucial topies. The Hugging Face community emphasizes three major themes within this section: Social Impact of Dataset, Discussion of Biases, and Other Known Limitations.
The Social Impact of Dataset aspect explores not only societal implications but also the potential benefits to technology and research communities. In this section, practitioners discuss issues like

---

Published as a conference paper at ICLR 2024
how the dataset can expand the scope of NLP research (Armstrong et al. (022), and increase access to natural language technology across diverse regions and cultures Taace et al. 2021). Additionally, the subsection covers sensitive topics related to polities, ethics, and culture within the social scope.
Discussion of Biases delves into subpopulation bias and data collection biases, highlighting the importance of addressing bias-related issues. Previous research have identified numerous technical and social biases such as subgroup bias (Buolamwini & Gebru 2018), data collection bias (Wang) al. 2019), and label bias (Jiang & Nachum 2020). Our topic modeling results reveal that two primary biases are discussed by practitioners in this subsection. The first is subpopulation bias, which includes biases related to gender, age, or race. For instance, an audio dataset (Nsoesie & Galea (2022 notes that female speakers are underrepresented, comprising less than 50% of the dataset. The second major bias arises from the data collection process, specifically the annotation process, which is often significant bottleneck and source of errors.
Lastly, Other Known Limitations focuses on technical limitations, particularly data quality and processing limitations. This comprehensive coverage underscores the multifaceted nature of consider- ations related to dataset usage. Data quality often focus in other disciplines, such as the social sciences and biomedicine, and there are many insights to draw upon (Paullada et al. [02]] Fedorov 2010; Fan & Geerts 2012). Mcanwhile, processing limitations encompass a broader range of issues beyond biases from the collection procedure, such as inaccuracies or the absence of some data points.
Emergence of the Usage Section Beyond the Community Template While Hugging Face's community-endorsed dataset card structure comprises five main sections, there are instances where practitioners encounter valuable information that doesn't 't neatly fit into these sections. These addi- tional sections, referred to as Other sections, can contain important content. Notably, among these Other sections, discussions related to Usage emerge as a frequent (nearly one-third of the time, 33.2%) and significant theme. These Usage sections offer diverse range of information, including details on downloading, version specifications, and general guidelines to maximize the dataset's 's utility. This highlights the importance of considering content that falls outside the predefined template and suggests potential area for improvement in dataset card templates.
Quantifying the Impact of Usage Section on Dataset Downloads To assess the influence of Usage section in dataset documentation, we conducted counterfactual analysis experiment (Appendix. C). We trained a BERT (Devlin et al. 2018) model using dataset card content and download counts, which were normalized fall within the range of [O. 1] for meaningful comparisons. When a dataset card that initially included a Usage section had this section removed, there was substantial decrease of 1.85% in downloads, with statistical significance. This result underscores the significant impact of the Usage section in bolstering dataset accessibility and popularity, emphasizing its pivotal role in enhancing the documentation and usability of datasets.
ANALYZING HUMAN PERCEIVED DATASET DOCUMENTATION QUALITY
Finding
Our human annotation evaluation emphasizes the pivotal role of comprehensive dataset content in shaping individuals' perceptions of a dataset card's overall quality.
Human Annotations for Comprehensive Evaluation of Dataset Card Quality We utilized human annotations to evaluate the quality of dataset cards, considering seven distinct aspects, drawing from prior research in dataset documentation literature and the Hugging Face
dataset card (Afzal et al., 2020; Gebru et al. 2021 Paaaayriakopoulls al., 2023 Barman et al. 2023
Costa-jussà et al. 2020): (1) Structural Organization, (2) Content Comprehensiveness, (3) Dataset Description, (4) Dataset Structure, (5) Dataset Preprocessing, (6) Usage Guidance, and (7) Additional Information. While Dataset Description, Dataset Structure, and Additional Information can be found in sections of community-endorsee dataset cards, we added evaluation aspects highlighted in the literature, like aspects that constitute the overall presentation (Structural Organization and Content Comprchensiveness)) Data Preprocessing and Usage Guidance. To conduct this assessment, we randomly selected subset containing 150 dataset cards and engaged five human annotators. These
7

---

Published as a conference paper at ICLR 2024
annotators were tasked with evaluating each dataset card across these seven aspects and providing an overall quality score within a range of 5 (Appendix [.2).. The overall quality is assessed through the subjective perception of human annotators, taking into account the seven aspects well their
overall impression. This evaluation approach aims to provide a comprehensive assessment of dataset card quality, reflecting the importance of these aspects in effective dataset documentation.
Human Perception of Documentation Quality Strongly Aligns with Quantitative Analysis Human annotation evaluation of dataset cards shows varying scores across different aspects. While Dataset Description (2.92/5), Structural Organization (2.82/5), Data Structure (2.7/5), and Content Comprehensiveness (2. (2.48/5) received relatively higher scores, areas like Data Preprocessing (1.21/5) and Usage Guidance (1. 1./55) scored lower. This aligns with the quantitative analysis that indicates a greater emphasis on the Dataset Description and Dataset Structure sections. Notably, even the highest-scoring aspect, Dataset Description, falls below 60% of the highest possible score, indicating room for improvement in dataset documentation.
Content Comprehensiveness has the strongest positive correlation with the overall quality of dataset card (Coefficient: 0.3935, p-value: 3.67E-07), emphasizing the pivotal role of comprehensive dataset content in shaping individuals' perceptions of dataset card's overall quality. Additionally, aspects like Dataset Description (Coefficient: 0.2137, p-value: 3.04E-07), Structural Organization (Coefficient: 0.1111, p-value: 2.17E-03), Data Structure (Coefficient: 0.0880, p- value: 6.49E-03), and Data Preprocessing (Coefficient: 0.0855, p-value: 2.27E-03) also significantly contribute to people's 's evaluations of dataset documentation quality. Moreover, the length a dataset card is positively related to Content Comprchensiveness (p-value: 1.89E-011), reinforcing the importance of detailed documentation in enhancing dataset quality and usability.
7 RELATED WORKS
Dataset has long been seen significant constraint in the realm of machine learning research (Halevy) ct al. 2009 Sun et al. 2017). The process of creating datasets remains arduous and time-intensive, primarily due the costs of curation and annotation (IBM 2020). Moreover, the quality of data assumes a pivotal role in shaping the outcomes of machine learning research (Liang et al.) 2022). Consequently, a profound understanding of datasets indispensable in the context of machine learning research, and this understanding is most effectively conveyed through comprehensive dataset documentation.
A long- standing problem in the literature is that there is no industry standard being formed about data documentation. Therefore, much existing work in the literature has been in exploring, conceptualizing and proposing different dataset documentation frameworks. Data-focused tools such as datasheets for datasets and data nutrition labels have been proposed to promote communication between dataset creators and users, and address the lack of industry-wide standards for documenting AI datasets (Bender & Friedman, 2018; [ender et al.] 202T: PPuskkarna et al.) 2022 Gebru et al.
2021 Holland Holland et al.] 2018| Chmielinski et al.) 2022 Papakyriakopoulos al. 2023). Additionally,
there are studies that concentrate on leveraging human-centered methods to scrutinize the design and evaluation aspects of dataset documentation (Fabris et al.) 2022) Mahajan & Shaikh 1 Hanley et al.) 2020; Hutiri et al.) 2022). In the library domain, numerous works have proposed methods tackle the absence of universally accepted guidelines for publishing library-linked data. These efforts are aimed at enhancing data quality, promoting interoperability. and facilitating the discoverability of data resources (Villazón- Terrazas al., 2011 |Hidalgo -Delgad et al. 2017 Abida et al. 2020). These tools and frameworks provide detailed information on the composition, collection process, recommended uses, and other contextual factors of datasets, promoting greater transparency, accountability, and reproducibility of AI results while mitigating unwanted biases in Al datasets. Additionally, they enable dataset creators to be more intentional throughout the dataset creation process. Consequently, datasheets and other forms of data documentation are now commonly included with datasets, helping researchers and practitioners to select the most appropriate dataset for their particular needs.
Despite the proliferation of dataset documentation tools and the growing emphasis on them, the current landscape of dataset documentation remains largely unexplored. In this paper, we present
8

---

Published as a conference paper at ICLR 2024
comprehensive analysis of Al dataset documentation Hugging Face to provide insights into current dataset documentation practices.
DISCUSSION
In this paper, we present a comprehensive large-scale analysis of 7,433 Al dataset documentation on Hugging Face. The analysis offers insights into the current state of adoption of dataset cards by the community, evaluates the effectiveness of current documentation efforts, and provides guidelines for writing effective dataset cards. Overall, our main findings cover 5 aspects:
Varied Adherence to Community- Endorsed Dataset Card: We observe that high high-downloaded dataset cards tend to adhere more closely to the community-cndone
dataset card structure.
Varied Emphasis on Sections: Our analysis of individual sections within dataset cards reveals that practitioners place varying levels of emphasis on different sections. For instance, among the top 100 downloaded dataset cards, Dataset Description and Dataset Structure sections receive the most attention. In contrast, the Considerations for Using the Data section garners notably lower engagement across all downloads, with only approximately 2% of dataset cards containing this section. This discrepancy can be attributed the section's 's content, which involves detailing limitations, biases, and the societal impact of datasets a more complex and nuanced endeavor. An internal user study conducted by Hugging Face (Hugging (202) also identified the Limitation section within this category as the most challenging to compose.
Topics Discussed in Each Section: Our examination of subsections within each section of dataset cards reveals a high completion rate for those suggested by the Hugging Face community. This highlights the effectiveness of the community-endorsee dataset card structure. In particular, our study places a special focus on the Considerations for Using the Data section, employing topic modeling identify key themes, including technical and social aspects of dataset limitations and impact.
Importance of Including Usage Sections: We observe that many dataset card creators go beyond the recommended structure by incorporating Usage sections, which provide instructions on effectively using the dataset. Our empirical experiment showcases the potential positive impact of these Usage sections in promoting datasets, underscoring their significance.
Human Evaluation of Dataset Card Quality: Our human evaluation of dataset card quality aligns well with our quantitative analysis. It underscores the pivotal role of Content Comprehensiveness in shaping people's assessments of dataset card quality. This finding offers clear guidance to practitioners, emphasizing the importance of creating comprehensive dataset cards. Moreover, we establish quantitative relationship between Content Comprehensiveness and the word length of dataset cards, providing a measurable method for evaluation.
Limitations and Future Works Our analysis of MI. dataset documentation relies on the distinctive community-curatte resource, Hugging Face, which may introduce biases and limitations due the platform's 's structure and coverage. For example, Hugging Face's NLP-oriented concentration could introduce biases into the dataset categories. However, our method is transferable and could easily be reproduced for another platform, facilitating future studies (Appendix. E). Additionally, analysis of completeness and informativeness is based on word count and topic modeling. which may not fully capture the nuances of the documentation. Furthermore, measuring dataset popularity based on downloads alone may not fully reflect the dataset's 's impact. Future research could consider additional factors, such the creation time of the dataset and research area of the dataset (Appendix. [D) Lastly, our human evaluation serves as a preliminary evaluation. Future analyses could involve more diverse group of annotators with varying backgrounds and perspectives.
Research Significance To summarize, our study uncovers the current community norms and practices in dataset documentation, and demonstrates the importance of comprehensive dataset documentation in promoting transparency, accessibility, and reproducibility in the AI community. We hope to offer foundation step in the large-scale empirical analysis of dataset documentation practices and contribute to the responsible and ethical use of AI while highlighting the importance of ongoing efforts to improve dataset documentation practices.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We have assembled a collection of dataset cards as a community resource, which includes extracted metadata such the number of downloads and textual analyses. This resource com Seaom/ooogxiinlllel
along with our analysis code can be accessed at /ettts:://gitiee The Hugging Face datasets can be accessed through the Hugging Face Hub API, which is available at httt:::: http://///hiinnaacccccco//// huggingface, hatpee__ubpppcaaacaa rewereeceeeeeeeeeeeeeeeeeeeeeeee
ACKNOWLEDGMENTS
We thank Yian Yin and Nazneen Rajani for their helpful comments and discussions. J.Z. is supported by the National Science Foundation (CCF 1763191 and CAREER 1942926), the US National Insti- tutes of Health (P30AG059307 and U01MH098953) and grants from the Silicon Valley Foundation and the Chan-Zuckerberg Initiative.
REFERENCES
Rabeb Abida, Emna Hachicha Belghith, and Anthony Cleve. An end-to-end framework for integrating and publishing linked open government data. In 2020 IEEE 29th International Conference Enabling Technologies: Infrustructure for Collaborative Enterprises (WETICE), pp. 257-262, 2020. doi: 10.1 1L1109/WETCCHH499222000000
Shazia Afzal, Rajmohan C, Manish Kesarwani, Sameep Mehta, and Hima Patel. Data readiness report, 2020.
Ruth-Ann Armstrong, John Hewitt, and Christopher Manning. Jampatoisnli: jamaican patois natural language inference dataset. arXiv preprint arXiv:2212.03417 2022.
Nabajeet Barman, Yuriy Reznik, and Maria Martini. Datasheet for subjective and objective quality assessment datasets, 2023.
Emily M Bender and Batya Friedman. Data statements for natural language processing: Toward miti- gating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018.
Emily M Bender, Batya Friedman, and Angelina McMillan-Major. guide for writing data statements for natural language processing, 2021.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018
Kasia S Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. The dataset nutrition label (2nd gen): Leveraging context mitigate harms in artificial intelligence. arXiv preprint arXiv:2201.03955. 2022.
Marta R. Costa-jussà, Roger Creus, Oriol Domingo, Albert Domínguez, Miquel Escobar, Cayetana López, Marina Garcia, and Margarita Geleta. Mt-adapted datasheets for datasets: Template and repository, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR,
2018. URL
httt:://axxi>
Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. Tackling doc- umentation debt: A survey algorithmic fairness datasets. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 450344999 doi:
10.1145/5551224.355222 URL https: : //w.i. org/10 /ttp://o........1//////555550
1el..1151131144400 3555286
Wenfei Fan and Floris Geerts. Foundations of data quality management. Synthesis Lectures on Data Management, 4(5):1-217, 2012.
10

---

Published as a conference paper at ICLR 2024
Valerii Fedorov. Optimal experimental design. Wiley Interdisciplinary Reviews: Computational Statistics, 2(5):581-589 2010.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64(12):86 92. nov 2021. ISSN 0001-0782. doi: 10/1345/34 URL https:://oi..orr//110000 3458723
Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Massive Analysis Quality Control (MAQC) Society Board of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susamna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38 Mason Christopher E. 39 Jones Wendell 40 Dopazo Joaquin 41 Furlanello Cesare 42, Levi Waldron, Bo Wang. Chris McIntosh, Anna Goldenberg, Anshul Kundaje, et al. Transparency and reproducibility in artificial intelligence. Nature, 586(7829)
2020.
Alon Halevy, Peter Norvig, and Fernando Pereira. The unreasonable effectiveness of data. IEEE intelligent systems, 24(2):8-12, 2009.
Margot Hanley, Apoorv Khandelwal, Hadar Averbuch-Elor, Noah Snavely, and Helen Nissenbaum. An ethical highlighter for people-centric dataset creation, 2020.
Yusniel Hidalgo-Delgadee Reina Estrada-Nelson, Bin Xu, Boris Villazon-Terrazas, Amed Leiva- Mederos, and Andrés Tello. Methodological guidelines for publishing library data as linked data. In 2017 International Conference Information Systems and Computer Science (INCISCOS), pp. 241-246, 2017. doi: 10.1109/ 2017.17.
Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The dataset nu- trition label: A framework to drive higher data quality standards. arXiv preprint 805.:7180..03677, 2018.
HuggingFace. Huggingface dataset card guidebook, 2021. URL UR/https///uugggggfce..co docs/ docs/datasetss. datasst card Accessed: 2023-05-23.
HuggingFace. Model card user studies. http:://hugginnaaceccoo//oo
model userr----e---- studies 2022.
Wiebke Toussaint Hutiri, Lauriane Gorce, and Aaron Yi Ding. Design guidelines for inclusive speaker verification evaluation datasets, 2022.
Matthew Hutson. Artificial intelligence faces reproducibility crisis, 2018
IBM. Overcome obstacles get to ai scale. https://www. 1.mm httpcccmddwwlccccecc ccas/
VEMPEQlN 2020.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias machine learning. In International Conference Artificial Intelligence and Statistics, pp. 702-712. PMLR, 2020.
Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster. Reduced, reused and recy-
cled: The life of dataset in machine learning research. In Thirty-fifth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
https::/cpenneeiiee net forum? 2
TK Koo and MY Li. guideline of selecting and reporting intraclass correlation coefficients for reliability research. chiropr med. 2016; 15 (2): 155-63, 2000.
Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho. L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669-677 2022.
Khyati Mahajan and Samira Shaikh. On the need for thoughtful data collection for multi-party dialogue: A survey of available corpora and collection methods. In Haizhou Li. Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefs, Yan Wu, and Junyi Jessy Li (eds.), Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 338-352, Singapore and Online, July 2021. Association for Computational L Linguisties. URL https: //a///aaaatttt logy. www..oog/200000 si gdi aidialll al-1.33
11

---

Published as a conference paper at ICLR 2024
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timmit Gebru. Model cards for model reporting. In Proceedings of the conference faimess, accountabillity, and transparency, pp. 220 (200222,,00 2019.
Elaine O Nsoesie and Sandro Galea. Towards better Data Science address racial bias and health equity. PNAS Nexus, I 1()), 07 2022. ISSN 2752-6542. doi: 10. 1tt.////aasneeusspgaaaaa 120. URL pgac120.
https://doi. htg.:/oo..rggg///oo 1111111111911000 http:///..../////aas/////a/aaaa.
Orestis Papakyriakopoulos, Anna Seo Gyeong Choi, William Thong, Dora Zhao, Jerone An- drews, Rebecca Bourke, Alice Xiang, and Allison Koenecke. Augmented datasheets for speech datasets and ethical decision-making. In 2023 ACM Conference on Fairness, Ac- countabilityy, and Transparency. ACM, jum 2023. doi: 10. 10.114//599503335540099 URL 3594444
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and in machine learning research. Patterns, 1001)0 nov 2021. doi: 10.1016/j wttt://..ater..21..1.00 20211...021.000333.. URL https://dol
2021. 100336
Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible ai. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1776-1826, 2022.
Victoria Stodden, Jennifer Seiler, and Zhaokun Ma. An empirical analysis journal policy effective- ness for computational reproducibility. Proceedings of the National Academy Sciences, 115(11): 2584-2589, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Anca Maria Tache, Mihaela Gaman, and Radu Tudor Ionescu. Clustering word embeddings with self-organizing maps. application on laroseda a large romanian sentiment data set. ArXiv, 2021.
Boris Tilazón-Teerazas Luis M Vilches- Blazzeez, Oscar Corcho, and Asunción Góme7-Pérez. Methodological guidelines for publishing government linked data. Linking government data, pp. 27-49, 2011.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai- Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias deep image representations. Proceedings of the IEEE/CVF International Conference Computer Vision pp. 5310-5319, 2019.
12

---

Published as a conference paper at ICLR 2024
A ILLUSTRATIONS FOR DATASET CARDS SUGGESTED BY HUGGING FACE COMMUNITY
a
Table of Contents
b
Dataont Supcr_slae
Bilee Creder Cuide
Lniung Contra
Canknis
Sipponet
Laagugge
chenl (TOM0) Convert dehoee stam froe bele
Dabior Srumter
Dats mntooa
Datarit Card "super,
Cate leids
Conuton
Source Deta
lios Duts Collecion for and foom on
Aupporid Tasla undderra
the sounce ling produces?
INtional and burt bec Mormutton
Consurte ofter's Datte
Soic Imppct of Crssies
Tilosedes Flroes
Plowe
inberaation
Tathoel Duriioo
Saat
Loand ng infermationn
Ciarton matdon
Ditsoee:: 3 orgl maur-eaarrrry Or
malx renssenmuyy TEOM End
Caliurno Vetival teacr with Higgl--lann 50300
Detuust Deacrition
d
Dutesets:
HuggingfuceeM4 cx4 werd-sytthtteettstteeee 9
warrsagy sigdl Chllnne
Unmer higuinece opeccal
enpastory
Detasst card
Files and seniions
Communiiy
Uedetoen
P nalki
DP4 synthctic assing READVE.nd
make of MartINA
bamset Summary
Kas IIESTME
Usdale FEAOre.
20uual vas supored io be dasill fardon but, cango ase dituen
Longage Erggages
datuctt be testing muttinoda tertiinage medel
Citatiin rebornati
The our sprronesplis ard 180, snague "ape popoutt 5900 unogue"
Minoort troeec Soarls "Eitertigg que qpeniiae eng "teecciee of liu Neus th he's dree toulysk taraaaaa winking Dnverementa 50002817 Lads Neww in Computel Sodeio, vol LSGA Softige
The eesoui onee ooe nwwnesentinggt the come 10unig monosee thine uentu
detoult spiil 200 Lous.
Contrbuttons
The full of datasot creatio is documentzd insie aa
Thenka to rig Ebeatro 185 adding th Caloes
Figure S1: Illustration of Adherence to Community-Endorse Dataset Card. (a) Community- Endorsed Dataset Card Struture: Hugging Face community provides suggested dataset card structure, which contains five main sections: Dataset Description, Dataset Structure, Dataset Creation, Considerations for Using the Data, and Additional Information. (b) Example of a Dataset Card Conforming to the Community Guidelines: A dataset card is considered to conform to the community guidelines when includes the five main sections outlined in the community guidelines, with the corresponding content provided for each section. (c) Example of Dataset Cards Not Following Community Guidelines (1): A dataset card is considered con-cooforming if it omits any of the five main sections provided in the suggested dataset card structure. (d) Example of Dataset Cards Not Following Community Guidelines (2): This dataset card contains only a few words and does not follow the structure at all.
13

---

Published as a conference paper at ICLR 2024
B METHOD
B.1 ACCESSING AND PARSING DATASET CARDS
In this work, we analyze datasets hosted on Hugging Face, popular platform that provides wealth of tools and resources for Al developers. One of its key features is the Hugging Face Hub API, which grants access
large library of pre-trained models and datasets for various tasks. With this API, we obtained all 24,065 datasets hosted on the Hub as of March 16th, 2023.
Dataset cards are Markdown files that serve as the README for dataset repository. They provide information about the dataset and are displayed the dataset's 's homepage. We downloaded all dataset repositories hosted on Hugging Face and extracted its README file to get the dataset cards. For further analysis of the documentation content, we utilized the Python package mistune (https) 7/mistune. re readt hedocs. wt/ee/////iee het///nnaaat/////e/aa/aa to parse the README file and extract the intended content. The structure of dataset cards typically consists of five sections: Dataset Description, Dataset Structure, Dataset Creation, Additional Information, and Considerations for Using the Data, as recommended by Hugging Face community. Examples of dataset cards, shown in Fig. ST illustrate the essential components and information provided by dataset cards. We identified and extracted different types of sections through parsing and word matching of the section heading. A significant 84% of the section titles in the 7,433 dataset cards matched one of the 27 titles suggested by the HuggingFace community using the exact keyword matching. This strong alignment underscores the effectiveness of exact keyword matching as an analytical tool.
B.2 HUMAN-ANNOTATE, DATASET CARD EVALUATION METHODOLOGGY AND CRITERIA
We conducted an evaluation on a sample of 150 dataset cards from a total of 7,433. The assessment involved five human annotators evaluate the dataset cards, who are PhI) students with solid background in AI fields such as NLP, Computer Vision, Human-AI, ML, and Data Science. Their extensive experience with datasets ensured deep understanding of dataset documentation. To confirm the reliability of our evaluation, we randomly sampled 30 dataset cards for the annotators to assess and achieved Intraclass Correlation Coefficient (ICC) of 0.76, which is considered good agreement (Koo & Li 2000). This high level of agreement, combined with the annotators' deep expertise AI research, substantially reinforces the trustworthiness of the annotation results. We focused on seven key aspects of the dataset cards drawing from prior research in dataset documentation and the Hugging Face community-endore dataset card:
Aspect
Description
Structural Organization
How well is the documentation structured with headings, sections, or subsections?
Content Comprehensiveness
How comprehensive is the information provided in the documentation?
Dataset Description
How effectively does the documentation describe the dataset?
Dataset Preprocessing
How well does the documentation describe any preprocessing steps applied the data?
Usage Guidance
How well does the documentation offer guidance on using the dataset?
Additional Information
How well does the documentation provide extra details such citations and references?
Table S1: Descriptions of Evaluation Aspects
14

---

Published as a conference paper at ICLR 2024
Each aspect received score scale from 0 to 5, with the following score metrics:
Score
Description
Comment
5
Exceptionally comprehensive and effective
Covers subsections in detail
4
Very good and thorough
Includes many subsections comprehensively
3
Moderately satisfactory
Covers some subsections adequately
2
Insufficient
Provides a basic, general overview
1
Poor and inadequate
Offers minimal, vague content
0
Absent
Lacks relevant content
Table S2: Metries of the Scores
C ADDITIONAL ANALYSIS OF Usage SECTION
Among 7,433 dataset cards, there are 567 dataset cards uploaded by 52 distinct practitioners that contain a Usage section, instructing how the dataset through text and codes. A specific example of Usage section is from aa4bhaara/nsamaaddnn which has 469 downloads and has a Usage section to instruct how the dataset (Fig. (S2).
Usage
You should have the 'diataset'' packages installed to be able to use tracket: HuggingFace datasets repository. Please following command install pip:
pip install datasets
To use the dataset, please use:
from datasets importt load dataset
hiner loed_dataset( "aidbhaaatt/// /ntastaadnnnnnn
Figure $2: Example of Usage Section
Intuitively, Usage section could give users quick instructions on how to use the dataset, which could make the dataset more accessible, transparent, and reproducible. To verify this intuition, we conduct an experiment to quantify how the Usage section will affect the dataset's popularity.
In our experiment, we trained a BERT (Devlin et al., 2018) Model using the content of dataset cards and their corresponding download counts. To ensure comparability, the download counts were normalized to a range of [0,1] and stratified monthly based on the dataset's creation time. This ranking system assigned rank of to the dataset with the highest downloads within given month, and a rank of 0 to the dataset with the lowest downloads.
Using the dataset card content, the trained BERT Model predicted the download counts. Subsequently, we conducted a test using 567 dataset cards that included Usage section. For this test, we deliberately removed the Usage section from the dataset cards and employed the BERT Model to predict the download counts for these modified cards. The resulting predictions are summarized in Table. S3 The average predicted score of downloads after removing the Usage section is 0.0185 lower compared to the original dataset card. This indicates decrease in the number of downloads, highlighting the negative impact of not including Usage section.
In future research, it would be valuable to further investigate the effect of adding a Usage section to the dataset cards that do not have one originally. randomized controlled trial (RCT) experiment could be conducted to assess whether the inclusion of a Usage section leads to an increase in downloads.
15

---

Published as a conference paper at ICLR 2024
Condition
Predicted Score of Downloads
With Usage Section
0.3917
Without Usage Section
0.3732
Change in Score
-0.0185
Table S3: Predicted Impact of Usage Section on Dataset Downloads. This table presents comparative analysis of predicted download scores for dataset cards, distinguishing between those that include a Usage Section and those from which it has been removed. It indicates a potential decrease download rates following the removal of the Usage Section.
D OPTIONAL METRICS FOR DATASETS
In our analysis, we employ downloads as a metric to gauge the popularity of the dataset. Numerous factors can influence the download count, including the dataset's publication date and its associated research field. Moreover, aside from dataset downloads, we can incorporate other indicators of dataset popularity, such as the count of models utilizing the datasets and the corresponding download counts.
To address the concerns of factors that might affect downloads, we expanded our dataset analysis by extracting more metadata from the Hugging Face dataset information. We collected data such as the models utilizing the corresponding dataset, the total number of downloads for these models, and the dataset's 's task domain. The primary dataset tasks recognized by Hugging Face encompass Multimodal, Computer Vision, Natural Language Processing, Audio, Tabular and Reinforcement Learning. Among the total 7, 7,433 dataset cards, 1,988 are categorized as NLP dataset cards, 198 related to computer vision, and 102 pertain multimodal datasets. We proceeded with additional analysis by employing the following metrics:
We integrated dataset downloads ("direct usage") with the downloads of models employing the dataset ("secondary usage").
2. A time range (measured in months) was selected, encompassing dataset cards created within the designated time frame and specified task domain.
3. Selected dataset cards ranked within each domain for each time range and then normalized to range of [0, 1].
By adopting this approach, we were able to compare dataset cards created the same month and task domain, assessing them based on the metries of direct and secondary usage metrics. We conducted word count analysis using this new metric and attained results consistent with our prior analysis that datasets with higher rankings tend to have longer dataset cards, as shown in Fig.
NLP Dotasot Card Contont Word Count
Computer Vision Dataset Card Content Word Count
Muttmocal Datasot Card Content Word Count
100
500
Word Count
500
100
2% 15%
25%
2%
20%
Tap A1
Tea Rank Upcated Mesicc
33%
Rank of Upsated Mctric
Rank of Upsated Mo3tic
Figure S3: Word Count Variation Based on Direct and Secondary Usage Rankings. This figure demonstrates the relationship between the length of dataset cards and their rankings in terms of direct and secondary usage. It reveals distinct pattern: dataset cards with higher rankings tend to have greater word count, suggesting a correlation with more thorough and detailed content.
The finding enables us contemplate an alternative metric option, factoring in publication time, research area, and secondary dataset usage. However, the results remain aligned with our previ- ous analysis, which solely considered download counts, highlighting the reasonableness of using download counts metrics.
16

---

Published as a conference paper at ICLR 2024
E APPLICABILITY ACROSS PLATFORMS: ADAPTING TO GITHUB
Our study demonstrates strong potential for application across various platforms. The foundational format of Hugging Face's dataset cards, essentially README files, prevalent documentation standard shared by many platforms, notably GitHub. This commonality implies that our approach to parsing and analyzing dataset cards can be readily adapted for broader studies. To illustrate, we present an example of how our analysis methodology can be effectively applied to GitHub, widely recognized open-source platform for data and code sharing.
Our expanded analysis involved sourcing datasets from a GitHub repository of Papers With Code We chose repositories linked to dataset-relevant papers and processed their README files using the pipeline proposed in our paper on Hugging Face dataset card analysis. This exploration revealed a more varied structure in GitHub's 's dataset cards. For example, 57% of the section titles on GitHub are unique, compared to just 3% Hugging Face. Due to their specificity, we excluded these unique sections and created a categorization list based on Hugging Face's 's community-endore endorsed dataset card structure, mapping GitHub's 's titles through keyword matching. This method successfully categorized 74% of GitHub's ' section titles.
As shown in Table. S4 analysis reveals that both platforms excel in Dataset Description and Additional Information sections but underperform in Dataser Creation and Considerations for Using the Data, underscoring points raised in our paper. notable difference is GitHub's s lower emphasis on Dataset Structure, highlighting the potentially positive impact of Hugging Face's 's community- endorsed dataset structure. Furthermore, the prevalence of Usage and Experiment sections on GitHub, absent in Hugging Face, highlights the practical value of these sections in promoting the usability of datasets. Adopting these sections, suggested our paper, could enrich the structure of Hugging Face's s dataset cards, making them more comprehensive and practically useful.
These results indicate our method's adaptability to other platforms and provide a benchmark for evaluating dataset documentation elsewhere. The insights from our Hugging Face study can guide the categorization and enhancement of dataset documentation across various platforms, especially in the current situation that most other platforms don't have standardized dataset card structure.
Section Type
GitHub
Hugging Face
Description
Dataset Description
0.62
0.46
Summary, leaderboard, languages, etc.
Dataset Structure
0.09
0.34
Format, fields, splits, etc.
Dataset Creation
0.08
0.15
Motivation, collection procedures, etc.
Considerations for Using the Data
0.02
0.08
Limitations, biases, disclaimers, etc.
Additional Information
0.62
0.58
Citations, acknowledgements, licensing, etc.
Experiment
0.57
Model experiments, training, evaluation on the dataset, etc.
Usage
0.38
Instructions for setup, installation, requirements, etc.
Table S4: Comparison of Fill-out Rate of Dataset Documentation on GitHub and Hugging Face. Dataset cards from both GitHub and Hugging Face perform well in the Dataset Description and Additional Information sections, but fall short in the Dataset Creation and Considerations for Using the Data sections. While GitHub places less emphasis on Dataset Structure, it shows a higher occurrence of Usage and Experiment sections.
17

---

Published as a conference paper at ICLR 2024
F ADDITIONAL FIGURES AND TABLES
Figure S4: Power Law Distribution Patterns in Dataset Usage across Task Domains. This figure illustrates the dataset usage distribution within each task domain, demonstrating a consistent power law distribution, despite the variations the number of datasets across different domains.
46.2%
Dataset Description
93.0%
34,1%
Dataset Structure
86.0%
15.4%
Dataset Creation
33.0%
7.6%
All Dalisset Cards
Considerations for Using the Data
Top 100
15.0%
58.0%
Additional Information
91.0%
0%
20%
40%
60%
80%
100%
Dalaset Card Section Filled-Out Rate (%)
Figure S5: Highly Downloaded Dataset Cards Exhibit Greater Completion across All Sections. This figure indicates that the top 100 downloaded dataset cards exhibit higher completion rate compared to all dataset cards in the sections recommended by the Hugging Face community. However, there a consistently low completion rate in the Dataset Creation and Considerations for Using the Data sections, regardless of the dataset cards' popularity.
Category
Description
Dataset Card Number
Adherence to Guidelines
Avg. Word Count
Industry organization
Companies (e.g. Hugging Face Facebook)
2,527
0.34
219
Academic organization
Universities. Research Labs (c.g. Stanford CRFM jhu-clsp)
985
0.31
427
Community
Non-profit Communities (e.g.
1,387
0.27
190
allenai bio datasstss
Industry professional
Engineers, Industry Scientists
985
0.25
256
Academic professional
Students, Postdocs, Faculty
672
0.16
180
All dataset cards
7,433 dataset cards analyzed
7,433
0.29
234
Table S5: Differences in the Practices of Dataset Documentation across Creators from Different Backgrounds. This table highlights the diverse documentation practices across creators from different backgrounds. Industry organizations, with the most creators, adhere to the guidelines best. Academics. though fewer, offer the most comprehensive documentation, while academic professionals exhibit lower guideline adherence and shorter word counts. The information about these creators gathered from their linked GitHub, Twitter, and personal websites their Hugging Face profiles.
18


Reviews:
[{"review_id": "vVO2w1mt0D", "reviewer": "Reviewer_gNxe", "summary": "The paper performs an analysis of the quality of dataset cards, a form of data documentation, for 150 datasets on Hugging Face.", "questions": "are the values given in the human eval averaged over the 5 annotators (each saw every dataset card?). If so, it would be interesting to see some inter-rater (dis)agreement, and some qualitative demonstration of e.g. which cards were high-agreement sort of gold standard, high agreement bad, and low agreement. \n\nWhat guidance were the raters given? What level of expertise did they have?\n\nThe correlation with content comprehensiveness in the human eval is not super high (~40%), and the rest are all quite low. It would be interesting to compare this with a qualitative description from the annotators of what they found made a better or worse card -- are there factors not detailed here they think it would be more informative to consider?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The motivation and context of the work are clearly explained. Dataset work is an important and undervalued aspect of machine learning research and this paper tackles an important topic given the prolifieration of ML datasets and platforms. The analysis and rationale are mostly presented clearly, such that it could easily be reproduced for another platform. The key insights are clearly highlighted, and they are interesting; the discussion of results is often insightful. The figures are beautiful and the captions well-done and self-contained (this is surprisingly rare, well done!). The discussion, limitations, and significance are well written. The metadata gathered (as far as I can tell) are formatted in a way that will enable future research in this direction.", "weaknesses": "Dataset curation, a very relevant domain, is not covered at all in related works. In general the related work lacks earlier works and perspective, e.g. from library science, which could have informed the analysis in the paper in some interesting ways. \nThe human eval is not given much space compared to the other aspects and I think it could have provided a lot more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RsFTtcMo3r", "reviewer": "Reviewer_eSEd", "summary": "Authors conduct an analysis on a large array of Hugging face datasets, and their associated documentation. Their aim is to better understand what results in the popularity of a particular datasets, and perhaps standardize the documentation procedure of datasets", "questions": "- The correlation between documentation length and popularity is not causal\n   - I’d emphasize on the quality of the documentation and what points these “popular cards” address in the written documentation there may be a more granular set of points mentioned in each that results in their popularity\n        - Data usability looks to have been a common key topic \n- It’s not clear how the power law with respect to data usage is useful\n   - Power-law is quite pervasive -- for example in “popularity”\n   - This power-law may simply represent the underlying distribution of people interested in deep learning topics/domains\n      - E.g. text (NLP) versus mortgage tabular data (Finance)\n- There’s a key component in the dataset documentation that is missing e.g. how the data is processed – perhaps this is covered in data usability?\n   - By having a raw data card with associated derivatives simply being a modified transformation of the raw could drastically help with reproducibility\n   - Further this way of representing allows for more comprehensive analysis on how various preprocessing steps effect benchmarks\n       - these are much more important considerations in time-series, and healthcare domains for example.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper is well written, easy to follow and digest. It is also interesting to study a large array of datasets.", "weaknesses": "- Most finding in the paper are “common sense” i.e. there always has been a high correlation with quality of documentation and usability of a particular tool/dataset\n- It’s not clear how this paper fits ICLR conference, although it the study does conduct a comprehensive overview of the hugging face dataset cards and provides an insight on what makes the datasets popular they aren’t significantly new and their insight is limited\n   - For example datasets may be simply popular because they are in a “hot” area of study\n   - Another example is the quality of the underlying dataset – it may be the case the reason one of the datasets is very popular is simply attributed to the fact that there are many studies leveraging this dataset (which in some sense is correlated with the quality of the data)\n- It seems a although there are several findings with respect to what makes a dataset popular, there isn’t a conclusive suggestion for improving dataset sharing/contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MgYwayST6", "reviewer": "Reviewer_kkaF", "summary": "This paper gives an analysis of dataset documentation practices by data creators. They focus on a Hugging Face’s data repository. The authors highlight findings such as frequently downloaded datasets are also often well documented, the fact that social concerns are some of the least documented and that users care about operational matters related to dataset - these may include, versioning, download details… etc. This work is a continuation of work championed by many ML researchers concerned with documentation and auditing to better understand ethical implications of our tools.", "questions": "- Were the authors able to find a correlation between well documented datasets and their creators? Eg, Do they come from other disciplines? Are they academic or industry researchers? Start-ups...etc\n- Is there a relationship between the features included in the dataset and whether there is going to be a limitations section? ie do datasets that contain more sensitive data likely to be well documented and include limitations sections?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is useful to provide these kinds of evaluations on new documentation processes. The ideas of how to document the different pieces of the ML process, though important, are still new and should change where there is a need. This type of study gives us a way to see where things are working and where improvements are needed. Eg. Inclusion of a uses section. Additionally, the authors started hinting on some of the relationships between what sections of the data card get filled in and the motivations of the data creators. Additionally,it is once again reminded  that we have to continue searching for better incentives to get dataset creators better document their datasets beyond describing what is in there. This work does not give us these answers and they are perhaps beyond the scope of this work but it is certainly another nudge that there is a lot more work here.", "weaknesses": "None for me. I think the paper did what it promised to do.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V44dDslHgJ", "reviewer": "Reviewer_ALk3", "summary": "The paper conducts a large scale analysis on dataset cards from HuggingFace. The paper contributes mainly 5 findings which offers a unique perspective on dataset documentations.", "questions": "See my questions in weakness section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well written with each finding clearly explained and backed by statistical data analysis\n2. Through various data analysis, the paper showed quite a few insights on the current status of dataset cards, which are often under-studied from previous works.\n3. The paper empirically verified the hypothesis proposed by employing human annotations. \n4. Through the study of HuggingFace dataset cards, the paper proposes new documentations practices that could be adopted by the industry to improve dataset documentations.", "weaknesses": "1. The paper only analyzes the documentations from HuggingFace, thus the potential for the findings to apply to other platforms remains unclear. Are there any suggestions from the authors on how to process dataset cards on other platforms?\n2. The paper uses exact keyword matching to identify corresponding subsections. Thus it's hard to know the proportion of dataset cards which covers the corresponding subsection but with different keywords. \n3. Although the paper proposes new standard that the industry should follow, are there any methods the authors think that could help achieve this goal? \n4. How are the human annotators selected? Are they machine learning domain experts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)