DOI: 10.5281/zenodo.15187256
title: Replication package for the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models"

description: This replication package contains datasets and scripts related to the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models"

root directory



statistics.r: R script used to compute the correlation between usage and downloads, and the RQ1/RQ2 inter-rater agreements


scripts/

Directory containing all the scripts used to collect and process data. For further details, see the README file inside the script directory.

further-analyses/

Directory containing scripts and data used for further analyses. For further details, see the README file inside the directory.

first-snaphot

Contains data related to the first snapshot (ICPC 2024 study)



modelsInfo.zip: zip file containing all the downloaded model cards (in JSON format)


Dataset

Dataset/DatasetHFmodellist.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipelinetag,pipelinecategory,License,licensemodelpermissivity Dataset/DatasetGitHubprjlistTransformers.txt: list of GitHub projects using the transformers library Dataset/DatasetGitHubprjlistDiffusers.txt: list of GitHub projects using the diffuserslibrary Dataset/DatasetGitHubprjfrompretrainedTransformers.txt: list of GitHub projects using the "frompretrained" transformers library Dataset/DatasetGitHubprjfrompretrainedDiffusers.txt: list of GitHub projects using the "frompretrained" diffusers library Dataset/DatasetGitHubprjmodelusedTransformers.csv: contains usage pairs: project, model for transformers library Dataset/DatasetGitHubprjmodelusedDiffusers.csv: contains usage pairs: project, model for diffusers library Dataset/DatasetIntersectedModels.csv : contains the models shared between the first and second snapshot for category Dataset/modelsReadme: contains the model cards belonging to the sample size Dataset/projectswith5ormorestars.csv: contains the projects with numStars major of 5 Dataset/projectsstars_summary.csv: contains the number of total projects with numStars RQ1

RQ1/RQ1datasetlistHF.txt: list of HF datasets RQ1/RQ1datasetTags.txt: list of models declaring the dataset tag RQ1/RQ1modelDataset.csv : list of models declaring the dataset tag with their respective datasets RQ1/RQ1datasetSample.csv: sample set of models used for the manual analysis of datasets RQ2

RQ2/RQ2biasclassification_sheet.csv: results of the manual labeling RQ3

RQ3/RQ3LicenseModels.csv: model license list,categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3prjmodellicensepermissivityTransformersDiffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffuserslibrary RQ3/RQ3prjmodellicensepermissivityTransformersDiffusersStarmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5 RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffuserslibrary in terms of licenses RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusersStarmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5 RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5 RQ4

RQ4/RQ4TermsofUseSnapshot1.csv: results of the manual labeling related to terms of use for the first snapshot RQ4/RQ4TermsofUseSnapshot2.csv: results of the manual labeling related to terms of use for the second snapshot

Dataset



Dataset/Dataset_HF-models-list.csv: list of HF models analyzed

Dataset/Dataset_github-prj-list.txt: list of GitHub projects using the transformers library

Dataset/Dataset_github-Prj_model-Used.csv: contains usage pairs: project, model

Dataset/Dataset_prj-num-models-reused.csv: number of models used by each GitHub project

Dataset/Dataset_model-download_num-prj_correlation.csv contains, for each model used by GitHub projects: the name, the task, the number of reusing projects, and the number of downloads


RQ1



RQ1/RQ1_dataset-list.txt: list of HF datasets

RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets

RQ1/RQ1_analyzeDatasetTags.py: Python script to analyze model tags for the presence of datasets. it requires to unzip the modelsInfo.zip in a directory with the same name (modelsInfo) at the root of the replication package folder. Produces the output to stdout. To redirect in a file fo be analyzed by the RQ2/countDataset.py script

RQ1/RQ1_countDataset.py: given the output of RQ2/analyzeDatasetTags.py (passed as argument) produces, for each model, a list of Booleans indicating whether (i) the model only declares HF datasets, (ii) the model only declares external datasets, (iii) the model declares both, and (iv) the model is part of the sample for the manual analysis

RQ1/RQ1_datasetTags.csv: output of RQ2/analyzeDatasetTags.py

RQ1/RQ1_dataset_usage_count.csv: output of RQ2/countDataset.py


RQ2



RQ2/tableBias.pdf: table detailing the number of occurrences of different types of bias by model Task

RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling

RQ2/RQ2_isBiased.csv: file to compute the inter-rater agreement of whether or not a model documents Bias

RQ2/RQ2_biasAgrLabels.csv: file to compute the inter-rater agreement related to bias categories

RQ2/RQ2_final_bias_categories_with_levels.csv: for each model in the sample, this file lists (i) the bias leaf category, (ii) the first-level category, and (iii) the intermediate category


RQ3



RQ3/RQ3_LicenseValidation.csv: manual validation of a sample of licenses

RQ3/RQ3_{NETWORK-RESTRICTIVE|RESTRICTIVE|WEAK-RESTRICTIVE|PERMISSIVE}-license-list.txt: lists of licenses with different permissiveness

RQ3/RQ3_prjs_license.csv: for each project linked to models, among other fields it indicates the license tag and name

RQ3/RQ3_models_license.csv: for each model, indicates among other pieces of info, whether the model has a license, and if yes what kind of license

RQ3/RQ3_model-prj-license_contingency_table.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows)

RQ3/RQ3_models_prjs_licenses_with_type.csv: pairs project-model, with their respective licenses and permissiveness level

RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types:



analysis\_by\_type.py: script to perform the analysis

list.csv: the list of licenses with their permissiveness

licensebytype.csv: licenses of all analyzed models




RQ4



RQ4/RQ4_Terms_of_Use_Snapshot1.csv: results of the manual labeling related to terms of use for the first snapshot


second-snaphot

Contains data related to the second snapshot.

Dataset



Dataset/Dataset_HF_model_list.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipeline_tag,pipeline_category,License,license_model_permissivity

Dataset/Dataset_GitHub_prj_list_Transformers.txt: list of GitHub projects using the transformers library

Dataset/Dataset_GitHub_prj_list_Diffusers.txt: list of GitHub projects using the diffusers library

Dataset/Dataset_GitHub_prj_frompretrained_Transformers.txt: list of GitHub projects using the "from_pretrained" transformers library

Dataset/Dataset_GitHub_prj_frompretrained_Diffusers.txt: list of GitHub projects using the "from_pretrained" diffusers library

Dataset/Dataset_GitHub_prj_model_used_Transformers.csv: contains usage pairs: project, model for transformerslibrary

Dataset/Dataset_GitHub_prj_model_used_Diffusers.csv: contains usage pairs: project, model for diffusers library

Dataset/Dataset_IntersectedModels.csv : contains the models shared between the first and second snapshot for category

Dataset/modelsReadme: contains the model cards belonging to the sample size

Dataset/projects_with_5_or_more_stars.csv: contains the projects with numStars major of 5

Dataset/projects_stars_summary.csv: contains the number of total projects with numStars


RQ1



RQ1/RQ1_dataset_list_HF.txt: list of HF datasets

RQ1/RQ1_datasetTags.txt: list of models declaring the dataset tag

RQ1/RQ1_modelDataset.csv : list of models declaring the dataset tag with their respective datasets

RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets


RQ2



RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling


RQ3



RQ3/RQ3_License_Models.csv: model license list,categorized by permissiveness, with the respective number of occurrences

RQ3/RQ3_License_prjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences

RQ3/RQ3_License_prjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences

RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library

RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers_Starmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5

RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness

RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses

RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5

RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5

RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types:



analysis\_by\_type.py: script to perform the analysis

list.csv: the list of licenses with their permissiveness

licensebytype.csv: licenses of all analyzed models




RQ4



RQ4/RQ4_Terms_of_Use_Snapshot2.csv: results of the manual labeling related to terms of use for the second snapshot