My experience with uploading a dataset on HuggingFace’s dataset-hub | by Kartik Godawat | Towards Data Science

HuggingFace’s datasets library is a one-liner python library to download and preprocess datasets from HuggingFace dataset hub. The library, as of now, contains around 1,000 publicly-available datasets.
In this post, I’ll share my experience in uploading and mantaining a dataset on the dataset-hub. The following meme summarizes the intent behind using datasets library:
With the help and guidance from folks at HuggingFace, I was able to download the metadata of information available on the model-hub(where, similar to datasets, HuggingFace hosts 10,000+ publicly available models) into a csv file. I then began the process to upload it as a dataset on dataset-hub.
$pip install datasets
There are two ways of adding a public dataset:
Community-provided: Dataset is hosted on dataset hub. It’s unverified and identified under a namespace or organization, just like a GitHub repo.
Canonical: Dataset is added directly to the datasets repo by opening a PR(Pull Request) to the repo. Usually, data isn’t hosted and one has to go through PR merge process.
Since I wanted to host the data, along with the pre-processing script(referred to as “dataset loading script” by the library), I chose to upload the dataset as a community-provided dataset.
There are two primary requirements for converting an existing dataset of csv, json, xml or any other format into a dataset:
Dataset loading script
Dataset metadata
There’s good documentation available on creating such a script. I, however preferred copy-pasting by using the scripts of pre-existing datasets similar to mine, which are presnt in the datasets library. eg: If my dataset is of csv type, I’ll start with a similar csv type script and modify it according to my needs.
The script primarily needs three components to be defined:
Information about column and data types (referred as Features)
URLs to download data from ( or local files ) and specifying train/test/validation splits
Yield 1 row of data, utilizing split and features
1. _info: The headers of the csv field need to be defined with their datatype. I ended up using string , int32 and large_string. Description of supported datatypes is available in features.
def _info(self):        return datasets.DatasetInfo(            description=_DESCRIPTION,            features=datasets.Features(                {                    "modelId": datasets.Value("string"),                    "lastModified": datasets.Value("string"),                    "tags": datasets.features.Sequence(datasets.Value("string")),                    "pipeline_tag": datasets.Value("string"),                    "files": datasets.features.Sequence(datasets.Value("string")),                    "publishedBy": datasets.Value("string"),                    "downloads_last_month": datasets.Value("int32"),                    "library": datasets.Value("string"),                    "modelCard": datasets.Value("large_string"),                }            ),            homepage=_HOMEPAGE,            license=_LICENSE,            citation=_CITATION,        )
The columns “tags” and “files” are arrays, as they can be multiple which are supported by library by dataset.features.Sequence.
2. URLs / local files:
Next step is to then define a URL, which in my case is a local file. Since the data is only exploratory and doesn’t have any target labels(not specifically meant for training anything), there wasn’t any need of test split. Hence, only train dataset split will do.
_URL = "huggingface-modelhub.csv"...def _split_generators(self, dl_manager):    """Returns SplitGenerators."""    data_file = dl_manager.download_and_extract(_URL)    return [        datasets.SplitGenerator(            name=datasets.Split.TRAIN,            gen_kwargs={                "filepath": data_file,            },),]
3. Yield a row:
The next step is to yield a single row of data. At runtime, appropriate generator (defined above) will pick the datasource from URL or local file and use it to generate a row. Here, since the data format was csv , we can use python’s inbuilt csv module and it’s function csv.reader to read the data from the file.
def _generate_examples(self, filepath):    """Yields examples."""    with open(filepath, encoding="utf-8") as f:        reader = csv.reader(f)        for id_, row in enumerate(reader):            if id_ == 0:                continue            yield id_, {                "modelId": row[0],                "lastModified": row[1],                "tags": ast.literal_eval(row[2]),                "pipeline_tag": row[3],                "files": ast.literal_eval(row[4]),                "publishedBy": row[5],                "downloads_last_month": float(row[6]) if row[6] else 0,                "library": row[7],                "modelCard": row[8]            }
ast.literal_eval is a handy function to parse the array present in a string into an actual array(list).
The script could be tested like this:
>>> from datasets import load_dataset>>> dataset = load_dataset('PATH/TO/MY/SCRIPT.py')>>> dataset["train"] # To access train generator>>> dataset["train"][0] #Access elements in dataset
With the script in place, we move on to adding the dataset metadata and getting the dataset ready for publishing.
The documentation explains in detail on how to prepare dataset for sharing. To add the metadata, there is a helper command available through datasets-cli , which gets installed when we install datasetslibrary.
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
Running the above command generates a file dataset_infos.json , which contains the metadata like dataset size, checksum etc.
Huggingface uses git and git-lfs behind the scenes to manage the dataset as a respository. To start, we need to create a new repository.
Once, the repository is ready, the standard git practices apply. i.e. from your project directory run:
$ git init .$ git remote add origin https://huggingface.co/datasets/<user>/<repo>$ git pull origin main
Now, we’ve synced our local machine with the repository. Next step is to add the following files and commit:
dataset files (csv): The data itself
dataset loading script: Loader for the data
dataset metadata: Metadata like size, citations etc
But there’s a catch! Conventional git systems aren’t suitable for handling large files. That is managed through git-lfs(Large File Storage). We don’t need to go deeper into how it works, we can simply run the following commands in order to be able to push large files to the repo:
$ git lfs install$ git lfs track huggingface-modelhub.csv$ git add dataset_infos.json huggingface-modelhub.csv huggingface-modelhub.py$ git commit -m "Commit message"$ git push origin main
The documentation covers lfs in detail as well. Since this is just a git repo, any other files like README could be committed as well. Dataset-hub UI also provides a quick way to update the README (referred to as datasetCard)
And that’s it. The dataset should be uploaded to dataset-hub. To access it, run:
>>> dataset = load_dataset("<user>/<repo>")
Since the repo of dataset is controllable using git, I believe it might be a good practice to first commit to a dev branch, test it completely and then perform the merge to main . That will help a lot in “accidentally” ruining an already existing and a stable, working dataset. This is very much analogous to standard git based software-release philosophy.
Another advantage is that dataset version also doesn’t mandatorily need to be serial( 1, 1.1, 1.2 etc). Different branches could hold different versions of the dataset.
>>> dataset = load_dataset("<user>/<repo>",>>>   script_version="dev")  # tag name, branch name, or commit hash
HuggingFace forum is very actively mantained and the team+community are very helpful and supportive. I personally found it to be more active and quick-responsive than stackoverflow. One could also open issues directly on github as well, but I prefer forums as they’re a bit informal.
No question is a dumb question! Never be afraid to ask.
That’s all folks!
I hope you enjoyed the post and found the datasets library to be useful.The dataset I uploaded with the loading script could be viewed here and used as a reference for csv formats.
Please share your views in comments. I’m also up for a discussion on twitter.
Have a great day. :)