Add raw data files to the Hub with GitHub LFS for canonical dataset
Opened by zlucia (closed)

I'm interested in sharing the CaseHOLD dataset (https://arxiv.org/abs/2104.08671) as a canonical dataset on the HuggingFace Hub and would like to add the raw data files to the Hub with GitHub LFS, since it seems like a more sustainable long term storage solution, compared to other storage solutions available to my team. From what I can tell, this option is not immediately supported if one follows the sharing steps detailed here: [https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset](https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset), since GitHub LFS is not supported for public forks. Is there a way to request this? Thanks!

Comments:
Hi @zlucia, I would actually suggest hosting the dataset as a huggingface.co-hosted dataset.

The only difference with a "canonical"/legacy dataset is that it's nested under an organization (here `stanford` or `stanfordnlp` for instance – completely up to you) but then you can upload your data using git-lfs (unlike "canonical" datasets where we don't host the data)

Let me know if this fits your use case!

cc'ing @osanseviero @lhoestq and rest of the team 🤗

Hi @zlucia,

As @julien-c pointed out, the way to store/host raw data files in our Hub is by using what we call "community" datasets:
- either at your personal namespace: `load_dataset("zlucia/casehold")`
- or at an organization namespace: for example, if you create the organization `reglab`, then `load_dataset("reglab/casehold")`

Please note that "canonical" datasets do not normally store/host their raw data at our Hub, but in a third-party server. For "canonical" datasets, we just host the "loading script", that is, a Python script that downloads the raw data from a third-party server, creates the HuggingFace dataset from it and caches it locally.

In order to create an organization namespace in our Hub, please follow this link: https://huggingface.co/organizations/new

There are already many organizations at our Hub (complete list here: https://huggingface.co/organizations), such as:
- Stanford CRFM: https://huggingface.co/stanford-crfm
- Stanford NLP: https://huggingface.co/stanfordnlp
- Stanford CS329S: Machine Learning Systems Design: https://huggingface.co/stanford-cs329s

Also note that you in your organization namespace:
- you can add any number of members
- you can store both raw datasets and models, and those can be immediately accessed using `datasets` and `transformers`

Once you have created an organization, these are the steps to upload/host a raw dataset: 
- The no-code procedure: https://huggingface.co/docs/datasets/upload_dataset.html
- Using the command line (terminal): https://huggingface.co/docs/datasets/share.html#add-a-community-dataset

Please, feel free to ping me if you have any further questions or need help.


Ah I see, I think I was unclear whether there were benefits to uploading a canonical dataset vs. a community provided dataset. Thanks for clarifying. I'll see if we want to create an organization namespace and otherwise, will upload the dataset under my personal namespace.