Created dataset card snli.md
State: closed | Author: mcmillanmajora (CONTRIBUTOR)
Labels: name: Dataset discussion

First draft of a dataset card using the SNLI corpus as an example.

This is mostly based on the [Google Doc draft](https://docs.google.com/document/d/1dKPGP-dA2W0QoTRGfqQ5eBp0CeSsTy7g2yM8RseHtos/edit), but I added a few sections and moved some things around. 

- I moved **Who Was Involved** to follow **Language**, both because I thought the authors should be presented more towards the front and because I think it makes sense to present the speakers close to the language so it doesn't have to be repeated.

- I created a section I called **Data Characteristics** by pulling some things out of the other sections. I was thinking that this would be more about the language use in context of the specific task construction. That name isn't very descriptive though and could probably be improved.
-- Domain and language type out of **Language**. I particularly wanted to keep the Language section as simple and as abstracted from the task as possible.
-- 'How was the data collected' out of **Who Was Involved** 
-- Normalization out of **Features/Dataset Structure** 
-- I also added an annotation process section.

- I kept the **Features** section mostly the same as the Google Doc, but I renamed it **Dataset Structure** to more clearly separate it from the language use, and added some links to the documentation pages. 

- I also kept **Tasks Supported**,  **Known Limitations**, and **Licensing Information** mostly the same. Looking at it again though, maybe **Tasks Supported** should come before **Data Characteristics**?

The trickiest part about writing a dataset card for the SNLI corpus specifically is that it's built on datasets which are themselves built on datasets so I had to dig in a lot of places to find information. I think this will be easier with other datasets and once there is more uptake of dataset cards so they can just link to each other. (Maybe that needs to be an added section?)

I also made an effort not to repeat information across the sections or to refer to a previous section if the information was relevant in a later one. Is there too much repetition still?

Comments: Adding a direct link to the rendered markdown:
https://github.com/mcmillanmajora/datasets/blob/add_dataset_documentation/datasets/snli/README.md


It would be amazing if we ended up with this much information on all of our datasets :) 

I don't think there's too much repetition, everything that is in here is relevant. The main challenge will be to figure out how to structure the sheet so that all of the information can be presented without overwhelming the reader. We'll also want to have as much of it as possible in structured form so it can be easily navigated.

@mcmillanmajora for now can you remove the prompts / quoted blocks so we can see what the datasheet would look like on its own?

Would also love to hear if @sgugger has some first impressions

I removed the prompts. It's definitely a little easier to read without them!

Should we name the file `README.md` for consistency with models?

Asked @sleepinyourhat for some insights too :) 

Thank you for taking the time to look through the card and for all your comments @sleepinyourhat ! I've incorporated them in the latest update.  

Be careful to keep the ‘sa’ term in the license. It’s something we
inherited from the Flickr captions.

On Thu, Oct 1, 2020 at 10:09 AM Julien Chaumond <notifications@github.com>
wrote:

> *@julien-c* commented on this pull request.
> ------------------------------
>
> In datasets/snli/README.md
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23discussion-5Fr498273172&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=WbEkKXCbL6j5Ui3sox_WqvzrbShbJn2WW-51SENL2ZQ&e=>
> :
>
> > +---
> +language:
> +- en
> +task:
> +- text-classification
> +purpose:
> +- NLI
> +size:
> +- ">100k"
> +language producers:
> +- crowdsourced
> +annotation:
> +- crowdsourced
> +tags:
> +- extended-from-other-datasets
> +license: "CC BY-SA 4.0"
>
> ⬇️ Suggested change
>
> -license: "CC BY-SA 4.0"
> +license: cc-by-4.0
>
> For models (documented at
> https://huggingface.co/docs#what-metadata-can-i-add-to-my-model-card
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__huggingface.co_docs-23what-2Dmetadata-2Dcan-2Di-2Dadd-2Dto-2Dmy-2Dmodel-2Dcard&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=ck3x8c_ujrwKReDTSGuWWgD9W6REHEPbZaO7S4GFRd4&e=>)
> we use the License keywords listed by GitHub at
> https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/licensing-a-repository#searching-github-by-license-type
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.github.com_en_free-2Dpro-2Dteam-40latest_github_creating-2Dcloning-2Dand-2Darchiving-2Drepositories_licensing-2Da-2Drepository-23searching-2Dgithub-2Dby-2Dlicense-2Dtype&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=dWBP-ZvtMErD-egoBiBTCKA4500mjDXVSk03oW1g16U&e=>
>
> (Hopefully we'll plug some sort of form validation for users at some point)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23pullrequestreview-2D500386385&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=HU2Hwi7HH9W2NtMoCIiQlhXxxEULLi8L9gnWU5PBAPY&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWL63W2LB7SBICA2GMTSISEPZANCNFSM4RWKAZRA&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=086__lKQLxTanHfjE8kOIpaJbaWPzBB9gGIt_prWeH8&e=>
> .
>


@sleepinyourhat You're right, wrong copy/paste

Question: Where does this standard come from? It looks similar to both
'Data Statements' and 'Datasheets for Datasets', but it doesn't look quite
like either.

On Mon, Oct 12, 2020 at 4:27 PM Yacine Jernite <notifications@github.com>
wrote:

> Merged #663
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=s4_X-BSEnTKgGg9rPLBt3cyVptyMX_iWD5Ql3UMBi-I&e=>
> into master.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23event-2D3868180429&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=elcM4umqReQfIrgHhpey9W_wPaq5QRgq7xNlubM47QI&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWJVGQRCR4OTTV27VTTSKNRBXANCNFSM4RWKAZRA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=NB6nEROnTPgwNyF3ZklOmHnvP7kOkOm7sEa740KbVCs&e=>
> .
>


@sleepinyourhat The schema is definitely drawing from Data Statements and Datasheets for Datasets but we also wanted to include some more general information to introduce the dataset to new users. If you have any suggestions for changes to the schema itself, please let us know!