Adding Microsoft CodeXGlue Datasets
State: closed | Author: ncoop57 (NONE)

Hi there, this is a new pull request to get the CodeXGlue datasets into the awesome HF datasets lib. Most of the work has been done in this PR #997 by the awesome @madlag. However, that PR has been stale for a while now and so I spoke with @lhoestq about finishing up the final mile and so he told me to open a new PR with the final changes :smile:. 

I believe I've met all of the changes still left in the old PR to do, except for the change to the languages. I believe the READMEs should include the different programming languages used rather than just using the tag "code" as when searching for datasets, SE researchers may specifically be looking only for what type of programming language and so being able to quickly filter will be very valuable. Let me know what you think of that or if you still believe it should be the "code" tag @lhoestq.

Comments: Oh one other thing. Mentioned in the PR was that I would need to regenerate the dataset_infos.json once the camel casing was done. However, I am unsure why this is the case since there is no reference to any object names in the dataset_infos.json file.

If it needs to be reran, I can try it do it on my own machine, but I've had a memory issues with a previous dataset due to my compute constraints so I'd prefer to hopefully avoid it all together if not necessary to regenerate.

Was just reviewing the `builder_name`s of each dataset and it seems like it is already following this format:

`CodeXGlueCcCloneDetectionBigCloneBenchMain -> code_x_glue_cc_clone_detection_big_clone_bench_main` Is there a location I am missing?

> Was just reviewing the `builder_name`s of each dataset and it seems like it is already following this format:
> 
> `CodeXGlueCcCloneDetectionBigCloneBenchMain -> code_x_glue_cc_clone_detection_big_clone_bench_main` Is there a location I am missing?

If it's already in this format then it's fine thanks ! It's all good then

To fix the CI you just need to add the `encoding=` parameters to the `open()` calls

@lhoestq I think everything should be good to go besides the code styling, which seem to be due to missing or unsupported metadata tags for the READMEs, is this something I should worry about since all the other datasets seem to be failing as well?

Awesome! Just committed your changes and I will begin on adding the TOCs and filling in the content for the new sections/subsections.

Also, I see that we are having to only use the `code` tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.

> Also, I see that we are having to only use the code tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.

Yes I agree. We'll be able to reuse the tags per programming language from this PR when we allow this feature

cc @yjernite what do you think about extending our languages taxonomy to programming languages ?

Hey @lhoestq, just finalizing the READMEs and testing them against the automated test. For the non, WIN tests, it seems like there is some dependency issue that doesn't have to do with the new datasets. For the WIN tests, it looks like some of the headings are mislabeled such as "Supported Tasks and Leaderboards" -> "Supported Tasks" in the TOC you posted. Should I base my TOC on the one you posted or on the one that the test script is using? Also, it throws errors for some of the fields being empty, such as "Source Data" in the `code_x_glue_tt_text_to_text` dataset. However, I am not familiar with this dataset, so I put the `[More Information Needed]` stub, similar to the other sections I couldn't easily answer. For some of the sections like "Source Data", is this info required?

Yes you're right, it is `Supported Tasks and Leaderboards` that we need to use, sorry about that

I also noticed the same for the splits section: we have to use `Data Splits` (not Data Splits Sample Size)


Some subsections are also missing: `Initial Data Collection and Normalization`, `Who are the source language producers?`.
If you are interested you can fill those sections as well, or leave them empty for now.
This will also fix the error regarding "Source Data"

You can see the template of the readme here:
https://github.com/huggingface/datasets/blob/9d8bf36fdb861d9b2922d7c782fb58f9f542997c/templates/README.md

> > Also, I see that we are having to only use the code tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.
> 
> Yes I agree. We'll be able to reuse the tags per programming language from this PR when we allow this feature
> 
> cc @yjernite what do you think about extending our languages taxonomy to programming languages ?

Sounds good, as long as they all share a prefix! maybe `code_cpp`, `code_java`, etc. ? 

I don't think we currently have `_` in language codes/names, but also don't see what it would break *a priori*

We don't use `_` but there are some languages that use `-` though like `en-US`. Let's use `-` maybe, to match the same hierarchy pattern ?

Hi guys, I just started working on https://github.com/huggingface/datasets/pull/997 this morning and I just realized that you were finishing it... You may want to get the dataset cards from https://github.com/madlag/datasets, and maybe some code too, as I did a few things like moving _CITATION and _DESCRIPTION to globals.



I am renaming the main classes to match the dataset names, for example : CodeXGlueTcTextToCodeMain -> CodeXGlueTcTextToCode . And I am regenerating the dataset_infos.json accordingly.

Thanks for renaming the classes and updating the dataset_infos.json ! This looks all clean now :)

This PR looks all good to me :) One just needs to merge master into this branch to make sure the CI is green with the latest changes. It should also fix the current CI issues that are not related to this PR

Woot woot :rocket:! All green, looks like it is ready for showtime. Thank you both @lhoestq and especially @madlag, I think these datasets are going to be a great new addition to :hugs: datasets and I can't wait to use them in my research :nerd_face:.

Thanks @ncoop57 for you contribution! It will be really cool to see those datasets used as soon as they are released !