add flue dataset
State: closed | Author: mariamabarham (CONTRIBUTOR)

This PR add the Flue dataset as requested in this issue #223  . @lbourdois made  a detailed description in that issue.


Comments: The dummy data file was wrong. I only fixed it for the book config. Even though the tests are all green here, this should also be fixed for all other configs. Could you take a look there @mariamabarham ? 

Hi @mariamabarham 

FLUE can indeed become a very interesting benchmark for french NLP !
Unfortunately, it seems that we've both been working on adding it to the repo...
I was going to open a pull request before I came across yours.
I didn't want to open a duplicate, that's why I'm commenting here (I hope it's not rude).

When I look at your code there is one issue that jump out at me: for both `vsd` and `nsd`, the labels are missing. I believe this is more a data issue, as they were not kept in the cleaned dataframes of #223. I think the *word sense disambiguation* task was a bit misunderstood. 

Maybe you should directly use the data provided by FLUE for these ?

Hi @TheophileBlard thanks for pointing this out. I will give a look at it or maybe if you already done it you can update this PR. Also I haven't added yet the parsing datasets, I submited a request to get access to them. If you already have them, you can also add them.

Hi,

As @TheophileBlard pointed out, the labels for the vsd and nsd stains are missing.

For the wsd, it is my mistake, I added the files containing the labels on the drive.
There is still the join to do between the files that I didn't have time to do. It can be done after importing the two files, however if you wish to have a single dataframe already containing all the information, I could do it but only when I have free time because I have a lot of work at the moment at INSERM with the covid.

For the nsd, I've downloaded the files at https://zenodo.org/record/3549806, and if you do the same you'll see that they don't contain any labels.
In the files, you can see that some words have a WN code. I don't know what it corresponds to. On the FLUE github, they say to use the disambiguate tool (https://github.com/getalp/disambiguate) but I don't understand what he's doing.

@mariamabarham  for the parsing datasets, I have them in my possession. What it does that I haven't shared them is that they are licensed and you have to make a request to their creators. They give them away very easily for research purposes. For another use, you have to ask a commercial licence. All this means that if the data is freely available on your librairy, their licence and their application form are no longer of interest, which is why I did not add them.
Afterwards, maybe the authors will change their policies and decide to make the data freely available through your librairy

@mariamabarham  @lbourdois, Yea I don't think we can had the parsing datasets without asking the authors permission first. I also hope they'll change their policy.

Regarding `vsd` and `nsd`, if I understand well the task, the labels are "word senses" and the goal is to find the correct word sense for each ambiguous word. For `vsd` there is one ambiguous verb per sentence, and the labels we manually annotated with "wiktionary senses". For `nsd`, there are multiple ambiguous word per sentence, and the labels are WordNet Princeton Identifiers (hence the WN tag). This dataset was translated in french & automatically aligned.

Imo, for these 2 datasets, each example should be made of:
-  a list of string tokens (the words of the sentence)
- a list of string labels (the word senses or 'O' when the word is not ambiguous.

In fact, for `vsd` it could be even simpler, with a single string label (as there is only one ambiguous verb), + some "idx" feature to indicate the location of the ambiguous verb.

Unfortunately, I cannot update your PR as I'm not a maintainer of the project. Maybe we could work together on a fork ? Here's [mine](https://github.com/TheophileBlard/nlp/commits/flue-benchmark).


Hi

Any news about this PR ?
Because thinking back FLUE basically offers only two new datasets : those for the Word Sense Disambiguation task (vsd and nsd).

Wouldn't it be more clever to make separate PRs to add the datasets of the other tasks which are multi-lingual (and therefore can be used for other languages) ?

XNLI being already present on your library, there would only be PAWS-X (datasets and bibtex available here : https://github.com/google-research-datasets/paws/tree/master/pawsx) and the Webis-CLS-10 dataset (dataset : https://zenodo.org/record/3251672#.XvCXN-d8taQ and bibtex : https://zenodo.org/record/3251672/export/hx#.XvCXZ-d8taQ) to do.

And next for the FLUE benchmark, all you would have to do would be to use your own library by making an nlp.load_dataset() (for example nlp.load_dataset('xnli') which is already present in your library) for each of the datasets of the benchmark tasks and to keep only the 'fr' data.



Also @mariamabarham , did you get any feedback for the parsing task dataset request?
In case of refusal from the authors, there are other datasets in French to perform this task and in this case, I would open a new topic


Hi @lbourdois ,
PAWS-X is also present in the lib, it's part of `xtreme` dataset, so it can be loaded by `nlp.load_dataset('xtreme', 'PAWS-X.fr')` for the french version.
I think the parsing and the Word Sense Disambiguation task datasets are the only missing in the lib now. 
I did not get a feedback yet for the parsing dataset.


By the way, @TheophileBlard  I commented some days ago in your fork. It would be great if you can maybe open a new PR with your code or if you have a better way  to make it available to others for review.

> By the way, @TheophileBlard I commented some days ago in your fork. It would be great if you can maybe open a new PR with your code or if you have a better way to make it available to others for review.

Yea sorry, missed that! I think @lbourdois has a point, it helps no one to have the same dataset in multiple places. I will try to find some time to adapt the code of my fork and open PRs for `Webis-CLS-10` and `nsd`/`vsd`. Maybe we should group `nsd`/`vsd` together ?

Shall we close this PR then ? @mariamabarham @TheophileBlard @lbourdois