Add OSCAR dataset
State: closed | Author: pjox (CONTRIBUTOR)

I don't know if tests pass, when I run them it tries to download the whole corpus which is around 3.5TB compressed and I don't have that kind of space. I'll really need some help with it 😅 

Thanks!

Comments: @pjox I think the tests don't pass because you haven't provided any dummy data (`dummy_data.zip`).

 

> @pjox I think the tests don't pass because you haven't provided any dummy data (`dummy_data.zip`).

But can I do the dummy data without running `python nlp-cli test datasets/<your-dataset-folder> --save_infos --all_configs` first? 🤔 

You make a good point! Do you know how big is it uncompressed?

Between 7T and 9T I think.

Hi ! I've been busy but I plan to compute the missing metadata soon !
Looking forward to be able to load a memory mapped version of OSCAR :) 

> Hi ! I've been busy but I plan to compute the missing metadata soon !
> Looking forward to be able to load a memory mapped version of OSCAR :)

Amazing! Thanks! 😄 

Hi there, are there any plans to complete this issue soon? I'm planning to use this dataset on a project. Let me know if there's anything I can do to help to finish this 🤗 

Yes it will be added soon :) 
Recently the OSCAR data files were moved to another host. We just need to update the script and compute the dataset_infos.json (it will probably take a few days).

@lhoestq I've seen in oscar.py that it isn't a dataset script with manual download way. Is that correct? 
Some time ago, @pjox had some troubles with his servers providing that dataset 'cause it's really huge. Providing it on an automatic download way seems to be a little bit dangerous for me 😄 

Now thanks to @pjox 's help OSCAR is hosted on HF's S3, which is probably more robust that the previous servers :)

Also small update on my side:
I launched the computation of the dataset_infos.json file, it will take a few days.

Now it seems to be a good plan for me 🤗 

But is there a plan to provide the OSCAR's unshuffled version too?

The one we have on S3 is currently the unshuffled version

I've thought that you won't provide the unshuffled version 'cause this comment on oscar.py:

`# TODO(oscar): Implement unshuffled OSCAR`



That TODO is normal, I haven't touched the python script in months (I haven't had the time, sorry), but I guess @lhoestq fixed the paths if he's already working on the metadata. In any case from now on, only the unshuffled versions of OSCAR will be distributed through the hf/datasets library as in any case it is the version most people use to train language models.

If for any reason, you need the shuffled version it will always be available on the [OSCAR website](https://oscar-corpus.com).

Also future versions of OSCAR will be unshuffled only.

Should we close this PR now that the other one was merged?

Sure.
Closing since #1694 is merged

@lhoestq just a little detail, is the Oscar version that HF offers the same one that was available on INRIA? By that I mean, have you done any further filtering or removing of data inside it? Thanks a lot! 

Hello @jchwenger, this is exactly the same (unshuffled) version that's available at Inria. Sadly no further filtering is provided, but after the latest OSCAR audit (https://arxiv.org/abs/2103.12028) we're already working on future versions of OSCAR that will be "filtered" and that will be available on the OSCAR website and hopefully here as well.

@pjox brilliant, in my case I was hoping it would be unfiltered, good news!