Add OSCAR
State: closed | Author: lhoestq (MEMBER)

Continuation of #348 
The files have been moved to S3 and only the unshuffled version is available.
Both original and deduplicated versions of each language are available.

Example of usage:
```python
from datasets import load_dataset

oscar_dedup_en = load_dataset("oscar", "unshuffled_deduplicated_en", split="train")
oscar_orig_fr = load_dataset("oscar", "unshuffled_original_fr", split="train")
```

cc @pjox @jonatasgrosman 

-------------

To make the metadata generation work in parallel I did a few changes in the `datasets-cli test` command to add the `num_proc` and `proc_rank` arguments. This way you can run multiple processes for the metadata computation.

```
datasets-cli test ./datasets/oscar --save_infos --all_configs --num_proc 4 --proc_rank 0 --clear_cache --cache_dir tmp0
```

-------------

ToDo: add the dummy_data

Comments: Hi @lhoestq, on the OSCAR dataset, the document boundaries are defined by an empty line. Are there any chances to keep this empty line or explicitly group the sentences of a document? I'm asking for this 'cause I need to know if some sentences belong to the same document on my current OSCAR dataset usage.

Indeed currently it yields one example per line and ignore the empty lines.
Maybe the best is to group them by paragraph then, and yield one example when an empty line is found.
What do you think ?

I think to group them is the best choice indeed, I actually did this on [brwac](https://github.com/huggingface/datasets/tree/master/datasets/brwac) dataset too, it's another huge textual dataset.

Ok I just launched the computation of the dataset_infos.json again by grouping lines in paragraphs.
The new _generate_examples is
```python
    def _generate_examples(self, filepaths):
        """This function returns the examples in the raw (text) form."""
        id_ = 0
        current_lines = []
        for filepath in filepaths:
            logging.info("generating examples from = %s", filepath)
            with gzip.open(filepath, "rt") as f:
                for line in f:
                    if len(line.strip()) > 0:
                        current_lines.append(line)
                    else:
                        feature = id_, {"id": id_, "text": "".join(current_lines)}
                        yield feature
                        id_ += 1
                        current_lines = []
                # last paragraph
                if current_lines:
                    feature = id_, {"id": id_, "text": "".join(current_lines)}
                    yield feature
```

Is there any chance to also keep the sentences raw (without the `"".join()`)?. This is useful if you wanna train models where one of the tasks you perform is document sentence permutation... that's my case :)

They are raw in the sense that nothing is changed from the raw file for each paragraph.
You can split sentences on new lines `\n` for example.

The first example for the unshuffled deduplicated english is going to be 
> Mtendere Village was inspired by the vision of Chief Napoleon Dzombe, which he shared with John Blanchard during his first visit to Malawi. Chief Napoleon conveyed the desperate need for a program to intervene and care for the orphans and vulnerable children (OVC) in Malawi, and John committed to help.
> Established in honor of John & Lindy’s son, Christopher Blanchard, this particular program is very dear to the Blanchard family. Dana Blanchard, or Mama Dana as she is more commonly referred to at Mtendere, lived on site during the initial development, and she returns each summer to spend the season with her Malawian family. The heart of the program is to be His hands and feet by caring for the children at Mtendere, and meeting their spiritual, physical, academic, and emotional needs.
> [...]
> 100X Development Foundation, Inc. is registered 503 (c)(3) nonprofit organization. Donations are deductable to the full extent allowable under IRS regulations.

I thought the line reader would omit the `\n` character. I can easily split the sentences as you suggested. Thanks @lhoestq! 😃 

The recomputation of the metadata finished a few days ago, I'll update the PR soon :) 

Let me know if you have comments @pjox @jonatasgrosman :) 

Otherwise we can merge it

Everything seems fine to me 😄