Add ability to load newer, cleaner version of Multi-News 
State: closed | Author: JohnGiorgi (CONTRIBUTOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

The [Multi-News dataloader points to the original version of the Multi-News dataset](https://github.com/huggingface/datasets/blob/12540dd75015678ec6019f258d811ee107439a73/datasets/multi_news/multi_news.py#L47), but this has [known errors in it](https://github.com/Alex-Fabbri/Multi-News/issues/11). There exists a [newer version which fixes some of these issues](https://drive.google.com/open?id=1jwBzXBVv8sfnFrlzPnSUBHEEAbpIUnFq).

Unfortunately I don't think you can just replace this old URL with the new one, otherwise this could lead to issues with reproducibility.

**Describe the solution you'd like**

Add a new version to the Multi-News dataloader that points to the updated dataset which has fixes for some known issues.

**Describe alternatives you've considered**

Replace the current URL to the original version to the dataset with the URL to the version with fixes.

**Additional context**

Would be happy to make a PR for this, could someone maybe point me to another dataloader that has multiple versions so I can see how this is handled in `datasets`?


Comments: Hi! Our versioning is based on Git revisions (the `revision` param in `load_dataset`), so you can just replace the old URL with the new one and open a PR :). I can also give you some pointers if needed.

@mariosasko Awesome thanks! I will do that. Looks like this new version of the data is not available as a zip but as three files (train/dev/test). How is this usually handled in HF Datasets, should `_URL` be a dict with keys `train`, `val`, `test` perhaps?

Yes! Let me help you with more detailed instructions.

In the first step, we need to update the URLs. One of the possible dictionary structures is as follows:
```python
_URLs = {
    "train": {"src": "https://drive.google.com/uc?export=download&id=1wHAWDOwOoQWSj7HYpyJ3Aeud8WhhaJ7P", "tgt": "https://drive.google.com/uc?export=download&id=1QVgswwhVTkd3VLCzajK6eVkcrSWEK6kq"}
    "val": ...
    "test": ...
}
```

(You can use this page to generate direct download links: https://sites.google.com/site/gdocs2direct/)

Then we move to the `split_generators` method:
```python
def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    files = dl_manager.download(_URLs)
    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={"src_file": files["train"]["src"], "tgt_file": files["train"]["tgt"]},
        ),
        ... # same for val and test
    ]
```
Finally, we adjust the signature of `_generate_examples`:
```python
def _generate_examples(self, src_file, tgt_file):
     """Yields examples."""
     with open(src_file, encoding="utf-8") as src_f, open(
         tgt_file, encoding="utf-8"
     ) as tgt_f:
         ... # the rest is the same
```

And that's it!

PS: Let me know if you need help updating the dummy data and regenerating the metadata file.

Awesome! Thanks for the detailed help, that was straightforward with your instruction. However, I think I am being blocked by this issue: https://github.com/huggingface/datasets/issues/4428

Feel free to open a PR, and I can fix this manually.

Awsome, done in #4451!