Added XL-Sum dataset
State: closed | Author: abhik1505040 (NONE)

Added XL-Sum dataset published in ACL-IJCNLP 2021. (https://aclanthology.org/2021.findings-acl.413/). The default timeout values in `src/datasets/utils/file_utls.py` were increased to enable downloading from the original google drive links.

Comments: Thanks for adding this one ! I just did some minor changes and set the timeout back to 100sec instead of 1000

The CI failure is unrelated to this PR - let me take a look

> Thanks for adding this one! I just did some minor changes and set the timeout back to 100sec instead of 1000

Thank you for updating the language tags. I tried timeout values up to 300 sec on my local machine, but some of the larger files still get timed out. Although this could have been a network issue on my end, have you verified that 100 sec works for all files?

Well the main issue with google drive - even before the time out issues - is that it has a daily quota of downloads per file.
Therefore if many people start downloading this dataset, it will be unavailable until the quota is reset the next day.

So ideally it would be nice if the data were hosted elsewhere than Google drive, to avoid the quota and time out issue.
HF can probably help with hosting the data if needed

> Well the main issue with google drive - even before the time out issues - is that it has a daily quota of downloads per file.
> Therefore if many people start downloading this dataset, it will be unavailable until the quota is reset the next day.
> 
> So ideally it would be nice if the data were hosted elsewhere than Google drive, to avoid the quota and time out issue.
> HF can probably help with hosting the data if needed

It'd be great if the dataset can be hosted in HF. How should I proceed here though? Upload the dataset files as a community dataset and update the links in this pull request or is there a more straightforward way?

Hi ! Ideally everything should be in the same place, so feel free to create a community dataset on the Hub and upload your data files as well as you dataset script (and also the readme.md and dataset_infos.json).

The only change you have to do in your dataset script is use a relative path to your data files instead of urls.
For example if your repository looks like this:
```
xlsum/
├── data/
│   ├── amharic_XLSum_v2.0.tar.bz2
│   ├── ...
│   └── yoruba_XLSum_v2.0.tar.bz2
├── xlsum.py
├── README.md
└── dataset_infos.json
```
Then you just need to pass `"data/amharic_XLSum_v2.0.tar.bz2"` to `dl_manager.download_and_extract(...)`, instead of an url.

Locally you can test that it's working as expected with
```python
load_dataset("path/to/my/directory/named/xlsum")
```

Then once it's on the Hub, you can load it with
```python
load_dataset("username/xlsum")
```

Let me know if you have questions :)

Thank you for your detailed response regarding the community dataset building process. However, will this pull request be merged into the main branch?

If XL-sum is available via the Hub we don't need to add it again in the `datasets` github repo ;)

The dataset has now been uploaded on HF hub. It's available at https://huggingface.co/datasets/csebuetnlp/xlsum. Closing this pull request. Thank you for your contributions. 

Thank you !