AniketTec/github-issues huggingface dataset:
Title: Can't call shape on the output of select()
State: closed

I get the error `TypeError: tuple expected at most 1 argument, got 2` when calling `shape` on the output of `select()`.
It's line 531 in shape in arrow_dataset.py that causes the problem:
``return tuple(self._indices.num_rows, self._data.num_columns)``
This makes sense, since `tuple(num1, num2)` is not a valid call.
 
Full code to reproduce:

```python
dataset = load_dataset("cnn_dailymail", "3.0.0")
train_set = dataset["train"]
t = train_set.select(range(10))
print(t.shape)

Comments:
Title: Can't filter language:EN on https://huggingface.co/datasets
State: closed

When visiting https://huggingface.co/datasets, I don't see an obvious way to filter only English datasets. This is unexpected for me, am I missing something? I'd expect English to be selectable in the language widget. This problem reproduced on Mozilla Firefox and MS Edge:

![screenshot](https://user-images.githubusercontent.com/4547987/102792244-892e1f00-43a8-11eb-9e89-4826ca201a87.png)


Comments:
Title: added TurkishMovieSentiment dataset
State: closed

This PR adds the **TurkishMovieSentiment: This dataset contains turkish movie reviews.**

- **Homepage:** [https://www.kaggle.com/mustfkeskin/turkish-movie-sentiment-analysis-dataset/tasks](https://www.kaggle.com/mustfkeskin/turkish-movie-sentiment-analysis-dataset/tasks)
- **Point of Contact:** [Mustafa Keskin](https://www.linkedin.com/in/mustfkeskin/)

Comments:
Title: Bug: Can't download TriviaQA with `load_dataset` - custom `cache_dir`
State: open

Hello,
I'm having issue downloading TriviaQA dataset with `load_dataset`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")
```

## The output:
1. Download begins:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to /cs/labs/gabis/sapirweissbuch/tr
ivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                                         
Downloading:  17%|███████████████████▉                                                                                                   | 446M/2.67G [00:37<04:45, 7.77MB/s]
```
2. 100% is reached
3. It got stuck here for about an hour, and added additional 30G of data to "./datasets" directory. I killed the process eventually.

A similar issue can be observed in Google Colab:

https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing

## Expected behaviour:
The dataset "TriviaQA" should be successfully downloaded.


Comments:
Title: shuffle with torch generator 
State: closed
Labels: description: New feature or request
name: enhancement

Hi
I need to shuffle mutliple large datasets with `generator = torch.Generator()` for a distributed sampler which needs to make sure datasets are consistent across different cores, for this, this is really necessary for me to use  torch generator, based on documentation this generator is not supported with datasets, I really need to make shuffle work with this generator and I was wondering what I can do about this issue, thanks for your help 

@lhoestq 

Comments:
Title: Not able to use 'jigsaw_toxicity_pred' dataset
State: closed

 When trying to use jigsaw_toxicity_pred dataset, like this in a [colab](https://colab.research.google.com/drive/1LwO2A5M2X5dvhkAFYE4D2CUT3WUdWnkn?usp=sharing):
```
from datasets import list_datasets, list_metrics, load_dataset, load_metric

ds = load_dataset("jigsaw_toxicity_pred")
```
 
I see below error:

> FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/jigsaw_toxicity_pred/jigsaw_toxicity_pred.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/jigsaw_toxicity_pred/jigsaw_toxicity_pred.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    280                 raise FileNotFoundError(
    281                     "Couldn't find file locally at {}, or remotely at {} or {}".format(
--> 282                         combined_path, github_file_path, file_path
    283                     )
    284                 )

FileNotFoundError: Couldn't find file locally at jigsaw_toxicity_pred/jigsaw_toxicity_pred.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/jigsaw_toxicity_pred/jigsaw_toxicity_pred.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/jigsaw_toxicity_pred/jigsaw_toxicity_pred.py

Comments:
Title: adding  ted_talks_iwslt
State: closed

UPDATE2: (2nd Jan) Wrote a long writeup on the slack channel. I don't think this approach is correct. Basically this created language pairs (109*108) 
Running the `pytest `went for more than 40+ hours and it was still running!  
So working on a different approach, such that the number of configs = number of languages. Will make a new pull request with that. 

UPDATE: This requires manual download dataset

This is a draft version 

Comments:
Title: added Semantic Scholar Open Research Corpus
State: closed

I picked up this dataset [Semantic Scholar Open Research Corpus](https://allenai.org/data/s2orc) but it contains 6000 files to be downloaded. I tried the current code with 100 files and it worked fine (took ~15GB space). For 6000 files it would occupy ~900GB space which I don’t have. Can someone from the HF team with that much of disk space help me with generate dataset_infos and dummy_data?

Comments:
Title: Navigation version breaking
State: closed

Hi, 

when navigating docs (Chrome, Ubuntu) (e.g. on this page: https://huggingface.co/docs/datasets/loading_metrics.html#using-a-custom-metric-script) the version control dropdown has the wrong string displayed as the current version: 

![image](https://user-images.githubusercontent.com/3007947/102632187-02cad080-414f-11eb-813b-28f3c8d80def.png)

**Edit:** this actually happens _only_ if you open a link to a concrete subsection.

IMO, the best way to fix this without getting too deep into the intricacies of retrieving version numbers from the URL would be to change [this](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L112) line to:
```
let label = (version in versionMapping) ? version : stableVersion
```
which delegates the check to the (already maintained) keys of the version mapping dictionary & should be more robust. There's a similar ternary expression [here](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L97) which should also fail in this case.

I'd also suggest swapping this [block](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L80-L90) to `string.contains(version) for version in versionMapping` which might be more robust. I'd add a PR myself but I'm by no means competent in JS :)

I also have a side question wrt. docs versioning: I'm trying to make docs for a project which are versioned alike to your dropdown versioning. I was wondering how do you handle storage of multiple doc versions on your server? Do you update what `https://huggingface.co/docs/datasets` points to for every stable release & manually create new folders for each released version?
So far I'm building & publishing (scping) the docs to the server with a github action which works well for a single version, but would ideally need to reorder the public files triggered on a new release.

Comments:
Title: Add retries to HTTP requests
State: closed
Labels: description: New feature or request
name: enhancement

## What does this PR do ?

Adding retries to HTTP GET & HEAD requests, when they fail with a `ConnectTimeout` exception.

The "canonical" way to do this is to use [urllib's Retry class](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry) and wrap it in a [HttpAdapter](https://requests.readthedocs.io/en/master/api/#requests.adapters.HTTPAdapter). Seems a bit overkill to me, plus it forces us to use the `requests.Session` object. I prefer this simpler implementation. I'm open to remarks and suggestions @lhoestq @yjernite 

Fixes #1102 

Comments:
Title: AttributeError: 'DatasetDict' object has no attribute 'train_test_split'
State: closed
Labels: description: Further information is requested
name: question

The following code fails with "'DatasetDict' object has no attribute 'train_test_split'" - am I doing something wrong?
```
from datasets import load_dataset
dataset = load_dataset('csv', data_files='data.txt')
dataset = dataset.train_test_split(test_size=0.1)
```

> AttributeError: 'DatasetDict' object has no attribute 'train_test_split'

Comments:
Title: connection error 
State: closed

Hi
I am hitting to this error, thanks 

```
> Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 379, in <module>
    main()
  File "finetune_t5_trainer.py", line 208, in main
    if training_args.do_eval or training_args.evaluation_strategy != EvaluationStrategy.NO
  File "finetune_t5_trainer.py", line 207, in <dictcomp>
    for task in data_args.eval_tasks}
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 66, in load_dataset
    return datasets.load_dataset(self.task.name, split=split, script_version="master")
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/boolq/boolq.py
el/0 I1217 01:11:33.898849 354161 main shadow.py:210 Current job status: FINISHED
```

Comments:
Title: Access to key in DatasetDict map
State: closed
Labels: description: New feature or request
name: enhancement

It is possible that we want to do different things in the `map` function (and possibly other functions too) of a `DatasetDict`, depending on the key. I understand that `DatasetDict.map` is a really thin wrapper of `Dataset.map`, so it is easy to directly implement this functionality in the client code. Still, it'd be nice if there can be a flag, similar to `with_indices`, that allows the callable to know the key inside `DatasetDict`.

Comments:
Title: IWSLT-17 Link Broken
State: closed
Labels: description: This issue or pull request already exists
name: duplicate

description: A bug in a dataset script provided in the library
name: dataset bug

```
FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz
```

Comments:
Title: FileNotFoundError for `amazon_polarity`
State: closed

Version: `datasets==v1.1.3`

### Reproduction
```python
from datasets import load_dataset
data = load_dataset("amazon_polarity")
```
crashes with
```bash
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py
```
and 
```bash
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```
and
```bash
FileNotFoundError: Couldn't find file locally at amazon_polarity/amazon_polarity.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```

Comments:
Title: Adding CLIMATE-FEVER dataset
State: closed

This PR request the addition of the CLIMATE-FEVER dataset:
A dataset adopting the FEVER methodology that consists of 1,535 real-world claims regarding climate-change collected on the internet. Each claim is accompanied by five manually annotated evidence sentences retrieved from the English Wikipedia that support, refute or do not give enough information to validate the claim totalling in 7,675 claim-evidence pairs. The dataset features challenging claims that relate multiple facets and disputed cases of claims where both supporting and refuting evidence are present.

More information can be found at:
- Homepage: <http://climatefever.ai>
- Paper: <https://arxiv.org/abs/2012.00614>



Comments:
Title: Add comet metric
State: closed

Hey! I decided to add our new Crosslingual Optimized Metric for Evaluation of Translation (COMET) to the list of the available metrics.

COMET was [presented at EMNLP20](https://www.aclweb.org/anthology/2020.emnlp-main.213/) and it is the highest performing metric, so far, on the WMT19 benchmark.

We also participated in the [WMT20 Metrics shared task ](http://www.statmt.org/wmt20/pdf/2020.wmt-1.101.pdf) where once again COMET was validated as a top-performing metric. 


I hope that this metric will help researcher's and industry workers to better validate their MT systems in the future 🤗 !

Cheers,
Ricardo





Comments:
Title: adding tmu-gfm-dataset
State: closed

Adding TMU-GFM-Dataset for Grammatical Error Correction.

https://github.com/tmu-nlp/TMU-GFM-Dataset

A dataset for GEC metrics with manual evaluations of grammaticality, fluency, and meaning preservation for system outputs.
More detail about the creation of the dataset can be found in [Yoshimura et al. (2020)](https://www.aclweb.org/anthology/2020.coling-main.573.pdf).

Comments:
Title: added air_dialogue
State: closed

UPDATE2 (3797ce5): Updated for multi-configs 

UPDATE (7018082): manually created the dummy_datasets. All tests were cleared locally. Pushed it to origin/master

DRAFT VERSION (57fdb20):  (_no longer draft_)
Uploaded the air_dialogue database. 
dummy_data creation was failing in local, since the original downloaded file has some nested folders. Pushing it since the tests with real data was cleared. Will re-check & update via manually creating some dummy_data

Comments:
Title: connection issue while downloading data
State: closed

Hi
I am running my codes on google cloud, and I am getting this error resulting in the failure of the codes when trying to download the data, could you assist me to solve this? also as a temporary solution, could you tell me how I can increase the number of retries and timeout to at least let the models run for now. thanks 

```
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 361, in <module>
    main()
  File "finetune_t5_trainer.py", line 269, in main
    add_prefix=False if training_args.train_adapters else True)
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 306, in load_dataset
    return datasets.load_dataset('glue', 'cola', split=split)
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 263, in prepare_module
    head_hf_s3(path, filename=name, dataset=dataset)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 200, in head_hf_s3
    return http_head(hf_bucket_url(identifier=identifier, filename=filename, use_cdn=use_cdn, dataset=dataset))
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 403, in http_head
    url, proxies=proxies, headers=headers, cookies=cookies, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f47db511e80>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))
```

Comments:
Title: added TTC4900: A Benchmark Data for Turkish Text Categorization dataset
State: closed

This PR adds the TTC4900 dataset which is a Turkish Text Categorization dataset by me and @basakbuluz. 

Homepage: [https://www.kaggle.com/savasy/ttc4900](https://www.kaggle.com/savasy/ttc4900)
Point of Contact: [Savaş Yıldırım](mailto:savasy@gmail.com) / @savasy


Comments:
Title: Add : Conv AI 2 (Messed up original PR)
State: closed

@lhoestq Sorry I messed up the previous 2 PR's -> https://github.com/huggingface/datasets/pull/1462 -> https://github.com/huggingface/datasets/pull/1383. So created a new one. Also, everything is fixed in this PR. Can you please review it ?
Thanks in advance. 

Comments:
Title: Inconsistent argument names.
State: closed

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.

Comments:
Title: adding snow_simplified_japanese_corpus
State: closed

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23

Comments:
Title: Add Turkish News Category Dataset (270K).Updates were made for review…
State: closed

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq 

Comments:
Title: Add id_nergrit_corpus
State: closed

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. 
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)

Comments:
Title: add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]
State: closed

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Comments:
Title: add thai_toxicity_tweet
State: closed

Thai Toxicity Tweet Corpus contains 3,300 tweets (506 tweets with texts missing) annotated by humans with guidelines including a 44-word dictionary. The author obtained 2,027 and 1,273 toxic and non-toxic tweets, respectively; these were labeled by three annotators. The result of corpus analysis indicates that tweets that include toxic words are not always toxic. Further, it is more likely that a tweet is toxic, if it contains toxic words indicating their original meaning. Moreover, disagreements in annotation are primarily because of sarcasm, unclear existing target, and word sense ambiguity.

Notes from data cleaner: The data is included into [huggingface/datasets](https://www.github.com/huggingface/datasets) in Dec 2020. By this time, 506 of the tweets are not available publicly anymore. We denote these by `TWEET_NOT_FOUND` in `tweet_text`.
Processing can be found at [this PR](https://github.com/tmu-nlp/ThaiToxicityTweetCorpus/pull/1).

Comments:
Title: FileNotFound remotly, can't load a dataset
State: closed

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy 

Comments:
Title: Create XML dummy data without loading all dataset in memory
State: closed

While I was adding one XML dataset, I noticed that all the dataset was loaded in memory during the dummy data generation process (using nearly all my laptop RAM).

Looking at the code, I have found that the origin is the use of `ET.parse()`. This method loads **all the file content in memory**.

In order to fix this, I have refactorized the code and use `ET.iterparse()` instead, which **parses the file content incrementally**.

I have also implemented a test.

Comments: