noamaanMulla-03/datasets-issues huggingface dataset:
Linguisticprobing

Adding Linguistic probing datasets from
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
 https://www.aclweb.org/anthology/P18-1198/

Labels: name: dataset contribution

Thanks for your contribution, @sileod.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

As you already created this dataset under your organization namespace (https://huggingface.co/datasets/metaeval/linguisticprobing), I think we can safely close this PR.

We would suggest you add a dataset card with the YAML tags, to make it searchable and discoverable.

Please, feel free to tell us if you need some help.
[WIP] complex_webqa - Error zipfile.BadZipFile: Bad CRC-32

Have a string `zipfile.BadZipFile: Bad CRC-32 for file 'web_snippets_train.json'` error when downloading the largest file from dropbox: `https://www.dropbox.com/sh/7pkwkrfnwqhsnpo/AABVENv_Q9rFtnM61liyzO0La/web_snippets_train.json.zip?dl=1`

Didn't managed to see how to solve that.

Putting aside for now.


Labels: name: dataset contribution

Thanks for your contribution, @thomwolf. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
CheckList

Sorry for the large pull request.
- Added checklists as datasets. I can't run `test_load_real_dataset` (see #474), but I can load the datasets successfully as shown in the example notebook
- Added a checklist wrapper 

Labels: name: dataset contribution

> Also, a little out of my depth there, but would there be a way to have the default pip install checklist command not require mysql and mariadb to be installed? Feels like that might be a source of confusion for users.

I removed the pattern dependency, mysql is not a requirement anymore. I'm not sure where mariadb is coming from. 

Thanks for your contribution, @marcotcr. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Revert task removal in folder-based builders

Reverts the removal of `task_templates` in the folder-based builders. I also added the `AudioClassifaction` task for consistency.

This is needed to fix https://github.com/huggingface/transformers/issues/19177.

I think we should soon deprecate and remove the current task API (and investigate if it's possible to integrate the `train eval index` API), but we need to update the Transformers examples before that so we don't break them.

cc @NielsRogge 


_The documentation is not available anymore as the PR was closed or merged._
Dataset Viewer issue for timit_asr

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Yes, the dataset viewer is based on `datasets`, and the following does not work:

```
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('timit_asr')
Downloading builder script: 7.48kB [00:00, 6.69MB/s]
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/timit_asr/43f9448dd5db58e95ee48a277f466481b151f112ea53e27f8173784da9254fb2/timit_asr.py", line 117, in _split_generators
    data_dir = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/posixpath.py", line 231, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

cc @huggingface/datasets 

Due to license restriction, this dataset needs manual downloading of the original data.

This information is in the dataset card: https://huggingface.co/datasets/timit_asr
> The dataset needs to be downloaded manually from https://catalog.ldc.upenn.edu/LDC93S1

Maybe a better error message for datasets that need manual downloading? @severo 

Maybe we can raise a specific excpetion as done from `load_dataset`...

Yes, ideally something like https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L81


The preview is now disabled (and a descriptive warning is displayed) for datasets requiring manual download. See:

![timit_asr-manual-download](https://user-images.githubusercontent.com/8515462/193578572-3d21b790-f848-4257-9e9b-7cab3d76a269.png)
passing parameters to the method passed to Dataset.from_generator()

Big thanks for providing dataset creation via a generator.
I want to ask whether there is any way that parameters can be passed to the method Dataset.from_generator() method, like as follows. 

```
from datasets import Dataset

def gen(param1):
    for idx in len(custom_dataset):
        yield custom_dataset[idx] + param1

ds = Dataset.from_generator(gen(param1))
```


Labels: name: enhancement

Hi! Yes, you can either use the `gen_kwargs` param in `Dataset.from_generator` (`ds = Dataset.from_generator(gen, gen_kwargs={"param1": val})`) or wrap the generator function with `functools.partial`
(`ds = Dataset.from_generator(functools.partial(gen, param1="val"))`) to pass custom parameters to it.
Fix backward compatibility for dataset_infos.json

While working on https://github.com/huggingface/datasets/pull/5018 I noticed a small bug introduced in #4926 regarding backward compatibility for dataset_infos.json

Indeed, when a dataset repo had both dataset_infos.json and README.md, the JSON file was ignored. This is unexpected: in practice it should be ignored only if the README.md has a dataset_info field, which has precedence over the data in the JSON file.


_The documentation is not available anymore as the PR was closed or merged._
Support loading a dataset from SQLite files

As requested by @julien-c, we could eventually support loading a dataset from SQLite files, like it is the case for JSON/CSV files.

Labels: name: enhancement

name: good second issue

for reference Kaggle has a good number of open source datasets stored in sqlite

Alternatively a tutorial or tool on how to convert from sqlite to parquet would be cool too

Hello, could we leverage [`pandas.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) for this? 

This would be basically the same as [`CSVBuilder`](https://github.com/huggingface/datasets/blob/7380140accf522a4363bb56c0b77a4190f49bed6/src/datasets/packaged_modules/csv/csv.py#L127)
, but uses `pandas.read_sql(..., chunksize=1)` instead of `pandas.read_csv(..., iterator=True)`

I'm happy to work on this :) 

self-assign
[GH->HF] Part 2: Remove all dataset scripts from github

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

- [x] Needs https://github.com/huggingface/datasets/pull/4973 to be merged first
- [x] and PR to be enabled on the Hub for non-namespaced datasets


_The documentation is not available anymore as the PR was closed or merged._

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

We are deprecating the metrics in `datasets` indeed and suggest users to switch to `evaluate` (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

I guess this is ready to merge ?

It should break nothing except one rare case:

If someone is using an old version of `datasets` to try to load a recent dataset. Indeed in that case it fetches the `main` branch on github to see if it exists. But since we're removing all the datasets, forward fetching won't work anymore.

e.g. if someone uses "imagenet-1k" with a version of `datasets` that didn't have it at that time. I checked on kibana and one single user would be affected with 4k downloads/months. It should still work for them though thanks to the `datasets` cache

But if they delete their cache, the workaround is... 🥁 update `datasets` 😅

Let's merge this on monday if we can, to make sure contributors who wanted to merge their dataset PRs here could do it

Alright, merging !
Fix CI hfh token warning

In our CI, we get warnings from `hfh` about using deprecated `token`: https://github.com/huggingface/datasets/actions/runs/3174626525/jobs/5171672431
```
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_private
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files_with_max_shard_size
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_overwrite_files
  C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\huggingface_hub\utils\_deprecation.py:97: FutureWarning: Deprecated argument(s) used in 'dataset_info': token. Will not be supported from version '0.12'.
    warnings.warn(message, FutureWarning)
```

This PR fixes the tests in `TestPushToHub` so that we fix these warnings.

Continuation of:
- #5031

CC: @lhoestq 


_The documentation is not available anymore as the PR was closed or merged._

good catch !
[Clean up] Datasets

Clean up `nlp/datasets` folder. 

As I understood, eventually the `nlp/datasets` shall not exist anymore at all. 

The folder `nlp/datasets/nlp` is kept for the moment, but won't be needed in the future, since it will live on S3 (actually it already does) at: `https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/?region=us-east-1` and the different `dataset downloader scripts will be added to `nlp/src/nlp` when downloaded by the user. 

The folder `nlp/datasets/checksums` is kept for now, but won't be needed anymore in the future. 

The remaining folders/ files are leftovers from tensorflow-datasets and are not needed. The can be looked up in the private tensorflow-dataset repo.


Yes!
[Map Function] add assert statement if map function does not return dict or None

IMO, if a function is provided that is not a print statement (-> returns variable of type `None`) or a function that updates the datasets (-> returns variable of type `dict`), then a `TypeError` should be raised. 

Not sure whether you had cases in mind where the user should do something else @thomwolf , but I think a lot of silent errors can be avoided with this assert statement.


Also added to an assert statement that if a dict is returned by function, all values of `dicts` are `lists`

Wait to merge until `make style` is set in place.

Updated the assert statements. Played around with multiple cases and it should be good now IMO.
Add metrics

This PR is a draft for adding metrics (sacrebleu and seqeval are added)

use case examples:
`import nlp`
**sacrebleu:**
```
refs = [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
sacrebleu = nlp.load_metrics('sacrebleu')
print(sacrebleu.score)
```

**seqeval:**
```
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = nlp.load_metrics('seqeval')
print(seqeval.accuracy_score(y_true, y_pred)
print(seqeval.f1_score(y_true, y_pred)
```
_examples are taken from the corresponding web page_

your comments and suggestions are more than welcomed
add metrics which require download files from github

To download files from github, I copied the `load_dataset_module` and its dependencies (without the builder) in `load.py` to `metrics/metric_utils.py`. I made the following changes:

- copy the needed files in a folder`metric_name` 
- delete all other files that are not needed

For metrics that require an external import, I first create a `<metric_name>_imports.py` file which contains all external urls. Then I create a `<metric_name>.py` in which I will load the external files using `<metric_name>_imports.py`
[Checksums] If no configs exist prevent to run over empty list

`movie_rationales` e.g. has no configs.


Whoops I fixed it directly on master before checking that you have done it in this PR. We may close it

Yeah, I saw :-) But I think we should add this as well since some datasets have an empty list of configs and then as the code is now it would fail. 

In this PR, I just make sure that the code jumps in the correct else if "there are no configs" as is the case for some datasets @mariamabarham 

Sorry, I thought you meant a different commit . Just saw this one: https://github.com/huggingface/nlp/commit/7c644f284e2303b57612a6e7c904fe13906d893f
.

All good then :-)
Mark CI tests as xfail when 502 error

To make CI more robust, we could mark as xfail when the Hub raises a 502 error (besides 500 error):

- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_skip_identical_files
  - https://github.com/huggingface/datasets/actions/runs/3174626525/jobs/5171672431
  ```
  >           raise HTTPError(http_error_msg, response=self)
  E           requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16648055339047.git/info/lfs/objects/batch
  ```

- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_overwrite_files
  - https://github.com/huggingface/datasets/actions/runs/3145587033/jobs/5113074889
  ```
  >           raise HTTPError(http_error_msg, response=self)
  E           requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16643866807322.git/info/lfs/objects/verify
  ```

Currently, we mark as xfail when 500 error:
- #4845


_The documentation is not available anymore as the PR was closed or merged._
Add add cc-by-sa-2.5 license tag

- [ ] add it to moon-landing
- [ ] add it to hub-docs 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4818). All of your documentation changes will be reflected on that endpoint.

I think we can close this PR because the `standard_licenses.tsv` file was removed from this repo and we no longer perform any dataset card validation.
Fix broken URL's (GEM)

This PR fixes the broken URL's in GEM. cc. @lhoestq, @albertvillanova 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5056). All of your documentation changes will be reflected on that endpoint.

Thanks, @manandey. We have removed all dataset scripts from this repo. Subsequent PRs should be opened directly on the Hugging Face Hub.
Add tedlium

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

name: speech

Hi @patrickvonplaten 
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0

Resolved via https://github.com/huggingface/datasets/pull/4309
Inspecting datasets per category

Hi
Is there a way I could get all NLI datasets/all QA datasets to get some understanding of available datasets per category? this is hard for me to inspect the datasets one by one in the webpage, thanks for the suggestions @lhoestq 


That's interesting, can you tell me what you think would be useful to access to inspect a dataset?

You can filter them in the hub with the search by the way: https://huggingface.co/datasets have you seen it?

Hi @thomwolf 
thank you, I was not aware of this, I was looking into the data viewer linked into readme page. 

This is exactly what I was looking for, but  this does not work currently, please see the attached 
I am selecting to see all nli datasets in english and it retrieves none. thanks

![5tarDHn9CP6ngeM](https://user-images.githubusercontent.com/53898419/103107612-1509aa80-4638-11eb-85b5-0c995a189969.png)





I see 4 results for NLI in English but indeed some are not tagged yet and missing (GLUE), we will focus on that in January (cc @yjernite): https://huggingface.co/datasets?filter=task_ids:natural-language-inference,languages:en

Hi! You can use `huggingface_hub`'s `list_datasets` for that now:
```python
import huggingface_hub # pip install huggingface_hub
huggingface_hub.list_datasets(filter="task_categories:question-answering")
# or
huggingface_hub.list_datasets(filter=("task_categories:natural-language-inference", "languages:"en"))
```
How to use similarity settings  other then "BM25" in Elasticsearch index ?

**QUESTION : How should we use other similarity algorithms supported by Elasticsearch other than "BM25"  ?**
**ES Reference**
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html
**HF doc reference:**
https://huggingface.co/docs/datasets/faiss_and_ea.html

**context :**
========

I used the latest Elasticsearch server  version 7.9.2
When I set DFR  which is one of the other similarity algorithms supported by elasticsearch  in the mapping, I get an error

For example DFR that I had tried in the first instance in mappings as below.,
`"mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "DFR"}}},`

I get the following error 
RequestError: RequestError(400, 'mapper_parsing_exception', 'Unknown Similarity type [DFR] for field [text]')

The other thing as another option I had tried was to declare "similarity": "my_similarity" within settings and then assigning "my_similarity" inside the mappings as below 

`es_config = {
        "settings": {
            "number_of_shards": 1,
             **"similarity":  "my_similarity"**: {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        } ,
            "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
            
        },
        "mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "my_similarity"}}},
    }`

For this , I got the following error
RequestError: RequestError(400, 'illegal_argument_exception', 'unknown setting [index.similarity] please check that any required plugins are installed, or check the breaking changes documentation for removed settings')




Datasets does not use elasticsearch API to define custom similarity. If you want to use a custom similarity, the best would be to run a curl request directly to your elasticsearch instance (see sample hereafter, directly from ES documentation), then you should be able to use `my_similarity` in your configuration passed to datasets

```
curl -X PUT "localhost:9200/index?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "similarity": {
        "my_similarity": {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        }
      }
    }
  }
}
'

```
Best practices for training tokenizers with nlp

Hi, thank you for developing this library. 

What do you think are the best practices for training tokenizers using `nlp`?  In the document and examples, I could only find pre-trained tokenizers used.


Docs that explain how to train a tokenizer with `datasets` are available here: https://huggingface.co/docs/tokenizers/training_from_memory#using-the-datasets-library
wmt download speed example

Continuing from the slack 1.0 roadmap thread w @lhoestq , I realized the slow downloads is only a thing sometimes. Here are a few examples, I suspect there are multiple issues. All commands were run from the same gcp us-central-1f machine.

```
import nlp
nlp.load_dataset('wmt16', 'de-en')
```
Downloads at 49.1 KB/S

Whereas 
```
pip install gdown # download from google drive
!gdown https://drive.google.com/uc?id=1iO7um-HWoNoRKDtw27YUSgyeubn9uXqj
```
Downloads at 127 MB/s. (The file is a copy of wmt-en-de raw).


```
nlp.load_dataset('wmt16', 'ro-en')
```
goes at 27 MB/s, much faster. 

if we wget the same data from s3 is the same download speed, but ¼ the file size:
```
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_packed_200_rand.tgz
```

Finally,
```
nlp.load_dataset('wmt19', 'zh-en')
```
Starts fast, but broken. (duplicate of #493 )





Thanks for creating the issue :)
The download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.
Is this mirror official ?

Also it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool

Also cc @patrickvonplaten 

Mirror is not official.

Shall we host the files ourselves or it is fine to use this mirror in your opinion ?

Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?

This could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).

@lhoestq I think we should host it ourselves. I'll put the subset of wmt (without preprocessed files) that we need on s3 and post a link over the weekend.

Is there a solution yet? The download speed is still too slow. 60-70kbps download for wmt16 and around 100kbps for wmt19. @sshleifer 

I'm working on mirror links which will provide high download speed :)
See https://github.com/huggingface/datasets/issues/1892

Resolved via https://github.com/huggingface/datasets/pull/1912