noamaanMulla-03/datasets-issues huggingface dataset:
Add SILICONE benchmark

My collaborators and I within the Affective Computing team at Telecom Paris would like to re-submit our spoken dialogue dataset for publication.

This is a new pull request relative to the [previously closed request](https://github.com/huggingface/datasets/pull/1712) which was reviewed by @lhoestq.




Thanks for the feedback. All your comments have been addressed!

Thank you for your constructive feedback! I now know how to best format future datasets that our team plans to publish in the near future :)

Awesome ! Looking forward to it :) 

Hi @lhoestq ! One last question. Our research team would like to distribute a link to this dataset amongst the spoken dialogue research community but the dataset does not show in the dropdown menu at huggingface.co. Is there anything else we must do in order to find the dataset there ?

Once the dataset does show in the dropdown menu, how can I affiliate it with the Telecom Paris organization that I already created at the website ?

The files are not located in the right place in the repo. Let me move them

I created a PR at https://github.com/huggingface/datasets/pull/1794

I just merged the change @eusip, now the dataset page is available at the url:
https://huggingface.co/datasets/silicone

Thank you for moving the folder for me :)
Support future datasets

If a dataset is available at the version of the local installation of `datasets` (e.g. 1.2.0), then loading this dataset means loading the script at this version.

However when trying to load a dataset that is only available on master, currently users have to specify `script_version="master"` in `load_dataset` to make it work.

However we could automatically get the dataset from master instead in this case.

I added this feature in this PR.
I also added a warning if a dataset is not available at the version of the local installation of `datasets` but is loaded from master:
```python
>>> load_dataset("silicone", "dyda_da")
Couldn't find file locally at silicone/silicone.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/silicone/silicone.py.
The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/silicone/silicone.py.
Downloading and preparing dataset silicone/dyda_da (download: 8.46 MiB, generated: 9.39 MiB, post-processed: Unknown size, total: 17.86 MiB) to /Users/quentinlhoest/.cache/huggingface/datasets/silicone/dyda_da/1.0.0/d41d8c0b73c6df035b1369c45774418f0051163ea689b5502b8bda783adf6342...
...
```
Add metrics usage examples and tests

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.
Adding SICK dataset

Hi
It would be great to include SICK dataset.

## Adding a Dataset
- **Name:** SICK
- **Description:** a well known entailment dataset 
- **Paper:** http://marcobaroni.org/composes/sick.html
- **Data:** http://marcobaroni.org/composes/sick.html
- **Motivation:** this is an important NLI benchmark

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).



thanks

Labels: name: dataset request
SICK dataset 

Hi, this would be great to have this dataset included. I might be missing something, but I could not find it in the list of already included datasets. Thank you. 

## Adding a Dataset
- **Name:** SICK
- **Description:**  SICK consists of about 10,000 English sentence pairs that include many examples of the lexical, syntactic, and semantic phenomena. 
- **Paper:** https://www.aclweb.org/anthology/L14-1314/
- **Data:** http://marcobaroni.org/composes/sick.html
- **Motivation:** This dataset is well-known in the NLP community used for recognizing entailment between sentences.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request
Add Tweet Eval Dataset

Closes Draft PR #1407. 

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.
Add Voxforge

## Adding a Dataset
- **Name:** *voxforge* 
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

name: speech
Adding initial version of cord-19 dataset

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### TODO:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding


Hi @ggdupont !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or when you're ready for a review

> Hi @ggdupont !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or when you're ready for a review

Yes I did, just busy period (and no time on weekend right now ;-) )

With some delay, reduced the dummy data and had t rebase

Thanks !

It looks like the rebase messed up the github diff for this PR (2.000+ files changed)
Could you create another branch and another PR please ?

Cleaned PR: https://github.com/huggingface/datasets/pull/1850
Enable logging propagation and remove logging handler

We used to have logging propagation disabled because of this issue: https://github.com/tensorflow/tensorflow/issues/26691
But since it's now fixed we should re-enable it. This is important to keep the default logging behavior for users, and propagation is also needed for pytest fixtures as asked in #1826 

I also removed the handler that was added since, according to the logging [documentation](https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library):
> It is strongly advised that you do not add any handlers other than NullHandler to your library’s loggers. This is because the configuration of handlers is the prerogative of the application developer who uses your library. The application developer knows their target audience and what handlers are most appropriate for their application: if you add handlers ‘under the hood’, you might well interfere with their ability to carry out unit tests and deliver logs which suit their requirements.

It could have been useful if we wanted to have a custom formatter for the logging but I think it's more important to keep the logging as default to not interfere with the users' logging management.

Therefore I also removed the two methods `datasets.logging.enable_default_handler` and `datasets.logging.disable_default_handler`.

cc @albertvillanova this should let you use capsys/caplog in pytest
cc @LysandreJik @sgugger if you want to do the same in `transformers`


Thank you @lhoestq. This logging configuration makes more sense to me.

Once propagation is allowed, the end-user can customize logging behavior and add custom handlers to the proper top logger in the hierarchy.

And I also agree with following the best practices and removing any custom handlers:
- it is the end user who has to implement any custom handlers
- indeed, the previous logging problem with TensorFlow was due to the fact that absl did not follow best practices and had implemented a custom handler

Our errors/warnings will be displayed anyway, even if we do not implement any custom handler. Since Python 3.2, logging has a built-in "default" handler (logging.lastResort) with the expected default behavior (sending error/warning messages to sys.stderr), which is used only if the end user has not configured any custom handler.
Add cord 19 dataset

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### Extras:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding


Cleaned-up version of previous PR: https://github.com/huggingface/datasets/pull/1129

@lhoestq FYI

Before merging I might tweak a little bit the dummy data to avoid having to check if the `document_parses` and `embeddings` directories exist or not. I'll do that later today

Looks all good now ! Thanks a lot @ggdupont :)
Merging
Provide better exception message when one of many files results in an exception

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)


Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
[Metrics] Add word error metric metric

This PR adds the word error rate metric to datasets. 
WER: https://en.wikipedia.org/wiki/Word_error_rate
for speech recognition. WER is the main metric used in ASR. 

`jiwer` seems to be a solid library (see https://github.com/asteroid-team/asteroid/pull/329#discussion_r525158939)


Feel free to merge once the CI is all green ;)
Update: SWDA - Fixed code to use all metadata features. Added comments and cleaned c…

This is a dataset I currently use my research and I realized some features are not being returned.

Previous code was not using all available metadata and was kind of messy

I fixed code to use all metadata and made some modification to be more efficient and better formatted.


Please let me know if I need to make any changes.


@yjernite Pushed all the changes you recommended. Thank you for your help!
test.json has been removed from the limit dataset repo (breaks dataset)

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`

Labels: name: dataset bug

Thanks for the heads up ! I'm opening a PR to fix that
Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`


Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.


```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```



Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR

I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)

Thanks for such a quick fix and merge to master, pip installed git master, tested all OK
Add Winogender Schemas

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

Nevermind, this one is already available on the hub under the name `'wino_bias'`: https://huggingface.co/datasets/wino_bias
Add OSCAR dataset card

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).


@lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅 

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔 

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :) 


I just updated the size in the table @pjox it should be good now :) 
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

Thank you so much @lhoestq !
Allow loading dataset in-memory

Allow loading datasets either from:
- memory-mapped file (current implementation)
- from file descriptor, copying data to physical memory

Close #708


I am wondering how to test their difference...

> ring how to test their difference...

Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.

Otherwise we can still check the difference of RAM used when loading a big chunk of data.

> Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.
> 
> Otherwise we can still check the difference of RAM used when loading a big chunk of data.

@lhoestq I think I found a way: `pa.total_allocated_bytes()` :smirk:
Add loading from the Datasets Hub + add relative paths in download manager

With the new Datasets Hub on huggingface.co it's now possible to have a dataset repo with your own script and data.
For example: https://huggingface.co/datasets/lhoestq/custom_squad/tree/main contains one script and two json files.

You can load it using
```python
from datasets import load_dataset

d = load_dataset("lhoestq/custom_squad")
```

To be able to use the data files that live right next to the dataset script on the repo in the hub, I added relative paths support for the DownloadManager. For example in the repo mentioned above, there are two json files that can be downloaded via
```python
_URLS = {
    "train": "train-v1.1.json",
    "dev": "dev-v1.1.json",
}
downloaded_files = dl_manager.download_and_extract(_URLS)
```

To make it work, I set the `base_path` of the DownloadManager to be the parent path of the dataset script (which comes from either a local path or a remote url).

I also had to add the auth header of the requests to huggingface.co for private datasets repos. The token is fetched from [huggingface_hub](https://github.com/huggingface/huggingface_hub).


I just added the steps to share a dataset on the datasets hub. It's highly inspired by the steps to share a model in the `transformers` doc.

Moreover once the new huggingface_hub is released we can update the version in the setup.py. We also need to update the command to create a dataset repo in the documentation

I added a few more tests with the "lhoestq/test" dataset I added on the hub and it works fine :) 

Here is the PR adding support for datasets repos in `huggingface_hub`: https://github.com/huggingface/huggingface_hub/pull/14