noamaanMulla-03/datasets-issues huggingface dataset:
Trying to get in touch regarding a security issue

Hey there!

I'd like to report a security issue but cannot find contact instructions on your repository.

If not a hassle, might you kindly add a `SECURITY.md` file with an email, or another contact method? GitHub [recommends](https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository) this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)


Hi @JamieSlome,

Thanks for reaching out. Yes, you are right: I'm opening a PR to add the `SECURITY.md` file and a contact method.

In the meantime, please feel free to report the security issue to: feedback@huggingface.co
Correctly update metadata to preserve features when concatenating datasets with axis=1

This PR correctly updates metadata to preserve higher-level feature types (e.g. `ClassLabel`) in `datasets.concatenate_datasets` when `axis=1`. Previously, we would delete the feature metadata in `datasets.concatenate_datasets` if `axis=1` and restore the feature types from the arrow table schema in `Dataset.__init__`. However, this approach only works for simple feature types (e.g. `Value`).

Fixes #3111
Update doc links to point to new docs

This PR:
* updates the README links and the ADD_NEW_DATASET template to point to the new docs (the new docs don't have a section with the list of all the possible features, so I added that info to the `Features` docstring, which is then referenced in the ADD_NEW_DATASET template)
* fixes some broken links in the `.rst` files (fixed with the `make linkcheck` tool)

Labels: name: documentation
Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?) 

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik

Labels: name: enhancement

@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer
Use ETag in streaming mode to detect resource updates

**Is your feature request related to a problem? Please describe.**

I want to cache data I generate from processing a dataset I've loaded in streaming mode, but I've currently no way to know if the remote data has been updated or not, thus I don't know when to invalidate my cache.

**Describe the solution you'd like**

Take the ETag of the data files into account and provide it (directly or through a hash) to give a signal that I can invalidate my cache.

**Describe alternatives you've considered**

None



Labels: name: enhancement

name: dataset-viewer
download_mode=`force_redownload` does not work on removed datasets

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1

Labels: name: bug

name: dataset-viewer
always requiring the username in the dataset name when there is one

Me and now another person have been bitten by the `datasets`'s non-strictness on requiring a dataset creator's username when it's due.

So both of us started with `stas/openwebtext-10k`, somewhere along the lines lost `stas/` and continued using `openwebtext-10k` and it all was good until we published the software and things broke, since there is no `openwebtext-10k`

So this feature request is asking to tighten the checking and not allow dataset loading if it was downloaded with the user prefix, but then attempted to be used w/o it.

The same in code:

```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Please let me know if I explained myself clearly.

Thank you!

Labels: name: enhancement

From what I can understand, you want the saved arrow file directory to have username as well instead of just dataset name if it was downloaded with the user prefix?

I don't think the user cares of how this is done, but the 2nd command should fail, IMHO, as its dataset name is invalid:
```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Moreover, if someone were to create `openwebtext-10k` w/o the prefix, they will now get the wrong dataset, if they previously downloaded `stas/openwebtext-10k`.

And if there are 2 users with the same dataset name `foo/ds` and `bar/ds` - currently this won't work to get the correct dataset.

So really there 3 unrelated issues hiding in the current behavior.

This has been fixed now, and we'll do a new release of the library today.

Now the stas/openwebtext-10k dataset is cached at `.cache/huggingface/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/datasets/openwebtext10k`. Since they are different, the cache won't fall back on loading the wrong one anymore.

Same for the python script used to generate the dataset: stas/openwebtext-10k is cached at `.cache/huggingface/modules/datasets_modules/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/modules/datasets_modules/datasets/openwebtext10k`

Amazing! Thank you for adding this improvement, @lhoestq!

(can be closed?)

Yes indeed :) thanks
Add a metric for the MATH dataset (competition_math).

This metric computes accuracy for the MATH dataset (https://arxiv.org/abs/2103.03874) after canonicalizing the prediction and the reference (e.g., converting "1/2" to "\\\\frac{1}{2}").


I believe the only failed test related to this PR is tests/test_metric_common.py::LocalMetricTest::test_load_metric_competition_math. It gives the following error:

ImportError: To be able to use this dataset, you need to install the following dependencies['math_equivalence'] using 'pip install git+https://github.com/hendrycks/math.git' for instance'

It fails along with (these fail with ImportError as well):
test_load_metric_bertscore
test_load_metric_bleurt
test_load_metric_comet
test_load_metric_coval

Let me know if there is anything I need to change.

Hi ! The script looks all good thanks :)

To fix the CI you just need to merge `master` into your branch
```
git fetch upstream/master
git merge upstream/master
```

Then you also need to add `math_equivalence` to the list of git packages installed for the tests in `additional-tests-requirements.txt`
https://github.com/huggingface/datasets/blob/ba831e4bcd175ae3d52afbf7d12c4f625bf541b0/additional-tests-requirements.txt#L1-L3

I ran:

git fetch upstream
git merge upstream/master

And I also added  math_equivalence to the list of git packages installed for the tests in additional-tests-requirements.txt

tests/test_metric_common.py fails with the same errors as before. tests/test_dataset_cards.py also fails, but it doesn't look related to this PR (it's an issue datasets/ami/README.md).

@lhoestq Anything else I can do? I re-merged again and am getting the same test failures as described in the previous comment.
Fix file/directory deletion on Windows  

Currently, on Windows, some attempts to delete a dataset file/directory will fail with the `PerimissionError`. 

Examples:
- download a dataset, then force redownload it in the same session while keeping a reference to the downloaded dataset
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset = load_dataset("sst", split="train", download_mode="force_redownload")
```
- try to clean up the cache files while keeping a reference to those files (via the mapped dataset):
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset_mapped = dset.map(lambda _: {"dummy_col": 1})
dset.cleanup_cache_files()
```
We should fix those.


Labels: name: bug
Provide a way to check if the features (in info) match with the data of a split

**Is your feature request related to a problem? Please describe.**

I understand that currently the data loaded has not always the type described in the info features

**Describe the solution you'd like**

Provide a way to check if the rows have the type described by info features

**Describe alternatives you've considered**

Always check it, and raise an error when loading the data if their type doesn't match the features.


Labels: name: enhancement

name: dataset-viewer

Related: #3144
Missing Zenodo 1.13.3 release

After `datasets` 1.13.3 release, this does not appear in Zenodo releases: https://zenodo.org/record/5570305

TODO:
- [x] Contact Zenodo support
- [x] Check it is fixed

Labels: name: bug

Zenodo has fixed on their side the 1.13.3 release: https://zenodo.org/record/5589150
"arabic_billion_words" dataset does not create the full dataset

## Describe the bug
When running: 
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB) 

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1



Labels: name: bug

Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.
Fix script of Arabic Billion Words dataset to return all data

The script has a bug and only parses and generates a portion of the entire dataset.

This PR fixes the loading script so that is properly parses the entire dataset. 

Current implementation generates the same number of examples as reported in the [original paper](https://arxiv.org/abs/1611.04033) for all configurations except for one:
- For "Youm7" we generate more examples (1172136) than the ones reported by the paper (1025027)

|                | Number of examples | Number of examples according to the source |
|:---------------|-------------------:|-----:|
| Alittihad      |             349342 |349342 |
| Almasryalyoum  |             291723 |291723 |
| Almustaqbal    |             446873 |446873 |
| Alqabas        |             817274 |817274 |
| Echoroukonline |             139732 |139732 |
| Ryiadh         |             858188 | 858188 |
| Sabanews       |              92149 |92149 |
| SaudiYoum      |             888068 |888068 |
| Techreen       |             314597 |314597 |
| Youm7          |            1172136 |1025027 |

Fix #3126.
VisibleDeprecationWarning when using `set_format("numpy")`

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```

Labels: name: bug

I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)
Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech

## Adding a Dataset
- **Name:** *openslr**
- **Description:** *Data set which contains male and female recordings of English from various dialects of the UK and Ireland.*
- **Paper:** *https://www.openslr.org/resources/83/about.html*
- **Data:** *Eleven separate data files can be found via https://www.openslr.org/resources/83/*
- **Motivation:** *Increase english ASR data with UK and Irish dialects*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

The *openslr* dataset already exists, this will add additional subset, *SLR83*.

Labels: name: dataset request

Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.
Add Google BLEU (aka GLEU) metric

This PR adds the NLTK implementation of Google BLEU metric. This is also a part of an effort to resolve an unfortunate naming collision between GLEU for machine translation and GLEU for grammatical error correction.
I used [this page](https://huggingface.co/docs/datasets/add_metric.html)  for reference. Please, point me to the right direction if I missed anything.
Fix string encoding for Value type

Some metrics have `string` features but currently it fails if users pass integers instead. Indeed feature encoding that handles the conversion of the user's objects to the right python type is missing a case for `string`, while it already works as expected for integers, floats and booleans

Here is an example code that didn't work previously, but that works with this fix:
```python
import datasets

# Note that 'id' is an integer while the SQuAD metric uses strings
predictions = [{'prediction_text': '1976', 'id': 5}]
references = [{'answers': {'answer_start': [97], 'text': ['1976']}, 'id': 5}] 

squad_metric = datasets.load_metric("squad") 
squad_metric.add_batch(predictions=predictions, references=references) 
results = squad_metric.compute()
# {'exact_match': 100.0, 'f1': 100.0}
```

cc @sgugger @philschmid 


That was fast!
request to mirror wmt datasets, as they are really slow to download

Would it be possible to mirror the wmt data files under hf? Some of them take hours to download and not because of the local speed. They are all quite small datasets, just extremely slow to download.

Thank you!


Yes that would be awesome. Not only the download speeds are awful, but also some files are missing.
We list all the URLs in the datasets/wmt19/wmt_utils.py so we can make a script to download them all and host on S3.
Also I think most of the materials are under the CC BY-NC-SA 3.0 license (must double check) so it should be possible to redistribute the data with no issues.

cc @patrickvonplaten who knows more about the wmt scripts

Yeah, the scripts are pretty ugly! A big refactor would make sense here...and I also remember that the datasets were veeery slow to download

I'm downloading them.
I'm starting with the ones hosted on http://data.statmt.org which are the slowest ones

@lhoestq better to use our new git-based system than just raw S3, no? (that way we have built-in CDN etc.)

Closing since the urls were changed to mirror urls in #1912 

Hi there! What about mirroring other datasets like [CCAligned](http://www.statmt.org/cc-aligned/) as well? All of them are really slow to download...
CLI test command throws NonMatchingSplitsSizesError when saving infos

When trying to generate a datset JSON metadata, a `NonMatchingSplitsSizesError` is thrown:
```
$ datasets-cli test datasets/arabic_billion_words --save_infos --all_configs
Testing builder 'Alittihad' (1/10)
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: Unknown size, post-processed: Unknown size, total: 332.13 MiB) to .cache\arabic_billion_words\Alittihad\1.1.0\8175ff1c9714c6d5d15b1141b6042e5edf048276bb81a9c14e35e149a7a62ae4...
Traceback (most recent call last):
  File "path\huggingface\datasets\.venv\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "path\huggingface\datasets\src\datasets\commands\datasets_cli.py", line 33, in main
    service.run()
  File "path\huggingface\datasets\src\datasets\commands\test.py", line 144, in run
    builder.download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "path\huggingface\datasets\src\datasets\utils\info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words')}]
```

This is due because a previous run generated a wrong `dataset_info.json`.

This error can be avoided by passing `--ignore_verifications`, but I think this should be assumed when passing `--save_infos`.

Labels: name: bug
datasets-cli: convertion of a tfds dataset to a huggingface one.

### Discussed in https://github.com/huggingface/datasets/discussions/3079

<div type='discussions-op-text'>

<sup>Originally posted by **vitalyshalumov** October 14, 2021</sup>
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/


and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


</div>


Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.
Make inspect.get_dataset_config_names always return a non-empty list of configs

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161


Labels: name: enhancement

name: dataset-viewer

Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?

Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)
add multi-proc in `to_csv`

This PR extends the multi-proc method used in #2747 for`to_json` to `to_csv` as well. 

Results on my machine post benchmarking on `ascent_kb` dataset (giving ~45% improvement when compared to num_proc = 1):
```
Time taken on 1 num_proc, 10000 batch_size  674.2055702209473
Time taken on 4 num_proc, 10000 batch_size  425.6553490161896

Time taken on 1 num_proc, 50000 batch_size  623.5897650718689
Time taken on 4 num_proc, 50000 batch_size  380.0402421951294

Time taken on 4 num_proc, 100000 batch_size  361.7168130874634
```
This is a WIP as writing tests is pending for this PR. 

I'm also exploring [this](https://arrow.apache.org/docs/python/csv.html#incremental-writing) approach for which I'm using `pyarrow-5.0.0`.



I think you can just add a test `test_dataset_to_csv_multiproc` in `tests/io/test_csv.py` and we'll be good

Hi @lhoestq, 
I've added `test_dataset_to_csv` apart from `test_dataset_to_csv_multiproc` as no test was there to check generated CSV file when `num_proc=1`. Please let me know if anything is also required!