noamaanMulla-03/datasets-issues huggingface dataset:
Add `@transmit_format` in `flatten`

As suggested by @mariosasko in https://github.com/huggingface/datasets/pull/4411, we should include the `@transmit_format` decorator to `flatten`, `rename_column`, and `rename_columns` so as to ensure that the value of `_format_columns` in an `ArrowDataset` is properly updated.

**Edit**: according to @mariosasko comment below, the decorator `@transmit_format` doesn't handle column renaming, so it's done manually for those instead.


@mariosasko please let me know whether we need to include some sort of tests to make sure that the decorator is working as expected. Thanks! 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4493). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Yes, please add (simple) tests so we can avoid any unexpected behavior in the future.

`@transmit_format` doesn't handle column renaming, so I removed it from `rename_column` and `rename_columns` and added a comment to explain this.

Oops, I thought this PR was already merged and deleted from the source repository, I'll be creating a new branch out of `main` so as to re-create this PR... My bad :weary:
Dataset Viewer issue for Jean-Baptiste/wikiner_fr

### Link

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr

### Description

```
Error code:   StreamingRowsError
Exception:    FileNotFoundError
Message:      [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 77, in get_rows
                 rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 718, in __iter__
                 for key, example in self._iter():
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 708, in _iter
                 yield from ex_iterable
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 112, in __iter__
                 yield from self.generate_examples_fn(**self.kwargs)
               File "/tmp/modules-cache/datasets_modules/datasets/Jean-Baptiste--wikiner_fr/683a580ba6ec769d508f7dfc603a651667b0ed3817b1ae5bfd45f97cc024923f/wikiner_fr.py", line 165, in _generate_examples
                 dataset = Dataset.load_from_disk(filepath)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1210, in load_from_disk
                 with open(Path(dataset_path, config.DATASET_STATE_JSON_FILENAME).as_posix(), encoding="utf-8") as state_file:
             FileNotFoundError: [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
```

Is it an error with the dataset script, or the data itself, @huggingface/datasets?

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/tree/main 

### Owner

No


The script uses `Dataset.load_from_disk`, which as you can expect, doesn't work in streaming mode.

It would probably be more practical to load the dataset locally using `Dataset.load_from_disk` first and then `push_to_hub` to upload it in Parquet on the Hub

I've transferred this issue to the Hub repo: https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/discussions/3

I'm closing this.
Fix Citation Information section in dataset cards

Fix Citation Information section in dataset cards:
- cc_news
- conllpp
- datacommons_factcheck
- gnad10
- id_panl_bppt
- jigsaw_toxicity_pred
- kinnews_kirnews
- kor_sarcasm
- makhzan
- reasoning_bg
- ro_sts
- ro_sts_parallel
- sanskrit_classic
- telugu_news
- thaiqa_squad
- wiki_movies

This PR partially fixes the Citation Information section in dataset cards. Subsequent PRs will follow to complete this task.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4879). All of your documentation changes will be reflected on that endpoint.
Update swiss judgment prediction

Hi,

I updated the dataset to include additional data made available recently. When I test it locally, it seems to work. However, I get the following error with the dummy data creation: 
`Dummy data generation done but dummy data test failed since splits ['train', 'validation', 'test'] have 0 examples for config 'fr'`. Do you know why this could be the case?

Cheers,
Joel

Labels: name: dataset contribution

Thank you very much for the detailed review @albertvillanova!

I updated the PR with the requested changes. 

At the end, I had to manually fix the conflict, so that CI tests are launched.

PLEASE NOTE: you should first pull to incorporate the previous commit
```shell
git pull
```

_The documentation is not available anymore as the PR was closed or merged._

Thank you very much for the detailed feedback and your time @albertvillanova! 
Yes, thanks. My other datasets are already on the hub: https://huggingface.co/joelito
Fix URLs of sbu_captions dataset

Forbidden
You don't have permission to access /~vicente/sbucaptions/sbu-captions-all.tar.gz on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5 mod_perl/2.0.11 Perl/v5.16.3 Server at [www.cs.virginia.edu](mailto:csroot@virginia.edu) Port 443

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._
Text strings are split into lists of characters in xcsr dataset

## Describe the bug
Text strings are split into lists of characters.

Example for "X-CSQA-en":
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': ['T',
   'h',
   'e',
   ' ',
   'd',
   'e',
   'n',
   't',
   'a',
   'l',
   ' ',
   'o',
   'f',
   'f',
   'i',
   'c',
   'e',
   ' ',
   'h',
   'a',
   'n',
   'd',
   'l',
   'e',
   'd',
   ' ',
   'a',
   ' ',
   'l',
   'o',
   't',
   ' ',
   'o',
   'f',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'w',
   'h',
   'o',
   ' ',
   'e',
   'x',
   'p',
   'e',
   'r',
   'i',
   'e',
   'n',
   'c',
   'e',
   'd',
   ' ',
   't',
   'r',
   'a',
   'u',
   'm',
   'a',
   't',
   'i',
   'c',
   ' ',
   'm',
   'o',
   'u',
   't',
   'h',
   ' ',
   'i',
   'n',
   'j',
   'u',
   'r',
   'y',
   ',',
   ' ',
   'w',
   'h',
   'e',
   'r',
   'e',
   ' ',
   'w',
   'e',
   'r',
   'e',
   ' ',
   't',
   'h',
   'e',
   's',
   'e',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'c',
   'o',
   'm',
   'i',
   'n',
   'g',
   ' ',
   'f',
   'r',
   'o',
   'm',
   '?'],
  'choices': [{'label': ['A'], 'text': ['t', 'o', 'w', 'n']},
   {'label': ['B'], 'text': ['m', 'i', 'c', 'h', 'i', 'g', 'a', 'n']},
   {'label': ['C'], 'text': ['h', 'o', 's', 'p', 'i', 't', 'a', 'l']},
   {'label': ['D'], 'text': ['s', 'c', 'h', 'o', 'o', 'l', 's']},
   {'label': ['E'],
    'text': ['o',
     'f',
     'f',
     'i',
     'c',
     'e',
     ' ',
     'b',
     'u',
     'i',
     'l',
     'd',
     'i',
     'n',
     'g']}]},
 'answerKey': 'C'}

## Steps to reproduce the bug
```python
ds = load_dataset("datasets/xcsr", "X-CSQA-en", split="validation", streaming=True)
item = next(iter(ds))
item
```

## Expected results
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': 'The dental office handled a lot of patients who experienced traumatic mouth injury, where were these patients coming from?',
  'choices': {'label': ['A', 'B', 'C', 'D', 'E'],
   'text': ['town', 'michigan', 'hospital', 'schools', 'office building']}},
 'answerKey': 'C'}
```


Labels: name: dataset bug
Hendrycks Checksum

Hi, 
The checksum for [hendrycks_test](https://huggingface.co/datasets/hendrycks_test) does not compare correctly, I guess it has been updated on the remote.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://people.eecs.berkeley.edu/~hendrycks/data.tar']
```


Labels: name: dataset bug

Thanks for reporting, @DanielHesslow. We are fixing it. 

@albertvillanova thanks for taking care of this so quickly!

The dataset metadata is fixed. You can download it normally.
Fix tar extraction vuln

Fix for CVE-2007-4559

Description:
Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.

I fixed it by using the solution proposed in https://stackoverflow.com/questions/10060069/safely-extract-zip-or-tar-using-python

It blocks extraction of files with an absolute path or double dots and symlinks.


_The documentation is not available anymore as the PR was closed or merged._
Fast dataset iter

Use `pa.Table.to_reader` to make iteration over examples/batches faster in `Dataset.{__iter__, map}`

TODO:
* [x] benchmarking (the only benchmark for now - iterating over (single) examples of `bookcorpus` (75 mil examples) in Colab is approx. 2.3x faster)
* [x] check if iterating over bigger chunks + slicing to fetch individual examples in `_iter` yields better performance





_The documentation is not available anymore as the PR was closed or merged._

I ran some benchmarks (focused on the data fetching part of `__iter__`) and it seems like the combination `table.to_reader(batch_size)` + `RecordBatch.slice` performs the best ([script](https://gist.github.com/mariosasko/0248288a2e3a7556873969717c1fe52b) with the results). I think we can choose (implicit) `batch_size=10` in the final implementation to avoid having problems with fetching large examples.
Update cats_vs_dogs size

It seems like 12 new examples have been added to the `cats_vs_dogs`. This PR updates the size in the card and the info file to avoid a verification error (reported by @stevhliu). 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3878). All of your documentation changes will be reflected on that endpoint.

Maybe `NonMatchingSplitsSizesError` errors should also tell the user to try using a more recent version of the dataset to get the fixes ?

@lhoestq Good idea. Will open a new PR to improve the error messages of NonMatchingSplitsSizesError, NonMatchingChecksumsError, ...

It seems there is still a problem. I am using datasets version 2.5.1. 
I just typed `ds = load_dataset("cats_vs_dogs")` and get the error below.

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=3893603, num_examples=23422, dataset_name='cats_vs_dogs'), 'recorded': SplitInfo(name='train', num_bytes=3891612, num_examples=23410, dataset_name='cats_vs_dogs')}]
```
It looks like the dataset still only has 23,410 examples....


Thanks for reporting, I opened https://github.com/huggingface/datasets/pull/5047
"no-token" is passed to `huggingface_hub` when token is `None`

## Describe the bug

In the 2 lines listed below, a token is passed to `huggingface_hub` to get information from a dataset. If no token is provided, a "no-token" string is passed. What is the purpose of it ? If no real, I would prefer if the `None` value could be sent directly to be handle by `huggingface_hub`. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753
https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

## Expected results
Pass `token=None` to `huggingface_hub`.

## Actual results
`token="no-token"` is passed.

## Environment info
`huggingface_hub v0.10.0dev`

Labels: name: bug

Hi @Wauplin, thanks for raising this potential issue.

The choice of passing `"no-token"` instead of `None` was made in this PR:
- #4536 

According to the PR description, the reason why it is passed is to avoid that `HfApi.dataset_info` uses the local token when no token should be used.

Hi @albertvillanova , thanks for finding the original issue :+1: 

As of next release of `huggingface_hub`, the `token` argument will be deprecated in favor of the `use_auth_token` argument in `dataset_info` method. This change as been done by @SBrandeis in https://github.com/huggingface/huggingface_hub/pull/928. `use_auth_token` is a bit different and allow the case "don't sent the cached token by default".

If you want to strictly avoid sending the cached token from `datasets`, you can use:
```py
# token=token if token else "no-token", <- will fail because token is not valid

use_auth_token=token if token else False,  # using the new `use_auth_token` parameter
```

And as a note, I am currently updating the "don't send the cached token by default"-rule to "don't send the cached token on public repos by default but use it in private ones" in https://github.com/huggingface/huggingface_hub/pull/1064. This will not change the fact that `use_auth_token=False` doesn't send the token at all.


What is current strategy in term of updating `huggingface_hub` version in `datasets` ? I don't want to break stuff in the next release so let's find a proper solution :) 

As soon as `token` is deprecated and hfh has a new release, we'll update `datasets` to use the new argument instead. Does it sound good to you ?

Perfect :ok_hand: 

Hi @Wauplin, thanks for the warning about the deprecation of `token` in favor of `use_auth_token`.

Indeed, in datasets we use internally `use_auth_token`, which in this case was transformed to `token` to call `HfApi.dataset_info`:
https://github.com/huggingface/datasets/blob/1a9385d7cc8a3241b44015145ef56a230fdadc51/src/datasets/load.py#L747

Therefore, for the new hfh release, the fix will be trivial: we will pass directly `use_auth_token`.

As discussed during our meeting yesterday, due to the fact that at datasets we support multiple hfh versions, I think we should handle passing `token` or `use_auth_token` depending on the hfh version.
Support hfh 0.10 implicit auth

In huggingface-hub 0.10 the `token` parameter is deprecated for dataset_info and list_repo_files in favor of use_auth_token.
Moreover if use_auth_token=None then the user's token is used implicitly.

I took those two changes into account

Close https://github.com/huggingface/datasets/issues/4990

TODO:
- [x] fix tests

We should wait hfh 0.10 to be relased first to make sure it works correctly before merging


_The documentation is not available anymore as the PR was closed or merged._

@lhoestq it is now released so you can move forward with it :) 

I took your comments into account @Wauplin :)
I also bumped the requirement to 0.2.0 because we're using `set_access_token`

cc @albertvillanova WDYT ? I edited the CI job to also check for our minimum supported version of hfh at the same time as the minimum pyarrow version

@lhoestq great, thanks ! :)
Update license to bookcorpus dataset card

Not entirely sure, following the links here, but it seems the relevant license is at https://github.com/soskek/bookcorpus/blob/master/LICENSE

Labels: name: dataset contribution

The smashwords ToS apply for this dataset, we did the same for https://github.com/huggingface/datasets/pull/3525

_The documentation is not available anymore as the PR was closed or merged._
Add oversampling strategy iterable datasets interleave

Hello everyone,
Following the issue #4893 and the PR #4831, I propose here an oversampling strategy for a `IterableDataset` list.  
The `all_exhausted` strategy stops building the new dataset as soon as all samples in each dataset have been added at least once. 
It follows roughly the same logic behind #4831, namely:
- if ``probabilities`` is `None` and the strategy is `all_exhausted`, it simply performs a round robin interleaving that stops when the longest dataset is out of samples. Here the new dataset length will be $maxLengthDataset*nbDataset$.
- if ``probabilities`` is not `None` and the strategy is `all_exhausted`, it keeps trace of the datasets which were out of samples but continues to add them to the new dataset, and stops as soons as every dataset runs out of samples at least once.

In order to be consistent and also to align with the `Dataset` behavior, please note that the behavior of the default strategy (`first_exhausted`) has been changed. Namely, it really stops when a dataset is out of samples whereas it used to stop when receiving the `StopIteration` error.  
To give an example of the last note, with the following snippet:
```
>>> from tests.test_iterable_dataset import *
>>> d1 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [0, 1, 2]])), {}))
>>> d2 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [10, 11, 12, 13]])), {}))
>>> d3 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [20, 21, 22, 23, 24]])), {}))
>>> dataset = interleave_datasets([d1, d2, d3])
>>> [x["a"] for x in dataset]
```
The result here will then be `[10, 0, 11, 1, 2]` instead of `[10, 0, 11, 1, 2, 20, 12, 13]`.

I modified the behavior because I found it to be consistent with the under/oversampling approach and because it unified the undersampling and oversampling code, but I stay open to any suggestions.



_The documentation is not available anymore as the PR was closed or merged._
Add MANtIS dataset

This PR adds MANtIS dataset.
Arxiv: [https://arxiv.org/abs/1912.04639](https://arxiv.org/abs/1912.04639)
Github: [https://github.com/Guzpenha/MANtIS](https://github.com/Guzpenha/MANtIS)

README and dataset tags are WIP.

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._

Thanks for your contribution, @bhavitvyamalik. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
datasets(RWTH-PHOENIX-Weather 2014 T): add initial loading script

This is the first sign language dataset in this repo as far as I know.
Following an old issue I opened https://github.com/huggingface/datasets/issues/302.

I added the dataset official REAMDE file, but I see it's not very standard, so it can be removed.



Labels: name: dataset contribution

This might be interesting to @kayoyin the author of https://github.com/kayoyin/transformer-slt – pinging you just in case :)

Thanks Amit, this is a great idea! I'm thinking of porting the SLT models from my paper here as well, having this dataset would be perfect for that :)

Thanks for your contribution, @AmitMY. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
`Dataset.unique` showing wrong output after filtering

## Describe the bug
After filtering a dataset, and if no samples remain, `Dataset.unique` will return the unique values of the unfiltered dataset.

## Steps to reproduce the bug
```python
from datasets import Dataset

dataset = Dataset.from_dict({'id': [0]})
dataset = dataset.filter(lambda _: False)
print(dataset.unique('id'))
```

## Expected results
The above code should return an empty list since the dataset is empty.

## Actual results
```bash
[0]
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.18.19-100.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.14
- PyArrow version: 7.0.0
- Pandas version: 1.3.5

Labels: name: bug

Hi! It seems like `flatten_indices` (called in `unique`) doesn't know how to handle empty indices mappings. I'm working on the fix.

Thanks, that was fast!
Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`

**Is your feature request related to a problem? Please describe.**

So this is more a readability improvement rather than a proposal, wouldn't it be better to use `assertTupleEqual` over the tuples rather than `assertEqual`? As `unittest` added that function in `v3.1`, as detailed at https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual, so maybe it's worth updating.

Find an example of an `assertEqual` over a tuple in 🤗 `datasets` unit tests over an `ArrowDataset` at https://github.com/huggingface/datasets/blob/0bb47271910c8a0b628dba157988372307fca1d2/tests/test_arrow_dataset.py#L570

**Describe the solution you'd like**

Start slowly replacing all the `assertEqual` statements with `assertTupleEqual` if the assertion is done over a Python tuple, as we're doing with the Python lists using `assertListEqual` rather than `assertEqual`.

**Additional context**

If so, please let me know and I'll try to go over the tests and create a PR if applicable, otherwise, if you consider this should stay as `assertEqual` rather than `assertSequenceEqual` feel free to close this issue! Thanks 🤗 


Labels: name: enhancement

Hi! If the only goal is to improve readability, it's better to use `assertTupleEqual` than `assertSequenceEqual` for Python tuples. Also, note that this function is called internally by `assertEqual`, but I guess we can accept a PR to be more verbose.

Hi @mariosasko, right! I'll update the issue title/desc with `assertTupleEqual` even though as you said it seems to be internally using `assertEqual` so I'm not sure whether it's worth it or not...

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual

I thought we were supposed to move gradually from `unittest` to `pytest`...
Fix bug with labels of eurlex config of lex_glue dataset

Fix for a critical bug in the EURLEX dataset label list to make LexGLUE EURLEX results replicable. 

In LexGLUE (Chalkidis et al., 2022), the following is mentioned w.r.t. EUR-LEX: _"It supports four different label granularities, comprising 21, 127, 567, 7390 EuroVoc concepts, respectively. We use the 100 most frequent concepts from level 2 [...]”._ The current label list has all 127 labels, which leads to different (lower) results, as communicated by users.

Thanks!

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._

@JamesLYC88 here is the fix! Thanks again!

Thanks, @albertvillanova. When do you expect that this change will take effect when someone downloads the dataset?

The change is immediately available now, since this change we made to our library:
- #4059