jamesLeeeeeee/datasets-github-issues huggingface dataset:
Text strings are split into lists of characters in xcsr dataset
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
Text strings are split into lists of characters.

Example for "X-CSQA-en":
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': ['T',
   'h',
   'e',
   ' ',
   'd',
   'e',
   'n',
   't',
   'a',
   'l',
   ' ',
   'o',
   'f',
   'f',
   'i',
   'c',
   'e',
   ' ',
   'h',
   'a',
   'n',
   'd',
   'l',
   'e',
   'd',
   ' ',
   'a',
   ' ',
   'l',
   'o',
   't',
   ' ',
   'o',
   'f',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'w',
   'h',
   'o',
   ' ',
   'e',
   'x',
   'p',
   'e',
   'r',
   'i',
   'e',
   'n',
   'c',
   'e',
   'd',
   ' ',
   't',
   'r',
   'a',
   'u',
   'm',
   'a',
   't',
   'i',
   'c',
   ' ',
   'm',
   'o',
   'u',
   't',
   'h',
   ' ',
   'i',
   'n',
   'j',
   'u',
   'r',
   'y',
   ',',
   ' ',
   'w',
   'h',
   'e',
   'r',
   'e',
   ' ',
   'w',
   'e',
   'r',
   'e',
   ' ',
   't',
   'h',
   'e',
   's',
   'e',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'c',
   'o',
   'm',
   'i',
   'n',
   'g',
   ' ',
   'f',
   'r',
   'o',
   'm',
   '?'],
  'choices': [{'label': ['A'], 'text': ['t', 'o', 'w', 'n']},
   {'label': ['B'], 'text': ['m', 'i', 'c', 'h', 'i', 'g', 'a', 'n']},
   {'label': ['C'], 'text': ['h', 'o', 's', 'p', 'i', 't', 'a', 'l']},
   {'label': ['D'], 'text': ['s', 'c', 'h', 'o', 'o', 'l', 's']},
   {'label': ['E'],
    'text': ['o',
     'f',
     'f',
     'i',
     'c',
     'e',
     ' ',
     'b',
     'u',
     'i',
     'l',
     'd',
     'i',
     'n',
     'g']}]},
 'answerKey': 'C'}

## Steps to reproduce the bug
```python
ds = load_dataset("datasets/xcsr", "X-CSQA-en", split="validation", streaming=True)
item = next(iter(ds))
item
```

## Expected results
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': 'The dental office handled a lot of patients who experienced traumatic mouth injury, where were these patients coming from?',
  'choices': {'label': ['A', 'B', 'C', 'D', 'E'],
   'text': ['town', 'michigan', 'hospital', 'schools', 'office building']}},
 'answerKey': 'C'}
```
Split is inferred from filename and overrides metadata.jsonl
Author: float-trip | Pull request: no | State: closed

## Describe the bug

Including the strings "test" or "train" anywhere in a filename causes `datasets` to infer the split and silently ignore all other files.

This behavior is documented for directory names but not filenames: https://huggingface.co/docs/datasets/image_dataset#imagefolder

## Steps to reproduce the bug
`metadata.jsonl`
```json
{"file_name": "photo of a cat.jpg", "text": "a photo of a cat"}
{"file_name": "photo of a dog.jpg", "text": "a photo of a dog"}
{"file_name": "photo of a train.jpg", "text": "a photo of a train"}
{"file_name": "photo of test tubes.jpg", "text": "a photo of test tubes"}
```

`bug.py`
```python
from datasets import load_dataset

dataset = load_dataset("dataset")

print(dataset)
# DatasetDict({
#     train: Dataset({
#         features: ['image', 'text'],
#         num_rows: 1
#     })
#     test: Dataset({
#         features: ['image', 'text'],
#         num_rows: 1
#     })
# })

for split in dataset:
    for n in dataset[split]:
        print(n['text'])
        # a photo of a train
        # a photo of test tubes
```

## Expected results
One single dataset with all four images / a warning for unused files / documentation of this behavior

## Actual results
Only the images with "test" or "train" in the name are loaded

## Environment info
- `datasets` version: 2.5.1
- Platform: macOS-12.5.1-x86_64-i386-64bit
- Python version: 3.10.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.0
Fix URLs of sbu_captions dataset
Author: donglixp | Pull request: yes | State: closed

Forbidden
You don't have permission to access /~vicente/sbucaptions/sbu-captions-all.tar.gz on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5 mod_perl/2.0.11 Perl/v5.16.3 Server at [www.cs.virginia.edu](mailto:csroot@virginia.edu) Port 443
Update swiss judgment prediction
Author: JoelNiklaus | Pull request: yes | State: closed

Hi,

I updated the dataset to include additional data made available recently. When I test it locally, it seems to work. However, I get the following error with the dummy data creation: 
`Dummy data generation done but dummy data test failed since splits ['train', 'validation', 'test'] have 0 examples for config 'fr'`. Do you know why this could be the case?

Cheers,
Joel
Create all YAML dataset_info
Author: lhoestq | Pull request: yes | State: closed

Following https://github.com/huggingface/datasets/pull/4926

Creates all the `dataset_info` YAML fields in the dataset cards
The JSON are also updated using the simplified backward compatible format added in https://github.com/huggingface/datasets/pull/4926

Needs https://github.com/huggingface/datasets/pull/4926 to be merged first
xcsr: X-CSQA simply uses english for all alleged non-english data
Author: thesofakillers | Pull request: no | State: closed

## Describe the bug
All the alleged non-english subcollections for the X-CSQA task in the [xcsr benchmark dataset ](https://huggingface.co/datasets/xcsr) seem to be copies of the english subcollection, rather than translations. This is in contrast to the data description:

> we automatically translate the original CSQA and CODAH datasets, which only have English versions, to 15 other languages, forming development and test sets for studying X-CSR
 
## Steps to reproduce the bug
```python
# let's say you want to load the french X-CSQA subcollection
french = datasets.load_dataset("xcsr", "X-CSQA-fr")
# for good measure, let's load english too
english = datasets.load_dataset("xcsr", "X-CSQA-en")

# let's inspect
"".join(english['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
"".join(french['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
# what? Why are they both in english?

# I've checked this for validation and train splits too, across many datapoints. It's all the same english dataset
# maybe i need to look better?
french['test'].unique('lang')
# output: ['en']
# no, it's all english

```

## Expected results
Accessing a subcollection in language X should return a subcollection containg samples in language X

## Actual results
Accessing a subcollection in language X returns a subcollection containing samples in English.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 2.5.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
Fix tar extraction vuln
Author: lhoestq | Pull request: yes | State: closed

Fix for CVE-2007-4559

Description:
Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.

I fixed it by using the solution proposed in https://stackoverflow.com/questions/10060069/safely-extract-zip-or-tar-using-python

It blocks extraction of files with an absolute path or double dots and symlinks.
Transfer dataset scripts to Hub
Author: albertvillanova | Pull request: no | State: closed

Before merging:
- #4974 

TODO:
- [x] Create label: ["dataset contribution"](https://github.com/huggingface/datasets/pulls?q=label%3A%22dataset+contribution%22)
- [x] Create project: [Datasets: Transfer datasets to Hub](https://github.com/orgs/huggingface/projects/22/)
- [x] PRs:
  - [x] Add dataset: we should recommend transfer all additions of datasets to the Hub, under the appropriate namespace; no more additions of datasets on GitHub
  - [x] Update dataset: in general, we should merge bug fixes; enhancements should be considered on a case-by-case basis, depending on whether there is a more suitable namespace on the Hub
- [ ] Issues

Finally:
- [x] #4974

Let me know what you think! :hugs:
Force JSON format regardless of file naming on S3
Author: junwang-wish | Pull request: no | State: closed

I have a file on S3 created by Data Version Control, it looks like `s3://dvc/ac/badff5b134382a0f25248f1b45d7b2` but contains a json file. If I run 
```python
dataset = load_dataset(
    "json",
    data_files='s3://dvc/ac/badff5b134382a0f25248f1b45d7b2' 
)
```
It gives me 
```
InvalidSchema: No connection adapters were found for 's3://dvc/ac/badff5b134382a0f25248f1b45d7b2'
```
However, I cannot go ahead and change the names of the s3 file. Is there a way to "force" load a S3 url with certain decoder (JSON, CSV, etc.) regardless of s3 URL naming?
Add deprecation warning to multilingual_librispeech dataset card
Author: albertvillanova | Pull request: yes | State: closed

Besides the current deprecation warning in the script of `multilingual_librispeech`, this PR adds a deprecation warning to its dataset card as well.

The format of the deprecation warning is aligned with the one in the library documentation when docstrings contain the `<Deprecated/>` tag.

Related to:
- #4060
Release 2.5.0 breaks transformers CI
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported by @lhoestq:

> see https://app.circleci.com/pipelines/github/huggingface/transformers/47634/workflows/b491886b-e66e-4edb-af96-8b459e72aa25/jobs/564563
this is used here: [https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55[…]torch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py](https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55250e7da/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py#L482-L488)
Add EmptyDatasetError
Author: lhoestq | Pull request: yes | State: closed

examples:

from the hub:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("lhoestq/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1171, in dataset_module_factory
    raise e1 from None
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1162, in dataset_module_factory
    download_mode=download_mode,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 760, in get_module
    else get_data_patterns_in_dataset_repository(hfh_dataset_info, self.data_dir)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 678, in get_data_patterns_in_dataset_repository
    ) from None
datasets.data_files.EmptyDatasetError: The dataset repository at 'lhoestq/empty' doesn't contain any data file.
```

from local directory:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("playground/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1107, in dataset_module_factory
    path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 625, in get_module
    else get_data_patterns_locally(base_path)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 460, in get_data_patterns_locally
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data file") from None
datasets.data_files.EmptyDatasetError: The directory at playground/empty doesn't contain any data file
```

Close https://github.com/huggingface/datasets/issues/4995
Don't add a tag on the Hub on release
Author: lhoestq | Pull request: yes | State: closed

Datasets with no namespace on the Hub have tags to redirect to the version of datasets where they come from.
I’m about to remove them all because I think it looks bad/unexpected in the UI and it’s not actually useful

Therefore I'm also disabling tagging.

Note that the CI job will be completely removed in https://github.com/huggingface/datasets/pull/4974 anyway
Add support for parsing JSON files in array form
Author: mariosasko | Pull request: yes | State: closed

Support parsing JSON files in the array form (top-level object is an array). For simplicity, `json.load` is used for decoding. This means the entire file is loaded into memory. If requested, we can optimize this by introducing a param similar to `lines` in [`pandas.read_json`](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html), which, if set to `True`, would allow us to read in chunks.

Fixes https://github.com/huggingface/datasets/issues/4963
Dataset Viewer issue for Jean-Baptiste/wikiner_fr
Author: severo | Pull request: no | State: closed

### Link

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr

### Description

```
Error code:   StreamingRowsError
Exception:    FileNotFoundError
Message:      [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 77, in get_rows
                 rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 718, in __iter__
                 for key, example in self._iter():
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 708, in _iter
                 yield from ex_iterable
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 112, in __iter__
                 yield from self.generate_examples_fn(**self.kwargs)
               File "/tmp/modules-cache/datasets_modules/datasets/Jean-Baptiste--wikiner_fr/683a580ba6ec769d508f7dfc603a651667b0ed3817b1ae5bfd45f97cc024923f/wikiner_fr.py", line 165, in _generate_examples
                 dataset = Dataset.load_from_disk(filepath)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1210, in load_from_disk
                 with open(Path(dataset_path, config.DATASET_STATE_JSON_FILENAME).as_posix(), encoding="utf-8") as state_file:
             FileNotFoundError: [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
```

Is it an error with the dataset script, or the data itself, @huggingface/datasets?

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/tree/main 

### Owner

No
Get a specific Exception when the dataset has no data
Author: severo | Pull request: no | State: closed

In the dataset viewer on the Hub (https://huggingface.co/datasets/glue/viewer), we would like (https://github.com/huggingface/moon-landing/issues/3882) to show a specific message when the repository lacks any data files.

In that case, instead of showing a complex traceback, we want to show a call to action to help the user upload data.

To do that, it would be very helpful to know for sure that the repository is missing any (supported) data files.

It could be done by raising a custom exception, for example, `NoDataError`.
delete the hardcoded license list in `datasets`
Author: julien-c | Pull request: no | State: closed

> Feel free to delete the license list in `datasets` [...]
>
> Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238401662_

> [...], in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

_Originally posted by @julien-c in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238390659_
fix: avoid casting tuples after Dataset.map
Author: szmoro | Pull request: yes | State: closed

This PR updates features.py to avoid casting tuples to lists when reading the results of Dataset.map as suggested by @lhoestq [here](https://github.com/huggingface/datasets/issues/4676#issuecomment-1187371367) in https://github.com/huggingface/datasets/issues/4676.
Fix missing tags in dataset cards
Author: albertvillanova | Pull request: yes | State: closed

Fix missing tags in dataset cards:
- aeslc
- empathetic_dialogues
- event2Mind
- gap
- iwslt2017
- newsgroup
- qa4mre
- scicite

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
- #4979
"no-token" is passed to `huggingface_hub` when token is `None`
Author: Wauplin | Pull request: no | State: closed

## Describe the bug

In the 2 lines listed below, a token is passed to `huggingface_hub` to get information from a dataset. If no token is provided, a "no-token" string is passed. What is the purpose of it ? If no real, I would prefer if the `None` value could be sent directly to be handle by `huggingface_hub`. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753
https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

## Expected results
Pass `token=None` to `huggingface_hub`.

## Actual results
`token="no-token"` is passed.

## Environment info
`huggingface_hub v0.10.0dev`
Running add_column() seems to corrupt existing sequence-type column info
Author: derek-rocheleau | Pull request: no | State: closed

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.
[doc] Fix broken snippet that had too many quotes
Author: tomaarsen | Pull request: yes | State: closed

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen
docs: ✏️ add links to the Datasets API
Author: severo | Pull request: yes | State: closed

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.
How to convert  torch.utils.data.Dataset to huggingface dataset?
Author: DEROOCE | Pull request: no | State: closed

I look through the huggingface dataset docs, and it seems that there is no offical support function to  convert  `torch.utils.data.Dataset`  to huggingface dataset. However, there is a way to convert huggingface dataset to  `torch.utils.data.Dataset`, like below:
```python
from datasets import Dataset
data = [[1, 2],[3, 4]]
ds = Dataset.from_dict({"data": data})
ds = ds.with_format("torch")
ds[0]
ds[:2]
```
So is there something I miss, or there IS no function to convert  `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert?
Thanks.
Fix missing tags in dataset cards
Author: albertvillanova | Pull request: yes | State: closed

Fix missing tags in dataset cards:
- amazon_us_reviews
- art
- discofuse
- indic_glue
- ubuntu_dialogs_corpus

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
Providing dataset size
Author: sashavor | Pull request: no | State: open

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq
Hope to adapt Python3.9 as soon as possible
Author: RedHeartSecretMan | Pull request: no | State: open

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.
Add `fn_kwargs` param to `IterableDataset.map`
Author: mariosasko | Pull request: yes | State: closed

Add the `fn_kwargs` parameter to `IterableDataset.map`.

("Resolves" https://discuss.huggingface.co/t/how-to-use-large-image-text-datasets-in-hugging-face-hub-without-downloading-for-free/22780/3)