kardwalker/my_dataset huggingface dataset:
GitHub issue #6334: datasets.filesystems: fix is_remote_filesystems
State: closed | Created: 2023-10-23T09:17:54Z

Close #6330, close #6333.

`fsspec.implementations.LocalFilesystem.protocol`
was changed from `str` "file" to `tuple[str,...]` ("file", "local") in `fsspec>=2023.10.0`

This commit supports both styles.
GitHub issue #6333: Support fsspec 2023.10.0
State: closed | Created: 2023-10-23T09:14:53Z

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:
- #6331

Related to issue:
- #6330

As @ZachNagengast suggested, the issue might be related to:
- https://github.com/fsspec/filesystem_spec/pull/1381
GitHub issue #6332: Replace deprecated license_file in setup.cfg
State: closed | Created: 2023-10-23T09:05:26Z

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
```
GitHub issue #6331: Temporarily pin fsspec < 2023.10.0
State: closed | Created: 2023-10-23T08:51:50Z

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```
GitHub issue #6330: Latest fsspec==2023.10.0 issue with streaming datasets
State: closed | Created: 2023-10-22T20:57:10Z

### Describe the bug

Loading a streaming dataset with this version of fsspec fails with the following error:

`NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.`

I suspect the issue is with this PR 

https://github.com/fsspec/filesystem_spec/pull/1381

### Steps to reproduce the bug

1. Upgrade fsspec to version `2023.10.0`
2. Attempt to load a streaming dataset e.g. `load_dataset("laion/gpt4v-emotion-dataset", split="train", streaming=True)`
3. Observe the following exception:

```
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/load.py", line 2146, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/builder.py", line 1318, in as_streaming_dataset
    raise NotImplementedError(
NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
```

### Expected behavior

Should stream the dataset as normal.

### Environment info

datasets@main
fsspec==2023.10.0
GitHub issue #6327: FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`
State: closed | Created: 2023-10-21T12:27:03Z

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`
GitHub issue #6324: Conversion to Arrow fails due to wrong type heuristic
State: closed | Created: 2023-10-20T23:20:58Z

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values. 

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.


### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
GitHub issue #6323: Loading dataset from large GCS bucket very slow since 2.14
State: open | Created: 2023-10-20T12:59:55Z

### Describe the bug

Since updating to >2.14 we have very slow access to our parquet files on GCS when loading a dataset (>30 min vs 3s). Our GCS bucket has many objects and resolving globs is very slow. I could track down the problem to this change:
https://github.com/huggingface/datasets/blame/bade7af74437347a760830466eb74f7a8ce0d799/src/datasets/data_files.py#L348 
The underlying implementation with gcsfs is really slow. Could you go back to the old way if we are simply giving the parquet files and no glob pattern?

Thank you.




### Steps to reproduce the bug

Load a dataset from a GCS bucket that has many files.

### Expected behavior

Used to be fast (3s) in 2.13

### Environment info

datasets==2.14.5
GitHub issue #6322: Fix regex `get_data_files` formatting for base paths
State: closed | Created: 2023-10-19T19:45:10Z

With this pr https://github.com/huggingface/datasets/pull/6309, it is formatting the entire base path into regex, which results in the undesired formatting error `doesn't match the pattern` because of the line in `glob_pattern_to_regex`: `.replace("//", "/")`:
- Input: `hf://datasets/...`
- Output: `hf:/datasets/...`

This fix will only convert the `split_pattern` to regex and keep the `base_path` unchanged.

cc @albertvillanova hopefully this still works with your implementation
GitHub issue #6320: Dataset slice splits can't load training and validation at the same time
State: closed | Created: 2023-10-19T16:09:22Z

### Describe the bug

According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:

`train_test_ds = datasets.load_dataset("bookcorpus", split="train+test")`

to load the train and test sets from the dataset. 

However executing the equivalent code:

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

only yields the following output:

> Dataset({
>     features: ['file', 'audio', 'label'],
>     num_rows: 54175
> })

Where loading the dataset without the split argument yields:

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     validation: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 6798
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

Thus, the API seems to be broken in this regard. 

This is a bit annoying since I want to be able to use the split argument with `split="train[:10%]+test[:10%]"` to have smaller dataset to work with when validating my model is working correctly. 

### Steps to reproduce the bug

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

### Expected behavior

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

### Environment info

```
import datasets
print(datasets.__version__)
```
> 2.14.5

```
import sys
print(sys.version)
```
> 3.9.17 (main, Jul  5 2023, 20:41:20) 
> [GCC 11.2.0]
GitHub issue #6319: Datasets.map is severely broken
State: open | Created: 2023-10-19T12:19:33Z

### Describe the bug

Regardless of how many cores I used, I have 16 or 32 threads, map slows down to a crawl at around 80% done, lingers maybe until 97% extremely slowly and NEVER finishes the job. It just hangs.

After watching this for 27 hours I control-C out of it. Until the end one process appears to be doing something, but it never ends.

I saw some comments about fast tokenizers using Rust and all and tried different variations. NOTHING works.



### Steps to reproduce the bug

Running it without breaking the dataset into parts results in the same behavior. The loop was an attempt to see if this was a RAM issue. 

for idx in range(100):
    dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
    dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False, num_proc=1, remove_columns=["text", "meta"])
    dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")


### Expected behavior

I expect map to run at more or less the same speed it starts with and FINISH its processing. 

### Environment info

Python 3.8, same with 3.10 makes no difference.
Ubuntu 20.04,
GitHub issue #6318: Deterministic set hash
State: closed | Created: 2023-10-19T12:19:13Z

Sort the items in a set according to their `datasets.fingerprint.Hasher.hash` hash to get a deterministic hash of sets.

This is useful to get deterministic hashes of tokenizers that use a trie based on python sets.

reported in https://github.com/huggingface/datasets/issues/3847
GitHub issue #6317: sentiment140 dataset unavailable
State: closed | Created: 2023-10-19T11:25:21Z

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3    
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
GitHub issue #6316: Fix loading Hub datasets with CSV metadata file
State: closed | Created: 2023-10-19T10:21:34Z

Currently, the reading of the metadata file infers the file extension (.jsonl or .csv) from the passed filename. However, downloaded files from the Hub don't have file extension. For example:
- the original file: `hf://datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5916a4-16977085077831/metadata.jsonl`
- corresponds to the downloaded path: `/tmp/pytest-of-username/pytest-46/cache/datasets/downloads/9f5374dbb470f711f6b89d66a5eec1f19cc96324b26bcbebe29138bda6cb20e6`, which does not have extension

In the case where the metadata file does not have an extension, the reader assumes it is a JSONL file, thus the reported error when trying to read a CSV file as a JSONL one: `ArrowInvalid: JSON parse error: Invalid value. in row 0`

This behavior was introduced by:
- #4837

This PR extracts the metadata file extension from the original filename (instead of the downloaded one) and passes it as a parameter to the read_metadata function.

Fix #6315.
GitHub issue #6315: Hub datasets with CSV metadata raise ArrowInvalid: JSON parse error: Invalid value. in row 0
State: closed | Created: 2023-10-19T10:11:29Z

When trying to load a Hub dataset that contains a CSV metadata file, it raises an `ArrowInvalid` error:
```
E   pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0

pyarrow/error.pxi:100: ArrowInvalid
```

See: https://huggingface.co/datasets/lukarape/public_small_papers/discussions/1
GitHub issue #6313: Fix commit message formatting in multi-commit uploads
State: closed | Created: 2023-10-19T07:53:56Z

Currently, the commit message keeps on adding:

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00000-of-00002) (part 00001-of-00002)`

Introduced in https://github.com/huggingface/datasets/pull/6269

This PR fixes this issue to have

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00001-of-00002)`
GitHub issue #6312: docs: resolving namespace conflict, refactored variable 
State: closed | Created: 2023-10-18T16:10:59Z

In docs of about_arrow.md, in the below example code
![image](https://github.com/huggingface/datasets/assets/74114936/fc70e152-e15f-422e-949a-1c4c4c9aa116)
The variable name 'time' was being used in a way that could potentially lead to a namespace conflict with Python's built-in 'time' module. It is not a good convention and can lead to unintended variable shadowing for any user re-using the example code.
To ensure code clarity, and prevent potential naming conflicts renamed the variable 'time' to 'elapsed_time' in the example  code.
GitHub issue #6310: Add return_file_name in load_dataset
State: closed | Created: 2023-10-17T13:36:57Z

Proposition to fix #5806.

Added an optional parameter `return_file_name` in the dataset builder config. When set to `True`, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:
- for arrow-based: a column is concatenated after the table is cast.
- for folder-based: `dataset.info.features` has the entry `file_name` and the original file name is passed to the `sample_metadata` dictionary. 

The difference in behavior might be a concern, also I do not know whether the `file_name` should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.
GitHub issue #6309: Fix get_data_patterns for directories with the word data twice
State: closed | Created: 2023-10-17T09:00:39Z

Before the fix, `get_data_patterns` inferred wrongly the split name for paths with the word "data" twice:
- For the URL path: `hf://datasets/piuba-bigdata/articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357/data/train-00001-of-00009.parquet` (note the org name `piuba-bigdata/` ending with `data/`)
- The inferred split name was: `articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357/data/train` instead of `train`

This PR fixes this issue by passing the `base_path` (`hf://datasets/piuba-bigdata/articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357`) to `_get_data_files_patterns` and prepending it to the regex split pattern (`data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*\\..*`).

Fix #6305.
Fix https://huggingface.co/datasets/piuba-bigdata/articles_and_comments/discussions/1
GitHub issue #6308: module 'resource' has no attribute 'error'
State: closed | Created: 2023-10-17T08:08:54Z

### Describe the bug

just run import:
`from datasets import load_dataset`

and then:
```

  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_dataset.py", line 66, in <module>
    from .arrow_reader import ArrowReader
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\__init__.py", line 10, in <module>
    from .streaming_download_manager import StreamingDownloadManager
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\streaming_download_manager.py", line 21, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\filesystems\__init__.py", line 8, in <module>
    import fsspec.asyn
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\fsspec\asyn.py", line 157, in <module>
    ResourceEror = resource.error
AttributeError: module 'resource' has no attribute 'error'

Process finished with exit code 1

```


and the error codes are:
```
try:
    import resource
except ImportError:
    resource = None
    ResourceError = OSError
else:
    ResourceEror = resource.error

```
1. miss spelling : "ResourceEror " should be "ResourceErorr"
2. module 'resource' has no attribute 'error'




### Steps to reproduce the bug

only one step:

`from datasets import load_dataset`

### Expected behavior

slove error: module 'resource' has no attribute 'error'

### Environment info

python=3.10

datasets==2.14.5
GitHub issue #6306: pyinstaller : OSError: could not get source code
State: closed | Created: 2023-10-17T01:41:51Z

### Describe the bug

I ran a package with pyinstaller and got the following error：



### Steps to reproduce the bug

```  
  ...
  File "datasets\__init__.py", line 52, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module
  File "datasets\inspect.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module
  File "datasets\load.py", line 58, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module
  File "datasets\packaged_modules\__init__.py", line 31, in <module>
  File "inspect.py", line 1147, in getsource
  File "inspect.py", line 1129, in getsourcelines
  File "inspect.py", line 958, in findsource
OSError: could not get source code
```

### Expected behavior

I have looked up the relevant information, but I can't find a suitable reason

### Environment info

```python
python 3.10
datasets 2.14.4
pyinstaller 5.6.2
```
GitHub issue #6303: Parquet uploads off-by-one naming scheme
State: open | Created: 2023-10-14T18:31:03Z

### Describe the bug

I noticed this numbering scheme not matching up in a different project and wanted to raise it as an issue for discussion, what is the actual proper way to have these stored?

<img width="425" alt="image" src="https://github.com/huggingface/datasets/assets/1981179/3ffa2144-7c9a-446f-b521-a5e9db71e7ce">

The `-SSSSS-of-NNNNN` seems to be used widely across the codebase. The section that creates the part in my screenshot is here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5287
There are also some edits to this section in the single commit branch.

### Steps to reproduce the bug

1. Upload a dataset that requires at least two parquet files in it
2. Observe the naming scheme

### Expected behavior

The couple options here are of course **1. keeping it as is**

**2. Starting the index at 1:**
train-00001-of-00002-{hash}.parquet
train-00002-of-00002-{hash}.parquet

**3. My preferred option** (which would solve my specific issue), dropping the total entirely:
train-00000-{hash}.parquet
train-00001-{hash}.parquet

This also solves an issue that will occur with an `append` variable for `push_to_hub` (see https://github.com/huggingface/datasets/issues/6290) where as you add a new parquet file, you need to rename everything in the repo as well. 

However, I know there are parts of the repo that use 0 as the starting file or may require the total, so raising the question for discussion.


### Environment info

- `datasets` version: 2.14.6.dev0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.18.0
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue #6302: ArrowWriter/ParquetWriter `write` method does not increase `_num_bytes` and hence datasets not sharding at `max_shard_size`
State: closed | Created: 2023-10-13T14:43:36Z

### Describe the bug

An example from [1], does not work when limiting shards with `max_shard_size`.

Try the following example with low `max_shard_size`, such as:

```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

The reason for this is that, in line [2] `writer._num_bytes > max_shard_size` is never true, because the `write` method of `ArrowWriter` [3] does not increase `self._num_bytes`.

Such that respective Arrow/Parquet shards are only written to file based on the `writer_batch_size` or `config.DEFAULT_MAX_BATCH_SIZE`, but not based on `max_shard_size`.


[1] https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

[2] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/builder.py#L1677

[3] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/arrow_writer.py#L459

### Steps to reproduce the bug

Get example from: https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

Call `builder.download_and_prepare` with low `max_shard_size` such as `10MB`, e.g.:
```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

### Expected behavior

Shards should be written based on `max_shard_size` instead of batch size.

### Environment info

```
>>> import datasets
>>> datasets.__version__
'2.14.6.dev0
```
GitHub issue #6299: Support for newer versions of JAX
State: closed | Created: 2023-10-12T10:03:46Z

### Feature request

Hi,

I like your idea of adapting the datasets library to be usable with JAX. Thank you for that.

However, in your [setup.py](https://github.com/huggingface/datasets/blob/main/setup.py), you enforce old versions of JAX <= 0.3... It is very cumbersome !

What is the rationale for such a limitation ? Can you remove it please ?

Thanks,

### Motivation

This library is unusable with new versions of JAX ?

### Your contribution

Yes.
GitHub issue #6293: Choose columns to stream parquet data in streaming mode
State: closed | Created: 2023-10-11T08:59:36Z

Currently passing columns= to load_dataset in streaming mode fails

```
Tried to load parquet data with columns '['link']' with mismatching features '{'caption': Value(dtype='string', id=None), 'image': {'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='null', id=None)}, 'link': Value(dtype='string', id=None), 'message_id': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None)}'
```

similar to https://github.com/huggingface/datasets/issues/6039

reported at https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65259a09617407d4520f4ad9
GitHub issue #6290: Incremental dataset (e.g. `.push_to_hub(..., append=True)`)
State: open | Created: 2023-10-10T15:18:03Z

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)
GitHub issue #6287: map() not recognizing "text"
State: closed | Created: 2023-10-09T10:27:30Z

### Describe the bug

The [map() documentation](https://huggingface.co/docs/datasets/v2.14.5/en/package_reference/main_classes#datasets.Dataset.map) reads:
`
ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)`

I have been trying to reproduce it in my code as:

`tokenizedDataset = dataset.map(lambda x: tokenizer(x['text']), batched=True)`

But it doesn't work as it throws the error:

> KeyError: 'text'

Can you please guide me on how to fix it?



### Steps to reproduce the bug

1. `from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi")`

2. Then this code: `from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")`
3. The line I quoted above (which I have been trying)

### Expected behavior

As mentioned in the documentation, it should run without any error and map the tokenization on the whole dataset.

### Environment info

Python 3.10.2
GitHub issue #6286: Create DefunctDatasetError
State: closed | Created: 2023-10-09T09:23:23Z

Create `DefunctDatasetError` as a specific error to be raised when a dataset is defunct and no longer accessible.

See Hub discussion: https://huggingface.co/datasets/the_pile_books3/discussions/7#6523c13a94f3a1a2092d251b
GitHub issue #6285: TypeError: expected str, bytes or os.PathLike object, not dict
State: open | Created: 2023-10-09T04:56:26Z

### Describe the bug

my dataset is in form : train- image /n -labels

and tried the code:
```
from datasets import load_dataset

data_files = {
    "train": "/content/datasets/PotholeDetectionYOLOv8-1/train/",
      "validation": "/content/datasets/PotholeDetectionYOLOv8-1/valid/",
      "test": "/content/datasets/PotholeDetectionYOLOv8-1/test/"
}
dataset = load_dataset("imagefolder", data_dir=data_files)
dataset
```
got error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-29-2ef1926f73d9>](https://localhost:8080/#) in <cell line: 8>()
      6       "test": "/content/datasets/PotholeDetectionYOLOv8-1/test/"
      7 }
----> 8 dataset = load_dataset("imagefolder", data_dir=data_files)
      9 dataset

6 frames
[/usr/lib/python3.10/pathlib.py](https://localhost:8080/#) in _parse_args(cls, args)
    576                 parts += a._parts
    577             else:
--> 578                 a = os.fspath(a)
    579                 if isinstance(a, str):
    580                     # Force-cast str subclasses to str (issue #21127)

TypeError: expected str, bytes or os.PathLike object, not dict
```

### Steps to reproduce the bug

as share above 

### Expected behavior

load images and labels , but my dataset only uploads images 
- https://huggingface.co/datasets/Andyrasika/potholes-dataset

### Environment info

colab pro
GitHub issue #6284: Add Belebele multiple-choice machine reading comprehension (MRC) dataset
State: closed | Created: 2023-10-06T06:58:03Z

### Feature request

Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Please refer to paper for more details, [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884).


## Composition

- 900 questions per language variant
- 488 distinct passages, there are 1-2 associated questions for each.
- For each question, there is 4 multiple-choice answers, exactly 1 of which is correct.
- 122 language/language variants (including English).
- 900 x 122 = 109,800 total questions.


### Motivation

official repo https://github.com/facebookresearch/belebele

### Your contribution

-
GitHub issue #6283: Fix array cast/embed with null values
State: closed | Created: 2023-10-05T15:24:05Z

Fixes issues with casting/embedding PyArrow list arrays with null values. It also bumps the required PyArrow version to 12.0.0 (over 9 months old) to simplify the implementation.

Fix #6280, fix #6311, fix #6360 

(Also fixes https://github.com/huggingface/datasets/issues/5430 to make Beam compatible with PyArrow>=12.0.0)
GitHub issue #6280: Couldn't cast array of type fixed_size_list to Sequence(Value(float64))
State: closed | Created: 2023-10-05T12:48:31Z

### Describe the bug

I have a dataset with an embedding column, when I try to map that dataset I get the following exception:

```
Traceback (most recent call last):
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3189, in map
    for rank, done, content in iflatmap_unordered(
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1387, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1387, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
TypeError: Couldn't cast array of type
fixed_size_list<item: float>[2]
to
Sequence(feature=Value(dtype='float32', id=None), length=2, id=None)
```

### Steps to reproduce the bug

Here's a simple repro script:

```
from datasets import Features, Value, Sequence, ClassLabel, Dataset

dataset_features = Features({
    'text': Value('string'),
    'embedding': Sequence(Value('double'), length=2),
    'categories': Sequence(ClassLabel(names=sorted([
        'one',
        'two',
        'three'
    ]))),
})

dataset = Dataset.from_dict(
    {
        'text': ['A'] * 10000,
        'embedding': [[0.0, 0.1]] * 10000,
        'categories': [[0]] * 10000,
    },
    features=dataset_features
)

def test_mapper(r):
    r['text'] = list(map(lambda t: t + ' b', r['text']))
    return r


dataset = dataset.map(test_mapper, batched=True, batch_size=10, features=dataset_features, num_proc=2)
```

Removing the embedding column fixes the issue!

### Expected behavior

The mapping completes successfully.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
GitHub issue #6279: Batched IterableDataset
State: open | Created: 2023-10-05T11:12:49Z

### Feature request

Hi, 

could you add an implementation of a batched `IterableDataset`. It already support an option to do batch iteration via `.iter(batch_size=...)` but this cannot be used in combination with a torch `DataLoader` since it just returns an iterator.

### Motivation

The current implementation loads each element of a batch individually which can be very slow in cases of a big batch_size. I did some experiments [here](https://discuss.huggingface.co/t/slow-dataloader-with-big-batch-size/57224) and using a batched iteration would speed up data loading significantly.

### Your contribution

N/A
GitHub issue #6278: No data files duplicates
State: closed | Created: 2023-10-05T10:31:58Z

I added a new DataFilesSet class to disallow duplicate data files.
I also deprecated DataFilesList.

EDIT: actually I might just add drop_duplicates=True to `.from_patterns`

close https://github.com/huggingface/datasets/issues/6259
close https://github.com/huggingface/datasets/issues/6272

TODO:
- [ ] tests
- [ ] preserve data files order
GitHub issue #6277: FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.
State: closed | Created: 2023-10-04T22:01:25Z

### Describe the bug

I'm encountering a "FileNotFoundError" while attempting to use the "paws-x" dataset to retrain the DistilRoBERTa-base model. The error message is as follows: 

FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.

### Steps to reproduce the bug

https://colab.research.google.com/drive/11xUUFxloClpmqLvDy_Xxfmo3oUzjY5nx#scrollTo=kUn74FigzhHm

### Expected behavior

The the trained model

### Environment info

colab, "paws-x" dataset , DistilRoBERTa-base model
GitHub issue #6274: FileNotFoundError for dataset with multiple builder config
State: closed | Created: 2023-10-01T23:45:56Z

### Describe the bug

When there is only one config and only the dataset name is entered when using datasets.load_dataset(), it works fine. But if I create a second builder_config for my dataset and enter the config name when using datasets.load_dataset(), the following error will happen.

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

The "XXX.incomplete folder" in the cache folder of my dataset will disappear before "generating test split", which does not happen when config name is not entered and the config name is "default"

C:\Users\chenx\.cache\huggingface\datasets\my_dataset\0_shot_multiple_choice\1.0.0    
The folder that is supposed to remain under the above directory will disappear, and the data generator will not have a place to generate data into. 

### Steps to reproduce the bug

test = load_dataset('my_dataset', '0_shot_multiple_choice')

### Expected behavior

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/chenx/.cache/huggingface/datasets/my_dataset/0_shot_multiple_choice/1.0.0/97c3854a012cfd6b045e3be4c864739902af2d818bb9235b047baa94c302e9a2.incomplete/my_dataset-test-00000-00000-of-NNNNN.arrow'

### Environment info

datasets                  2.14.5
python                    3.8.18
GitHub issue #6273: Broken Link to PubMed Abstracts dataset .
State: open | Created: 2023-10-01T19:08:48Z

### Describe the bug

The link provided for the dataset is broken,
data_files = 
[https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst](url)

The 

### Steps to reproduce the bug

Steps to reproduce:

1) Head over to [https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt#big-data-datasets-to-the-rescue](url)

2) In the Section  "What is the Pile?", you can see a code snippet that contains the broken link.

### Expected behavior

The link should Redirect to the  "PubMed Abstracts dataset" as expected .

### Environment info

.
GitHub issue #6272: Duplicate `data_files` when named `<split>/<split>.parquet`
State: closed | Created: 2023-10-01T15:43:56Z

e.g. with `u23429/stock_1_minute_ticker`

```ipython
In [1]: from datasets import *

In [2]: b = load_dataset_builder("u23429/stock_1_minute_ticker")
Downloading readme: 100%|██████████████████████████| 627/627 [00:00<00:00, 246kB/s]

In [3]: b.config.data_files
Out[3]: 
{NamedSplit('train'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet'],
 NamedSplit('validation'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet'],
 NamedSplit('test'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet']}

```

This bug issue is present in the current `datasets` 2.14.5 and also on `main` even after https://github.com/huggingface/datasets/pull/6244 cc @mariosasko
GitHub issue #6271: Overwriting Split overwrites data but not metadata, corrupting dataset
State: closed | Created: 2023-09-30T22:37:31Z

### Describe the bug

I want to be able to overwrite/update/delete splits in my dataset. Currently the only way to do is to manually go into the dataset and delete the split. If I try to overwrite programmatically I end up in an error state and (somewhat) corrupting the dataset. Read below.

**Current Behavior**
When I push to an existing split I get this error:
`ValueError: Split complexRoofLocation_01Apr2023_to_31May2023test already present` 
This seems to suggest that the library doesn't support overwriting splits. 

**Potential Bug**
What’s strange is that datasets, despite the operation erroring out with the ValueError above, does, in fact, overwrite the split:

`Pushing dataset shards to the dataset hub: 100% [.....................] 1/1 [00:00<00:00, 55.04it/s]` 
Even though you got an error message and your code fails, your dataset is now changed. That seems like a bug. Either don't change the dataset, or don't throw the error and allow the script to proceed. 

Additional Bug
While it overwrites the split, it doesn’t overwrite the split’s information. Because of this when you pull down the dataset you may end up getting a `NonMatchingSplitsSizesError` if the size of the dataset during the overwrite is different. For example, my original split had 5 rows, but on my overwrite, I only had 4. Then when I try to download the dataset, I get a `NonMatchingSplitsSizesError` because the dataset's data.json states there’s 5 but only 4 exist in the split.

Expected Behavior
This corrupts the dataset rendering it unusable (until you take manual intervention). Either the library should let the overwrite happen (which it does but should also update the metadata) or it shouldn’t do anything.

### Steps to reproduce the bug

[Colab Notebook](https://colab.research.google.com/drive/1bqVkD06Ngs9MQNdSk_ygCG6y1UqXA4pC?usp=sharing)


### Expected behavior

The split should be overwritten and I should be able to use the new version of the dataset without issue. 

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue #6269: Reduce the number of commits in `push_to_hub`
State: closed | Created: 2023-09-29T16:22:31Z

Reduces the number of commits in `push_to_hub` by using the `preupload` API from https://github.com/huggingface/huggingface_hub/pull/1699. Each commit contains a maximum of 50 uploaded files.

A shard's fingerprint no longer needs to be added as a suffix to support resuming an upload, meaning the shards' naming scheme is the same as the initial one.

Also, it adds support for the following params: `create_pr`, `commit_message` and `revision` (`branch` deprecated; unlike the previous implementation, this one creates a branch if the branch does not exist to be consistent with `transformers`). 

(Nit) This implementation keeps the markdown section of the generated README.md empty to enable importing the card template (when the card is accessed on the Hub).

Fixes https://github.com/huggingface/datasets/issues/5492, fixes https://github.com/huggingface/datasets/issues/6257, fixes https://github.com/huggingface/datasets/issues/5045, fixes https://github.com/huggingface/datasets/issues/6271

TODO:
- [x] set the minimal version to the next `hfh` release (once it's published)
GitHub issue #6268: Add repo_id to DatasetInfo
State: open | Created: 2023-09-29T10:24:55Z

```python
from datasets import load_dataset

ds = load_dataset("lhoestq/demo1", split="train")
ds = ds.map(lambda x: {}, num_proc=2).filter(lambda x: True).remove_columns(["id"])
print(ds.repo_id)
# lhoestq/demo1
```

- repo_id is None when the dataset doesn't come from the Hub, e.g. from Dataset.from_dict
- repo_id is set to None when concatenating datasets with different repo ids

related to https://github.com/huggingface/datasets/issues/4129

TODO:
- [ ] discuss if it's ok for now
- [ ] tests
GitHub issue #6267: Multi label class encoding
State: open | Created: 2023-09-27T22:48:08Z

### Feature request

I have a multi label dataset and I'd like to be able to class encode the column and store the mapping directly in the features just as I can with a single label column.  `class_encode_column` currently does not support multi labels.

Here's an example of what I'd like to encode:

```
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.class_encode_column('labels')
```

I did some digging into the code base to evaluate the feasibility of this (note I'm very new to this code base) and from what I noticed the `ClassLabel` feature is still stored as an underlying raw data type of int so I thought a `MultiLabel` feature could similarly be stored as a Sequence of ints, thus not requiring significant serialization / conversion work to / from arrow.

I did a POC of this [here](https://github.com/huggingface/datasets/commit/15443098e9ce053943172f7ec6fce3769d7dff6e) and included a simple test case (please excuse all the commented out tests, going for speed of POC here and didn't want to fight IDE to debug a single test).   In the test I just assert that `num_classes` is the same to show that things are properly serializing, but if you break after loading from disk you'll see the dataset correct and the dataset feature is as expected.

After digging more I did notice a few issues
- After loading from disk I noticed type of the `labels` class is `Sequence` not `MultiLabel` (though the added `feature` attribute came through).  This doesn't happen for `ClassLabel` but I couldn't find the encode / decode code paths that handle this.
- I subclass `Sequence` in `MultiLabel` to leverage existing serialization, but this does miss the custom encode logic that `ClassLabel` has.  I'm not sure of the best way to approach this as I haven't fully understood the encode / decode flow for datasets.  I suspect my simple implementation will need some improvement as it'll require a significant amount of repeated logic to mimic `ClassLabel` behavior.



### Motivation

See above - would like to support multi label class encodings.

### Your contribution

This would be a big help for us and we're open to contributing but I'll likely need some guidance on how to implement to fit the encode / decode flow.  Some suggestions on tests / would be great too, I'm guessing in addition to the class encode tests (that I'll need to expand) we'll need encode / decode tests.
GitHub issue #6266: Use LibYAML with PyYAML if available
State: open | Created: 2023-09-27T21:13:36Z

PyYAML, the YAML framework used in this library, allows the use of LibYAML to accelerate the methods `load` and `dump`. To use it, a user would need to first install a PyYAML version that uses LibYAML (not available in PyPI; needs to be manually installed). Then, to actually use them, PyYAML suggests importing the LibYAML version of the `Loader` and `Dumper` and falling back to the default ones. This PR implements this change. See [PyYAML docs](https://pyyaml.org/wiki/PyYAMLDocumentation) for more info.

This change was motivated after trying to use any of [the SugarCREPE datasets in the Hub](https://huggingface.co/datasets?search=sugarcrepe) provided by [the org HuggingFaceM4](https://huggingface.co/datasets/HuggingFaceM4). Such datasets save a lot of information (~1MB) in the YAML metadata from the `README.md` file and I noticed this slowed down the data loading process. BTW, I also noticed cache files for it is also slow because it tries to hash an instance of `DatasetInfo`, which in turn has all this metadata.

Also, I changed two list comprehensions into generator expressions to avoid allocating extra memory unnecessarily.

And BTW, there's [an issue in PyYAML suggesting to make this automatic](https://github.com/yaml/pyyaml/issues/437).
GitHub issue #6265: Remove `apache_beam` import in `BeamBasedBuilder._save_info`
State: closed | Created: 2023-09-27T13:56:34Z

... to avoid an `ImportError` raised in `BeamBasedBuilder._save_info` when `apache_beam` is not installed (e.g., when downloading the processed version of a dataset from the HF GCS)

Fix https://github.com/huggingface/datasets/issues/6260