kardwalker/my_dataset huggingface dataset:
GitHub issue #5775: ArrowDataset.save_to_disk lost some logic of remote
State: closed | Created: 2023-04-20T16:58:01Z

### Describe the bug

https://github.com/huggingface/datasets/blob/e7ce0ac60c7efc10886471932854903a7c19f172/src/datasets/arrow_dataset.py#L1371

Here is the bug point, when I want to save from a `DatasetDict` class and the items of the instance is like `[('train', Dataset({features: ..., num_rows: ...}))]` , there is no guarantee that there exists a directory name `train` under `dataset_dict_path`.

### Steps to reproduce the bug

1. Mock a DatasetDict with items like what I said.
2. using save_to_disk with storage_options, u can use local sftp. code may like below

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset.save_to_disk('sftp:///tmp', storage_options={'host': 'localhost', 'username': 'admin'})
```
I suppose u can reproduce the bug by these steps. 

### Expected behavior

Should create the folder if it does not exists, just like we do locally.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-6.2.10-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5773: train_dataset does not implement __len__
State: open | Created: 2023-04-20T04:37:05Z

when train using data  precessored by the datasets, I get follow warning and it leads to that I can not set epoch numbers:
`ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.`
GitHub issue #5772: Fix JSON builder when missing keys in first row 
State: closed | Created: 2023-04-19T14:32:57Z

Until now, the JSON builder only considered the keys present in the first element of the list:
- Either explicitly: by passing index 0 in `dataset[0].keys()`
- Or implicitly: `pa.Table.from_pylist(dataset)`, where "schema (default None): If not passed, will be inferred from the first row of the mapping values"

This PR fixes the bug by considering the union of the keys present in all the rows.


Fix #5726.
GitHub issue #5771: Support cloud storage for loading datasets
State: closed | Created: 2023-04-19T12:43:53Z

### Feature request

It seems that the the current implementation supports cloud storage only for `load_from_disk`. It would be nice if a similar functionality existed in `load_dataset`.

### Motivation

Motivation is pretty clear -- let users work with datasets located in the cloud. 

### Your contribution

I can help implementing this.
GitHub issue #5769: Tiktoken tokenizers are not pickable 
State: closed | Created: 2023-04-18T16:07:40Z

### Describe the bug

Since tiktoken tokenizer is not pickable, it is not possible to use it inside `dataset.map()` with multiprocessing enabled. However, you [made](https://github.com/huggingface/datasets/issues/5536) tiktoken's tokenizers pickable in `datasets==2.10.0` for caching. For some reason, this logic does not work in dataset processing and raises `TypeError: cannot pickle 'builtins.CoreBPE' object`


### Steps to reproduce the bug

```
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
    num_proc=2,
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

starts processing dataset 

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-1021-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.0
GitHub issue #5768: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1
State: closed | Created: 2023-04-18T07:10:56Z

### Describe the bug

There is an issue that seems to be unique to the "squad" dataset, in which it cannot be loaded using standard methods. This issue is most quickly reproduced from the command line, using the HF examples to verify a dataset is loaded properly.

This is not a problem with "squad_v2" dataset for example.

### Steps to reproduce the bug

cmd line
> $ python -c "from datasets import load_dataset; print(load_dataset('squad', split='train')[0])"

OR

Python IDE
> from datasets import load_dataset
> load_dataset("squad")

### Expected behavior

I expected to either see the output described here from running the very same command in command line ([https://huggingface.co/docs/datasets/installation]), or any output that does not raise Python's TypeError.

There is some funky behaviour in the dataset builder portion of the codebase that means it is trying to import the squad dataset with an incorrect path, or the squad dataset couldn't be downloaded. I'm not really sure what the problem is beyond that. Messing around with caching I did manage to get it to load the dataset once, and then couldn't repeat this.

### Environment info

datasets=2.7.1 **or** 2.10.1, python=3.10.8, Linux 3.10.0-1160.36.2.el7.x86_64 **or** Windows 10-64
GitHub issue #5767: How to use Distill-BERT with different datasets?
State: closed | Created: 2023-04-18T06:25:12Z

### Describe the bug

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Steps to reproduce the bug

I recently read [this](https://huggingface.co/docs/transformers/quicktour#train-with-tensorflow:~:text=The%20most%20important%20thing%20to%20remember%20is%20you%20need%20to%20instantiate%20a%20tokenizer%20with%20the%20same%20model%20name%20to%20ensure%20you%E2%80%99re%20using%20the%20same%20tokenization%20rules%20a%20model%20was%20pretrained%20with.) and was wondering how to use distill-BERT (which is pre-trained with imdb dataset) with a different dataset (for eg. [this](https://huggingface.co/datasets/yhavinga/imdb_dutch) dataset)?

### Expected behavior

Distill-BERT should work with different datasets.

### Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 11.0.0
GitHub issue #5766: Support custom feature types
State: open | Created: 2023-04-17T15:46:41Z

### Feature request

I think it would be nice to allow registering custom feature types with the 🤗 Datasets library. For example, allow to do something along the following lines:

```
from datasets.features import register_feature_type  # this would be a new function

@register_feature_type
class CustomFeatureType:
    def encode_example(self, value):
        """User-provided logic to encode an example of this feature."""
        pass

    def decode_example(self, value, token_per_repo_id=None):
        """User-provided logic to decode an example of this feature."""
        pass
```

### Motivation

Users of 🤗 Datasets, such as myself, may want to use the library to load datasets with unsupported feature types (i.e., beyond `ClassLabel`, `Image`, or `Audio`). This would be useful for prototyping new feature types and for feature types that aren't used widely enough to warrant inclusion in 🤗 Datasets.

At the moment, this is only possible by monkey-patching 🤗 Datasets, which obfuscates the code and is prone to breaking with library updates. It also requires the user to write some custom code which could be easily avoided.

### Your contribution

I would be happy to contribute this feature. My proposed solution would involve changing the following call to `globals()` to an explicit feature type registry, which a user-facing `register_feature_type` decorator could update.

https://github.com/huggingface/datasets/blob/fd893098627230cc734f6009ad04cf885c979ac4/src/datasets/features/features.py#L1329

I would also provide an abstract base class for custom feature types which users could inherit. This would have at least an `encode_example` method and a `decode_example` method, similar to `Image` or `Audio`.

The existing `encode_nested_example` and `decode_nested_example` functions would also need to be updated to correctly call the corresponding functions for the new type.
GitHub issue #5764: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1
State: closed | Created: 2023-04-17T09:08:18Z

### Describe the bug

I want to use this (https://huggingface.co/datasets/josianem/imdb) dataset therefore I am trying to load it using the following code:

```
dataset = load_dataset("josianem/imdb")
```
The dataset is not getting loaded and gives the error message as the following:

```
Traceback (most recent call last):
  File "sample.py", line 3, in <module>
    dataset = load_dataset("josianem/imdb")
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 704, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/saurav/.cache/huggingface/modules/datasets_modules/datasets/imdb/cc6ab4acab2799be15d5d217c24548b856156dafdc850165fdc4f2031f27ff2f/imdb.py", line 79, in _split_generators
    archive = dl_manager.download(_DOWNLOAD_URL)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 289, in cached_path
    output_path = get_from_cache(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 606, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1
```

### Steps to reproduce the bug

You can reproduce the error by using the following code:

```
from datasets import load_dataset, load_metric

dataset = load_dataset("josianem/imdb")
```

### Expected behavior

The dataset should get loaded (I am using this dataset for the first time so not much aware of the exact behavior).

### Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 11.0.0
GitHub issue #5762: Not able to load the pile
State: closed | Created: 2023-04-17T03:09:10Z

### Describe the bug

Got this error when I am trying to load the pile dataset 

```
TypeError: Couldn't cast array of type
struct<file: string, id: string>
to
{'id': Value(dtype='string', id=None)}

```


### Steps to reproduce the bug

Please visit the following sample notebook 

https://colab.research.google.com/drive/1JHcjawcHL6QHhi5VcqYd07W2QCEj2nWK#scrollTo=ulJP3eJCI-tB

### Expected behavior

The pile should work

### Environment info

-  `datasets` version: 2.11.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue #5760: Multi-image loading in Imagefolder dataset
State: open | Created: 2023-04-16T16:01:05Z

### Feature request

Extend the `imagefolder` dataloading script to support loading multiple images per dataset entry.

This only really makes sense if a metadata file is present.

Currently you can use the following format (example `metadata.jsonl`:
```
{'file_name': 'path_to_image.png', 'metadata': ...}
...
```
which will return a batch with key `image` and any other metadata.

I would propose extending `file_name` to also accept a list of files, which would return a batch with key `images` and any other metadata.

### Motivation

This is useful for example in segmentation tasks in computer vision models, or in text-to-image models that also accept conditioning signals such as another image, feature map, or similar. Currently if I want to do this, I would need to write a custom dataset, rather than just use `imagefolder`.

### Your contribution

Would be open to doing a PR, but also happy for someone else to take it as I am not familiar with the datasets library.
GitHub issue #5759: Can I load in list of list of dict format? 
State: open | Created: 2023-04-16T13:50:14Z

### Feature request

my jsonl dataset has following format:
```
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
```

I try to use `datasets.load_dataset('json', data_files=path)` or `datasets.Dataset.from_json`, it raises
```
  File "site-packages/datasets/arrow_dataset.py", line 1078, in from_json
    ).read()
  File "site-packages/datasets/io/json.py", line 59, in read
    self.builder.download_and_prepare(
  File "site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "site-packages/datasets/builder.py", line 967, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "site-packages/datasets/builder.py", line 1749, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "site-packages/datasets/builder.py", line 1892, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Motivation

I wanna use features like `Datasets.map` or `Datasets.shuffle`, so i need the dataset in memory to be `arrow_dataset.Datasets` format

### Your contribution

PR
GitHub issue #5757: Tilde (~) is not supported
State: closed | Created: 2023-04-16T11:48:10Z

### Describe the bug

It seems that `~` is not recognized correctly in local paths. Whenever I try to use it I get an exception

### Steps to reproduce the bug

```python

load_dataset("imagefolder", data_dir="~/data/my_dataset")

```

Will generate the following error:
``` 
EmptyDatasetError: The directory at /path/to/cwd/~/data/datasets/clementine_tagged_per_cam doesn't contain any data files
```

### Expected behavior

Load the dataset.

### Environment info

datasets==2.11.0
GitHub issue #5756: Calling shuffle on a IterableDataset with streaming=True, gives "ValueError: cannot reshape array"
State: closed | Created: 2023-04-16T04:59:47Z

### Describe the bug

When calling shuffle on a IterableDataset with streaming=True, I get the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 937, in __iter__
    for key, example in ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 627, in __iter__
    for x in self.ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 138, in __iter__
    yield from self.generate_examples_fn(**kwargs_with_shuffled_shards)
  File "/home/administrator/.cache/huggingface/modules/datasets_modules/datasets/mnist/fda16c03c4ecfb13f165ba7e29cf38129ce035011519968cdaf74894ce91c9d4/mnist.py", line 111, in _generate_examples
    images = np.frombuffer(f.read(), dtype=np.uint8).reshape(size, 28, 28)
ValueError: cannot reshape array of size 59992 into shape (60000,28,28)

```
Tested with the fashion_mnist and mnist datasets

### Steps to reproduce the bug

Code to reproduce
```python
from datasets import load_dataset

SHUFFLE_SEED = 42
SHUFFLE_BUFFER_SIZE = 10_000

dataset = load_dataset('fashion_mnist', streaming=True).shuffle(seed=SHUFFLE_SEED, buffer_size=SHUFFLE_BUFFER_SIZE)
next(iter(dataset['train']))
```

### Expected behavior

A random item from the dataset and no error

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue #5755: ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
State: closed | Created: 2023-04-14T23:28:54Z

### Describe the bug

The module moved to new place?

### Steps to reproduce the bug

in the import step,

```python
from datasets.utils.deprecation_utils import DeprecatedEnum
```

error:

```
ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
```

### Expected behavior

import successfully

### Environment info

python==3.9.16
datasets==1.18.3
GitHub issue #5754: Minor tqdm fixes
State: closed | Created: 2023-04-14T18:15:14Z

`GeneratorBasedBuilder`'s TQDM bars were not used as context managers. This PR fixes that (missed these bars in https://github.com/huggingface/datasets/pull/5560). 

Also, this PR modifies the single-proc `save_to_disk` to fix the issue with the TQDM bar not accumulating the progress in the multi-shard setting (again, this bug was introduced by me in the linked PR 😎)
GitHub issue #5753: [IterableDatasets] Add column followed by interleave datasets gives bogus outputs
State: closed | Created: 2023-04-14T17:32:31Z

### Describe the bug

If we add a new column to our iterable dataset using the hack described in #5752, when we then interleave datasets the new column is pinned to one value.

### Steps to reproduce the bug

What we're going to do here is:
1. Load an iterable dataset in streaming mode (`original_dataset`)
2. Add a new column to this dataset using the hack in #5752 (`modified_dataset_1`)
3. Create another new dataset by adding a column with the same key but different values (`modified_dataset_2`)
4. Interleave our new datasets (`modified_dataset_1` + `modified_dataset_2`)
5. Check the value of our newly added column (`new_column`)

```python
from datasets import load_dataset

# load an iterable dataset
original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# now add a new column to our streaming dataset using our hack from 5752
name = "new_column"
column = [f"new dataset 1, row {i}" for i in range(50)]

new_features = original_dataset.features.copy()
new_features[name] = new_features["file"]  # I know that "file" has the right column type to match our new feature

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset_1 = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

# now create a second modified dataset using the same trick
column = [f"new dataset 2, row {i}" for i in range(50)]

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset_2 = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

# interleave these datasets
interleaved_dataset = interleave_datasets([modified_dataset_1, modified_dataset_2])

# now check what the value of the added column is
for i, sample in enumerate(interleaved_dataset):
    print(sample["new_column"])
    if i == 10:
        break
```
**Print Output:**
```
new dataset 2, row 0
new dataset 2, row 0
new dataset 2, row 1
new dataset 2, row 1
new dataset 2, row 2
new dataset 2, row 2
new dataset 2, row 3
new dataset 2, row 3
new dataset 2, row 4
new dataset 2, row 4
new dataset 2, row 5
```
We see that we only get outputs from our second dataset.

### Expected behavior

We should interleave between dataset 1 and 2 and increase in row value:
```
new dataset 1, row 0
new dataset 2, row 0
new dataset 1, row 1
new dataset 2, row 1
new dataset 1, row 2
new dataset 2, row 2
...
```

### Environment info

- datasets version: 2.10.2.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.13.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue #5752: Streaming dataset looses `.feature` method after `.add_column`
State: open | Created: 2023-04-14T16:39:50Z

### Describe the bug

After appending a new column to a streaming dataset using `.add_column`, we can no longer access the list of dataset features using the `.feature` method.

### Steps to reproduce the bug

```python
from datasets import load_dataset

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
print(original_dataset.features.keys())

# now add a new column to our streaming dataset
modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
print(modified_dataset.features.keys())
```
**Print Output:**
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 8
      6 # now add a new column to our streaming dataset
      7 modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
----> 8 print(modified_dataset.features.keys())

AttributeError: 'NoneType' object has no attribute 'keys'
```

We see that we get the features for the original dataset, but not the modified one with the added column.


### Expected behavior

Features should be persevered after adding a new column, i.e. calling:
```python
print(modified_dataset.features.keys())
```

Should return:
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id', 'new_column'])
```

### Environment info

- `datasets` version: 2.10.2.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.13.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue #5751: Consistent ArrayXD Python formatting + better NumPy/Pandas formatting
State: closed | Created: 2023-04-14T14:13:59Z

Return a list of lists instead of a list of NumPy arrays when converting the variable-shaped `ArrayXD` to Python. Additionally, improve the NumPy conversion by returning a numeric NumPy array when the offsets are equal or a NumPy object array when they aren't, and allow converting the variable-shaped `ArrayXD` to Pandas.

(Reported in https://github.com/huggingface/datasets/issues/5719#issuecomment-1507579671)
GitHub issue #5750: Fail to create datasets from a generator when using Google Big Query
State: closed | Created: 2023-04-14T13:50:59Z

### Describe the bug

Creating a dataset from a generator using `Dataset.from_generator()` fails if the generator is the [Google Big Query Python client](https://cloud.google.com/python/docs/reference/bigquery/latest). The problem is that the Big Query client is not pickable. And the function `create_config_id` tries to get a hash of the generator by pickling it. So the following error is generated:

```
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
```


### Steps to reproduce the bug

1. Install the big query client and datasets `pip install google-cloud-bigquery datasets`
2. Run the following code:

```py
from datasets import Dataset
from google.cloud import bigquery

client = bigquery.Client()

# Perform a query.
QUERY = (
    'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
    'WHERE state = "TX" '
    'LIMIT 100')
query_job = client.query(QUERY)  # API request
rows = query_job.result()  # Waits for query to finish

ds = Dataset.from_generator(rows)

for r in ds:
    print(r)
```

### Expected behavior

Two options:
1. Ignore the pickle errors when computing the hash
2. Provide a scape hutch so that we can avoid calculating the hash for the generator. For example, allowing to provide a hash from the user. 

### Environment info

python 3.9
google-cloud-bigquery         3.9.0
datasets                      2.11.0
GitHub issue #5744: [BUG] With Pandas 2.0.0, `load_dataset` raises `TypeError: read_csv() got an unexpected keyword argument 'mangle_dupe_cols'`
State: closed | Created: 2023-04-13T20:21:28Z

The `load_dataset` function with Pandas `1.5.3` has no issue (just a FutureWarning) but crashes with Pandas `2.0.0`.
For your convenience, I opened a draft Pull Request to fix it quickly: https://github.com/huggingface/datasets/pull/5745

---

* The FutureWarning mentioned above: 
```
FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
```
GitHub issue #5743: dataclass.py in virtual environment is   overriding the stdlib module "dataclasses"
State: closed | Created: 2023-04-13T17:28:33Z

### Describe the bug

"e:\Krish_naik\FSDSRegression\venv\Lib\dataclasses.py" is overriding the stdlib module "dataclasses"

### Steps to reproduce the bug

module issue

### Expected behavior

 overriding the stdlib module "dataclasses"

### Environment info

VS code
GitHub issue #5740: Fix CI mock filesystem fixtures
State: closed | Created: 2023-04-12T08:52:35Z

This PR fixes the fixtures of our CI mock filesystems.

Before, we had to pass `clobber=True` to `fsspec.register_implementation` to overwrite the still present previously added "mock" filesystem. That meant that the mock filesystem fixture was not working properly, because the previously added "mock" filesystem, should have been deleted by the fixture.

This PR fixes the mock filesystem fixtures, so that the "mock" filesystem is properly deleted from the inner `fsspec` registry.

Tests were added to check the correct behavior of the mock filesystem fixtures.

Related to:
- #5733
GitHub issue #5739: weird result during dataset split when data path starts with `/data`
State: open | Created: 2023-04-12T04:51:35Z

### Describe the bug

The regex defined here https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/utils/py_utils.py#L158 

will cause a weird result during dataset split when data path starts with `/data`

### Steps to reproduce the bug

1.  clone dataset into local path
```
cd /data/train/raw/
git lfs clone  https://huggingface.co/datasets/deepmind/code_contests.git

ls /data/train/raw/code_contests
# README.md  data  dataset_infos.json

ls /data/train/raw/code_contests/data
# test-00000-of-00001-9c49eeff30aacaa8.parquet
# train-[0-9]+-of-[0-9]+-xx.parquet
# valid-00000-of-00001-5e672c5751f060d3.parquet

```

2. loading data from local 
```
from datasets import load_dataset
dataset = load_dataset('/data/train/raw/code_contests')

FileNotFoundError: Unable to resolve any data file that matches '['data/train/raw/code_contests/data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']' at /data/train/raw/code_contests with any supported extension
```

weird path `data/train/raw/code_contests/data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*`

While dive deep into `LocalDatasetModuleFactoryWithoutScript` defined in [load.py](https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/load.py#L627) and _get_data_files_patterns https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/data_files.py#L228. I found the weird behavior caused by `string_to_dict`

3. check `string_to_dict`
```
p = '/data/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
split_pattern = 'data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'

string_to_dict(p, split_pattern)
# {'split': 'train/raw/code_contests/data/test'}

p = '/data2/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
string_to_dict(p, split_pattern)
{'split': 'test'}
```

go deep into string_to_dict https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/utils/py_utils.py#L158. 

4. test the regex:
<img width="680" alt="image" src="https://user-images.githubusercontent.com/1772912/231351129-75179f01-fb9f-4f12-8fa9-0dfcc3d5f3bd.png">

<img width="679" alt="image" src="https://user-images.githubusercontent.com/1772912/231351025-009f3d83-2cf3-4e15-9ed4-6b9663dcb2ee.png">




### Expected behavior

statement in `steps to reproduce the bug`

3. check `string_to_dict`
```
p = '/data/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
split_pattern = 'data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'

string_to_dict(p, split_pattern)
# {'split': 'train/raw/code_contests/data/test'}

p = '/data2/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
string_to_dict(p, split_pattern)
{'split': 'test'}
```


### Environment info

- linux(debian)
- python 3.7
- datasets 2.8.0
GitHub issue #5738: load_dataset("text","dataset.txt") loads the wrong dataset!
State: closed | Created: 2023-04-12T01:07:46Z

### Describe the bug

I am trying to load my own custom text dataset using the load_dataset function.  My dataset is a bunch of ordered text, think along the lines of shakespeare plays.  However, after I load the dataset and I inspect it, the dataset is a table with a bunch of latitude and longitude values!  What in the world??

### Steps to reproduce the bug

my_dataset = load_dataset("text","TextFile.txt")
my_dataset

### Expected behavior

I expected the dataset to contain the actual data from the text document that I used.

### Environment info

Google Colab
GitHub issue #5737: ClassLabel Error 
State: closed | Created: 2023-04-11T17:14:13Z

### Describe the bug

I still getting the error "call() takes 1 positional argument but 2 were given" even after ensuring that the value being passed to the label object is a single value and that the ClassLabel object has been created with the correct number of label classes

### Steps to reproduce the bug

from datasets import ClassLabel, Dataset

1. Create the ClassLabel object with 3 label values and their corresponding names
label_test = ClassLabel(num_classes=3, names=["label_1", "label_2", "label_3"])

2. Define a dictionary with text and label fields
data = {
    'text': ['text_1', 'text_2', 'text_3'],
    'label': [1, 2, 3],
}

3. Create a Hugging Face dataset from the dictionary
dataset = Dataset.from_dict(data)
print(dataset.features)
4. Map the label values to their corresponding label names using the label object
dataset = dataset.map(lambda example: {'text': example['text'], 'label': label_test(example['label'])})

5. Print the resulting dataset
print(dataset)

### Expected behavior

I hope my label type is class label instead int. 

### Environment info

python 3.9
google colab
GitHub issue #5736: FORCE_REDOWNLOAD raises "Directory not empty" exception on second run
State: open | Created: 2023-04-11T11:29:15Z

### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2
GitHub issue #5735: Implement sharding on merged iterable datasets
State: closed | Created: 2023-04-11T10:02:25Z

This PR allows sharding of merged iterable datasets.

Merged iterable datasets with for instance the `interleave_datasets` command are comprised of multiple sub-iterable, one for each dataset that has been merged.

With this PR, sharding a merged iterable will result in multiple merged datasets each comprised of sharded sub-iterable, ensuring that there is no duplication of data.

As a result it is now possible to set any amount of workers in the dataloader as long as it is lower or equal to the lowest amount of shards amongst the datasets. Before it had to be set to 0.

I previously talked about this issue on the forum [here](https://discuss.huggingface.co/t/interleaving-iterable-dataset-with-num-workers-0/35801)
GitHub issue #5733: Unpin fsspec
State: closed | Created: 2023-04-11T08:52:12Z

In `fsspec--2023.4.0` default value for clobber when registering an implementation was changed from True to False. See:
- https://github.com/fsspec/filesystem_spec/pull/1237

This PR recovers previous behavior by passing clobber True when registering mock implementations.

This PR also removes the temporary pin introduced by:
- #5731

Fix #5734.
GitHub issue #5732: Enwik8 should support the standard split
State: closed | Created: 2023-04-11T08:38:53Z

### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR.
GitHub issue #5728: The order of data split names is nondeterministic
State: closed | Created: 2023-04-11T07:31:25Z

After this CI error: https://github.com/huggingface/datasets/actions/runs/4639528358/jobs/8210492953?pr=5718
```
FAILED tests/test_data_files.py::test_get_data_files_patterns[data_file_per_split4] - AssertionError: assert ['random', 'train'] == ['train', 'random']
  At index 0 diff: 'random' != 'train'
  Full diff:
  - ['train', 'random']
  + ['random', 'train']
```
I have checked locally and found out that the data split order is nondeterministic.

This is caused by the use of `set` for sharded splits.
GitHub issue #5726: Fallback JSON Dataset loading does not load all values when features specified manually
State: closed | Created: 2023-04-10T15:22:14Z

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10
GitHub issue #5725: How to limit the number of examples in dataset, for testing?
State: closed | Created: 2023-04-10T08:41:43Z

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special
GitHub issue #5721: Calling datasets.load_dataset("text" ...) results in a wrong split.
State: open | Created: 2023-04-08T23:55:12Z

### Describe the bug

When creating a text dataset, the training split should have the bulk of the examples by default. Currently, testing does.

### Steps to reproduce the bug

I have a folder with 18K text files in it. Each text file essentially consists in a document or article scraped from online. Calling the following codeL
```
folder_path = "/home/cyril/Downloads/llama_dataset"
data = datasets.load_dataset("text", data_dir=folder_path)
data.save_to_disk("/home/cyril/Downloads/data.hf")

data = datasets.load_from_disk("/home/cyril/Downloads/data.hf")
print(data)
```

Results in the following split:
```
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 2114
    })
    test: Dataset({
        features: ['text'],
        num_rows: 200882
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 152
    })
})
```

It seems to me like the train/test/validation splits are in the wrong order since test split >>>> train_split

### Expected behavior

Train split should have the bulk of the training examples.

### Environment info

datasets 2.11.0, python 3.10.6
GitHub issue #5719: Array2D feature creates a list of list instead of a numpy array
State: closed | Created: 2023-04-07T21:04:08Z

### Describe the bug

I'm not sure if this is expected behavior or not. When I create a 2D array using `Array2D`, the data has list type instead of numpy array. I think it should not be the expected behavior especially when I feed a numpy array as input to the data creation function. Why is it converting my array into a list?
Also if I change the first dimension of the `Array2D` shape to None, it's returning array correctly.

### Steps to reproduce the bug

Run this code:
```py
from datasets import Dataset, Features, Array2D
import numpy as np

# you have to change the first dimension of the shape to None to make it return an array
features = Features(dict(seq=Array2D((2,2), 'float32')))  
ds = Dataset.from_dict(dict(seq=[np.random.rand(2,2)]), features=features)
a = ds[0]['seq']
print(a)
print(type(a))
```

The following will be printed in stdout:
```
[[0.8127174377441406, 0.3760348856449127], [0.7510159611701965, 0.4322739541530609]]
<class 'list'>
```

### Expected behavior

Each indexed item should be a list or numpy array.  Currently, `Array((2,2))` yields a list but `Array((None,2))` yields an array.

### Environment info

- `datasets` version: 2.11.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.9.13
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 1.4.4
GitHub issue #5717: Errror when saving to disk a dataset of images
State: open | Created: 2023-04-07T11:59:17Z

### Describe the bug

Hello!

I have an issue when I try to save on disk my dataset of images. The error I get is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1442, in save_to_disk
    for job_id, done, content in Dataset._save_to_disk_single(**kwargs):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1473, in _save_to_disk_single
    writer.write_table(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_writer.py", line 570, in write_table
    pa_table = embed_table_storage(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2268, in embed_table_storage
    arrays = [
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2269, in <listcomp>
    embed_array_storage(table[name], feature) if require_storage_embed(feature) else table[name]
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2142, in embed_array_storage
    return feature.embed_storage(array)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/features/image.py", line 269, in embed_storage
    storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
  File "pyarrow/array.pxi", line 2766, in pyarrow.lib.StructArray.from_arrays
  File "pyarrow/array.pxi", line 2961, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```

My dataset is around 50K images, is this error might be due to a bad image?

Thanks for the help.

### Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset["train"].save_to_disk("./myds", num_shards=40)
```

### Expected behavior

Having my dataset properly saved to disk.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue #5716: Handle empty audio
State: closed | Created: 2023-04-07T09:51:40Z

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`