konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Calling shuffle on a IterableDataset with streaming=True, gives "ValueError: cannot reshape array"

### Describe the bug

When calling shuffle on a IterableDataset with streaming=True, I get the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 937, in __iter__
    for key, example in ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 627, in __iter__
    for x in self.ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 138, in __iter__
    yield from self.generate_examples_fn(**kwargs_with_shuffled_shards)
  File "/home/administrator/.cache/huggingface/modules/datasets_modules/datasets/mnist/fda16c03c4ecfb13f165ba7e29cf38129ce035011519968cdaf74894ce91c9d4/mnist.py", line 111, in _generate_examples
    images = np.frombuffer(f.read(), dtype=np.uint8).reshape(size, 28, 28)
ValueError: cannot reshape array of size 59992 into shape (60000,28,28)

```
Tested with the fashion_mnist and mnist datasets

### Steps to reproduce the bug

Code to reproduce
```python
from datasets import load_dataset

SHUFFLE_SEED = 42
SHUFFLE_BUFFER_SIZE = 10_000

dataset = load_dataset('fashion_mnist', streaming=True).shuffle(seed=SHUFFLE_SEED, buffer_size=SHUFFLE_BUFFER_SIZE)
next(iter(dataset['train']))
```

### Expected behavior

A random item from the dataset and no error

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0


Hi! I've merged a PR on the Hub with a fix: https://huggingface.co/datasets/fashion_mnist/discussions/3

Thanks, this appears to have fixed the issue.

I've created a PR for the same change in the mnist dataset: https://huggingface.co/datasets/mnist/discussions/3/files
ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 

### Describe the bug

The module moved to new place?

### Steps to reproduce the bug

in the import step,

```python
from datasets.utils.deprecation_utils import DeprecatedEnum
```

error:

```
ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
```

### Expected behavior

import successfully

### Environment info

python==3.9.16
datasets==1.18.3

update the version. fix
[BUG FIX] Issue 5744

A temporal fix for https://github.com/huggingface/datasets/issues/5744.

Have met the same problem with datasets==2.8.0, pandas==2.0.0. It could be solved by installing the latest version of datasets or using datasets==2.8.0, pandas==1.5.3.

Pandas 2.0.0 has removed support to passing `mangle_dupe_cols`.

However, our `datasets` library does not use this parameter: it only passes it to pandas if the user passes it to `load_dataset`.

You should better:
- Either "take steps to stop the use of 'mangle_dupe_cols'" (as it was suggested in the deprecation warning in pandas-1.5.3)
- Or pin pandas (< 2.0.0) in your local requirements file

Please note that from `datasets` library, we don't want to force users to use a specific pandas version. We would like to support users as well:
- that use pandas < 1.5.3
- that use pandas >= 2.0.0 and that do not pass the 'mangle_dupe_cols' parameter

`datasets` 2.11 doesn't pass `mangle_dupe_cols` unless the user specifies it indeed, so I think we're fine
dataclass.py in virtual environment is   overriding the stdlib module "dataclasses"

### Describe the bug

"e:\Krish_naik\FSDSRegression\venv\Lib\dataclasses.py" is overriding the stdlib module "dataclasses"

### Steps to reproduce the bug

module issue

### Expected behavior

 overriding the stdlib module "dataclasses"

### Environment info

VS code

We no longer depend on `dataclasses` (for almost a year), so I don't think our package is the problematic one. 

I think it makes more sense to raise this issue in the `dataclasses` repo: https://github.com/ericvsmith/dataclasses.
load_dataset("text","dataset.txt") loads the wrong dataset!

### Describe the bug

I am trying to load my own custom text dataset using the load_dataset function.  My dataset is a bunch of ordered text, think along the lines of shakespeare plays.  However, after I load the dataset and I inspect it, the dataset is a table with a bunch of latitude and longitude values!  What in the world??

### Steps to reproduce the bug

my_dataset = load_dataset("text","TextFile.txt")
my_dataset

### Expected behavior

I expected the dataset to contain the actual data from the text document that I used.

### Environment info

Google Colab

You need to provide a text file as `data_files`, not as a configuration:

```python
my_dataset = load_dataset("text", data_files="TextFile.txt")
```

Otherwise, since `data_files` is `None`, it picks up Colab's sample datasets from the `content` dir.
ClassLabel Error 

### Describe the bug

I still getting the error "call() takes 1 positional argument but 2 were given" even after ensuring that the value being passed to the label object is a single value and that the ClassLabel object has been created with the correct number of label classes

### Steps to reproduce the bug

from datasets import ClassLabel, Dataset

1. Create the ClassLabel object with 3 label values and their corresponding names
label_test = ClassLabel(num_classes=3, names=["label_1", "label_2", "label_3"])

2. Define a dictionary with text and label fields
data = {
    'text': ['text_1', 'text_2', 'text_3'],
    'label': [1, 2, 3],
}

3. Create a Hugging Face dataset from the dictionary
dataset = Dataset.from_dict(data)
print(dataset.features)
4. Map the label values to their corresponding label names using the label object
dataset = dataset.map(lambda example: {'text': example['text'], 'label': label_test(example['label'])})

5. Print the resulting dataset
print(dataset)

### Expected behavior

I hope my label type is class label instead int. 

### Environment info

python 3.9
google colab

Hi, you can use the `cast_column` function to change the feature type from a `Value(int64)` to `ClassLabel`:

```py
dataset = dataset.cast_column("label", ClassLabel(names=["label_1", "label_2", "label_3"]))
print(dataset.features)
{'text': Value(dtype='string', id=None),
 'label': ClassLabel(names=['label_1', 'label_2', 'label_3'], id=None)}
```

thank you @stevhliu, its worked.
FORCE_REDOWNLOAD raises "Directory not empty" exception on second run

### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2


Hi ! I couldn't reproduce your issue :/

It seems that `shutil.rmtree` failed. It is supposed to work even if the directory is not empty, but you still end up with `OSError: [Errno 39] Directory not empty:`. Can you make sure another process is not using this directory at the same time ?

I have the same error with `datasets==2.14.5` and `pyarrow==13.0.0`. Python 3.10.13

I have same error. Any workaround?
Enwik8 should support the standard split

### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR. 

#self-assign

The Enwik8 pipeline is not present in this codebase, and is hosted elsewhere. I have opened a PR [there](https://huggingface.co/datasets/enwik8/discussions/4) instead.
The order of data split names is nondeterministic

After this CI error: https://github.com/huggingface/datasets/actions/runs/4639528358/jobs/8210492953?pr=5718
```
FAILED tests/test_data_files.py::test_get_data_files_patterns[data_file_per_split4] - AssertionError: assert ['random', 'train'] == ['train', 'random']
  At index 0 diff: 'random' != 'train'
  Full diff:
  - ['train', 'random']
  + ['random', 'train']
```
I have checked locally and found out that the data split order is nondeterministic.

This is caused by the use of `set` for sharded splits.
Fallback JSON Dataset loading does not load all values when features specified manually

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10

Thanks for reporting, @myluki2000.

I am working on a fix.
How to limit the number of examples in dataset, for testing?

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special

Hi! You can use the `nrows` parameter for this:
```python
data = load_dataset("json", data_files=data_path, nrows=10)
```

@mariosasko I get:

`TypeError: __init__() got an unexpected keyword argument 'nrows'`

I misread the format in which the dataset is stored -  the `nrows` parameter works for CSV, but not JSON.

This means the only option is first to create a DataFrame and then convert it to a Dataset object:
```python
import pandas as pd
from datasets import Dataset

df = pd.read_json(data_path, lines=True, nrows=10)
ds = Dataset.from_pandas(df)
```
Handle empty audio

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`

Hi! Can you share one of the problematic audio files with us?

I tried to reproduce the error with the following code: 
```python
import soundfile as sf
import numpy as np
from datasets import Audio

sf.write("empty.wav",  np.array([]), 16000)
Audio(sampling_rate=24000).decode_example({"path": "empty.wav", "bytes": None})
```
But without success.

Also, what version of `librosa` is installed in your env? (You can get this info with `python -c "import librosa; print(librosa.__version__)`)



I'm closing this issue as the reproducer hasn't been provided.
Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks

Hi! 

You can use [`.set_format("np")`](https://huggingface.co/docs/datasets/process#format) to get NumPy arrays (or Pytorch tensors with `.set_format("torch")`) in `__getitem__`.

Also, have you been able to reproduce the linked PyTorch issue with a HF dataset?
Dataset sizes are in MiB instead of MB in dataset cards

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations

Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5

looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`

I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.

yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">


I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files

Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example

First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.

The bulk edit parsed 751 canonical datasets and updated 166.

Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">


I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6

should we force merge the PR and close this issue?

I merged the PRs for "scicite" and "scifact".
Getting next item from IterableDataset took forever.

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12

Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...

Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
5537 speedup load

I reimplemented fsspec.spec.glob() in `hffilesystem.py` as `_glob`,  used it in `_resolve_single_pattern_in_dataset_repository` only, and saw a 20% speedup in times to load the config, on average. 

That's not much when usually this step takes only 2-3 seconds for most datasets, but in this particular case,  `bigcode/the-stack-dedup` , the loading time to get the config (not download the entire 6tb dataset, of course), went from ~170 secs to ~20 secs. 

What makes this work is this code in `_glob`:

```  
      if self.dir_cache is not None:
            allpaths = self.dir_cache
        else:
            allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
```

I also had to `import glob.has_magic( )` for `_glob()` (confusing, I know).

I hope there is no issue with copying most of the code from `fsspec.spec.glob`, as it is a BSD 3-Clause License, 
and I left a comment about this in the docstring of` _glob()`, that we may want to delete.

As mentioned, I evaluated the speedup across a random selection of about 1000 datasets (not all 27k+), and verified that old_config.eq(new_method_config) with the build in method, but deleted this test and related code changes on the subsequent commit. It's in the commit history if anyone wants to see it. (Note this does not include the outlier of  `bigcode/the-stack-dedup`

|   |   old_time     |    new _time |     diff   |  pct_diff |
| -- | -- | -- | -- | -- |
| mean  |    3.340 |    2.642 |    0.698 |   18.404 |
| min   |    2.024   |  1.976   |  -0.840  | -37.634 |
| max   |  66.582  |   41.517 |   30.927  |  85.538 |

Awesome ! cc @mariosasko :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5704). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this!

Your solution only works if the `root` is `""`, e.g., this would yield an incorrect result:
```python
dset = load_dataset("user/hf-dataset-repo", data_dir="path/to/data_dir")
```

Also, the `HfFileSystem` implementation in `datasets` will be replaced with the more powerful [one](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py) from `huggingface_hub` soon (I plan to open a PR that makes `find` much faster in the coming days). 

So I don't think we want to merge this PR in the current state, but thanks again for the effort.

 (I'll comment on the original issue to propose a cleaner solution)

Ooof. Sorry, I should have checked that more thoroughly then! I would say we could just add that check and only use my approach if the root is "", which would still be faster in many cases, but it sounds like you have a better solution on the way. Thanks for the feedback Mario.
[WIP][Test, Please ignore] Investigate performance impact of using multiprocessing only


`multiprocess` uses `dill` instead of `pickle` for pickling shared objects and, as such, can pickle more types than `multiprocessing`. And I don't think this is something we want to change :).

That makes sense to me, and I don't think you should merge this change. I was only curious about the performance impact. I saw the benchmarks that was produced in other PRs, and wanted to get a better understanding of it. I created this PR to see if it got automatically added here.

Is there a way I can generate those benchmarks myself?

You can find some speed comparisons between dill and pickle on SO if you google "dill vs pickle speed".

And for the benchmarks, you can generate them locally with DVC running this code from the repo root: https://github.com/huggingface/datasets/blob/0803a006db1c395ac715662cc6079651f77c11ea/.github/workflows/benchmarks.yaml#L23-L47.

Thanks for the help @mariosasko!
Add Qdrant as another search index

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.

@mariosasko I'd appreciate your feedback on this.
Shuffle a sharded iterable dataset without seed can lead to duplicate data

As reported in https://github.com/huggingface/datasets/issues/5360

If `seed=None` in `.shuffle()`, shuffled datasets don't use the same shuffling seed across nodes.

Because of that, the lists of shards is not shuffled the same way across nodes, and therefore some shards may be assigned to multiple nodes instead of exactly one.

This can happen only when you have a number of shards that is a factor of the number of nodes.

The current workaround is to always set a `seed` in `.shuffle()`
Dataset configuration

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md

Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit

From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).

Current format:
```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
Document to compress data files before uploading

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1

Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example). 
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

Perfect, thanks for making the distinction between compression and data extensions!
Add information about patterns search order to the doc about structuring repo

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....

Good idea, I think I've seen this a couple of times before too on the forums. I can work on this :)

Closed in #5693
Allow load_dataset to take a working dir for intermediate data

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.

Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.

In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache. 

You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk

If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?
Add support to create a Dataset from spark dataframe

### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.

if i read spark Dataframe , got an error on  multi-node Spark cluster.
Did the Api (Dataset.from_spark) support Spark cluster, read dataframe and save_to_disk?

Error: 
_pickle.PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforma
tion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
23/06/16 21:17:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)



How to perform predictions on Dataset object in Spark with multi-node cluster parallelism?

Addressed in #5701

Hi ! for your information we are working on some more documentation on how to use Spark with HF Datasets repositories (without the need for the `datasets` library) 
https://github.com/huggingface/datasets/issues/5678

Cc @lu-wang-dl @maddiedawson let me know what you think !

sorry, wrong link: https://github.com/huggingface/hub-docs/pull/1392
Filter datasets by language code

Hi! I use the language search field on https://huggingface.co/datasets
However, some of the datasets tagged by ISO language code are not accessible by this search form.
For example, [myv_ru_2022](https://huggingface.co/datasets/slone/myv_ru_2022) is has `myv` language tag but it is not included in Languages search form.
I've also noticed the same problem with `mhr` (see https://huggingface.co/datasets/AigizK/mari-russian-parallel-corpora)

The dataset still can be found, if instead of using the search form you just enter the language code in the url, like https://huggingface.co/datasets?language=language:myv. 

But of course having a more complete list of languages in the search form (or just a fallback to the language codes, if they are missing from the code=>language mapping) would be much more convenient!

Hi! I've opened a PR to make these languages searchable on the Hub.

Thanks @mariosasko!
Do you think it is possible to turn this into a more scalable pipeline? Such as:
1. Looping through all the datasets on the hub and collecting the set of all their language codes;
2. Selecting the codes not covered yet in `Language.ts`
3. Looking up their codes at https://iso639-3.sil.org/code_tables/639/data
4. Adding all the newly found language codes to `Language.ts`

@avidale This has been discussed in https://github.com/huggingface/datasets/issues/4881, so also feel free to share your opinion there.
How to use `load_dataset('glue', 'cola')`

### Describe the bug

I'm new to use HuggingFace datasets but I cannot use `load_dataset('glue', 'cola')`.

- I was stacked by the following problem:

```python
from datasets import load_dataset
cola_dataset = load_dataset('glue', 'cola')

---------------------------------------------------------------------------
InvalidVersion                            Traceback (most recent call last)
File <timed exec>:1

(Omit because of long error message)

File /usr/local/lib/python3.8/site-packages/packaging/version.py:197, in Version.__init__(self, version)
    195 match = self._regex.search(version)
    196 if not match:
--> 197     raise InvalidVersion(f"Invalid version: '{version}'")
    199 # Store the parsed out pieces of the version
    200 self._version = _Version(
    201     epoch=int(match.group("epoch")) if match.group("epoch") else 0,
    202     release=tuple(int(i) for i in match.group("release").split(".")),
   (...)
    208     local=_parse_local_version(match.group("local")),
    209 )

InvalidVersion: Invalid version: '0.10.1,<0.11'
```

- You can check this full error message in my repository: [MLOps-Basics/week_0_project_setup/experimental_notebooks/data_exploration.ipynb](https://github.com/makinzm/MLOps-Basics/blob/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup/experimental_notebooks/data_exploration.ipynb)


### Steps to reproduce the bug

- This is my repository to reproduce: [MLOps-Basics/week_0_project_setup](https://github.com/makinzm/MLOps-Basics/tree/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup)

1. cd `/DockerImage` and command `docker build . -t week0`
2. cd `/` and command `docker-compose up`
3. Run `experimental_notebooks/data_exploration.ipynb`

----

Just to be sure, I wrote down Dockerfile and requirements.txt

- Dockerfile
```Dockerfile
FROM python:3.8

WORKDIR /root/working

RUN apt-get update && \
    apt-get install -y python3-dev python3-pip python3-venv && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .

RUN pip3 install --no-cache-dir jupyter notebook && pip install --no-cache-dir -r requirements.txt

CMD ["bash"]
```

- requirements.txt
```txt
pytorch-lightning==1.2.10
datasets==1.6.2
transformers==4.5.1
scikit-learn==0.24.2
```


### Expected behavior

There is no bug to implement `load_dataset('glue', 'cola')`

### Environment info

I already wrote it.

Sounds like an issue with incompatible `transformers` dependencies versions.

Can you try to update `transformers` ?

EDIT: I checked the `transformers` dependencies and it seems like you need `tokenizers>=0.10.1,<0.11` with `transformers==4.5.1`

EDIT2: this old version of `datasets` seems to import `transformers` but it's no longer the case, so you could also simply update `datasets` and `transformers` won't be imported

Thank you for advising me to update these libraries versions.

I can implement codes using `datasets==2.10.1` and `transformers==4.27.3`
Support for downloading only provided split

We can pass split to `_split_generators()`.
But I'm not sure if it's possible to solve cache issues, mostly with `dataset_info.json`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5668). All of your documentation changes will be reflected on that endpoint.

My previous comment didn't create the retro-link in the PR. I write it here again.

You can check the context and the discussions we had about this feature enhancement in this PR:
- #2249
Feature request: IterableDataset.push_to_hub

### Feature request

It'd be great to have a lazy push to hub, similar to the lazy loading we have with `IterableDataset`.

Suppose you'd like to filter [LAION](https://huggingface.co/datasets/laion/laion400m) based on certain conditions, but as LAION doesn't fit into your disk, you'd like to leverage streaming:
```
from datasets import load_dataset

dataset = load_dataset("laion/laion400m", streaming=True, split="train")
```
Then you could filter the dataset based on certain conditions:
```
filtered_dataset = dataset.filter(lambda example: example['HEIGHT'] > 400)
```

In order to persist this dataset and push it back to the hub, one currently needs to first load the entire filtered dataset on disk and then push:

```
from datasets import Dataset

Dataset.from_generator(filtered_dataset.__iter__).push_to_hub(...)
```
It would be great if we can instead lazy push to the data to the hub (basically stream the data to the hub), not being limited by our disk size:
```
filtered_dataset.push_to_hub("my-filtered-dataset")
```

### Motivation

This feature would be very useful for people that want to filter huge datasets without having to load the entire dataset or a filtered version thereof on their local disk.

### Your contribution

Happy to test out a PR :)

+1

+1

+1, should be possible now? :) https://huggingface.co/blog/xethub-joins-hf

Haha we're working hard to integrate Xet in the HF back-end, it will enable cool use cases :)

Anyway about `IterableDataset.push_to_hub`, I'd be happy to to provide guidance and answer questions if anyone wants to start a first simple implementation of this

+1

+1

+1

+1

Currently running into this when filtering [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) for Dutch entries.

Extra points for somehow making it resumable on error. 11 TB is a lot of data to stream on a home connection without encountering any sort of errors along the way.

If it can help, IterableDataset does implement `.state_dict()` and `.load_state_dict()` that you can use to resume a stream already

+1

+1

+1

Just added a first implementation for `IterableDataset.push_to_hub()` :)

I'll do a new release soon, in the meantime feel free to install `datasets` from source to try it out !
CI is broken: ModuleNotFoundError: jax requires jaxlib to be installed

CI test_py310 is broken: see https://github.com/huggingface/datasets/actions/runs/4498945505/jobs/7916194236?pr=5662
```
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_in_memory - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_on_disk - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_audio - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_device - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_image - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_jnp_array_kwargs - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/features/test_features.py::CastToPythonObjectsTest::test_cast_to_python_objects_jax - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
===== 8 failed, 2147 passed, 10 skipped, 37 warnings in 228.69s (0:03:48) ======
```
integration with imbalanced-learn

### Feature request

Wouldn't it be great if the various class balancing operations from imbalanced-learn were available as part of datasets?

### Motivation

I'm trying to use imbalanced-learn to balance a dataset, but it's not clear how to get the two to interoperate - what would be great would be some examples.  I've looked online, asked gpt-4, but so far not making much progress.

### Your contribution

If I can get this working myself I can submit a PR with example code to go in the docs

You can convert any dataset to pandas to be used with imbalanced-learn using `.to_pandas()`

Otherwise if you want to keep a `Dataset` object and still use e.g. [make_imbalance](https://imbalanced-learn.org/stable/references/generated/imblearn.datasets.make_imbalance.html#imblearn.datasets.make_imbalance), you just need to pass the list of rows ids and labels:

```python
row_indices = list(range(len(dataset)))
resampled_row_indices, _ = make_imbalance(
    row_indices,
    dataset["label"],
    sampling_strategy={0: 25, 1: 50, 2: 50},
    random_state=RANDOM_STATE,
)

resampled_dataset = dataset.select(resampled_row_indices)
```
Doc: save_to_disk, `num_proc` will affect `num_shards`, but it's not documented

### Describe the bug

[`num_proc`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_proc) will affect `num_shards`, but it's not documented


### Steps to reproduce the bug

Nothing to reproduce

### Expected behavior

[document of `num_shards`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_shards) explicitly says that it depends on `max_shard_size`, it should also mention `num_proc`.


### Environment info

datasets main document

I agree this should be documented