mengta666/huggingface_github huggingface dataset:
Issue: Error using `load_datasets`

### Describe the bug

I tried to load a dataset using the `datasets` library in a conda jupyter notebook and got the below error. 

```
ImportError: dlopen(/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib
  Referenced from: <65B094A2-59D7-31AC-A966-4DB9E11D2A15> /Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so
  Reason: tried: '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file, not in dyld cache)
```

### Steps to reproduce the bug

Run the `load_datasets` function

### Expected behavior

I expected the dataset to be loaded into my notebook.

### Environment info

name: review_sense
channels:
  - apple
  - conda-forge
dependencies:
  - python=3.8
  - pip>=19.0
  - jupyter
  - tensorflow-deps
  #- scikit-learn
  #- scipy
  - pandas
  - pandas-datareader
  - matplotlib
  - pillow
  - tqdm
  - requests
  - h5py
  - pyyaml
  - flask
  - boto3
  - ipykernel
  - seaborn
  - pip:
      - tensorflow-macos==2.9
      - tensorflow-metal==0.5.0
      - bayesian-optimization
      - gym
      - kaggle
      - huggingface_hub
      - datasets
      - numpy
      - huggingface


Comments:
It looks like an issue with your installation of scipy, can you try reinstalling it ?

Sorry for the late reply, but that worked @lhoestq . Thanks for the assist.
Issue: Schrödinger's dataset_dict

### Describe the bug

If you use load_dataset('json', data_files="path/test.json"), it will return DatasetDict({train:...}).

And if you use load_dataset("path"), it will return DatasetDict({test:...}).

Why can't the output behavior be unified?

### Steps to reproduce the bug

as description above.

### Expected behavior

consistent predictable output.

### Environment info

'2.11.0'

Comments:
Hi ! Passing `data_files="path/test.json"` is equivalent to `data_files={"train": ["path/test.json"]}`, that's why you end up with a train split. If you don't pass `data_files=`, then split names are inferred from the data files names
Issue: ArrowDataset.save_to_disk lost some logic of remote

### Describe the bug

https://github.com/huggingface/datasets/blob/e7ce0ac60c7efc10886471932854903a7c19f172/src/datasets/arrow_dataset.py#L1371

Here is the bug point, when I want to save from a `DatasetDict` class and the items of the instance is like `[('train', Dataset({features: ..., num_rows: ...}))]` , there is no guarantee that there exists a directory name `train` under `dataset_dict_path`.

### Steps to reproduce the bug

1. Mock a DatasetDict with items like what I said.
2. using save_to_disk with storage_options, u can use local sftp. code may like below

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset.save_to_disk('sftp:///tmp', storage_options={'host': 'localhost', 'username': 'admin'})
```
I suppose u can reproduce the bug by these steps. 

### Expected behavior

Should create the folder if it does not exists, just like we do locally.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-6.2.10-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments:
We just fixed this on `main` and will do a new release soon :)
Issue: train_dataset does not implement __len__

when train using data  precessored by the datasets, I get follow warning and it leads to that I can not set epoch numbers:
`ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.`

Comments:
Thanks for reporting, @v-yunbin.

Could you please give more details, the steps to reproduce the bug, the complete error back trace and the environment information (`datasets-cli env`)?

this is a detail error info from transformers：
```
Traceback (most recent call last):
  File "finetune.py", line 177, in <module>
    whisper_finetune(traindir,devdir,outdir)
  File "finetune.py", line 161, in whisper_finetune
    trainer = Seq2SeqTrainer(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 56, in __init__
    super().__init__(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer.py", line 567, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```


How did you create `train_dataset`? The `datasets` library does not appear in your stack trace.

We need more information in order to reproduce the issue...

```
def asr_dataset(traindir,devdir):
    we_voice = IterableDatasetDict()
    #we_voice["train"] = load_from_disk(traindir,streaming=True)
    #we_voice["test"]= load_from_disk(devdir,streaming=True)
    we_voice["train"] = load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train",streaming=True)
    #print(load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train"))
    we_voice["test"] = load_dataset("csv",data_files=os.path.join(devdir,"dev.csv"), split="train",streaming=True)
    we_voice = we_voice.remove_columns(["id"])
    we_voice = we_voice.cast_column("audio", Audio(sampling_rate=16000))
    return we_voice

```

As you are using iterable datasets (`streaming=True`), their length is not defined.

You should:
- Either use non-iterable datasets, which have a defined length: use `DatasetDict` and not passing `streaming=True`
- Or pass `args.max_steps` to the `Trainer`

I don't know how to give a reasonable args.max_steps...........................

Then you should not use streaming.

@albertvillanova I think @v-yunbin, myself, and others might be slightly confused about max_steps and how it differs from num_train_epochs.

@lkurlandski A **step** is referring to optimizer's update after back propagation, and it's associated with a batch of data. For example, if a dataset contains 64 examples and you have an overall batch size of 4, then an epoch will have 64/4=16 batches. Therefore, in one epoch, you will have 16 optimizer **steps**.
Issue: Support cloud storage for loading datasets

### Feature request

It seems that the the current implementation supports cloud storage only for `load_from_disk`. It would be nice if a similar functionality existed in `load_dataset`.

### Motivation

Motivation is pretty clear -- let users work with datasets located in the cloud. 

### Your contribution

I can help implementing this. 

Comments:
A duplicate of https://github.com/huggingface/datasets/issues/5281
Issue: Tiktoken tokenizers are not pickable 

### Describe the bug

Since tiktoken tokenizer is not pickable, it is not possible to use it inside `dataset.map()` with multiprocessing enabled. However, you [made](https://github.com/huggingface/datasets/issues/5536) tiktoken's tokenizers pickable in `datasets==2.10.0` for caching. For some reason, this logic does not work in dataset processing and raises `TypeError: cannot pickle 'builtins.CoreBPE' object`


### Steps to reproduce the bug

```
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
    num_proc=2,
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

starts processing dataset 

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-1021-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.0

Comments:
Thanks for reporting, @markovalexander.

Unfortunately, I'm not able to reproduce the issue: the `tiktoken` tokenizer can be used within `Dataset.map`, both in my local machine and in a Colab notebook: https://colab.research.google.com/drive/1DhJroZgk0sNFJ2Mrz-jYgrmh9jblXaCG?usp=sharing

Are you sure you are using `datasets` version 2.11.0?
Issue: How to use Distill-BERT with different datasets?

### Describe the bug

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Steps to reproduce the bug

I recently read [this](https://huggingface.co/docs/transformers/quicktour#train-with-tensorflow:~:text=The%20most%20important%20thing%20to%20remember%20is%20you%20need%20to%20instantiate%20a%20tokenizer%20with%20the%20same%20model%20name%20to%20ensure%20you%E2%80%99re%20using%20the%20same%20tokenization%20rules%20a%20model%20was%20pretrained%20with.) and was wondering how to use distill-BERT (which is pre-trained with imdb dataset) with a different dataset (for eg. [this](https://huggingface.co/datasets/yhavinga/imdb_dutch) dataset)?

### Expected behavior

Distill-BERT should work with different datasets.

### Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 11.0.0

Comments:
Closing this one in favor of the same issue opened in the `transformers` repo.
Issue: Not able to load the pile

### Describe the bug

Got this error when I am trying to load the pile dataset 

```
TypeError: Couldn't cast array of type
struct<file: string, id: string>
to
{'id': Value(dtype='string', id=None)}

```


### Steps to reproduce the bug

Please visit the following sample notebook 

https://colab.research.google.com/drive/1JHcjawcHL6QHhi5VcqYd07W2QCEj2nWK#scrollTo=ulJP3eJCI-tB

### Expected behavior

The pile should work

### Environment info

-  `datasets` version: 2.11.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Thanks for reporting, @surya-narayanan.

I see you already started a discussion about this on the Community tab of the corresponding dataset: https://huggingface.co/datasets/EleutherAI/the_pile/discussions/10
Let's continue the discussion there!
Issue: Can I load in list of list of dict format? 

### Feature request

my jsonl dataset has following format:
```
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
```

I try to use `datasets.load_dataset('json', data_files=path)` or `datasets.Dataset.from_json`, it raises
```
  File "site-packages/datasets/arrow_dataset.py", line 1078, in from_json
    ).read()
  File "site-packages/datasets/io/json.py", line 59, in read
    self.builder.download_and_prepare(
  File "site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "site-packages/datasets/builder.py", line 967, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "site-packages/datasets/builder.py", line 1749, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "site-packages/datasets/builder.py", line 1892, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Motivation

I wanna use features like `Datasets.map` or `Datasets.shuffle`, so i need the dataset in memory to be `arrow_dataset.Datasets` format

### Your contribution

PR

Comments:
Thanks for reporting, @LZY-the-boys.

Could you please give more details about what is your intended dataset structure? What are the names of the columns and the value of each row?

Currently, the JSON-Lines format is supported:
- Each line correspond to one row of the dataset
- Each line is composed of one JSON object, where the names are the names of the columns, and the values are the values for the row-column pair.
Issue: Calling shuffle on a IterableDataset with streaming=True, gives "ValueError: cannot reshape array"

### Describe the bug

When calling shuffle on a IterableDataset with streaming=True, I get the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 937, in __iter__
    for key, example in ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 627, in __iter__
    for x in self.ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 138, in __iter__
    yield from self.generate_examples_fn(**kwargs_with_shuffled_shards)
  File "/home/administrator/.cache/huggingface/modules/datasets_modules/datasets/mnist/fda16c03c4ecfb13f165ba7e29cf38129ce035011519968cdaf74894ce91c9d4/mnist.py", line 111, in _generate_examples
    images = np.frombuffer(f.read(), dtype=np.uint8).reshape(size, 28, 28)
ValueError: cannot reshape array of size 59992 into shape (60000,28,28)

```
Tested with the fashion_mnist and mnist datasets

### Steps to reproduce the bug

Code to reproduce
```python
from datasets import load_dataset

SHUFFLE_SEED = 42
SHUFFLE_BUFFER_SIZE = 10_000

dataset = load_dataset('fashion_mnist', streaming=True).shuffle(seed=SHUFFLE_SEED, buffer_size=SHUFFLE_BUFFER_SIZE)
next(iter(dataset['train']))
```

### Expected behavior

A random item from the dataset and no error

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0


Comments:
Hi! I've merged a PR on the Hub with a fix: https://huggingface.co/datasets/fashion_mnist/discussions/3

Thanks, this appears to have fixed the issue.

I've created a PR for the same change in the mnist dataset: https://huggingface.co/datasets/mnist/discussions/3/files
Issue: ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 

### Describe the bug

The module moved to new place?

### Steps to reproduce the bug

in the import step,

```python
from datasets.utils.deprecation_utils import DeprecatedEnum
```

error:

```
ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
```

### Expected behavior

import successfully

### Environment info

python==3.9.16
datasets==1.18.3

Comments:
update the version. fix
Issue: dataclass.py in virtual environment is   overriding the stdlib module "dataclasses"

### Describe the bug

"e:\Krish_naik\FSDSRegression\venv\Lib\dataclasses.py" is overriding the stdlib module "dataclasses"

### Steps to reproduce the bug

module issue

### Expected behavior

 overriding the stdlib module "dataclasses"

### Environment info

VS code

Comments:
We no longer depend on `dataclasses` (for almost a year), so I don't think our package is the problematic one. 

I think it makes more sense to raise this issue in the `dataclasses` repo: https://github.com/ericvsmith/dataclasses.
Issue: load_dataset("text","dataset.txt") loads the wrong dataset!

### Describe the bug

I am trying to load my own custom text dataset using the load_dataset function.  My dataset is a bunch of ordered text, think along the lines of shakespeare plays.  However, after I load the dataset and I inspect it, the dataset is a table with a bunch of latitude and longitude values!  What in the world??

### Steps to reproduce the bug

my_dataset = load_dataset("text","TextFile.txt")
my_dataset

### Expected behavior

I expected the dataset to contain the actual data from the text document that I used.

### Environment info

Google Colab

Comments:
You need to provide a text file as `data_files`, not as a configuration:

```python
my_dataset = load_dataset("text", data_files="TextFile.txt")
```

Otherwise, since `data_files` is `None`, it picks up Colab's sample datasets from the `content` dir.
Issue: ClassLabel Error 

### Describe the bug

I still getting the error "call() takes 1 positional argument but 2 were given" even after ensuring that the value being passed to the label object is a single value and that the ClassLabel object has been created with the correct number of label classes

### Steps to reproduce the bug

from datasets import ClassLabel, Dataset

1. Create the ClassLabel object with 3 label values and their corresponding names
label_test = ClassLabel(num_classes=3, names=["label_1", "label_2", "label_3"])

2. Define a dictionary with text and label fields
data = {
    'text': ['text_1', 'text_2', 'text_3'],
    'label': [1, 2, 3],
}

3. Create a Hugging Face dataset from the dictionary
dataset = Dataset.from_dict(data)
print(dataset.features)
4. Map the label values to their corresponding label names using the label object
dataset = dataset.map(lambda example: {'text': example['text'], 'label': label_test(example['label'])})

5. Print the resulting dataset
print(dataset)

### Expected behavior

I hope my label type is class label instead int. 

### Environment info

python 3.9
google colab

Comments:
Hi, you can use the `cast_column` function to change the feature type from a `Value(int64)` to `ClassLabel`:

```py
dataset = dataset.cast_column("label", ClassLabel(names=["label_1", "label_2", "label_3"]))
print(dataset.features)
{'text': Value(dtype='string', id=None),
 'label': ClassLabel(names=['label_1', 'label_2', 'label_3'], id=None)}
```

thank you @stevhliu, its worked.
Issue: FORCE_REDOWNLOAD raises "Directory not empty" exception on second run

### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2


Comments:
Hi ! I couldn't reproduce your issue :/

It seems that `shutil.rmtree` failed. It is supposed to work even if the directory is not empty, but you still end up with `OSError: [Errno 39] Directory not empty:`. Can you make sure another process is not using this directory at the same time ?

I have the same error with `datasets==2.14.5` and `pyarrow==13.0.0`. Python 3.10.13

I have same error. Any workaround?
Issue: Enwik8 should support the standard split

### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR. 

Comments:
#self-assign

The Enwik8 pipeline is not present in this codebase, and is hosted elsewhere. I have opened a PR [there](https://huggingface.co/datasets/enwik8/discussions/4) instead.
Issue: Fallback JSON Dataset loading does not load all values when features specified manually

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10

Comments:
Thanks for reporting, @myluki2000.

I am working on a fix.
Issue: How to limit the number of examples in dataset, for testing?

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special

Comments:
Hi! You can use the `nrows` parameter for this:
```python
data = load_dataset("json", data_files=data_path, nrows=10)
```

@mariosasko I get:

`TypeError: __init__() got an unexpected keyword argument 'nrows'`

I misread the format in which the dataset is stored -  the `nrows` parameter works for CSV, but not JSON.

This means the only option is first to create a DataFrame and then convert it to a Dataset object:
```python
import pandas as pd
from datasets import Dataset

df = pd.read_json(data_path, lines=True, nrows=10)
ds = Dataset.from_pandas(df)
```
Issue: Handle empty audio

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`

Comments:
Hi! Can you share one of the problematic audio files with us?

I tried to reproduce the error with the following code: 
```python
import soundfile as sf
import numpy as np
from datasets import Audio

sf.write("empty.wav",  np.array([]), 16000)
Audio(sampling_rate=24000).decode_example({"path": "empty.wav", "bytes": None})
```
But without success.

Also, what version of `librosa` is installed in your env? (You can get this info with `python -c "import librosa; print(librosa.__version__)`)



I'm closing this issue as the reproducer hasn't been provided.
Issue: Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks

Comments:
Hi! 

You can use [`.set_format("np")`](https://huggingface.co/docs/datasets/process#format) to get NumPy arrays (or Pytorch tensors with `.set_format("torch")`) in `__getitem__`.

Also, have you been able to reproduce the linked PyTorch issue with a HF dataset?
Issue: Dataset sizes are in MiB instead of MB in dataset cards

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations

Comments:
Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5

looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`

I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.

yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">


I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files

Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example

First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.

The bulk edit parsed 751 canonical datasets and updated 166.

Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">


I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6

should we force merge the PR and close this issue?

I merged the PRs for "scicite" and "scifact".
Issue: Getting next item from IterableDataset took forever.

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12

Comments:
Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...

Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
Issue: Add Qdrant as another search index

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.

Comments:
@mariosasko I'd appreciate your feedback on this.
Issue: Dataset configuration

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md

Comments:
Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit

From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).

Current format:
```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
Issue: Document to compress data files before uploading

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1

Comments:
Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example). 
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

Perfect, thanks for making the distinction between compression and data extensions!
Issue: Add information about patterns search order to the doc about structuring repo

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....

Comments:
Good idea, I think I've seen this a couple of times before too on the forums. I can work on this :)

Closed in #5693
Issue: Allow load_dataset to take a working dir for intermediate data

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.

Comments:
Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.

In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache. 

You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk

If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?
Issue: Add support to create a Dataset from spark dataframe

### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.

Comments:
if i read spark Dataframe , got an error on  multi-node Spark cluster.
Did the Api (Dataset.from_spark) support Spark cluster, read dataframe and save_to_disk?

Error: 
_pickle.PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforma
tion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
23/06/16 21:17:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)



How to perform predictions on Dataset object in Spark with multi-node cluster parallelism?

Addressed in #5701

Hi ! for your information we are working on some more documentation on how to use Spark with HF Datasets repositories (without the need for the `datasets` library) 
https://github.com/huggingface/datasets/issues/5678

Cc @lu-wang-dl @maddiedawson let me know what you think !

sorry, wrong link: https://github.com/huggingface/hub-docs/pull/1392
Issue: Filter datasets by language code

Hi! I use the language search field on https://huggingface.co/datasets
However, some of the datasets tagged by ISO language code are not accessible by this search form.
For example, [myv_ru_2022](https://huggingface.co/datasets/slone/myv_ru_2022) is has `myv` language tag but it is not included in Languages search form.
I've also noticed the same problem with `mhr` (see https://huggingface.co/datasets/AigizK/mari-russian-parallel-corpora)

Comments:
The dataset still can be found, if instead of using the search form you just enter the language code in the url, like https://huggingface.co/datasets?language=language:myv. 

But of course having a more complete list of languages in the search form (or just a fallback to the language codes, if they are missing from the code=>language mapping) would be much more convenient!

Hi! I've opened a PR to make these languages searchable on the Hub.

Thanks @mariosasko!
Do you think it is possible to turn this into a more scalable pipeline? Such as:
1. Looping through all the datasets on the hub and collecting the set of all their language codes;
2. Selecting the codes not covered yet in `Language.ts`
3. Looking up their codes at https://iso639-3.sil.org/code_tables/639/data
4. Adding all the newly found language codes to `Language.ts`

@avidale This has been discussed in https://github.com/huggingface/datasets/issues/4881, so also feel free to share your opinion there.
Issue: How to use `load_dataset('glue', 'cola')`

### Describe the bug

I'm new to use HuggingFace datasets but I cannot use `load_dataset('glue', 'cola')`.

- I was stacked by the following problem:

```python
from datasets import load_dataset
cola_dataset = load_dataset('glue', 'cola')

---------------------------------------------------------------------------
InvalidVersion                            Traceback (most recent call last)
File <timed exec>:1

(Omit because of long error message)

File /usr/local/lib/python3.8/site-packages/packaging/version.py:197, in Version.__init__(self, version)
    195 match = self._regex.search(version)
    196 if not match:
--> 197     raise InvalidVersion(f"Invalid version: '{version}'")
    199 # Store the parsed out pieces of the version
    200 self._version = _Version(
    201     epoch=int(match.group("epoch")) if match.group("epoch") else 0,
    202     release=tuple(int(i) for i in match.group("release").split(".")),
   (...)
    208     local=_parse_local_version(match.group("local")),
    209 )

InvalidVersion: Invalid version: '0.10.1,<0.11'
```

- You can check this full error message in my repository: [MLOps-Basics/week_0_project_setup/experimental_notebooks/data_exploration.ipynb](https://github.com/makinzm/MLOps-Basics/blob/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup/experimental_notebooks/data_exploration.ipynb)


### Steps to reproduce the bug

- This is my repository to reproduce: [MLOps-Basics/week_0_project_setup](https://github.com/makinzm/MLOps-Basics/tree/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup)

1. cd `/DockerImage` and command `docker build . -t week0`
2. cd `/` and command `docker-compose up`
3. Run `experimental_notebooks/data_exploration.ipynb`

----

Just to be sure, I wrote down Dockerfile and requirements.txt

- Dockerfile
```Dockerfile
FROM python:3.8

WORKDIR /root/working

RUN apt-get update && \
    apt-get install -y python3-dev python3-pip python3-venv && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .

RUN pip3 install --no-cache-dir jupyter notebook && pip install --no-cache-dir -r requirements.txt

CMD ["bash"]
```

- requirements.txt
```txt
pytorch-lightning==1.2.10
datasets==1.6.2
transformers==4.5.1
scikit-learn==0.24.2
```


### Expected behavior

There is no bug to implement `load_dataset('glue', 'cola')`

### Environment info

I already wrote it.

Comments:
Sounds like an issue with incompatible `transformers` dependencies versions.

Can you try to update `transformers` ?

EDIT: I checked the `transformers` dependencies and it seems like you need `tokenizers>=0.10.1,<0.11` with `transformers==4.5.1`

EDIT2: this old version of `datasets` seems to import `transformers` but it's no longer the case, so you could also simply update `datasets` and `transformers` won't be imported

Thank you for advising me to update these libraries versions.

I can implement codes using `datasets==2.10.1` and `transformers==4.27.3`