jamesLeeeeeee/datasets-github-issues huggingface dataset:
Issue with train_test_split maintaining the same underlying PyArrow Table
Author: Oziel14 | Pull request: no | State: open

### Describe the bug

I've been using the train_test_split method in the datasets module to split my HuggingFace Dataset into separate training, validation, and testing subsets. However, I've noticed an issue where the split datasets appear to maintain the same underlying PyArrow Table.

### Steps to reproduce the bug

1. Load any dataset  ```dataset = load_dataset("lhoestq/demo1")``` 
2. Try the next code:
```python
from datasets import Dataset, DatasetDict

train_size = 0.6

split_train = dataset["train"].train_test_split(
    train_size=train_size,
)

separate_dataset_dict = DatasetDict({
    "train": split_train["train"],
    "test": split_train["test"],
})
```
3.  The next code ```print(separate_dataset_dict)``` when printing the dataset it gives the indication that they have 3 and 2 rows respectively.
4. But the next code: 
 ```python
print(len(separate_dataset_dict["train"].data['id']))
print(len(separate_dataset_dict["test"].data['id'])) 
```

 Indicates that both tables still have 5 rows.

### Expected behavior

However, I've noticed that train_test_split["train"].data, test_val_split["train"].data, and test_val_split["test"].data are identical, suggesting that they all point to the same underlying PyArrow Table. This means that the split datasets are not independent, as I expected.

I believe this is a bug in the train_test_split implementation, as I would expect this function to return datasets with separate underlying PyArrow Tables. Could you please help me understand if this is expected behavior, or if there's a workaround to create truly independent split datasets?

I would appreciate any assistance with this issue. Thank you.

### Environment info

I tried in Colab:

- `datasets` version: 2.13.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

and my PC:

- `datasets` version: 2.13.0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
IterableDataset: split by node and map may preprocess samples that will be skipped anyway
Author: johnchienbronci | Pull request: no | State: open

              There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051_
read metric glue.py from local file 
Author: JiazhaoLi | Pull request: no | State: closed

### Describe the bug

Currently, The server is off-line.  I am using the glue metric from the local file downloaded from the hub. 
I download / cached datasets using `load_dataset('glue','sst2', cache_dir='/xxx')` to cache them and then in the off-line mode, I use `load_dataset('xxx/glue.py','sst2', cache_dir='/xxx')`. I can successfully reuse cached datasets.

My problem is about the load_metric. 
When I run `load_dataset('xxx/glue_metric.py','sst2',cache_dir='/xxx')` , it returns 

`  File "xx/lib64/python3.9/site-packages/datasets/utils/deprecation_utils.py", line 46, in wrapper
    return deprecated_function(*args, **kwargs)
  File "xx//lib64/python3.9/site-packages/datasets/load.py", line 1392, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable`

Thanks in advance for help! 
### Steps to reproduce the bug

N/A

### Expected behavior

N/A

### Environment info

`datasets == 2.12.0`
Bad error message when trying to download gated dataset
Author: patrickvonplaten | Pull request: no | State: closed

### Describe the bug

When I attempt to download a model from the Hub that is gated without being logged in, I get a nice error message. E.g.:

E.g.
```sh
Repository Not Found for url: https://huggingface.co/api/models/DeepFloyd/IF-I-XL-v1.0.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password..
Will try to load from local cache.
```

If I do the same for a gated dataset on the Hub, I'm not gated a nice error message IMO:

```sh
File ~/hf/lib/python3.10/site-packages/fsspec/implementations/http.py:430, in HTTPFileSystem._info(self, url, **kwargs)
    427     except Exception as exc:
    428         if policy == "get":
    429             # If get failed, then raise a FileNotFoundError
--> 430             raise FileNotFoundError(url) from exc
    431         logger.debug(str(exc))
    433 return {"name": url, "size": None, **info, "type": "file"}

FileNotFoundError: https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0/resolve/main/n_shards.json
```

### Steps to reproduce the bug

```
huggingface-cli logout
```

and then:

```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# Swahili
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "sw", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
sw_sample = next(iter(stream_data))["audio"]["array"]
```

### Expected behavior

Better error message

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.12.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
What is the Right way to use discofuse dataset??
Author: akesh1235 | Pull request: no | State: closed

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?
Support for data with instance-wise dictionary as features
Author: richardwth | Pull request: no | State: open

### Feature request

I notice that when loading data instances with feature type of python dictionary, the dictionary keys would be broadcast so that every instance has the same set of keys. Please see an example in the Motivation section.

It is possible to avoid this behavior, i.e., load dictionary features as it is and do not broadcast the keys among instances? Please note that these dictionaries would have to be processed dynamically at each training iteration into strings (and tokenized).

### Motivation

I am trying to load a dataset from a json file. Each instance of the dataset has a feature that is a dictionary but its keys depend on the instance. Every two instances may have different keys. For example, imagine a dataset that contains a set of math expressions from a bunch of mutually redundant expressions:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
    }
},
...
{
    "index": 9999,
    "feature": {
        "x >= 6": ["x >= 6", "x >= 0", "x >= -1"],
        ...
    }
},
...
```
When directly loading the dataset using `data = load_dataset("json", data_files=file_paths, split='train')`, each instance would have all the keys from other instances and None as values. That is, instance of index 0 becomes:
```
{
    "index": 0,
    "feature": {
        "2 * x + y >= 3": ["2 * x + y >= 3", "4 * x + 2 * y >= 6"],
        ...
        "x >= 6": None,  # keys from other instances
        ...
    }
},
```
This is not desirable. Moreover, issue would be raised if I attempt to combine two such datasets using `data = concatenate_datasets(multi_datasets)`, perhaps because their dictionary features contain different keys.

A solution I can think of is to store the dictionary features as a long string, and evaluate it later. Please kindly suggest any other solution using existing methods of datasets.

### Your contribution

N/A
Replace metadata utils with `huggingface_hub`'s RepoCard API
Author: mariosasko | Pull request: yes | State: closed

Use `huggingface_hub`'s RepoCard API instead of `DatasetMetadata` for modifying the card's YAML, and deprecate `datasets.utils.metadata` and `datasets.utils.readme`.

After removing these modules, we can also delete `datasets.utils.resources` since the moon landing repo now stores its own version of these resources for the metadata UI.

PS: this change requires bumping `huggingface_hub` to 0.13.0 (Transformers requires 0.14.0, so should be ok)
Return the audio filename when decoding fails due to corrupt files
Author: wetdog | Pull request: no | State: open

### Feature request

Return the audio filename when the audio decoding fails. Although currently there are some checks for mp3 and opus formats with the library version there are still cases when the audio decoding could fail, eg. Corrupt file. 

### Motivation

When you try to load an object file dataset and the decoding fails you can't know which file is corrupt
```

raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7f5ab7e38290>: Format not recognised.
```

### Your contribution

Make a PR to Add exceptions for LIbsndfileError to return the audio filename or path when soundfile decoding fails.
Failing to upload dataset to the hub
Author: Ar770 | Pull request: no | State: closed

### Describe the bug

Trying to upload a dataset of hundreds of thousands of audio samples (the total volume is not very large, 60 gb) to the hub with push_to_hub, it doesn't work.
From time to time one piece of the data (parquet) gets pushed and then I get RemoteDisconnected even though my internet is stable.
Please help.
I'm trying to upload the dataset for almost a week.
Thanks

### Steps to reproduce the bug

not relevant 

### Expected behavior

Be able to upload thedataset

### Environment info

python: 3.9
Pass datasets-cli additional args as kwargs to DatasetBuilder in `run_beam.py`
Author: graelo | Pull request: yes | State: open

Hi,

Following this <https://discuss.huggingface.co/t/how-to-preprocess-a-wikipedia-dataset-using-dataflowrunner/41991/3>, here is a simple PR to pass any additional args to datasets-cli as kwargs in the DatasetBuilder in `run_beam.py`.

I also took the liberty to add missing setup steps to the `beam.mdx` docs in order to help everyone.

@lhoestq
Load Data Sets Too Slow In Train Seq2seq Model
Author: xyx361100238 | Pull request: no | State: closed

### Describe the bug


step 'Generating train split' in load_dataset is too slow：
![image](https://github.com/huggingface/datasets/assets/19569322/d9b08eee-95fe-4741-a346-b70416c948f8)


### Steps to reproduce the bug

Data： own data，16K16B Mono wav
Oficial Script:[ run_speech_recognition_seq2seq.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py)
Add Code：
    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)
 （change cache_dir to other path ，ex:/DATA/cache）

### Expected behavior

load data fast,at least 1000+
`Generating train split: 387875 examples [32:24:45, 1154.83 examples/s]`

### Environment info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
Make get_from_cache use custom temp filename that is locked
Author: albertvillanova | Pull request: yes | State: closed

This PR ensures that the temporary filename created is the same as the one that is locked, while writing to the cache.

This PR stops using `tempfile` to generate the temporary filename.

Additionally, the behavior now is aligned for both `resume_download` `True` and `False`.

Refactor temp_file_manager so that it uses the filename that is locked: 
- Use: `cache_path + ".incomplete"`, when the locked one is `cache_path + ".lock"`

Before it was using `tempfile` inside `cache_dir`, which was not locked: although very improbable name collision (8 random characters), this was not impossible when huge number of multiple processes.

Maybe related to "Stale file handle" issues caused by `tempfile`: 
- [ ] https://huggingface.co/datasets/tapaco/discussions/4
- [ ] https://huggingface.co/datasets/xcsr/discussions/1
- [ ] https://huggingface.co/datasets/covost2/discussions/3
```
Error code:   ConfigNamesError
Exception:    OSError
Message:      [Errno 116] Stale file handle
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 61, in compute_config_names_response
                  for config in sorted(get_dataset_config_names(path=dataset, use_auth_token=use_auth_token))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1188, in dataset_module_factory
                  return HubDatasetModuleFactoryWithScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 907, in get_module
                  dataset_readme_path = self.download_dataset_readme_file()
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 896, in download_dataset_readme_file
                  return cached_path(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 183, in cached_path
                  output_path = get_from_cache(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 611, in get_from_cache
                  http_get(
                File "/usr/local/lib/python3.9/tempfile.py", line 496, in __exit__
                  result = self.file.__exit__(exc, value, tb)
              OSError: [Errno 116] Stale file handle
```
- the stale file handle error can be raised when `tempfile` tries to close (when exiting its context manager) a filename that has been already closed by other process
  - note that `tempfile` filenames are randomly generated but not locked in our code

CC: @severo
Fix `to_numpy` when None values in the sequence
Author: qgallouedec | Pull request: yes | State: closed

Closes #5927 
I've realized that the error was overlooked during testing due to the presence of only one None value in the sequence.
Unfortunately, it was the only case where the function works as expected. When the sequence contained more than one None value, the function failed. Consequently, I've updated the tests to include sequences with multiple None values.
`datasets.map` not reusing cached copy by default
Author: bhavitvyamalik | Pull request: no | State: closed

### Describe the bug

When I load the dataset from local directory, it's cached copy is picked up after first time. However, for `map` operation, the operation is applied again and cached copy is not picked up. Is there any way to pick cached copy instead of processing it again? The only solution I could think of was to use `save_to_disk` after my last transform and then use that in my DataLoader pipeline. Are there any other solutions for the same?

One more thing, my dataset is occupying 6GB storage memory after I use `map`, is there any way I can reduce that memory usage?


### Steps to reproduce the bug

```
# make sure that dataset decodes audio with correct sampling rate
dataset_sampling_rate = next(iter(self.raw_datasets.values())).features["audio"].sampling_rate
if dataset_sampling_rate != self.feature_extractor.sampling_rate:
    self.raw_datasets = self.raw_datasets.cast_column(
        "audio", datasets.features.Audio(sampling_rate=self.feature_extractor.sampling_rate)
    )

vectorized_datasets = self.raw_datasets.map(
    self.prepare_dataset,
    remove_columns=next(iter(self.raw_datasets.values())).column_names,
    num_proc=self.num_workers,
    desc="preprocess datasets",
)
# filter data that is longer than max_input_length
self.vectorized_datasets = vectorized_datasets.filter(
    self.is_audio_in_length_range,
    num_proc=self.num_workers,
    input_columns=["input_length"],
        )

def prepare_dataset(self, batch):
    # load audio
    sample = batch["audio"]
    inputs = self.feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])

    batch["labels"] = self.tokenizer(batch["target_text"]).input_ids
    return batch

```

### Expected behavior

`map` to use cached copy and if possible an alternative technique to reduce memory usage after using `map`

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
Importing PyTorch reduces multiprocessing performance for map
Author: Maxscha | Pull request: no | State: closed

### Describe the bug

I noticed that the performance of my dataset preprocessing with `map(...,num_proc=32)` decreases when PyTorch is imported.

### Steps to reproduce the bug

I created two example scripts to reproduce this behavior:

```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
Takes around 4 seconds on my machine.

While the same code, but with an `import torch`:
```
import datasets
datasets.disable_caching()

from datasets import Dataset
import time
import torch
    
PROC=32

if __name__ == "__main__":
    dataset = [True] * 10000000
    dataset = Dataset.from_dict({'train': dataset})
    

    start = time.time()
    dataset.map(lambda x: x, num_proc=PROC)
    end = time.time()
    print(end - start)
```
takes around 22 seconds.



### Expected behavior

I would expect that the import of torch to not have such a significant effect on the performance of map using multiprocessing.


### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
- torch: 2.0.1
Breaking API change in datasets.list_datasets caused by change in HfApi.list_datasets
Author: mtkinit | Pull request: no | State: closed

### Describe the bug

Hi all,

after an update of the `datasets` library, we observer crashes in our code. We relied on `datasets.list_datasets` returning a `list`. Now, after the API of the HfApi.list_datasets was changed and it returns a `list` instead of an `Iterable`, the `datasets.list_datasets` now sometimes returns a `list` and somesimes an `Iterable`. 

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

Thanks,
Martin 

### Steps to reproduce the bug

Here, the code crashed after we updated the `datasets` library:

```python
# list_datasets no longer returns a list, which leads to an error when one tries to slice it
for datasets.list_datasets(with_details=True)[:limit]:
    ...
```

### Expected behavior

It would be helpful to indicate that by the return type of the `datasets.list_datasets` function.

### Environment info

Ubuntu 22.04
datasets 2.12.0
Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility
Author: ehuangc | Pull request: no | State: closed

### Describe the bug

When trying to import datasets, I get a pyarrow ValueError:

Traceback (most recent call last):
  File "/Users/edward/test/test.py", line 1, in <module>
    import datasets
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 65, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_reader.py", line 28, in <module>
    import pyarrow.parquet as pq
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 20, in <module>
    from .core import *
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 45, in <module>
    from pyarrow.fs import (LocalFileSystem, FileSystem, FileType,
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/fs.py", line 49, in <module>
    from pyarrow._gcsfs import GcsFileSystem  # noqa
  File "pyarrow/_gcsfs.pyx", line 1, in init pyarrow._gcsfs
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

`import datasets`

### Expected behavior

Successful import

### Environment info

Conda environment, MacOS
python 3.9.12
datasets 2.12.0