noamaanMulla-03/datasets-issues huggingface dataset:
Error when loading a big local json file

### Describe the bug

When trying to load big json files from a local directory, `load_dataset` throws the following error

```
Traceback (most recent call last):
  File "/miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/builder.py", line 1989, in _prepare_split_single
    writer.write_table(table)
  File "miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/arrow_writer.py", line 573, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 3638, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
```

### Steps to reproduce the bug

1. Download a big file, e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-train.json.gz`
2. Load it like `data = load_dataset("json", data_files=["nq-train.json"], split="train")`

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-train.json"], split="train")
```

A similarly formatted but smaller file, e.g. e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-dev.json.gz` is loaded without issues

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-dev.json"], split="train")
```

### Expected behavior

It should load normally

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


I get similar when dealing with a large jsonl file (6k lines), 

> TypeError: Couldn't cast array of type timestamp[us] to null

Yet when I split it into 1k lines, files, load_dataset works fine!

https://github.com/huggingface/course/issues/692



What's the proposed solution? :-)
Dataset.from_pandas with a DataFrame of PIL.Images

Currently type inference doesn't know what to do with a Pandas Series of PIL.Image objects, though it would be nice to get a Dataset with the Image type this way

Labels: name: enhancement

A duplicate of https://github.com/huggingface/datasets/issues/4796.

We could get this for free by implementing the `Image` feature as an extension type, as shown in [this](https://colab.research.google.com/drive/1Uzm_tXVpGTwbzleDConWcNjacwO1yxE4?usp=sharing) Colab (example with UUIDs).


+1 to this
Calling this line with a df that contains a PIL image (as they are returned from load_dataset)
`ds = Dataset.from_pandas(df)`
Results in this error:
`ArrowInvalid: ('Could not convert <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x2B41F2D70> with type PngImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column image with type object')`

I found something that can be used as solution.

I have the same problem when I've try to load the images from a pamdas dataset

If you have all on a pandas dataset try 
Dataset.from_dict( your_df.reset_index(drop=True).to_dict(orient='list'), split=set_your_split)

And this avoid the error
Let server decide default repo visibility

Until now, all repos were public by default when created without passing the `private` argument. This meant that passing `private=False` or `private=None` was strictly the same. This is not the case anymore. Enterprise Hub offers organizations to set a default visibility setting for new repos. This is useful for organizations forbidding public repos for security matters. This PR mostly updates docstrings + default values so that `private=None` is always passed when users don't set it manually.

This PR doesn't create any breaking change. The real update has been done server-side when introducing the new Enterprise Hub feature. Related to https://github.com/huggingface/huggingface_hub/pull/2679.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

No need for a specific version of huggingface_hub to avoid a breaking change no (it's a server-side change)
wrong return type for `IterableDataset.shard()`

### Describe the bug

`IterableDataset.shard()` has the wrong typing for its return as `"Dataset"`. It should be `"IterableDataset"`. Makes my IDE unhappy.

### Steps to reproduce the bug

look at [the source code](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py#L2668)?

### Expected behavior

Correct return type as `"IterableDataset"`

### Environment info

datasets==3.1.0


Oops my bad ! thanks for reporting
fix: update elasticsearch version

This should fix the `test_py311 (windows latest, deps-latest` errors.

```
=========================== short test summary info ===========================
ERROR tests/test_search.py - AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.
ERROR tests/test_search.py - AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.
===== 2822 passed, 54 skipped, 10 warnings, 2 errors in 373.36s (0:06:13) =====
Error: Process completed with exit code 1.
```

The elasticsearch version used is `elasticsearch==7.9.1`, which is 4 years old and uses the removed `numpy.float_`.

elasticsearch fixed this in [https://github.com/elastic/elasticsearch-py/pull/2551](https://github.com/elastic/elasticsearch-py/pull/2551) and released in 8.15.0 (August 2024) and 7.17.12 (September 2024).



May I request a review @lhoestq

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7300). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
OSError: Not enough disk space.

## Describe the bug
I'm trying to download `natural_questions` dataset from the Internet, and I've specified the cache_dir which locates in a mounted disk and has enough disk space. However, even though the space is enough, the disk space checking function still reports the space of root `/` disk having no enough space. 

The file system structure is like below. The root `/` has `115G` disk space available, and the `sda1` is mounted to `/mnt`, which has `1.2T` disk space available:
```
/
/mnt/sda1/path/to/args.dataset_cache_dir 
```

## Steps to reproduce the bug
```python
dataset_config = DownloadConfig(
    cache_dir=os.path.abspath(args.dataset_cache_dir),
    resume_download=True,
)
dataset = load_dataset("natural_questions", download_config=dataset_config)
```

## Expected results

Can download the dataset without an error.

## Actual results

The following error raised:
```
OSError: Not enough disk space. Needed: 134.92 GiB (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Ubuntu 18.04
- Python version: 3.8.10
- PyArrow version:


Labels: name: bug

Maybe we can change the disk space calculating API from `shutil.disk_usage` to  `os.statvfs` in UNIX-like system, which can provide correct results.
```
statvfs = os.statvfs('path')
avail_space_bytes = statvfs.f_frsize * statvfs.f_bavail
```

Hi @qqaatw, thanks for reporting.

Could you please try:
```python
dataset = load_dataset("natural_questions", cache_dir=os.path.abspath(args.dataset_cache_dir))
```

@albertvillanova it works! Thanks for your suggestion. Is that a bug of `DownloadConfig`?

`DownloadConfig` only sets the location to download the files. On the other hand, `cache_dir` sets the location for both downloading and caching the data. You can find more information here: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory

I had encountered the same error when running a command `ds = load_dataset('food101')` in a docker container. The error I got: `OSError: Not enough disk space. Needed: 9.43 GiB (download: 4.65 GiB, generated: 4.77 GiB, post-processed: Unknown size)`

In case anyone encountered the same issue, this was my fix:

```sh
# starting the container (mount project directory onto /app, so that the code and data in my project directory are available in the container)
docker run -it --rm -v $(pwd):/app my-demo:latest bash
```


```python
# other code ...
ds = load_dataset('food101', cache_dir="/app/data") # set cache_dir to the absolute path of a directory (e.g. /app/data) that's mounted from the host (MacOS in my case) into the docker container

# this assumes ./data directory exists in your project folder. If not, create it or point it to any other existing directory where you want to store the cache
```

Thanks @albertvillanova for posting the fix above :-) 

import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
Not enough disk space (Needed: Unknown size) when caching on a cluster

I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?


Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ?

Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
```

That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ?

I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue

Were you able to figure it out?

@philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package

@olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import?

I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```

import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
Creating new dataset from list loses information. (Audio Information Lost - either Datatype or Values).

### Describe the bug

When creating a dataset from a list of datapoints, information is lost of the individual items.

Specifically, when creating a dataset from a list of datapoints (from another dataset). Either the datatype is lost or the values are lost. See examples below. 

-> What is the best way to create a dataset from a list of datapoints?

---
e.g.:
**When running this code:**
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```

**We get the following**:
---
1. `datapoint`: (the original datapoint)
```
'audio': {'path': 'it_test_0/common_voice_it_23606167.mp3', 'array': array([0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ...,
       2.21619011e-05, 2.72628222e-05, 0.00000000e+00]), 'sampling_rate': 48000}
 ```
Original Dataset Features:
```
>>> commonvoice_data.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None)
```
 - Here we see column "audio", has the proper values (both `path` & and `array`) and has the correct datatype (Audio).

 
 ----
 2. new_data[0]:
```
# Cannot be printed (as it prints the entire array).
```
New Dataset 1 Features:
```
>>> new_data.features
'audio': {'array': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'path': Value(dtype='string', id=None), 'sampling_rate': Value(dtype='int64', id=None)}
```
 - Here we see that the column "audio", has the correct values, but is not the Audio datatype anymore.

---
3. new_data2[0]:
```
'audio': {'path': None, 'array': array([0., 0., 0., ..., 0., 0., 0.]), 'sampling_rate': 48000},
```
New Dataset 2 Features:
```
>>> new_data2.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None),
```
 - Here we see that the column "audio", has the correct datatype, but all the array & path values were lost!



### Steps to reproduce the bug

## Run:
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```





### Expected behavior

## Expected:
```datapoint == new_data[0]```

AND

```datapoint == new_data2[0]```

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
Inconsistent output in documentation example: `num_classes` not displayed in `ClassLabel` output

In the documentation for [ClassLabel](https://huggingface.co/docs/datasets/v2.21.0/en/package_reference/main_classes#datasets.ClassLabel), there is an example of usage with the following code:

````
from datasets import Features
features = Features({'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'])})
features
````

which expects to output (as stated in the documentation):

````
{'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'], id=None)}
````

but it generates the following

````
{'label': ClassLabel(names=['bad', 'ok', 'good'], id=None)}
````

If my understanding is correct, this happens because although num_classes is used during the init of the object, it is afterward ignored:

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/features/features.py#L975

I would like to work on this issue if this is something needed 😄
Updated inconsistent output in documentation examples for `ClassLabel`

fix #7129 

@stevhliu


Updated! 😄 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7293). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq, can you help with this failing test please? 🙏
Faster parquet streaming + filters with predicate pushdown

ParquetFragment.to_batches uses a buffered stream to read parquet data, which makes streaming faster (x2 on my laptop).

I also added the `filters` config parameter to support filtering with predicate pushdown, e.g.

```python
from datasets import load_dataset

filters = [('problem_source', '==', 'math')]
ds = load_dataset("nvidia/OpenMathInstruct-2", streaming=True, filters=filters)
first_example = next(iter(ds["train"]))
print(first_example["problem_source"])
# 'math'
```

cc @allisonwang-db this is a nice plus for usage in spark


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7309). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Audio Features - DO NOT MERGE] PoC for adding an offset+sliced reading to audio file.

This is a proof of concept for #7310 . The idea is to enable the access to others column of the dataset row when loading an audio file into a table. This is to allow sliced reading. As stated in the issue, many people have very long audio files and use start and stop slicing in this audio file.

Right now, this code work as a PoC on my dataset. However, this is **just to illustrate** the idea. Many things are messed up, the first being that the shards have wildly varying sizes. 

Could be of interest to @lhoestq and @sanchit-gandhi ? 

Happy to test better ideas locally.
incompatibily issue when using load_dataset with datasets==3.0.1

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10


The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


Hi! I'm also getting the same issue - have you been able to find a solution to this? 

From what I remember, I stayed at the "downgraded" version of dataset (2.21.0)
Enable the Audio Feature to decode / read with an offset + duration

### Feature request

For most large speech dataset, we do not wish to generate hundreds of millions of small audio samples. Instead, it is quite common to provide larger audio files with frame offset (soundfile start and stop arguments). We should be able to pass these arguments to Audio() (column ID corresponding in the dataset row).

### Motivation

I am currently generating a fairly big dataset to .parquet(). Unfortunately, it does not work because all existing functions load the whole .wav file corresponding to the row. All my attempts at bypassing this failed. We should be able to put in the Table only the bytes corresponding to what soundfile reads with an offset (and subset of the audio file). 

### Your contribution

I can totally test whatever code on my large dataset creation script. 

Labels: name: enhancement

Hi ! What about having audio + start + duration columns and enable something like this ?

```python
for example in ds:
    array = example["audio"].read(start=example["start"], frames=example["duration"])
```

Hi @lhoestq, this would work with a file-based dataset but would be terrible for a sharded one as it would duplicate the large audio file many times. Also, very long audio files are not embedded very well in the parquet file, even with large_binary(). It crashed a few times for me until I switched to one sample == one file :-(
More docs to from_dict to mention that the result lives in RAM

following discussions at https://discuss.huggingface.co/t/how-to-load-this-simple-audio-data-set-and-use-dataset-map-without-memory-issues/17722/14


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7316). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.