noamaanMulla-03/datasets-issues huggingface dataset:
Overwriting Split overwrites data but not metadata, corrupting dataset

### Describe the bug

I want to be able to overwrite/update/delete splits in my dataset. Currently the only way to do is to manually go into the dataset and delete the split. If I try to overwrite programmatically I end up in an error state and (somewhat) corrupting the dataset. Read below.

**Current Behavior**
When I push to an existing split I get this error:
`ValueError: Split complexRoofLocation_01Apr2023_to_31May2023test already present` 
This seems to suggest that the library doesn't support overwriting splits. 

**Potential Bug**
What’s strange is that datasets, despite the operation erroring out with the ValueError above, does, in fact, overwrite the split:

`Pushing dataset shards to the dataset hub: 100% [.....................] 1/1 [00:00<00:00, 55.04it/s]` 
Even though you got an error message and your code fails, your dataset is now changed. That seems like a bug. Either don't change the dataset, or don't throw the error and allow the script to proceed. 

Additional Bug
While it overwrites the split, it doesn’t overwrite the split’s information. Because of this when you pull down the dataset you may end up getting a `NonMatchingSplitsSizesError` if the size of the dataset during the overwrite is different. For example, my original split had 5 rows, but on my overwrite, I only had 4. Then when I try to download the dataset, I get a `NonMatchingSplitsSizesError` because the dataset's data.json states there’s 5 but only 4 exist in the split.

Expected Behavior
This corrupts the dataset rendering it unusable (until you take manual intervention). Either the library should let the overwrite happen (which it does but should also update the metadata) or it shouldn’t do anything.

### Steps to reproduce the bug

[Colab Notebook](https://colab.research.google.com/drive/1bqVkD06Ngs9MQNdSk_ygCG6y1UqXA4pC?usp=sharing)


### Expected behavior

The split should be overwritten and I should be able to use the new version of the dataset without issue. 

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
Parquet uploads off-by-one naming scheme

### Describe the bug

I noticed this numbering scheme not matching up in a different project and wanted to raise it as an issue for discussion, what is the actual proper way to have these stored?

<img width="425" alt="image" src="https://github.com/huggingface/datasets/assets/1981179/3ffa2144-7c9a-446f-b521-a5e9db71e7ce">

The `-SSSSS-of-NNNNN` seems to be used widely across the codebase. The section that creates the part in my screenshot is here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5287
There are also some edits to this section in the single commit branch.

### Steps to reproduce the bug

1. Upload a dataset that requires at least two parquet files in it
2. Observe the naming scheme

### Expected behavior

The couple options here are of course **1. keeping it as is**

**2. Starting the index at 1:**
train-00001-of-00002-{hash}.parquet
train-00002-of-00002-{hash}.parquet

**3. My preferred option** (which would solve my specific issue), dropping the total entirely:
train-00000-{hash}.parquet
train-00001-{hash}.parquet

This also solves an issue that will occur with an `append` variable for `push_to_hub` (see https://github.com/huggingface/datasets/issues/6290) where as you add a new parquet file, you need to rename everything in the repo as well. 

However, I know there are parts of the repo that use 0 as the starting file or may require the total, so raising the question for discussion.


### Environment info

- `datasets` version: 2.14.6.dev0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.18.0
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


You can find the reasoning behind this naming scheme [here](https://github.com/huggingface/transformers/pull/16343#discussion_r931182168).

This point has been raised several times, so I'd be okay with starting with `00001-` (also to be consistent with the `transformers` sharding), but I'm not sure @lhoestq agrees.

We start at 0 in `datasets` for consistency with Apache Spark, Apache Beam, Dask and others.

Also note `transformers` isn't a good reference on this topic. I talked with the maintainers when they added shards but it was already released this way. Though we found that there is a backward-compatible way in `transformers` to start at 0, but no request from `transformers` users to changes this AFAIK.

not sure it would be a good idea to break the consistency now, IMO

Makes sense to start at 0 for plenty of good reasons so I'm on board.

What about the second part `-of-0000X`? With single commit PR #6269  just getting merged, there was a note about issues with 100+ file edits https://github.com/huggingface/datasets/pull/6269#issuecomment-1755428581.

That would be my last remaining concern in the context of the `push_to_hub(..., append=True)` work to be done, where appending a single file to the full dataset will require renaming every other existing file in the dataset. If it doesn't seem like a big issue for this work then all the better 👍
ArrowWriter/ParquetWriter `write` method does not increase `_num_bytes` and hence datasets not sharding at `max_shard_size`

### Describe the bug

An example from [1], does not work when limiting shards with `max_shard_size`.

Try the following example with low `max_shard_size`, such as:

```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

The reason for this is that, in line [2] `writer._num_bytes > max_shard_size` is never true, because the `write` method of `ArrowWriter` [3] does not increase `self._num_bytes`.

Such that respective Arrow/Parquet shards are only written to file based on the `writer_batch_size` or `config.DEFAULT_MAX_BATCH_SIZE`, but not based on `max_shard_size`.


[1] https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

[2] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/builder.py#L1677

[3] https://github.com/huggingface/datasets/blob/3e8d420808718c9a1453a2e7ee3484ca12c9c70d/src/datasets/arrow_writer.py#L459

### Steps to reproduce the bug

Get example from: https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage

Call `builder.download_and_prepare` with low `max_shard_size` such as `10MB`, e.g.:
```python
builder.download_and_prepare(output_dir, storage_options=storage_options, file_format="parquet", max_shard_size="10MB")
```

### Expected behavior

Shards should be written based on `max_shard_size` instead of batch size.

### Environment info

```
>>> import datasets
>>> datasets.__version__
'2.14.6.dev0
```


`writer._num_bytes` is updated every `writer_batch_size`-th call to the `write` method (default `writer_batch_size` is 1000 (examples)). You should be able to see the update by passing a smaller `writer_batch_size` to the `load_dataset_builder`.

We could improve this by supporting the string `writer_batch_size` version as we do with `max_shard_size`, and capping `writer_batch_size` to `max_shard_size` in scenarios where the default `writer_batch_size` > `max_shard_size`. 

Thanks, reducing `writer_batch_size` solved my problem :)
sentiment140 dataset unavailable

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3    
- PyArrow version: 13.0.0
- Pandas version: 2.1.1


Thanks for reporting. We are investigating the issue.

We have opened an issue in the corresponding Hub dataset: https://huggingface.co/datasets/sentiment140/discussions/3

Let's continue the discussion there.
Hub datasets with CSV metadata raise ArrowInvalid: JSON parse error: Invalid value. in row 0

When trying to load a Hub dataset that contains a CSV metadata file, it raises an `ArrowInvalid` error:
```
E   pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0

pyarrow/error.pxi:100: ArrowInvalid
```

See: https://huggingface.co/datasets/lukarape/public_small_papers/discussions/1

Labels: name: bug
support squad structure datasets using a YAML parameter

### Feature request

Since the squad structure is widely used, I think it could be beneficial to support it using a YAML parameter.
could you implement automatic data loading of squad-like data using squad JSON format, to read it from JSON files and view it in the correct squad structure.
The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers

### Motivation

Dataset repo requires arbitrary Python code execution

### Your contribution

The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers
train and dev sets in squad structure JSON files

Labels: name: enhancement
FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`


You can clone the `togethercomputer/RedPajama-Data-1T-Sample` repo and load the dataset with `load_dataset("path/to/cloned_repo")` to use it offline.

@mariosasko Thank you for your kind reply! I'll try it as a workaround.
Does that mean that currently it's not supported to simply load with a short name?

It is, but manually downloading repo files to the cache can easily lead to failure (the HF cache is not meant to be modified by a user besides deleting the files 🙂), as in your case. Hence, the clone + `load_dataset("path/to/cloned_repo")` workflow should be used instead.
Conversion to Arrow fails due to wrong type heuristic

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values. 

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.


### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1


Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix. 

JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon. 

Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

> Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix.
> 
> JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon.
> 
> Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

Ok many thanks, I was able to mitigate the problem by manually checking and converting all problematic fields now.
Incorrect example code in 'Create a dataset' docs

### Describe the bug

On [this](https://huggingface.co/docs/datasets/create_dataset) page, the example code for loading in images and audio is incorrect.

Currently, examples are:

``` python
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")
```
and 
``` python
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I'm pretty sure the imports are wrong and should be:

``` python 
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I am happy to update this if this is right but just wanted to check before making any changes.

### Steps to reproduce the bug

Go to https://huggingface.co/docs/datasets/create_dataset

### Expected behavior

N/A

### Environment info

N/A


This was fixed in https://github.com/huggingface/datasets/pull/6247. You can find the fix in the `main` version of the docs

Ah great, thanks :)
module 'resource' has no attribute 'error'

### Describe the bug

just run import:
`from datasets import load_dataset`

and then:
```

  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_dataset.py", line 66, in <module>
    from .arrow_reader import ArrowReader
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\__init__.py", line 10, in <module>
    from .streaming_download_manager import StreamingDownloadManager
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\streaming_download_manager.py", line 21, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\filesystems\__init__.py", line 8, in <module>
    import fsspec.asyn
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\fsspec\asyn.py", line 157, in <module>
    ResourceEror = resource.error
AttributeError: module 'resource' has no attribute 'error'

Process finished with exit code 1

```


and the error codes are:
```
try:
    import resource
except ImportError:
    resource = None
    ResourceError = OSError
else:
    ResourceEror = resource.error

```
1. miss spelling : "ResourceEror " should be "ResourceErorr"
2. module 'resource' has no attribute 'error'




### Steps to reproduce the bug

only one step:

`from datasets import load_dataset`

### Expected behavior

slove error: module 'resource' has no attribute 'error'

### Environment info

python=3.10

datasets==2.14.5



This (Windows) issue was fixed in `fsspec` in https://github.com/fsspec/filesystem_spec/pull/1275.  So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

> This (Windows) issue was fixed in `fsspec` in [fsspec/filesystem_spec#1275](https://github.com/fsspec/filesystem_spec/pull/1275). So, to avoid the error, update the `fsspec` installation with `pip install -U fsspec`.

after I run `pip install -U fsspec`

it occurs a new error:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict
s.
datasets 2.14.5 requires fsspec[http]<2023.9.0,>=2023.1.0, but you have fsspec 2023.9.2 which is incompatible.

```

The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

> The `fsspec<2023.9.0` upper bound will be removed in the next release. The `ResourceError` fix is also present in version 2023.6.0, so use that version in the meantime (`pip install fsspec==2023.6.0`).

thanks for reply！
Ecoset

## Adding a Dataset
- **Name:** *Ecoset*
- **Description:** *https://www.kietzmannlab.org/ecoset/*
- **Paper:** *https://doi.org/10.1073/pnas.2011417118*
- **Data:** *https://codeocean.com/capsule/9570390/tree/v1*
- **Motivation:**

**Ecoset** was created as a clean and ecologically valid alternative to **Imagenet**.

It is a large image recognition dataset, similar to Imagenet in size and structure. However, the authors of ecoset claim several improvements over Imagenet, like:
- more ecologically valid classes (e.g. not over-focussed on distinguishing different dog breeds)
- less NSFW content
- 'pre-packed image recognition models' that come with the dataset and can be used for validation of other models.


I am working for one of the authors of the paper with the aim of bringing Ecoset to huggingface datasets. Therefore I can work on this issue personally, but could use some help from devs and experienced users if the dataset is of interest to them. I phrased some of my questions on [discuss.huggingface](https://discuss.huggingface.co/t/handling-large-image-datasets/19373).


Labels: name: dataset request

Hi! Very cool dataset! I answered your questions on the forum. Also, feel free to comment `#self-assign` on this issue to self-assign it.

The dataset lives on the Hub [here](https://huggingface.co/datasets/kietzmannlab/ecoset), so I'm closing this issue.

Hey There, thanks for closing 🤗 

Forgot the issue existed, so I didn't close it after implementing the downloader :)
Different objects are returned from calls that should be returning the same kind of object.

### Describe the bug

        1.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
        2.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.


`load_dataset` returns a `DatasetDict` object unless `split`  is defined, in which case it returns a `Dataset` (or a list of datasets if `split` is a list). We've discussed dropping `DatasetDict` from the API in https://github.com/huggingface/datasets/issues/5189 to always return the same type in `load_dataset` and support datasets without (explicit) splits. IIRC the main discussion point is deciding what to return when loading a dataset with multiple splits, but `split` is not specified. What would you expect as a return value in that scenario?

> `load_dataset` returns a `DatasetDict` object unless `split` is defined, in which case it returns a `Dataset` (or a list of datasets if `split` is a list). We've discussed dropping `DatasetDict` from the API in #5189 to always return the same type in `load_dataset` and support datasets without (explicit) splits. IIRC the main discussion point is deciding what to return when loading a dataset with multiple splits, but `split` is not specified. What would you expect as a return value in that scenario?

Wouldn't a dataset with multiple splits already have keys and their related data arrays?

Lets say the dataset has "train" : trainset, "valid": validset and "test": testset

So a dictionary can be returned,, i.e.

{ 
"train": trainset,
"valid": validset,
"test": testset
}

if a split is provided split=['train[:80%]', 'valid[80%:90%]', 'test[90%:100%]']

would also return the same dictionary as above.

split='train[:10%]' should return the same value as split=['train[:10%]']

{
"train": trainset
}
expanduser in save_to_disk

### Describe the bug

save_to_disk() does not expand `~`

1. `dataset = load_datasets("any dataset")`
2. `dataset.save_to_disk("~/data")`
3. a folder named "~" created in current folder
4. FileNotFoundError is raised, because the expanded path does not exist (`/home/<user>/data`)

related issue https://github.com/huggingface/transformers/issues/10628

### Steps to reproduce the bug

As described above.

### Expected behavior

expanduser correctly

### Environment info

- datasets 2.10.1
- python 3.10

Labels: name: good first issue

`save_to_disk` should indeed expand `~`. Marking it as a "good first issue".

#self-assign

File path to code: 

https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_dataset.py#L1364

@RmZeta2718 I created a pull request for this issue. 

Hello, 
It says `save_to_disk` is deprecated in 2.8.0, so the alternative to this will be `storage_options`? 

https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.save_to_disk

@ashikshafi08 I think you misunderstood the warning. The method `save_to_disk` is not deprecated only the optional parameter `fs`.
Also @benjaminbrown038 as I cannot find your PR I would like to work on this if you don't mind.

@mariosasko It's been several months and the PR is not reviewed. Could you please take a look? I assume this is not complicated and could be merged fairly soon.