noamaanMulla-03/datasets-issues huggingface dataset:
is it possible to make slice to be more compatible like python list and numpy?

Hi,
see below error:
```
AssertionError: Requested slice [:10000000000000000] incompatible with 20 examples.
```


Hi ! Thanks for reporting.
I am working on changes in the way data are sliced from arrow. I can probably fix your issue with the changes I'm doing.
If you have some code to reproduce the issue it would be nice so I can make sure that this case will be supported :)
I'll make a PR in a few days 

Good if you can take care at your side.
Here is the [colab notebook](https://colab.research.google.com/drive/19c-abm87RTRYgW9G1D8ktfwRW95zDYBZ?usp=sharing)
cnn_dailymail repeats itself

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13


Hi ! We recently updated `cnn_dailymail` and now `datasets>=2.14` is needed to load it.

You can update `datasets` with

```
pip install -U datasets
```
ELI5 no longer available, but referenced in example code

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.


Addressed in https://github.com/huggingface/transformers/pull/28715.
Impossible to only download a test split

I've spent a significant amount of time trying to locate the split object inside my _split_generators() custom function.
Then after diving [in the code](https://github.com/huggingface/datasets/blob/5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54/src/datasets/load.py#L2558) I realized that `download_and_prepare` is executed before! split is passed to the dataset builder in `as_dataset`.

If I'm not missing something, this seems like bad design, for the following use case:

> Imagine there is a huge dataset that has an evaluation test set and you want to just download and run just to compare your method.

Is there a current workaround that can help me achieve the same result?

Thank you,


The only way right now is to load with streaming=True

This feature has been proposed for a long time. I'm looking forward to the implementation. On clusters `streaming=True` is not an option since we do not have Internet on compute nodes. See: https://github.com/huggingface/datasets/discussions/1896#discussioncomment-2359593
After `2.16.0` version, there are `PermissionError` when users use shared cache_dir

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)



We'll do a new release of `datasets` in the coming days with a fix !

@lhoestq Thank you very much!
Sign Language Support

### Feature request

Currently, there are only several Sign Language labels, I would like to propose adding all the Signed Languages as new labels which are described in this ISO standard: https://www.evertype.com/standards/iso639/sign-language.html

### Motivation

Datasets currently only have labels for several signed languages. There are more signed languages in the world. Furthermore, some signed languages that have a lot of online data cannot be found because of this reason (for instance, German Sign Language, and there is no German Sign Language label on huggingface datasets even though there are a lot of readily available sign language datasets exist for German Sign Language, which are used very frequently in Sign Language Processing papers, and models.)

### Your contribution

I can submit a PR for this as well, adding the ISO codes and languages to the labels in datasets.

Labels: name: enhancement
wiki_dpr.py error (ID mismatch between lines {id} and vector {vec_id}

### Describe the bug

I'm trying to run a rag example, and the dataset is wiki_dpr.
wiki_dpr download and extracting have been completed successfully.
However, at the generating train split stage, an error from wiki_dpr.py keeps popping up.

Especially in "_generate_examples" :
1. The following error occurs in the line **id, text, title = line.strip().split("\t")**
ValueError: not enough values ​​to unpack (expected 3, got 2)
-> This part handles exceptions so that even if an error occurs, it passes.

2. **ID mismatch between lines {id} and vector {vec_id}**
This error seems to occur at the line " assert int(id) == int(vec_id),".
After I handled the exception in the split error, generating train split progressed to 80%, but an id mismatch error occurred at about the 16200000th vector id.

Debugging is even more difficult because it takes a long time to download and split wiki_dpr. I need help. thank you in advance!!

### Steps to reproduce the bug

Occurs in the generating train split step when running the rag example in the transformers repository.
Specifically, it is an error in wiki_dpr.py.

### Expected behavior

.

### Environment info

python 3.8


Thanks for reporting, @kiehls90.

As this seems an issue with the specific "wiki_dpr" dataset, I am transferring the issue to the corresponding dataset page: https://huggingface.co/datasets/wiki_dpr/discussions/13
Differently dataset object saved than it is loaded.

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(


### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1


I see now, that I have to use `load_from_disk`, in order to load dataset properly, not `load_dataset`. Why is this behavior split? Why do we need both, `load_dataset` and `load_from_disk`?

Unless answered, I believe this might be helpful for other hf datasets newbies.

Anyway, made a `load_dataset` compatible dataset in a following way. I created a directory, and just copied jsonl there as `train.jsonl/test.jsonl`.
```python
output_folder = os.path.join(args.output_folder, f"{task_meta_type}_{task_type}")
os.makedirs(output_folder, exist_ok=True)
file = f"{task_meta_type}_{task_type}_train.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "train.jsonl"))
# now test
file = f"{task_meta_type}_{task_type}_test.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "test.jsonl"))
```


Hi @MFajcik, 

You can find information about save_to_disk/load_from_disk in our docs:
- https://huggingface.co/docs/datasets/v2.16.1/en/process#save
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.save_to_disk
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.load_from_disk
Wrong path for cache directory in offline mode

### Describe the bug

Dear huggingfacers,

I'm trying to use a subset of the-stack dataset. When I run the command the first time

```
dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )
```

It downloads the files and caches them normally.

Nevertheless, since my compute nodes are not online (`HF_DATASETS_OFFLINE=1`) . Whenever I try to run the command again, the library is passing the wrong cache path:


`Cache directory for the-stack doesn't exist at /Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data%2Ffortran-data_dir=data%2Ffortran`

when the right path is:

`'/Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data\%2Ffortran`

Not sure why those redundancies are included in the path.  If I try adding the correct path through the the cache_dir argument it throws an error:

ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'bigcode/the-stack': Offline mode is enabled.


Your help with this issue is greatly appreciated. Thanks a lot for the great work.


### Steps to reproduce the bug

1: 

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


2: 

`HF_DATASETS_OFFLINE=1`

3:

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


### Expected behavior

being able to use the cached data

### Environment info

several different systems


+1

same error in 2.16.1

@kongjiellx any luck with the issue?

I opened https://github.com/huggingface/datasets/pull/6632 to fix this issue. Once it's merged we'll do a new release of `datasets`

Thanks @lhoestq !
Add support for `Sequence(Audio/Image)` feature in `push_to_hub`

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature

Labels: name: enhancement

This issue stems from https://github.com/huggingface/datasets/blob/6d2f2a5e0fea3827eccfd1717d8021c15fc4292a/src/datasets/table.py#L2203-L2205

I'll address it as part of https://github.com/huggingface/datasets/pull/6283.

In the meantime, this should work

```python
import pyarrow as pa
from datasets import Image

dataset = dataset.with_format("arrow")

def embed_images(pa_table):
    images_arr = pa.chunked_array(
        [
            pa.ListArray.from_arrays(chunk.offsets, Image().embed_storage(chunk.values), mask=chunk.is_null())
            for chunk in pa_table["images"].chunks
        ]
    )
    return pa_table.set_column(pa_table.schema.get_field_index("images"), "images", images_arr)

dataset = dataset.map(embed_images, batched=True)

dataset = dataset.with_format("python")

dataset.push_to_hub(...)
```
Support fsspec 2023.10.0

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:
- #6331

Related to issue:
- #6330

As @ZachNagengast suggested, the issue might be related to:
- https://github.com/fsspec/filesystem_spec/pull/1381


Hi @albertvillanova @lhoestq 

I believe the pull request that pins the fsspec version (https://github.com/huggingface/datasets/pull/6331) was merged by mistake. Another fix for the issue was merged on the same day an hour apart. See https://github.com/huggingface/datasets/pull/6334

I'm now having an issue in my project where I can't use newer versions of fsspec.

Can we remove the pin?

Have a nice day! :)

Hi @tomscholz,

Thanks for pointing this out. I think you are right.

I am doing some cross-checks and fixing it. 

Hi again, @tomscholz.

After a more cautious investigation, I think the pin is OK because there are other reasons for it. Chronologically:
- #6331 
- #6334
- #6336 
- #6337 

The reason is that after version 2023.10.0, they changed again the behavior of their `glob` function. See: https://github.com/huggingface/datasets/pull/6337#issuecomment-1774930135
We are working on our side to support both previous and new glob behavior.

Note:
- First pin was < 2023.10.0
- Last pin is <= 2023.10.0

Fixed by #6334 and #6336.
multi-GPU map does not work

### Describe the bug

Here is the code for single-GPU processing: https://pastebin.com/bfmEeK2y
Here is the code for multi-GPU processing: https://pastebin.com/gQ7i5AQy

Here is the video showing that the multi-GPU mapping does not work as expected (there are so many things wrong here, it's better to watch the 3-minute video than explain here):

https://youtu.be/RNbdPkSppc4

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

x2 RTX A4000


This should now be fixed by https://github.com/huggingface/datasets/pull/6550 and updated with https://github.com/huggingface/datasets/pull/6646

Feel free to re-open if you're still having issues :)
Update loading.mdx to include "jsonl" file loading.

* A small update to the documentation, noting the ability to load jsonl files. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6647). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for adding the explicit loading command.
> 
> However, I would move it just below, where we present the JSON-Lines example.
> 
> * Maybe adding that this format is called JSON-Lines
> * Add the example after the JSON-Lines data example
> 
> https://github.com/huggingface/datasets/blob/14d9afbb7ae1b787c450261ca0ff374551993031/docs/source/loading.mdx#L135-L138

Thank you @albertvillanova for the feedback! I moved the jsonl file loading example to a more appropriate location.
Logs are delayed when doing .map when `docker logs`

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...


Hi! `tqdm` doesn't work well in non-interactive environments, so there isn't much we can do about this. It's best to [disable it](https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/utilities#datasets.disable_progress_bars) in such environments and instead use logging to track progress.
Fix imagefolder with one image

A dataset repository with one image and one metadata file was considered a JSON dataset instead of an ImageFolder dataset. This is because we pick the dataset type with the most compatible data file extensions present in the repository and it results in a tie in this case.

e.g. for https://huggingface.co/datasets/multimodalart/repro_1_image

I fixed this by deprioritizing metadata files in the count.

fix https://github.com/huggingface/datasets/issues/6545


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6556). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixed in dataset viewer: https://huggingface.co/datasets/multimodalart/repro_1_image

<img width="682" alt="Capture d’écran 2024-02-12 à 22 57 08" src="https://github.com/huggingface/datasets/assets/1676121/be9a8dbc-2d78-4ffc-aed4-293a7c57bc0d">
Can't align optional features of struct

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.


Friendly bump, I would be happy to work on this issue once I get the go-ahead from the dev team. 

Thanks for the PR!

I'm struggling with this as well and would love to see this PR merged. My case is slightly different, with keys completely missing rather than being `None`:

```
ds = Dataset.from_dict({'speaker': [{'name': 'Ben'}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

I would expect this to work as well because other Dataset functions already handle this situation well. For example, this works just as expected:

```
ds = Dataset.from_dict({'n': [1,2]})
ds_mapped = ds.map(lambda x: {
    'speaker': {'name': 'Ben'} if x['n'] == 1 else {'name': 'Fred', 'email': 'abc@aol.com'}
})
print(ds_mapped)
```

@vova-cyberhaven can you check with the new release if it fixes your issue?
`write_examples_on_file` and `write_batch` are broken in `ArrowWriter`

### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.

> Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.

I feel that'd be good, but it'd be great to release a hotfix ASAP (a revert is a fast thing to do) so people can continue using this library and then focus on still applying the improvement.

Fixed by #6664