SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: load_dataset can't work correct with my image data 

Comment:
@WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
```
 Fuaimeanna2/
├─ test
│   ├─ label_0
│   │   ├── filename_0.jpg
│   │   └── filename_1.jpg
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
├─ train
│   ├─ label_0
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
└── validation
      ├─ label_0
      │   └── ...
      ├─ label_1
      │   └── ...
      ├─ label_2
      │   └── ...
      └─ label_3
            └── ...
```
Issue: load_dataset can't work correct with my image data 

Comment:
> @WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
> ```
>  Fuaimeanna2/
> ├─ test
> │   ├─ label_0
> │   │   ├── filename_0.jpg
> │   │   └── filename_1.jpg
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> ├─ train
> │   ├─ label_0
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> └── validation
>       ├─ label_0
>       │   └── ...
>       ├─ label_1
>       │   └── ...
>       ├─ label_2
>       │   └── ...
>       └─ label_3
>             └── ...
> ```

I have read this documentation more than once. It just wasn't a problem before.
Issue: load_dataset can't work correct with my image data 

Comment:
Hi,

You need to use:
```
from datasets import load_dataset

dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
```
instead of 
```
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```
To create an image dataset from your local folders.
Issue: load_dataset can't work correct with my image data 

Comment:
> Hi,
> 
> You need to use:
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
> ```
> 
> instead of
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("my_folder_name", split="train")
> ```
> 
> To create an image dataset from your local folders.

Thank you, but even using the method that you wrote above absolutely nothing changes, especially without using data_dir on my other data everything works fine
Issue: load_dataset can't work correct with my image data 

Comment:
@WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 


> even using the method that you wrote above absolutely nothing changes

fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.
Issue: load_dataset can't work correct with my image data 

Comment:
Oh I didn't know that! OK but in any case, not sure why the image builder isn't working for @WiNE-iNEFF. But it's hard for us to help if we can't reproduce. I'd just check the structure of the folders, see if the splits are correctly set up, etc.
Issue: load_dataset can't work correct with my image data 

Comment:
> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I'll try to try your method over the next few days, then I'll write it turned out
Issue: load_dataset can't work correct with my image data 

Comment:
> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I tried creating a `train` folder and put my image folders in it. As a result, all 18,000 images were loaded.
Issue: load_dataset can't work correct with my image data 

Comment:
@WiNE-iNEFF great! So to explain what happened according to my assumptions:

When you use a standard packaged loader (like `imagefolder`, `csv`, `jsonl`, and so on) and load your data like `load_dataset("my_folder_name")` or `load_dataset("imagefolder", data_dir="my_folder_name"`, the library searches for patterns to divide files into splits. This is described a bit in [this doc](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names). And the order to search for patterns is the following:
1. first it checks for [pattern like `data/<split_name>-xxxxx-of-xxxxx`](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#custom-split-names) (which allows to pass custom split names)
2. then for directories named as splits (if you have directories named `train`, `test` etc.)
3. then for [splits in filenames](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names) (like if you have files named `train-image.jpg`, `test_0.jpg`, ...)
4. then if no pattern was found, it treats all files as belonging to a single `train` split

The code is [here](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L215).
So I assume that in your case, since you didn't have directories for splits (pattern 2), some files that included split keywords (pattern 3) were included and others were ignored as not matching the pattern. And when you added `train` directory, the pattern for directories (pattern 2) was triggered first and everything worked as expected. Everything worked in your previous cases probably because you didn't have split names keywords in filenames, so all the files ended up being a part of a single train split (pattern 4).

Another way to mitigate this apart from structuring your data according to the patterns is to explicitly state with files belong to which splits by passing them with `data_files` parameter:
```python
load_dataset("my_folder_name", data_files={"train": "**"})   # to tell that all files should be included  
```

Now I see that this order should be explained in documentation and also referenced in sections for packaged modules like `imagefolder`, thank you for pointing this out.
Issue: load_dataset can't work correct with my image data 

Comment:
@NielsRogge @polinaeterna I have a similar problem when reading my dataset. I want to use DETR for object detection, but my data is in YOLO format. With a dataset of 10k images, yolo format involves having 10k labels. As far as I read regarding [COCO format](https://auto.gluon.ai/stable/tutorials/multimodal/object_detection/data_preparation/convert_data_to_coco_format.html), there must be one JSON per split. However, as I post in the [Hugging Face forum](https://discuss.huggingface.co/t/prepare-dataset-from-yolo-format-to-coco-for-detr/34894), when it is read, the number of rows is 1, which does not make sense. 
The instruction to read the train-val-test splits are: 
```python
from datasets import load_dataset
data_files = {
	"train": './train_labels.json',
	"validation": './val_labels.json',
	"test": './test_labels.json'
}
dataset = load_dataset("json", data_files=data_files)
```
An example of the short version of the json file I read, to reproduce my error, is the following: 

``` json
{
    "info": {},
    "licenses": [],
    "images": [
        {
            "id": 1,
            "file_name": "aceca_100.mp4frame21.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
            },
        {
            "id": 2,
            "file_name": "aceca_100.mp4frame24.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
        },
        {
            "id": 3,
            "file_name": "aceca_100.mp4frame25.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null}],
            "annotations": [
                {
                    "id": 1,
                    "image_id": 1,
                    "category_id": 0,
                    "bbox": [0.0, 278.21896388398557, 86.94096523844935, 156.0293445072134],
                    "area": 13565.341816979679,
                    "iscrowd": 0
                },
                {
                    "id": 2,
                    "image_id": 2,
                    "category_id": 0,
                    "bbox": [149.28851295721816, 297.6359759754418, 34.76802347007475, 98.03908698442889],
                    "area": 3408.625277259324,
                    "iscrowd": 0
                },
                {
                    "id": 3,
                    "image_id": 3,
                    "category_id": 0,
                    "bbox": [153.3817197549372, 300.168969412891, 31.787555842913775, 89.69583163436312],
                    "area": 2851.2112569539095,
                    "iscrowd": 0
                }
            ],
            "categories": [
                {
                    "id": 0, "name": "person"
                }
            ]
        }
```
If full files required, my email is aruigui98@gmail.com
Issue: load_dataset can't work correct with my image data 

Comment:
Hi @Alberto1404, to load an object detection dataset it's recommended to make use of the metadata feature as explained [here](https://huggingface.co/docs/datasets/image_dataset#object-detection).
Issue: load_dataset can't work correct with my image data 

Comment:
You can now refer to https://huggingface.co/docs/datasets/repository_structure to learn about the `datasets`' data files inference, so I'm closing this issue.
Issue: The index column created with .to_sql() is dependent on the batch_size when writing

Comment:
Thanks for reporting, @lsb. 

We are investigating it.

On the other hand, please note that in the next `datasets` release, the index will not be created by default (see #5583). If you would like to have it, you will need to explicitly pass `index=True`.
Issue: The index column created with .to_sql() is dependent on the batch_size when writing

Comment:
I think this is low enough priority for me to close this as Won't Fix. If I need any primary keys I can generate them beforehand. Feel free to reopen.
Issue: flatten_indices doesn't work with pandas format

Comment:
Thanks for reporting! This can be fixed by setting the format to `arrow` in `flatten_indices` and restoring the original format after the flattening. I'm working on a PR that reduces the number of the `flatten_indices` calls in our codebase and makes `flatten_indices` a no-op when a dataset does not have an indices mapping, so I'll incorporate the fix in that PR.
Issue: Make all print statements optional

Comment:
We now log these messages instead of printing them (addressed in #6019), so I'm closing this issue.
Issue: Datasets map and select(range()) is giving dill error

Comment:
It looks like an error that we observed once in https://github.com/huggingface/datasets/pull/5166

Can you try to update `datasets` ?

```
pip install -U datasets
```

if it doesn't work, can you make sure you don't have packages installed that may modify `dill`'s behavior, such as `apache-beam` ?
Issue: xPath to implement all operations for Path

Comment:
 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).
Issue: xPath to implement all operations for Path

Comment:
Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?
Issue: xPath to implement all operations for Path

Comment:
`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.
Issue: xPath to implement all operations for Path

Comment:
Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.
Issue: xPath to implement all operations for Path

Comment:
It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)
Issue: IterableDataset with_format does not support 'device' keyword for jax

Comment:
Hi! Yes, only `torch` is currently supported. Unlike `Dataset`, `IterableDataset` is not PyArrow-backed, so we cannot simply call `to_numpy` on the underlying subtables to format them numerically. Instead, we must manually convert examples to (numeric) arrays while preserving consistency with `Dataset`, which is not trivial, so this is still a to-do.
Issue: IterableDataset with_format does not support 'device' keyword for jax

Comment:
Any plans to support it in the future? Or would streaming dataset be left without support for jax and tensorflow?
Issue: Not all progress bars are showing up when they should for downloading dataset

Comment:
Hi! 

By default, tqdm has `leave=True` to "keep all traces of the progress bar upon the termination of iteration". However, we use `leave=False` in some places (as of recently), which removes the bar once the iteration is over.

I feel like our TQDM bars are noisy, so I think we should always set `leave=False` and also use the `delay` parameter to display progress bars only for tasks that take time (e.g., more than 3s). What do you think about this? Do you find these bars useful (after the dataset generation is over)?
Issue: Cannot import datasets

Comment:
Okay, the issue was likely caused by mixing `conda` and `pip` usage - I forgot that I have already used `pip` in this environment previously and that it was 'spoiled' because of it. Creating another environment and installing `datasets` by pip with other packages from the `requirements.txt` file solved the problem.
Issue: Dataset cannot convert too large dictionnary

Comment:
Answered on the forum:

> To fix the overflow error, we need to merge [support LargeListArray in pyarrow by xwwwwww · Pull Request #4800 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4800), which adds support for the large lists. However, before merging it, we need to come up with a cleaner API for large lists. I hope to find some time to address this before Datasets 3.0.
Issue: Custom split names

Comment:
Hi!

You can also use names other than "train", "validation"  and "test". As an example, check the [script](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0/blob/e095840f23f3dffc1056c078c2f9320dad9ca74d/common_voice_11_0.py#L139) of the Common Voice 11 dataset.
Issue: load_dataset gives "403" error when using Financial phrasebank

Comment:
Hi! You seem to be using an outdated version of `datasets` that downloads the older script version. To avoid the error, you can either pass `revision="main"` to `load_dataset` (this can fail if a script uses newer features of the lib) or update your installation with `pip install -U datasets` (better solution).
Issue: Unable to load AutoTrain-generated dataset from the hub

Comment:
The AutoTrain format is not supported right now. I think it would require a dedicated dataset builder
Issue: Unable to load AutoTrain-generated dataset from the hub

Comment:
Okay, good to know. Thanks for the reply. For now I will just have to
manage the split manually before training, because I can’t find any way of
pulling out file indices or file names from the autogenerated split. The
file names field of the image dataset (loaded directly from arrow file) is
missing, just fyi (for anyone else this might be relevant too).

On Fri, Mar 10, 2023 at 7:02 PM Quentin Lhoest ***@***.***>
wrote:

> The AutoTrain format is not supported right now. I think it would require
> a dedicated dataset builder
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5627#issuecomment-1464734308>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACBJ4F5A353MCZ76OGRJ6CTW3PFI7ANCNFSM6AAAAAAVWXNUTE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Issue: Allow "jsonl" data type signifier

Comment:
You can use "json" instead. It doesn't work by extension names, but rather by dataset builder names, e.g. "text", "imagefolder", etc. I don't think the example in `transformers` is correct because of that
Issue: Allow "jsonl" data type signifier

Comment:
Yes, I understand the reasoning but this issue is to propose that the example in transformers (while incorrect) "makes sense" in terms of user expectation. So the question is whether it would be possible to add "aliases" for common types (like "json" and "text") based on common extensions (like jsonl and txt)?
Issue: glue datasets returning -1 for test split

Comment:
Hi @lithafnium, thanks for reporting.

Please note that you can use the "Community" tab in the corresponding dataset page to start any discussion: https://huggingface.co/datasets/glue/discussions

Indeed this issue was already raised there (https://huggingface.co/datasets/glue/discussions/5) and answered: https://huggingface.co/datasets/glue/discussions/5#63907885937867f0cb3cde31
> The test labels are not public.
>
> Note this dataset belongs to a benchmark: people send their predictions for the test split to GLUE (https://gluebenchmark.com/) and then they get a score in their leaderboard...
Issue: Version mismatch with multiprocess and dill on Python 3.10

Comment:
Reopening, since I think the docs should inform the user of this problem. For example, [this page](https://huggingface.co/docs/datasets/installation) says 
> Datasets is tested on Python 3.7+.

but it should probably say that Beam Datasets do not work with Python 3.10 (or link to a known issues page).
Issue: Version mismatch with multiprocess and dill on Python 3.10

Comment:
Same problem on Colab using a vanilla setup running :
Python 3.10.11 
apache-beam 2.47.0
datasets 2.12.0
Issue: Arrow map type in parquet files unsupported

Comment:
I'm attaching a minimal reproducible example:
```python
from datasets import load_dataset
import pyarrow as pa
import pyarrow.parquet as pq

table_with_map = pa.Table.from_pydict(
    {"a": [1, 2], "b": [[("a", 2)], [("b", 4)]]},
    schema=pa.schema({"a": pa.int32(), "b": pa.map_(pa.string(), pa.int32())})
)
pq.write_table(table_with_map, "parquet_with_map.parquet")
dset = load_dataset("parquet", data_files="parquet_with_map.parquet", split="train") # error unless streaming=True
``` 

For a dataset generated with the packaged loaders (CSV, JSON, Parquet), `streaming=True` sets the dataset's features to `None` (unless explicitly provided in `load_dataset`), hence no error will be thrown as long as the features stay "unresolved" (resolving the features with `_resolve_features` will lead to an error).
Issue: use datasets streaming mode in trainer ddp mode cause memory leak

Comment:
Same problem, 
transformers 4.28.1
datasets 2.12.0

leak around 100Mb per 10 seconds when use dataloader_num_werker > 0 in training argumennts for transformer train, possile bug in transformers repo, but still not found solution :(
Issue: use datasets streaming mode in trainer ddp mode cause memory leak

Comment:
found an article described a problem, may be helpful for somebody:
https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/
I confirm, it`s not memory leak, after some time memory growing has stopped
Issue: `load_from_disk` vs `load_dataset` performance.

Comment:
Hi! We've recently made some improvements to `save_to_disk`/`list_to_disk` (100x faster in some scenarios), so it would help if you could install `datasets` directly from `main` (`pip install git+https://github.com/huggingface/datasets.git`) and re-run the "benchmark".
Issue: `load_from_disk` vs `load_dataset` performance.

Comment:
@mariosasko is that fix released to pip in the meantime? Asking cause im facing still the same issue (regarding loading images from local paths):
```
dataset = load_dataset("csv", cache_dir="cache", data_files=["/STORAGE/DATA/mijam/vit/code/list_filtered.csv"], num_proc=16, split="train").cast_column("image", Image())
dataset = dataset.class_encode_column("label")
```
quite fast. 

Then I do `save_to_disk()` and some time later:
```
dataset = load_from_disk('/STORAGE/DATA/mijam/accel/saved_arrow_big')
```
really slow. In theory it should be quicked since it only loads arrow files, no conversions and so on.
Issue: `load_from_disk` vs `load_dataset` performance.

Comment:
@mjamroz I assume your CSV file stores image file paths. This means `save_to_disk` needs to embed the image bytes resulting in a much bigger Arrow file (than the initial one). Maybe specifying `num_shards` to make the Arrow files smaller can help (large Arrow files on some systems take a long time to load).
Issue: audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files. 

Comment:
Hi!

> naming convention of mp3 files

Yes, this could be the problem. MP3 files should end with  `.mp3`/`.MP3` to be recognized as audio files.

If the file names are not the culprit, can you paste the audio folder's directory structure to help us reproduce the error (e.g., by running the `tree "x"` command)?