SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Allow load_dataset to take a working dir for intermediate data

Comment:
You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk
Issue: Allow load_dataset to take a working dir for intermediate data

Comment:
If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?
Issue: Add support to create a Dataset from spark dataframe

Comment:
if i read spark Dataframe , got an error on  multi-node Spark cluster.
Did the Api (Dataset.from_spark) support Spark cluster, read dataframe and save_to_disk?

Error: 
_pickle.PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforma
tion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
23/06/16 21:17:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
Issue: Filter datasets by language code

Comment:
The dataset still can be found, if instead of using the search form you just enter the language code in the url, like https://huggingface.co/datasets?language=language:myv. 

But of course having a more complete list of languages in the search form (or just a fallback to the language codes, if they are missing from the code=>language mapping) would be much more convenient!
Issue: Filter datasets by language code

Comment:
Thanks @mariosasko!
Do you think it is possible to turn this into a more scalable pipeline? Such as:
1. Looping through all the datasets on the hub and collecting the set of all their language codes;
2. Selecting the codes not covered yet in `Language.ts`
3. Looking up their codes at https://iso639-3.sil.org/code_tables/639/data
4. Adding all the newly found language codes to `Language.ts`
Issue: Filter datasets by language code

Comment:
@avidale This has been discussed in https://github.com/huggingface/datasets/issues/4881, so also feel free to share your opinion there.
Issue: Pushing dataset to hub crash

Comment:
Hi ! It's been fixed by https://github.com/huggingface/datasets/pull/5598. We're doing a new release tomorrow with the fix and you'll be able to push your 100k images ;)

Basically `push_to_hub` used to fail if the remote repository already exists and has a README.md without dataset_info in the YAML tags.

In the meantime you can install datasets from source
Issue: Pushing dataset to hub crash

Comment:
Hi @lhoestq ,

What version of datasets library fix this case? I am using the last `v2.10.1` and I get the same error.
Issue: How to use `load_dataset('glue', 'cola')`

Comment:
Sounds like an issue with incompatible `transformers` dependencies versions.

Can you try to update `transformers` ?

EDIT: I checked the `transformers` dependencies and it seems like you need `tokenizers>=0.10.1,<0.11` with `transformers==4.5.1`

EDIT2: this old version of `datasets` seems to import `transformers` but it's no longer the case, so you could also simply update `datasets` and `transformers` won't be imported
Issue: How to use `load_dataset('glue', 'cola')`

Comment:
Thank you for advising me to update these libraries versions.

I can implement codes using `datasets==2.10.1` and `transformers==4.27.3`
Issue: Almost identical datasets, huge performance difference

Comment:
Hi! 

The first dataset stores images as bytes (the "image" column type is `datasets.Image()`) and decodes them as `PIL.Image` objects and the second dataset stores them as variable-length lists (the "image" column type is `datasets.Sequence(...)`)), so I guess going from `arrow bytes -> NumPy -> decoding as PIL.Image -> PyTorch` is faster than going from `arrow list -> NumPy -> PyTorch`. 

To store image bytes in the second example, you can do the following:

```python
def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"])
    return example

features = dataset.features.copy()
del features["image"]
features["image2"] = datasets.Image()
dataset2 = dataset.map(transform, remove_columns=["image"], features=features)

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass
```
Issue: Almost identical datasets, huge performance difference

Comment:
Thanks, @mariosasko. I could not understand why a (decoded) sequence should be MUCH slower than an encoded image (that must be decoded every time). At any rate, I tried you suggestion. It made the `map` step to run extremely slow (consumes all the 16GB of memory and starts swapping)

I tried also the easiest (as I see it) scenario, where images are kept as bytes, but it made things even worse: not only it was extremely slow, but also crashes

```python

def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"]).tobytes()
    return example

dataset2 = dataset.map(transform, remove_columns=["image"])

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass


Resource temporarily unavailable (src/thread.cpp:269)
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/virtenvs/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1133, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1132 try:
-> 1133     data = self._data_queue.get(timeout=timeout)
   1134     return (True, data)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/queues.py:113, in Queue.get(self, block, timeout)
    112 timeout = deadline - time.monotonic()
--> 113 if not self._poll(timeout):
    114     raise Empty

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:257, in _ConnectionBase.poll(self, timeout)
    256 self._check_readable()
--> 257 return self._poll(timeout)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:424, in Connection._poll(self, timeout)
    423 def _poll(self, timeout):
--> 424     r = wait([self], timeout)
    425     return bool(r)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:931, in wait(object_list, timeout)
    930 while True:
--> 931     ready = selector.select(timeout)
    932     if ready:
...
-> 1146     raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
   1147 if isinstance(e, queue.Empty):
   1148     return (False, None)

RuntimeError: DataLoader worker (pid(s) 195393) exited unexpectedly
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
```
Issue: Almost identical datasets, huge performance difference

Comment:
Correction: the `beans` dataset stores the image file paths, not the bytes.

For your use case, I think it makes  more sense to use `with_tranform` than `map` and lazily decode images with `cv2.imread` when indexing an example/batch:
```python
import cv2

def transform(batch):
    batch["image2"] = np.stack([cv2.imread(image_file_path) for image_file_path in batch["image_file_path"]])
    return batch

dataset = dataset.with_transform(transform)
```
Issue: Almost identical datasets, huge performance difference

Comment:
This is incorrect.

Did you try to run it?  dataset[0] returns a tensor of numbers. dataset2[0]
returns the same tensor, but after a few long seconds.  Looping over a
thousand of images cannot take 15 minutes.

On Fri, 24 Mar 2023 at 19:28 Mario Šaško ***@***.***> wrote:

> Correction: the beans dataset stores the image file paths, not the bytes.
>
> For your use case, I think it makes more sense to use with_tranform than
> map and lazily decode images with cv2.imread when accessing an
> example/batch:
>
> import cv2
> def transform(batch):
>     batch["image2"] = np.stack([cv2.imread(image_file_path) for image_file_path in batch["image_file_path"]])
>     return batch
> dataset = dataset.with_transform(transform)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5669#issuecomment-1483084347>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AASS73SHRWXIQX6SCYCJ7ITW5XDUDANCNFSM6AAAAAAWFSHWEM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Issue: Almost identical datasets, huge performance difference

Comment:
I updated the transform with the NumPy -> PyTorch conversion.

I'm sharing the entire code:
```python
import cv2
import numpy as np
import datasets
import torch
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("beans", split="train")

def transform(batch):
    # # Pillow decodes as RGB
    # batch["image"] = torch.stack([torch.from_numpy(cv2.cvtColor(cv2.imread(image_file_path), cv2.COLOR_BGR2RGB)) for image_file_path in batch["image_file_path"]])
    batch["image"] = torch.stack([torch.from_numpy(cv2.imread(image_file_path)) for image_file_path in batch["image_file_path"]])
    batch["labels"] = torch.tensor(batch["labels"])
    return batch

dataset2 = dataset.cast_column("image", datasets.Image(decode=False)).with_transform(transform)

for x in DataLoader(dataset2, batch_size=16, shuffle=True, num_workers=8):
    pass
```

This code is ≈ 10% faster on my machine than the default decoding with Pillow and `.with_format("torch")`.
Issue: Almost identical datasets, huge performance difference

Comment:
Thanks, @mariosasko 
My question remain unanswered though. Why is the `map`ed dataset  so slow? My understanding is that a dataset of numpy arrays should be must faster than a dataset that has to decode images into numpy arrays every time one accesses an item.
Issue: integration with imbalanced-learn

Comment:
You can convert any dataset to pandas to be used with imbalanced-learn using `.to_pandas()`

Otherwise if you want to keep a `Dataset` object and still use e.g. [make_imbalance](https://imbalanced-learn.org/stable/references/generated/imblearn.datasets.make_imbalance.html#imblearn.datasets.make_imbalance), you just need to pass the list of rows ids and labels:

```python
row_indices = list(range(len(dataset)))
resampled_row_indices, _ = make_imbalance(
    row_indices,
    dataset["label"],
    sampling_strategy={0: 25, 1: 50, 2: 50},
    random_state=RANDOM_STATE,
)

resampled_dataset = dataset.select(resampled_row_indices)
```
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
@sanchit-gandhi can you please also post the logs of `pip install soundfile==0.12.1`? To check what wheel is being installed or if it's being built from source (I think it's the latter case). 
Required `libsndfile` binary **should** be bundeled with `soundfile` wheel but I assume it **might not** be the case for some non standard Linux distributions. 
The only solution for using `soundfile` here is to build [`libsndfile`](https://github.com/libsndfile/libsndfile) from source:

```bash
git clone https://github.com/libsndfile/libsndfile.git
cd libsndfile/
autoreconf -vif
./configure --enable-werror 
make
make install
```
for this, some building libraries should be installed, for Debian/Ubuntu it's like:
```bash
apt install autoconf autogen automake build-essential libasound2-dev \
  libflac-dev libogg-dev libtool libvorbis-dev libopus-dev libmp3lame-dev \
  libmpg123-dev pkg-config python
```
but for other Linux distributions it might be different.

When the binary is compiled, it should be put into location where `soundfile` would search for it (the directory is named `_soundfile_data`), it depends on where`libsdfile` (from the previous step) and  `soundfile` were installed, might be something like this:

```bash
cp /usr/local/lib/libsndfile.so /usr/local/lib/python3.7/dist-packages/_soundfile_data/
cp /usr/local/lib/libsndfile.la /usr/local/lib/python3.7/dist-packages/_soundfile_data/
```

Another solution is to not use `soundfile` and apply custom processing function with `torchaudio` while setting `decode=False` in `Audio` feature and passing custom function to `.map`.
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
Not sure if it may help, but you could also try updating `pip` before installing soundfile
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
@lhoestq @sanchit-gandhi. I encountered the same error (also on the TPU v4) when trying to run `datasets` from source.

Downgrading soundfile with `pip install soundfile==0.12.0` seems to fix the issue for me.
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
Maybe let's open an issue at https://github.com/bastibe/python-soundfile/issues in case they might know why you get `OSError: cannot load library 'libsndfile.so'` ?
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
> @sanchit-gandhi can you please also post the logs of `pip install soundfile==0.12.1`? To check what wheel is being installed or if it's being built from source (I think it's the latter case). Required `libsndfile` binary **should** be bundeled with `soundfile` wheel but I assume it **might not** be the case for some non standard Linux distributions. The only solution for using `soundfile` here is to build [`libsndfile`](https://github.com/libsndfile/libsndfile) from source:
> 
> ```shell
> git clone https://github.com/libsndfile/libsndfile.git
> cd libsndfile/
> autoreconf -vif
> ./configure --enable-werror 
> make
> make install
> ```

This fixed the issue for me. After installing libsndfile as described above, I had to uninstall soundfile and re-install it with this command. `pip install "soundfile>=0.12.1"`
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
Thank you so much for the comprehensive instructions @polinaeterna! Also confirming that they worked for me 🤗 In my case, I had to run several of these commands under "sudo" for privileges, but otherwise this workaround gave a successful `libsndfile` install:

1. Grab source code:
```
git clone https://github.com/libsndfile/libsndfile.git
```

2. Set up a build environment:
```
sudo apt install autoconf autogen automake build-essential libasound2-dev \
  libflac-dev libogg-dev libtool libvorbis-dev libopus-dev libmp3lame-dev \
  libmpg123-dev pkg-config python
```

3. Build and test `libsndfile`:

```
autoreconf -vif
./configure --enable-werror
sudo make
sudo make check
```

4. Create `_soundfile_data` submodule (if it does not exist already):
```
sudo mkdir /usr/local/lib/python3.8/dist-packages/_soundfile_data/
```

5. Copy `libsndfile` files into submodule:
```
sudo cp /usr/local/lib/libsndfile.* /usr/local/lib/python3.8/dist-packages/_soundfile_data/
```
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
On a different machine, I also tried separately by first upgrading pip, then installing soundfile. This worked too! Thanks @lhoestq 🙌
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
> @sanchit-gandhi can you please also post the logs of `pip install soundfile==0.12.1`? To check what wheel is being installed or if it's being built from source (I think it's the latter case). Required `libsndfile` binary **should** be bundeled with `soundfile` wheel but I assume it **might not** be the case for some non standard Linux distributions. The only solution for using `soundfile` here is to build [`libsndfile`](https://github.com/libsndfile/libsndfile) from source:
> 
> ```shell
> git clone https://github.com/libsndfile/libsndfile.git
> cd libsndfile/
> autoreconf -vif
> ./configure --enable-werror 
> make
> make install
> ```
> 
> for this, some building libraries should be installed, for Debian/Ubuntu it's like:
> 
> ```shell
> apt install autoconf autogen automake build-essential libasound2-dev \
>   libflac-dev libogg-dev libtool libvorbis-dev libopus-dev libmp3lame-dev \
>   libmpg123-dev pkg-config python
> ```
> 
> but for other Linux distributions it might be different.
> 
> When the binary is compiled, it should be put into location where `soundfile` would search for it (the directory is named `_soundfile_data`), it depends on where`libsdfile` (from the previous step) and `soundfile` were installed, might be something like this:
> 
> ```shell
> cp /usr/local/lib/libsndfile.so /usr/local/lib/python3.7/dist-packages/_soundfile_data/
> cp /usr/local/lib/libsndfile.la /usr/local/lib/python3.7/dist-packages/_soundfile_data/
> ```
> 
> Another solution is to not use `soundfile` and apply custom processing function with `torchaudio` while setting `decode=False` in `Audio` feature and passing custom function to `.map`.

Thanks, the solution solved my problem. 

1. Purge uninstall libsndfile, uninstall python-soundfile.
2. Build libsndfile from source code and install.
3. Build python-soundfile from source code and install
4. Well done.
Issue: [Audio] Soundfile/libsndfile requirements too stringent for decoding mp3 files

Comment:
> Thank you so much for the comprehensive instructions @polinaeterna! Also confirming that they worked for me 🤗 In my case, I had to run several of these commands under "sudo" for privileges, but otherwise this workaround gave a successful `libsndfile` install:
> 
> 1. Grab source code:
> 
> ```
> git clone https://github.com/libsndfile/libsndfile.git
> ```
> 
> 2. Set up a build environment:
> 
> ```
> sudo apt install autoconf autogen automake build-essential libasound2-dev \
>   libflac-dev libogg-dev libtool libvorbis-dev libopus-dev libmp3lame-dev \
>   libmpg123-dev pkg-config python
> ```
> 
> 3. Build and test `libsndfile`:
> 
> ```
> autoreconf -vif
> ./configure --enable-werror
> sudo make
> sudo make check
> ```
> 
> 4. Create `_soundfile_data` submodule (if it does not exist already):
> 
> ```
> sudo mkdir /usr/local/lib/python3.8/dist-packages/_soundfile_data/
> ```
> 
> 5. Copy `libsndfile` files into submodule:
> 
> ```
> sudo cp /usr/local/lib/libsndfile.* /usr/local/lib/python3.8/dist-packages/_soundfile_data/
> ```

I had to run 'make install' or the `/usr/local/lib/libsndfile.*` files didn't exist.

It's working though!
Issue: Offset overflow when executing Dataset.map

Comment:
Upd. the above code works if we replace `25` with `1`, but the result value at key "hr" is not a tensor but a list of lists of lists of uint8.

Adding `train_data.set_format("torch")` after map fixes this, but the original issue remains
Issue: Offset overflow when executing Dataset.map

Comment:
As a workaround, one can replace
`return {"hr": torch.stack([crop_transf(tensor) for _ in range(25)])}`
with
`return {f"hr_crop_{i}": crop_transf(tensor) for i in range(25)}`
and then choose appropriate crop randomly in further processing, but I still don't understand why the original approach doesn't work(
Issue: expanduser in save_to_disk

Comment:
Hello, 
It says `save_to_disk` is deprecated in 2.8.0, so the alternative to this will be `storage_options`? 

https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.save_to_disk
Issue: expanduser in save_to_disk

Comment:
@ashikshafi08 I think you misunderstood the warning. The method `save_to_disk` is not deprecated only the optional parameter `fs`.
Also @benjaminbrown038 as I cannot find your PR I would like to work on this if you don't mind.
Issue: load_dataset can't work correct with my image data 

Comment:
> Can you post a reproducible code snippet of what you tried to do?
> 
> 

```python
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```
Issue: load_dataset can't work correct with my image data 

Comment:
hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?
Issue: load_dataset can't work correct with my image data 

Comment:
> hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?

All file have format .png converted in RGBA. 
In main folder "MyData" contain 4 folder with images. In function load_dataset i use folder "MyData"
Issue: load_dataset can't work correct with my image data 

Comment:
@WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
What would be the output if you omit `split` while loading? Like just
```python
ds = load_dataset("MyData")
print(ds)
```
Issue: load_dataset can't work correct with my image data 

Comment:
> @WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
> What would be the output if you omit `split` while loading? Like just
> ```python
> ds = load_dataset("MyData")
> print(ds)
> ```
> 
> 

```python
Dataset({
    features: ['image', 'label'],
    num_rows: 4
})
```
Issue: load_dataset can't work correct with my image data 

Comment:
@WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).

If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples.
Issue: load_dataset can't work correct with my image data 

Comment:
> @WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).
> 
> If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples. 

I checked my files and some of them do have the words train, valid and test in their names, but the number of such images is more than 500, not 4.