SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Support `fsspec` in `Dataset.to_<format>` methods

Comment:
I'm assuming this should just cover `to_csv`, `to_parquet`, and `to_json`, right? As `to_list` and `to_dict` just return Python objects, `to_pandas` returns a `pandas.DataFrame` and `to_sql` just inserts into a SQL DB, is that right?
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
When the process starts to hang, can you interrupt it with CTRL + C and paste the error stack trace here?
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
PyTorch tensors cannot store `Decimal` objects. Casting the column with decimals to `float` should fix the issue.
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
I already have cast in collate_fn, in which I perform .astype(float) for each numerical field.
On the same instance, I installed a conda env with python 3.6, and this works well.

Sample:

```
def streaming_data_collate_fn(batch):
    df = pd.DataFrame.from_dict(batch)
    feat_vals = torch.FloatTensor(np.nan_to_num(np.array(df[feats].astype(float))))

```
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
`collate_fn` is applied after the `torch` formatting step, so I think the only option when working with an `IterableDataset` is to remove the `with_format` call and perform the conversion from Python values to PyTorch tensors in `collate_fn`. The standard `Dataset` supports `with_format("numpy")`, which should make this conversion faster.
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
Thanks! 
Python 3.10 conda-env: After replacing with_format("torch") with with_format("numpy"), the error went away. However, it was still taking over 2 minutes to load a very small batch of 64 samples with num_workers set to 32. Once I removed with_format call altogether, it is finishing in 11 seconds.

Python 3.6 based conda-env: When I switch the kernel , neither of the above work, and with_format("torch") is the only thing that works, and executes in 1.6 seconds.

I feel something else is also amiss here.
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
Can you share the `datasets` and `torch` versions installed in these conda envs?

> Once I removed with_format call altogether, it is finishing in 11 seconds.

Hmm, that's surprising. What are your dataset's `.features`?
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
Python 3.6: 
datasets.__version__ 2.4.0
torch.__version__ 1.10.1+cu102

Python 3.10:
datasets.__version__ 2.14.0
torch.__version__ 2.0.0

Anonymized features are of the form (subset shown here):
{
'string_feature_i': Value(dtype='string', id=None),
'numerical_feature_i': Value(dtype='decimal128(38, 0)', id=None),
'numerical_feature_series_i': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None),
}


There is no output from .features in python 3.6 kernel BTW.
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
One more thing, in python 3.10 based kernel, interestingly increasing num_workers seem to be increasing the runtime of iterating I was trying out. In python 3.10 kernel execution, I do not even see multiple CPU cores spiking unlike in 3.6.

512 batch size on 32 workers executes in 2.4 seconds on python 3.6 kernel, while it takes ~118 seconds on 3.10!
Issue: Iterating over DataLoader based on HF datasets is stuck forever

Comment:
**Update**: It seems the latency part is more of a multiprocessing issue with torch and some host specific issue, and I had to scourge through relevant pytorch issues, when I stumbled across these threads:
1. https://github.com/pytorch/pytorch/issues/102494
2. https://github.com/pytorch/pytorch/issues/102269
3. https://github.com/pytorch/pytorch/issues/99625

Out of the suggested solutions, the one that worked in my case was:
```
os.environ['KMP_AFFINITY'] = "disabled"
```
It is working for now, though I have no clue why, just I hope it does not get stuck when I do actual model training, will update by tomorrow.
Issue: resume_download with streaming=True

Comment:
Currently, it's not possible to efficiently resume streaming after an error. Eventually, we plan to support this for Parquet (see https://github.com/huggingface/datasets/issues/5380).
Issue: Mapping gets stuck at 99%

Comment:
The `MAX_MAP_BATCH_SIZE = 1_000_000_000` hack is bad as it loads the entire dataset into RAM when performing `.map`. Instead, it's best to use `.iter(batch_size)` to iterate over the data batches and compute `mean` for each column. (`stddev` can be computed in another pass).

Also, these arrays are big, so it makes sense to reduce `batch_size`/`writer_batch_size` to avoid RAM issues and slow IO.
Issue: Mapping gets stuck at 99%

Comment:
Hi @mariosasko !

I agree, it's an ugly hack, but it was convenient since the resulting `mean_std` could be cached by the library. For my large dataset (which doesn't fit in RAM), I'm actually using something similar to what you suggested. I got rid of the first mapping in the above scripts and replaced it with an iterator, but the issue with the second mapping still persists.
Issue: Mapping gets stuck at 99%

Comment:
Have you tried to reduce `batch_size`/`writer_batch_size` in the 2nd `.map`? Also, can you interrupt the process when it gets stuck and share the error stack trace?
Issue: Error loading music files using `load_dataset`

Comment:
This code behaves as expected on my local machine or in Colab. Which version of `soundfile` do you have installed? MP3 requires `soundfile>=0.12.1`.
Issue: version2.3.2   load_dataset()data_files can't include .xxxx in path

Comment:
Version 2.3.2 is over one year old, so please use the latest release (2.14.0) to get the expected behavior. Version 2.3.2 does not contain some fixes we made to fix resolving hidden files/directories (starting with a dot).
Issue: storage_options provided to load_dataset not fully piping through since datasets 2.14.0

Comment:
Hi ! Thanks for reporting, I opened a PR to fix this

What filesystem are you using ?
Issue: storage_options provided to load_dataset not fully piping through since datasets 2.14.0

Comment:
Hi @lhoestq ! Thank you so much 🙌 

It's a bit of a custom setup, but in practice I am using a [pyarrow.fs.S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) (wrapped in a `fsspec.implementations.arrow.ArrowFSWrapper` [to make it](https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec) `fsspec` compatible). I also register it as an entrypoint with `fsspec` so that it's the one that gets automatically resolved when looking for filesystems for the `s3` protocol

In my case the `storage_option` that seemed not getting piped through was the filesystem's `endpoint_override` that I use in some tests to point at a mock S3 bucket
Issue: KeyError: dataset has no key "image"

Comment:
You can list the dataset's columns with `ds.column_names` before `.map` to check whether the dataset has an `image` column. If it doesn't, then this is a bug. Otherwise, please paste the line with the `.map` call.
Issue: KeyError: dataset has no key "image"

Comment:
The `remove_columns=["image"]` argument in the `.map` call removes the `image` column from the output, so drop this argument to preserve it.
Issue: KeyError: dataset has no key "image"

Comment:
The problem is not with the removal of the image key. The bug is why only the labels are sent to be process, instead of all the featues or dictionary keys.

P.S. I just dropped the removal argument as you've suggested, but that didn't solve the problem, because only the labels are being sent to be processed
Issue: KeyError: dataset has no key "image"

Comment:
All the `image_dataset.column_names` after the `map` call should also be present in `preprocess_train `/`preprocess_val` unless (input) `columns` in `set_transform` are specified.

If that's not the case, we need a full reproducer (not snippets) with the environment info.
Issue: KeyError: dataset has no key "image"

Comment:
I have resolved the error after including a collate function as indicated in the Quick Start session of the Datasets docs.:

Here is what I did:
```
data_transforms = utils.get_data_augmentation(args)
image_dataset = utils.load_image_dataset(args.dataset)

def preprocess_train(example_batch):
    example_batch["pixel_values"] = [
        data_transforms["train"](image.convert("RGB")) for image in example_batch["image"]
    ]
    return example_batch

def preprocess_val(example_batch):
    example_batch["pixel_values"] = [
        data_transforms["val"](image.convert("RGB")) for image in example_batch["image"]
    ]
    return example_batch

def collate_fn(examples):
    images = []
    labels = []
    for example in examples:
        images.append((example["pixel_values"]))
        labels.append(example["label"])

    pixel_values = torch.stack(images)
    labels = torch.tensor(labels)
    return {"pixel_values": pixel_values, "label": labels}

train_dataset = image_dataset["train"].with_transform(preprocess_train)
val_dataset = image_dataset["validation"].with_transform(preprocess_val)

image_datasets = {
    "train": train_dataset,
    "val": val_dataset
}

samplers = {
    "train": data.RandomSampler(train_dataset),
    "val": data.SequentialSampler(val_dataset),
}

dataloaders = {
    x: data.DataLoader(
        image_datasets[x],
        collate_fn=collate_fn,
        batch_size=batch_size,
        sampler=samplers[x],
        num_workers=args.num_workers,
        worker_init_fn=utils.set_seed_for_worker,
        generator=g,
        pin_memory=True,
    )
    for x in ["train", "val"]
}

train_loader, val_loader = dataloaders["train"], dataloaders["val"]
```
Everything runs fine without any bug now.
Issue: AttributeError: '_tqdm_cls' object has no attribute '_lock'

Comment:
Hi ! I opened https://github.com/huggingface/datasets/pull/6067 to add the missing `_lock`

We'll do a patch release soon, but feel free to install `datasets` from source in the meantime
Issue: AttributeError: '_tqdm_cls' object has no attribute '_lock'

Comment:
I have tested the latest main, it does not work.

I add more logs to reproduce this issue, it looks like a multi threading bug:

```python
@contextmanager
def ensure_lock(tqdm_class, lock_name=""):
    """get (create if necessary) and then restore `tqdm_class`'s lock"""
    import os
    import threading
    print(os.getpid(), threading.get_ident(), "ensure_lock", tqdm_class, lock_name)
    old_lock = getattr(tqdm_class, '_lock', None)  # don't create a new lock
    lock = old_lock or tqdm_class.get_lock()  # maybe create a new lock
    lock = getattr(lock, lock_name, lock)  # maybe subtype
    tqdm_class.set_lock(lock)
    print(os.getpid(), threading.get_ident(), "set_lock")
    yield lock
    if old_lock is None:
        print(os.getpid(), threading.get_ident(), "del tqdm_class")
        del tqdm_class._lock
    else:
        tqdm_class.set_lock(old_lock)
```
output
```
64943 8424758784 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 8424758784 set_lock
64943 8424758784 del tqdm_class
64943 8424758784 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 8424758784 set_lock
64943 8424758784 del tqdm_class
64943 11638370304 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11638370304 set_lock
64943 11568967680 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11568967680 set_lock
64943 11638370304 del tqdm_class
64943 11638370304 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11638370304 set_lock
64943 11638370304 del tqdm_class
64943 11568967680 del tqdm_class
```

Thread `11638370304` del the _lock from tqdm_class first, then thread `11568967680` del _lock failed.
Issue: AttributeError: '_tqdm_cls' object has no attribute '_lock'

Comment:
Maybe it is a bug of tqdm? I think simply use `try ... except AttributeError ...` wraps `del tqdm_class._lock` should work.
Issue: AttributeError: '_tqdm_cls' object has no attribute '_lock'

Comment:
Yes it looks like a bug on their end indeed, do you want to open a PR on tqdm ?

Let me see if I can find a workaround in the meantime
Issue: Dataset.map() execute twice when in PyTorch DDP mode

Comment:
Sorry for asking a duplicate question about `num_proc`, I searched the forum and find the solution.

But I still can't make the trick with `torch.distributed.barrier()` to only map at the main process work. The [post on forum]( https://discuss.huggingface.co/t/slow-processing-with-map-when-using-deepspeed-or-fairscale/7229/7) didn't help.
Issue: Dataset.map() execute twice when in PyTorch DDP mode

Comment:
If it does the `map` twice then it means the hash of your map function is not some same between your two processes.

Can you make sure your map functions have the same hash in different processes ?

```python
from datasets.fingerprint import Hasher

print(Hasher.hash(lambda x: cut_reorder_keys(x, num_stations_list=args.num_stations_list, is_pad=True, is_train=True)))
print(Hasher.hash(lambda x: random_shift(x, shift_range=(-160, 0), feature_scale=16)))
```

You can also set the fingerprint used to reload the resulting dataset by passing `new_finegrprint=` in `map`, see https://huggingface.co/docs/datasets/v2.13.1/en/about_cache#the-cache. This will force the different processes to use the same fingerprint used to locate the resulting dataset in the cache.
Issue: Dataset.map() execute twice when in PyTorch DDP mode

Comment:
Thanks for help! I find the fingerprint between processes don't have same hash:
```
Rank 0: Gpu 0 cut_reorder_keys fingerprint c7f47f40e9a67657
Rank 0: Gpu 0 random_shift fingerprint 240a0ce79831e7d4

Rank 1: Gpu 1 cut_reorder_keys fingerprint 20edd3d9cf284001
Rank 1: Gpu 1 random_shift fingerprint 819f7c1c18e7733f
```
But my functions only process the example one by one and don't need rank or other arguments. After all it can work in the test for dataset and dataloader.
I'll try to set `new_fingerprint` to see if it works and figure out the reason of different hash.
Issue: laion-coco download error

Comment:
This can also mean one of the files was not downloaded correctly.

We log an erroneous file's name before raising the reader's error, so this is how you can find the problematic file. Then, you should delete it and call `load_dataset` again.

(I checked all the uploaded files, and they seem to be valid Parquet files, so I don't think this is a bug on their side)
Issue: Why is the speed difference of gen example so big?

Comment:
Hi!

It's hard to explain this behavior without more information. Can you profile the slower version with the following code
```python
import cProfile, pstats
from datasets import load_dataset

with cProfile.Profile() as profiler:
    ds = load_dataset(...)

stats = pstats.Stats(profiler).sort_stats("cumtime")
stats.print_stats()
```
and share the output?
Issue: Skipping shard in the remote repo and resume upload

Comment:
Hi! `_select_contiguous` fetches a (zero-copy) slice of the dataset's Arrow table to build a shard, so I don't think this part is the problem. To me, the issue seems to be the step where we embed external image files' bytes (a lot of file reads). You can use `.map` with multiprocessing to perform this step before `push_to_hub` in a faster manner and cache it to disk:
```python
from datasets.table import embed_table_storage
# load_dataset(...)
format = dataset.format
dataset = dataset.with_format("arrow")
dataset = dataset.map(embed_table_storage, batched=True)
dataset = dataset.with_format(**format)
# push_to_hub(...)
```

(In Datasets 3.0, these external bytes will be written to an Arrow file when generating a dataset to avoid this "embed" step)
Issue: Skipping shard in the remote repo and resume upload

Comment:
Hi, thanks, this solution saves some time.
But can't we avoid embedding all external image files bytes with each push, skipping the images that have already been pushed into the repo?

Edit: Ok I missed the part of cache it manually on the disk the first time, this solves the problem. Thank you
Issue: when i use datasets.load_dataset, i encounter the http connect error!

Comment:
The `audiofolder` loader is not available in version `2.3.2`, hence the error. Please run the `pip install -U datasets` command to update the `datasets` installation to make `load_dataset("audiofolder", ...)` work.
Issue: Compression kwargs have no effect when saving datasets as csv

Comment:
Hello @exs-avianello, I have reproduced the bug successfully and have understood the problem. But I am confused regarding this part of the statement, "`pandas.DataFrame.to_csv` is always called with a buf-like `path_or_buf`".

Can you please elaborate on it?

Thanks!
Issue: Compression kwargs have no effect when saving datasets as csv

Comment:
Hi @aryanxk02 ! Sure, what I actually meant is that when passing a path-like `path_or_buf` here

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/arrow_dataset.py#L4708-L4714 

it gets converted to a file object behind the scenes here

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/io/csv.py#L92-L94

and the eventual pandas `.to_csv()` calls that write to it always get `path_or_buf=None`, making pandas ignore the `compression` kwarg in the `to_csv_kwargs`

https://github.com/huggingface/datasets/blob/14f6edd9222e577dccb962ed5338b79b73502fa5/src/datasets/io/csv.py#L107-L109
Issue: Compression kwargs have no effect when saving datasets as csv

Comment:
@exs-avianello When `path_or_buf` is set to None, the `to_csv()` method will return the CSV data as a string instead of saving it to a file. Hence the compression doesn't take place. I think setting `path_or_buf=self.path_or_buf` should work. What you say?
Issue:   File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 992, in _download_and_prepare     if str(split_generator.split_info.name).lower() == "all": AttributeError: 'str' object has no attribute 'split_info'. Did you mean: 'splitlines'?

Comment:
Instead of writing the loading script, you can use the built-in loader to [load JSON files](https://huggingface.co/docs/datasets/loading#json):
```python
from datasets import load_dataset
ds = load_dataset("json", data_files={"train": os.path.join(data_dir["train"]), "dev": os.path.join(data_dir["dev"])})
```
Issue: Documentation links to examples are broken

Comment:
These docs are outdated (version 1.2.1 is over two years old). Please refer to [this](https://huggingface.co/docs/datasets/dataset_script) version instead.

Initially, we hosted datasets in this repo, but now you can find them [on the HF Hub](https://huggingface.co/datasets) (e.g. the [`ag_news`](https://huggingface.co/datasets/ag_news/blob/main/ag_news.py) script)
Issue: load_dataset hangs on WSL

Comment:
Even if a dataset is cached, we still make requests to check whether the cache is up-to-date. [This](https://huggingface.co/docs/datasets/v2.13.1/en/loading#offline) section in the docs explains how to avoid them and directly load the cached version.
Issue: load_dataset hangs on WSL

Comment:
Thanks - that works! However it doesn't resolve the original issue (but I am not sure if it is a WSL problem)
Issue: load_dataset hangs on WSL

Comment:
We use `requests` to make HTTP requests (and `aiohttp` in the streaming mode), so I don't think we can provide much help regarding the socket issue (it probably has something to do with WSL).
Issue: DownloadConfig.proxies not work when load_dataset_builder calling HfApi.dataset_info  

Comment:
`HfApi` comes from the `huggingface_hub` package. You can use [this](https://huggingface.co/docs/huggingface_hub/v0.16.3/en/package_reference/utilities#huggingface_hub.configure_http_backend) utility to change the `huggingface_hub`'s `Session` proxies (see the example).

We plan to implement https://github.com/huggingface/datasets/issues/5080 and make this behavior more consistent eventually.
Issue: Using a dataset for a use other than it was intended for.

Comment:
I've opened a PR with a fix. In the meantime, you can avoid the error by deleting `task_templates` with `dataset.info.task_templates = None` before the `interleave_datasets` call.
`
Issue: Inconsistent "The features can't be aligned" error when combining map, multiprocessing, and variable length outputs

Comment:
This scenario currently requires explicitly passing the target features (to avoid the error): 
```python
import datasets

...

features = dataset.features
features["output"] = = [{"test": datasets.Value("int64")}]
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32, features=features)
```
Issue: Request to Share/Update Dataset Viewer Code

Comment:
Hi ! The huggingface/dataset-viewer code was not maintained anymore because we switched to a new dataset viewer that is deployed available for each dataset the Hugging Face website.

What are you using this old repository for ?
Issue: Request to Share/Update Dataset Viewer Code

Comment:
I think these parts are outdated:

* https://github.com/huggingface/datasets-viewer/blob/8efad8eae313a891f713469983bf4c744786f26e/run.py#L126-L131
* https://github.com/huggingface/datasets-viewer/blob/8efad8eae313a891f713469983bf4c744786f26e/run.py#L145-L150

To make the viewer work, the first one should be replaced with the following:
```python
dataset_module = datasets.load.dataset_module_factory(path)
builder_cls = datasets.load.import_main_class(dataset_module.module_path)
confs = builder_cls.BUILDER_CONFIGS
```
And the second one:
```python
dataset_module = datasets.load.dataset_module_factory(path)
builder_cls = datasets.load.import_main_class(dataset_module.module_path)
if conf:
    builder_instance = builder_cls(name=conf, cache_dir=path if path_to_datasets is not None else None)
else:
    builder_instance = builder_cls(cache_dir=path if path_to_datasets is not None else None)
```

But as @lhoestq suggested, it's better to use the `datasets-server` API nowadays to [fetch the rows](https://huggingface.co/docs/datasets-server/rows).
Issue: Request to Share/Update Dataset Viewer Code

Comment:
> The dataset viewer on the Hugging Face website is incredibly useful

@mariosasko i think @lilyorlilypad wants to run the new dataset-viewer, not the old one
Issue: Request to Share/Update Dataset Viewer Code

Comment:
> wants to run the new dataset-viewer, not the old one

Thanks for the clarification for me. I do want to run the new dataset-viewer.
Issue: Request to Share/Update Dataset Viewer Code

Comment:
It should be possible to run it locally using the HF datasets-server API (docs [here](https://huggingface.co/docs/datasets-server)) but the front end part is not open source (yet ?)

The back-end is open source though if you're interested: https://github.com/huggingface/datasets-server
It automatically converts datasets on HF to Parquet, which is the format we use to power the viewer.
Issue: Request to Share/Update Dataset Viewer Code

Comment:
the new frontend would probably be hard to open source, as is, as it's quite intertwined with the Hub's code.

However, at some point it would be amazing to have a community-driven open source implementation of a frontend to datasets-server!