SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Add ProgressBar for `to_parquet`

Comment:
That would be awesome ! You can comment `#self-assign` to assign you to this issue and open a PR :) Will be happy to review
Issue: Sharding error with Multilingual LibriSpeech

Comment:
Thanks for reporting, @Nithin-Holla.

This is a known issue for multiple datasets and we are investigating it:
- See e.g.: https://huggingface.co/datasets/ami/discussions/3
Issue: concatenate_datasets fails when two dataset with shards > 1 and unequal shard numbers

Comment:
Hi ! Thanks for reporting :)

I managed to reproduce the hub using
```python

from datasets import concatenate_datasets, Dataset, load_from_disk

Dataset.from_dict({"a": range(9)}).save_to_disk("tmp/ds1")
ds1 = load_from_disk("tmp/ds1")
ds1 = concatenate_datasets([ds1, ds1])

Dataset.from_dict({"b": range(6)}).save_to_disk("tmp/ds2")
ds2 = load_from_disk("tmp/ds2")
ds2 = concatenate_datasets([ds2, ds2, ds2])

concatenate_datasets([ds1, ds2], axis=1)
```
and I get
```python
Traceback (most recent call last):                                                                                                    
  File "test.py", line 98, in <module>
    dds = concatenate_datasets([ds1, ds2], axis=1)
  File "/Users/.../datasets/combine.py", line 182, in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)
  File "/Users/.../datasets/arrow_dataset.py", line 5499, in _concatenate_map_style_datasets
    table = concat_tables([dset._data for dset in dsets], axis=axis)
  File "/Users/.../datasets/table.py", line 1778, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/Users/.../datasets/table.py", line 1483, in from_tables
    blocks = _extend_blocks(blocks, table_blocks, axis=axis)
  File "/Users/.../datasets/table.py", line 1477, in _extend_blocks
    result[i].extend(row_blocks)
IndexError: list index out of range
```

It appears to happen when the two datasets have a number of shards that is not the same
Issue: load_dataset() cannot find dataset_info.json with multiple training runs in parallel

Comment:
Hi ! It fails because the dataset is already being prepared by your first run. I'd encourage you to prepare your dataset before using it for multiple trainings.

You can also specify another cache directory by passing `cache_dir=` to `load_dataset()`.
Issue: load_dataset() cannot find dataset_info.json with multiple training runs in parallel

Comment:
Thank you! What do you mean by prepare it beforehand? I am unclear how to conduct dataset preparation outside of using the `load_dataset` function.
Issue: load_dataset() cannot find dataset_info.json with multiple training runs in parallel

Comment:
You can have a separate script that does load_dataset + map + save_to_disk to save your prepared dataset somewhere. Then in your training script you can reload the dataset with load_from_disk
Issue: load_dataset() cannot find dataset_info.json with multiple training runs in parallel

Comment:
Thank you! I believe I was running additional map steps after loading, resulting in the cache conflict.
Issue: dataset map function could not be hash properly

Comment:
Hi ! On macos I tried with
- py 3.9.11
- datasets 2.8.0
- transformers 4.25.1
- dill 0.3.4

and I was able to hash `prepare_dataset` correctly:
```python
from datasets.fingerprint import Hasher
Hasher.hash(prepare_dataset)
```

What version of transformers do you have ? Can you try to call `Hasher.hash` on the the tokenizer and the feature extractor to see which one can't be hashed ?
Issue: dataset map function could not be hash properly

Comment:
Thanks for your prompt reply.

I update datasets version to 2.8.0 and the warning is gong.
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
Hi ! I just tested locally and or colab and it works fine for 2.9 on `sst2`.

Also the code that is shown in your stack trace is not present in the 2.9 source code - so I'm wondering how you installed `datasets` that could cause this ? (you can check by searching for `[0:{label_ids[-1] + 1}]` in the [2.9 codebase](https://github.dev/huggingface/datasets/tree/b5672a956d5de864e6f5550e493527d962d6ae55) - it doesn't find anything)

Anyway you can try uninstalling `datasets` and install it again
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
For what it's worth, I've also gotten this error on 2.9.0, and I've tried uninstalling an reinstalling
![Screenshot 2023-02-01 at 11 06 55 AM](https://user-images.githubusercontent.com/22944438/216126466-6934e8f8-0be4-41f4-9822-8436dfafd61c.png)

I'm very new to this package (I was following this tutorial: https://huggingface.co/docs/transformers/training), so there's a good chance I was doing something wrong 😅 but thought I'd pass along the feedback
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
@ntrpnr @mtwichel Did you install `datasets` with conda ?

I suspect that `datasets` 2.9 on conda still have this issue for some reason. When I install `datasets` with `pip` I don't have this error.
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
> @ntrpnr @mtwichel Did you install datasets with conda ?

I did yeah, I wonder if that's the issue
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
I just checked on conda at https://anaconda.org/HuggingFace/datasets/files

and everything looks fine, I got
```python

f"ClassLabel expected a value for all label ids [0:{int(label_ids[-1]) + 1}] but some ids are missing."
```
as expected in features.py line 1760 (notice the "int()") to not have the TypeError.

From where on conda did you install `datasets` ? You should use the `HuggingFace` official channel

edit: the conda-forge one [here](https://anaconda.org/conda-forge/datasets/files) seems ok as well
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
Could you also try this in your notebook ? In case your python kernel doesn't match the `pip` environment in your shell
```python
import datasets; datasets.__version__
```
and
```
!which python
```
```python
import sys; sys.executable
```
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
Mmmm, just a potential clue:

Where are you running your Python code? Is it the Spyder IDE?

I have recently seen some users reporting conflicting Python environments while using Spyder...

Maybe related:
- #5487
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
Other potential clue:
- Had you already imported `datasets` before pip-updating it? You should first update datasets, before importing it. Otherwise, you need to restart the kernel after updating it.
Issue: [2.6.1][2.7.0] Upgrade `datasets` to fix `TypeError: can only concatenate str (not "int") to str`

Comment:
I installed `datasets` with Conda using `conda install datasets` and got this issue.

Then I tried to reinstall using
`
conda install -c huggingface -c conda-forge datasets
`
The issue is now fixed.
Issue: size_in_bytes the same for all splits

Comment:
Hi @Breakend,

Indeed, the attribute `size_in_bytes` refers to the size of the entire dataset configuration, for all splits (size of downloaded files + Arrow files), not the specific split.
This is also the case for `download_size` (downloaded files) and `dataset_size` (Arrow files).

The size of the Arrow files for a specific split can be accessed: e.g. size of the "test" split only
```python
ds["train"].info.splits["test"].num_bytes
```

I agree this is confusing and maybe we should improve it.
Issue: Missing state.json when creating a cloud dataset using a dataset_builder

Comment:
`load_from_disk` must be used on datasets saved using `save_to_disk`: they correspond to fully serialized datasets including their state.

On the other hand, `download_and_prepare` just downloads the raw data and convert them to arrow (or parquet if you want). We are working on allowing you to reload a dataset saved on S3 with `download_and_prepare` using `load_dataset` in #5281 

For now I'd encourage you to keep using `save_to_disk`
Issue: Missing state.json when creating a cloud dataset using a dataset_builder

Comment:
Thanks, I'll follow that issue. 

I was following the [cloud storage](https://huggingface.co/docs/datasets/filesystems) docs section and perhaps I'm missing some part of the flow; start with `load_dataset_builder` + `download_and_prepare`. You say I need an explicit `save_to_disk` but what object needs to be saved? the builder? is that related to the other issue?
Issue: Missing state.json when creating a cloud dataset using a dataset_builder

Comment:
Right now `load_dataset_builder` + `download_and_prepare` is to be used with tools like dask or spark, but `load_dataset` will support private cloud storage soon as well so you'll be able to reload the dataset with `datasets`.

Right now the only function that can load a dataset from a cloud storage is `load_from_disk`, that must be used with a dataset serialized with `save_to_disk`.
Issue: CI error: TypeError: dataclass_transform() got an unexpected keyword argument 'field_specifiers'

Comment:
@MFatnassi, this issue and the corresponding fix only affect our Continuous Integration testing environment.

Note that `datasets` does not depend on `spacy`.
Issue: Whisper Event - RuntimeError: The size of tensor a (504) must match the size of tensor b (448) at non-singleton dimension 1 100% 1000/1000 [2:52:21<00:00, 10.34s/it]

Comment:
Hey @catswithbats! Super sorry for the late reply! This is happening because there is data with label length (504) that exceeds the model's max length (448). 

There are two options here:
1. Increase the model's `max_length` parameter: 
```python
model.config.max_length = 512
```
2. Filter data with labels longer than max length: https://discuss.huggingface.co/t/open-to-the-community-whisper-fine-tuning-event/26681/21?u=sanchit-gandhi

Note that the datasets repo is reserved for issues directly related to the HF datasets library. Issues related to custom fine-tuning implementations are more applicable to the HF Forum: https://discuss.huggingface.co. You're more likely to get a response by posting your issue in the most applicable place and boost the chance of someone sharing a working solution!
Issue: Whisper Event - RuntimeError: The size of tensor a (504) must match the size of tensor b (448) at non-singleton dimension 1 100% 1000/1000 [2:52:21<00:00, 10.34s/it]

Comment:
@sanchit-gandhi Thank you for all your work on this topic.

I'm finding that changing the `max_length` value does not make this error go away.
Issue: Error when pushing to the CI hub

Comment:
Hmmm, git bisect tells me that the behavior is the same since https://github.com/huggingface/datasets/commit/67e65c90e9490810b89ee140da11fdd13c356c9c (3 Oct), i.e. https://github.com/huggingface/datasets/pull/4926
Issue: Error when pushing to the CI hub

Comment:
Maybe the current version of moonlanding in Hub CI is the issue.

I relaunched tests that were working two days ago: now they are failing. https://github.com/huggingface/datasets-server/commit/746414449cae4b311733f8a76e5b3b4ca73b38a9 for example

cc @huggingface/moon-landing
Issue: Error when pushing to the CI hub

Comment:
Hi! I don't think this has anything to do with `datasets`. Hub CI seems to be the culprit - the identical failure can be found in  [this](https://github.com/huggingface/datasets/pull/5389) PR (with unrelated changes) opened today.
Issue: Getting Value Error while loading a dataset..

Comment:
Hi! I can't reproduce this error locally (Mac) or in Colab. What version of `datasets` are you using?
Issue: Getting Value Error while loading a dataset..

Comment:
@valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.

Please, restart your Colab runtime and execute the install commands before importing `datasets`:
```python
!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))
```
Issue: Getting Value Error while loading a dataset..

Comment:
> @valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.
> 
> Please, restart your Colab runtime and execute the install commands before importing `datasets`:
> 
> ```python
> !pip install datasets
> !pip install zstandard
> 
> from datasets import load_dataset
> 
> ds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(ds))
> ```

I guess that was the problem, importing datasets before the installation of zstandard. Thank you for the feedback.
Issue: Missing documentation page : improve-performance

Comment:
Hi! Our documentation builder does not support links to sections, hence the bug. This is the link it should point to https://huggingface.co/docs/datasets/v2.8.0/en/cache#improve-performance.
Issue: `max_shard_size` in `datasets.push_to_hub()` breaks with large files

Comment:
Hi! 

This behavior stems from the fact that we don't always embed image bytes in the underlying arrow table, which can lead to bad size estimation (we use the first 1000 table rows to [estimate](https://github.com/huggingface/datasets/blob/9a7272cd4222383a5b932b0083a4cc173fda44e8/src/datasets/arrow_dataset.py#L4627) the external file size). We plan to address this in the next major release by always embedding external bytes. In the meantime, you can either shuffle the dataset with `.shuffle().flatten_indices()` to make the estimation more precise or embed the bytes in the table like so:
```python
from datasets.table import embed_table_storage
format = ds.format
ds = ds.with_format("arrow")
ds = ds.map(embed_table_storage, batched=True)
ds = ds.with_format(**format)
...
ds.push_to_hub(...)
```
Issue: Is `fs=` deprecated in `load_from_disk()` as well?

Comment:
Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR?
Issue: Is `fs=` deprecated in `load_from_disk()` as well?

Comment:
> Hi! Yes, we should deprecate the `fs` param here. Would you be interested in submitting a PR?

Yeah I can do that sometime next week. Should the storage_options be a new arg here?  I’ll look around for anywhere else where fs is an arg.
Issue: IterableDataset missing column_names, differs from Dataset interface

Comment:
Another example is that `IterableDataset.map` does not have `fn_kwargs`, among other arguments. It makes it harder to convert code from Dataset to IterableDataset.
Issue: IterableDataset missing column_names, differs from Dataset interface

Comment:
Hi! `fn_kwargs` was added to `IterableDataset.map` in `datasets 2.5.0`, so please update your installation (`pip install -U datasets`) to use it.

Regarding `column_names`, I agree we should add this property to `IterableDataset`. In the meantime, you can use `list(dataset.features.keys())` instead.
Issue: IterableDataset missing column_names, differs from Dataset interface

Comment:
Thanks! That's great news.

On Thu, Dec 22, 2022, 07:48 Mario Šaško ***@***.***> wrote:

> Hi! fn_kwargs was added to IterableDataset.map in datasets 2.5.0, so
> please update your installation (pip install -U datasets) to use it.
>
> Regarding column_names, I agree we should add this property to
> IterableDataset. In the meantime, you can use
> list(dataset.features.keys()) instead.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5383#issuecomment-1362993633>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHD6N2EQUFEOUFDW3VHSILWORZ45ANCNFSM6AAAAAATGKWVGM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Issue: IterableDataset missing column_names, differs from Dataset interface

Comment:
I'm marking this issue as a "good first issue", as it makes sense to have `IterableDataset.column_names` in the API. Besides the case when `features` are `None` (e.g., `features` are `None` after `map`), in which we can also return `column_names` as `None`, adding this property should be straightforward,
Issue: Wrong URL for the_pile dataset

Comment:
Hi! This error can happen if there is a local file/folder with the same name as the requested dataset. And to avoid it, rename the local file/folder.

Soon, it will be possible to explicitly request a Hub dataset as follows:https://github.com/huggingface/datasets/issues/5228#issuecomment-1313494020
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
Hi! I agree `skip` can be inefficient to use in the current state.

To make it fast, we could use "statistics" stored in Parquet metadata and read only the chunks needed to form a dataset. 

And thanks to the "datasets-server" project, which aims to store the Parquet versions of the Hub datasets (only the smaller datasets are covered currently), this solution can also be applied to datasets stored in formats other than Parquet. (cc @severo)
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
@mariosasko do the current parquet files created by the datasets-server already have the required "statistics"? If not, please open an issue on https://github.com/huggingface/datasets-server with some details to make sure we implement it.
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
Yes, nothing has to be changed on the datasets-server side. What I mean by "statistics" is that we can use the "row_group" metadata embedded in a Parquet file (by default) to fetch the requested rows more efficiently.
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
Glad to see the feature could be of interest. 

I'm sure there are many possible ways to implement this feature. I don't know enough about the datasets-server, but I guess that it is not instantaneous, in the sense that user-owned private datasets might need hours or days until they are ported to the datasets-server (if at all), which could be cumbersome. Having optionally that information in the `dataset_infos.json` file would make it easier for users to control the skip process a bit.
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
re: statistics:

- https://arrow.apache.org/docs/python/generated/pyarrow.parquet.FileMetaData.html
- https://arrow.apache.org/docs/python/generated/pyarrow.parquet.RowGroupMetaData.html

```python
>>> import pyarrow.parquet as pq
>>> import hffs
>>> fs = hffs.HfFileSystem("glue", repo_type="dataset", revision="refs/convert/parquet")
>>> metadata = pq.read_metadata("ax/glue-test.parquet", filesystem=fs)
>>> metadata
<pyarrow._parquet.FileMetaData object at 0x7f4537cec400>
  created_by: parquet-cpp-arrow version 7.0.0
  num_columns: 4
  num_rows: 1104
  num_row_groups: 2
  format_version: 1.0
  serialized_size: 2902
>>> metadata.row_group(0)
<pyarrow._parquet.RowGroupMetaData object at 0x7f45564bcbd0>
  num_columns: 4
  num_rows: 1000
  total_byte_size: 164474
>>> metadata.row_group(1)
<pyarrow._parquet.RowGroupMetaData object at 0x7f455005c400>
  num_columns: 4
  num_rows: 104
  total_byte_size: 13064
```
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
> user-owned private datasets might need hours or days until they are ported to the datasets-server (if at all)

private datasets are not supported yet (https://github.com/huggingface/datasets-server/issues/39)
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
@versae `Dataset.push_to_hub` writes shards in Parquet, so this solution would also work for such datasets (immediately after the push).
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
@mariosasko that is right. However, there are still a good amount of datasets for which the shards are created manually. In our very specific case, we create medium-sized datasets (rarely over 100-200GB) of both text and audio, we prepare the shards by hand and then upload then. It would be great to have immediate access to this download skipping feature for them too.
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
From looking at Arrow's source, it seems Parquet stores metadata at the end, which means one needs to iterate over a Parquet file's data before accessing its metadata. We could mimic Dask to address this "limitation" and write metadata in a `_metadata`/`_common_metadata` file in `to_parquet`/`push_to_hub`, which we could then use to optimize reads (if present). Plus, it's handy that PyArrow can also parse these metadata files.
Issue: Improve dataset `.skip()` speed in streaming mode

Comment:
So if Parquet metadata needs to be in its own file anyway, why not implement this skipping feature by storing the example counts per shard in `dataset_infos.json`? That would allow:
- Support both private and public datasets
- Immediate access to the feature upon uploading of shards
- Use any dataset, not only those uploaded using `.push_to_hub()`

A proper Parquet metadata file could still be created and "overwrite" the `dataset_infos.json` info in the datasets-server.
Issue: The dataset "the_pile", subset "enron_emails" , load_dataset() failure

Comment:
Thanks for reporting @shaoyuta. We are investigating it.

We are transferring the issue to "the_pile" Community tab on the Hub: https://huggingface.co/datasets/the_pile/discussions/4
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
The data files are hosted on HF at https://huggingface.co/datasets/allenai/c4/tree/main

You have 200 runs streaming the same files in parallel. So this is probably a Hub limitation. Maybe rate limiting ? cc @julien-c 

Maybe you can also try to reduce the number of HTTP requests by increasing the block size of each request. This can be done by increasing `DEFAULT_BLOCK_SIZE` in `fsspec.implementations.http`. Default is `5 * 2**20` (5MiB)

Anyway maybe it's just better to save the dataset locally in that case ?
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
you don't get an HTTP error code or something in your stack trace? Kinda hard to debug with this info
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
You could try to re-run using this `datasets` branch: [raise-err-when-disconnect](https://github.com/huggingface/datasets/compare/raise-err-when-disconnect?expand=1)
It should raise the fsspec error
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
The weird thing is that I already have it saved locally & it seems to indeed be using the cached one 🧐 ; I'm also using offline mode, so I don't think it has something to do with the Hub.
```
WARNING:datasets.load:Using the latest cached version of the module from /users/muennighoff/.cache/huggingface/modules/datasets_modules/datasets/c4/df532b158939272d032cc63ef19cd5b83e9b4d00c922b833e4cb18b2e9869b01 (last modified on Mon Dec 12 10:45:02 2022) since it couldn't be found locally at c4.
```
Issue: Using too many threads results in: Got disconnected from remote data host. Retrying in 5sec

Comment:
No, you passed `streaming=True` so it streams the data from the Hub.
This message just shows that you use the cached version of the `c4` **module**, aka the python script that is run to generate the examples from the raw data files.

Maybe the offline mode should also disable `fsspec`/`aiohttp` HTTP calls in `datasets` and not just the `requests` ones.