SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: how to limit the size of memory mapped file?

Comment:
Hi! Can you share the error this reproducer throws in your environment? `streaming=True` streams the dataset as it's iterated over without creating a memory-map file.
Issue: how to limit the size of memory mapped file?

Comment:
This issue has previously been reported here: https://github.com/huggingface/datasets/issues/5710. Reporting it in the Arrow repo makes more sense as they have control over memory mapping.

PS: this is the API to reduce the size of the generated Arrow file:
```python
from datasets import load_dataset
builder = load_dataset_builder("c4", "en")
builder.download_and_prepare(max_shard_size="5GB")
dataset = builder.as_dataset()
```

If this resolves the issue, we can consider exposing `max_shard_size` in `load_dataset`.
Issue: Make Dataset streaming queries retryable

Comment:
Hi! The streaming mode also retries requests - `datasets.config.STREAMING_READ_MAX_RETRIES` (20 sec by default) controls the number of retries and `datasets.config.STREAMING_READ_RETRY_INTERVAL` (5 sec) the sleep time between retries.

> At step 1800 I got a 504 HTTP status code error from Huggingface hub for my pytorch dataloader

A minor Hub outage that we experienced yesterday could be the cause.
Issue: Configurations in yaml not working

Comment:
Unfortunately, I cannot reproduce this behavior on my machine or Colab - the reproducer returns `['main_data', 'additional_data']` as expected.
Issue: Configurations in yaml not working

Comment:
Thank you for looking into this, Mario. Is this on [my repository](https://huggingface.co/datasets/tsor13/test), or on another one that you have reproduced? Would you mind pointing me to it if so?
Issue: Configurations in yaml not working

Comment:
Whoa, in colab I received the correct behavior using my dataset. It must have something to do with my local copy of `datasets` (which again just failed).

I've tried uninstalling/reinstnalling to no avail
Issue: Configurations in yaml not working

Comment:
hi @tsor13 , I haven't been able to reproduce your issue on `tsor13/test` dataset locally either. reinstalling doesn't help?
Issue: load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields

Comment:
Hi ! Feel free to open a discussion at https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/discussions to ask the file to be fixed (or directly open a PR with the fixed file)

`datasets` expects all the examples to have the same fields
Issue: load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields

Comment:
@lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570). Maybe setting `streaming=True` can workaround this problem. Would you agree with my statement?
Issue: load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields

Comment:
> @lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570).

Correct. Therefore any example that doesn't follow the inferred schema will make the code fail.

> Maybe setting streaming=True can workaround this problem. Would you agree with my statement?

You'll meet the same problem but later - when streaming and arriving at the problematic example
Issue: load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields

Comment:
@lhoestq I just run below test with streaming=True and is not failing at the problematic example
```python
ds = load_dataset('json', data_files='/path_to_local_RedPajamaData/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl', streaming=True)
count = 0
for i in ds['train']:
    count += 1
    print(count)
```

and completes the 262241 samples successfully. It does error our when streaming is not used
Issue: DatasetInfo.__init__() got an unexpected keyword argument '_column_requires_decoding'

Comment:
Thanks for reporting, but we can only fix this issue if you can provide a reproducer that consistently reproduces it.
Issue: DatasetInfo.__init__() got an unexpected keyword argument '_column_requires_decoding'

Comment:
Does this error occur even if you change the cache directory (the `cache_dir` parameter in `load_dataset`)?
Issue: Why not use self._epoch as seed to shuffle in distributed training with IterableDataset

Comment:
`_effective_generator` returns a RNG that takes into account `self._epoch` and the current dataset's base shuffling RNG (which can be set by specifying `seed=` in `.shuffle() for example`).

To fix your error you can pass `seed=` to `.shuffle()`. And the shuffling will depend on both this seed and `self._epoch`
Issue: FolderBase Dataset automatically resolves under current directory when data_dir is not specified

Comment:
Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)
Issue: FolderBase Dataset automatically resolves under current directory when data_dir is not specified

Comment:
I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.
Issue: FolderBase Dataset automatically resolves under current directory when data_dir is not specified

Comment:
This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed
Issue: Faster sorting for single key items

Comment:
`Dataset.sort` essentially does the same thing except it uses `pyarrow.compute.sort_indices` which doesn't involve copying the data into python objects (saving memory)

```python
sort_keys = [(col, "ascending") for col in column_names]
indices = pc.sort_indices(self.data, sort_keys=sort_keys)
return self.select(indices)
```
Issue: Allow dataset implement .take

Comment:
```
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset  # to force dataset.take(batch_size) to work in non-streaming mode
   ```
Issue: Allow dataset implement .take

Comment:
Feel free to work on this. In addition, `IterableDataset` supports `skip`, so we should also add this method to `Dataset`.
Issue: Dataset.from_parquet cannot load subset of columns

Comment:
Looks like this regression was introduced in `datasets==2.13.0` (`2.12.0` could load a subset of columns)

This does not appear to be fixed by https://github.com/huggingface/datasets/pull/6045 (bug still exists on `main`)
Issue: ValueError when running BeamBasedBuilder with GCS path in cache_dir

Comment:
The cause of the error seems to be that `datasets` adds "gcs://" as a schema, while `beam` checks only "gs://".

datasets: https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/builder.py#L822

beam: [link](https://github.com/apache/beam/blob/25e1a64641b1c8a3c0a6c75c6e86031b87307f22/sdks/python/apache_beam/io/filesystems.py#L98-L101)
```
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
```
Issue: DatasetGenerationError when load glue benchmark datasets from `load_dataset` 

Comment:
This issue can happen if there is a directory named "glue" relative to the Python script with the `load_dataset` call (similar issue to this one: https://github.com/huggingface/datasets/issues/5228). Is this the case?
Issue: custom load dataset to hub

Comment:
> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them
Issue: the-stack-dedup fails to generate

Comment:
It seems that some parquet files have additional columns.

I ran a scan and found that two files have the additional `__id__` column:

1. `hf://datasets/bigcode/the-stack-dedup/data/numpy/data-00000-of-00001.parquet`
2. `hf://datasets/bigcode/the-stack-dedup/data/omgrofl/data-00000-of-00001.parquet`

We should open a PR to fix those two files
Issue: the-stack-dedup fails to generate

Comment:
The files have been fixed ! I'm closing this one but feel free to re-open if you still have the issue
Issue: TypeError: ClientSession._request() got an unexpected keyword argument 'https'

Comment:
Hi! I cannot reproduce this error on my machine or in Colab. Which version of `fsspec` do you have installed?
Issue: Offline dataset viewer

Comment:
Hi, thanks for the suggestion. It's not possible at the moment. The viewer is part of the Hub codebase and only works on public datasets. Also, it relies on [Datasets Server](https://github.com/huggingface/datasets-server/), which prepares the data and provides an API to access the rows, size, etc.

If you're interested in hosting your data as a private dataset on the Hub, you might want to look at https://github.com/huggingface/datasets-server/issues/39.
Issue: `datasets` cannot be installed alongside `apache-beam`

Comment:
I noticed that this is actually covered by issue #5613, which for some reason I didn't see when I searched the issues in this repo the first time.
Issue: Dataset is slower after calling `to_iterable_dataset`

Comment:
It's roughly the same code between the two so we can expected roughly the same speed, could you share a benchmark ?
Issue: default config name doesn't work when config kwargs are specified.

Comment:
What should be the behavior in this case ? Should it override the default config with the added parameter ?
Issue: default config name doesn't work when config kwargs are specified.

Comment:
I know why it should be treated as a new config if overriding parameters are passed. But in some case, I just pass in some common fields like `data_dir`.

For example, I want to extend the FolderBasedBuilder as a multi-config version, the `data_dir` or `data_files` are always passed by user and should not be considered as overriding the default config. In current state, I cannot leverage the feature of default config since passing `data_dir` will disable the default config.
Issue: default config name doesn't work when config kwargs are specified.

Comment:
Thinking more about it I think the current behavior is the right one.

Provided parameters should be passed to instantiate a new BuilderConfig.

What's the error you're getting ?
Issue: default config name doesn't work when config kwargs are specified.

Comment:
For example, this works to use default config with name '_all_':
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train")
```
while this failed to use default config
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train", data_dir='.')
```
After manually specifying it, it works again.
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", "_all_", split="train", data_dir='.')
```
Issue: default config name doesn't work when config kwargs are specified.

Comment:
It should work if you explicitly ask for the config you want to override

```python
load_dataset('/dataset/with/multiple/config', 'name_of_the_default_config', some_field_in_config='some')
```

Alternatively you can have a BuilderConfig class that when instantiated returns a config with the right default values. In this case this code would instantiate this config with the default values except for the parameter to override:

```python
load_dataset('/dataset/with/multiple/config', some_field_in_config='some')
```
Issue: default config name doesn't work when config kwargs are specified.

Comment:
@lhoestq Yes. But it doesn't work for me.

Here's my dataset for example.
```
lass MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, name: str, version: str, **kwargs):
        self.option1 = kwargs.pop("option1", False)
        self.option2 = kwargs.pop("option2", 5)

        super().__init__(
            name=name,
            version=datasets.Version(version),
            **kwargs)


class MyDataset(datasets.GeneratorBasedBuilder):
    DEFAULT_CONFIG_NAME = "v1"

    BUILDER_CONFIGS = [
        UnifiedTtsDatasetConfig(
            name="v1",
            version="1.0.0",
            description="Initial version of the dataset"
        ),
    ]

    def _info(self) -> DatasetInfo:
        _ = self.option1
        ....
```

Here it's okay to use `load_dataset('my_dataset.py')` for loading the default config `v1`.

But if I want to override the default values in config with `load_dataset('my_dataset.py', option2=3)`, it failed to find my default config `v1.

Unless I use `load_dataset('my_dataset.py', 'v1', option2=3)`

So according to your advice, how can I modify my dataset to be able to override default config without manually specifying it.
Issue: default config name doesn't work when config kwargs are specified.

Comment:
@lhoestq The error is
```
def _info(self) -> DatasetInfo:
        _ = self.option1  <-
        ....
AttributeError: 'BuilderConfig' object has no attribute 'option1'
```
which seems to find another unknown config.

You can try this line `datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train", data_dir='.')`, it's a multi-config dataset on HF hub and the error is the same.

My insights:
https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518
if `config_kwargs` is provided here, the if branch is skipped.
Issue: default config name doesn't work when config kwargs are specified.

Comment:
I see, you just have to set this class attribute to your builder class :)

```python
BUILDER_CONFIG_CLASS = MyDatasetConfig
```
Issue: default config name doesn't work when config kwargs are specified.

Comment:
So what does this attribute do? In most cases it's not used and the [documents for multi-config dataset](https://huggingface.co/docs/datasets/main/en/image_dataset#multiple-configurations) never mentioned that.
Issue: default config name doesn't work when config kwargs are specified.

Comment:
It tells which builder config class to instantiate if additional config parameters are passed to load_dataset
Issue: default config name doesn't work when config kwargs are specified.

Comment:
@lhoestq maybe we can enhance the document to say something about the common attributes of `DatasetBuilder`
Issue: IndexError: Invalid key: 88 is out of bounds for size 0

Comment:
Hi @TomasAndersonFang,

I guess in this case it may be an issue with `transformers` (or `PyTorch`). I would recommend you open an issue on their repo.
Issue: Private datasets do not load when passing token

Comment:
Our CI did not catch this issue because with current implementation, stored token in `HfFolder` (which always exists) is used by default.
Issue: Private datasets do not load when passing token

Comment:
I can confirm this and have the same problem (and just went almost crazy because I couldn't figure out the source of this problem because on another computer everything worked well even with `DownloadMode.FORCE_REDOWNLOAD`).
Issue: Private datasets do not load when passing token

Comment:
We are planning to do a patch release today, after the merge of the fix:
- #6127

In the meantime, the problem can be circumvented by passing `download_config` instead:
```python
from datasets import DownloadConfig, load_dataset

load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
```
Issue: Private datasets do not load when passing token

Comment:
> We are planning to do a patch release today, after the merge of the fix:
> 
>     * [Fix authentication issues #6127](https://github.com/huggingface/datasets/pull/6127)
> 
> 
> In the meantime, the problem can be circumvented by passing `download_config` instead:
> 
> ```python
> from datasets import DownloadConfig, load_dataset
> 
> load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
> ```

This did not work for me (there was some other error with the split being an unexpected size 0). Downgrading to 2.13 fixed it....
Issue: Datasets crashing runs due to KeyError

Comment:
i once had the same error and I could fix that by pushing a fake or a dummy commit on my hugging face dataset repo
Issue: Datasets crashing runs due to KeyError

Comment:
Hi! We need a reproducer to fix this. Can you provide a link to the dataset (if it's public)?
Issue: Datasets crashing runs due to KeyError

Comment:
> Hi! We need a reproducer to fix this. Can you provide a link to the dataset (if it's public)?

Hi Mario,

Unfortunately, the dataset in question is currently private until the model is trained and released.

This is not happening with one dataset but numerous hosted private datasets.

I am only loading the dataset and doing nothing else currently. It seems to happen completely sporadically.

Thank you,

Enrico
Issue: Inaccurate Bounding Boxes in "wildreceipt" Dataset

Comment:
Hi! Thanks for the investigation, but we are not the authors of these datasets, so please report this on the Hub instead so that the actual authors can fix it.
Issue: Lookahead streaming support?

Comment:
In which format is your dataset? We could expose the `pre_buffer` flag for Parquet to use PyArrow's background thread pool to speed up loading.
Issue: IterableDataset.from_generator() fails with pickle error when provided a generator or iterator

Comment:
Hi! `IterableDataset.from_generator` expects a generator function, not the object (to be consistent with `Dataset.from_generator`).

You can fix the above snippet as follows:
```python
train_dataset = IterableDataset.from_generator(line_generator, fn_kwargs={"files": model_training_files})
```
Issue: Cache not being used when loading commonvoice 8.0.0

Comment:
You can avoid this by using the `revision` parameter in `load_dataset` to always force downloading a specific commit (if not specified it defaults to HEAD, hence the redownload).
Issue: Cache not being used when loading commonvoice 8.0.0

Comment:
Thanks @mariosasko this works well, looks like I should have read the documentation a bit more carefully.  

It is still a bit confusing which hash I should provide:  passing `revision = c8fd66e85f086e3abb11eeee55b1737a3d1e8487` from https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/commits/main caused the cached version at `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a` to be loaded, so I had to know that it was the previous commit unless I've missed something else.
Issue: load_dataset() fails with streamlit caching inside docker

Comment:
Hi! This should be fixed in the latest (patch) release (run `pip install -U datasets` to install it). This behavior was due to a bug in our authentication logic.
Issue: yaml error using push_to_hub with generated README.md

Comment:
Thanks for reporting! This is a bug in converting the `ArrayXD` types to YAML. It will be fixed soon.
Issue: raise FileNotFoundError("Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory." )

Comment:
This should work: `load_dataset("path/to/downloaded_repo")`

`load_from_disk` is intended to be used on directories created with `Dataset.save_to_disk` or `DatasetDict.save_to_disk`
Issue: [BUG] Dataset initialized from in-memory data does not create cache.

Comment:
This is expected behavior. You must provide `cache_file_name` when performing `.map` on an in-memory dataset for the result to be cached.
Issue: Problems in downloading Amazon reviews from HF

Comment:
Thanks for reporting, @610v4nn1.

Indeed, the source data files are no longer available. We have contacted the authors of the dataset and they report that Amazon has decided to stop distributing the multilingual reviews dataset.

We are adding a notification about this issue to the dataset card.

See: https://huggingface.co/datasets/amazon_reviews_multi/discussions/4#64c3898db63057f1fd3ce1a0
Issue: Loading local datasets got strangely stuck

Comment:
Yesterday I waited for more than 12 hours to make sure it was really **stuck** instead of proceeding too slow.
Issue: Loading local datasets got strangely stuck

Comment:
I've had similar weird issues with `load_dataset` as well. Not multiple files, but dataset is quite big, about 50G.
Issue: Loading local datasets got strangely stuck

Comment:
We use a generic multiprocessing code, so there is little we can do about this - unfortunately, turning off multiprocessing seems to be the only solution. Multithreading would make our code easier to maintain and (most likely) avoid issues such as this one, but we cannot use it until the GIL is dropped (no-GIL Python should be released in 2024, so we can start exploring this then)
Issue: load local json_file as dataset

Comment:
Hi! We use PyArrow to read JSON files, and PyArrow doesn't allow different value types in the same column. #5776 should address this.

In the meantime, you can combine `Dataset.from_generator` with the above code to cast the values to the same type.
Issue: How do i get "amazon_us_reviews

Comment:
Seems like the problem isn't with the library, but the dataset itself hosted on AWS S3.

Its [homepage](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) returns an `AccessDenied` XML response, which is the same thing you get if you try to log the `record` that triggers the exception

```python
try:
    example = self.info.features.encode_example(record) if self.info.features is not None else record
except Exception as e:
    print(record)
```

⬇️

```
{'<?xml version="1.0" encoding="UTF-8"?>': '<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>N2HFJ82ZV8SZW9BV</RequestId><HostId>Zw2DQ0V2GdRmvH5qWEpumK4uj5+W8YPcilQbN9fLBr3VqQOcKPHOhUZLG3LcM9X5fkOetxp48Os=</HostId></Error>'}
```
Issue: How do i get "amazon_us_reviews

Comment:
I have figured it out. there was an option of **parquet formated files**  i downloaded some from there.
Issue: How do i get "amazon_us_reviews

Comment:
Thanks for reporting, @IqraBaluch.

We contacted the authors and unfortunately they reported that Amazon has decided to stop distributing this dataset.
Issue: How do i get "amazon_us_reviews

Comment:
I noticed that some book data is missing, we can only get Books_v1_02 data. 
Is there any way we can get the Books_v1_00 and Books_v1_01? 
Really appreciate !!!
Issue: How do i get "amazon_us_reviews

Comment:
@albertvillanova will this dataset be retired given the data are no longer hosted on S3? What is done in cases such as these?
Issue: AssertionError: daemonic processes are not allowed to have children

Comment:
We could add a "threads" parallel backend to `datasets.parallel.parallel_backend` to support downloading with threads but note that `download_and_extract` also decompresses archives, and this is a CPU-intensive task, which is not ideal for (Python) threads (good for IO-intensive tasks).
Issue: AssertionError: daemonic processes are not allowed to have children

Comment:
> We could add a "threads" parallel backend to `datasets.parallel.parallel_backend` to support downloading with threads but note that `download_and_extract` also decompresses archives, and this is a CPU-intensive task, which is not ideal for (Python) threads (good for IO-intensive tasks).

Great! Download takes more time than extract, multiple threads can download in parallel, which can speed up a lot.