SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
@lhoestq thnx for getting back to me! i've tested the suggested method, but unfortunately the memory consumption is the very same:

```
from datasets import Dataset, Features, Array2D, Array3D, load_from_disk
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data}, features=Features({column_name: Array3D(shape=array_shape, dtype="float64")}))
dataset.save_to_disk("foo")

foo_db = load_from_disk("foo")
colum_value = foo_db[column_name]
```

the very same happens when you create the dataset, but dont specify the feature type.

i've tried running this on different envs (macOS, linux) and it's behaving the very same way.
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
When you call `colum_value = foo_db[column_name]`, you load the full column in memory.

If you want to avoid filling up your memory, you can access chunks of data instead
```python
embeddings = dataset[i:i + chunk_size]["embeddings"]
```
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
@lhoestq yeah that's intentional, i.e. i really want to load the whole column into the memory. but as said above there's an unreasonable amount of overhead for the memory. the np array itself is using about 1G of memory:
```
>>> getsizeof(data)/1024/1024
937.5001525878906
```
that accessing of column above is using 10x memory compared to the original numpy array.
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
The dataset must be twice as big because we use regular arrow ListArray under the hood and not FixedSizeListArray. Basically we store unnecessary offsets.

And this should affect performance as well. When we developed this, FixedSizeListArray still had some issues but they should be resolved on the PyArrow side now
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
A doubling would be fine. My very basic understanding of PyArrow is that using ListArray is probably related to the issue though. Using a multi-dimensional array in datasets is storing everything as strange nested 1d object arrays, which I imagine is creating the massive overhead.

I think it should be a PyArrow Tensor, no?
Issue: Column of arrays (2D+) are using unreasonably high memory

Comment:
PyArrow tensors are not part of the Arrow format AFAIK:

> There is no direct support in the arrow columnar format to store Tensors as column values.

source: https://github.com/apache/arrow/issues/4802#issuecomment-508494694
Issue: Dataset without script does not support regular JSON data file

Comment:
Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.
Issue: fsspec 2022.8.2 breaks xopen in streaming mode

Comment:
loading `fsspec==2022.7.1` fixes this issue, setup.py would need to be changed to prevent users from using the latest version of fsspec.
Issue: fsspec 2022.8.2 breaks xopen in streaming mode

Comment:
Hi @DCNemesis, thanks for reporting.

That was a temporary issue in `fsspec` releases 2022.8.0 and 2022.8.1. But they fixed it in their patch release 2022.8.2 (and yanked both previous versions). See:
- https://github.com/huggingface/transformers/pull/18846

Are you sure you have version 2022.8.2 installed?
```shell
pip install -U fsspec
```
Issue: fsspec 2022.8.2 breaks xopen in streaming mode

Comment:
@albertvillanova I was using a temporary Google Colab instance, but checking it again today it seems it was loading 2022.8.1 rather than 2022.8.2. It's surprising that colab is using the version that was replaced the same day it was released. Testing with 2022.8.2 did work. It appears Colab [will be fixing it](https://github.com/googlecolab/colabtools/issues/3055) on their end too.
Issue: fsspec 2022.8.2 breaks xopen in streaming mode

Comment:
Thanks for the additional information.

Once we know 2022.8.2 works, I'm closing this issue. Feel free to reopen it if necessary.
Issue: BioASQ  AttributeError: 'BuilderConfig' object has no attribute 'schema'

Comment:
Following worked:

```
data_dir = "/Users/dlituiev/repos/datasets/bioasq/"
bioasq_task_b = load_dataset("aps/bioasq_task_b", data_dir=data_dir, name="bioasq_9b_source")
```

Would maintainers be open to one of the following:
- automating this with a latest default config (e.g.  `bioasq_9b_source`); how can this be generalized to other datasets?
- providing an actionable error message that lists available `name` values? I only got available `name` values once I've provided something there (`name="aps/bioasq_task_b"`), before it would not even mention that it requires `name` argument
Issue: BioASQ  AttributeError: 'BuilderConfig' object has no attribute 'schema'

Comment:
Hi ! In general the list of available configurations is prompted. I think this is an issue with this specific dataset.

Feel free to open a new discussions at https://huggingface.co/datasets/aps/bioasq_task_b/discussions

cc @apsdehal

In particular it sounds like the `BUILDER_CONFIG_CLASS= BigBioConfig ` class attribute is missing and the _info should account for schema being None and raise an error
Issue: Trec Dataset has incorrect labels

Comment:
Thanks for reporting, @wmpauli. 

Indeed we recently fixed this issue:
- #4801 

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.
Issue: vivos (Vietnamese speech corpus) dataset not accessible

Comment:
If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)
Issue: Dataset Viewer issue for ubuntu_dialogs_corpus

Comment:
The dataset maintainers (https://huggingface.co/datasets/ubuntu_dialogs_corpus) decided to forbid the dataset from being downloaded automatically (https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download), and the dataset viewer respects this.
We will try to improve the error display though. Thanks for reporting.
Issue: Dataset Viewer issue for indonesian-nlp/librivox-indonesia

Comment:
Thanks @albertvillanova for checking the issue. Actually, I can use the dataset like following:
```
>>> from datasets import load_dataset
>>> ds=load_dataset("indonesian-nlp/librivox-indonesia")
No config specified, defaulting to: librivox-indonesia/all
Reusing dataset librivox-indonesia (/root/.cache/huggingface/datasets/indonesian-nlp___librivox-indonesia/all/1.0.0/9a934a42bfb53dc103003d191618443b8a786bea2bd7bb0bc2d9454b8494521e)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 500.87it/s]
>>> ds
DatasetDict({
    train: Dataset({
        features: ['path', 'language', 'reader', 'sentence', 'audio'],
        num_rows: 7815
    })
})
>>> ds["train"][0]
{'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'language': 'sun', 'reader': '3174', 'sentence': 'pernyataan umum ngeunaan hak hak asasi manusa sakabeh manusa', 'audio': {'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.02419001,
       -0.01957154, -0.01502833], dtype=float32), 'sampling_rate': 44100}}

```
It would be just nice if I also can see it using dataset viewer.
Issue: Dataset Viewer issue for indonesian-nlp/librivox-indonesia

Comment:
Yes, the issue arises when streaming (that is used by the viewer): your script does not support streaming and to support it in this case there are some subtleties that we are explaining better in our docs in a work-in progress pull request:
- #4872

Just note that when streaming, `local_extracted_archive` is None, and this code line generates the error:
```python
filepath = local_extracted_archive + "/librivox-indonesia/audio_transcription.csv"
```

For a proper implementation, you could have a look at: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py

You can test your script locally by passing `streaming=True` to `load_dataset`:
```python
ds = load_dataset("indonesian-nlp/librivox-indonesia", split="train", streaming=True); item = next(iter(ds)); item
```
Issue: Dataset Viewer issue for indonesian-nlp/librivox-indonesia

Comment:
Hi @albertvillanova , I just add the streaming functionality and it works in the first try :-) Thanks a lot!
Issue: Dataset/DatasetDict.filter() cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

Comment:
Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.

In your case, something like
```python
from datasets import load_dataset


ds_mc4_ja = load_dataset("mc4", "ja")  # This will take 6+ hours... perhaps test it with a toy dataset instead?
ds_mc4_ja_2020 = ds_mc4_ja.filter(
  lambda batch: [timestamp[:4] == "2020" for timestamp in batch["timestamp"]],
  batched=True,
)
```

Let me know if it helps !
Issue: Dataset/DatasetDict.filter() cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

Comment:
> Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.
> [...]
> Let me know if it helps !

Hi @lhoestq,

Ah, my bad, I totally forgot that part...
Sorry for the trouble and thank you for the kind help!
Issue: Dataset Viewer issue for bigscience-biomedical/biosses

Comment:
Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```
Issue: Dataset Viewer issue for bigscience-biomedical/biosses

Comment:
thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here 

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens.
Issue: Dataset Viewer issue for bigscience-biomedical/biosses

Comment:
closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )
Issue: Unable to load local tsv files through load_dataset method

Comment:
Hi @DataNoob0723,

Under the hood, we use `pandas` to load CSV/TSV files. Therefore, you should use "csv" and pass `sep="\t"`, as explained in our docs: https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/loading_methods#from-files
```python
ds = load_dataset('csv', sep="\t", data_files=data_files)
```
Issue: Dataset Viewer issue for pysentimiento/spanish-targeted-sentiment-headlines

Comment:
Thanks for reporting, it's fixed now (I refreshed it manually). It's a known issue; we hope it will be fixed permanently in a few days.

<img width="1508" alt="Capture d’écran 2022-09-05 à 18 31 22" src="https://user-images.githubusercontent.com/1676121/188489762-0ed86a7e-dfb3-46e8-a125-43b815a2c6f4.png">
Issue: Keys mismatch: make error message more informative

Comment:
Good idea ! I think this can be improved in `Features.reorder_fields_as()` indeed at

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/features.py#L1739-L1740

Is it something you would be interested in contributing ?
Issue: Keys mismatch: make error message more informative

Comment:
Is this open to work on? I'd love to take on this as my first issue.
Issue: FileNotFoundError while downloading wikipedia dataset for any language

Comment:
Hi @Shilpac20,

As explained in the Wikipedia dataset card: https://huggingface.co/datasets/wikipedia
> You can find the full list of languages and dates [here](https://dumps.wikimedia.org/backup-index.html).

This means that, before passing a specific date, you should first make sure it is available online, as Wikimedia only keeps last X months (depending on the size of the corresponding language dump)): e.g. to see which dates "aa" Wikipedia is available online, see https://dumps.wikimedia.org/aawiki/ (as of today 2022-08-31, the available dates are from [20220401](https://dumps.wikimedia.org/aawiki/20220401/) to [20220820](https://dumps.wikimedia.org/aawiki/20220820/)).
Issue: FileNotFoundError while downloading wikipedia dataset for any language

Comment:
Hi, the date that I have specified "20220401" is available for the language "aa". The error persists for any other available dates as present in https://dumps.wikimedia.org/aawiki/. The error is mainly due to apache beam not able to write the downloaded files. Any help on this?
Issue: FileNotFoundError while downloading wikipedia dataset for any language

Comment:
I am struggling with basically the same issue. I am trying to download the German Wikipedia dump.

As per the [documentation](https://huggingface.co/datasets/wikipedia), `"20220301.de"` should be available as a pre-processed dataset.

Issuing the command mentioned in the documentation cited above

    from datasets import load_dataset
    load_dataset("wikipedia", "20220301.de")

raises the following `FileNotFound` error

    FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/dewiki/20220301/dumpstatus.json

Using the ([undocumented](https://huggingface.co/docs/datasets/v1.2.1/package_reference/loading_methods.html#datasets.load_dataset)?) call to `load_dataset()` with `language` and `date` parameters

    load_dataset("wikipedia", language="de", date="20220301", beam_runner="DirectRunner")

produces the same error.

EDIT: as I am using `datasets` v2.7.1, I should be looking at [that version's documentation](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/loading_methods#datasets.load_dataset)! It is mentioned there, that additional `kwargs` are "passed to the [BuilderConfig](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/builder_classes#datasets.BuilderConfig) and used in the [DatasetBuilder](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/builder_classes#datasets.DatasetBuilder)". So I guess that is how `language` and `date` are used.

As I can see a folder `20221130` on `https://dumps.wikimedia.org/dewiki/`, I also tried

    from datasets import load_dataset
    load_dataset("wikipedia", "20221130.de")

which throws another error:

    ValueError: BuilderConfig 20221120.de not found. Available: ['20220301.aa', ... '20220301.de', ...

basically telling me that the dataset I originally requested (`'20220301.de'`) is available...

It seems that `load_dataset` is not handling the vanishing older dumps for Wikipedia correctly?
Issue: FileNotFoundError while downloading wikipedia dataset for any language

Comment:
I am able to start downloading the dataset when trying anything with the recent dumps for 20221201. But obviously, those are the big wiki dumps and I need the smaller preloaded version.

I am now getting some error when the files show up in my cache but it will say FileNotFoundError at the end of the download for some reason. The cache directory to the datasets\wikipedia\date.bn\ had something in it, then when the error came up it disappeared. 

It is easy to test with the langauge "bn" because the amount of files is low.

dataset = load_dataset('wikipedia', date="20221201", language="bn", split='train', beam_runner='DirectRunner')
Issue: datasets map() handles all data at a stroke and takes long time

Comment:
Hi ! Interesting question ;)

> Which is better? Process in map() or in data-collator

As you said, both can be used in practice: map() if you want to preprocess before training, or a data-collator (or the equivalent `dataset.set_transform`) if you want to preprocess on-the-fly during training. Both options are great and really depend on your case.

To choose between the two, here are IMO the main caveats of each approach:
- if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
- on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

> Why huggingface advises map() function? There should be some advantages to using map()

To get the best throughput when training a model, it is often recommended to preprocess your dataset before training. Note that preprocessing may include other steps before tokenization such as data filtering, cleaning, chunking etc. which are often done before training.
Issue: datasets map() handles all data at a stroke and takes long time

Comment:
Thanks for your clear explanation @lhoestq ! 
> *  if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
> * on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

I really agree with you. There should be some trade-off between processing before and during the train loop.
Besides, I find `map()` function can cache the results once it has been executed. Very useful!
Issue: datasets map() handles all data at a stroke and takes long time

Comment:
@lhoestq How to preprocess on-the-fly during training？my data is about 1w hours, when I use map to  preprocess, and It's not finished yet， but all disk space(2T) is full.
Issue: [Tests] Ensure `datasets` supports renamed repositories

Comment:
You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
I am getting similar error - `TypeError: type object got multiple values for keyword argument 'name'` while following this [tutorial](https://huggingface.co/docs/datasets/dataset_script#create-a-dataset-loading-script). I am getting this error with the `dataset-cli test` command.

`datasets` version: 2.4.0
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
In my case, this was happening because I defined multiple `BuilderConfig` for multiple types, but didn't had all the data files that are requierd by those configs. 

I think this is different than the original issue by @bablf .
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
Hi ! I think this can be fixed by letting the config_kwargs take over the builder kwargs here:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/load.py#L1533-L1534

maybe something like this ?
```python
    **{**builder_kwargs, **config_kwargs}
```

Let me know if you'd like to contribute and fix this bug, so I can assign you :)

> In my case, this was happening because I defined multiple BuilderConfig for multiple types, but didn't had all the data files that are requierd by those configs.
> 
> I think this is different than the original issue by @bablf .

Feel free to to open an new issue, I'd be happy to help
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
Cool thank you ! Let me know if you have questions or if I can help
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
@lhoestq On second thoughts, I think this might be expected behavior; although a better error message might help.

Reasoning: Given n configs, if no data file is provided for any config, then it should be an error. Then why it should not be the case if out of n configs, for some data files are provided but not for others. Also, I was using `--all_configs` flag with `dataset-cli test`.
Issue: Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

Comment:
Ok I see - maybe we should check the values of builder_kwargs raise an error if any key in config_kwargs tries to overwrite it ? The builder kwargs are determined from the builder's type and location (in some cases it forces the base_path, data_files and config name for example)
Issue: None Type error for swda datasets

Comment:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?
Issue: None Type error for swda datasets

Comment:
Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.
Issue: Can't import datasets AttributeError: partially initialized module 'datasets' has no attribute 'utils' (most likely due to a circular import)

Comment:
Thanks for reporting, @OPterminator.

However, we are not able to reproduce this issue.

There might be 2 reasons why you get this exception:
- Either the name of your local Python file: if it is called  `datasets.py` this could generate a circular import when trying to import the Hugging Face `datasets` library.
  - You could try to rename it and run it again.
- Another cause could be the simultaneous use of the packages `nlp` and `datasets`. Please note that we renamed the Hugging Face `nlp` library to `datasets` more than 2 years ago: they are 2 versions of the same library.
  - Please try to update your script and use only `datasets` (`nlp` name is no longer in use and is out of date).