SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
Hi ! Could it be because you need to free the memory used by `tarfile` by emptying the tar `members` by any chance ?
```python
        yield key, {"audio": {"path": audio_name, "bytes": audio_file_obj.read()}}
        audio_tarfile.members = []  # free memory
        key += 1
```

and then you can set `DEFAULT_WRITER_BATCH_SIZE` to whatever value makes more sense for your dataset.

Let me know if the issue persists (which could happen, given that you managed to run your generator without RAM issues and using os.walk didn't solve the issue)
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I also run out of memory when loading `mozilla-foundation/common_voice_8_0` that also uses `tarfile` via `dl_manager.iter_archive`. There seems to be some data files that stay in memory somewhere

I don't have the issue with other compression formats like gzipped files
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I'm facing a similar memory leak issue when loading cv8. As you said @lhoestq 

`load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True, writer_batch_size=1)`

This issue is happening on a 32GB RAM machine. 

Any updates on how to fix this?
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I've run a memory profiler to see where's the leak comes from:

![image](https://user-images.githubusercontent.com/5097052/165101712-e7060ae5-77b2-4f6a-92bd-2996dbd60b36.png)

... it seems that it's related to the tarfile lib buffer reader. But I don't know why it's only happening on the huggingface script
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I have the same problem when loading video into numpy. 
```
yield id,{ 
    "video": imageio.v3.imread(video_path),
    "label": int(label)
}
```
Since video files are heavy, it can only processes a dozen samples before OOM.
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
For video datasets I think you can just define the max number of video that can stay in memory by adding this class attribute to your dataset builer:
```py
DEFAULT_WRITER_BATCH_SIZE = 8  # only 8 videos at a time in memory before flushing the dataset writer
```
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
same thing happens for me with `load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True, writer_batch_size=1)` on azure ml. seems to fill up `tmp` and not release that memory until OOM
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I'll add that I'm encountering the same issue with
`load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train')`.
Same for `'es'` in place of `'ceb'`.
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
> I'll add that I'm encountering the same issue with
> load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train').
> Same for 'es' in place of 'ceb'.

This is because the Apache Beam `DirectRunner` runs with the full data in memory unfortunately. Optimizing the `DirectRunner` is not in the scope of the `datasets` library, but rather in the Apache Beam project I believe. If you have memory issues with the `DirectRunner`, please consider switching to a machine with more RAM, or to distributed processing runtimes like Spark, Flink or DataFlow. There is a bit of documentation here: https://huggingface.co/docs/datasets/beam
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
> > I'll add that I'm encountering the same issue with
> > `load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train')`.
> > Same for `'es'` in place of `'ceb'`.
> 
> This is because the Apache Beam `DirectRunner` runs with the full data in memory unfortunately. Optimizing the `DirectRunner` is not in the scope of the `datasets` library, but rather in the Apache Beam project I believe. If you have memory issues with the `DirectRunner`, please consider switching to a machine with more RAM, or to distributed processing runtimes like Spark, Flink or DataFlow. There is a bit of documentation here: https://huggingface.co/docs/datasets/beam

Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM! I have also tried with `Runner='Flink'` on an environment with 51GB of RAM, which also failed.

Apache Beam has tons of open tickets already - is it worth submitting one to them over this?
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
> Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM!

What, wikipedia is not even bigger than 20GB

cc @albertvillanova
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
> > Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM!
> 
> What, wikipedia is not even bigger than 20GB
> 
> cc @albertvillanova

Luckily, on Colab you can watch the call stack at the bottom of the screen - much of the time and space complexity seems to come from `_parse_and_clean_wikicode()` rather than the actual download process. As far as I can tell, the script is loading the full dataset and then cleaning it all at once, which is consuming a lot of memory.
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I think we are mixing many different bugs in this Issue page:
- TAR archive with audio files
- video file
- distributed parsing of Wikipedia using Apache Beam

@dan-the-meme-man may I ask you to open a separate Issue for your problem? Then I will address it. It is important to fix it because we are currently working on a Datasets enhancement to be able to provide all Wikipedias already preprocessed.

On the other hand, I think we could keep this Issue page for the original problem: TAR archive with audio files. That is not fixed yet either.
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
Is there an update on the TAR archive issue with audio files? Happy to lend a hand in fixing this :)
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I found the issue with Common Voice 8 and opened a PR to fix it: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/discussions/2

Basically the `metadata` dict that contains the transcripts per audio file was continuously getting filled with bytes from `f.read()` because of this code:
```python
result = metadata[path]
result["audio"] = {"path": path, "bytes": f.read()}
```
copying the result with `result = dict(metadata[path])` fixes it: the bytes are no longer added to `metadata`

I also opened PRs to the other CV datasets
Issue: `load_dataset` consumes too much memory for audio + tar archives

Comment:
I'm closing this one for now, but feel free to reopen if you encounter other memory issues with audio datasets
Issue: Unexpected behavior of _TempDirWithCustomCleanup

Comment:
Hi ! Would setting TMPDIR at the beginning of your python script/session work ? I mean, even before importing transformers, datasets, etc. and using them ? I think this would be the most robust solution given any library that uses `tempfile`. I don't think we aim to support environment variables to be changed at run time
Issue: Unexpected behavior of _TempDirWithCustomCleanup

Comment:
Hi, yeah setting the environment variable before the imports / as environment variable outside is another way to fix this. I am just arguing that `datasets` already uses its own global variable to track temporary files: `_TEMP_DIR_FOR_TEMP_CACHE_FILES`, and the creation of this global variable should respect TMPDIR instead of relying on tempfile to do so.
Issue: metric = metric_cls( TypeError: 'NoneType' object is not callable

Comment:
Hi @klyuhang9,

I'm sorry but I can't reproduce your problem:
```python
In [2]: metric = load_metric('glue', 'rte')
Downloading builder script: 5.76kB [00:00, 2.40MB/s]
```

Could you please, retry to load the metric? Sometimes there are temporary connectivity issues.

Feel free to re-open this issue of the problem persists.
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/glue/glue.py

Comment:
Hi @klyuhang9,

I'm sorry but I can't reproduce your problem:
```python
In [4]: ds = load_dataset("glue", "sst2", download_mode="force_redownload")
Downloading builder script: 28.8kB [00:00, 9.15MB/s]                                                                                                                                                       
Downloading metadata: 28.7kB [00:00, 10.7MB/s]                                                                                                                                                             
Downloading and preparing dataset glue/sst2 (download: 7.09 MiB, generated: 4.78 MiB, post-processed: Unknown size, total: 11.88 MiB) to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.44M/7.44M [00:01<00:00, 4.12MB/s]
Dataset glue downloaded and prepared to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.  
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1047.96it/s]

In [5]: ds
Out[5]: 
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
})
```

Please, note that sometimes GitHub has some temporary connectivity issues. Feel free to retry and re-open this issue if the problem persists.
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/glue/glue.py

Comment:
> Are you able to access the URL in your web browser?

Yes, with or without a VPN, we (people in China) can access the URL. And we can even use wget to download these files. We can download the pretrained language model automatically with the code.
However, we CANNOT access glue.py & metric.py automatically. Every time, it will raise ConnectionError, and we have to download datasets manually (SQuAD is extremely hard to preprocess) and replace metric.py with scipy.metrics. If this problem is solved, many Chinese will save a lot of time.
Issue: ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/glue/glue.py

Comment:
> ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/glue/glue.py
> 
> I don't know why; it is ok when I use

If you would query the question `ConnectionError: Couldn't reach` in www.baidu.com (Chinese Google, Google is banned and some people cannot access it), you will find that there are so many questions about accessing `https://raw.githubusercontent.com`. There are some solutions like adding `185.199.108.133 raw.githubusercontent.com` to `C:/windows/systen32/drives/etc/hosts`, but it is time-consuming, hard for green-hand, and invalid sometimes.
Issue: Split size error on `amazon_us_reviews` / `PC_v1_00` dataset

Comment:
Follow-up: I have confirmed there are no duplicate lines via `sort amazon_reviews_us_PC_v1_00.tsv | uniq -cd` after extracting the raw file.
Issue: Split size error on `amazon_us_reviews` / `PC_v1_00` dataset

Comment:
Hi @trentonstrong, thanks for reporting!

I confirm that loading this dataset configuration throws a `NonMatchingSplitsSizesError`:
```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=350242049, num_examples=785730, dataset_name='amazon_us_reviews'), 'recorded': SplitInfo(name='train', num_bytes=3982712078, num_examples=6908554, dataset_name='amazon_us_reviews')}]
```

Also thank you for your offer to fix this. You can find information about how to update the metadata JSON file here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
```shell
datasets-cli test datasets/amazon_us_reviews --save_infos --all_configs
```
Please, feel free to open a PR with this fix. And do not hesitate to ping me if you need any help.
Issue: Dataset.unique(column: str) -> ArrowNotImplementedError

Comment:
Hi @orkenstein, thanks for reporting.

Please note that for this case, our `datasets` library uses under the hood the Apache Arrow `unique` function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique

And currently the Apache Arrow `unique` function is only implemented for these input types (see info in their [docs](https://arrow.apache.org/docs/cpp/compute.html#array-wise-vector-functions)): Boolean, Null, Numeric, Temporal, Binary- and String-like.

However, the data types of the `wikiann` dataset are all `list<item: string>` (see its [dataset card](https://huggingface.co/datasets/wikiann#data-fields)), and thus, not yet supported by the Apache Arrow `unique` function.
Issue: Dataset.unique(column: str) -> ArrowNotImplementedError

Comment:
As a workaround solution you can use pandas:
```python
from datasets import load_dataset

dataset = load_dataset('wikiann', 'en', split='train')
df = dataset.to_pandas()
unique_df = df[~df.tokens.apply(tuple).duplicated()]  # from https://stackoverflow.com/a/46958336/17517845
```

Note that pandas loads the dataset in memory (this one is small so it's fine).
Issue: Add support for IIIF in datasets 

Comment:
Hi! Thanks for the detailed analysis of adding IIIF support. I like the idea of "using IIIF through datasets scripts" due to its ease of use. Another approach that I like is yielding image ids and using the `piffle` library (which offers a bit more flexibility) + `map` to download + cache images. We can handle bad URLs in `map`  by returning `None`. Plus, we can add a `Dataset Preprocessing` section with the code that explains this approach to the card of such datasets. WDYT?

> currently, IIIF is mainly used by cultural heritage organizations (museums, archives etc.) The adoption of IIIF in this sector has been growing but it's possible that adoption won't be extended to other industries which may also be a source of image data for training ML models.

This is why (currently) adding a new feature type would be overkill, IMO.
Issue: Error while building documentation

Comment:
After some investigation, maybe the bug is in `doc-builder`.

I've opened an issue there:
- huggingface/doc-builder#160
Issue: Cannot load the dataset conll2012_ontonotesv5

Comment:
Hi @cathyxl, thanks for reporting.

Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists.
Issue: Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

Comment:
Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```
Issue: Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

Comment:
wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?
Issue: Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

Comment:
`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index` 

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index
Issue: ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
Hi, @MoritzLaurer, thanks for reporting.

Normally this is due to a mismatch between the versions of your Elasticsearch client and server:
- your ES client is passing only keyword arguments to your ES server
- whereas your ES server expects a positional argument called 'scheme'

In order to fix this, you should align the major versions of both Elasticsearch client and server.

You can have more info:
- on this other issue page: https://github.com/huggingface/datasets/issues/3956#issuecomment-1072115173
- Elasticsearch client docs: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/overview.html#_compatibility

Feel free to re-open this issue if the problem persists.

Duplicate of:
- #3956
Issue: ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
1. Check elasticsearch version
```
import elasticsearch
print(elasticsearch.__version__)
```
Ex: 7.9.1
2. Uninstall current elasticsearch package
`pip uninstall elasticsearch`
3. Install elasticsearch 7.9.1 package
`pip install elasticsearch==7.9.1`
Issue: Can not correctly parse the classes with imagefolder

Comment:
I found that the problem arises because the image files in my folder are actually symbolic links (for my own reasons).  After modifications, the classes can now be correctly parsed. Therefore, I close this issue.
Issue: Cannot preview "hazal/Turkish-Biomedical-corpus-trM"

Comment:
Hi @hazalturkmen, thanks for reporting.

Note that your dataset repository does not contain any loading script; it only contains a data file named `tr_article_2`.

When there is no loading script but only data files, the `datasets` library tries to infer how to load the data by looking at the data file extensions. However, your data file does not have any extension.

Note that current supported data file extensions are: 'csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip'.

You have more info on our docs: [How to share a dataset](https://huggingface.co/docs/datasets/share).
Issue: set_format does not work with multi dimension tensor

Comment:
Hi! Use the `ArrayXD` feature type (where X is the number of dimensions) to get correctly formated tensors. So in your case, define the dataset as follows :
```python
ds = Dataset.from_dict({"A": [torch.rand((2, 2))]}, features=Features({"A": Array2D(shape=(2, 2), dtype="float32")}))
```
Issue: set_format does not work with multi dimension tensor

Comment:
Hi @mariosasko I'm facing the same issue and the only work around I've found so far is to convert my `DatasetDict` to a dictionary and then create new objects with `Dataset.from_dict`.
```
dataset = load_dataset("my_dataset.py")
dataset = dataset.map(lambda example: blabla(example))
dict_dataset_test = dataset["test"].to_dict()
...
dataset_test = Dataset.from_dict(dict_dataset_test, features=Features(features))
```
However, converting a `Dataset` object to a dict takes quite a lot of time and memory... Is there a way to directly create an `Array2D` without having to transform the original `Dataset` to a dict?
Issue: set_format does not work with multi dimension tensor

Comment:
Hi! Yes, you can directly pass the `Features` dictionary as `features` in `map` to cast the column to `Array2D`:
```python
dataset = dataset.map(lambda example: blabla(example), features=Features(features))
```
Or you can use `cast` after `map` to do that:
```python
dataset = dataset.map(lambda example: blabla(example))
dataset = dataset.cast(Features(features))
```
Issue: Yelp not working

Comment:
Yet another issue related to google drive not being nice. Most likely your IP has been banned from using their API programmatically. Do you know if we are allowed to host and redistribute the data ourselves ?
Issue: Yelp not working

Comment:
Hi,

Facing the same issue while loading the dataset: 

`Error: {NonMatchingChecksumError}Checksums didn't match for dataset source files`

Thanks
Issue: Yelp not working

Comment:
> Facing the same issue while loading the dataset:
> 
> Error: {NonMatchingChecksumError}Checksums didn't match for dataset source files

Thanks for reporting. I think this is the same issue. Feel free to try again later, once Google Drive stopped blocking you. You can retry by passing `download_mode="force_redownload"` to `load_dataset`
Issue: Yelp not working

Comment:
I noticed that FastAI hosts the Yelp dataset at https://s3.amazonaws.com/fast-ai-nlp/yelp_review_full_csv.tgz (from their catalog [here](https://course.fast.ai/datasets))

Let's update the yelp dataset script to download from there instead of Google Drive
Issue: Yelp not working

Comment:
I updated the link to not use Google Drive anymore, we will do a release early next week with the updated download url of the dataset :)
Issue: ASSIN2 dataset checksum bug

Comment:
That's true. Steps to reproduce the bug on Google Colab:

```
git clone https://github.com/huggingface/datasets.git
cd datasets
pip install -e .
python -c "from datasets import load_dataset; print(load_dataset('assin2')['train'][0])"
```

However the dataset will load without any problems if you just install version 2.0.0:

 ```
pip install datasets
python -c "from datasets import load_dataset; print(load_dataset('assin2')['train'][0])"
```

Any thoughts @lhoestq ?
Issue: ASSIN2 dataset checksum bug

Comment:
Right indeed ! Let me open a PR to fix this.
The dataset_infos.json file that stores some metadata about the dataset to download (and is used to verify it was correctly downloaded) hasn't been updated correctly
Issue: ASSIN2 dataset checksum bug

Comment:
Not sure what the status of this is, but personally I am still getting this error, with glue.
Issue: How to use generate this multitask dataset for SQUAD? I am getting a value error. 

Comment:
Hi! Replacing `nlp.<obj>` with `datasets.<obj>` in the script should fix the problem. `nlp` has been renamed to `datasets` more than a year ago, so please use `datasets` instead to avoid weird issues.
Issue: How to use generate this multitask dataset for SQUAD? I am getting a value error. 

Comment:
But I request you to please fix the same in the dataset hub explorer as well...