SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Enum used in map functions will raise a RecursionError with dill.

Comment:
I have the same bug.
the code is as follows:
![image](https://user-images.githubusercontent.com/84262181/139785849-620dd4ac-86ce-4212-8163-942bbca305aa.png)
the error is: 
![image](https://user-images.githubusercontent.com/84262181/139785899-88a9bd75-c60b-45a5-b819-830c7c096f3d.png)

Look for the solution for this bug.
Issue: Enum used in map functions will raise a RecursionError with dill.

Comment:
Hi ! I think your RecursionError comes from a different issue @BitcoinNLPer , could you open a separate issue please ?

Also which dataset are you using ? I tried loading `CodedotAI/code_clippy` but I get a different error
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1615, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1446, in load_dataset_builder
    builder_cls = import_main_class(dataset_module.module_path)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 101, in import_main_class
    module = importlib.import_module(module_path)
  File "/Users/quentinlhoest/.virtualenvs/hf-datasets/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/quentinlhoest/.cache/huggingface/modules/datasets_modules/datasets/CodedotAI___code_clippy/d332f69d036e8c80f47bc9a96d676c3fa30cb50af7bb81e2d4d12e80b83efc4d/code_clippy.py", line 66, in <module>
    url_elements = results.find_all("a")
AttributeError: 'NoneType' object has no attribute 'find_all'
```
Issue: Support multi-worker with streaming dataset (IterableDataset).

Comment:
Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step
Issue: load_dataset("financial_phrasebank") NonMatchingChecksumError

Comment:
Hi! It's probably because this dataset is stored on google drive and it has a per day quota limit. It should work if you retry, I was able to initiate the download.

Similar issue [here](https://github.com/huggingface/datasets/issues/2646)
Issue: load_dataset("financial_phrasebank") NonMatchingChecksumError

Comment:
Hi ! Loading the dataset works on my side as well.
Feel free to try again and let us know if it works for you know
Issue: load_dataset("financial_phrasebank") NonMatchingChecksumError

Comment:
Thank you! I've been trying periodically for the past month, and no luck yet with this particular dataset. Just tried again and still hitting the checksum error.

Code:

`dataset = load_dataset("financial_phrasebank", "sentences_allagree") `

Traceback:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-55cc2144f31e> in <module>
----> 1 dataset = load_dataset("financial_phrasebank", "sentences_allagree")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    859         ignore_verifications=ignore_verifications,
    860         try_from_hf_gcs=try_from_hf_gcs,
--> 861         use_auth_token=use_auth_token,
    862     )
    863 

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    582                     if not downloaded_from_gcs:
    583                         self._download_and_prepare(
--> 584                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    585                         )
    586                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    642         if verify_infos:
    643             verify_checksums(
--> 644                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    645             )
    646 

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip']
```
Issue: Progress bars are not properly rendered in Jupyter notebook

Comment:
To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.
Issue: Progress bars are not properly rendered in Jupyter notebook

Comment:
Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Issue: Load datasets from the Hub without requiring a dataset script

Comment:
This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
Issue: Integration with AugLy

Comment:
Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```
Issue: Integration with AugLy

Comment:
Preprocessing functions/augmentations, unless super generic, should be defined in separate libraries, so I'm closing this issue.
Issue: `filelock.py` Error

Comment:
Hi @liyucheng09, thanks for reporting.

Apparently this issue has to do with your environment setup. One question: is your data in an NFS share? Some people have reported this error when using `fcntl` to write to an NFS share... If this is the case, then it might be that your NFS just may not be set up to provide file locks. You should ask your system administrator, or try these commands in the terminal:
```shell
sudo systemctl enable rpc-statd
sudo systemctl start rpc-statd
```
Issue: Jsonlines export error

Comment:
For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.
Issue: Jsonlines export error

Comment:
@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?
Issue: Jsonlines export error

Comment:
@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898
Issue: Jsonlines export error

Comment:
Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)
Issue: Jsonlines export error

Comment:
I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄
Issue: Streaming local gzip compressed JSON line files is not working

Comment:
Hi @thomwolf, thanks for reporting.

It seems this might be due to the fact that the JSON Dataset builder uses `pyarrow.json` (`paj.read_json`) to read the data without using the Python standard `open(file,...` (which is the one patched with `xopen` to work in streaming mode).

This has to be fixed.
Issue: Streaming local gzip compressed JSON line files is not working

Comment:
Sorry for reopening this, but I'm having the same issue as @thomwolf when streaming a gzipped JSON Lines file from the hub. Or is that just not possible by definition?
I installed `datasets`in editable mode from source (so probably includes the fix from #2608 ?): 
```
>>> datasets.__version__
'1.9.1.dev0'
```

```
>>> msmarco = datasets.load_dataset("webis/msmarco", "corpus", streaming=True)
Using custom data configuration corpus-174d3b7155eb68db
>>> msmarco_iter = iter(msmarco['train'])
>>> print(next(msmarco_iter))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 338, in __iter__
    for key, example in self._iter():
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 335, in _iter
    yield from ex_iterable
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 78, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/home/christopher/.cache/huggingface/modules/datasets_modules/datasets/msmarco/eb63dff8d83107168e973c7a655a6082d37e08d71b4ac39a0afada479c138745/msmarco.py", line 96, in _generate_examples
    with gzip.open(file, "rt", encoding="utf-8") as f:
  File "/usr/lib/python3.6/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'https://huggingface.co/datasets/webis/msmarco/resolve/main/msmarco_doc_00.gz'
```

Loading the dataset without streaming set to True, works fine.
Issue: Streaming local gzip compressed JSON line files is not working

Comment:
Hi ! To make the streaming work, we extend `open` in the dataset builder to work with urls.

Therefore you just need to use `open` before using `gzip.open`:
```diff
- with gzip.open(file, "rt", encoding="utf-8") as f:
+ with gzip.open(open(file, "rb"), "rt", encoding="utf-8") as f:
```

You can see that it is the case for oscar.py and c4.py for example:

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/oscar/oscar.py#L358-L358

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/c4/c4.py#L88-L88
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or 
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Also, if I delete the extracted files they need to be re-extracted again instead of loading from the Arrow cache files
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
I think we already opened an issue about this topic (suggested by @stas00): duplicated of #2481?

This is in our TODO list... 😅
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
I think the deletion of each extracted file could be implemented in our CacheManager and ExtractManager (once merged to master: #2295, #2277). 😉
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Nevermind @thomwolf, I just mentioned the other issue so that both appear linked in GitHub and we do not forget to close both once we make the corresponding Pull Request... That was the main reason! 😄
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant). 

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Note that I'm confirming that with the current master branch of dataset, deleting extracted files (without deleting the arrow cache file) lead to **re-extracting** these files when reloading the dataset instead of directly loading the arrow cache file.
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
Hi ! That's weird, it doesn't do that on my side (tested on master on my laptop by deleting the `extracted` folder in the download cache directory). You tested with one of the files at https://huggingface.co/datasets/thomwolf/github-python that you have locally ?
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
@thomwolf I'm sorry but I can't reproduce this problem. I'm also using: 
```python
ds = load_dataset("json", split="train", data_files=data_files, cache_dir=cache_dir)
```
after having removed the extracted files:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I get the logging message:
```shell
WARNING  datasets.builder:builder.py:531 Reusing dataset json ...
```
Issue: Add option to delete temporary files (e.g. extracted files) when loading dataset

Comment:
> 
> 
> Do you confirm the extracted folder stays empty after reloading?

Yes, I have the above mentioned assertion on the emptiness of the extracted folder:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```
Issue: Unable to download omp dataset

Comment:
Hi @erikadistefano , thanks for reporting the issue.

I have created a Pull Request that should fix it. 

Once merged into master, feel free to update your installed `datasets` library (either by installing it from our GitHub master branch or waiting until our next release) to be able to load omp dataset.
Issue: Transformer Class on dataset

Comment:
Hi ! Do you have an example in mind that shows how this could be useful ?
Issue: Transformer Class on dataset

Comment:
Example:

Merge 2 datasets into one datasets

Label extraction from dataset

dataset(text, label)
   —> dataset(text, newlabel)

TextCleaning.


For image dataset, 
Transformation are easier (ie linear algebra).






> On Jul 6, 2021, at 17:39, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> Hi ! Do you have an example in mind that shows how this could be useful ?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
Issue: Transformer Class on dataset

Comment:
There are already a few transformations that you can apply on a dataset using methods like `dataset.map()`.
You can find examples in the documentation here:
https://huggingface.co/docs/datasets/processing.html

You can merge two datasets with `concatenate_datasets()` or do label extraction with `dataset.map()` for example
Issue: Transformer Class on dataset

Comment:
Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to 
     Dataset Class


 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)


What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?


thanks









> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
> 
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
Issue: Transformer Class on dataset

Comment:
I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.

I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
Issue: Transformer Class on dataset

Comment:
Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?








> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
> 
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
Issue: Transformer Class on dataset

Comment:
We can be pretty flexible and not impose any constraints for transforms.

Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like `map` work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
Issue: Transformer Class on dataset

Comment:
Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)


The main issue is serialization 
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?




Maybe, not, since the output is aimed to feed model in memory (?)








> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> We can be pretty flexible and not impose any constraints for transforms.
> 
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
Issue: Transformer Class on dataset

Comment:
I'm not sure I understand, could you elaborate a bit more please ?

Each dataset is a wrapper of a PyArrow Table that contains all the data. The table is loaded from an arrow file on the disk.
We have an ArrowWriter and ArrowReader class to write/read arrow tables on disk or in in-memory buffers.
Issue: ModuleNotFoundError: No module named 'datasets.tasks' while importing common voice datasets

Comment:
Hi @profsatwinder.

It looks like you are using an old version of `datasets`. Please update it with `pip install -U datasets` and indicate if the problem persists.
Issue: ModuleNotFoundError: No module named 'datasets.tasks' while importing common voice datasets

Comment:
@albertvillanova Thanks for the information. I updated it to 1.9.0 and the issue is resolved. Thanks again.
Issue: Cached dataset overflowing disk space

Comment:
I'm using the datasets concatenate dataset to combine the datasets and then train.
train_dataset = concatenate_datasets([dataset1, dataset2, common_voice_train])
Issue: Cached dataset overflowing disk space

Comment:
Hi @BirgerMoell.

You have several options:
- to set caching to be stored on a different path location, other than the default one (`~/.cache/huggingface/datasets`):
  - either setting the environment variable `HF_DATASETS_CACHE` with the path to the new cache location
  - or by passing it with the parameter `cache_dir` when loading each of the datasets: `dataset = load_dataset(..., cache_dir=your_new_location)`

  You can get all the information in the docs: https://huggingface.co/docs/datasets/loading_datasets.html#cache-directory
- I wouldn't recommend disabling caching, because current implementation generates cache files anyway, although in a temporary directory and they are deleted when the session closes. See details here: https://huggingface.co/docs/datasets/processing.html#enable-or-disable-caching
- You could alternatively load the datasets in streaming mode. This is a new feature which allows loading the datasets without downloading the entire files. More information here: https://huggingface.co/docs/datasets/dataset_streaming.html
Issue: Cached dataset overflowing disk space

Comment:
Hi @BirgerMoell,

We are planning to add a new feature to datasets, which could be interesting in your case: Add the option to delete temporary files (decompressed files) from the cache directory (see: #2481, #2604).

We will ping you once this feature is implemented, so that the size of your cache directory will be considerably reduced.
Issue: sqaud_v2 dataset contains misalignment between the answer text and the context value at the answer index

Comment:
Hi @mmajurski, thanks for reporting this issue.

Indeed this misalignment arises because the source dataset context field contains leading blank spaces (and these are counted within the answer_start), while our datasets loading script removes these leading blank spaces.

I'm going to fix our script so that all leading blank spaces in the source dataset are kept, and there is no misalignment between the answer text and the answer_start within the context.