SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: to_tf_dataset keeps a reference to the open data somewhere, causing issues on windows

Comment:
I did some investigation and, as it seems, the bug stems from [this line](https://github.com/huggingface/datasets/blob/8004d7c3e1d74b29c3e5b0d1660331cd26758363/src/datasets/arrow_dataset.py#L325). The lifecycle of the dataset from the linked line is bound to one of the returned `tf.data.Dataset`. So my (hacky) solution involves wrapping the linked dataset with `weakref.proxy` and adding a custom `__del__` to `tf.python.data.ops.dataset_ops.TensorSliceDataset` (this is the type of a dataset that is returned by `tf.data.Dataset.from_tensor_slices`; this works for TF 2.x, but I'm not sure `tf.python.data.ops.dataset_ops` is a valid path for TF 1.x) that deletes the linked dataset, which is assigned to the dataset object as a property. Will open a draft PR soon!
Issue: "File name too long" error for file locks

Comment:
Hi, the filename here is less than 255
```python
>>> len("_home_garrett_.cache_huggingface_datasets_csv_test-7c856aea083a7043_0.0.0_9144e0a4e8435090117cea53e6c7537173ef2304525df4a077c435d8ee7828ff.incomplete.lock")
154
```
so not sure why it's considered too long for your filesystem.
(also note that the lock files we use always have smaller filenames than 255)

https://github.com/huggingface/datasets/blob/5d1a9f1e3c6c495dc0610b459e39d2eb8893f152/src/datasets/utils/filelock.py#L135-L135
Issue: "File name too long" error for file locks

Comment:
Yes, you're right! I need to get you more info here. Either there's something going with the name itself that the file system doesn't like (an encoding that blows up the name length??) or perhaps there's something with the path that's causing the entire string to  be used as a name. I haven't seen this on any system before and the Internet's not forthcoming with any info.
Issue: "File name too long" error for file locks

Comment:
Perhaps this could be exposed as a config setting so you could change it manually?

https://github.com/huggingface/datasets/blob/5d1a9f1e3c6c495dc0610b459e39d2eb8893f152/src/datasets/utils/filelock.py#L135-L135

Rather than hard-code 255, default it to 255, and allow it to be changed, the same way is done for `datasets.config.IN_MEMORY_MAX_SIZE`:

https://github.com/huggingface/datasets/blob/12b7e13bc568b9f92705f64b249e148f3bc9a9ea/src/datasets/config.py#L171-L173

In fact there already appears to be an existing variable to do so:

https://github.com/huggingface/datasets/blob/12b7e13bc568b9f92705f64b249e148f3bc9a9ea/src/datasets/config.py#L187

It's used here:

https://github.com/huggingface/datasets/blob/efe89edd36e4ffa562fc3eebaf07a5fec26e6dac/src/datasets/builder.py#L163-L165

Perhaps it could be set based on a test (trying to create a 255 char length named lock file and seeing if it fails)
Issue: "File name too long" error for file locks

Comment:
Hi @lmmx @gar1t ,

it would be helpful if you could run the following code and copy-paste the output here:
```python
import datasets
import os
os.statvfs(datasets.config.HF_DATASETS_CACHE)
```
Issue: "File name too long" error for file locks

Comment:
Hi @lmmx,

Thanks for providing the result of the command. I've opened a PR, and it would be great if you could verify that the fix works on your system. To install the version of the datasets with the fix, please run the following command:
```
pip install git+https://github.com/huggingface/datasets.git@fix-2924
```

Btw, I saw your PR, and I appreciate your effort. However, my approach is a bit simpler for the end-user, so that's why I decided to fix the issue myself.
Issue: "File name too long" error for file locks

Comment:
No problem Mario I didn't know that was where that value was recorded so I learnt something :smiley: I just wanted to get a local version working, of course you should implement whatever fix is best for HF. Yes can confirm this fixes it too. Thanks!
Issue: `Can not decode content-encoding: gzip` when loading `scitldr` dataset with streaming

Comment:
Hi @SBrandeis, thanks for reporting! ^^

I think this is an issue with `fsspec`: https://github.com/intake/filesystem_spec/issues/389

I will ask them if they are planning to fix it...
Issue: `Can not decode content-encoding: gzip` when loading `scitldr` dataset with streaming

Comment:
Code to reproduce the bug: `ClientPayloadError: 400, message='Can not decode content-encoding: gzip'`
```python
In [1]: import fsspec

In [2]: import json

In [3]: with fsspec.open('https://raw.githubusercontent.com/allenai/scitldr/master/SciTLDR-Data/SciTLDR-FullText/test.jsonl', encoding="utf-8") as f:
   ...:     for row in f:
   ...:         data = json.loads(row)
   ...:
---------------------------------------------------------------------------
ClientPayloadError                        Traceback (most recent call last)
```
Issue: windows download abnormal

Comment:
Hi ! Is there some kind of proxy that is configured in your browser that gives you access to internet ? If it's the case it could explain why it doesn't work in the code, since the proxy wouldn't be used
Issue: timit_asr dataset only includes one text phrase

Comment:
Hi @margotwagner, 
This bug was fixed in #1995. Upgrading the datasets should work (min v1.8.0 ideally)
Issue: timit_asr dataset only includes one text phrase

Comment:
Hi @margotwagner,

Yes, as @bhavitvyamalik has commented, this bug was fixed in `datasets` version 1.5.0. You need to update it, as your current version is 1.4.1:
> Environment info
> - `datasets` version: 1.4.1
Issue: FORCE_REDOWNLOAD does not work

Comment:
Hi ! Thanks for reporting. The error seems to happen only if you use compressed files.

The second dataset is prepared in another dataset cache directory than the first - which is normal, since the source file is different. However, it doesn't uncompress the new data file because it finds the old uncompressed data in the extraction cache directory.

If we fix the extraction cache mechanism to uncompress a local file if it changed then it should fix the issue.
Currently the extraction cache mechanism only takes into account the path of the compressed file, which is an issue.
Issue: FORCE_REDOWNLOAD does not work

Comment:
Facing the same issue, is there any way to overtake this issue until it will be fixed?
Issue: Add WIT Dataset

Comment:
WikiMedia is now hosting the pixel values directly which should make it a lot easier!
The files can be found here:
https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/
https://analytics.wikimedia.org/published/datasets/one-off/caption_competition/training/image_pixels/
Issue: Add WIT Dataset

Comment:
> @hassiahk is working on it #2810

Thank you @bhavitvyamalik! Added this issue so we could track progress 😄 . Just linked the PR as well for visibility.
Issue: Add WIT Dataset

Comment:
Hey folks, we are now hosting the merged pixel values + embeddings + metadata ourselves. I gave it a try - [nateraw/wit](https://huggingface.co/datasets/nateraw/wit)

**⚠️ - Make sure you add `streaming=True` unless you're prepared to download 400GB of data!**

```python
from datasets import load_dataset

ds = load_dataset('nateraw/wit', streaming=True)
example = next(iter(ds))
```

```python
>>> example = next(iter(ds['train']))
>>> example.keys()
dict_keys(['b64_bytes', 'original_width', 'image_url', 'wit_features', 'original_height', 'metadata_url', 'mime_type', 'caption_attribution_description', 'embedding'])
>>> example['wit_features'].keys()
dict_keys(['hierarchical_section_title', 'language', 'attribution_passes_lang_id', 'context_section_description', 'is_main_image', 'page_title', 'caption_title_and_reference_description', 'caption_alt_text_description', 'caption_reference_description', 'page_url', 'context_page_description', 'section_title', 'page_changed_recently'])
```
Issue: Add WIT Dataset

Comment:
Hi! `datasets` now hosts two versions of the WIT dataset:
* [`google/wit`](https://huggingface.co/datasets/google/wit): Google's version with the image URLs
* [`wikimedia/wit_base`](https://huggingface.co/datasets/wikimedia/wit_base): Wikimedia's version with the images + ResNet embeddings, but with less data than Google's version
Issue: Incompatibility with pytest

Comment:
Sorry, my bad... When implementing `xpathopen`, I just considered the use case in the COUNTER dataset... I'm fixing it!
Issue: Error when encoding a dataset with None objects with a Sequence feature

Comment:
This has been fixed by https://github.com/huggingface/datasets/pull/2900
We're doing a new release 1.12 today to make the fix available :)
Issue: v1.11.1 release date

Comment:
@albertvillanova i think this issue is still valid and should not be closed till `>1.11.0` is published :)
Issue: Adding an Elastic Search index to a Dataset

Comment:
Hi, is this bug deterministic in your poetry env ? I mean, does it always stop at 90% or is it random ?

Also, can you try using another version of Elasticsearch ? Maybe there's an issue with the one of you poetry env
Issue: Adding an Elastic Search index to a Dataset

Comment:
I face similar issue with oscar dataset on remote ealsticsearch instance. It was mainly due to timeout of batch indexing requests and I solve these by adding large request_timeout param in `search.py`

```
       for ok, action in es.helpers.streaming_bulk(
            client=self.es_client,
            index=index_name,
            actions=passage_generator(),
            request_timeout=3600,
        )
        ```
Issue: Adding an Elastic Search index to a Dataset

Comment:
Hi @MotzWanted - are there any errors in the Elasticsearch cluster logs? Since it works in your local environment and the cluster versions are different between your poetry env and your local env, it is possible that it is some difference in the cluster - either settings or the cluster being under a different load etc that has this effect, so it would be useful to see if any errors are thrown in the cluster's logs when you try to ingest. 
Which elasticsearch client method is the function `add_elasticsearch_index` from your code using under the hood? Is it `helpers.bulk` or is the indexing performed using something else? You can try adding a timeout to the indexing method to see if this helps. Also, you mention that it stops at around 90%  - do you know if the timeout/hanging happens always when a particular document is being indexed or does it happen randomly at around 90% completeness but on different documents?
Issue: `load_dataset('docred')` results in a `NonMatchingChecksumError` 

Comment:
Hi @tmpr, thanks for reporting.

Two weeks ago (23th Aug), the host of the source `docred` dataset updated one of the files (`dev.json`): you can see it [here](https://drive.google.com/drive/folders/1c5-0YwnoJx8NS6CV2f-NoTHR__BdkNqw).

Therefore, the checksum needs to be updated.

Normally, in the meantime, you could avoid the error by passing `ignore_verifications=True` to `load_dataset`. However, as the old link points to a non-existing file, the link must be updated too.

I'm fixing all this.
Issue: In v1.4.1, all TIMIT train transcripts are "Would such an act of refusal be useful?"

Comment:
Hi @rcgale, thanks for reporting.

Please note that this bug was fixed on `datasets` version 1.5.0: https://github.com/huggingface/datasets/commit/a23c73e526e1c30263834164f16f1fdf76722c8c#diff-f12a7a42d4673bb6c2ca5a40c92c29eb4fe3475908c84fd4ce4fad5dc2514878

If you update `datasets` version, that should work.

On the other hand, would it be possible for @patrickvonplaten to update the [blog post](https://huggingface.co/blog/fine-tune-wav2vec2-english) with the correct version of `datasets`?
Issue: In v1.4.1, all TIMIT train transcripts are "Would such an act of refusal be useful?"

Comment:
I just proposed a change in the blog post.

I had assumed there was a data format change that broke a previous version of the code, since presumably @patrickvonplaten tested the tutorial with the version they explicitly referenced. But that fix you linked suggests a problem in the code, which surprised me.

I still wonder, though, is there a way for downloads to be invalidated server-side? If the client can announce its version during a download request, perhaps the server could reject known incompatibilities? It would save much valuable time if `datasets` raised an informative error on a known problem ("Error: the requested data set requires `datasets>=1.5.0`."). This kind of API versioning is a prudent move anyhow, as there will surely come a time when you'll need to make a breaking change to data.
Issue: Don't keep the dummy data folder or dataset_infos.json when resolving data files

Comment:
Hi @lhoestq I am new to huggingface datasets, I would like to work on this issue!
Issue: Don't keep the dummy data folder or dataset_infos.json when resolving data files

Comment:
Thanks for the help :) 

As mentioned in the PR, excluding files named "dummy_data.zip" is actually more general than excluding the files inside a "dummy" folder. I just did the change in the PR, I think we can merge it now
Issue: datasets.config.PYARROW_VERSION has no attribute 'major'

Comment:
Hi @bwang482,

I'm sorry but I'm not able to reproduce your bug.

Please note that in our current master branch, we made a commit (d03223d4d64b89e76b48b00602aba5aa2f817f1e) that simultaneously modified:
- test_dataset_common.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-a1bc225bd9a5bade373d1f140e24d09cbbdc97971c2f73bb627daaa803ada002L289 that introduces the usage of `datasets.config.PYARROW_VERSION.major`
- but also changed config.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-e021fcfc41811fb970fab889b8d245e68382bca8208e63eaafc9a396a336f8f2L40, so that `datasets.config.PYARROW_VERSION.major` exists
Issue: datasets.config.PYARROW_VERSION has no attribute 'major'

Comment:
Reopening this. Although the `test_dataset_common.py` script works fine now.

Has this got something to do with my pull request not passing `ci/circleci: run_dataset_script_tests_pyarrow` tests?

https://github.com/huggingface/datasets/pull/2873
Issue: datasets.config.PYARROW_VERSION has no attribute 'major'

Comment:
Hi @bwang482,

If you click on `Details` (on the right of your non passing CI test names: `ci/circleci: run_dataset_script_tests_pyarrow`), you can have more information about the non-passing tests.

For example, for ["ci/circleci: run_dataset_script_tests_pyarrow_1" details](https://circleci.com/gh/huggingface/datasets/46324?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link), you can see that the only non-passing test has to do with the dataset card (missing information in the `README.md` file): `test_changed_dataset_card`
```
=========================== short test summary info ============================
FAILED tests/test_dataset_cards.py::test_changed_dataset_card[swedish_medical_ner]
= 1 failed, 3214 passed, 2874 skipped, 2 xfailed, 1 xpassed, 15 warnings in 175.59s (0:02:55) =
```

Therefore, your PR non-passing test has nothing to do with this issue.
Issue: TypeError: 'NoneType' object is not callable

Comment:
Hi, @Chenfei-Kang.

I'm sorry, but I'm not able to reproduce your bug:
```python
from datasets import load_dataset

ds = load_dataset("glue", 'cola')
ds
```
```
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 8551
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1043
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1063
    })
})
```

Could you please give more details and environment info (platform, PyArrow version)?
Issue: TypeError: 'NoneType' object is not callable

Comment:
> Hi, @Chenfei-Kang.
> 
> I'm sorry, but I'm not able to reproduce your bug:
> 
> ```python
> from datasets import load_dataset
> 
> ds = load_dataset("glue", 'cola')
> ds
> ```
> 
> ```
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 8551
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 1043
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 1063
>     })
> })
> ```
> 
> Could you please give more details and environment info (platform, PyArrow version)?

Sorry to reply you so late.
platform: pycharm 2021 + anaconda with python 3.7
PyArrow version: 5.0.0
huggingface-hub: 0.0.16
datasets: 1.9.0
Issue: TypeError: 'NoneType' object is not callable

Comment:
- For the platform, we need to know the operating system of your machine. Could you please run the command `datasets-cli env` and copy-and-paste its output below?
- In relation with the error, you just gave us the error type and message (`TypeError: 'NoneType' object is not callable`). Could you please copy-paste the complete stack trace, so that we know exactly which part of the code threw the error?
Issue: TypeError: 'NoneType' object is not callable

Comment:
> * For the platform, we need to know the operating system of your machine. Could you please run the command `datasets-cli env` and copy-and-paste its output below?
> * In relation with the error, you just gave us the error type and message (`TypeError: 'NoneType' object is not callable`). Could you please copy-paste the complete stack trace, so that we know exactly which part of the code threw the error?

1. For the platform, here are the output:
        - datasets` version: 1.11.0
        - Platform: Windows-10-10.0.19041-SP0
        - Python version: 3.7.10
        - PyArrow version: 5.0.0
2. For the code and error：
     ```python
     from datasets import load_dataset, load_metric
     dataset = load_dataset("glue", "cola")
    ```
    ```python
    Traceback (most recent call last):
    ....
    ....
    File "my_file.py", line 2, in <module>
    dataset = load_dataset("glue", "cola")
    File "My environments\lib\site-packages\datasets\load.py", line 830, in load_dataset
    **config_kwargs,
     File "My environments\lib\site-packages\datasets\load.py", line 710, in load_dataset_builder
    **config_kwargs,
    TypeError: 'NoneType' object is not callable
    ```
   Thank you!
Issue: TypeError: 'NoneType' object is not callable

Comment:
For that environment, I am sorry but I can't reproduce the bug: I can load the dataset without any problem.
Issue: TypeError: 'NoneType' object is not callable

Comment:
One naive question: do you have internet access from the machine where you execute the code?
Issue: TypeError: 'NoneType' object is not callable

Comment:
> For that environment, I am sorry but I can't reproduce the bug: I can load the dataset without any problem.

But I can download other task dataset such as `dataset = load_dataset('squad')`. I don't know what went wrong.  Thank you so much!
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
Hi @severo, thanks for reporting.

Just note that currently not all canonical datasets support streaming mode: this is one case!

All datasets that use `pathlib` joins (using `/`) instead of `os.path.join` (as in this dataset) do not support streaming mode yet.
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
OK. Do you think it's possible to detect this, and raise an exception (maybe `NotImplementedError`, or a specific `StreamingError`)?
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
We should definitely support datasets using `pathlib` in streaming mode...

For non-supported datasets in streaming mode, we have already a request of raising an error/warning: see #2654.
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
Hi @severo, please note that "counter" dataset will be streamable (at least until it arrives at the missing file, error already in normal mode) once these PRs are merged:
- #2874
- #2876
- #2880

I have tested it. 😉
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
Now (on master), we get:

```
import datasets as ds
ds.load_dataset('counter', split="train", streaming=False)
```

```
Using custom data configuration default
Downloading and preparing dataset counter/default (download: 1.29 MiB, generated: 2.48 MiB, post-processed: Unknown size, total: 3.77 MiB) to /home/slesage/.cache/huggingface/datasets/counter/default/1.0.0/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9...
Traceback (most recent call last):
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 726, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 1124, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/slesage/hf/datasets/.venv/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/counter/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9/counter.py", line 161, in _generate_examples
    with derived_file.open(encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 728, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'
```

The error is now the same with or without streaming. I close the issue, thanks @albertvillanova and @lhoestq!
Issue: "counter" dataset raises an error in normal mode, but not in streaming mode

Comment:
Note that we might want to open an issue to fix the "counter" dataset by itself, but I let it up to you.