SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory

Comment:
Thanks for reporting, @jason-brian-anderson.

Yes, this is a known issue: the [CodeSearchNet](https://github.com/github/CodeSearchNet) repo has been archived (Apr 11, 2023) and their source data files are no longer accessible in their S3: e.g. https://s3.amazonaws.com/code-search-net/CodeSearchNet/v2/python.zip gives 403 Forbidden error. See:
- https://huggingface.co/datasets/code_search_net/discussions/3

We have contacted one of the authors of the dataset to find a solution. I'll keep you informed.

CC: @hamelsmu
Issue: datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory

Comment:
This issue is fixed because we are hosting the CodeSearchNet data files in the Hugging Face Hub. See: https://huggingface.co/datasets/code_search_net/discussions/7
Issue: datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory

Comment:
Thanks @hamelsmu for the Zenodo link. I am adding it to the dataset card on the Hugging Face Hub, so that the community knows about this "official" source data. I guess this link is not well known, because some community members already hosted an "unofficial" version of the data on Zenodo as well: https://zenodo.org/record/7857872
Issue: train_dataset does not implement __len__

Comment:
Thanks for reporting, @v-yunbin.

Could you please give more details, the steps to reproduce the bug, the complete error back trace and the environment information (`datasets-cli env`)?
Issue: train_dataset does not implement __len__

Comment:
this is a detail error info from transformers：
```
Traceback (most recent call last):
  File "finetune.py", line 177, in <module>
    whisper_finetune(traindir,devdir,outdir)
  File "finetune.py", line 161, in whisper_finetune
    trainer = Seq2SeqTrainer(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 56, in __init__
    super().__init__(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer.py", line 567, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```
Issue: train_dataset does not implement __len__

Comment:
How did you create `train_dataset`? The `datasets` library does not appear in your stack trace.

We need more information in order to reproduce the issue...
Issue: train_dataset does not implement __len__

Comment:
```
def asr_dataset(traindir,devdir):
    we_voice = IterableDatasetDict()
    #we_voice["train"] = load_from_disk(traindir,streaming=True)
    #we_voice["test"]= load_from_disk(devdir,streaming=True)
    we_voice["train"] = load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train",streaming=True)
    #print(load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train"))
    we_voice["test"] = load_dataset("csv",data_files=os.path.join(devdir,"dev.csv"), split="train",streaming=True)
    we_voice = we_voice.remove_columns(["id"])
    we_voice = we_voice.cast_column("audio", Audio(sampling_rate=16000))
    return we_voice

```
Issue: train_dataset does not implement __len__

Comment:
As you are using iterable datasets (`streaming=True`), their length is not defined.

You should:
- Either use non-iterable datasets, which have a defined length: use `DatasetDict` and not passing `streaming=True`
- Or pass `args.max_steps` to the `Trainer`
Issue: train_dataset does not implement __len__

Comment:
@albertvillanova I think @v-yunbin, myself, and others might be slightly confused about max_steps and how it differs from num_train_epochs.
Issue: train_dataset does not implement __len__

Comment:
@lkurlandski A **step** is referring to optimizer's update after back propagation, and it's associated with a batch of data. For example, if a dataset contains 64 examples and you have an overall batch size of 4, then an epoch will have 64/4=16 batches. Therefore, in one epoch, you will have 16 optimizer **steps**.
Issue: Tiktoken tokenizers are not pickable 

Comment:
Thanks for reporting, @markovalexander.

Unfortunately, I'm not able to reproduce the issue: the `tiktoken` tokenizer can be used within `Dataset.map`, both in my local machine and in a Colab notebook: https://colab.research.google.com/drive/1DhJroZgk0sNFJ2Mrz-jYgrmh9jblXaCG?usp=sharing

Are you sure you are using `datasets` version 2.11.0?
Issue: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1

Comment:
I am not able to reproduce your issue: the dataset loads perfectly on my local machine and on a Colab notebook: https://colab.research.google.com/drive/1Fbdoa1JdNz8DOdX6gmIsOK1nCT8Abj4O?usp=sharing
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("squad")
Downloading builder script: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.27k/5.27k [00:00<00:00, 3.22MB/s]
Downloading metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.36k/2.36k [00:00<00:00, 1.60MB/s]
Downloading readme: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.67k/7.67k [00:00<00:00, 4.58MB/s]
Downloading and preparing dataset squad/plain_text to ...t/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...
Downloading data: 30.3MB [00:00, 91.8MB/s]                                                                                                                                           | 0/2 [00:00<?, ?it/s]
Downloading data: 4.85MB [00:00, 75.3MB/s]                                                                                                                                                                 
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.31it/s]
Extracting data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2157.01it/s]
Dataset squad downloaded and prepared to .../.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 463.95it/s]

In [3]: ds
Out[3]: 
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 87599
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 10570
    })
})
```
Issue: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1

Comment:
I'm back on linux machine 1 (login node) now. After submitting that as a job to machine 2 and it failing with TypeError, linux machine 1 now produces identical traceback to machine 2:

> (arkroyal) [yr3g17@cyan52 squad_qanswering]$ python
> Python 3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>
> from datasets import load_dataset
> load_dataset("squad")
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/yr3g17/.conda/envs/arkroyal/lib/python3.10/site-packages/datasets/load.py", line 1759, in load_dataset
>     builder_instance = load_dataset_builder(
>   File "/home/yr3g17/.conda/envs/arkroyal/lib/python3.10/site-packages/datasets/load.py", line 1522, in load_dataset_builder
>     builder_instance: DatasetBuilder = builder_cls(
> TypeError: 'NoneType' object is not callable

I thought it might be useful to provide you with my cache file structure:

>_home_yr3g17_.cache_huggingface_datasets_casino_default_1.1.0_302c3b1ac78c48091deabe83a11f4003c7b472a4e11a8eb92799653785bd5da1.lock
>_home_yr3g17_.cache_huggingface_datasets_imdb_plain_text_1.0.0_2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1.lock
>_home_yr3g17_.cache_huggingface_datasets_squad_plain_text_1.0.0_d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453.lock
>_home_yr3g17_.cache_huggingface_datasets_yelp_review_full_yelp_review_full_1.0.0_e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf.lock
> casino
> downloads
> imdb
> json
> squad
> squad_v2
> yelp_review_full

The inside of squad/plain_text/1.0.0/ looks like

> d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453
> d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453.incomplete_info.lock
> d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453_builder.lock
Issue: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1

Comment:
I see this is quite a complex use case...

Let's try multiple things:
- First, update `datasets` and make sure you use the same version in all machines, so that we can easily compare different behaviors.
  ```
  pip install -U datasets
  ```
- Second, wherever you run the `load_dataset("squad")` command, make sure there is not a local directory named "squad". The datasets library gives priority to any local file/directory over the datasets on the Hugging Face Hub
  - I tell you this, because in one of your trace backs, it seems it refers to a local directory:
    ```
    Downloading and preparing dataset json/squad to /home/yr3g17/.cache/huggingface/datasets/json/squad-d733af945be1d2c2/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
    ```
- Third, to use the "squad" dataset from the Hub, you need to have internet connection, so that you can download the "squad" Python loading script from the Hub. Do all your machines have internet connection?
  - I ask this because of this error message:
    ```
    Using the latest cached version of the module from /home/yr3g17/.cache/huggingface/modules/datasets_modules/datasets/squad/8730650fed465361f38ac4d810ccdd16e8fc87b56498e52fb7e2cadaefc1f177 (last modified on Tue Feb 14 10:12:56 2023) since it couldn't be found locally at squad., or remotely on the Hugging Face Hub.
    ```
- Fourth, to be sure that we avoid any issues with the cache, it is a good idea to remove it and regenerate it. Remove `.cache/huggingface/datasets` and also `.cache/huggingface/modules`
- Fifth, as an additional debugging tool, let's be sure we use the latest "squad" Python loading script by passing the revision parameter:
  ```
  ds = load_dataset("squad", revision="5fe18c4c680f9922d794e3f4dd673a751c74ee37")
  ```
Issue: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1

Comment:
Additionally, we just had an infrastructure issue on the Hugging Face Hub at around 11:30 today. That might have contributed to the connectivity issue... It is fixed now.

https://status.huggingface.co/
Issue: load_dataset("squad") doesn't work in 2.7.1 and 2.10.1

Comment:
Hi again, thanks for your help and insight Albert Villanova.

It's all working now, so thank you for that. For the benefit of anyone else who ends up in this thread, I solved the problem by addressing Albert's advice:

(1) Both Windows and Linux machine 1 (have internet access) and can now access the SQuAD dataset. The supercomputer login node can only access version 2.7.1, but my Windows laptop is running on datasets 2.11.0 just fine. I suspect it was just a perfect storm alongside the aforementioned "infrastructure issue".

(2) I did have a local directory called squad, because I was using a local copy of evaluate's "SQuAD" metric. The supercomputer compute nodes do not have internet access and treat `metric = evaluate.load('<x>')` as a way of loading a metric at the local path `./<x>/<x>.py`, which could've been a related issue as I was storing the metric under `squad/squad.py`. Don't be lazy like me and store the evaluation code under a path with a name that can be misinterpreted.

(3) I can't give internet access to the supercomputer compute nodes, so local files do just fine here.

(4) The windows and Linux machine 1 can both access the internet and were getting fresh copies of the dataset from the huggingface hub. Linux machine 2 was working after I cleared the contents of ~/.cache/huggingface/....

I feel silly now, knowing it was all so simple! Sorry about that Albert, and thanks again for the help. I've not raised a Github issue like this before, so I'm not sure if I should be close my own issues or if this is something you guys do?
Issue: Support custom feature types

Comment:
Hi ! Interesting :) What kind of new types would you like to use ?

Note that you can already implement your own decoding by using `set_transform` that can decode data on-the-fly when rows are accessed
Issue: Support custom feature types

Comment:
An interesting proposal indeed. 

Pandas and Polars have the "extension API", so doing something similar on our side could be useful, too. However, this requires defining a common interface for the existing feature types before discussing the API/workflow for defining/sharing custom feature types, and this could take some time.

It would also be nice if the datasets viewer could render these custom types.
Issue: Support custom feature types

Comment:
Thank you for your replies! @lhoestq I have a use case involving whole-slide images in digital pathology. These are very large images (potentially gigapixel scale), so standard image tools are not suitable. Essentially, encoding/decoding can be done from/to [`OpenSlide`](https://openslide.org/api/python/) objects. Though there may be interest in this use case from the digital pathology community, it may not be sufficiently useful to suggest adding the feature type, but there will likely be many other use cases for a generic custom feature type.

Thank you for pointing out `set_transform`! I will make sure to keep this in mind in the future.

@mariosasko An "extension API" sounds like a good idea, though I understand that this needs to be properly defined, and that you will need to discuss it internally. Support from the viewer would be awesome, too, though the generalization to arbitrary types sounds challenging.

For now, happy to know that you're considering the feature. Feel free to let me know if I can do anything to support the process.
Issue: ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['text']

Comment:
You need to remove the `text` and `text_en` columns before passing the dataset to the `DataLoader` to avoid this error:
```python
tokenized_datasets = tokenized_datasets.remove_columns(["text", "text_en"])
```
Issue: ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['text']

Comment:
Please also remove/comment these lines:
```python
tokenized_datasets = tokenized_datasets.remove_columns("attention_mask")
tokenized_datasets = tokenized_datasets.remove_columns("input_ids")
tokenized_datasets = tokenized_datasets.remove_columns("label")
```
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
Thanks for reporting, @sauravtii.

Unfortunately, I'm not able to reproduce the issue:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("josianem/imdb")

In [2]: ds
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25799
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})
```

Could you please retry to load the dataset? Maybe there was a temporary connection issue to Dropbox.
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
Thanks @albertvillanova. I am facing another issue now

```
Traceback (most recent call last):
  File "sample.py", line 4, in <module>
    dataset = load_dataset("josianem/imdb")
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 738, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=34501348, num_examples=25799, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='test', num_bytes=32650697, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67106814, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

This is my code

```
from datasets import load_dataset, load_metric

dataset = load_dataset("josianem/imdb")
```
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
Your connection didn't work and you got an empty dataset (`num_bytes=0, num_examples=0`):
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: 
[
    {
        'expected': SplitInfo(name='train', num_bytes=34501348, num_examples=25799, dataset_name='imdb'), 
        'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')
    }, 
    {
        'expected': SplitInfo(name='test', num_bytes=32650697, num_examples=25000, dataset_name='imdb'), 
        'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, dataset_name='imdb')
    }, 
    {
        'expected': SplitInfo(name='unsupervised', num_bytes=67106814, num_examples=50000, dataset_name='imdb'), 
        'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')
    }
]
```

Could you please try the link in your browser and see if it works? https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1
- If it does not work, you should contact the author of the dataset in their Community tab (https://huggingface.co/datasets/josianem/imdb/discussions) and inform them, so that they can host their data elsewhere, for example on the Hugging Face Hub itself

If the link works, you should try to load the dataset but forcing the re-download of the data files (so that the cache is refreshed with the actual data file), by passing `download_mode="force_redownload"`:
```python
dataset = load_dataset("josianem/imdb", download_mode="force_redownload")
```
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
After pasting the link in the browser, it did start the download so it seems that the link is working. But even after including the `download_mode` in my code I am facing the same issue:

Error:
```
Traceback (most recent call last):
  File "sample.py", line 4, in <module>
    dataset = load_dataset("josianem/imdb", download_mode="force_redownload")
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/builder.py", line 704, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/saurav/.cache/huggingface/modules/datasets_modules/datasets/imdb/cc6ab4acab2799be15d5d217c24548b856156dafdc850165fdc4f2031f27ff2f/imdb.py", line 79, in _split_generators
    archive = dl_manager.download(_DOWNLOAD_URL)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 289, in cached_path
    output_path = get_from_cache(
  File "/home/saurav/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 606, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1
```

My code:
```
from datasets import load_dataset, load_metric

dataset = load_dataset("josianem/imdb", download_mode="force_redownload")
```
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
I have tried again to reproduce your issue without success: the dataset loads perfectly, both in my local machine and in a Colab notebook.
- See: https://colab.research.google.com/drive/1dky3T0XGFuldggy22NNQQN-UqOFqvnuY?usp=sharing

I think the cause maight be that you are using a very old version of `datasets`. Please, could you update it and retry?
```
pip install -U datasets
```
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
That worked!! Thanks @albertvillanova : )

```
Downloading builder script: 100%|███████| 4.20k/4.20k [00:00<00:00, 6.69MB/s]
Downloading metadata: 100%|█████████████| 2.60k/2.60k [00:00<00:00, 3.41MB/s]
Downloading readme: 100%|███████████████| 7.52k/7.52k [00:00<00:00, 12.6MB/s]
Downloading and preparing dataset imdb/plain_text to /home/saurav/.cache/huggingface/datasets/josianem___imdb/plain_text/1.0.0/cc6ab4acab2799be15d5d217c24548b856156dafdc850165fdc4f2031f27ff2f...
Downloading data: 100%|███████████████████| 301M/301M [01:32<00:00, 3.25MB/s]
Dataset imdb downloaded and prepared to /home/saurav/.cache/huggingface/datasets/josianem___imdb/plain_text/1.0.0/cc6ab4acab2799be15d5d217c24548b856156dafdc850165fdc4f2031f27ff2f. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████| 3/3 [00:00<00:00, 794.83it/s]
```

The code I used:
```
from datasets import load_dataset, load_metric

dataset = load_dataset("josianem/imdb", download_mode="force_redownload")

```

But when I remove `download_mode="force_redownload"` I get the same error. Any guess on that?
Issue: ConnectionError: Couldn't reach https://www.dropbox.com/s/zts98j4vkqtsns6/aclImdb_v2.tar?dl=1

Comment:
That is because the cache got the "empty" download file the first time you tried and got the connection error.

Then, once you no longer get the connection error, you need to refresh the cache by passing `download_mode="force_redownload"`.
Issue: Not able to load the pile

Comment:
Thanks for reporting, @surya-narayanan.

I see you already started a discussion about this on the Community tab of the corresponding dataset: https://huggingface.co/datasets/EleutherAI/the_pile/discussions/10
Let's continue the discussion there!