SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Load local dataset error

Comment:
I'm getting this problem. Script has been stuck at this part for the past 15 or so minutes:
 
`Resolving data files: 100%|█████████████████████████████████████████| 107/107 [00:00<00:00, 472.74it/s]`

I had everything working fine on an AWS EC2 node with a single GPU. Then I created an image based on the single GPU machine, and spun up a new one with 4 GPUs, so I got all of the training data ready at .cache. 

Turned off all checks with `verification_mode='no_checks'`. Logged in with huggingface-cli again just to be sure.

Interrupting shows the code is stuck here:

```
File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 200, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 336, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 357, in read_table
    return table_cls.from_file(filename)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/table.py", line 1059, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/table.py", line 66, in _memory_mapped_arrow_table_from_file
    pa_table = opened_stream.read_all()
```

Is it just going to take a while or am I going to run out of money? :sweat_smile: 

edit: ping @mariosasko
Issue: Medium-sized dataset conversion from pandas causes a crash 

Comment:
Hi ! It looks like an issue with pyarrow, could you try updating pyarrow and try again ?
Issue: Medium-sized dataset conversion from pandas causes a crash 

Comment:
I´m getting the same problem with some files, @albertvillanova did you find a solution to this?
Issue: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
Hi @amirj, thanks for reporting.

At first sight, your issue seems a version incompatibility between your Elasticsearch client and your Elasticsearch server.

Feel free to have a look at Elasticsearch client docs: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/overview.html#_compatibility
> Language clients are forward compatible; meaning that clients support communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.
Issue: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
@albertvillanova It doesn't seem a version incompatibility between the client and server, since the following code is working:

```
from elasticsearch import Elasticsearch
es_client = Elasticsearch("http://localhost:9200")
dataset.add_elasticsearch_index(column="e1", es_client=es_client, es_index_name="e1_index")
```
Issue: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
Hi @amirj, 

I really think it is a version incompatibility issue between your Elasticsearch client and server:
- Your Elasticsearch server NodeConfig expects a positional argument named 'scheme'
- Whereas your Elasticsearch client passes only keyword arguments: `NodeConfig(**options)`

Moreover:
- Looking at your stack trace, I deduce you are using Elasticsearch client **"8"** major version:
  - the Elasticsearch file "elasticsearch/_sync/client/utils.py" was created in version "8.0.0a1": https://github.com/elastic/elasticsearch-py/commit/21fa13b0f03b7b27ace9e19a1f763d40bd2e2ba4
  - you can check your Elasticsearch client version by running this Python code:
    ```python
    import elasticsearch
    print(elasticsearch.__version__)
    ```

- However, in the *Environment info*, you informed that the major version of your Eleasticsearch cluster server is **"7"** ("7.10.2-SNAPSHOT")

Could you please align the Elasticsearch client/server major versions (as pointed out in Elasticsearch docs) and check if the problem persists?
Issue: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
```
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
```
```
TypeError                                 Traceback (most recent call last)
<ipython-input-8-675c6ffe5293> in <module>
      1 #es = Elasticsearch([{'host':'localhost', 'port':9200}])
      2 from elasticsearch import Elasticsearch
----> 3 es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\_sync\client\__init__.py in __init__(self, hosts, cloud_id, api_key, basic_auth, bearer_auth, opaque_id, headers, connections_per_node, http_compress, verify_certs, ca_certs, client_cert, client_key, ssl_assert_hostname, ssl_assert_fingerprint, ssl_version, ssl_context, ssl_show_warn, transport_class, request_timeout, node_class, node_pool_class, randomize_nodes_in_pool, node_selector_class, dead_node_backoff_factor, max_dead_node_backoff, serializer, serializers, default_mimetype, max_retries, retry_on_status, retry_on_timeout, sniff_on_start, sniff_before_requests, sniff_on_node_failure, sniff_timeout, min_delay_between_sniffing, sniffed_node_callback, meta_header, timeout, randomize_hosts, host_info_callback, sniffer_timeout, sniff_on_connection_fail, http_auth, maxsize, _transport)
    310 
    311         if _transport is None:
--> 312             node_configs = client_node_configs(
    313                 hosts,
    314                 cloud_id=cloud_id,

C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\_sync\client\utils.py in client_node_configs(hosts, cloud_id, **kwargs)
     99     else:
    100         assert hosts is not None
--> 101         node_configs = hosts_to_node_configs(hosts)
    102 
    103     # Remove all values which are 'DEFAULT' to avoid overwriting actual defaults.

C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\_sync\client\utils.py in hosts_to_node_configs(hosts)
    142 
    143         elif isinstance(host, Mapping):
--> 144             node_configs.append(host_mapping_to_node_config(host))
    145         else:
    146             raise ValueError(

C:\ProgramData\Anaconda3\lib\site-packages\elasticsearch\_sync\client\utils.py in host_mapping_to_node_config(host)
    209         options["path_prefix"] = options.pop("url_prefix")
    210 
--> 211     return NodeConfig(**options)  # type: ignore
    212 
    213 

TypeError: __init__() missing 1 required positional argument: 'scheme'
```
Issue: TypeError: __init__() missing 1 required positional argument: 'scheme'

Comment:
@raj713335, thanks for reporting.

Please note that in your code example, you are not using our `datasets` library. 

Thus, I think you should report that issue to `elasticsearch` library: https://github.com/elastic/elasticsearch-py
Issue: The dataset preview is not available for tdklab/Hebrew_Squad_v1.1 dataset

Comment:
Hi @MatanBenChorin, thanks for reporting.

Please, take into account that the preview may take some time until it properly renders (we are working to reduce this time).

Maybe @severo can give more details on this.
Issue: The dataset preview is not available for tdklab/Hebrew_Squad_v1.1 dataset

Comment:
I imagine the dataset has been moved to https://huggingface.co/datasets/tdklab/Hebrew_Squad_v1, which still has an issue:

```
Server Error

Status code:   400
Exception:     NameError
Message:       name 'HebrewSquad' is not defined
```
Issue: The dataset preview is not available for tdklab/Hebrew_Squad_v1.1 dataset

Comment:
The issue is not related to the dataset viewer but to the loading script (cc @albertvillanova @lhoestq @mariosasko)

```python
>>> import datasets as ds
>>> hf_token = "hf_..." # <- required because the dataset is gated
>>> d = ds.load_dataset('tdklab/Hebrew_Squad_v1', use_auth_token=hf_token)
...
NameError: name 'HebrewSquad' is not defined
```
Issue: The dataset preview is not available for tdklab/Hebrew_Squad_v1.1 dataset

Comment:
Yes indeed there is an error in [Hebrew_Squad_v1.py:L40](https://huggingface.co/datasets/tdklab/Hebrew_Squad_v1/blob/main/Hebrew_Squad_v1.py#L40)

Here is the fix @MatanBenChorin :

```diff
-         HebrewSquad(
+         HebrewSquadConfig(
```
Issue: Checksum error for glue sst2, stsb, rte etc datasets

Comment:
Hi, @ravindra-ut.

I'm sorry but I can't reproduce your problem:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("glue", "sst2")
Downloading builder script: 28.8kB [00:00, 11.6MB/s]                                                                                                                                                       
Downloading metadata: 28.7kB [00:00, 12.9MB/s]                                                                                                                                                             
Downloading and preparing dataset glue/sst2 (download: 7.09 MiB, generated: 4.81 MiB, post-processed: Unknown size, total: 11.90 MiB) to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.44M/7.44M [00:01<00:00, 5.82MB/s]
Dataset glue downloaded and prepared to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.  
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 895.96it/s]

In [3]: ds
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
})
``` 

Moreover, I see in your traceback that your error was for an URL at https://firebasestorage.googleapis.com
However, the URLs were updated on Sep 16, 2020 (`datasets` version 1.0.2) to https://dl.fbaipublicfiles.com: https://github.com/huggingface/datasets/commit/2f03041a21c03abaececb911760c3fe4f420c229

Could you please try to update `datasets`
```shell
pip install -U datasets
```
and then force redownload
```python
ds = load_dataset("glue", "sst2", download_mode="force_redownload")
```
to update the cache?

Please, feel free to reopen this issue if the problem persists.
Issue: Forked streaming datasets try to `open` data urls rather than use network

Comment:
Thanks for reporting this second issue as well. We definitely want to make streaming datasets fully working in a distributed setup and with the best performance. Right now it only supports single process.

In this issue it seems that the streaming capabilities that we offer to dataset builders are not transferred to the forked process (so it fails to open remote files and start streaming data from them). In particular `open` is supposed to be mocked by our `xopen` function that is an extended open that supports remote files. Let me try to fix this
Issue: Streaming Datasets don't work with Transformers Trainer when dataloader_num_workers>1

Comment:
Hi, thanks for reporting. This could be related to https://github.com/huggingface/datasets/issues/3148 too

We should definitely make `TorchIterableDataset` picklable by moving it in the main code instead of inside a function. If you'd like to contribute, feel free to open a Pull Request :)

I'm also taking a look at your second issue, which is more technical
Issue: reddit_tifu dataset: Checksums didn't match for dataset source files

Comment:
Hi @XingxingZhang, 

We have already fixed this. You should update `datasets` version to at least 1.18.4:
```shell
pip install -U datasets
```
And then force the redownload:
```python
load_dataset("...", download_mode="force_redownload")
```

Duplicate of:
- #3773
Issue: reddit_tifu dataset: Checksums didn't match for dataset source files

Comment:
thanks @albertvillanova . by upgrading to 1.18.4 and using `load_dataset("...", download_mode="force_redownload")` fixed 
 the bug.

using the following as you suggested in another thread can also fixed the bug
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Issue: reddit_tifu dataset: Checksums didn't match for dataset source files

Comment:
The latter solution (installing from GitHub) was proposed because the fix was not released yet. But last week we made the 1.18.4 patch release (with the fix), so no longer necessary to install from GitHub.

You can now install from PyPI, as usual:
```shell
pip install -U datasets
```
Issue: billsum dataset: Checksums didn't match for dataset source files:

Comment:
Hi @XingxingZhang, thanks for reporting.

This was due to a change in Google Drive service:
- #3786 

We have already fixed it:
- #3787

You should update `datasets` version to at least 1.18.4:
```shell
pip install -U datasets
```
And then force the redownload:
```python
load_dataset("...", download_mode="force_redownload")
```
Issue: Source links broken

Comment:
Thanks for reporting @qqaatw.

@mishig25 @sgugger do you think this can be tweaked in the new doc framework?
- From: https://github.com/huggingface/datasets/blob/v2.0.0/
- To: https://github.com/huggingface/datasets/blob/2.0.0/
Issue: Source links broken

Comment:
@qqaatw thanks a lot for notifying about this issue!

in comparison, transformers tags start with `v` like [this one](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/bert/configuration_bert.py#L54).

Therefore, we have to do one of 2 options below:
1. Make necessary changes on doc-builder side
OR
2. Make [datasets tags](https://github.com/huggingface/datasets/tags) start with `v`, just like [transformers](https://github.com/huggingface/transformers/tags) (so that tag naming can be consistent amongst hf repos)

I'll let you decide @albertvillanova @lhoestq @sgugger
Issue: Source links broken

Comment:
I think option 2 is the easiest and would provide harmony in the HF ecosystem but we can also add a doc config parameter to decide whether the default version has a v or not if `datasets` folks prefer their tags without a v :-)
Issue: Source links broken

Comment:
For me it is OK to conform to the rest of libraries and tag/release with a preceding "v", rather than adding an extra argument to the doc builder just for `datasets`.

Let me know if it is also OK for you @lhoestq.
Issue: Source links broken

Comment:
We could add a tag for each release without a 'v' but it could be confusing on github to see both tags `v2.0.0` and `2.0.0` IMO (not sure if many users check them though). Removing the tags without 'v' would break our versioning for github datasets: the library looks for dataset scripts at the URLs like `https://raw.githubusercontent.com/huggingface/datasets/{revision}/datasets/{path}/{name}` where `revision` is equal to `datasets.__version__` (which doesn't start with a 'v') for all released versions of `datasets`.

I think we could just have a parameter for the documentation - and having different URLs schemes for the source links that the users don't even see (they simply click on a button) is probably fine
Issue: Source links broken

Comment:
This is done in #3943 to go along with [doc-builder#146](https://github.com/huggingface/doc-builder/pull/146).

Note that this will only work for future versions, so once those two are merged, the actual v2.0.0 doc should be fixed. The easiest is to cherry-pick this commit on the v2.0.0 release branch (or on a new branch created from the 2.0.0 tag, with a name that triggers the doc building job, for instance v2.0.0-release)
Issue: Missing languages in lvwerra/github-code dataset

Comment:
That seems to be an oversight - I originally planned to include them in the dataset and for some reason they were in the list of languages but not in the query. Since there is an issue with the deduplication step I'll rerun the pipeline anyway and will double check the query.

Thanks for reporting this @Eytan-S!
Issue: Missing languages in lvwerra/github-code dataset

Comment:
Can confirm that the two languages are indeed missing from the dataset. Here are the file counts per language:
```Python
{'Assembly': 82847,
 'Batchfile': 236755,
 'C': 14127969,
 'C#': 6793439,
 'C++': 7368473,
 'CMake': 175076,
 'CSS': 1733625,
 'Dockerfile': 331966,
 'FORTRAN': 141963,
 'GO': 2259363,
 'Haskell': 340521,
 'HTML': 11165464,
 'Java': 19515696,
 'JavaScript': 11829024,
 'Julia': 58177,
 'Lua': 576279,
 'Makefile': 679338,
 'Markdown': 8454049,
 'PHP': 11181930,
 'Perl': 497490,
 'PowerShell': 136827,
 'Python': 7203553,
 'Ruby': 4479767,
 'Rust': 321765,
 'SQL': 655657,
 'Scala': 0,
 'Shell': 1382786,
 'TypeScript': 0,
 'TeX': 250764,
 'Visual Basic': 155371}
 ```
Issue: Missing languages in lvwerra/github-code dataset

Comment:
@Eytan-S check out v1.1 of the `github-code` dataset where issue should be fixed:

|    | Language     |File Count| Size (GB)|
|---:|:-------------|---------:|-------:|
|  0 | Java         | 19548190 | 107.7  |
|  1 | C            | 14143113 | 183.83 |
|  2 | JavaScript   | 11839883 |  87.82 |
|  3 | HTML         | 11178557 | 118.12 |
|  4 | PHP          | 11177610 |  61.41 |
|  5 | Markdown     |  8464626 |  23.09 |
|  6 | C++          |  7380520 |  87.73 |
|  7 | Python       |  7226626 |  52.03 |
|  8 | C#           |  6811652 |  36.83 |
|  9 | Ruby         |  4473331 |  10.95 |
| 10 | GO           |  2265436 |  19.28 |
| 11 | TypeScript   |  1940406 |  24.59 |
| 12 | CSS          |  1734406 |  22.67 |
| 13 | Shell        |  1385648 |   3.01 |
| 14 | Scala        |   835755 |   3.87 |
| 15 | Makefile     |   679430 |   2.92 |
| 16 | SQL          |   656671 |   5.67 |
| 17 | Lua          |   578554 |   2.81 |
| 18 | Perl         |   497949 |   4.7  |
| 19 | Dockerfile   |   366505 |   0.71 |
| 20 | Haskell      |   340623 |   1.85 |
| 21 | Rust         |   322431 |   2.68 |
| 22 | TeX          |   251015 |   2.15 |
| 23 | Batchfile    |   236945 |   0.7  |
| 24 | CMake        |   175282 |   0.54 |
| 25 | Visual Basic |   155652 |   1.91 |
| 26 | FORTRAN      |   142038 |   1.62 |
| 27 | PowerShell   |   136846 |   0.69 |
| 28 | Assembly     |    82905 |   0.78 |
| 29 | Julia        |    58317 |   0.29 |
Issue: Load a local dataset twice

Comment:
Hi @caush, thanks for reporting:

In order to load local CSV files, you can use our "csv" loading script: https://huggingface.co/docs/datasets/loading#csv
```python
dataset = load_dataset("csv", data_files=["data/file1.csv", "data/file2.csv"])
```
OR:
```python
dataset = load_dataset("csv", data_dir="data")
```

Alternatively, you may also use:
```python
dataset = load_dataset("data")
Issue: Frugal score deprecations

Comment:
Hi @Ierezell, thanks for reporting.

I'm making a PR to suppress those logs from the terminal.
Issue: 'datasets.features' is not a package

Comment:
Hi @Arij-Aladel,

You are using a very old version of our library `datasets`: 1.8.0
Current version is 2.0.0 (and the previous one was 1.18.4)

Please, try to update `datasets` library and check if the problem persists:
```shell
/env/bin/pip install -U datasets
Issue: 'datasets.features' is not a package

Comment:
The problem I can no I have build my project on this version and old version on transformers. I have preprocessed the data again to use it. Thank for your reply
Issue: AttributeError: 'DatasetDict' object has no attribute 'features'

Comment:
You are likely trying to get the `features` from a `DatasetDict`, a dictionary containing `Datasets`. You probably first want to index into a particular split from your `DatasetDict` i.e. `dataset['train'].features`. 

For example 

```python 
ds = load_dataset('mnist')
ds.features
```
Returns 
```python
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

[<ipython-input-39-791c1f9df6c2>](https://localhost:8080/#) in <module>()
----> 1 ds.features

AttributeError: 'DatasetDict' object has no attribute 'features'
```

If we look at the dataset variable, we see it is a `DatasetDict`:

```python 
print(ds)
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 60000
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 10000
    })
})
```

We can grab the features from a split by indexing into `train`:
```python
ds['train'].features
{'image': Image(decode=True, id=None),
 'label': ClassLabel(num_classes=10, names=['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'], id=None)}
```

Hope that helps
Issue: datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files

Comment:
Hi @willowdong! These issues were fixed on master. We will have a new release of `datasets` later today. In the meantime, you can avoid these issues by installing `datasets` from master as follows:
```bash
pip install git+https://github.com/huggingface/datasets.git
```
Issue: Error loading file audio when downloading the Common Voice dataset directly from the Hub

Comment:
Hi ! It could an issue with torchaudio, which version of torchaudio are you using ? Can you also try updating `datasets` to 2.0.0 and see if it works ?
Issue: Error loading file audio when downloading the Common Voice dataset directly from the Hub

Comment:
Hi ! In `datasets` 2.0 can access the audio array with `librispeech_eval[0]["audio"]["array"]` already, no need to use `map_to_array`. See our documentation on [how to process audio data](https://huggingface.co/docs/datasets/audio_process) :)

cc @patrickvonplaten we will need to update the readme at [facebook/s2t-small-librispeech-asr](https://huggingface.co/facebook/s2t-small-librispeech-asr) as well as https://huggingface.co/docs/transformers/model_doc/speech_to_text
Issue: Error loading file audio when downloading the Common Voice dataset directly from the Hub

Comment:
Thanks!

And sorry for posting this problem in what turned on to be an unrelated thread.

I rewrote the code, and the model works. The WER is 0.137 however, so I'm not sure if I have missed a step. I will look further into that at a later point. The transcriptions look good through manual inspection.

The rewritten code:
```python
from datasets import load_dataset, load_metric
from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor, Wav2Vec2Processor

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
wer = load_metric("wer")

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)

def map_to_pred(batch):
    audio = batch["audio"]
    features = processor(audio["array"], sampling_rate=audio["sampling_rate"], padding=True, return_tensors="pt")
    input_features = features.input_features.to("cuda")
    attention_mask = features.attention_mask.to("cuda")

    gen_tokens = model.generate(input_features=input_features, attention_mask=attention_mask)
    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)
    return batch

result = librispeech_eval.map(map_to_pred)#, batched=True, batch_size=8)

print("WER:", wer.compute(predictions=result["transcription"], references=result["text"]))
```