MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each.

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

@lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

->
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
```

if i do
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Hi ! You can pass dataset.data.table instead of dataset.data to pq.write_table


MemoryError when computing WER metric

MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

This seems to be working so far! Thanks!


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
Hi @XuhuiZhou, thanks for reporting this issue.

Indeed, the old links are no longer valid (404 Not Found error), and the script must be updated with the new links to Google Drive.


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
It would be nice to update the urls indeed !

To do this, you just need to replace the urls in `iwslt2017.py` and then update the dataset_infos.json file with
```
datasets-cli test ./datasets/iwslt2017 --all_configs --save_infos --ignore_verifications
```


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
Is this a command to update my local files or fix the file Github repo in general? (I am not so familiar with the datasets-cli command here)

I also took a brief look at the **Sharing your dataset** section, looks like I could fix that locally and push it to the repo? I guess we are "canonical" category?


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
This command will update your local file. Then you can open a Pull Request to push your fix to the github repo :)
And yes you are right, it is a "canonical" dataset, i.e. a dataset script defined in this github repo (as opposed to dataset repositories of users on the huggingface hub)


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
Hi, thanks for the answer.

I gave a try to the problem today. But I encountered an upload error:

```
git push -u origin fix_link_iwslt
Enter passphrase for key '/home2/xuhuizh/.ssh/id_rsa':
ERROR: Permission to huggingface/datasets.git denied to XuhuiZhou.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Any insight here?

By the way, when I run the datasets-cli command, it shows the following error, but does not seem to be the error coming from `iwslt.py`

```
Traceback (most recent call last):
  File "/home2/xuhuizh/anaconda3/envs/UMT/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/datasets_cli.py", line 35, in main
    service.run()
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/test.py", line 141, in run
    try_from_hf_gcs=False,
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 639, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/xuhuizh/projects/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz'}
```


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
Hi ! To create a PR on this repo your must fork it and create a branch on your fork. See how to fork the repo [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#start-by-preparing-your-environment).
And to make the command work without the `ExpectedMoreDownloadedFiles` error, you just need to use the `--ignore_verifications` flag.


Issue: Dataset download error

Issue: Dataset download error
The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?
Hi @XuhuiZhou,

As @lhoestq has well explained, you need to fork HF's repository, create a feature branch in your fork, push your changes to it and then open a Pull Request to HF's upstream repository. This is so because at HuggingFace Datasets we follow a development model called "Fork and Pull Model". You can find more information here:
- [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
- [Forking Projects](https://guides.github.com/activities/forking/)

Alternatively, if you find all these steps too complicated, you can use the GitHub official command line tool: [GitHub CLI](https://cli.github.com/). Once installed, in order to create a Pull Request, you only need to use this command:
```shell
gh pr create --web
```
This utility will automatically create the fork, push your changes and open a Pull Request, under the hood.


ConnectionError: Couldn't reach common_voice.py
When I run:
from datasets import load_dataset, load_metric

common_voice_train = load_dataset("common_voice", "zh-CN", split="train+validation")
common_voice_test = load_dataset("common_voice", "zh-CN", split="test")

Got:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/common_voice/common_voice.py

Version:
1.4.1

Thanks!  @lhoestq @LysandreJik @thomwolf
Hi @LifaSun, thanks for reporting this issue.

Sometimes, GitHub has some connectivity problems. Could you confirm that the problem persists?


ConnectionError: Couldn't reach common_voice.py
When I run:
from datasets import load_dataset, load_metric

common_voice_train = load_dataset("common_voice", "zh-CN", split="train+validation")
common_voice_test = load_dataset("common_voice", "zh-CN", split="test")

Got:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/common_voice/common_voice.py

Version:
1.4.1

Thanks!  @lhoestq @LysandreJik @thomwolf
@albertvillanova Thanks! It works well now.


Multiprocessing is slower than single process
```python
# benchmark_filter.py
import logging
import sys
import time

from datasets import load_dataset, set_caching_enabled

if __name__ == "__main__":
    set_caching_enabled(False)
    logging.basicConfig(level=logging.DEBUG)

    bc = load_dataset("bookcorpus")

    now = time.time()
    try:
        bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
    except Exception as e:
        print(f"cancelled: {e}")
    elapsed = time.time() - now

    print(elapsed)
```

Running `python benchmark_filter.py 1` (20min+) is faster than `python benchmark_filter.py 2` (2hrs+)
dupe of #1992


ArrowInvalid issue for squad v2 dataset

ArrowInvalid issue for squad v2 dataset
Hello, I am using the huggingface official question answering example notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

In the prepare_validation_features function, I made some modifications to tokenize a new set of quesions with the original contexts and save them in three different list called candidate_input_dis, candidate_attetion_mask and candidate_token_type_ids. When I try to run the next cell for dataset.map, I got the following error:

`ArrowInvalid: Column 1 named candidate_attention_mask expected length 1180 but got length 1178`

My code is as follows:

```
def generate_candidate_questions(examples):
  val_questions = examples["question"]
  candididate_questions = random.sample(datasets["train"]["question"], len(val_questions))
  candididate_questions = [x[:max_length] for x in candididate_questions]
  return candididate_questions

def prepare_validation_features(examples, use_mixing=False):
  pad_on_right = tokenizer.padding_side == "right"
  tokenized_examples = tokenizer(
      examples["question" if pad_on_right else "context"],
      examples["context" if pad_on_right else "question"],
      truncation="only_second" if pad_on_right else "only_first",
      max_length=max_length,
      stride=doc_stride,
      return_overflowing_tokens=True,
      return_offsets_mapping=True,
      padding="max_length",
  )
  if use_mixing:
    candidate_questions = generate_candidate_questions(examples)
    tokenized_candidates = tokenizer(
        candidate_questions if pad_on_right else examples["context"],
        examples["context"] if pad_on_right else candidate_questions,
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

  sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

  tokenized_examples["example_id"] = []

  if use_mixing:
    tokenized_examples["candidate_input_ids"] = tokenized_candidates["input_ids"]
    tokenized_examples["candidate_attention_mask"] = tokenized_candidates["attention_mask"]
    tokenized_examples["candidate_token_type_ids"] = tokenized_candidates["token_type_ids"]

  for i in range(len(tokenized_examples["input_ids"])):
      sequence_ids = tokenized_examples.sequence_ids(i)
      context_index = 1 if pad_on_right else 0

      sample_index = sample_mapping[i]
      tokenized_examples["example_id"].append(examples["id"][sample_index])

      tokenized_examples["offset_mapping"][i] = [
          (o if sequence_ids[k] == context_index else None)
          for k, o in enumerate(tokenized_examples["offset_mapping"][i])
      ]

  return tokenized_examples

validation_features = datasets["validation"].map(
    lambda xs: prepare_validation_features(xs, True),
    batched=True,
    remove_columns=datasets["validation"].column_names
)
```

I guess this might happen because of the batched=True. I see similar issues in this repo related to arrow table length mismatch error, but in their cases, the numbers vary a lot. In my case, this error always happens when the expected length and unexpected length are very close. Thanks for the help!
Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

cc @philschmid


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`.
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

Could paste the code you use the start your training job and the fine-tuning script you run?


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

@sivakhno this should be now fixed in `datasets>=1.5.0`.


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

@philschmid Recently released tensorflow-macos seems to be missing.


PyTorch not available error on SageMaker GPU docker though it is installed
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!

I've created a PR to add this.


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')

updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop

I was able to copy some of the shell
This is repeating every half second
Win 10, Anaconda with python 3.8, datasets installed from main branche
```

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    exitcode = _main(fd, parent_sentinel)
    raise RuntimeError('''
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,
    prepare(preparation_data)

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    self._popen = self._Popen(self)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 327, in _Popen
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    return Popen(process_obj)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\popen_spawn_win32.py", line 45, in __init__
    return self._repopulate_pool_static(self._ctx, self.Process,
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...
```


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
Thanks this is really helpful !
I'll try to reproduce on my side and come back to you


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
if __name__ == '__main__':

This line before calling the map function stops the error but the script still repeats endless


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
It's already on the newest version


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
```
Traceback (most recent call last):
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\tmpx9fl_jg8' -> 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\cvtrain.py", line 243, in <module>
    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1339, in map
    return self._map_single(
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1646, in _map_single
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 805, in move
    copy_function(src, real_dst)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0%|                                                                                | 0/27771 [00:00<?, ?ex/s]
 File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'
```

I was adding freeze support before calling the mapping function like this
if __name__ == '__main__':
    freeze_support()
   dataset.map(....)


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?


Multiprocessing windows error

Multiprocessing windows error
As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop
Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Hi ! Thanks for reporting.

Currently there's no way to specify this.

When loading/processing a dataset, the arrow file is written using a temporary file. Then once writing is finished, it's moved to the cache directory (using `shutil.move` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646))

That means it keeps the permissions specified by the `tempfile.NamedTemporaryFile` object, i.e. `-rw-------` instead of `-rw-r--r--`. Improving this could be a nice first contribution to the library :)


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Hi @lhoestq,
I looked into this and yes you're right. The `NamedTemporaryFile` is always created with mode 0600, which prevents group from reading the file. Should we change the permissions of `tmp_file.name` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1871) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1590), post creation to 0644 inorder for group and others to read it?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Good idea :) we could even update the permissions after the file has been moved by shutil.move [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1899) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646) actually.
Apparently they set the default 0600 for temporary files for security reasons, so let's update the umask only after the file has been moved


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Would it be possible to actually set the umask based on a user provided argument? For example, a popular usecase my team has is using a shared file-system for processing datasets. This may involve writing/deleting other files, or changing filenames, which a -rw-r--r-- wouldn't fix.


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Note that you can get the cache files of a dataset with the `cache_files` attributes.
Then you can `chmod` those files and all the other cache files in the same directory.

Moreover we can probably keep the same permissions after each transform. This way you just need to set the permissions once after doing `load_dataset` for example, and then all the new transformed cached files will have the same permissions.
What do you think ?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
This means we'll check the permissions of other `cache_files` already created for a dataset before setting permissions for new  `cache_files`?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
You can just check the permission of `dataset.cache_files[0]` imo


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
> This way you just need to set the permissions once after doing load_dataset for example, and then all the new transformed cached files will have the same permissions.

I was referring to this. Ensuring that newly generated `cache_files` have the same permissions


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Yes exactly

I imagine users can first do `load_dataset`, then chmod on the arrow files. After that all the new cache files could have the same permissions as the first arrow files. Opinions on this ?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Sounds nice but I feel this is a sub-part of the approach mentioned by @siddk. Instead of letting the user set new permissions by itself first and then making sure newly generated files have same permissions why don't we ask the user initially only what they want? What are your thoughts?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Yes sounds good. Should this be a parameter in `load_dataset` ? Or an env variable ? Or use the value of `os.umask` ?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Ideally it should be a parameter in `load_dataset` but I'm not sure how important it is for the users (considering only important things should go into `load_dataset` parameters)


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
I think it's fairly important; for context, our team uses a shared file-system where many folks run experiments based on datasets that are cached by other users.

For example, I might start a training run, downloading a dataset. Then, a couple of days later, a collaborator using the same repository might want to use the same dataset on the same shared filesystem, but won't be able to under the default permissions.

Being able to specify directly in the top-level `load_dataset()` call seems important, but an equally valid option would be to just inherit from the running user's `umask` (this should probably be the default anyway).

So basically, argument that takes a custom set of permissions, and by default, use the running user's umask!


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Maybe let's start by defaulting to the user's umask !
Do you want to give it a try @bhavitvyamalik ?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Yeah sure! Instead of using default `0o644` should I first extract umask of current user and then use `os.umask` on it? We can do it inside `Dataset` class so that all folders/files created during the call use running user's umask


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
You can get the umask using `os.umask` and then I guess you can just use `os.chmod` as in your previous PR, but with the right permissions depending on the umask.


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
FWIW, we have this issue with other caches - e.g. `transformers` model files. So probably will need to backport this into `transformers` as well.

thanks @thomwolf for the pointer.


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
Hi @stas00,
For this should we use the same umask code in the respective model directory inside `TRANSFORMERS_CACHE`?


Only user permission of saved cache files, not group
Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?
That sounds very right to me, @bhavitvyamalik


Cannot load udpos subsets from xtreme dataset using load_dataset()
Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'

@lhoestq Adding "_" to the class labels in the dataset script will fix the issue.

The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.


Cannot load udpos subsets from xtreme dataset using load_dataset()
Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'

Hi ! Thanks for reporting @adzcodez

> @lhoestq Adding "_" to the class labels in the dataset script will fix the issue.
>
> The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

You're right: "_" should be added to the list of labels, and the examples must be sequences of tokens, not singles tokens.


Cannot load udpos subsets from xtreme dataset using load_dataset()
Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'

@lhoestq Can you please label this issue with the "good first issue" label? I'm not sure I'll find time to fix this.

To resolve it, the user should:
1. add `"_"` to the list of labels
2. transform the udpos subset to the conll format (I think the preprocessing logic can be borrowed from [the original repo](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204))
3. update the dummy data
4. update the dataset info
5. [optional] add info about the data fields structure of the udpos subset to the dataset readme


Cannot load udpos subsets from xtreme dataset using load_dataset()
Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'

I tried fixing this issue, but its working fine in the dev version : "1.6.2.dev0"

I think somebody already fixed it.