SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: How to disable making arrow tables in load_dataset ?

Comment:
Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.
Issue: How to disable making arrow tables in load_dataset ?

Comment:
@lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?
Issue: How to disable making arrow tables in load_dataset ?

Comment:
We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub
Issue: How to disable making arrow tables in load_dataset ?

Comment:
Hi! You can now use `Dataset.push_to_hub` to store preprocessed files on the Hub.

And to avoid downloading preprocessed files, you can use streaming by setting `streaming=True` in `load_dataset`.
Issue: Add documentaton for dataset README.md files

Comment:
Hi ! We are using the [datasets-tagging app](https://github.com/huggingface/datasets-tagging) to select the tags to add.

We are also adding the full list of tags in #2107 
This covers multilinguality, language_creators, licenses, size_categories and task_categories.

In general if you want to add a tag that doesn't exist (for example for a custom license) you must make it start with `other-` and then a custom tag name.

edit (@theo-m) if you ever find yourself resorting to adding an `other-*` tag, please do ping us somewhere so we can think about adding it to the "official" list :)
Issue: Add documentaton for dataset README.md files

Comment:
@lhoestq hmm - ok thanks for the answer.
To be honest I am not sure if this issue can be closed now.
I just wanted to point out that this should either be documented or linked in the documentation.
If you feel like it is (will be) please just close this.
Issue: Add documentaton for dataset README.md files

Comment:
We're still working on the validation+documentation in this.
Feel free to keep this issue open till we've added them
Issue: Add documentaton for dataset README.md files

Comment:
Hi ! There's the tagging app at https://huggingface.co/datasets/tagging/ that you can use.
It shows the list of all the tags you can use.

It is based on all the tag sets defined in this folder:
https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources
Issue: Add documentaton for dataset README.md files

Comment:
I don't think so. Feel free to take a look at the tags of other models (example [here](https://huggingface.co/bert-base-uncased/blob/main/README.md)). But we should definitely have some docs or an app to write the tags. Feel free to open an issue in the `transformers` repo or in the `huggingface_hub` repo so we can discuss this
Issue: Add documentaton for dataset README.md files

Comment:
When modifying a README file, the Hub now displays a special UI with allowed values (see https://huggingface.co/docs/datasets/main/en/upload_dataset#create-a-dataset-card).
Issue: `concatenate_datasets` throws error when changing the order of datasets to concatenate

Comment:
Hi,

this bug is related to `Dataset.{remove_columns, rename_column, flatten}` not propagating the change to the schema metadata when the info features are updated, so this line is the culprit:
```python
common_voice_train = common_voice_train.remove_columns(['client_id', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment'])

``` 
The order is important because the resulting dataset inherits the schema metadata of the first dataset passed to the `concatenate_datasets(...)` function (`pa.concat_tables` [docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)). I'll try to fix this ASAP.
Issue: Multidimensional arrays in a Dataset

Comment:
Hi !

This is actually supported ! but not yet in `from_pandas`.
You can use `from_dict` for now instead:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset)
```

This will work but to use it with the torch formatter you must specify the `Array2D` feature type in order to tell the shape:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset, features=Features({
    "bbox": Array2D(shape=(3, 4), dtype="int64"),
    "input_ids": Value("int64")
}))
dataset.set_format("torch")
print(dataset[0]['bbox'])
# tensor([[1, 2, 3, 4],
#         [1, 2, 3, 4],
#         [1, 2, 3, 4]])
```
If you don't specify the `Array2D` feature type, then the inferred type will be Sequence(Sequence(Value("int64"))) and therefore the torch formatter will return list of tensors
Issue: Multidimensional arrays in a Dataset

Comment:
Thanks for the explanation. 
With my original DataFrame, I did
```
dataset = dataset.to_dict("list")
```
and then the rest of the transformation from dictionary works just fine.
Issue: MemoryError when computing WER metric

Comment:
Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94
Issue: MemoryError when computing WER metric

Comment:
Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.
Issue: MemoryError when computing WER metric

Comment:
I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)
Issue: MemoryError when computing WER metric

Comment:
Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago
Issue: MemoryError when computing WER metric

Comment:
Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?
Issue: MemoryError when computing WER metric

Comment:
Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?
Issue: MemoryError when computing WER metric

Comment:
One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each. 

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.
Issue: MemoryError when computing WER metric

Comment:
Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.
Issue: MemoryError when computing WER metric

Comment:
@lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

-> 
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
``` 

if i do 
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0
Issue: Issue: Dataset download error

Comment:
Hi @XuhuiZhou, thanks for reporting this issue. 

Indeed, the old links are no longer valid (404 Not Found error), and the script must be updated with the new links to Google Drive.
Issue: Issue: Dataset download error

Comment:
It would be nice to update the urls indeed !

To do this, you just need to replace the urls in `iwslt2017.py` and then update the dataset_infos.json file with
```
datasets-cli test ./datasets/iwslt2017 --all_configs --save_infos --ignore_verifications
```
Issue: Issue: Dataset download error

Comment:
Is this a command to update my local files or fix the file Github repo in general? (I am not so familiar with the datasets-cli command here)

I also took a brief look at the **Sharing your dataset** section, looks like I could fix that locally and push it to the repo? I guess we are "canonical" category?
Issue: Issue: Dataset download error

Comment:
This command will update your local file. Then you can open a Pull Request to push your fix to the github repo :)
And yes you are right, it is a "canonical" dataset, i.e. a dataset script defined in this github repo (as opposed to dataset repositories of users on the huggingface hub)
Issue: Issue: Dataset download error

Comment:
Hi, thanks for the answer. 

I gave a try to the problem today. But I encountered an upload error: 

```
git push -u origin fix_link_iwslt
Enter passphrase for key '/home2/xuhuizh/.ssh/id_rsa': 
ERROR: Permission to huggingface/datasets.git denied to XuhuiZhou.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Any insight here? 

By the way, when I run the datasets-cli command, it shows the following error, but does not seem to be the error coming from `iwslt.py`

```
Traceback (most recent call last):
  File "/home2/xuhuizh/anaconda3/envs/UMT/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/datasets_cli.py", line 35, in main
    service.run()
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/test.py", line 141, in run
    try_from_hf_gcs=False,
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 639, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/xuhuizh/projects/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz'}
```
Issue: Issue: Dataset download error

Comment:
Hi ! To create a PR on this repo your must fork it and create a branch on your fork. See how to fork the repo [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#start-by-preparing-your-environment).
And to make the command work without the `ExpectedMoreDownloadedFiles` error, you just need to use the `--ignore_verifications` flag.
Issue: Issue: Dataset download error

Comment:
Hi @XuhuiZhou,

As @lhoestq has well explained, you need to fork HF's repository, create a feature branch in your fork, push your changes to it and then open a Pull Request to HF's upstream repository. This is so because at HuggingFace Datasets we follow a development model called "Fork and Pull Model". You can find more information here:
- [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
- [Forking Projects](https://guides.github.com/activities/forking/)

Alternatively, if you find all these steps too complicated, you can use the GitHub official command line tool: [GitHub CLI](https://cli.github.com/). Once installed, in order to create a Pull Request, you only need to use this command:
```shell
gh pr create --web
```
This utility will automatically create the fork, push your changes and open a Pull Request, under the hood.
Issue: ConnectionError: Couldn't reach common_voice.py

Comment:
Hi @LifaSun, thanks for reporting this issue.

Sometimes, GitHub has some connectivity problems. Could you confirm that the problem persists?
Issue: ArrowInvalid issue for squad v2 dataset

Comment:
Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.
Issue: PyTorch not available error on SageMaker GPU docker though it is installed 

Comment:
Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`
Issue: PyTorch not available error on SageMaker GPU docker though it is installed 

Comment:
Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`. 
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.
Issue: PyTorch not available error on SageMaker GPU docker though it is installed 

Comment:
Could paste the code you use the start your training job and the fine-tuning script you run?
Issue: Multiprocessing windows error

Comment:
Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..
Issue: Multiprocessing windows error

Comment:
```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')


updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```
Issue: Multiprocessing windows error

Comment:
Thanks this is really helpful !
I'll try to reproduce on my side and come back to you
Issue: Multiprocessing windows error

Comment:
if __name__ == '__main__':


This line before calling the map function stops the error but the script still repeats endless
Issue: Multiprocessing windows error

Comment:
Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?
Issue: Multiprocessing windows error

Comment:
Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?
Issue: Multiprocessing windows error

Comment:
Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again
Issue: Only user permission of saved cache files, not group

Comment:
Hi ! Thanks for reporting.

Currently there's no way to specify this.

When loading/processing a dataset, the arrow file is written using a temporary file. Then once writing is finished, it's moved to the cache directory (using `shutil.move` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646))

That means it keeps the permissions specified by the `tempfile.NamedTemporaryFile` object, i.e. `-rw-------` instead of `-rw-r--r--`. Improving this could be a nice first contribution to the library :)
Issue: Only user permission of saved cache files, not group

Comment:
Hi @lhoestq,
I looked into this and yes you're right. The `NamedTemporaryFile` is always created with mode 0600, which prevents group from reading the file. Should we change the permissions of `tmp_file.name` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1871) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1590), post creation to 0644 inorder for group and others to read it?
Issue: Only user permission of saved cache files, not group

Comment:
Good idea :) we could even update the permissions after the file has been moved by shutil.move [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1899) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646) actually.
Apparently they set the default 0600 for temporary files for security reasons, so let's update the umask only after the file has been moved
Issue: Only user permission of saved cache files, not group

Comment:
Would it be possible to actually set the umask based on a user provided argument? For example, a popular usecase my team has is using a shared file-system for processing datasets. This may involve writing/deleting other files, or changing filenames, which a -rw-r--r-- wouldn't fix.
Issue: Only user permission of saved cache files, not group

Comment:
Note that you can get the cache files of a dataset with the `cache_files` attributes.
Then you can `chmod` those files and all the other cache files in the same directory.

Moreover we can probably keep the same permissions after each transform. This way you just need to set the permissions once after doing `load_dataset` for example, and then all the new transformed cached files will have the same permissions.
What do you think ?
Issue: Only user permission of saved cache files, not group

Comment:
This means we'll check the permissions of other `cache_files` already created for a dataset before setting permissions for new  `cache_files`?
Issue: Only user permission of saved cache files, not group

Comment:
> This way you just need to set the permissions once after doing load_dataset for example, and then all the new transformed cached files will have the same permissions.

I was referring to this. Ensuring that newly generated `cache_files` have the same permissions
Issue: Only user permission of saved cache files, not group

Comment:
Yes exactly

I imagine users can first do `load_dataset`, then chmod on the arrow files. After that all the new cache files could have the same permissions as the first arrow files. Opinions on this ?
Issue: Only user permission of saved cache files, not group

Comment:
Sounds nice but I feel this is a sub-part of the approach mentioned by @siddk. Instead of letting the user set new permissions by itself first and then making sure newly generated files have same permissions why don't we ask the user initially only what they want? What are your thoughts?
Issue: Only user permission of saved cache files, not group

Comment:
Yes sounds good. Should this be a parameter in `load_dataset` ? Or an env variable ? Or use the value of `os.umask` ?
Issue: Only user permission of saved cache files, not group

Comment:
Ideally it should be a parameter in `load_dataset` but I'm not sure how important it is for the users (considering only important things should go into `load_dataset` parameters)
Issue: Only user permission of saved cache files, not group

Comment:
I think it's fairly important; for context, our team uses a shared file-system where many folks run experiments based on datasets that are cached by other users.

For example, I might start a training run, downloading a dataset. Then, a couple of days later, a collaborator using the same repository might want to use the same dataset on the same shared filesystem, but won't be able to under the default permissions.

Being able to specify directly in the top-level `load_dataset()` call seems important, but an equally valid option would be to just inherit from the running user's `umask` (this should probably be the default anyway).

So basically, argument that takes a custom set of permissions, and by default, use the running user's umask!
Issue: Only user permission of saved cache files, not group

Comment:
Maybe let's start by defaulting to the user's umask !
Do you want to give it a try @bhavitvyamalik ?
Issue: Only user permission of saved cache files, not group

Comment:
Yeah sure! Instead of using default `0o644` should I first extract umask of current user and then use `os.umask` on it? We can do it inside `Dataset` class so that all folders/files created during the call use running user's umask
Issue: Only user permission of saved cache files, not group

Comment:
You can get the umask using `os.umask` and then I guess you can just use `os.chmod` as in your previous PR, but with the right permissions depending on the umask.
Issue: Only user permission of saved cache files, not group

Comment:
FWIW, we have this issue with other caches - e.g. `transformers` model files. So probably will need to backport this into `transformers` as well.

thanks @thomwolf for the pointer.
Issue: Only user permission of saved cache files, not group

Comment:
Hi @stas00,
For this should we use the same umask code in the respective model directory inside `TRANSFORMERS_CACHE`?
Issue: Only user permission of saved cache files, not group

Comment:
The cluster I am working on does not allow me to change the permission of the files with os.chmod. I was wondering if there is any workaround for this? My cache is in a GCP bucket and I can't change file permissions once I mount it.