SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Only user permission of saved cache files, not group

Comment:
I get a permission denied error on https://github.com/huggingface/datasets/blob/b8363e0539c6f0cb5de49af32962cf2eb4c47395/src/datasets/arrow_dataset.py#L2799. I suspect I don't have permissions to change group permissions. I am mounting a GCP bucket through [gcsfuse](https://github.com/GoogleCloudPlatform/gcsfuse).
Issue: Only user permission of saved cache files, not group

Comment:
What @lhoestq is asking for is the full multi-line traceback  - it's almost never enough to show the last line - a full stack is needed to get the context. Thank you!

I wonder if a workaround is to try/except and then issue a warning if this fails?
Issue: Only user permission of saved cache files, not group

Comment:
Python files are stored in the modules caches, that you can modify by setting `HF_MODULES_CACHE` as well and set appropriate permissions

They are in different locations because the `HF_MODULES_CACHE` is added to the python path to be able to import the dataset scripts :)
Issue: Only user permission of saved cache files, not group

Comment:
Thanks a lot for the tip @lhoestq ! I didn't know about this extra cache - thanks :)
Issue: Cannot load udpos subsets from xtreme dataset using load_dataset()

Comment:
@lhoestq Adding "_" to the class labels in the dataset script will fix the issue.

The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.
Issue: Cannot load udpos subsets from xtreme dataset using load_dataset()

Comment:
Hi ! Thanks for reporting @adzcodez 


> @lhoestq Adding "_" to the class labels in the dataset script will fix the issue.
> 
> The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

You're right: "_" should be added to the list of labels, and the examples must be sequences of tokens, not singles tokens.
Issue: Cannot load udpos subsets from xtreme dataset using load_dataset()

Comment:
@lhoestq Can you please label this issue with the "good first issue" label? I'm not sure I'll find time to fix this.

To resolve it, the user should:
1. add `"_"` to the list of labels
2. transform the udpos subset to the conll format (I think the preprocessing logic can be borrowed from [the original repo](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204))
3. update the dummy data
4. update the dataset info
5. [optional] add info about the data fields structure of the udpos subset to the dataset readme
Issue: Cannot load udpos subsets from xtreme dataset using load_dataset()

Comment:
I tried fixing this issue, but its working fine in the dev version : "1.6.2.dev0"

I think somebody already fixed it.
Issue: Cannot load udpos subsets from xtreme dataset using load_dataset()

Comment:
Hi,

after #2326, the lines with pos tags equal to `"_"` are filtered out when generating the dataset, so this fixes the KeyError described above. However, the udpos subset should be in the conll format i.e. it should yield sequences of tokens and not single tokens, so it would be great to see this fixed (feel free to borrow the logic from [here](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204) if you decide to work on this).
Issue: Error while following docs to load the `ted_talks_iwslt` dataset

Comment:
This has been fixed in #2064  by @mariosasko (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)
Issue: Is it possible to convert a `tfds` to HuggingFace `dataset`?

Comment:
Hi! You can either save the TF dataset to one of the formats supported by datasets (`parquet`, `csv`, `json`, ...) or pass a generator function to `Dataset.from_generator` that yields its examples.
Issue: issue with opus100/en-fr dataset 

Comment:
@lhoestq  I also deleted the cache and redownload the file and still the same issue, I appreciate any help on this. thanks
Issue: issue with opus100/en-fr dataset 

Comment:
Here please find the minimal code to reproduce the issue @lhoestq  note this only happens with MT5TokenizerFast

```
from datasets import load_dataset
from transformers import MT5TokenizerFast

def get_tokenized_dataset(dataset_name, dataset_config_name, tokenizer):
    datasets = load_dataset(dataset_name, dataset_config_name, script_version="master")
    column_names = datasets["train"].column_names
    text_column_name = "translation"
    def process_dataset(datasets):
        def process_function(examples):
            lang = "fr"
            return {"src_texts": [example[lang] for example in examples[text_column_name]]}
        datasets = datasets.map(
            process_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True,
        )
        return datasets
    datasets = process_dataset(datasets)
    text_column_name = "src_texts"
    column_names = ["src_texts"]
    def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
    tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True
    )

if __name__ == "__main__":
     tokenizer_kwargs = {
        "cache_dir": None,
        "use_fast": True,
        "revision": "main",
        "use_auth_token": None
     }
     tokenizer = MT5TokenizerFast.from_pretrained("google/mt5-small", **tokenizer_kwargs)
     get_tokenized_dataset(dataset_name="opus100", dataset_config_name="en-fr", tokenizer=tokenizer)
~     
```
Issue: issue with opus100/en-fr dataset 

Comment:
as per https://github.com/huggingface/tokenizers/issues/626 this looks like to be the tokenizer bug, I therefore, reported it there https://github.com/huggingface/tokenizers/issues/626 and  I am closing this one.
Issue: is there a way to override a dataset object saved with save_to_disk?

Comment:
I tried this way, but when there is a mapping process to the dataset, it again uses a random cache name. atm, I am trying to use the following method by setting an exact cache file,

```
            dataset_with_embedding =csv_dataset.map(
                partial(self.embed, ctx_encoder=ctx_encoder, ctx_tokenizer=self.context_tokenizer),
                batched=True,
                batch_size=1,
                features=new_features,
                cache_file_name=cache_arrow_path,
                load_from_cache_file=False
            )
```
So here we set a cache_file_name , after this it uses the same  file name when saving again and again.
Issue: is there a way to override a dataset object saved with save_to_disk?

Comment:
I'm not sure I understand your issue, can you elaborate ?

`cache_file_name` is indeed an argument you can set to specify the cache file that will be used for the processed dataset. By default the file is named with something like `cache-<fingerprint>.arrow` where the fingerprint is a hash.
Issue: is there a way to override a dataset object saved with save_to_disk?

Comment:
Let's say I am updating a set of embedding in a dataset that is around 40GB inside a training loop every 500 steps (Ex: calculating the embeddings in updated ctx_encoder in RAG and saving it to the passage path).  So when we use **dataset_object.save_to_disk('passage_path_directory')** it will save the new dataset object every time with a random file name, especially when we do some transformations to dataset objects such as map or shards. This way, we keep collecting unwanted files that will eventually eat up all the disk space. 

But if we can save the dataset object every time by a single name like **data_shard_1.arrow**, it will automatically remove the previous file and save the new one in the same directory.  I found the above-mentioned code snippet useful to complete this task. 

Is this clear?
Issue: Could not find file for ZEST dataset

Comment:
This has been fixed in #2057 by @matt-peters (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)
Issue: Timit_asr dataset repeats examples

Comment:
Hi,

this was fixed by #1995, so you can wait for the next release or install the package directly from the master branch with the following command: 
```bash
pip install git+https://github.com/huggingface/datasets
```
Issue: Build custom dataset to fine-tune Wav2Vec2

Comment:
Sure you can use the json loader
```python
data_files = {"train": "path/to/your/train_data.json", "test": "path/to/your/test_data.json"}
train_dataset = load_dataset("json", data_files=data_files, split="train")
test_dataset = load_dataset("json", data_files=data_files, split="test")
```

You just need to make sure that the data contain the paths to the audio files.
If not, feel free to use `.map()` to add them.
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
I think faiss automatically sets the number of threads to use to build the index.
Can you check how many CPU cores are being used when you build the index in `use_own_knowleldge_dataset` as compared to this script ? Are there other programs running (maybe for rank>0) ?
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
Hi,
 I am running the add_faiss_index during the training process of the RAG from the master process (rank 0). But at the exact moment, I do not run any other process since I do it in every 5000 training steps. 
 
 I think what you say is correct. It depends on the number of CPU cores. I did an experiment to compare the time taken to finish the add_faiss_index process on use_own_knowleldge_dataset.py vs the training loop thing.  The training loop thing takes 40 mins more. It might be natural right? 
 
 
 at the moment it uses around 40 cores of a 96 core machine (I am fine-tuning the entire process).
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
Can you try to set the number of threads manually ?
If you set the same number of threads for both the `use_own_knowledge_dataset.py` and RAG training, it should take the same amount of time.
You can see how to set the number of thread in the faiss wiki: https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
Ok, I will report the details too soon. I am the first one on the list and currently add_index being computed for the 3rd time in the loop. Actually seems like the time is taken to complete each interaction is the same, but around 1 hour more compared to running it without the training loop. A the moment this takes 5hrs and 30 mins.  If there is any way to faster the process, an end-to-end rag will be perfect. So I will also try out with different thread numbers too.  

![image](https://user-images.githubusercontent.com/16892570/111453464-798c5f80-8778-11eb-86d0-19d212f58e38.png)
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
@lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
@lhoestq 

Hi,  I executed the **use_own_dataset.py** script independently and ask a few of my friends to run their programs in the HPC machine at the same time.  

 Once there are so many other processes are running the add_index function gets slows down naturally.  So basically the speed of the add_index depends entirely on the number of CPU processes.  Then I set the number of threads as you have mentioned and got actually the same time for RAG training and independat running.  So you are correct! :) 

 
 Then I added this [issue in Faiss repostiary](https://github.com/facebookresearch/faiss/issues/1767). I got an answer saying our current **IndexHNSWFlat** can get slow for 30 million vectors and it would be better to use alternatives. What do you think?
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
It's a matter of tradeoffs.
HSNW is fast at query time but takes some time to build.
A flat index is flat to build but is "slow" at query time.
An IVF index is probably a good choice for you: fast building and fast queries (but still slower queries than HSNW).

Note that for an IVF index you would need to have an `nprobe` parameter (number of cells to visit for one query, there are `nlist` in total) that is not too small in order to have good retrieval accuracy, but not too big otherwise the queries will take too much time. From the faiss documentation:
> The nprobe parameter is always a way of adjusting the tradeoff between speed and accuracy of the result. Setting nprobe = nlist gives the same result as the brute-force search (but slower).

From my experience with indexes on DPR embeddings, setting nprobe around 1/4 of nlist gives really good retrieval accuracy and there's no need to have a value higher than that (or you would need to brute-force in order to see a difference).
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
@lhoestq 

Thanks a lot for sharing all this prior knowledge. 

Just asking what would be a good nlist of parameters for 30 million embeddings?
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
When IVF is used alone, nlist should be between `4*sqrt(n)` and `16*sqrt(n)`.
For more details take a look at [this section of the Faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset)
Issue: add_faisis_index  gets very slow when doing it interatively  

Comment:
@lhoestq  Thanks a lot for the help you have given to solve this issue. As per my experiments, IVF index suits well for my case and it is a lot faster. The use of this can make the entire RAG end-to-end trainable lot faster.  So I will close this issue. Will do the final PR soon.
Issue: ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Comment:
Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?
They should both have a path to an arrow file

Also note that from #2025 concatenating datasets will no longer have such restrictions.
Issue: ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Comment:
Sure, thanks for the fast reply!

For dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`
For dataset B: `[]`

No clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'
Issue: ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Comment:
In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).

For now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with
```python
dataset = dataset.flatten_indices()
```
Issue: ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Comment:
Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot!
Issue: outdated dataset_infos.json might fail verifications

Comment:
Hi ! Thanks for reporting.

To update the dataset_infos.json you can run:
```
datasets-cli test ./datasets/doc2dial --all_configs --save_infos --ignore_verifications
```
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Dear @lhoestq for wikipedia dataset I also get the same error, I greatly appreciate if you could have a look into this dataset as well. Below please find the command to reproduce the error:

```
dataset = load_dataset("wikipedia", "20200501.bg")
print(dataset)
```

Your library is my only chance to be able training  the models at scale and I am grateful for your help.
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hi @dorost1234,
Try installing this library first, `pip install 'apache-beam[gcp]' --use-feature=2020-resolver` followed by loading dataset like this using beam runner.

`dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

 I also read in error stack trace that:

> Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc.

Worked perfectly fine after this (Ignore these warnings)

![image](https://user-images.githubusercontent.com/19718818/110908410-c7e2ce00-8334-11eb-8d10-7354359e9ec3.png)
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
For wikipedia dataset, looks like the files it's looking for are no longer available. For `bg`, I checked [here](https://dumps.wikimedia.org/bgwiki/). For this I think `dataset_infos.json` for this dataset has to made again? You'll have to load this dataset also using beam runner.
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hello @dorost1234,

Indeed, Wikipedia datasets need a lot of preprocessing and this is done using Apache Beam. That is the reason why it is required that you install Apache Beam in order to preform this preprocessing.

For some specific default parameters (English Wikipedia), Hugging Face has already preprocessed the dataset for you (and it is stored in the cloud). That is the reason why you do not get the error for English: the preprocessing is already done by HF and you just get the preprocessed dataset; Apache Beam is not required in that case.
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hi
I really appreciate if huggingface can kindly provide preprocessed
datasets, processing these datasets require sufficiently large resources
and I do not have unfortunately access to, and perhaps many others too.
thanks

On Fri, Mar 12, 2021 at 9:04 AM Albert Villanova del Moral <
***@***.***> wrote:

> Hello @dorost1234 <https://github.com/dorost1234>,
>
> Indeed, Wikipedia datasets need a lot of preprocessing and this is done
> using Apache Beam. That is the reason why it is required that you install
> Apache Beam in order to preform this preprocessing.
>
> For some specific default parameters (English Wikipedia), Hugging Face has
> already preprocessed the dataset for you (and it is stored in the cloud).
> That is the reason why you do not get the error for English: the
> preprocessing is already done by HF and you just get the preprocessed
> dataset; Apache Beam is not required in that case.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-797310899>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXACFQZAGMK4VGXRETTDHDI3ANCNFSM4ZA5R2UA>
> .
>
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hi everyone
thanks for the helpful pointers, I did it as @bhavitvyamalik suggested, for me this freezes on this command for several hours, 

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
`

Do you know how long this takes? Any specific requirements the machine should have? like very large memory or so? @lhoestq 

thanks
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
HI @dorost1234, 
The dataset size is 631.84 MiB so depending on your internet speed it'll take some time. You can monitor your internet speed meanwhile to see if it's downloading the dataset or not (use `nload` if you're using linux/mac to monitor the same). In my case it took around 3-4 mins. Since they haven't used `download_and_extract` here that's why there's no download progress bar.
Issue: wiki40b/wikipedia for almost all languages cannot be downloaded

Comment:
Hi
thanks, my internet speed should be good, but this really freezes for me, this is how I try to get this dataset:

`from datasets import load_dataset
dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

the output I see if different also from what you see after writing  this command:

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...`

do you have any idea why it might get freezed? anything I am missing @lhoestq @bhavitvyamalik. Do I need maybe to set anything special for apache-beam? 

thanks a lot 

On Tue, Mar 16, 2021 at 9:03 AM Bhavitvya Malik ***@***.***>
wrote:

> HI @dorost1234 <https://github.com/dorost1234>,
> The dataset size is 631.84 MiB so depending on your internet speed it'll
> take some time. You can monitor your internet speed meanwhile to see if
> it's downloading the dataset or not (use nload if you're using linux/mac
> to monitor the same). In my case it took around 3-4 mins. Since they
> haven't used download_and_extract here that's why there's no download
> progress bar.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800044303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMQIHNNLM2LGG6QKZ73TD4GDJANCNFSM4ZA5R2UA>
> .
>