dynamic masking for RoBERTa model
Author: mingyang3github
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->
<!-- Description of your issue -->
I saw that "dynamic masking" was mentioned on the README file for language modeling:
"In accordance to the RoBERTa paper, we use dynamic masking rather than static masking. The model may, therefore, converge slightly slower (over-fitting takes more epochs)."

I couldn't find which class this method is implemented in and how to enable this feature during pre-training using the Trainer class. Could someone please help me?

Thank you very much in advance.


Comments:
Hi @mingyang3github 
masking is implemented in ` DataCollatorForLanguageModeling`'s `mask_tokens` method, [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L103)

For pre-training you can use the language modeling script, it takes of the dataset and masking.  

> Hi @mingyang3github
> masking is implemented in ` DataCollatorForLanguageModeling`'s `mask_tokens` method, [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L103)
> 
> For pre-training you can use the language modeling script, it takes of the dataset and masking.

Hi @patil-suraj ,
Thank you for your reply. If I understand this correctly, DataCollatorForLanguageModeling's method is static masking, right?

Based on RoBERTa model's paper, the dynamic masking refers to "training data was duplicated 10 times so that each sequence is masked in 10 different ways over the 40 epochs of training."
Has this method implemented been implemented in DataCollatorForLanguageModeling ?

It is dynamic masking, as the masking is handled in the collate function, the same examples gets masked differently. Because every batch first goes through collate function before the batch is returned from the loader, so each time the examples goes through collate it gets masked differently.

Hi @patil-suraj , is that mean we **always use dynamic masking**,  when use `DataCollatorForLanguageModeling`, no matter pre-training for Bert, RoBERTa or else.

@buaapengbo
Yes, the `DataCollatorForLanguageModeling` always does dynamic masking no matter the model

I thought I responded, but it turns out I didn't. Sorry.
@patil-suraj Thank you very much! That makes sense! 


Thanks for the answer.  For the original roberta, they seem to use the same mask per sentence 4 times during training and 10 different masks for the same sentence during the whole training. But this function always returns different masks for the same sentence during training. This means the model always receives different masks for the same sentence during the whole training process right?