Inquiry about Batch Size and Total Steps Used for MLM Task on xlm-roberta?

#26

by jrjrjrjr - opened Feb 16

Feb 16

Hello,

I'm currently exploring the xlm-roberta model and have a specific interest in understanding some of the training details, particularly regarding the Masked Language Model (MLM) task. Could you provide insights or details about the batch size and total number of steps used during the training process for the MLM task on XLM-R and XLM-R-base?

ybelkada

Feb 21

Hi @jrjrjrjr !
Thanks for the issue
You can read more about the training protocol of that model in the original repository: https://github.com/facebookresearch/fairseq/tree/main/examples/xlmr and you can also try out MLM task yourself in one of our official scripts: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment