Exact and F1 score do not increase when fine-tunes XLM on the SQuAD dataset
Author: ZhengWeiH
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using XLM:

Language I am using the model on English:

The problem arise when using:
 [CUDA_VISIBLE_DEVICES=0 python run_squad.py   --model_type xlm   --model_name_or_path xlm-mlm-tlm-xnli15-1024   --do_train   --do_eval   --train_file $SQUAD_DIR/train-v1.1.json   --predict_file $SQUAD_DIR/dev-v1.1.json   --per_gpu_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 100   --max_seq_length 384   --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training  --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
![WeChat Image_20191112113346](https://user-images.githubusercontent.com/43492059/68639732-752ab380-0540-11ea-8543-7279a40d553a.png)

The tasks I am working on is an official SQUaD task.

All the parameters I set are in accordance with the example in the pytorch-transformers document. But Exact and F1 scores have barely increased. And the loss of each epoch also drops very slowly. Each epoch drops by about 0.01.
![68477603-d6b10080-0268-11ea-947f-995b157da8d3](https://user-images.githubusercontent.com/43492059/68640118-b96a8380-0541-11ea-8ac1-8e0e72cdc001.png)



Is there a problem with my parameter settings? Or do I need to adjust some parts of the model when I use XLM?



Comments:
I haven't yet tried XLM for SQUad but I did try to finetune it on a similar (private) dataset. I managed to make it converge however the F1 is much worse than that of BERT (~0.75 vs ~0.80). XLM training seems to be very learning rate sensitive so you may want to tinker with that a bit.

Thank you for your answer. 
Did you have any changes to the code released in pytorch-transformers when you test on your own dataset? Could you tell me if you did any changes? Or cloud you share your parameter settings when you test on your own data set?
Thank you.

> Thank you for your answer.
> Did you have any changes to the code released in pytorch-transformers when you test on your own dataset? Could you tell me if you did any changes? Or cloud you share your parameter settings when you test on your own data set?
> Thank you.

I think the only changes I made was adding language embeddings since I was finetuning `xlm-mlm-tlm-xnli15-1024` which requires that.

> > Thank you for your answer.
> > Did you have any changes to the code released in pytorch-transformers when you test on your own dataset? Could you tell me if you did any changes? Or cloud you share your parameter settings when you test on your own data set?
> > Thank you.
> 
> I think the only changes I made was adding language embeddings since I was finetuning `xlm-mlm-tlm-xnli15-1024` which requires that.

Hi,  I've tried all the XLM pre-training models and got random results. loss does not converge. can you help me with this problem? thanks @suicao @ZhengWeiH 

> > Thank you for your answer.
> > Did you have any changes to the code released in pytorch-transformers when you test on your own dataset? Could you tell me if you did any changes? Or cloud you share your parameter settings when you test on your own data set?
> > Thank you.
> 
> I think the only changes I made was adding language embeddings since I was finetuning `xlm-mlm-tlm-xnli15-1024` which requires that.

Thanks @suicao !
Where did you obtain language embeddings? Doesn't pre-trained `XLM` already come with a multilingual BPE embedding dictionary?

Ok, I think I may have found the issue in this particular model: it requires language IDs to be specified, but the default is `0`, which for `xlm-mlm-tlm-xnli15-1024` is Arabic, not English (which SQuAD is in). English is `4`, based on the `XLMConfig` class.
I think this will require passing an additional input tensor to all training and evaluation batches, that's all `4` `int64`s.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.