Error when Fine-tuning XLM on SQuA
Author: ZhengWeiH
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (XLM):

Language I am using the model on (English and Chinese):

The problem arise when using:

[CUDA_VISIBLE_DEVICES=2 python run_squad.py   --model_type xlm  --model_name_or_path xlm-mlm-tlm-xnli15-1024   --do_train   --do_eval     --train_file $SQUAD_DIR/Crosslingual_QZH_train.json   --predict_file $SQUAD_DIR/Crosslingual_QZH_valid.json   --per_gpu_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 100   --max_seq_length 384   --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training  --output_dir /home/weihua/Sqad/transformers/xlm_out/ ] 


error:
Traceback (most recent call last):
  File "run_squad.py", line 569, in <module>
    main()
  File "run_squad.py", line 515, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 179, in train
    results = evaluate(args, model, tokenizer)
  File "run_squad.py", line 275, in evaluate
    args.version_2_with_negative, tokenizer, args.verbose_logging)
  File "/home/weihua/Sqad/transformers/examples/utils_squad.py", line 814, in write_predictions_extended
    final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
AttributeError: 'XLMTokenizer' object has no attribute 'do_lower_case'

![WeChat Image_20191108182243](https://user-images.githubusercontent.com/43492059/68469281-d0188e00-0254-11ea-928a-8ab4a7140838.png)
![WeChat Image_20191108182248](https://user-images.githubusercontent.com/43492059/68469288-d3ac1500-0254-11ea-9116-9f0dd15b65cd.png)


How can I deal with this problem?




Comments:
> ##  Bug
> Model I am using (XLM):
> 
> Language I am using the model on (English and Chinese):
> 
> The problem arise when using:
> 
> [CUDA_VISIBLE_DEVICES=2 python run_squad.py --model_type xlm --model_name_or_path xlm-mlm-tlm-xnli15-1024 --do_train --do_eval --train_file $SQUAD_DIR/Crosslingual_QZH_train.json --predict_file $SQUAD_DIR/Crosslingual_QZH_valid.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 100 --max_seq_length 384 --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
> 
> error:
> Traceback (most recent call last):
> File "run_squad.py", line 569, in 
> main()
> File "run_squad.py", line 515, in main
> global_step, tr_loss = train(args, train_dataset, model, tokenizer)
> File "run_squad.py", line 179, in train
> results = evaluate(args, model, tokenizer)
> File "run_squad.py", line 275, in evaluate
> args.version_2_with_negative, tokenizer, args.verbose_logging)
> File "/home/weihua/Sqad/transformers/examples/utils_squad.py", line 814, in write_predictions_extended
> final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
> AttributeError: 'XLMTokenizer' object has no attribute 'do_lower_case'
> 
> ![WeChat Image_20191108182243](https://user-images.githubusercontent.com/43492059/68469281-d0188e00-0254-11ea-928a-8ab4a7140838.png)
> ![WeChat Image_20191108182248](https://user-images.githubusercontent.com/43492059/68469288-d3ac1500-0254-11ea-9116-9f0dd15b65cd.png)
> 
> How can I deal with this problem?

Look [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_xlm.py). This is the source code of the XLM tokenizer. In the ___init__()_ method, you can see that the _do_lower_case_ parameter doesn't exist, but it exists the **do_lowercase_and_remove_accent** parameter.

Said this, I tell you a solution to your problem (a wordaround).
If you see the source code of [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py) at rows 487-489, you see that here the tokenizer (different according to the model chosen by you - in your case XLM) is loaded, passing the _do_lower_case_ parameter. But as I said before, the tokenizer for XLM model accepts a different parameter. So, you can insert an if/else statement (in pseudo-code):
```
if args.model_type == 'XLM':
    tokenizer = tokenizer_class.from_pretrained(..., do_lower_case_and_remove_accent=args.do_lower_case, ...)
else:
    tokenizer = tokenizer_class.from_pretrained(..., do_lower_case=args.do_lower_case, ...)
```
N.B. if you want to be more comfortable of my solution to your problem, please see the source code of the `tokenization_xlm.py` in your HuggingFace version whether it contains the _do_lowercase_and_remove_accent_ parameter and not the _do_lowercase_ parameter. 

Keep us updated on your problem! Good luck

Actually, I just replace the line 814 in utils_squad.py with 'final_text = get_final_text(tok_text, orig_text,verbose_logging)'. It means I no longer pass ‘tokenizer.do_lower_case’ to this function. Because I find that in the tokenization_xlm.py, the parameter 'do_lower_case_and_remove_accent' is already set. Although this allows my program to run, it does not give good results. So I suspect that my change is wrong.
I also tried to follow your suggestion above. But the results did not improve. 
This is the result of the 700th step. 
![WeChat Image_20191108204610](https://user-images.githubusercontent.com/43492059/68477603-d6b10080-0268-11ea-947f-995b157da8d3.png)
The exact and f1 is very low. I used bert-base-multilingual-cased to do SQuAD with the same data set before. The exact and f1 increased very fast. At around 200th step, the f1 can reach 40+, and exact is about 30.
Is it wrong with my parameter settings that caused these two scores to rise slowly when I use XLM?
Or is it necessary to take longer to finetune the SQuAD task in XLM than in BERT?
This is my first time to do the SQuAD. I sincerely hope that you can give me some advice.
Thank you.

I also tried to train and test on the official dataset,
that is 
![WeChat Image_20191109175459](https://user-images.githubusercontent.com/43492059/68526753-13890000-031a-11ea-9f38-e9841bb0ac42.png)
in the examples document.

but the situation was the same as what I got on my own dataset.
At the same time, I find that when I use the XLM model with learning-rate= 3e-5, the loss decreases very slowly. 
Is this due to the inappropriateness of my parameter settings? Or is it because the loss function of the model needs to be changed?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.