The acc of RACE is always low by roberta model
Author: csliangchen
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
My scripts like follows:
python ./examples/run_multiple_choice.py \
        --model_type roberta \
        --task_name race \
        --model_name_or_path roberta-base \
        --do_train \
        --do_eval \
        --do_lower_case \
        --do_test \
        --data_dir $RACE_DIR \
        --learning_rate 5e-5 \
        --num_train_epochs 5 \
        --max_seq_length 512 \
        --output_dir ./race_base_1115 \
        --per_gpu_eval_batch_size=2 \
        --per_gpu_train_batch_size=2 \
        --gradient_accumulation_steps 1 \
        --overwrite_output_dir \
        --logging_steps 1000\
        --save_steps 1000\
        --evaluate_during_training \

but my acc result is always under 30%, both test and evaluate :
eval_acc = 0.2833400891771382
eval_loss = 1.386294308898901

Maybe it be that the token length is too long?  More than 3W input sentences' length is longer than 512.  It would truncated context.
But this problem didn't happen in fairseq(original Roberta code). i wanna know why.
Ask for help. Thanks a lot. 

Comments:
Same.
model           =roberta-base
total batch size=8
train num epochs=5
fp16            =False
max seq length  =512
eval_acc = 0.29568242275424594
eval_loss = 1.3862943781258896
@spolu 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


In my experience, it's because loss not getting down.
refer to fb repo issue https://github.com/pytorch/fairseq/issues/1114#issue-490144245
It seems that learning rate needs to be smaller and require larger batch size

Use the config below:
```
python ./examples/run_multiple_choice.py --model_type roberta --task_name race --model_name_or_path roberta-base-openai-detector  --do_eval  --data_dir $RACE_DIR --learning_rate 1e-5 --num_train_epochs 10 --max_seq_length 512 --output_dir ./roberta-base-openai-race --model_name_or_path ./roberta-base-openai-race/ --per_gpu_eval_batch_size=9 --per_gpu_train_batch_size=9 --gradient_accumulation_steps 2 --save_steps 4882 --eval_all_checkpoints --fp16 --seed 77 --do_lower_case
```

I can get 
***** Eval results checkpoint-24410 is test:False *****                                                                                                                                                                                                                 
eval_acc = 0.6758747697974218                                                                                                                                                                                                                                         
eval_loss = 0.9408586282721337  

It‘s getting close, but still faraway from current result......

Thank you very much, I will try it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.