XLNet-large-cased: hyper-parameters for fine-tuning on SST-2
Author: avostryakov
Pull request: no
Labels: name: wontfix

I tried to finetune XLNet on one of the classification tasks from GLUE (Ubuntu, GPU Titan RTX, CUDA 10.0, pytorch 1.1):

export GLUE_DIR=/path/to/glue

python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --do_eval   \
    --task_name=sst-2     \
    --data_dir=${GLUE_DIR}/SST-2  \
    --output_dir=./proc_data/sst-2   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=1200  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120

Training and evaluation work without errors but it looks like accuracy doesn't increase during training, I evaluated every 500 steps:

07/16/2019 22:29:30 - INFO - __main__ -   ***** Eval results  *****
07/16/2019 22:29:30 - INFO - __main__ -     acc = 0.5091743119266054
                                                                                                                                                                                                                              
07/16/2019 22:32:16 - INFO - __main__ -   Loading features from cached file glue_data/SST-2/cached_dev_xlnet-large-cased_128_sst-2              | 999/8419 [05:37<41:47,  2.96it/s]
07/16/2019 22:32:17 - INFO - __main__ -   ***** Running evaluation  *****
07/16/2019 22:32:17 - INFO - __main__ -     Num examples = 872
07/16/2019 22:32:17 - INFO - __main__ -     Batch size = 8
                                                                                                                                                                                                                              
07/16/2019 22:32:25 - INFO - __main__ -   ***** Eval results  *****
07/16/2019 22:32:25 - INFO - __main__ -     acc = 0.5091743119266054

Finally the same acc:

07/16/2019 22:33:59 - INFO - __main__ -   ***** Eval results  *****
07/16/2019 22:33:59 - INFO - __main__ -     acc = 0.5091743119266054

The same situation is with my own classification dataset. Accuracy wasn't changed during training. Something is wrong with finetuning of XLNet

Comments:
I also tried to finetune xlnet base on squad 2.0 but the numbers on dev are pretty bad
`Results: {'exact': 3.0405120862461046, 'f1': 6.947601433150003, 'total': 11873, 'HasAns_exact': 6.056005398110662, 'HasAns_f1': 13.881388632893048, 'HasAns_total': 5928, 'NoAns_exact': 0.0336417157275021, 'NoAns_f1': 0.0336417157275021, 'NoAns_total': 5945, 'best_exact': 50.07159100480081, 'best_exact_thresh': 0.0, 'best_f1': 50.07159100480081, 'best_f1_thresh': 0.0}`

I suspect something is wrong with the evaluation code. Looking into it now.

@tbright17 Nothing wrong with evaluation. Accuracy and evaluation loss aren't changed during training. I used my own evaluation script, I used old BertAdam or OpenAIAdam optimizers without success.
@thomwolf Can you help?

I'll give a look, I've only tested XLNet on STS-B for the moment. You should check the hyper-parameters as well, they probably won't be the same as the ones of STS-B (some are mentioned in the XLNet paper).

First thing that comes to mind is that SST-2 is ~10 times bigger than STS-B (see the [GLUE paper](https://arxiv.org/abs/1804.07461)) so you need to increase the number of training step a lot if you want to do at least one full epoch on SST-2 training dataset (here you use the value for STS-B). And you should probably do several epochs, e.g. we do 6-7 epochs on STS-B). Check some examples of recommended hyper-parameters table 8 of the [xlnet paper](http://arxiv.org/abs/1906.08237).

You can also directly specify the number of epochs instead of the maximum number of steps in the script. You can see all the hyper-parameters of the script with `python ./run_glue.py --help`.

> First thing that comes to mind is that SST-2 is ~10 times bigger than STS-B (see the [GLUE paper](https://arxiv.org/abs/1804.07461)) so you need to increase the number of training step a lot if you want to do at least one full epoch on SST-2 training dataset (here you use the value for STS-B). And you should probably do several epochs, e.g. we do 6-7 epochs on STS-B). Check some examples of recommended hyper-parameters table 8 of the [xlnet paper](http://arxiv.org/abs/1906.08237).
> 
> You can also directly specify the number of epochs instead of the maximum number of steps in the script. You can see all the hyper-parameters of the script with `python ./run_glue.py --help`.

I trained STS-B task with the same problem. You can see the following output with evaluation of every 100 steps (I added train and evaluation loss in output):

```
07/17/2019 13:09:55 - INFO - __main__ -   ***** Running evaluation  *****
07/17/2019 13:09:55 - INFO - __main__ -     Num examples = 1500
07/17/2019 13:09:55 - INFO - __main__ -     Batch size = 8
07/17/2019 13:10:09 - INFO - __main__ -   ***** Eval results  *****
07/17/2019 13:10:09 - INFO - __main__ -     corr = -0.05367882385720809
07/17/2019 13:10:09 - INFO - __main__ -     eval_loss = 2.8412214481133096##################################################################################################################| 188/188 [00:14<00:00, 13.41it/s]
07/17/2019 13:10:09 - INFO - __main__ -     pearson = -0.041275192
07/17/2019 13:10:09 - INFO - __main__ -     spearmanr = -0.06608245566229025
07/17/2019 13:10:09 - INFO - __main__ -   Training loss: 307.258519500494
                                                                                                                                                                                                                              07/17/2019 13:10:41 - INFO - __main__ -   Loading features from cached file ...glue_data/STS-B/cached_dev_xlnet-large-cased_128_sts-b               | 199/719 [01:18<03:25,  2.53it/s]
07/17/2019 13:10:41 - INFO - __main__ -   ***** Running evaluation  *****
07/17/2019 13:10:41 - INFO - __main__ -     Num examples = 1500
07/17/2019 13:10:41 - INFO - __main__ -     Batch size = 8
07/17/2019 13:10:56 - INFO - __main__ -   ***** Eval results  *****
07/17/2019 13:10:56 - INFO - __main__ -     corr = 0.13943037650184956
07/17/2019 13:10:56 - INFO - __main__ -     eval_loss = 2.3762524007482733##################################################################################################################| 188/188 [00:14<00:00, 13.29it/s]
07/17/2019 13:10:56 - INFO - __main__ -     pearson = 0.13502572
07/17/2019 13:10:56 - INFO - __main__ -     spearmanr = 0.1438350282350605
07/17/2019 13:10:56 - INFO - __main__ -   Training loss: 533.9101385176182
                                                                                                                                                                                                                              07/17/2019 13:11:28 - INFO - __main__ -   Loading features from cached file .../glue_data/STS-B/cached_dev_xlnet-large-cased_128_sts-b               | 299/719 [02:05<02:56,  2.39it/s]
07/17/2019 13:11:28 - INFO - __main__ -   ***** Running evaluation  *****
07/17/2019 13:11:28 - INFO - __main__ -     Num examples = 1500
07/17/2019 13:11:28 - INFO - __main__ -     Batch size = 8
07/17/2019 13:11:42 - INFO - __main__ -   ***** Eval results  *****
07/17/2019 13:11:42 - INFO - __main__ -     corr = -0.0830871973267994
07/17/2019 13:11:42 - INFO - __main__ -     eval_loss = 2.5565993221516305##################################################################################################################| 188/188 [00:14<00:00, 13.20it/s]
07/17/2019 13:11:42 - INFO - __main__ -     pearson = -0.08915693
07/17/2019 13:11:42 - INFO - __main__ -     spearmanr = -0.077017461524765
07/17/2019 13:11:42 - INFO - __main__ -   Training loss: 761.6802722513676
                                                                                                                                                                                                                              07/17/2019 13:12:15 - INFO - __main__ -   Loading features from cached file .../glue_data/STS-B/cached_dev_xlnet-large-cased_128_sts-b               | 399/719 [02:52<02:18,  2.32it/s]
07/17/2019 13:12:15 - INFO - __main__ -   ***** Running evaluation  *****
07/17/2019 13:12:15 - INFO - __main__ -     Num examples = 1500
07/17/2019 13:12:15 - INFO - __main__ -     Batch size = 8
07/17/2019 13:12:29 - INFO - __main__ -   ***** Eval results  *****
07/17/2019 13:12:29 - INFO - __main__ -     corr = -0.08715267932681456
07/17/2019 13:12:29 - INFO - __main__ -     eval_loss = 2.398741365113157###################################################################################################################| 188/188 [00:14<00:00, 13.12it/s]
07/17/2019 13:12:29 - INFO - __main__ -     pearson = -0.08428703
07/17/2019 13:12:29 - INFO - __main__ -     spearmanr = -0.09001832616862088
07/17/2019 13:12:29 - INFO - __main__ -   Training loss: 974.8287971913815
```

How you can see training loss is increasing, eval loss is almost the same, other metrics fluctuate around 0.

@thomwolf So, it looks like training is happening but in opposite direction for some reason

Maybe you haven't fully read the [explanation](https://github.com/huggingface/pytorch-transformers#fine-tuning-xlnet-model-on-the-sts-b-regression-task) accompanying the STS-B example in the readme?

It says "On this machine we thus have a batch size of 32, please increase `gradient_accumulation_steps` to reach the same batch size if you have a smaller machine."

@avostryakov Did you try to reduce the learning rate? I had a similar issue training with the TensorFlow version XLNet on only one GPU. I tried reducing the learning rate from 5e-5 to 1e-5, and it worked. Wish this can help you.

@thomwolf @tbright17 I got similar numbers like you Squad 2.0. Seems that the model probably isn't learning much. I'll print out the losses to explore. Also should we change the LR as well? 
 : the best I got with fine-tuning on Squad 2.0 with a `train_batch_size=8` and `gas=1` all others are default on a single v100 gpu was:
 `07/16/2019 16:21:43 - INFO - __main__ -   Results: {'exact': 26.438136949380947, 'f1': 28.470459931964722, 'total': 11873, 'HasAns_exact': 0.08434547908232119, 'HasAns_f1': 4.154819630940996, 'HasAns_total': 5928, 'NoAns_exact': 52.716568544995795, 'NoAns_f1': 52.716568544995795, 'NoAns_total': 5945, 'best_exact': 50.07159100480081, 'best_exact_thresh': 0.0, 'best_f1': 50.07159100480081, 'best_f1_thresh': 0.0}`

May also be a problem of batch size, the authors use a batch size between 32 and 128 in the paper. 

What effective batch size do you have (printed during training)?

While we reproduce the official XLNet number on STS-B, I still have to work a bit on the SQuAD example for XLNet, the XLNet authors used a complex pre- and post-processing of the data (smarter than Bert's) that I haven't fully integrated into our `run_squad` example yet.

> Maybe you haven't fully read the [explanation accompanying the STS-B example in the readme](https://github.com/huggingface/pytorch-transformers#fine-tuning-xlnet-model-on-the-sts-b-regression-task)?
> 
> It says "On this machine we thus have a batch size of 32, please increase `gradient_accumulation_steps` to reach the same batch size if you have a smaller machine."

@thomwolf  You are right, STS-B started to train with batch size 32 and gradient_accumulation_steps = 2. Now I'm wondering why it so heavily depends on batch size. But it doesn't help for STS-2, I set max_steps=5000 (it's 5 epochs) and training and evaluation loss didn't change at all during training. I'm trying to train with learning rate 1e-5 how it was recommended by @alexpython1988 

@thomwolf maybe. Also my sequence length is `384`: the authors did mention they prolly did 512.  Here's my batch size related printout: I think the number of examples seem a lil low. No? I think Squad has about 150K examples (ha and na questions) and with the `doc_stride` I think it should be more than 150k examples (I think).

`07/15/2019 13:23:32 - INFO - __main__ -   ***** Running training *****`
`07/15/2019 13:23:32 - INFO - __main__ -     Num examples = 133947`
`07/15/2019 13:23:32 - INFO - __main__ -     Num Epochs = 3`
`07/15/2019 13:23:32 - INFO - __main__ -     Instantaneous batch size per GPU = 4`
`07/15/2019 13:23:32 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4`
`07/15/2019 13:23:32 - INFO - __main__ -     Gradient Accumulation steps = 1`
`07/15/2019 13:23:32 - INFO - __main__ -     Total optimization steps = 100461`


I saw in the [renatoviolin's repo](https://github.com/renatoviolin/xlnet/blob/master/run_squad_GPU.py) that they have the following which gives them `86F1` on a RTX2080: 
`flags.DEFINE_integer("max_seq_length",
                     default=512, help="Max sequence length")
flags.DEFINE_integer("max_query_length",
                     default=64, help="Max query length")
flags.DEFINE_integer("doc_stride",
                     default=128, help="Doc stride")
flags.DEFINE_integer("max_answer_length",
                     default=64, help="Max answer length")`

Also, lr is different than ours (`5e-5` in this repo):
`flags.DEFINE_float("learning_rate", default=3e-5, help="initial learning rate")`


Learning rate = 1e-5 helps to train STS-2 together with batch size 32 and accumulation steps = 2. I need more experiments but it works. Thanks, @thomwolf, and @alexpython1988!

Great to hear, good job and good luck @avostryakov! Feel free to share good hyper-parameters if you find a nice set and I can add them to the documentation (with credits).

> May also be a problem of batch size, the authors use a batch size between 32 and 128 in the paper.
> 
> What effective batch size do you have (printed during training)?
> 
> While we reproduce the official XLNet number on STS-B, I still have to work a bit on the SQuAD example for XLNet, the XLNet authors used a complex pre- and post-processing of the data (smarter than Bert's) that I haven't fully integrated into our `run_squad` example yet.

I was using per_gpu_train_batch 8 for squad 2.0. Powerful model is hard to tune maybe

> Great to hear, good job and good luck @avostryakov! Feel free to share good hyper-parameters if you find a nice set and I can add them to the documentation (with credits).

@thomwolf My the best result for SST-2 so far is 94.15 of accuracy (in xlnet's article 95.6). It's better than BERT-large. I trained with the following parameters:

```
python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --evaluate_during_training \
    --do_eval   \
    --logging_steps 500 \
    --save_steps 3000 \
    --task_name=sst-2     \
    --data_dir=${GLUE_DIR}/SST-2  \
    --output_dir=./proc_data/sst-2   \
    --max_seq_length=128   \
    --learning_rate 1e-5 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=16000  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120 \
    --fp16
```

@thomwolf Ok, the last result for SST-2 almost matched with XLNet article: Accuracy 95.4:

```
python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --evaluate_during_training \
    --do_eval   \
    --logging_steps 400 \
    --save_steps 3000 \
    --task_name=sst-2     \
    --data_dir=${GLUE_DIR}/SST-2  \
    --output_dir=./proc_data/sst-2   \
    --max_seq_length=128   \
    --learning_rate 1e-5 \
    --per_gpu_eval_batch_size=16   \
    --per_gpu_train_batch_size=16   \
    --gradient_accumulation_steps=1 \
    --max_steps=8000  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120 \
    --fp16
```

Thank you for your work!

This is great @avostryakov! Thanks for sharing the results!
I'm editing the issue title until I've time to add the hyperparameters to the doc.

Hi, how could I finetune the model for text generation? Is it possible just having raw text for the finetuning?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.