Fine tune Xlnet
Author: AhmedBahaaElDinMohammed
Pull request: no
Labels: name: wontfix

Can anybody guide me on how to fine tune xlnet for simple text classification task or any reference code because i am lost.

Comments:
As far as I know, the pytorch code of XLNet is not completely ready now. But you could find it in the branch `xlnet` and the classifier code is nearly ready in the file `example/run_xlnet_classifier.py`. I have successfully fine-tuned it on the SST-2 task (which belongs to GLUE) with following args:

```shell
python run_xlnet_classifier.py \
--data_dir ..\glue_data\SST-2 \
--task_name sst-2 \
--output_dir sst_model \
--do_train \
--do_eval \
--max_seq_length 128 \
--train_batch_size 64 \
--learning_rate 5e-6
```

@SivilTaram Cant be fine tuned on external data ? is the tensorflow version ready ?

@AhmedBahaaElDinMohammed Sure you could fine-tune it on external data, which means you should process your data and construct train/validate `examples` as SST-2 does. You could see `example/utlis_glue.py` for more details to handle your external data :)

The tensorflow is ready, you could refer to the original repo for help. This repo is only for pytorch version, thanks.

> I have successfully fine-tuned it on the SST-2 task (which belongs to GLUE) 

@SivilTaram Would it be possible to fine-tune it on SQuAD 2.0? or alternatively, convert a fine-tuned model from the original repo/tensorflow?

@edanweis Not ready now. Please wait the author to complete the awesome work :) Or you could watch the updates of PR [here](https://github.com/huggingface/pytorch-pretrained-BERT/pull/711).

Has anyone tried fp16 for xlnet?
I tried it and found that the memory was half, but it was slower than fp32(even when I used the same GPU memory). 
Environment: v100, cuda 10.0, torch 1.1
The environment is ok, because I tried bert + fp16 and it was much faster than fp32.
I thought it is the problem of torch.einsum, but I am not that sure. 
Guys, do you have the same problem ?

@SivilTaram Following the latest release 0.6.2, I am trying to convert my tf checkpoints:

```
export TRANSFO_XL_CHECKPOINT_PATH=home/edanweis/xlnet/model/squad
export TRANSFO_XL_CONFIG_PATH=home/edanweis/xlnet/model/squad
export FINETUNING_TASK=squad


pytorch_transformers xlnet \
  $TRANSFO_XL_CHECKPOINT_PATH \
  $TRANSFO_XL_CONFIG_PATH \
  $PYTORCH_DUMP_OUTPUT \
  $FINETUNING_TASK \
```
But getting `pytorch_transformers/__main__.py", line 111, in main FINETUNING_TASK) UnboundLocalError: local variable 'FINETUNING_TASK' referenced before assignment`

@SivilTaram Did you try to finetune XLNet with the last code (release 1.0) using examples/run_glue.py? Everything works but accuracy didn't change and every time is around 0.50? It looks like it didn't train at all. I used the following script:

```
export GLUE_DIR=/path/to/glue

python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --do_eval   \
    --evaluate_during_training \
    --logging_steps 500 \
    --save_steps 1000 \
    --task_name=sst-2     \
    --data_dir=${GLUE_DIR}/SST-2  \
    --output_dir=./proc_data/sst-2   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=8000  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120
```

@avostryakov I do not yet. I guess you could explore if the loss decrease as expected? There should be loss logs, along with tensorboard logs.

@SivilTaram Evaluation loss isn't changed, training loss is increased. It looks like something wrong with the optimization process during training.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.