Fix token order in xlnet preprocessing for SQuAD
Author: hlums
Pull request: yes

#947 
My current result on SQuAD 1.1
{
  "exact": 85.45884578997162,
  "f1": 92.5974600601065,
  "total": 10570,
  "HasAns_exact": 85.45884578997162,
  "HasAns_f1": 92.59746006010651,
  "HasAns_total": 10570
}

My code validation command
```
python /data/home/hlu/transformers/examples/run_squad.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file /data/home/hlu/notebooks/NLP/examples/question_answering/train-v1.1.json \
    --predict_file /data/home/hlu/notebooks/NLP/examples/question_answering/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ./wwm_cased_finetuned_squad/ \
    --per_gpu_eval_batch_size=4  \
    --per_gpu_train_batch_size=4   \
    --save_steps 5000
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=h1) Report
> Merging [#1549](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8a62835577a2a93642546858b21372e43c1a1ff8?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1549/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1549   +/-   ##
=======================================
  Coverage   85.14%   85.14%           
=======================================
  Files          94       94           
  Lines       13920    13920           
=======================================
  Hits        11852    11852           
  Misses       2068     2068
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=footer). Last update [8a62835...9a3b173](https://codecov.io/gh/huggingface/transformers/pull/1549?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is great, thanks @hlums.
I'm adding information on the runs/results in the example's readme and merging.

@ShengeBelmendo I think we set them to the right values here as well (cf lines 309-310 in `run_squad`).

Ok LGTM, merging this PR, thanks @hlums 

@thomwolf oh, I'm sorry for missing that.

And another small question. @hlums Have you tested whether set the `do_lower_case` option or not ? It seems that we shouldn't set this option cuz the model is the cased version and it also didn't be set in official code.

I'm trying to reproduce the results of xlnet but obviously there are some problems especially for squad2.0, it's still a long way to go.

> @thomwolf oh, I'm sorry for missing that.
> 
> And another small question. @hlums Have you tested whether set the `do_lower_case` option or not ? It seems that we shouldn't set this option cuz the model is the cased version and it also didn't be set in official code.
> 
> I'm trying to reproduce the results of xlnet but obviously there are some problems especially for squad2.0, it's still a long way to go.

You are right. I shouldn't have set the do_lower_case flag. I can give it a try some time this week. 
Have you tried the latest code on squad 2.0? I thought put the cls token at the right place would help a lot because it's used for unanswerable question classification. 

> > @thomwolf oh, I'm sorry for missing that.
> > And another small question. @hlums Have you tested whether set the `do_lower_case` option or not ? It seems that we shouldn't set this option cuz the model is the cased version and it also didn't be set in official code.
> > I'm trying to reproduce the results of xlnet but obviously there are some problems especially for squad2.0, it's still a long way to go.
> 
> You are right. I shouldn't have set the do_lower_case flag. I can give it a try some time this week.
> Have you tried the latest code on squad 2.0? I thought put the cls token at the right place would help a lot because it's used for unanswerable question classification.

@hlums Sorry for late reply. The latest code doesn't work, you will get a f1 score closed to 0 on unanswerable questions. But I have found the reason. 

The following is a piece of code in forward function of xlnet model, which obviously is the key point of training the model on unanswerable questions using cls token representations.  But the default value of tensor `is_impossible`(using to indicate whether this example is answerable) is none, and we also hadn't passed this tensor into forward function. That's the problem. 

```        
 if cls_index is not None and is_impossible is not None:
                # Predict answerability from the representation of CLS and START
                cls_logits = self.answer_class(hidden_states, start_positions=start_positions, cls_index=cls_index)
                loss_fct_cls = nn.BCEWithLogitsLoss()
                cls_loss = loss_fct_cls(cls_logits, is_impossible)
                total_loss += cls_loss * 0.5
```

I added the `is_impossible` tensor to TensorDataset and model inputs, and got a reasonable result, f1: 84, EM: 80. Maybe I can creat a PR for this, maybe after I find more discrepancies and get better results. I'm working hard to reproduce the results of xlnet on squad2.0, so I hope you can tell me if you have some new ideas or finds.Thanks! 

@hlums before your fix:
`xlnet-large-cased`, SQuAD 1.1, 2 epochs, MSL: 512, BS: 48
{
  "exact": 75.01419110690634,
  "f1": 82.13017516396678,
  "total": 10570,
  "HasAns_exact": 75.01419110690634,
  "HasAns_f1": 82.13017516396678,
  "HasAns_total": 10570
}
Post fix, awesome:
{
  "exact": 85.1371807000946,
  "f1": 92.24219729313499,
  "total": 10570,
  "HasAns_exact": 85.1371807000946,
  "HasAns_f1": 92.24219729313499,
  "HasAns_total": 10570
}
Thanks again for your fix!

`xlnet-large-cased`, SQuAD 2.0, max_steps: 8000, MSL: 512, BS: 48
{
  "exact": 40.95005474606249,
  "f1": 45.305949189220875,
  "total": 11873,
  "HasAns_exact": 81.96693657219973,
  "HasAns_f1": 90.69121705864026,
  "HasAns_total": 5928,
  "NoAns_exact": 0.050462573591253154,
  "NoAns_f1": 0.050462573591253154,
  "NoAns_total": 5945
}

Hopefully `xlnet-large-cased` on _SQuAD 2.0_ for the holidays: @ShengeBelmendo  https://github.com/huggingface/transformers/pull/1803
Limited what I can contribute to the code/logic, but I can run tests 24 x 7.

@ShengeBelmendo I tried turning do_lower_case off, but the model performance didn't change much.
![image](https://user-images.githubusercontent.com/16907204/68711380-82a76400-0567-11ea-85be-d582bac6212e.png)

Sorry I'm not actively working QA anymore so probably won't be able to contribute to the improvement on squad 2.0

Another thing mentioned in the XLNet paper is layer-wise learning rate decay. I actually tried implementing it, but it didn't help with the performance on 1.1 for me. See #1198 

The pre-processing code in the XLNet repo also looks much more complicated than here. I'm not sure if it has anything to do with the performance discrepancy though. 


@hlums ok, tks again. I will check more carefully about the pre-processing part and maybe read all the official code for comparison.