PyTorch-Transformers 1.0 - w. XLNet and XLM model - Standard API - Torchscript compatibility
Author: thomwolf
Pull request: yes

Current status:
- [x] model with commented code and pretrained loading logic
- [x] tokenizer
- [x] tests for model and tokenizer
- [x] checking standard deviation of hidden states with TF model is ok (max dev btw 1e-4 & 1e-5 until last layer, last layer 1e-3, higher than bert but should be ok, investigated this in details, comes from the conjunction of layer_norm and slight differences in internal PT vs. TF ops. Add some graphs to readme)
- [x] converting and uploading model to S3

Model/tokenizer are usable, now just need to
- [x] check the model behave well under various conditions and in a few corner cases
- [ ] add `XLNetForQuestionAnswering` classes variants
- [x] add `XLNetForSequenceClassification` classes variants
- [ ] add a finetuning example with results close to TF
- [ ] add models in README
- [ ] add models on torch.hub

Comments:
@thomwolf I could get the `XLNetLMHeadModel` running, but I have some issues with the "normal" `XLNetModel` implementation:

```python
import torch
from pytorch_pretrained_bert import XLNetTokenizer, XLNetModel

import logging
logging.basicConfig(level=logging.INFO)

tokenizer = XLNetTokenizer.from_pretrained("xlnet-large-cased")

text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.encode(text)

indexed_tokens = torch.tensor([tokenized_text])

model = XLNetModel.from_pretrained("xlnet-large-cased")
model.eval()

with torch.no_grad():
    hidden_states, mems = model(indexed_tokens)
    print(hidden_states)
```

-> is currently not working, `hidden_states = model(indexed_tokens)` always returns `nan`s for the embeddings. 

Thanks :heart: 

Indeed `from_pretrained()` was not loading weights in `XLNetModel` (did you see that in the logs?). Should be fixed now. This is all very WIP so beware @stefan-it!

# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=h1) Report
> Merging [#711](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/c304593d8fa93f25febe1458c63497a846749c89?src=pr&el=desc) will **increase** coverage by `16.62%`.
> The diff coverage is `81.32%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master     #711       +/-   ##
===========================================
+ Coverage   62.27%   78.89%   +16.62%     
===========================================
  Files          18       34       +16     
  Lines        3979     6180     +2201     
===========================================
+ Hits         2478     4876     +2398     
+ Misses       1501     1304      -197
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tests/modeling\_openai\_test.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfb3BlbmFpX3Rlc3QucHk=) | `84.21% <ø> (ø)` | |
| [pytorch\_transformers/tests/conftest.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvY29uZnRlc3QucHk=) | `90% <ø> (ø)` | |
| [pytorch\_transformers/tests/modeling\_xlnet\_test.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfeGxuZXRfdGVzdC5weQ==) | `95.89% <ø> (ø)` | |
| [pytorch\_transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX29wZW5haS5weQ==) | `82.2% <ø> (ø)` | |
| [pytorch\_transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `96.66% <ø> (ø)` | |
| [pytorch\_transformers/tests/modeling\_gpt2\_test.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfZ3B0Ml90ZXN0LnB5) | `84.21% <ø> (ø)` | |
| [pytorch\_transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbS5weQ==) | `82.05% <ø> (ø)` | |
| [...torch\_transformers/tests/tokenization\_gpt2\_test.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2dwdDJfdGVzdC5weQ==) | `96.87% <ø> (ø)` | |
| [...h\_transformers/tests/tokenization\_tests\_commons.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3Rlc3RzX2NvbW1vbnMucHk=) | `100% <ø> (ø)` | |
| [...ytorch\_transformers/tests/tokenization\_xlm\_test.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbV90ZXN0LnB5) | `96.77% <ø> (ø)` | |
| ... and [55 more](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=footer). Last update [c304593...8ad7e5b](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/711?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for the fix - it's working now :)

Sorry, if that's not the right place here:
I've implemented a new embedding class in `flair`. But when the complete model is going to be saved (via `torch.save`), the following error message is shown:

```bash
Traceback (most recent call last):
  File "train_xlnet.py", line 36, in <module>
    max_epochs=500)
  File "/mnt/flair/flair/trainers/trainer.py", line 341, in train
    self.model.save(base_path / "best-model.pt", pickle_module=self.pickle_module)
  File "/mnt/flair/flair/nn.py", line 86, in save
    self.save_torch_model(model_state, str(model_file), pickle_module)
  File "/mnt/flair/flair/nn.py", line 76, in save_torch_model
    torch.save(model_state, str(model_file), pickle_protocol=pickle_protocol)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 149, in _with_file_like
    return body(f)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 297, in _save
    pickler.dump(obj)
TypeError: can't pickle SwigPyObject objects
```

This error message does not appear, when e.g. using the GPT1 embeddings. Do you have any hint, what's going wrong here 🤔



Am I correct that model.eval() does not work with this yet? It seems all predictions generated are the same. 

This is how im doing it:

```
config = XLNetConfig('config.json')
model = XLNetForSequenceClassification(config, num_labels=3)
model.load_state_dict(torch.load("xlnet_pytorch.bin"))
model.to(device)
for param in model.parameters():
    param.requires_grad = False
model.eval()
```

Or am i doing something wrong?

@thomwolf I think this is branch still WIP yes? Because the `run_xlnet_squad.py` still refers to BertForQuestionAnswering. SHould I just change that to XLnetForQuestionAnswering and it should work? or maybe I should wait more?

Yeah, I'm still on it, finishing the tests of XLNetForSequenceClassification so you should wait more (or help me code the XLNetForQuestionAnswering, haha).
I'll make the description of the PR more explicit that only the base model is up now and `XLNetForSequenceClassification` and `XLNetForQuestionAnswering` are not ready yet.

I think I found the root cause of the serialization problem (described in one of the previous comments here):

The sentencepiece processor object cannot be correctly serialized. I found a similar issue in the xnmt library:

https://github.com/neulab/xnmt/pull/351

> Yeah, I'm still on it, finishing the tests of XLNetForSequenceClassification so you should wait more (or help me code the XLNetForQuestionAnswering, haha).


I've started to work on it with whatever time I've :(
So far I'm getting an error in the `modeling_xlnet.py` in line 752: 
`    attention_mask = attention_mask.transpose(0, 1).contiguous() if attention_mask is not None else None`

Whoever finishes first should let here know..But appreciate your work here @thomwolf.

@stefan-it do you know if there is a workaround to the sentencepiece serialization issue?

@thomwolf I think this can be fixed with:

```python
    def __getstate__(self):
        state = self.__dict__.copy()
        state["sp_model"] = None
        return state

    def __setstate__(self, d):
        self.__dict__ = d
        try:
            import sentencepiece as spm
        except ImportError:
            logger.warning("You need to install SentencePiece to use XLNetTokenizer: https://github.com/google/sentencepiece"
                           "pip install sentencepiece")
        self.sp_model = spm.SentencePieceProcessor()
        self.sp_model.Load(self.vocab_file)
```

In the `XLNetTokenizer` class. A nice test case would be:

```python
import pickle
from pytorch_pretrained_bert import XLNetTokenizer

tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')

text = "Munich and Berlin are nice cities"
filename = "tokenizer.bin"

subwords = tokenizer.tokenize(text)

pickle.dump(tokenizer, open(filename, "wb"))

tokenizer_new = pickle.load(open(filename, "rb"))
subwords_loaded = tokenizer_new.tokenize(text)

assert subwords == subwords_loaded
```

I'm hitting a bug where the head mask created here:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/xlnet/pytorch_pretrained_bert/modeling_xlnet.py#L854

is a list of None values instead of just None, which eventually results in an error: 

```
  File "../pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling_xlnet.py", line 397, in rel_attn_core
    attn_prob = attn_prob * head_mask
TypeError: mul(): argument 'other' (position 1) must be Tensor, not list
```

I think the fix is to either make the mask a single None value, or to index it with the layer number when it's used.

----

I also ran into one more error:

```
File "../pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling_xlnet.py", line 555, in forward
  outputs = [output_h, output_g] + outputs[2:]  # Add again attentions if there are there
TypeError: can only concatenate list (not "tuple") to list
```
Not sure what's causing this, but casting `outputs[2:]` to a list seems to fix things.

@stefan-it Great, thanks!

@nikitakit Yeah I'm refactoring the API among models to make it more consistent/simpler to switch among models. Will be finished soon.

While modifying `run_xlnet_squad.py` I looked through the `run_squad.py` of the xlnet repo. It seems he made a bunch of changes to `convert_examples_to_features` (adding lcs etc.) Also, I didn't see that he put the cls at the end; rather p before than q (I think; hope I'm not wrong). Have you started to work on it @thomwolf ? It's interesting :) 

One quick update in case someone else is also working on the Squad XLnet fine tuning. I got this error and working on it seems that the end logits in the QA layer is buggy.

`Traceback (most recent call last):
  File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 478, in <module>
    main()
  File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 470, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/run_squad.py", line 219, in evaluate
    args.null_score_diff_threshold)
  File "/dccstor/avistor4/squad/expts/xlnet_branch_hf/pytorch-pretrained-BERT/examples/utils_squad.py", line 463, in write_predictions
    feature_null_score = result.start_logits[0] + result.end_logits[0]
TypeError: unsupported operand type(s) for +: 'float' and `'list'`


Is there any rough estimation as to when "a finetuning example with results close to TF" will be available?

> 
> 
> Thanks for the fix - it's working now :)
> 
> Sorry, if that's not the right place here:
> I've implemented a new embedding class in `flair`. But when the complete model is going to be saved (via `torch.save`), the following error message is shown:
> 
> ```shell
> Traceback (most recent call last):
>   File "train_xlnet.py", line 36, in <module>
>     max_epochs=500)
>   File "/mnt/flair/flair/trainers/trainer.py", line 341, in train
>     self.model.save(base_path / "best-model.pt", pickle_module=self.pickle_module)
>   File "/mnt/flair/flair/nn.py", line 86, in save
>     self.save_torch_model(model_state, str(model_file), pickle_module)
>   File "/mnt/flair/flair/nn.py", line 76, in save_torch_model
>     torch.save(model_state, str(model_file), pickle_protocol=pickle_protocol)
>   File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in save
>     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
>   File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 149, in _with_file_like
>     return body(f)
>   File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in <lambda>
>     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
>   File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 297, in _save
>     pickler.dump(obj)
> TypeError: can't pickle SwigPyObject objects
> ```
> 
> This error message does not appear, when e.g. using the GPT1 embeddings. Do you have any hint, what's going wrong here 🤔

Hi, have you solved this issue? I'm trying to implement XLMRoberta in my model based on flair and meet the same issue when saving the model.