Transformer-XL: hidden states are nan
Author: stefan-it
Pull request: no

Hi,

I followed the code in the Transformer-XL section:

```python
import torch
from pytorch_pretrained_bert import TransfoXLTokenizer, TransfoXLModel, TransfoXLLMHeadModel

# Load pre-trained model tokenizer (vocabulary from wikitext 103)
tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')

# Tokenized input
text_1 = "Who was Jim Henson ?"
text_2 = "Jim Henson was a puppeteer"
tokenized_text_1 = tokenizer.tokenize(text_1)
tokenized_text_2 = tokenizer.tokenize(text_2)

# Convert token to vocabulary indices
indexed_tokens_1 = tokenizer.convert_tokens_to_ids(tokenized_text_1)
indexed_tokens_2 = tokenizer.convert_tokens_to_ids(tokenized_text_2)

# Convert inputs to PyTorch tensors
tokens_tensor_1 = torch.tensor([indexed_tokens_1])
tokens_tensor_2 = torch.tensor([indexed_tokens_2])

model = TransfoXLModel.from_pretrained('transfo-xl-wt103')
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor_1 = tokens_tensor_1
tokens_tensor_2 = tokens_tensor_2

with torch.no_grad():
    # Predict hidden states features for each layer
    hidden_states_1, mems_1 = model(tokens_tensor_1)
    # We can re-use the memory cells in a subsequent call to attend a longer context
    hidden_states_2, mems_2 = model(tokens_tensor_2, mems=mems_1)

    print(hidden_states_1)
    print(hidden_states_2)
```

(One modification: I'm running this on CPU). The hidden states of both sentences are:

```bash
tensor([[[nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan]]])
```

Is this expected? I wanted to get the embeddings of the two sentences 🤔

Tested with PyTorch 1.0.1 and Python 3.7.

Comments:
This is only Transformer-XL related. For GPT the output is:

```python
tensor([[[ 0.1963,  0.0367, -0.2051,  ...,  0.7062, -0.2786,  0.1352],
         [-0.4705,  0.1581,  0.0452,  ...,  0.7809, -0.2519,  0.4257],
         [-0.2602, -0.7126, -0.7966,  ...,  0.6364, -0.1560, -0.6084],
         ...,
         [-0.3665,  1.2743, -2.4027,  ..., -1.7271, -1.7892,  0.7689],
         [-1.3017,  2.7999, -2.8868,  ..., -1.3412,  0.2787, -0.0605],
         [ 0.2648,  0.3508,  0.2894,  ..., -0.7471,  0.1855, -0.0492]]])
```

Same situation, on GPU.



Indeed, there was a bug in the loading of the `TransfoXLModel` from the S3 dump (which is a converted `TransfoXLLMHeadModel`) so the weights were not loaded.

You can see that the weights are not loaded if you activate the logger before loading the model:
```python
import logging
logging.basicConfig(level=logging.INFO)
```

I've fixed it in release 0.5.1.
I've also fixed another issue you (@stefan-it) mentions in https://github.com/zalandoresearch/flair/issues/68 which is the dependency of `OpenAIGPTTokenizer` on SpaCy and ftfy by adding a fallback on BERT's `BasicTokenizer` (should be fine for normal usage, SpaCy+ftfy were included to exactly reproduce the paper's pre-processing steps).

Publishing 0.5.1 as soon as all the tests are checked.

Ok 0.5.1 is published: https://github.com/huggingface/pytorch-pretrained-BERT/releases/tag/v0.5.1