Problem loading finetuned XLNet model
Author: igormis
Pull request: no

After fine-tuning an XLNet classification model and obtaining TF checkpoints I converted the checkpoint to pytorch_model.bin and config.json. I need to make prediction on input text, but I have problems loading the models correctly. Any help?

Comments:
What task did you fine-tuned it on?
You can convert it by running the `convert_xlnet_checkpoint_to_pytorch.py` script with a `--finetuning_task` argument (see [here](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/convert_xlnet_checkpoint_to_pytorch.py#L94-L97))


binary classification (sentiment) on my dataset
I tried the following
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetModel
config = XLNetConfig.from_pretrained('./')
tokenizer = XLNetTokenizer.from_pretrained('./')
model = XLNetModel(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]`
however, the output is not binary, the output tensor is :
tensor([[[-1.0645, -1.1170, -0.5549,  ...,  0.7089,  0.2862, -0.2995],
         [ 0.0198, -1.6598, -0.1760,  ...,  0.1167, -0.1545,  0.1777],
         [ 0.6514, -0.5614, -1.2180,  ...,  1.0796,  1.1217,  1.0362]]],
       grad_fn=<PermuteBackward>)



the converting was done using:
`pytorch_transformers xlnet $TRANSFO_XL_CHECKPOINT_PATH $TRANSFO_XL_CONFIG_PAH $PYTORCH_DUMP_OUTPUT . `

I did not specified the task, because I though that it is not important.

It is important to match the last (classification) layer otherwise the conversion will fail.
If you give me more information on your TF training (like which script you usde for instance) I may be able to help.

I was using the:
`train_command = "python xlnet/run_classifier.py \
  --do_train=True \
  --do_eval=True \
  --eval_all_ckpt=True \
  --task_name=spam \
  --data_dir="+DATA_DIR+" \
  --output_dir="+OUTPUT_DIR+" \
  --model_dir="+CHECKPOINT_DIR+" \
  --uncased=False \
  --spiece_model_file="+PRETRAINED_MODEL_DIR+"/spiece.model \
  --model_config_path="+PRETRAINED_MODEL_DIR+"/xlnet_config.json \
  --init_checkpoint="+PRETRAINED_MODEL_DIR+"/xlnet_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=8 \
  --eval_batch_size=8 \
  --num_hosts=1 \
  --num_core_per_host=1 \
  --learning_rate=2e-5 \
  --train_steps=4000 \
  --warmup_steps=500 \
  --save_steps=500 \
  --iterations=500"

! {train_command}`
It is similar to the imdb sentiment task
The colab is here
https://colab.research.google.com/drive/1nfWEEDxPOE8myb-hwGdoXs9nVXM3AVcz

I have modified the:
class ImdbProcessor(DataProcessor):

Here is the model after model.eval():
`XLNetModel(
  (word_embedding): Embedding(32000, 1024)
  (layer): ModuleList(
    (0): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (1): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (2): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (3): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (4): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (5): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (6): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (7): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (8): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (9): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (10): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (11): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (12): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (13): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (14): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (15): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (16): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (17): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (18): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (19): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (20): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (21): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (22): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (23): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
  )
  (dropout): Dropout(p=0.1)
)
`

You'll have to modify the conversion script because this is not a standard task and the number of labels won't be found in the list (see top of the conversion script) you should add `'spam': 2` in the list (if you have two labels indeed).

The other option is to directly load the TF model in PyTorch and save the pytorch model afterwards with something like this:
```
config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='spam')
model = XLNetForSequenceClassification.from_pretrained('path/to/your/tf/model.ckpt.index', config=config, from_tf=True)
model.save_pretrained('pytorch_model_saving_directory')
```

Perfect, tnx.
Just a minor question if I have classification model with 4 labels, I do the same changes spam:4?

Yes, change my `2` to `4`

Hi Thom,
I did:
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetForSequenceClassification
config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='sentiment')
model = XLNetForSequenceClassification.from_pretrained('../model.ckpt-2500', config=config, from_tf=True)
model.save_pretrained('./test')`

Afterwards I get in the test folder the config.json and pytorch_model.bin.

I tried to run:
'import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetModel
config = XLNetConfig.from_pretrained('./')
tokenizer = XLNetTokenizer.from_pretrained('./')
model = XLNetModel(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]
print(last_hidden_states)


There was an error:
Model name './' was not found in model name list (xlnet-base-cased, xlnet-large-cased). We assumed './' was a path or url but couldn't find tokenizer filesat this path or url.

So I included the spiece.model into the same test folder, but still the results is:
tensor([[[ 0.0156, -0.9046,  0.9789,  ..., -0.7764,  1.1309,  0.2862],
         [ 0.4416,  0.0665,  2.0020,  ...,  0.4117,  0.9779, -1.0588],
         [ 0.6908, -0.0000,  0.1479,  ...,  0.0032,  0.9871, -0.9482]]],
       grad_fn=<PermuteBackward>)

Do u know what I am doing wrong? 

While converting this is some of the warnings:
Weights not copied to PyTorch model: beta1_power, beta2_power, global_step, model/classification_finsent/logit/bias, model/classification_finsent/logit/bias/Adam, model/classification_finsent/logit/bias/Adam_1, model/classification_finsent/logit/kernel, model/classification_finsent/logit/kernel/Adam, model/classification_finsent/logit/kernel/Adam_1

In addition, I have tried the other approach, to modify the script, but the results are the same. I think that the problem is that the weights from the classification_finsent are not copied to pytorch model. Any suggestions on this

I have tried also in this way:
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification

config = XLNetConfig.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/PyTorch/pytorch_transformer_script/test')
tokenizer = XLNetTokenizer.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/PyTorch/pytorch_transformer_script/test')
config.output_hidden_states=True
model = XLNetForSequenceClassification(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase rapidly")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
print(outputs[0])`

and the output now is:
tensor(0.3944, grad_fn=<NllLossBackward>)


In the last attempt I did:
```python
import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification

config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='finsent')
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') #I am using the default tokenizer

model = XLNetForSequenceClassification.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/model.ckpt-2500',config=config, from_tf=True) #this is my finetuned model
model.eval() #in evaluation mode
input_ids = torch.tensor(tokenizer.encode("Apple stock increase and they are overwhelmed with the success")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, logits = outputs[:2]
print(torch.nn.functional.softmax(logits.data))
```

However I am getting different prediction for the same input data, I did model.eval() to stop the dropout, but still the inference looks random.

In addition here is my config file printed:
```json
{
  "attn_type": "bi",
  "bi_data": false,
  "clamp_len": -1,
  "d_head": 64,
  "d_inner": 4096,
  "d_model": 1024,
  "dropatt": 0.1,
  "dropout": 0.1,
  "end_n_top": 5,
  "ff_activation": "gelu",
  "finetuning_task": "finsent",
  "init": "normal",
  "init_range": 0.1,
  "init_std": 0.02,
  "initializer_range": 0.02,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "mem_len": null,
  "n_head": 16,
  "n_layer": 24,
  "n_token": 32000,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "reuse_len": null,
  "same_length": false,
  "start_n_top": 5,
  "summary_activation": "tanh",
  "summary_last_dropout": 0.1,
  "summary_type": "last",
  "summary_use_proj": true,
  "torchscript": false,
  "untie_r": true
}
```
It looks that I getting closer, but still the inference is strange

@thomwolf I finally succeed to import the checkpoint model and infer. Still I am not sure if this a valid approach:
```python
import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification


seed = 0
torch.manual_seed(seed)
if torch.cuda.is_available():
	torch.cuda.manual_seed_all(seed)
#load the initial config file from the XLNet model
config = XLNetConfig.from_pretrained('xlnet_config.json', num_labels=2, finetuning_task='finsent')
#the tokenizer I am using is the initial one (spiece.model)
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')

model = XLNetForSequenceClassification.from_pretrained('/sentiment_over_imdb/model.ckpt-4000',config=config, from_tf=True)
model.eval()

def sentiment(data):
	input_ids = torch.tensor(tokenizer.encode(data)).unsqueeze(0)  # Batch size 1
	labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1

	outputs = model(input_ids, labels=labels)
	loss, logits = outputs[:2]
	output = torch.nn.functional.softmax(logits.data, dim=1)
	sent = output.tolist()[0][1]
        return sent
```