[RuntimeError: sizes must be non-negative] in run_squad.py using xlnet large model
Author: ShuGao0810
Pull request: no

 [RuntimeError: sizes must be non-negative] 
run_squad.py in main
global_step, tr_loss = train(args, train_dataset, model, tokenizer)

run_squad.py in train
outputs = model(**inputs)

modeling_xlnet.py
mems_mask = torch.zeros([data_mask.shape[0], mlen, bsz]).to(data_mask), in which
mlen = 0 resulting from "mems = None". 

Comments:
I have encountered a similar problem:
Just copy the code
`
import torch
#from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
from pytorch_transformers import XLNetLMHeadModel, XLNetTokenizer,XLNetConfig
import numpy as np
import math

config = XLNetConfig.from_pretrained('xlnet-large-cased')
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetLMHeadModel(config)

We show how to setup inputs to predict a next token using a bi-directional context.
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is very ")).unsqueeze(0) # We will predict the masked token
perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token
target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float) # Shape [1, 1, seq_length] => let's predict one token
target_mapping[0, 0, -1] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token)
outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
next_token_logits = outputs[0] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
print(next_token_logits)
`


results the same Runtime Error

@Nealcly the code you posted runs without any errors for me. @ShuGao0810 Can you both post the full stack trace?

> @Nealcly the code you posted runs without any errors for me. @ShuGao0810 Can you both post the full stack trace?

Traceback (most recent call last):
  File "run_squad.py", line 527, in <module>
    main()
  File "run_squad.py", line 473, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 142, in train
    outputs = model(**inputs)
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in__call__
    result = self.forward(*input, **kwargs)
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
    raise output
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 53, in _worker
    output = module(*input, **kwargs)
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in__call__
    result = self.forward(*input, **kwargs)
  File "/data/gaoshu562/SQuAD_v2.0/pytorch_version/xlnet_large/pytorch-transformers-master/pytorch_transformers/modeling_xlnet.py", line 1242, in forward
    head_mask=head_mask)
  File "/data/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in__call__
    result = self.forward(*input, **kwargs)
  File "/data/gaoshu562/SQuAD_v2.0/pytorch_version/xlnet_large/pytorch-transformers-master/pytorch_transformers/modeling_xlnet.py", line 900, in forward
    mems_mask = torch.zeros([data_mask.shape[0], mlen, bsz]).to(data_mask)
RuntimeError: sizes must be non-negative


I also run the glue.sh
To be clear: I use Pytorch 0.4.1 Python 3.6.2 
It yields the same error:

traceback (most recent call last):                       | 0/360 [00:00<?, ?it/s]
  File "./examples/run_glue.py", line 478, in <module>
    main()
  File "./examples/run_glue.py", line 432, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "./examples/run_glue.py", line 129, in train
    outputs = model(**inputs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
    raise output
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 53, in _worker
    output = module(*input, **kwargs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/pytorch_transformers/modeling_xlnet.py", line 1129, in forward
    head_mask=head_mask)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/neal/anaconda3/envs/allennlp/lib/python3.6/site-packages/pytorch_transformers/modeling_xlnet.py", line 891, in forward
    mems_mask = torch.zeros([data_mask.shape[0], mlen, bsz]).to(data_mask)
**RuntimeError: sizes must be non-negative**

If it’s not too much trouble, try cloning your conda env and replacing your torch version with the latest (1.1.0?), and then running again. 

Yes, I gave a deeper look and we are definitely not compatible anymore with PyTorch 0.4.1 at this point.

Maintaining compatibility would be too difficult and not really worth it so I'll update the readme to remove PyTorch 0.4.1 and indicate we start at PyTorch 1.0.0 from now on.