XLnet loss and accuracy not decreasing
Author: jetjodh
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLnet base cased

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
Can check the notebook below:
https://colab.research.google.com/drive/132r5kb1G5oG0yi-qnymBsMBPCGP5Gu85
Can only give access to a few people.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Binary classification.
## To reproduce

Steps to reproduce the behavior:
Preprocessing:
```
from transformers import XLNetTokenizer 
from keras.preprocessing.sequence import pad_sequences
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

 encoded_sent = tokenizer.encode(
             text, # Sentence to encode.
             add_special_tokens = True, 
             )
MAX_LEN = 128
encoded_sent = pad_sequences([encoded_sent], maxlen=MAX_LEN, dtype="long", 
         value=0, truncating="post", padding="post")
attention_masks=[]
att_mask = [int(token_id > 0) for token_id in encoded_sent[0]]
attention_masks.append(att_mask)
```

Model definition:
```
from transformers import BertForSequenceClassification, XLNetForSequenceClassification, AdamW, BertConfig

# bert = BertForSequenceClassification.from_pretrained(
#  "bert-base-uncased", 
#  num_labels = 2, .
#  output_attentions = False,
#  output_hidden_states = False,

xlnet = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased',
num_labels = 2, 
output_attentions = False, 
output_hidden_states = False, 
)
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.features2 = xlnet
        self.softmax = nn.LogSoftmax(dim=1)
    def forward(self, x2, x3):
        x2 = x2.to(torch.int64)
        x2 = self.features2(x2,x3)[0]
        x = self.softmax(x2)
        return x


model = MyModel()
torch.cuda.empty_cache()
model.to('cuda')
criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer = optim.AdamW(model.parameters(), lr=0.0005)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Loss and accuracy should decrease but are not changing at all( both training and valid).
This script worked while training BERT model.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version: Latest
- Platform: Colab
- Python version: 3.6
- PyTorch version (GPU?): Latest
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
I recommend to change your preproccessing to:

```python
from transformers import XLNetTokenizer 
from keras.preprocessing.sequence import pad_sequences
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

encoded_sent = tokenizer.encode_plus(
             text, # Sentence to encode.
             return_tensors='pt',
             max_length=128)
# ...
output = model(**encoded_sent )
```

However, the real problem is probably due to hyperparameters. You cannot simply use different models with the same hyperparameters and immediately expect results. You'll have to fiddle with the hyperparameters and find something that works for your case. 


I am converting the text to tensor in a later step.
I also tried changing the lr ranging from 0.05 to 5e-8 but still loss did not changing and also applied a lr sheduler. Maybe I should try other optimizers other than AdamW?

Since this is not a bug I am closing this. It is impossible for us to help with this any further since hyperparamter optimization is different for each task. This is something that you have to test yourself.

For a starting point, you can have a look at Table 8 in [the original paper](https://arxiv.org/pdf/1906.08237.pdf) where they suggest some good hyperparameter settings. But again, even then it depends on your specific case what would help and what wouldn't. Try and test!