The accuracy of XLNet
Author: chlorane
Pull request: no
Labels: name: wontfix

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....):
XLNet
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
 [ ] the official example scripts: (give details)
 [ * ] my own modified scripts: (give details)
I use my own scripts under this library 

The tasks I am working on is:
[ ] an official GLUE/SQUaD task: (give the name)
[ * ] my own task or dataset: (give details)
TVQA dataset, for question answering task

Details of the issue:

I was using pytorch-pretrained-BERT library on my scripts. Last week, I switched to the "transformer" library. After that, I tried to use BERT with "bert-base-cased" (do_lower_case=true) model, the valid acc is around 64% after 800 it (batch size=8). However, when I modify the model type to "xlnet" and the model with "xlnet-base-cased" (I have set left padding as true and pad_token_segment_id=4. The other parts are kept the same as using BERT), the valid acc is only about 28% after 800 it (batch size=8). When more iterations are made, the valid acc will drop to about 22%. I think this is quite strange.

## Environment

* OS: Windows 10
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): The newest? (Downloaded last Monday)
* Using GPU ? Yes
* Distributed or parallel setup ? No
* Any other relevant information:

## Checklist

- [ * ] I have read the migration guide in the readme.
- [ * ] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Are you sure that you are using the right models and not just `BertModel`? You also have to change the tokenizer completely.

Instead of something like

```python
model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```

You should use something like this:

```python
model = XLNetModel.from_pretrained('xlnet-base-cased')
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
```

For tokenisation, I recommend that you use the `encode` method so that special tokens are added correctly and automatically.

```python
input_ids = tokenizer.encode(text, return_tensors='pt')
```

It is normal that not all models behave the same way. You can't just use the same hyperparameters and expect the same or bettter results. Try longer finetuning, other learning rate, stuff like that.

Finally, you may wish to update your PyTorch version if possible. We're already at 1.4.



> 
> 
> Are you sure that you are using the right models and not just `BertModel`? You also have to change the tokenizer completely.
> 
> Instead of something like
> 
> ```python
> model = BertModel.from_pretrained('bert-base-uncased')
> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
> ```
> 
> You should use something like this:
> 
> ```python
> model = XLNetModel.from_pretrained('xlnet-base-cased')
> tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
> ```
> 
> For tokenisation, I recommend that you use the `encode` method so that special tokens are added correctly and automatically.
> 
> ```python
> input_ids = tokenizer.encode(text, return_tensors='pt')
> ```
> 
> It is normal that not all models behave the same way. You can't just use the same hyperparameters and expect the same or bettter results. Try longer finetuning, other learning rate, stuff like that.
> 
> Finally, you may wish to update your PyTorch version if possible. We're already at 1.4.

Yes, I have made these modifications, but the acc is still low. After training for an epoch, it is 28% still.
I'm using:
```
    MODEL_CLASSES = {"bert": (BertConfig, BertForMultipleChoice, BertTokenizer),"xlnet": (XLNetConfig, XLNetForMultipleChoice, XLNetTokenizer),"roberta": (RobertaConfig, RobertaForMultipleChoice, RobertaTokenizer)}
    ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, XLNetConfig, RobertaConfig)), ())
    config_class, model_class, tokenizer_class = MODEL_CLASSES["xlnet"]
    transformer_tokenizer = tokenizer_class.from_pretrained("xlnet-base-cased", do_lower_case=True, cache_dir=None)
```

and

```
            text_a=context_tokens_choice
            text_b=ending_tokens
            inputs=transformer_tokenizer.encode_plus(text_a, text_b, add_special_tokens=True, max_length=max_seq_length)
            input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
            attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
            padding_length = max_seq_length - len(input_ids)
            if pad_on_left:
                input_ids = ([pad_token] * padding_length) + input_ids
                segment_ids = ([0 if mask_padding_with_zero else 1] * padding_length) + attention_mask
                input_mask = ([pad_token_segment_id] * padding_length) + token_type_ids
            else:
                input_ids = input_ids + ([pad_token] * padding_length)
                segment_ids = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)
                input_mask = token_type_ids + ([pad_token_segment_id] * padding_length)        
```    

and

` self.sub_model=model_class.from_pretrained("xlnet_base-cased",from_tf=bool(".ckpt" in "xlnet-base-cased"),config=config,cache_dir=None)`

`out=self.sub_model(all_input_ids,all_segment_ids,all_input_mask)`

I thought it is because segment_ids and input_mask are in reversed order, but after switching them, the acc is still low.

I tried RoBERTa, the acc tends to be normal.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, @chlorane ,I got the same problem,and did you solve it???