BUG for XLNet: Low GPU usage and High CPU usage, very low running speed!
Author: Weili-NLP
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: XLNet used as encoder for seq2seq generation 

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Centos 7.6
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.2.0
* Using GPU ? V100, 16G
* Distributed of parallel setup ? Use multiprocessing for multi GPU training
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi, could you provide a minimal code that reproduces the issue so that we may help debug/check on our side?

hey @LysandreJik,

I think I am experiencing the same/similar issue. Below are the sample code, a screenshot showing the high CPU and low GPU usage, and the `requirements.txt` file.

In terms of technical setup, I used a docker container deployed with [vast.ai](https://vast.ai/) with RTX 2070S, CUDA 10.1, AMD Ryzen 5 2600X (6c/12t), 16GB. It used the [latest PyTorch image](https://hub.docker.com/r/pytorch/pytorch) as of the time of writing. From my observations the fluctuating CPU usage seems to max out at 600%, hinting at the 6 physical cores. However. I think I have experienced the issue with 2080 Ti and 1080 Ti, as well.

Thank you!

```python
# -*- coding: utf-8 -*-

import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel
import requests

convai1_data = requests.get('http://convai.io/2017/data/train_full.json').json()

for dial in convai1_data:
    utterances = [thread_line['text'] for thread_line in dial['thread']]
    dial['utterances'] = utterances
    dial['predictions'] = dict()

convai1_data = [dial for dial in convai1_data if len(dial['utterances']) > 2]

# https://github.com/huggingface/transformers/issues/917#issuecomment-525297746
def xlnet_sent_probability(PADDING_TEXT, text):
    tokenize_text = model_tokenizer.tokenize(text)[:512]
    tokenize_input = model_tokenizer.tokenize(PADDING_TEXT)[:511] + ['<eod>'] + tokenize_text

    sentence_word_probs = list()
    sentence_best_word_probs = list()
    best_words = list()

    for max_word_id in range((len(tokenize_input)-len(tokenize_text)), (len(tokenize_input))):

        sent = tokenize_input[:]

        input_ids = torch.tensor([model_tokenizer.convert_tokens_to_ids(sent)])

        perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
        perm_mask[:, :, max_word_id:] = 1.0 

        target_mapping = torch.zeros((1, 1, input_ids.shape[1]), dtype=torch.float)
        target_mapping[0, 0, max_word_id] = 1.0

        if torch.cuda.is_available():
            input_ids = input_ids.cuda()
            perm_mask = perm_mask.cuda()
            target_mapping = target_mapping.cuda()

        with torch.no_grad():
            outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
            next_token_logits = outputs[0]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
            predicted_prob = torch.softmax(next_token_logits[0][-1], dim=-1)

        predicted_prob = predicted_prob.detach().cpu().numpy()

        word_id = model_tokenizer.convert_tokens_to_ids([tokenize_input[max_word_id]])[0]
        word_prob = predicted_prob[word_id].item()

        sentence_word_probs.append(word_prob)

    return sentence_word_probs

for XLNET_MODEL in ['xlnet-base-cased', 'xlnet-large-cased']:

    model_tokenizer = XLNetTokenizer.from_pretrained(XLNET_MODEL)
    model = XLNetLMHeadModel.from_pretrained(XLNET_MODEL)

    if torch.cuda.is_available():
        model = model.cuda()

    model = model.eval()

    for dial in convai1_data:
        utterances = dial['utterances']

        sentences_word_probs = list()

        for u1, u2 in zip(utterances[:-1], utterances[1:]):
            sentence_word_probs = xlnet_sent_probability(u1, u2)

            sentences_word_probs.append(sentence_word_probs)

        dial['predictions'][XLNET_MODEL+'_sentences_word_probs'] = sentences_word_probs

```

[requirements.txt](https://github.com/huggingface/transformers/files/3883834/requirements.txt)
[cpu_and_gpu_usage](https://user-images.githubusercontent.com/8143425/69497875-e9226f80-0ee1-11ea-8098-f81df3972f3e.png)





This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


The same problem. How to solve it?