[Benchmark] TFGPT2LMHeadModel is five times slower than GPT2LMHeadModel
Author: bjourne
Pull request: no
Labels: name: wontfix

Here are two scripts I ran. 
```python
from time import time
from transformers import TFGPT2LMHeadModel, GPT2Tokenizer
import tensorflow as tf

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = tf.constant([generated])
past = None
start = time()
for i in range(100):
    output, past = model(context, past = past)
    logits = output[0, -1, :]
    tok = tf.argmax(logits)
    generated.append(tok.numpy())
    context = tf.expand_dims(tf.expand_dims(tok, 0), 0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)
```
and
```python
from time import time
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')

text = "What are you doing after you have finished working?"
generated = tokenizer.encode(text)
context = torch.tensor([generated])
past = None
start = time()
for i in range(100):
    output, past = model(context, past=past)
    token = torch.argmax(output[..., -1, :])
    generated += [token.tolist()]
    context = token.unsqueeze(0)
sequence = tokenizer.decode(generated)
print(time() - start, sequence)
```
On my computer with the models running on the cpu, the PyTorch version finishes in about six seconds while the TensorFlow version takes 30 seconds. So something must be wrong with the TF implementation because it shouldn't be that much slower.

Comments:
That's probably because you're in eager mode in your TensorFlow script. You can read about eager mode [here](https://www.tensorflow.org/guide/eager).

[Here's](https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit#gid=0) a spreadsheet showcasing several model performances, you can check it out for GPT-2.

The script runs even slower in graph execution mode.

Same thing here:
```python
from time import time
from transformers import (TFGPT2LMHeadModel, GPT2Tokenizer,
                          GPT2LMHeadModel,
                          pipeline)

seed = "What are you doing after you have finished working?"
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
gen = pipeline('text-generation',
               model = model,
               tokenizer = tokenizer)
start = time()
out = gen(seed, max_length = 100, num_return_sequences = 1)
print(time() - start, out)
```
Just changing `GPT2LMHeadModel` to `TFGPT2LMHeadModel` makes the program run 5 times slower.

Oh, I see. Thanks for opening an issue, we're looking into it now.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


same question

with @gante's new TF generate method this should be much faster now no? :-)

Hi @shiyongde 👋 Yeah, we have just released a much faster TF generation. Check our blog post [here](https://huggingface.co/blog/tf-xla-generate).

Note that it is not yet compatible with `pipeline` (it is in our TODO list)