Batch inference runtime slows down for inputs with different length sentences
Author: alexdauenhauer
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@LysandreJik @patrickvonplaten (sorry if wrong tags, this is for Luke model, but it is not listed)

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): LukeForEntityPairClassification

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. generate batched inputs for the LukeTokenizer with **identical** sentences in each batch (i.e. no padding required)
2. tokenize each batch by passing the batch to the tokenizer
3. run inference on each batch on GPU and notice that runtime is the same for each batch
4. generate batched inputs for the LukeTokenizer with sentences of **different length** in each batch (i.e. padding is required)
5. tokenize each batch by passing the batch to the tokenizer with `padding=True`
6. run inference on each batch on GPU and notice that runtime increases substantially for subsequent batches after first batch

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
import torch
from transformers import LukeForEntityPairClassification, LukeTokenizer
import time
text1 = "Beyoncé lives in Los Angeles."
entity_spans1 = [(0, 7), (17, 28)]
text2 = "Kevin Love has urged the Cleveland Cavaliers to fight to regain their form following LeBron James' move to the Los Angeles Lakers."
entity_spans2 = [(85, 97), (111, 129)]

# experiment 1 - sentence length is identical across the full batch
text = [[text1] * 10, [text2] * 10]
entity_spans = [[entity_spans1] * 10, [entity_spans2] * 10]
model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
tokenized_inputs = []
for text_batch, span_batch in zip(text, entity_spans):
    inputs = tokenizer(text_batch, entity_spans=span_batch, return_tensors="pt", padding=True, truncation=True)
    tokenized_inputs.append(inputs)
device = torch.device('cuda')
model.to(device)
model.eval()
for i, batch in enumerate(tokenized_inputs):
    with torch.no_grad():
        start = time.time()
        batch.to(device)
        outputs = model(**batch)
        print(f"runtime batch {i}: ", time.time() - start)


# experiment 2 - sentence length alternates in length across the batch
text = [[text1, text2] * 10] * 2
entity_spans = [[entity_spans1, entity_spans2] * 10] * 2
model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
tokenized_inputs = []
for text_batch, span_batch in zip(text, entity_spans):
    inputs = tokenizer(text_batch, entity_spans=span_batch, return_tensors="pt", padding=True, truncation=True)
    tokenized_inputs.append(inputs)
device = torch.device('cuda')
model.to(device)
model.eval()
for i, batch in enumerate(tokenized_inputs):
    with torch.no_grad():
        start = time.time()
        batch.to(device)
        outputs = model(**batch)
        print(f"runtime batch {i}: ", time.time() - start)

# results - Tesla T4
runtime batch 0:  0.028860092163085938
runtime batch 1:  0.03273129463195801
runtime batch 0:  0.028328895568847656
runtime batch 1:  0.09934639930725098
```

## Expected behavior

I expect the runtime to be the same for an identical batch of inputs
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Pinging @NielsRogge as he might have an idea of what's going on with LUKE

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This has not been resolved as far as I know. Please do not close this issue

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

There seems to be something odd happening indeed. Will investigate this.

Also cc'ing @ikuyamada.

This may be related to the asynchronous execution on GPU. When adding `torch.cuda.synchronize(device)` after calling `model(**batch)`, the runtime was approximately consistent across batches on my local machine.

```python
for i, batch in enumerate(tokenized_inputs):
    with torch.no_grad():
        start = time.time()
        batch.to(device)
        outputs = model(**batch)
        torch.cuda.synchronize(device)
        print(f"runtime batch {i}: ", time.time() - start)
```

thanks for the suggestion @ikuyamada ! I'll give it a shot

I tested with `torch.cuda.synchronize(device)` this isn't really the solution I want. I agree that it did seem to resolve the runtime issue, but it increased all runtimes to the longest runtime (now in the second example batch 0 and batch 1 both have execution time of ~0.1 s). This is the opposite of what I would hope to accomplish. The runtime of executing the first batch without `synchronize` is `0.03 s` so I am still not understanding why calling the exact same data a second time would result in a longer runtime. If this is because of async execution, can you please explain to me why?

@alexdauenhauer If I understand torch correctly, torch returns the result without completing the computation. When you use the result (e.g., `print(outputs)`), the *synchronization* happens and the result is computed. Therefore, the following code should give similar results to the code above.

```python
for i, batch in enumerate(tokenized_inputs):
    with torch.no_grad():
        start = time.time()
        batch.to(device)
        outputs = model(**batch)
        print(outputs[0][0])
        print(f"runtime batch {i}: ", time.time() - start)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.