Inference time difference between pipeline and with standalone model and tokenizer
Author: Arjunsankarlal
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

# 🐛 Bug

## Information

Model I am using: distilbert-base-cased-distilled-squad

Language I am using the model on : English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD
* [ ] my own task or dataset: (give details below)

## To reproduce

With the two examples provided in the Usage section [here](https://huggingface.co/transformers/usage.html#extractive-question-answering), the first method using the pipeline takes more time for the same context and question.

I modified the contents and question same for both the example and I used the same finetuned SQuAD model for both.

```
from transformers import pipeline
from time import perf_counter

nlp = pipeline("question-answering", model="/Users/arjun/Datasets/QuestionAnswering/distilbert-base-cased-distilled-squad",tokenizer="/Users/arjun/Datasets/QuestionAnswering/distilbert-base-cased-distilled-squad")

context = r"""
🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose
architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural
Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
TensorFlow 2.0 and PyTorch.
"""

st = perf_counter()
nlp(question="How many pretrained models are available in Transformers?", context=context)
nlp(question="What does Transformers provide?", context=context)
nlp(question="Transformers provides interoperability between which frameworks?", context=context)
print(f'Time taken is {perf_counter()-st}')
```

The output was,
> Time taken is 1.5614857940000002

With some added debugged code inside the pipeline code, it seems that the 99% of the time is taken when the input_ids are fed to the model for getting the start and end tensors.

The next example with standalone model and tokenizers is pretty fast, 

```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

tokenizer = AutoTokenizer.from_pretrained("/Users/arjun/Datasets/QuestionAnswering/distilbert-base-cased-distilled-squad")
model = AutoModelForQuestionAnswering.from_pretrained("/Users/arjun/Datasets/QuestionAnswering/distilbert-base-cased-distilled-squad")

text = r"""
🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose
architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural
Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
TensorFlow 2.0 and PyTorch.
"""

questions = [
    "How many pretrained models are available in Transformers?",
    "What does Transformers provide?",
    "Transformers provides interoperability between which frameworks?",
]
st = perf_counter()
for question in questions:
    inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="pt")
    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score

    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

    print(f"Question: {question}")
    print(f"Answer: {answer}\n")
print(f'Time taken is {perf_counter()-st}')
```

The output is,

> Time taken is 0.4920176359999999

I would like to know if my understanding is wrong here. Because inside the pipeline flow, the model and tokenizers are initialized with AutoModelForQuestionAnswering and AutoTokenizer only.  

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I was expecting that both the examples should take relatively equal amount of time.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Darwin-18.0.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 0.4.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Comments:
Anyone looking into this issue ? :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.