Transformer pipeline loading model and tokenizer on every prediction request
Author: Sharathmk99
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
I'm trying to build API which accept question and context, response should be the answer. I'm trying to use Transformer Question Answering pipeline task with BERT model. 
I'm initializing the model and pipeline object on __init__ method and trying to use pipeline object in another method, so that on server startup the model is loaded to memory and prediction will be fast. 
But on every prediction request, __init__method is called, when i debugged the flow, i think its because of feature calculation is happening using Pool threads.
https://github.com/huggingface/transformers/blob/68e19f1c228c92d5d800533f558faff24b57127a/src/transformers/data/processors/squad.py#L318

Can you please to resolve the issue?

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] my own modified scripts: Own script

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD

## To reproduce

Steps to reproduce the behavior:

Source Code:
```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch
import time
from transformers import pipeline

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using {device} for QNA Model prediction")

class QNAModel:
    def __init__(self):
        start_time = time.time()
        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        self.model = BertForQuestionAnswering.from_pretrained(
            'bert-large-uncased-whole-word-masking-finetuned-squad')
        self.pipeline = pipeline("question-answering", model=self.model, tokenizer=self.tokenizer)
        print("--- %s seconds model load time ---" % (time.time() - start_time))

    def get_answers(self, passages, question):
        start_time = time.time()
        answers = []
        for passage in passages:
            context = passage["passage"]
            print(self.pipeline(question=question, context=context))
        print("--- %s seconds Total QNA model prediction---" % (time.time() - start_time))
        return answers
```

Testing
```
qna_model = QNAModel()
qna_model.get_answers(passages1, question1)
qna_model.get_answers(passages1, question2)
```

Logs,
```
Using cpu for QNA Model prediction
--- 16.12714695930481 seconds model load time ---
convert squad examples to features: 100%|██████████| 1/1 [00:00<00:00, 58.58it/s]
add example index and unique id: 100%|██████████| 1/1 [00:00<?, ?it/s]
{'score': 0.39260570705747, 'start': 364, 'end': 380, 'answer': '<answer1>'}
Using cpu for QNA Model prediction
--- 14.559933185577393 seconds model load time ---
convert squad examples to features: 100%|██████████| 1/1 [00:00<00:00, 187.58it/s]
add example index and unique id: 100%|██████████| 1/1 [00:00<?, ?it/s]
{'score': 0.2136769815853059, 'start': 0, 'end': 84, 'answer': '<answer2>'}
--- 51.28193521499634 seconds Total QNA model prediction---
```

## Expected behavior
Model should load once so that prediction is fast

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.7.0
- Platform: Windows
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0+cpu
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi @Sharathmk99, can post the snippet of your server code where you are initialising `QNAModel()` 

I am also facing the same issue? please help

Any fix for this?

This windows issue has not fixed even in the latest versions of transformers.  @patil-suraj  Please help!
```
from transformers import pipeline

model_name = "deepset/roberta-base-squad2"
nlp = pipeline("question-answering", model=model_name, tokenizer=model_name, framework="pt")
nlp(question=question, context=text)
```
This only happens on Windows

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.