Multiprocessing for pipeline
Author: sharejing
Pull request: no

I am using the question-answering pipeline provided by huggingface. I am trying to perform multiprocessing to parallelize the question answering, but it stucks!!! Eexcution does not end. 

from transformers import pipeline
from torch.multiprocessing import Pool, Process, set_start_method
set_start_method('spawn', force = True)

model_name = "deepset/roberta-base-squad2"
reader = pipeline('question-answering', model=model_name, tokenizer=model_name, device = -1)


def get_answer(input_dict):
  return reader(input_dict)

input_list = []
for i in range(3):
  QA_input = {
          'question': val_questions[i],
          'context': val_contexts[i]
          }
  input_list.append(QA_input)
  
if __name__ == '__main__':
    result =[]
    multi_pool = Pool(processes=3)
    predictions = multi_pool.map(get_answer, input_list)
    multi_pool.close() 
    multi_pool.join()
    print(predictions)

Comments:
same problem! @sharejing any insight so far? 

I was trying to do something like
```
# this is required otherwise it will complain "AssertionError: daemonic processes are not allowed to have children"
class NoDaemonProcess(multiprocessing.Process):
    @property
    def daemon(self):
        return False

    @daemon.setter
    def daemon(self, value):
        pass

class NoDaemonContext(type(multiprocessing.get_context("fork"))):
    Process = NoDaemonProcess

from transformers import pipeline
model_name = "deepset/roberta-base-squad2"
reader = pipeline('question-answering', model=model_name, tokenizer=model_name, device = -1)

def func(input_dict):
     reader(input_dict)

mp_context = NoDaemonContext()
pool = mp_context.Pool(multiprocessing.cpu_count())
async_res = pool.apply_async(func, (input_dict))
async_res.wait()
res = async_res.get()
```
and it got stuck on the `async_res.wait()` line. 

@sharejing 
This code seems to work

```python
from transformers import pipeline
from torch.multiprocessing import Pool, Process, set_start_method

set_start_method("spawn", force=True)

model_name = "deepset/roberta-base-squad2"
reader = pipeline("question-answering", model=model_name, tokenizer=model_name, device=-1)


def get_answer(input_dict):
    print("Input", input_dict)
    return reader(input_dict)


input_list = []
for i in range(3):
    QA_input = {"question": "This is a test", "context": "This is a context"}
    input_list.append(QA_input)

if __name__ == "__main__":
    result = []
    multi_pool = Pool(processes=3)
    predictions = multi_pool.map(get_answer, input_list)
    multi_pool.close()
    multi_pool.join()
    print(predictions)

```

However, I would suggest that instead of sharing `pipelines` (and so `models`) across processes you load them on each thread instead since it will prevent any error on sharing the model across processes (which for instance seems impossible with TF).

```python
from transformers import pipeline
from torch.multiprocessing import Pool, Process, set_start_method

set_start_method("spawn", force=True)

model_name = "deepset/roberta-base-squad2"


PIPE = None


def get_pipe():
    # This will load the pipeline on demand on the current PROCESS/THREAD.
    # And load it only once.
    global PIPE
    if PIPE is None:
        PIPE = pipeline("question-answering", model=model_name, tokenizer=model_name, device=-1)
    return PIPE


def get_answer(input_dict):
    reader = get_pipe()
    print("Input", input_dict)
    return reader(input_dict)


input_list = []
for i in range(3):
    QA_input = {"question": "This is a test", "context": "This is a context"}
    input_list.append(QA_input)

if __name__ == "__main__":
    result = []
    multi_pool = Pool(processes=3)
    predictions = multi_pool.map(get_answer, input_list)
    multi_pool.close()
    multi_pool.join()
    print(predictions)
```

```

Other linked issue :https://github.com/huggingface/transformers/issues/15038

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.