How to use several pipelines in parallel
Author: giladpn
Pull request: no

Hi there. I apologize in advance for posting here, I was not able to get a response on the forum.

I am using several HF pipelines. Boiled down, we are using two pipelines in the same code. Pseudo-code:

```
pipe1 = pipeline("question-answering", model=model_name1)
results1 = pipe1(question=["Who?","What?","Why?"], context=sequences_to_classify)
pipe2 = pipeline("zero-shot-classification", model=model_name2)
results2 = pipe2(sequences_to_classify, candidate_labels=candidate_labels)
```

I am looking for code examples on how to run these two tasks in parallel. I did a search on the community forum and did not find an answer. The basic hope is that it can be done in

Python
+ a RESTful service built with Flask or FastAPI
+ a strong enough GPU can run both tasks with true parallelism

Thank you in advance for any pointers to existing code examples or help!

Comments:
Maybe @Narsil has good pointers!

Not sure what you mean with parallelism here (GPU on CPU parallelism ?)

You can probably put both pipelines on the GPU

```python
pipe1 = pipeline("question-answering", model=model_name1, device=0)
pipe2 = pipeline("zero-shot-classification", model=model_name2, device=0)
```

That should be enough to use your GPU's parallelism.

Depending on the context I would suggest leveraging the `DataLoader` streaming to the GPU (you can pass a dataset pointing to a queue for instance) which should be able to feed the GPU fast enough.

Using both of these you should be pretty close to maximum GPU utilization and it's a good starting point.

Other thoughts:
- Using both pipelines you have less GPU RAM for inference, so longer inferences will trigger errors most likely on either.
- Depending on load/model size data, you could enable batching, but as using 2 pipelines, more GPU utilization means careful with doing too big batch_sizes as it will eat up GPU RAM and might not necessarily speed up. Also if you're doing this, and have a queue dataset mecanism, make sure you're not doing the same work on each worker (i would use `num_workers=1` to be on the safe side first).

Does that help ?

@Narsil Thanks, this helps a great deal. Just to explain context, we're using HF right now for a project and I am coding this code, but I am not an NLP or HF expert. We are learning as we proceed. HF is a great help (thanks).

Some follow on questions: I understand your suggestion to mean the following, please advise if I got it correctly.

1. I should implement the code as follows:
```
pipe1 = pipeline("question-answering", model=model_name1, device=0)
pipe2 = pipeline("zero-shot-classification", model=model_name2, device=0)
results1 = pipe1(question=["Who?","What?","Why?"], context=[context1,context2,context3], num_workers=1)
results2 = pipe2([sequence1,sequence2,sequence3], candidate_labels=[label1,label2,label3], num_workers=1)
```

2. the above will still calculate `result1` and `result2` *one after the other*, so no parallelism there; but your advice is that each step will probably fully use the GPU parallelism so in any case no reason to try to make them even more parallel. Did I get that right?

3. My understanding is that by passing the questions as a list `["Who?","What?","Why?"]` to the question-answering pipeline, and with context sequences `[context1,context2,context3]`, the internal implementation will make a dataset out of the `(question, context)` pairs; so I am supposed to get your proposed "queue dataset mechanism" out of the box. Is that understanding correct, or do I need to change the above code?

4. Ditto for the second pipeline: my understanding is that by passing the sequences to classify as a list `[sequence1,sequence2,sequence3]` to the zero-shot-classification pipeline, and with labels `[label1,label2,label3]`, the internal implementation will make a dataset out of the `(sequence, label)` pairs; so I am supposed to get your proposed "queue dataset mechanism" out of the box once again. Is that understanding correct, or do I need to change the above code?

Many thanks.

> @Narsil Thanks, this helps a great deal. Just to explain context, we're using HF right now for a project and I am coding this code, but I am not an NLP or HF expert. We are learning as we proceed. HF is a great help (thanks).
> 
> Some follow on questions: I understand your suggestion to mean the following, please advise if I got it correctly.
> 
>     1. I should implement the code as follows:
> 
> 
> ```
> pipe1 = pipeline("question-answering", model=model_name1, device=0)
> pipe2 = pipeline("zero-shot-classification", model=model_name2, device=0)
> results1 = pipe1(question=["Who?","What?","Why?"], context=[context1,context2,context3], num_workers=1)
> results2 = pipe2([sequence1,sequence2,sequence3], candidate_labels=[label1,label2,label3], num_workers=1)
> ```
> 
>     1. the above will still calculate `result1` and `result2` _one after the other_, so no parallelism there; but your advice is that each step will probably fully use the GPU parallelism so in any case no reason to try to make them even more parallel. Did I get that right?

These will indeed be calculated sequentially, but my understanding was that you were using something like flask, which would be generating workers and handling the parallelism by using threads/processes for you.

```python
@route(/model1):
def bla(request):
    pipe1 = pipeline(...)
    return JSON(pipe1(request.load)

@route(/model2):
def bla(request):
    pipe2 = pipeline(...)
    return JSON(pipe1(request.load)
```

should work out of the box with thread parallelism if you deploy flask standardly.
You'll soon hit problems. Here for example you would be loading the models on every requests, which is bad, you want to cache the pipeline in someway (look at caching for your webserver of choice).


> 
>     2. My understanding is that by passing the questions as a list `["Who?","What?","Why?"]` to the question-answering pipeline, and with context sequences `[context1,context2,context3]`, the internal implementation will make a dataset out of the `(question, context)` pairs; so I am supposed to get your proposed "queue dataset mechanism" out of the box. Is that understanding correct, or do I need to change the above code?

No you need to change it a bit. I can't say exactly what's your best solution for your use case so I'll give you hints instead.
It's the second caveat with ML on webservers on GPU, you want to get 100% GPU utilization continuously when hammering the server, this requires a specific setup to achieve (naive solution from above won't work, because the GPU won't be fed fast enough most likely, check it first, if indeed your are not hitting 100% come read this.)

For `Dataset` you need to implement yourself with something like

```python
from torch.utils.data import Dataset
class MyDataset(Dataset):
   def __init__(self):
       ....

   def __len__(self):
     ....

  def __getitem__(self, i):
      return {"question": "xxx", "context": "bbb"}
```
There are multiple ways to achieve your dataset depending on your context exactly. If you are using flask server, my guess is that you can't use a `Dataset` (they have a static length) but more an `IterableDataset` (https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset).
Which is a bit more standard for webserver as it doesn't have a specific length associated (but try to read the docs it has caveats you have to be aware of).

You want to setup this way, because sending data from CPU to GPU takes time, and while the GPU is processing one request, you can prepare the next item, that is essential to get good GPU utilization.  `Dataset` and `IterableDataset` are just built-in of Pytorch that play nicely with `DataLoader` which alleviates you (and this library) the pain of making sure the parallelism works properly. `batch_size` and `num_workers` are your essential parameters to tune everything (start with everything at 1, you still should get good GPU usage without those)

To be honest there are multiple ways you could set this up, with different level of complexity and effectiveness.
Having 1 thread per pipeline that just waits on a queue and run inference whenever elements come in and your webserver sending to the appropriate queue is one of them (maybe the simplest)

The core thing when testing your server, is to check that if you overload it you are actually using 100% of your GPU (or something close enough like 90+). That's a big bottleneck effectively wasting a good amount of resources if not done properly (and it's easy to mess up).


> 
>     3. Ditto for the second pipeline: my understanding is that by passing the sequences to classify as a list `[sequence1,sequence2,sequence3]` to the zero-shot-classification pipeline, and with labels `[label1,label2,label3]`, the internal implementation will make a dataset out of the `(sequence, label)` pairs; so I am supposed to get your proposed "queue dataset mechanism" out of the box once again. Is that understanding correct, or do I need to change the above code?

Same answer as `2.`.

For completeness's sake, there's ongoing work to enable `batch_size` for those two pipelines which are a bit specific.
For reference: https://github.com/huggingface/transformers/pull/14225 (but you shouldn't need `batch_size` as a first step anyway)
> 
> 
> Many thanks.



That is a lot of information, thanks! We'll chew on it for a while. Just FYI your reference: #14225 really does sound interesting for us for the subsequent steps, appreciated!

As a bonus question please :-)  I've implemented a class that inherits from your zero-shot classifier in order to do some extra code that we need in the post-processing. So basically all other methods are untouched but in my code I

1. created a class that inherits from `ZeroShotClassificationPipeline`
2. overloaded `postprocess()` and wrote what we need in there
3. added my class to the list of pipelines in `transformers.pipelines.SUPPORTED_TASKS` (only *in memory*, I *didn't* change the code in `transformers.pipelines`!).
4. used that as a normal pipeline:
```
pipe = pipeline("my-zero-shot-classification", model=hg_model_hub_name)
```

Reading your code (nice code by the way), that seemed a good way to reuse. But let me know if I am on shaky ground here. Thanks again!

That's nice modification !

It seems overloading pipelines is getting trendy I have seen it pop up in other issues, maybe we can make the process even easier. I'll see what can be done.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.