Zeroshot pipeline performance worse on CPU when processing multiple texts as "batch"
Author: buhrmann
Pull request: no

Hi, I'm getting weird performance results using the zeroshot pipeline on a laptop with CPU. Essentially piping 5 texts through it at the same time is about 3x _slower_ than just iterating over the texts one by one:

``` python
texts = ...
classifier = pipeline("zero-shot-classification", model="joeddav/xlm-roberta-large-xnli")
t0 = time()
res1 = classifier(texts, labels, template, multi_label=False)
t1 = time()
res2 = [classifier(txt, labels, template, multi_label=False) for txt in texts]
t2 = time()
print(t1-t0, t2-t1)
``` 
```
>>> 85.13976335525513 27.092346906661987
```

The results are the same (other than some decimals in probabilities). In both cases 4 CPUs are utilized pretty much constantly at 100%. I don't know the code, but perhaps there is an attempt to parallelize at the level of texts, which is being blocked by the GIL or something? Perhaps it's just a documentation issue and batch processing is not supported on CPU?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Linux-5.6.0-1055-oem-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@joeddav 

## Information

Model I am using: joeddav/xlm-roberta-large-xnli

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

See example above. For reference, the 5 texts are random samples from the 20 newsgroups dataset:

```python
['I have to disagree with you on this one. It is anything BUT common. In the 4 or 5 years I have been watching hockey I have NEVER seen this happen EVER. I am not sure what league you have been watching. :-) Anyone else agree with this?',
 'About a month ago there was a photo posted on alt.binaries.pictures.misc of a 17.5-inch Northern Pike which had been caught on a lure made of 256K SIMMs. --',
 "You can't. But good luck trying.",
 ": The cops/feds do *not* need to be able to get hold of your private key to : listen in to cellular conversations. Encryption is not end-to-end, but : cellphone to base-station - it *has* to be this way so that cellular users : and fixed installations can talk to each other. For cellular to cellular : calls, the transmission is decrypted at the base-station, passed to another : base-station and re-encrypted. The cops/feds can listen to the unscrambled : call *provided* they get a warrant to tap into the cellular provider's : equipment. The only reason for wanting a crackable system is so they can : listen without having to obtain a warrant. : But, maybe the Clipper system is secure, and they really do need a warrant : to get the key out of escrow before they can listen in using a scanner (see : above - they don't *have* to go down this route anyway). I have my doubts, : but even if true once they have the key they will *never* again need a : warrant to tap into that particular phone whenever they want. `Well, Judge, : it appears he wasn't a drug-dealer after all, so naturally we'll stop : listening in'... That was true for the UK Paul, but I'm fairly sure they're talking about building end-to-end encryption phones out of this chip. It's *not* for cellular (though it certainly could be used there in the way you suggest)",
 'I am trying to get a copy of the _official_ rules of baseball. Someone once sent me the ISBN number of it, but I have since lost it. Can anyone give me this information, or tell me where I can find the book? None of my local bookstores have it.']
```

## Expected behavior

Piping multiple texts through the pipeline should be at least as fast, and ideally faster, than iterating over individual texts.


Comments:
The reason is likely padding. When passed through as a batch, the shorter sequences have to be padded to the length of the longest sequence. On GPU, you'll still get a speedup because batching allows so much more parallel computing to happen that it makes up for the padding. But on CPU you just end up with more pad tokens to be processed without as much parallelization speedup. I bet if your 5 sequences were all approx the same length, the difference in compute times would be far smaller or the batched might even be faster.

That makes perfect sense. Thanks for the quick response! May be a good idea to try breaking up larger texts into similarly sized chunks then I guess. I'll try that if I get around to it.

I think on CPU, the simplest and best solution is probably going to be to just pass each sequence one at a time (at least if you have wildly variable-length sequences like in your example).

That's what I'm doing for now, thanks. I think you can close this issue then (unless you want to keep it around to add something to the docs at some point, but I guess low-resource inference using CPU only is not the typical use case anyway).