Can’t run Parallel inference
Author: dabasmoti
Pull request: no

- `transformers` version: 4.12.3
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.0
- PyTorch version (GPU?): 1.10.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: parallel

Hi,
I have warning message 
`[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
`
while using pipeline

```
pipe = pipeline("sentiment-analysis", model=model, padding=True, max_length=200, truncation=True)
results = pipe(texts)
```
It happened  on both models:
- "distilbert-base-uncased-finetuned-sst-2-english"
- "cardiffnlp/twitter-roberta-base-sentiment"

Only one cpu is executed!

Any suggestions  ?
Thanks

@Narsil @LysandreJik 

Comments:
Hi @dabasmoti,

Can you share the full reproducible script ?
It seems that something early on is messing things up.

If you remove `model=model` then it works perfectly. The model creation is likely to be the issue.

@Narsil 
Thank you for your comment.

Here is the script

```
import pandas as pd
from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
import numpy as np
from scipy.special import softmax
import time
from transformers import pipeline



def using_pip(texts, model):

    print('Starting Pipe')
    pipe = pipeline("sentiment-analysis", model=model, padding=True, max_length=200, truncation=True)
    time0 = time.time()
    results = pipe(texts)
    print(f'Using Pipe: {time.time() - time0}')

def inference(texts, model):
    print('Starting Casual')
    cls = AutoModelForSequenceClassification.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    time0 = time.time()
    preds = []
    for i in texts:
        encoded_input = tokenizer(i, return_tensors='pt', padding=True, max_length=200, truncation=True)
        output = cls(**encoded_input)
        scores = output[0][0].detach().cpu().numpy()
        scores = softmax(scores)
        preds.append(np.argmax(scores))
    print(f'Using Casual: {time.time() - time0}')

def main(texts, model):
    using_pip(texts, model)
    #inference(texts, model)

if __name__ == '__main__':
    df = pd.read_csv('tagged_data/Sentiment_TrainSet_Review_PosNeg_20190107.xlsx - a1401422.csv',nrows=100)
    #df = df[df.label=='neutral']one
    print(f'DataFrame shape: {df.shape}')
    model = None #cardiffnlp/twitter-roberta-base-sentiment
    main(df.Description.to_list(), model)
```

As I said above
I checked it on  the default(model=None) and on - 'cardiffnlp/twitter-roberta-base-sentiment'.
and now the transformer-version is - `4.12.5`


```
DataFrame shape: (100, 21)
Starting Pipe
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
Using Pipe: 8.930904865264893
```





Hi, I couldn't reproduce with pandas since it triggers an error too `AttributeError: 'DataFrame' object has no attribute 'Description'
`. (pandas==1.3.4)

So I modified to 
`model = "cardiffnlp/twitter-roberta-base-sentiment"`
`main(["This is a test", "This is another test"], model)`

And I have as a result
```bash
Starting Pipe
Using Pipe: 0.07680535316467285
Starting Casual
Using Casual: 0.08454775810241699
```

What seems super odd, is that if `texts` is indeed a list of `str` then both should be equivalent. 
Can you tweak your example without pandas and reproduce (or with a file + pandas version necessary to reproduce) ?

Hey,

Tweaked version
```

from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
import numpy as np
from scipy.special import softmax
import time
from transformers import pipeline



def using_pip(texts, model):

    print('Starting Pipe')
    pipe = pipeline("sentiment-analysis", model=model, padding=True, max_length=200, truncation=True)
    time0 = time.time()
    results = pipe(texts, num_workers=8)
    print(f'Using Pipe: {time.time() - time0}')

def inference(texts, model):
    print('Starting Inference')
    cls = AutoModelForSequenceClassification.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    time0 = time.time()
    preds = []
    for i in texts:
        encoded_input = tokenizer(i, return_tensors='pt', padding=True, max_length=200, truncation=True)
        output = cls(**encoded_input)
        scores = output[0][0].detach().cpu().numpy()
        scores = softmax(scores)
        preds.append(np.argmax(scores))
    print(f'Using Casual: {time.time() - time0}')

def main(texts, model):
    using_pip(texts, model)
    inference(texts, model)

if __name__ == '__main__':

    texts = ["I am in Love"] *1000
    #default
    model = "distilbert-base-uncased-finetuned-sst-2-english"  #'cardiffnlp/twitter-roberta-base-sentiment'
    main(texts, model)
```



I get:

`distilbert-base-uncased-finetuned-sst-2-english`
```bash
Starting Pipe
Using Pipe: 25.281177520751953
Starting Inference
Using Casual: 23.430365562438965
```

`cardiffnlp/twitter-roberta-base-sentiment`
```
Starting Pipe
Using Pipe: 45.36654043197632
Starting Inference
Using Casual: 46.86234784126282
```

Could you share you pytorch version + hardware specifications maybe ? (This ran on `pytorch 1.10 + ubuntu 20.04 on i7-4790` so nothing too beafy)
TBH, aside from the `DataLoader` the `pipe` and `inference` should be extremely similar in performance since, they do basically the same thing. (and `DataLoader` shines only on GPU)

torch - 1.10.0
Processor - 2.3 GHz Quad-Core Intel Core i7

Is it a custom build of pytorch ?
Python version maybe (I am in 3.9) ?

We need to single out what makes your environment not working otherwise it'll be hard for me to help.

As I wrote in the first place

transformers version: 4.12.3
Platform: Darwin-20.6.0-x86_64-i386-64bit
Python version: 3.7.0
PyTorch version (GPU?): 1.10.0 (False)

Thanks for your help!

Okay, I tried every single thing I could but I am unable to reproduce.

I am guessing the issues lies in `Darwin` at that point.
I looked into pytorch issues which seem relevant (but cannot try to confirm at this time)
https://github.com/pytorch/pytorch/issues/58585
https://github.com/pytorch/pytorch/issues/46409

@Narsil 
Thanks
I suspected that the issue with `fork` or `spawn` ,but I couldn't find any related issues
Now, I can't find how to change the constructor to `spawn`

This comment seemed helpful to some users:https://github.com/pytorch/pytorch/issues/46409#issuecomment-726495303

If you can manage to make the fix, we'll try to integrate back into pipeline too (the trick part will be identifying to necessary conditions to trigger if we don't understand the bug properly :( )

What a strange behavior!
I have updated the python version to `3.8.9`
and now the 3.8 version is much slower with `pipe` but the warning is gone

```
Loading model took: 10.609
Starting Pipe
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 27.492
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 12.097
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 2.857

```

On 3.7

```
Running Python Version : 3.7.0
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 10.049
Starting Pipe
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:214] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 3.945
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 8.762
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 2.617

```

Script to reproduce
```

from transformers import AutoModelForSequenceClassification
from transformers import AutoTokenizer
import numpy as np
from scipy.special import softmax
import time
from transformers import pipeline
import platform




def using_pip(texts, model):

    print(f'Loading Model - {model}')
    time0 = time.time()
    pipe = pipeline("sentiment-analysis",
                    model=model,
                    padding=True,
                    max_length=200,
                    truncation=True)
    print(f'Loading model took: {round(time.time() - time0,3)}')
    time0 = time.time()
    print('Starting Pipe')
    results = pipe(texts)
    print(f'Using Pipe on {model}-model: {round(time.time() - time0,3)}')

def inference(texts, model):
    print(f'Loading Model - {model}')
    time0 = time.time()
    cls = AutoModelForSequenceClassification.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    print(f'Loading model took: {round(time.time() - time0,3)}')
    time0 = time.time()
    print('Starting Loop Inference')
    preds = []
    for i in texts:
        encoded_input = tokenizer(i,
                                  return_tensors='pt',
                                  padding=True,
                                  max_length=200,
                                  truncation=True)
        output = cls(**encoded_input)
        scores = output[0][0].detach().cpu().numpy()
        scores = softmax(scores)
        preds.append(np.argmax(scores))
    print(f'Using loop on {model}-model: {round(time.time() - time0,3)}')

def main(texts, model):
    using_pip(texts, model)
    inference(texts, model)

if __name__ == '__main__':
    print(f'Running Python Version : {platform.python_version()}')
    texts = ["I am in Love"] *100
    model = "distilbert-base-uncased-finetuned-sst-2-english"  #'cardiffnlp/twitter-roberta-base-sentiment'
    main(texts, model)

```





I ran the same experiments on my MacBook (BigSur, 2.3 GHz Dual-Core Intel Core i5):
```
Running Python Version : 3.8.5
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 10.064
Starting Pipe
/Users/leandro/git/test-pipe-speed/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4 (`cpuset` is not taken into account), which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 27.524
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 7.503
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 3.213
```
The warning made me try `pipe(texts, num_workers=4)`:
```
Running Python Version : 3.8.5
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 8.089
Starting Pipe
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 23.554
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 7.388
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 2.979
```
A little faster. So I also tried `pipe(texts, num_workers=1)`:
```
Running Python Version : 3.8.5
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 8.23
Starting Pipe
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 5.636
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 6.898
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 2.977
```
Finally, `num_workers=0` seems to resolve the issue:
```
Running Python Version : 3.8.5
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 9.538
Starting Pipe
Using Pipe on distilbert-base-uncased-finetuned-sst-2-english-model: 2.921
Loading Model - distilbert-base-uncased-finetuned-sst-2-english
Loading model took: 6.953
Starting Loop Inference
Using loop on distilbert-base-uncased-finetuned-sst-2-english-model: 2.798
```

So it seems to be an issue with parallelism - potentially in the `DataLoader`?


Yes it seems an error in `DataLoader` interferring somehow with inference with some Intel parallelism for tensor calculation. 

I guess at that point disabling DataLoader is the most sensible thing to do (num_workers=0 achieves that).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.