Deadlock in Image Processor of ViT by using OpenMP and Kserve
Author: harshyadav17
Pull request: no

### System Info

- `transformers` version: 4.28.0
- Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.2.5
- Python version: 3.8.13
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: Yes (using OpenMP)
- Kserve version: 0.9.0
- g++|gcc (Debian 10.2.1-6) 10.2.1 20210110
- cv2: 4.5.5
- numpy: 1.21.6

### Who can help?

- vision models: @amyeroberts 
- tokenizer: @ArthurZucker 
- PyTorch: @sgu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

We face the deadlock at AutoImageProcessor (forward pass) with the following settings:
1. If we use OMP_NUM_THREADS > 1 (to activate intra-op parallelism in pytorch) and Kserve workers > 1

The code works in the following setting:
1. If OMP_NUM_THREADS == 1, then irrespective of the kserve workers the script works, but the inference time increases (by ~2x).

CODE SNIPPET: https://gist.github.com/harshyadav17/149f1c990c17111d8340fcf2e89a5b88

**In the above code, the deadlock is happening at line 67.** 

### Expected behavior

Successful model prediction with OpenMP variables for optimised inference.

If we continue with non-deadlock (with OMP_NUM_THREADS = 1) setting then there is an increase in the inference time by 2x. We have setup OMP_NUM_THREADS to decrease the latency. The best latency can be checked with optimum number of OMP_NUM_THREADS (set according to the machine, ideally it is num_physical_cores).

Comments:
@amyeroberts can you please help on this ?


By searching "OMP_NUM_THREADS" "deadlock" on Google, it seems it's a general issue when `OMP_NUM_THREADS > 1`. Unfortunately, I am afraid there is no doable fix on `transformers` side.


Furthermore, this issue also involves the usage of `KServe`: it fits better on [HF Forum](https://discuss.huggingface.co/) to see if any user has the same issue and if some workaround is found, or maybe better, on `OpenMP` or `KServe` pages/forums.

Hi @ydshieh
thanks for the prompt response.

I face an interesting correlation with this issue. Whenever we initialise the gcp vision client in the predict method, the pipeline works perfectly.

This is how we can replicate the same.

`from google.cloud import vision`
`os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "<path_to_service_account_key>"`
`_ = vision.ImageAnnotatorClient()`

In order to get the inference, following is the code (to be run from another kernel):

`import requests`
`service_response = requests.post('http://localhost:8095/v1/models/test1234:predict', json={})`

I hope we can make something out of it and contribute to the open source community. This issue happens with most of the Image Processors of Hugging Face.

Thanks!

@harshyadav17 Is it possible for you to remove the parts that involve `Kserve`, and just keep  `OMP_NUM_THREADS > 1` to see if the issue is still there? If we can reproduce in this case, it might be much easier to dive into.

And it's also nice to see if `Kserve workers = 1` will give the issue or not.

@ydshieh with kserve workers = 1, the script runs perfectly. We won't be able replicate the issue if Kserve is removed from the script. 

Moreover, if we add that gcp vision client, the script works with every setting possible. So can we please have a look at why this is happening. GCP client is completely unrelated over here but the Huggingface processor doesn't show us a deadlock. This can help us in implementing the solution to HF processors.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.