Poor CPU Inference Scalability Possibly Due to  Disk IO
Author: jrsperry
Pull request: no

### System Info

Environment: 
Python: 3.10.8
Pytorch: 1.12.1
Transformers: 4.24.0
Model Used: cardiffnlp/twitter-roberta-base-sentiment

Tests Performed: 

I created a simple docker image based on this [github repo](https://github.com/jrsperry/transformers-sentiment-test) which would make predictions on sentences and print out the time to make said predictions. I did this in a kubernetes cluster on aws with node types of c6i.8xlarge, m6a.4xlarge, and several others. 

When running with only 1 pod replica with a limit of 4 cpu I got the expected performance. When I run multiple pods per node (more than 2) with the same limits set the performance per pod absolutely falls off a cliff. All the pods are still getting the same amount of cpu as the first test with 1 pod, and there's no clear ram pressure, but some of the performance per pod is 10 times slower than before. I've confirmed that all pods were getting the same amount of cpu (4) as well. 

I ended up adding a higher class of storage to the kubernetes node to see if it was IO bound and my performance improved dramatically and was much closer to being inline with my expectations given what I observed with the single pod test. 

In my test image I don't write to disk so I'm a bit perplexed as to where the disk pressure is coming from, whether it's expected, and if there's any way to optimize around it.

I've run many of these tests with different base images and slightly different transformers and pytorch versions and all of them suffered the same kind of performance drop off. If I provide the same amount of cpu to each pod, I would expect a similar amount of performance. It's possible this isn't IO based, it's just the only thing that brought performance back closer to expectations, although confusingly not in every situation, and varied in its impact.  The environment that responded best to the ssd drive had python 3.10.8, torch 1.12.1 and transformers 4.24.0. 




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

-  Run multiple instances of the sperry1996/transformers-sentiment-test:0.0.1 image on a kubernetes node and compare the results vs running 1 instance per node (with equal requests and limits set).  Alternatively you can run multiple containers locally with the command `docker run --cpus="0.75"  sperry1996/transformers-sentiment-test:0.0.1` (on an x86 machine) or `docker run --cpus="0.75"  sperry1996/transformers-sentiment-test:0.0.1-CPU-MULTI` (on an arm machine). I experienced slowdowns of around 4x (when running more than 1 container) on an m1 mac with docker desktop and around 25-30% on a windows with docker desktop (mac docker desktop has worse disk io performance than windows, there may be other factors as well). 

### Expected behavior

I would expect fairly linear scaling of the instances of the pods given they have equal requests and limits set. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.