Reducing CPU usage during decoding
Author: bloodraven66
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-1032-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in

### Who can help?

@sanchit-gandhi I'm looking at doing CPU decoding with Wav2Vec2ForCTC, and Wav2Vec2Processor, using a fixed number of CPU cores. I've tried setting num_workers = 1 but it does not decrease the CPU load. I could not find any solution by looking at the documentation.

Are there any inbuilt args/kwargs to define CPU load? Or should I use external tooling for it?


### Information

- [X] The official example scripts

### Reproduction

```
processor = Wav2Vec2Processor.from_pretrained(wav2vec_hf_key, repo_type="model")
model = Wav2Vec2ForCTC.from_pretrained(wav2vec_hf_key)

audio_input, sample_rate = sf.read(wav_file)
input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.decode(predicted_ids[0], skip_special_tokens=True)
```

### Expected behavior

I'm hoping that there is a way to control CPU load while decoding with transformers. I do not want the code to access all system resources.

Comments:
Hi @bloodraven66, thanks for raising this issue. 

For us to be able to help you'll need to provide some more information and context: 
* How and where is `num_workers=1` being set?
* Could you provide a code snippet which produces the conditions seen? We should be able to copy-paste the code and run it directly. 
* What do you mean by 'does not reduce the CPU load'? Is it the number of processes or RAM being used which doesn't reduce? 
* Is the memory spike being seeing during processing? Forward pass?

You can control the max memory to use on CPU for the model when loading with `from_pretrained` with `max_memory`. You'll also need to provide an offload folder. 

```
model = Wav2VecForCTC.from_pretrained(checkpoint, max_memory={"cpu": max_cpu_ram}, offload_folder=offload_folder, offload_state_dict=True)
```



Indeed an end-to-end reproducible code snippet to emulate the behaviour would be most useful! Note that you can also easily reduce the amount of CPU usage through the better transformers integration: https://huggingface.co/docs/transformers/perf_infer_cpu#bettertransformer-for-faster-inference

Better transformers will enable flash attention, which gives a nice memory gain and latency improvement during inference

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.