Wav2Vec2ProcessorWithLM degraded performance when transcribing multiple files
Author: falcaopetri
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.21.0.dev0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@patrickvonplaten, @anton-l

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This snippet uses https://github.com/falcaopetri/transformers/commit/4d0d36ef66b1fd52942721096665f8bc9574c2b0 to allow setting a pool in `batch_decode`. Full colab example [here](https://colab.research.google.com/drive/1j4UNdqcafKH8WQUYIr871xc8h2A97B_z?usp=sharing).

```python
# based on https://huggingface.co/patrickvonplaten/wav2vec2-large-960h-lv60-self-4-gram
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torch
from jiwer import wer

model_id = "patrickvonplaten/wav2vec2-base-100h-with-lm"
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

model = AutoModelForCTC.from_pretrained(model_id).to("cuda")
processor = AutoProcessor.from_pretrained(model_id)

def map_to_pred(batch, pool=None):
    inputs = processor(batch["audio"]["array"], sampling_rate=16_000, return_tensors="pt")

    inputs = {k: v.to("cuda") for k,v in inputs.items()}

    with torch.no_grad():
        logits = model(**inputs).logits

    transcription = processor.batch_decode(logits.cpu().numpy(), pool=pool).text[0]
    batch["transcription"] = transcription
    return batch


# Current implementation. Pool with 2 workers will be created for each dataset instance
# Running this a second time to reuse cache does not significantly improve runtime (it's still > 15s)
result = ds.map(map_to_pred, remove_columns=["audio"])

print(wer(result["text"], result["transcription"]))
# 100% 73/73 [00:29<00:00, 3.48ex/s]
# 0.057391304347826085

from multiprocessing import get_context

# Alternative implementation. User-managed pool is reused for all instances
with get_context("fork").Pool(None) as pool:
    result = ds.map(map_to_pred, remove_columns=["audio"], fn_kwargs={"pool": pool})

print(wer(result["text"], result["transcription"]))
# 100% 73/73 [00:04<00:00, 17.12ex/s]
# 0.057391304347826085
```


### Expected behavior

I'd expect that instantiating a `Wav2Vec2ProcessorWithLM` allowed me to apply it to multiple audio instances, and that increasing `batch_decode`'s `num_processes` would bring performance improvements for all calls. Current implementation of `batch_decode` creates a `multiprocessing.Pool` at every call, leading to an overhead when decoding multiple files and when increasing `num_processes`.

-----

https://github.com/falcaopetri/transformers/commit/4d0d36ef66b1fd52942721096665f8bc9574c2b0 implements a POC that allows `Wav2Vec2ProcessorWithLM` to reuse the same `multiprocessing.Pool` across multiple `batch_decode` calls. Performance gains can be checked in the previous Colab link.

Allowing the user to manage their own `Pool` is equivalent to how `pyctcdecode` implements [decode_batch](https://github.com/kensho-technologies/pyctcdecode/blob/33478761427b3faad2652ca5b46b158566d88bab/pyctcdecode/decoder.py#L609), but we could also consider having a `Wav2Vec2ProcessorWithLM`-managed pool. For example, as a user I'd expect out-of-the-box performance gains when using more `num_processes`. We should be aware about docs on [multiprocessing.pool.Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) though: 
> Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called (see [object.__del__()](https://docs.python.org/3/reference/datamodel.html#object.__del__) for more information).
> ...
> A frequent pattern found in other systems (such as Apache, mod_wsgi, etc) to free resources held by workers is to allow a worker within a pool to complete only a set amount of work before being exiting, being cleaned up and a new process spawned to replace the old one. The maxtasksperchild argument to the [Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) exposes this ability to the end user.



Comments:
Pool creation might be required to be platform-dependent in the future (https://github.com/huggingface/transformers/pull/17070#issuecomment-1117695494), which means this would be users' responsibility if we go with a user-managed pool scenario.

Hey @falcaopetri,

Thanks for the well-explained issue here! I agree that it would be nicer to let the user pass the pool to the function as an argument. Would you be interested in opening a PR for this? :-)

Sure, I'd be glad to help. I'll add some tests and docs and create a PR over the next few days.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.