Run some TF Whisper tests in subprocesses to avoid GPU OOM
Author: ydshieh
Pull request: yes

# What does this PR do?

Run some TF Whisper tests in subprocesses to avoid GPU OOM (in the PT tests, which run after the TF tests)

After TF loading `openai/whisper-large`, it takes almost all GPU and won't release it, causing some PT tests GPU OOM later.

Current failing CI report [here](https://github.com/huggingface/transformers/actions/runs/3278475639/jobs/5396961322)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger There are 2 considerations:

- `openai/whisper-large` is 6 GB (as file), quite large.
- Also, for `Whisper` model, `TFWhisper` tests run before (Pytorch) `Whisper` tests, due to the `_tf_whisper.py` and `_whisper.py` order (`t` vs `w`). If this order was reversed (and with potentially torch.cuda.empty.cache), we might get rid of the GPU OOM. 

I think this is a combination of the ordering plus very large model which cause the problem. For other models, either PT runs before TF tests, or the models are not large enough to cause problems.

I'm very reluctant to skip those slow tests though, so anything else you might think of to fix the issue would be better.

Close for now while try to find a way to make @sgugger happy 🧠 

@sgugger Could you take a quick look (no need review in detail) if this hacky way of using child process to run the actual testing is acceptable. The downside is: this makes the debugging more difficult (can't set `pdb` in the child process, otherwise fail). Maybe there is some workaround to do the debugging in this case though.

If this is acceptable, I can apply the same change to other TF test methods that use large checkpoints.

There is something `execute_subprocess_async` in `transformers`, but my first impression is it's mainly for launching a python command (training script for example). It won't make things easier here IMO.

Also, the target function (i.e. the child process to run) can't have `self` (i.e. the `TestCase` collected by `pytest`), otherwise it hangs, so it should go outside the test class (I haven't done extensive research on this, just my experience). Therefore, a `decorator` doesn't seems possible.

Since there are very few such use cases, we can focus on current approach, then see if there is a way to make the process easier :-) 🙏 

SGTM!

close for now to not run the tests during dev.

@sgugger @amyeroberts Ready for review :-)

After trying to do with `decorator` (for target function running in subprocess), I find the original `run_test_in_subprocess` as a simple function is much cleaner.

Otherwise, we will end up to prepare the following in `test_large_logits_librispeech` (and in every involved test methods), and pass them to `_test_large_logits_librispeech`
```python
    start_methohd = "spawn"
    ctx = multiprocessing.get_context(start_methohd)
    input_queue = ctx.Queue(1)
    output_queue = ctx.JoinableQueue(1)
```
I would prefer all these constructions being done just once in a single place.

Furthermore, while inside the decorator, we still need the `ctx` in order to do `process = ctx.Process`, which I believe it should be the same `ctx` as the one ` ctx = multiprocessing(...)` shown above, but it is not passed to `_test_large_logits_librispeech` and not available in the `wrapper`.

Therefore I will merge as it is 🙏 . If you have any better approach, we can discuss in another PR, thank you 🙏 


FYI: due to the overhead (2 python processes with TF), the full test suite for this model will fail at `test_large_batched_generation`. I have to split the batch into 2 smaller batches to pass that test, see the last 2 commits.

(when running some subset of the test suite, we don't need to split this way)