Deadlock when loading the model in multiprocessing context
Author: vikigenius
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.2
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes.

Models:

- BART EncoderDecoder: @patrickvonplaten

 -->

## Information

Model I am using (BART):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce

```python
import torch
from pathlib import Path
import multiprocessing as mp
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

queue = mp.Queue()


def load_model(filename):
    device = queue.get()
    print('Loading')
    model = AutoModelForSeq2SeqLM.from_pretrained('models/sqgen').to(device)
    print('Loaded')
    queue.put(device)


def parallel():
    num_gpus = torch.cuda.device_count()

    with mp.get_context('spawn').Pool(processes=num_gpus) as pool:
        for gpu_id in range(num_gpus):
            queue.put('cuda:{0}'.format(gpu_id))
        pool = mp.Pool(processes=num_gpus)
        flist = list(Path('data').glob('*.json'))
        pool.map(
            load_model,
            flist,
        )
        pool.close()
        pool.join()


if __name__ == '__main__':
    parallel()
```


Steps to reproduce the behavior:

1. Run the above script.
2. Script hangs when loading the model.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
1. Script should not hang.

Comments:
Hey @vikigenius,

Could you please use the forum: https://discuss.huggingface.co/ instead for this error? This seems to be quite a special case and we are trying to use Transformers issues only for issues that are directly related to Transformers models. We cannot guarantee that they work for every specific use case such as this one where the model loading is wrapped into Python's multiprocessing functions. 

If you are looking into doing distributed training on GPU, does this doc maybe help: https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision

@patrickvonplaten Thanks for the suggestion, I will post in the forum.

But to give you more context, I am not trying to do distributed training. I am trying to do distributed inference. I have multiple files that I want to generate questions on in parallel. So I am trying to assign each file to some GPU, and then once done, I release the GPU (to the queue) so that others use it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Did someone ever find a proper solution to this? My current hack is to `time.sleep(ordinal * 10)` which prevents the weird race condition