Bug when re-initializing CUDA in forked subprocess
Author: cassianlewis
Pull request: no

### System Info

```
- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.10.178-162.673.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
```

### Who can help?

@younesbelkada @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Following 3 cells in Jupyter Noteboook:

```
from transformers import AutoModelForSeq2SeqLM
from accelerate import notebook_launcher
import torch
torch.cuda.is_initialized()
```
output: `False`

```
def run():
    model = AutoModelForSeq2SeqLM.from_pretrained(
            't5-small',
            load_in_8bit = 'True',
            cache_dir='model_cache')
    print(model)
```

```
notebook_launcher(run, num_processes = 2)
```

Note: runs fine if I remove the entire `load_in_8bit = 'True',` line.

### Expected behavior

Following error:

```
---------------------------------------------------------------------------
ProcessRaisedException                    Traceback (most recent call last)
File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/launchers.py:137, in notebook_launcher(function, args, num_processes, mixed_precision, use_port)
    136 try:
--> 137     start_processes(launcher, args=args, nprocs=num_processes, start_method="fork")
    138 except ProcessRaisedException as e:

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py:197, in start_processes(fn, args, nprocs, join, daemon, start_method)
    196 # Loop on join until it returns True or raises an exception.
--> 197 while not context.join():
    198     pass

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py:160, in ProcessContext.join(self, timeout)
    159 msg += original_trace
--> 160 raise ProcessRaisedException(msg, error_index, failed_process.pid)

ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/utils/launch.py", line 543, in __call__
    self.launcher(*args)
  File "/tmp/ipykernel_30342/662585938.py", line 4, in run
    model = AutoModelForSeq2SeqLM.from_pretrained(
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained
    trust_remote_code = resolve_trust_remote_code(
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2330, in from_pretrained
    device_map = {"": torch.cuda.current_device()}
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/cuda/__init__.py", line 679, in current_device
    _lazy_init()
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/cuda/__init__.py", line 235, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method


The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
Cell In[21], line 1
----> 1 notebook_launcher(run, num_processes = 2)

File ~/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/launchers.py:140, in notebook_launcher(function, args, num_processes, mixed_precision, use_port)
    138         except ProcessRaisedException as e:
    139             if "Cannot re-initialize CUDA in forked subprocess" in e.args[0]:
--> 140                 raise RuntimeError(
    141                     "CUDA has been initialized before the `notebook_launcher` could create a forked subprocess. "
    142                     "This likely stems from an outside import causing issues once the `notebook_launcher()` is called. "
    143                     "Please review your imports and test them when running the `notebook_launcher()` to identify "
    144                     "which one is problematic."
    145                 ) from e
    147 else:
    148     # No need for a distributed launch otherwise as it's either CPU, GPU or MPS.
    149     if is_mps_available():

RuntimeError: CUDA has been initialized before the `notebook_launcher` could create a forked subprocess. This likely stems from an outside import causing issues once the `notebook_launcher()` is called. Please review your imports and test them when running the `notebook_launcher()` to identify which one is problematic.
```

Comments:
CC @SunMarc @younesbelkada, `transformers` may need a solution similar to what I had to do here: https://github.com/huggingface/accelerate/pull/1813

Sorry for the goose-chase, this is indeed accelerate: https://github.com/huggingface/accelerate/pull/1833