Dynamic module import error when using ddp
Author: jinzhen-lin
Pull request: no

### System Info

- `transformers` version: 4.27.3
- Platform: Linux-5.4.0-72-generic-x86_64-with-glibc2.10
- Python version: 3.8.15
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Create a file `a.py` with the following content

```python
from transformers import AutoConfig
AutoConfig.from_pretrained("THUDM/glm-10b", trust_remote_code=True)
```

Run it with `torchrun`

```
torchrun --nproc-per-node 8 a.py
```

Then we would get this error sometimes

```
Traceback (most recent call last):
  File "a.py", line 5, in <module>
    AutoConfig.from_pretrained("THUDM/glm-10b", trust_remote_code=True)
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 911, in from_pretrained
    config_class = get_class_from_dynamic_module(
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 399, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 157, in get_class_in_module
    shutil.copy(f"{module_dir}/{module_file_name}", tmp_dir)
  File "/home/linjinzhen/.miniconda3/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/linjinzhen/.miniconda3/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/home/linjinzhen/.cache/huggingface/modules/transformers_modules/THUDM/glm-10b/696788d4f82ac96b90823555f547d1e754839ff4/configuration_glm.py'
```

or 

```
Traceback (most recent call last):
  File "a.py", line 5, in <module>
    AutoConfig.from_pretrained("THUDM/glm-10b", trust_remote_code=True)
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 911, in from_pretrained
    config_class = get_class_from_dynamic_module(
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 399, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/home/linjinzhen/.miniconda3/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 177, in get_class_in_module
    module = importlib.import_module(module_path)
  File "/home/linjinzhen/.miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers_modules.THUDM.glm-10b.696788d4f82ac96b90823555f547d1e754839ff4.configuration_glm'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '/home/linjinzhen/.cache/huggingface/modules/transformers_modules/THUDM/glm-10b/696788d4f82ac96b90823555f547d1e754839ff4/configuration_glm.py'
```


It seems that it is a multiprocess-relalted issue. 
https://github.com/huggingface/transformers/blob/v4.27.4/src/transformers/dynamic_module_utils.py#L147-L179

### Expected behavior

Dynamic module can be imported successfully when using ddp.


Comments:
Thanks for raising this issue. I think this is linked to #21646, I will push a fix shortly.