get_class_from_dynamic_module may throw exception in multiple process
Author: maofagui
Pull request: no

### System Info

- `transformers` version: 4.22.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no need
- Using distributed or parallel set-up in script?: no need


### Who can help?

@ArthurZucker  @sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To simplify the problem, I design a simple code fragment to reproduce the bug: 

```
import multiprocessing as mp


def func():
    from transformers import AutoTokenizer, AutoModel

    local_dir = "/Users/mfg/Code/huggingface/glm-10b" # change to your local dir
    tokenizer = AutoTokenizer.from_pretrained(local_dir, trust_remote_code=True)
    model = AutoModel.from_pretrained(local_dir, trust_remote_code=True)
    print(tokenizer)


if __name__ == '__main__':
    procs = []
    for i in range(10):
        p = mp.Process(target=func)
        p.start()
        procs.append(p)

    for p in procs:
        p.join()

    print("done")

```
All files in dir "/Users/mfg/Code/huggingface/glm-10b" can be found in https://huggingface.co/THUDM/glm-10b/tree/main .
(no need to download the large file [pytorch_model.bin](https://huggingface.co/THUDM/glm-10b/blob/main/pytorch_model.bin) for that the exception happens before loading model)

After you run, you may meet the exception as below:
```
Process Process-6:
Process Process-5:
Traceback (most recent call last):
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mfg/Code/torch_submit/mp_load_model/main.py", line 7, in func
    tokenizer = AutoTokenizer.from_pretrained("/Users/mfg/Code/huggingface/glm-10b", trust_remote_code=True)
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 594, in from_pretrained
    tokenizer_class = get_class_from_dynamic_module(
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 375, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 148, in get_class_in_module
    return getattr(module, class_name)
AttributeError: module 'transformers_modules.local.tokenization_glm' has no attribute 'GLMChineseTokenizer'
Traceback (most recent call last):
copy /Users/mfg/Code/huggingface/glm-10b/tokenization_glm.py /Users/mfg/.cache/huggingface/modules/transformers_modules/local/tokenization_glm.py
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mfg/Code/torch_submit/mp_load_model/main.py", line 7, in func
    tokenizer = AutoTokenizer.from_pretrained("/Users/mfg/Code/huggingface/glm-10b", trust_remote_code=True)
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 594, in from_pretrained
    tokenizer_class = get_class_from_dynamic_module(
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 375, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
Process Process-10:
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 148, in get_class_in_module
    return getattr(module, class_name)
Traceback (most recent call last):
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mfg/Code/torch_submit/mp_load_model/main.py", line 7, in func
    tokenizer = AutoTokenizer.from_pretrained("/Users/mfg/Code/huggingface/glm-10b", trust_remote_code=True)
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 594, in from_pretrained
    tokenizer_class = get_class_from_dynamic_module(
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 375, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/opt/miniconda3/envs/py38_torch/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 148, in get_class_in_module
    return getattr(module, class_name)
AttributeError: module 'transformers_modules.local.tokenization_glm' has no attribute 'GLMChineseTokenizer'
AttributeError: module 'transformers_modules.local.tokenization_glm' has no attribute 'GLMChineseTokenizer'
```

In my opinion, the bug is caused by running `shutil.copy` and `importlib.import_module(module_path)` concurrently. 

```
    # lib/python3.8/site-packages/transformers/dynamic_module_utils.py #get_cached_module_file
    if submodule == "local":
        # We always copy local files (we could hash the file to see if there was a change, and give them the name of
        # that hash, to only copy when there is a modification but it seems overkill for now).
        # The only reason we do the copy is to avoid putting too many folders in sys.path.
        shutil.copy(resolved_module_file, submodule_path / module_file)
        for module_needed in modules_needed:
            module_needed = f"{module_needed}.py"
            shutil.copy(os.path.join(pretrained_model_name_or_path, module_needed), submodule_path / module_needed)
```

```
# lib/python3.8/site-packages/transformers/dynamic_module_utils.py
def get_class_in_module(class_name, module_path):
    """
    Import a module on the cache directory for modules and extract a class from it.
    """
    module_path = module_path.replace(os.path.sep, ".")
    module = importlib.import_module(module_path)
    return getattr(module, class_name)
```

Looking forward to your reply. Thanks a lot.

### Expected behavior

no exception

Comments:
I will try to fix this bug and push a commit soon.

Awesome thanks for reporting this! 

This will be fixed by #22537

> This will be fixed by #22537

OK~


I met the same `AttributeError` issue. But the problem still exist after I update the master branch code with this fix https://github.com/huggingface/transformers/pull/22537. I'm wondering if that pr really fix this issue?

<img width="1140" alt="image" src="https://user-images.githubusercontent.com/17028350/231663726-b80068a8-10f3-44d8-94aa-9a96102aed08.png">

Some information:

I saved glm-10b model files on NFS storage, launch 8 process one node. Add debug code as blow:
```python
# /opt/conda/lib/python3.8/site-packages/transformers/dynamic_module_utils.py
def get_class_in_module(class_name, module_path):
    """
    Import a module on the cache directory for modules and extract a class from it.
    """
    module_path = module_path.replace(os.path.sep, ".")
    module = importlib.import_module(module_path)
    try:
        return getattr(module, class_name)
    except:
        with open('/root/.cache/huggingface/modules/transformers_modules/glm-10b-chinese/configuration_glm.py', 'r') as f:
            print('print /root/.cache configuration_glm.py')
            print(f.read())
        raise
```

When the `AttributeError` exception happened, I found this code print `configuration_glm.py` file in `/root/.cache` is empty.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.