Fix dynamic module import error
Author: ydshieh
Pull request: yes

# What does this PR do?

### Issue

We have failing test 
```bash
FAILED tests/models/auto/test_modeling_auto.py::AutoModelTest::test_from_pretrained_dynamic_model_distant

ModuleNotFoundError: No module named 'transformers_modules.local.modeling'
```
The full trace is given at the end.

After a long debug process, it turns out that, when reloading from the saved model
```python
model = AutoModel.from_pretrained("hf-internal-testing/test_dynamic_model", trust_remote_code=True)
with tempfile.TemporaryDirectory() as tmp_dir:
    model.save_pretrained(tmp_dir)
    reloaded_model = AutoModel.from_pretrained(tmp_dir, trust_remote_code=True)
```
if `configuration.py` appears in the dynamic module directory (here `transformers_modules/local`), sometimes it interferes the import of `transformers_modules.local.modeling`. I have no clear reason for this situation however.

### What this PR fixes

This PR therefore tries to avoid the appearance of other module files while the code imports a specific module file, around this line
```
def get_class_in_module():
    ...
    module = importlib.import_module(module_path)
    ...
```

### Result

Running the reproduce code snippet (provided in the comment below) in a loop of 300 times:

 - with this PR: this issue doesn't appear, [job run](https://app.circleci.com/pipelines/github/huggingface/transformers/57824/workflows/fb3b74ed-9231-41f8-80c9-7d43fb871a35/jobs/702257/steps)
 
 - without the fix: this issue appears with 50% probability [job run](https://app.circleci.com/pipelines/github/huggingface/transformers/57826/workflows/1dee1636-9aa3-433c-a044-0c4c4c9dcbff/jobs/702277/steps)

#### Full traceback
```bash
Traceback (most recent call last):
    ...
    reloaded_model = AutoModel.from_pretrained(tmp_dir, trust_remote_code=True)
  File "/home/circleci/.pyenv/versions/3.7.12/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py", line 463, in from_pretrained
    pretrained_model_name_or_path, module_file + ".py", class_name, **hub_kwargs, **kwargs
  File "/home/circleci/.pyenv/versions/3.7.12/lib/python3.7/site-packages/transformers/dynamic_module_utils.py", line 367, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/home/circleci/.pyenv/versions/3.7.12/lib/python3.7/site-packages/transformers/dynamic_module_utils.py", line 147, in get_class_in_module
    module = importlib.import_module(module_path)
  File "/home/circleci/.pyenv/versions/3.7.12/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'transformers_modules.local.modeling'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Run the following commpand
```python
python run_debug.py
```
with the 2 files

### run_debug.py
```python
import os

for i in range(300):
    print(i)
    with open("output.txt", "a+") as fp:
        fp.write(str(i) + "\n")
    os.system("python3 debug.py")
```
(we need to run the debugging code `foo` (contained in file `debug.py`) in difference processes each time, instead of running the script `debug.py` with a for loop defined inside it - as this will be always in the same process)

### debug.py
```python
import time, traceback, tempfile, os
from transformers.utils import HF_MODULES_CACHE


def foo():
    from transformers import AutoModel

    model = AutoModel.from_pretrained("hf-internal-testing/test_dynamic_model", trust_remote_code=True)
    # Test model can be reloaded.
    with tempfile.TemporaryDirectory() as tmp_dir:
        model.save_pretrained(tmp_dir)
        try:
            reloaded_model = AutoModel.from_pretrained(tmp_dir, trust_remote_code=True)
        except Exception as e:
            print(e)
            with open("output.txt", "a+") as fp:
                fp.write(f"{traceback.format_exc()}" + "\n")


if __name__ == "__main__":
    timeout = os.environ.get("PYTEST_TIMEOUT", 10)
    timeout = int(timeout)
    for i in range(1):
        time.sleep(1)
        print(i)
        with open("output.txt", "a+") as fp:
            fp.write(str(i) + "\n")
        try:
            os.system(f'rm -rf "{HF_MODULES_CACHE}"')
        except:
            pass
        foo()
        print("=" * 80)
        with open("output.txt", "a+") as fp:
            fp.write("=" * 80 + "\n")
```

Thanks for working on this! I was going to have a look at it when back from vacation but if you beat me to it ;-)

My solution would have been to change the way the local module works: for now I dumb every file there without structure, I wanted to add a folder per model (so given by `pretrained_model_name_or_path`) which would also fix this issue I believe.

@sgugger I am open to explore further, but I have a bit doubt regarding

>  I wanted to add a folder per model  (so given by `pretrained_model_name_or_path`) which would also fix this issue I believe.

While I am debugging (this single test), the only model appears

```
transformers_modules/hf-internal-testing/test_dynamic_model/12345678901234567890.../
transformers_modules/local/
```
so I don't see multiple models sharing the same folder, but the issue still occurs. So, I am not sure how to proceed with the solution you mentioned above.

Hmm, there seems to affect other related tests. I will have to take a look 😭 

I believe the conflict is between two files in local being written/deleted concurrently (but I might be wrong) hence making sure we things like
```
transformers_modules/local/123456...
transformers_modules/local/777888...
```
might fix the issue.

> I believe the conflict is between two files in local being written/deleted concurrently

On (circleci) CI, we have `pytest -n 8`, which might cause the situation you mentioned. But I am debugging by running the following function in a loop (and the issue still appears), so I kinda feel the issue is not from the concurrently read/write/delete operations

```python
def foo():
    from transformers import AutoModel

    model = AutoModel.from_pretrained("hf-internal-testing/test_dynamic_model", trust_remote_code=True)
    # Test model can be reloaded.
    with tempfile.TemporaryDirectory() as tmp_dir:
        model.save_pretrained(tmp_dir)
        reloaded_model = AutoModel.from_pretrained(tmp_dir, trust_remote_code=True)
```

I could explore anyway - but maybe let me finalize the current PR (make CI green) first 

Finally get it:

- we don't need to remove other files (config, `__init__.py`) or `__pycache__` folder
- the point is: we need to remove the `module_file_name` in a subprocess, then copy it back
  - os.system("rm -rf ...") works: as it is in another process
  - os.system(f"python3 -c '{cmd}'"): same, but we don't use Linux specific command --> way to go
  - os.remove(...): not working! I could not explain (as I don't know the reason behind) 😢 


Don't know why we get an error where a module is not a python file, but a package. See below.
Can't reproduce so far, but the fix works for the auto model dynamic loading test.

```bash
FAILED tests/models/auto/test_image_processing_auto.py::AutoImageProcessorTest::test_from_pretrained_dynamic_image_processor

 - ModuleNotFoundError: No module named 'transformers_modules.local__tmp_tmpkcj_lb5j'
```

This PR is ready for review.

There is one failure thtat I can't reproduce with the same code snippet. See [this comment](https://github.com/huggingface/transformers/pull/21646#issuecomment-1433336303). It seems this happens much rarely. And probably we can investigate it if it happens again. 



Thanks for investigating so deeply this issue!