Suddenly 'FileNotFoundError: [Errno 2] No such file or directory' error came, the other issues were of no help
Author: vhdmoradi
Pull request: no

I was doing fine with the sentense-transformers using BAAI/bge-m3 model, when out of nowhere it started giving me errors on my docker container:
```
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/backend/products/urls.py", line 3, in <module>
    from .views import ProductViewSet
  File "/backend/products/views.py", line 13, in <module>
    from .utils import search, functions
  File "/backend/products/utils/search.py", line 9, in <module>
    from .functions import generate_embeddings, qdrant_client, generate_slug
  File "/backend/products/utils/functions.py", line 16, in <module>
    encoder = SentenceTransformer("BAAI/bge-m3")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 190, in __init__
    modules = self._load_sbert_model(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 1175, in _load_sbert_model
    module = module_class.load(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentence_transformers/models/Pooling.py", line 227, in load
    with open(os.path.join(input_path, "config.json")) as fIn:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/hub/models--BAAI--bge-m3/snapshots/50f9396f75618b3389c1fd1068a1ff58dc7b5b26/1_Pooling/config.json'

```
Then I tried to run things on my vps server as I normally do when testing, but there also I encountered this problem. 

In my vps, I try first by activating the .venv which I have sentense-transformers installed, then I type 'python' to open the interpretor:
```
Python 3.12.2 (main, Mar  4 2024, 06:25:05) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
```
Then I try to import the package and point to the model:
```
>>> from sentence_transformers import SentenceTransformer
>>> encoder = SentenceTransformer("BAAI/bge-m3")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cowsel/www/e-commerce/backend/searchai/vector_search/.venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 191, in __init__
    modules = self._load_sbert_model(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cowsel/www/e-commerce/backend/searchai/vector_search/.venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 1246, in _load_sbert_model
    module = module_class.load(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cowsel/www/e-commerce/backend/searchai/vector_search/.venv/lib/python3.12/site-packages/sentence_transformers/models/Pooling.py", line 227, in load
    with open(os.path.join(input_path, "config.json")) as fIn:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/cowsel/.cache/huggingface/hub/models--BAAI--bge-m3/snapshots/50f9396f75618b3389c1fd1068a1ff58dc7b5b26/1_Pooling/config.json'
```

Whis is really confusing, I tried deleting the cache folder, and doing things from start, but still no success!

Comments:
cc @tomaarsen this is unrelated to `transformers` and linked to `sentence_transformers `

Got the same error as you, seems to be fixed now.

> cc @tomaarsen this is unrelated to `transformers` and linked to `sentence_transformers `

So sorry, I got the names confused. But it seems it has been automatically solved as it came out of nowhere in the first place. 

I'm afraid I can't transfer the issue to Sentence Transformers as it falls under another organization, so I'll answer here.

I actually encountered the same issue while testing yesterday, and it also disappeared for me after a minute or 2. Perhaps some API was down temporarily which resulted in this odd crash.

I'll dive into it further if it happens again.

- Tom Aarsen 

> I'm afraid I can't transfer the issue to Sentence Transformers as it falls under another organization, so I'll answer here.
> 
> I actually encountered the same issue while testing yesterday, and it also disappeared for me after a minute or 2. Perhaps some API was down temporarily which resulted in this odd crash.
> 
> I'll dive into it further if it happens again.
> 
> * Tom Aarsen

Thank you for you answer, yes it is now solved with no problem. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Same thing's happening to me - trying to run the new training on AWS using DDP and got the same error two times (yesterday and today). I'll try and add some more tracing

```
ErrorMessage "RuntimeError: unable to open file </root/.cache/huggingface/hub/models--sentence-transformers--all-mpnet-base-v2/snapshots/84f2bcc00d77236f9e89c8a360a00fb1139bf47d/model.safetensors> in read-only mode: No such file or directory (2)
```

So I "fixed" this by adding

```
ENV HF_HOME=/root/.cache/huggingface
```

To my custom container - I've not investigated what the cause was, I switched to training with DDP and made a few other changes such as adding

```
# Set this enviroment variable for SageMaker to launch SMDDP correctly.
ENV SAGEMAKER_TRAINING_MODULE=sagemaker_pytorch_container.training:main
```

My guess is this is causing the container to download the model to one location before forking the process then looking in another after forking but I'm not totally sure - either way having HF_HOME relative to `/` rather than `~` seems to be a good practice in containers.

@david-waterworth I get this on DDP training runs too, but only the very first time I run the script. Afterwards, it'll run totally normally again, presumably because it's still cached. I have no idea what causes it.

- Tom Aarsen 

@tomaarsen I think what might be happening here is the model is being downloaded 4 times to the same folder causing a race - I would assume only the rank 0 process should download the model and the rest wait but that doesn't appear to be happening. I can see 4 different entries for each log message

So I added this - hard to tell if it fixes the problem as it's intermittent.

```
    # ensure different rank processes don't step on each other?
    cache_dir = f"/root/.cache/{uuid.uuid4().hex}"
    logging.debug("using cache_dir=%s", cache_dir)
    word_embedding_model = models.Transformer(args.model_path, max_seq_length=args.max_seq_length, cache_dir=cache_dir)
``