Unable to load wavlm-large from pretrained in offline mode
Author: Robinysh
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 4.40.0
- Platform: Linux-5.14.0-362.24.1.el9_3.0.1.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.9
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The code below still tries to connect to the internet. I am unsure whats so special about `wavlm-large` thats causing this issue.
```python
import os
os.environ['HF_DATASETS_OFFLINE'] = '1'
os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
from transformers import AutoModel
AutoModel.from_pretrained('microsoft/wavlm-large', trust_remote_code=True)
```

output
```
{
	"name": "ConnectionError",
	"message": "HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/wavlm-large/resolve/main/model.safetensors (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9ebc1447d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))",
	"stack": "---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connection.py:196, in HTTPConnection._new_conn(self)
    195 try:
--> 196     sock = connection.create_connection(
    197         (self._dns_host, self.port),
    198         self.timeout,
    199         source_address=self.source_address,
    200         socket_options=self.socket_options,
    201     )
    202 except socket.gaierror as e:

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84 try:
---> 85     raise err
     86 finally:
     87     # Break explicitly a reference cycle

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
     72     sock.bind(source_address)
---> 73 sock.connect(sa)
     74 # Break explicitly a reference cycle

OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

NewConnectionError                        Traceback (most recent call last)
File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connectionpool.py:789, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    788 # Make the request on the HTTPConnection object
--> 789 response = self._make_request(
    790     conn,
    791     method,
    792     url,
    793     timeout=timeout_obj,
    794     body=body,
    795     headers=headers,
    796     chunked=chunked,
    797     retries=retries,
    798     response_conn=response_conn,
    799     preload_content=preload_content,
    800     decode_content=decode_content,
    801     **response_kw,
    802 )
    804 # Everything went great!

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connectionpool.py:490, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    489         new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 490     raise new_e
    492 # conn.request() calls http.client.*.request, not the method in
    493 # urllib3.request. It also calls makefile (recv) on the socket.

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connectionpool.py:466, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    465 try:
--> 466     self._validate_conn(conn)
    467 except (SocketTimeout, BaseSSLError) as e:

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connectionpool.py:1095, in HTTPSConnectionPool._validate_conn(self, conn)
   1094 if conn.is_closed:
-> 1095     conn.connect()
   1097 # TODO revise this, see https://github.com/urllib3/urllib3/issues/2791

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connection.py:615, in HTTPSConnection.connect(self)
    614 sock: socket.socket | ssl.SSLSocket
--> 615 self.sock = sock = self._new_conn()
    616 server_hostname: str = self.host

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connection.py:211, in HTTPConnection._new_conn(self)
    210 except OSError as e:
--> 211     raise NewConnectionError(
    212         self, f\"Failed to establish a new connection: {e}\"
    213     ) from e
    215 # Audit hooks are only available in Python 3.8+

NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9ebc1447d0>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/adapters.py:667, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    666 try:
--> 667     resp = conn.urlopen(
    668         method=request.method,
    669         url=url,
    670         body=request.body,
    671         headers=request.headers,
    672         redirect=False,
    673         assert_same_host=False,
    674         preload_content=False,
    675         decode_content=False,
    676         retries=self.max_retries,
    677         timeout=timeout,
    678         chunked=chunked,
    679     )
    681 except (ProtocolError, OSError) as err:

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/connectionpool.py:843, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    841     new_e = ProtocolError(\"Connection aborted.\", new_e)
--> 843 retries = retries.increment(
    844     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    845 )
    846 retries.sleep()

File ~/.conda/envs/cuda/lib/python3.11/site-packages/urllib3/util/retry.py:519, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    518     reason = error or ResponseError(cause)
--> 519     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    521 log.debug(\"Incremented Retry for (url='%s'): %r\", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/wavlm-large/resolve/main/model.safetensors (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9ebc1447d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[1], line 10
      7 from huggingface_hub import snapshot_download
      8 # AutoModel.from_pretrained(snapshot_download('microsoft/wavlm-large'), local_files_only=True)
      9 # AutoModel.from_pretrained('distil-whisper/distil-large-v3')
---> 10 AutoModel.from_pretrained('microsoft/wavlm-large', trust_remote_code=True)

File ~/.conda/envs/cuda/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:563, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    561 elif type(config) in cls._model_mapping.keys():
    562     model_class = _get_model_class(config, cls._model_mapping)
--> 563     return model_class.from_pretrained(
    564         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    565     )
    566 raise ValueError(
    567     f\"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\
\"
    568     f\"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}.\"
    569 )

File ~/.conda/envs/cuda/lib/python3.11/site-packages/transformers/modeling_utils.py:3372, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   3355         has_file_kwargs = {
   3356             \"revision\": revision,
   3357             \"proxies\": proxies,
   3358             \"token\": token,
   3359         }
   3360         cached_file_kwargs = {
   3361             \"cache_dir\": cache_dir,
   3362             \"force_download\": force_download,
   (...)
   3370             **has_file_kwargs,
   3371         }
-> 3372         if not has_file(pretrained_model_name_or_path, safe_weights_name, **has_file_kwargs):
   3373             Thread(
   3374                 target=auto_conversion,
   3375                 args=(pretrained_model_name_or_path,),
   3376                 kwargs={\"ignore_errors_during_conversion\": True, **cached_file_kwargs},
   3377                 name=\"Thread-autoconversion\",
   3378             ).start()
   3379 else:
   3380     # Otherwise, no PyTorch file was found, maybe there is a TF or Flax model file.
   3381     # We try those to give a helpful error message.

File ~/.conda/envs/cuda/lib/python3.11/site-packages/transformers/utils/hub.py:638, in has_file(path_or_repo, filename, revision, proxies, token, **deprecated_kwargs)
    635 url = hf_hub_url(path_or_repo, filename=filename, revision=revision)
    636 headers = build_hf_headers(token=token, user_agent=http_user_agent())
--> 638 r = requests.head(url, headers=headers, allow_redirects=False, proxies=proxies, timeout=10)
    639 try:
    640     hf_raise_for_status(r)

File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/api.py:100, in head(url, **kwargs)
     89 r\"\"\"Sends a HEAD request.
     90 
     91 :param url: URL for the new :class:`Request` object.
   (...)
     96 :rtype: requests.Response
     97 \"\"\"
     99 kwargs.setdefault(\"allow_redirects\", False)
--> 100 return request(\"head\", url, **kwargs)

File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    584 send_kwargs = {
    585     \"timeout\": timeout,
    586     \"allow_redirects\": allow_redirects,
    587 }
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    700 start = preferred_clock()
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)
    706 elapsed = preferred_clock() - start

File ~/.conda/envs/cuda/lib/python3.11/site-packages/requests/adapters.py:700, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    696     if isinstance(e.reason, _SSLError):
    697         # This branch is for urllib3 v1.22 and later.
    698         raise SSLError(e, request=request)
--> 700     raise ConnectionError(e, request=request)
    702 except ClosedPoolError as e:
    703     raise ConnectionError(e, request=request)

ConnectionError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/wavlm-large/resolve/main/model.safetensors (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9ebc1447d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"
}
```

Other models such as `distil-whisper/distil-large-v3` loads fine offline.
```python
import os
os.environ['HF_DATASETS_OFFLINE'] = '1'
os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
from transformers import AutoModel
AutoModel.from_pretrained('distil-whisper/distil-large-v3')
```

### Expected behavior

I expect `microsoft/wavlm-large` to load offline.

Comments:
Hi @Robinysh, thanks for opening an issue! 

If the model is defined on the hub i.e. its modeling code is on the hub, then you'll first need to download the code and checkpoint at least once i.e. run 

```py
from transformers import AutoModel
AutoModel.from_pretrained('microsoft/wavlm-large', trust_remote_code=True)
```

in online mode. From there, you should be able to run in offline mode. From the error message, it looks like it's still trying to download the model in offline mode, which is unexpected. Do you experience this with any other checkpoint? There's a known issue with offline mode and models with adapter weights, due to how downloads are handled in the PEFT library. 

Running the repro script on the development branch, I'm unable to replicate this issue. Could you try upgrading your transformers version and see if this resolves the issue? 

After upgrading `transformers` from v4.40.0 to v4.43.1 I am not encountering the bug anymore. Thank you for the suggestion.