Repository: huggingface/peft

Issue:
peft/utils/save_and_load.py try to connect to the hub even when HF_HUB_OFFLINE=1
### System Info

peft 0.8.2
axolotl v0.4.0

export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_HUB_OFFLINE=1

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder
- [X] My own task or dataset (give details below)

### Reproduction

When I am FT my models on a GPU offline using lora and peft with axolotl library, I get an error from peft/utils/save_and_load.py", line 146, in get_peft_model_state_dict
    has_remote_config = file_exists(model_id, "config.json")
The function try to connect to the HUB before saving the checkpoint locally, which make crash the run and lose the FT model.

Here is the full error message:

```
Traceback (most recent call last):
  File "/gpfslocalsup/pub/anaconda-py3/2021.05/envs/python-3.9.12/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/gpfslocalsup/pub/anaconda-py3/2021.05/envs/python-3.9.12/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/gpfsdswork/projects/rech/vqt/ule33dt/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    fire.Fire(do_cli)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/gpfsdswork/projects/rech/vqt/ule33dt/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
    return do_train(parsed_cfg, parsed_cli_args)
  File "/gpfsdswork/projects/rech/vqt/ule33dt/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/gpfsdswork/projects/rech/vqt/ule33dt/axolotl/src/axolotl/train.py", line 163, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1561, in train
    return inner_training_loop(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 1968, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2340, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2416, in _save_checkpoint
    self.save_model(staging_output_dir, _internal_call=True)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2927, in save_model
    self._save(output_dir)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2999, in _save
    self.model.save_pretrained(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/peft/peft_model.py", line 216, in save_pretrained
    output_state_dict = get_peft_model_state_dict(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/peft/utils/save_and_load.py", line 146, in get_peft_model_state_dict
    has_remote_config = file_exists(model_id, "config.json")
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 2386, in file_exists
    get_hf_file_metadata(url, token=token)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1631, in get_hf_file_metadata
    r = _request_wrapper(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 385, in _request_wrapper
    response = _request_wrapper(
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 408, in _request_wrapper
    response = get_session().request(method=method, url=url, **params)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/linkhome/rech/genrqo01/ule33dt/.local/lib/python3.9/site-packages/huggingface_hub/utils/_http.py", line 78, in send
    raise OfflineModeIsEnabled(
huggingface_hub.utils._http.OfflineModeIsEnabled: Cannot reach https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/resolve/main/config.json: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.
```

I manage to fix the bug by commenting those lines in save_and_load.py:
```
if model_id is not None:
            try:
                has_remote_config = file_exists(model_id, "config.json")
            except (HFValidationError, EntryNotFoundError):
                warnings.warn(
                    f"Could not find a config file in {model_id} - will assume that the vocabulary was not modified."
                )
                has_remote_config = False
```

### Expected behavior

It seems that `file_exists(model_id, "config.json")` should not call the hub when the variable 'export `HF_HUB_OFFLINE=1` is set


Discussion:
Even excluding `HF_HUB_OFFLINE=1`, I'm surprised that peft is calling out to the hub for a `config.json` in the middle of training runs—this is crashing my training jobs with a coinflip at each eval step currently. Shouldn't all the model config info be available locally in the middle of the training loop?

Resolving patch:
diff --git a/src/peft/auto.py b/src/peft/auto.py
--- a/src/peft/auto.py
+++ b/src/peft/auto.py
@@ -18,8 +18,6 @@
 import os
 from typing import Optional
 
-from huggingface_hub import file_exists
-from huggingface_hub.utils import HfHubHTTPError, HFValidationError
 from transformers import (
     AutoModel,
     AutoModelForCausalLM,
@@ -42,6 +40,7 @@
     PeftModelForTokenClassification,
 )
 from .utils.constants import TOKENIZER_CONFIG_NAME
+from .utils.other import check_file_exists_on_hf_hub
 
 
 class _BaseAutoPeftModel:
@@ -112,16 +111,13 @@ def from_pretrained(
             if token is None:
                 token = kwargs.get("use_auth_token", None)
 
-            try:
-                tokenizer_exists = file_exists(
-                    repo_id=pretrained_model_name_or_path,
-                    filename=TOKENIZER_CONFIG_NAME,
-                    revision=kwargs.get("revision", None),
-                    repo_type=kwargs.get("repo_type", None),
-                    token=token,
-                )
-            except (HfHubHTTPError, HFValidationError):  # not on the Hub, so probably local repo
-                pass
+            tokenizer_exists = check_file_exists_on_hf_hub(
+                repo_id=pretrained_model_name_or_path,
+                filename=TOKENIZER_CONFIG_NAME,
+                revision=kwargs.get("revision", None),
+                repo_type=kwargs.get("repo_type", None),
+                token=token,
+            )
 
         if tokenizer_exists:
             tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)
diff --git a/src/peft/utils/other.py b/src/peft/utils/other.py
--- a/src/peft/utils/other.py
+++ b/src/peft/utils/other.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 import copy
 import inspect
+import os
 import warnings
 from contextlib import nullcontext
 from typing import Optional, Tuple
@@ -21,6 +22,8 @@
 import torch
 from accelerate.hooks import add_hook_to_module, remove_hook_from_module
 from accelerate.utils import is_npu_available, is_xpu_available
+from huggingface_hub import file_exists
+from huggingface_hub.utils import EntryNotFoundError, HFValidationError
 from safetensors.torch import storage_ptr, storage_size
 
 from ..import_utils import is_auto_gptq_available, is_torch_tpu_available
@@ -537,3 +540,39 @@ def cast_mixed_precision_params(model, dtype):
             p.data = p.to(dtype)
         else:
             p.data = p.to(torch.float32)
+
+
+def str_to_bool(value: str) -> int:
+    """
+    Converts a string representation of truth to `True` (1) or `False` (0).
+
+    True values are `y`, `yes`, `t`, `true`, `on`, and `1`; False value are `n`, `no`, `f`, `false`, `off`, and `0`;
+    """
+    # same as function as in accelerate.utils, which replaces the deprecated distutils.util.strtobool
+    value = value.lower()
+    if value in ("y", "yes", "t", "true", "on", "1"):
+        return 1
+    elif value in ("n", "no", "f", "false", "off", "0"):
+        return 0
+    else:
+        raise ValueError(f"invalid truth value {value}")
+
+
+def check_file_exists_on_hf_hub(repo_id: str, filename: str, **kwargs) -> Optional[bool]:
+    """Check if a file exists on HF Hub, if check was not successful returns None instead of erroring.
+
+    Respect offline mode if set.
+
+    """
+    exists: Optional[bool] = None
+    if str_to_bool(os.environ.get("HF_HUB_OFFLINE", "0")):
+        # user set offline mode, cannot check
+        return exists
+
+    try:
+        exists = file_exists(repo_id, filename, **kwargs)
+    except (HFValidationError, EntryNotFoundError):
+        # error, exists stays None
+        pass
+
+    return exists
diff --git a/src/peft/utils/save_and_load.py b/src/peft/utils/save_and_load.py
--- a/src/peft/utils/save_and_load.py
+++ b/src/peft/utils/save_and_load.py
@@ -17,10 +17,16 @@
 
 import torch
 from huggingface_hub import file_exists, hf_hub_download
-from huggingface_hub.utils import EntryNotFoundError, HFValidationError
+from huggingface_hub.utils import EntryNotFoundError
 from safetensors.torch import load_file as safe_load_file
 
-from .other import EMBEDDING_LAYER_NAMES, SAFETENSORS_WEIGHTS_NAME, WEIGHTS_NAME, infer_device
+from .other import (
+    EMBEDDING_LAYER_NAMES,
+    SAFETENSORS_WEIGHTS_NAME,
+    WEIGHTS_NAME,
+    check_file_exists_on_hf_hub,
+    infer_device,
+)
 from .peft_types import PeftType
 
 
@@ -140,14 +146,17 @@ def get_peft_model_state_dict(
         # we need to make sure we can download that config.
         has_remote_config = False
 
+        # ensure that this check is not performed in HF offline mode, see #1452
         if model_id is not None:
-            try:
-                has_remote_config = file_exists(model_id, "config.json")
-            except (HFValidationError, EntryNotFoundError):
+            exists = check_file_exists_on_hf_hub(model_id, "config.json")
+            if exists is None:
+                # check failed, could not determine if it exists or not
                 warnings.warn(
                     f"Could not find a config file in {model_id} - will assume that the vocabulary was not modified."
                 )
                 has_remote_config = False
+            else:
+                has_remote_config = exists
 
         # check if the vocab size of the base model is different from the vocab size of the finetuned model
         if (


Test patch: