[Tokenizer][OFFLINE] chat_template.jinja not downloaded in cache
Author: khatrimann
Pull request: no
Labels: name: bug

### System Info

**Issue Title**
`AutoTokenizer` ignores `HF_HUB_OFFLINE=1` and attempts to download `chat_template.jinja` for Llama/LLM models

**Description**
I am encountering an issue where `AutoTokenizer.from_pretrained()` attempts to connect to the Hugging Face Hub to download `chat_template.jinja`, even when the environment variable `HF_HUB_OFFLINE=1` is explicitly set.

While this workflow works correctly for older architectures (e.g., `bert-base-uncased`), it fails for Llama 3.1 and 3.2 models. I have also attempted to manually place the `chat_template.jinja` file in the cache revision directory, but the library still attempts an online connection.

**Reproduction***
*Script (`repro_script.py`):**

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# MODEL_NAME = "bert-base-uncased"    # MODEL IS LOADED AND TOKENIZER TOO
MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct"    # MODEL IS LOADED BUT NOT TOKENIZER
MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct"    # MODEL IS LOADED BUT NOT TOKENIZER
token = "hf_*"

# model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)    # WORKING
# tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)    # FAILS WITH LLAMA
# tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, local_files_only=True)    # FAILS WITH LLAMA
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, token=token)    # FAILS WITH LLAMA
# tokenizer.save_pretrained("./dummy_tokenizer")    # SAVE TO COPY FILES MANUALLY TO CACHE

print("Model and tokenizer loaded successfully.", tokenizer)
```

**Command:**

```bash
HF_HUB_OFFLINE=1 python repro_script.py

```

**Expected Behavior**
The tokenizer should load entirely from the local cache without attempting any network connections, respecting the `HF_HUB_OFFLINE=1` flag.

**Actual Behavior**
The script fails with a connection error/timeout (depending on network status) indicating it is trying to retrieve `chat_template.jinja` from the Hub.

**Error Log**
```text
huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.
```

**Environment Information*** `transformers` version: (Run `transformers-cli env` and paste results here)
* `python`: 3.10.19 
* `transformers`: 4.57.3
* Platform: Linux

**EDIT**: I tried running this in `transformers==4.55.2`, it worked but their was a new problem -- Unable to apply chat_template

```text
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Paste the given script and run the given command

### Expected behavior

No Error

Comments:
**Solved**
Steps:
1. Cleared cache (`rm ~/.cache`)
2. Ran the script and used `save_pretrained()` to save the current tokenizer. It saves `chat_template.jinja`. Then ran the same script with with `HF_HUB_OFFLINE=1 script.py`.
3. Copy the `chat_template.jinja` along with rest of the files to the cache snapshot. For me it looked like `cp dummy_tokenizer/chat_template.jinja ~/.cache/huggingface/hub/models--meta-llama--Llama-3.2-1B-Instruct/snapshots/9213176726f574b556790deb65791e0c5aa438b6/` (You can skip this step if till step 2 works for you)

**NOTE: I downgraded `transformers` to `4.55.2` from `4.57.3` and didn't upgraded back, so I don't know if it's working in newer versions**

I believe `HF_HUB_OFFLINE` should be respected in newer versions! Please check the latest version from `main` - if you or anyone else are seeing file download attempts when `HF_HUB_OFFLINE` is set, please let us know. I think the other issues you saw were just your cache being out of date and missing files, which couldn't be updated because you were in offline mode.

> I believe `HF_HUB_OFFLINE` should be respected in newer versions! Please check the latest version from `main` - if you or anyone else are seeing file download attempts when `HF_HUB_OFFLINE` is set, please let us know. I think the other issues you saw were just your cache being out of date and missing files, which couldn't be updated because you were in offline mode.

Hi,
Apologies for late reply but I tried this setup on a new server with transformers version 4.57.3 and it didn't worked but was working with 4.55.2

Can you try `v5` (install with the pre-release flag) or `main`?

> Can you try `v5` (install with the pre-release flag) or `main`?

Yes, it's working with `transformers==5.0.0rc1`

Okay! In that case, I think whatever bug caused this is fixed. We can't really go back and backport bugfixes to older versions, so I think there's not much we can do. If this is causing problems for you, just update!