GLM/ChatGLM badly broken in HF
Author: Qubitium
Pull request: no
Labels: name: Usage

name: bug

name: Remote code

### System Info

Ubuntu 24.04
Transformers 4.46.2

### Who can help?

@ArthurZucker @Cyrilvallez 

[[Cyrilvallez](https://github.com/Cyrilvallez)](https://github.com/huggingface/transformers/pull/33823)

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The official example in `glm.md` https://github.com/huggingface/transformers/pull/33823/files#diff-ad450d58db47b10054ff156f6bbb980437fd0d2769f60be2aa417772f9e10b6d doesn't run. 

There are multiple problems with the code.

* `trust_remote_code` is force checked for this model if using official model from THUD causing the PR to never even take effect. https://github.com/huggingface/transformers/blob/768f3c016eec88a00f2a991c7017a8a5423c4b06/src/transformers/models/auto/configuration_auto.py#L1018
* `has_local_code` is false since model type in `config.json` can be both `glm` and `chatglm` but `chatglm` is never checked. Again, this is all from ofificlal THUD models repos on HF.  https://github.com/huggingface/transformers/blob/768f3c016eec88a00f2a991c7017a8a5423c4b06/src/transformers/models/auto/configuration_auto.py#L1019
* Even if we force/fix the above 2, the model crashes with following on safetensor load:

```python
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4225, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4706, in _load_pretrained_model
    state_dict = load_state_dict(
                 ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 555, in load_state_dict
    with safe_open(checkpoint_file, framework="pt") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
```

Example code, same as `glm.md`:
```
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("THUDM/glm-4-9b-chat", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4-9b-chat")

prompt = "Give me a short introduction to large language model."

messages = [{"role": "user", "content": prompt}]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)

generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]
```

The fix is complex because THUD never updated their model code to be fully compatible with latest transformers and the merged GLM model loader will only work with a custom `glm` model or modified official repo, either modify config files and/or module tensors. 

If official HF repo from model makers doesn't work with this PR, then someone need to contact THUD to make sure their fix their model config files to be compatible or HF needs to be monkey patch the configs which is not very desirable. 

My suggestion is to revert the PR and rely on more reliable trust_remote_code=True for THUD glm models. 

What do you guys think?

### Expected behavior

Load an unmodified HF THUD glm/chatglm model without error. 

Comments:
Btw, THUD glm/chatglm modeling (trust_remote_true) was never updated to be compatbile to be full/100% compatbile with HF transformers. I sent an PR to them long time ago but never got attention. If THUD doesn't care about their models, it should not be added to HF Transformers, especially in the broken state, since they host the official models and everyone should rightfully download official models especially for those that require remote_code=True.

We fixed this for GPTQModel via monkey patch: https://github.com/ModelCloud/GPTQModel/pull/555
PR for ChatGLM since June 7th: https://huggingface.co/THUDM/glm-4-9b-chat/discussions/28

Seriously, if THUD doesn't care about their model compat, should HF auto fix it for them?

Hey! I think the way to make this work for you is to us the correct revision: `refs/pr/81`:
```diff
- model = AutoModelForCausalLM.from_pretrained("THUDM/glm-4-9b-chat", device_map="auto")
- tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4-9b-chat")
+ model = AutoModelForCausalLM.from_pretrained("THUDM/glm-4-9b-chat", device_map="auto", revision="refs/pr/81")
+ tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4-9b-chat", revision="refs/pr/81")
```
<img width="985" alt="image" src="https://github.com/user-attachments/assets/e9c9c653-cd0d-4a00-a3b5-c2f161357e4b">

Produced:
```python
In [5]: tokenizer.batch_decode(generated_ids)
Out[5]: ['\nA Large Language Model (LLM) is an artificial intelligence model designed to understand and generate human-like text. These models are trained on vast amounts of text data, which enables them to comprehend context, nuances, and generate coherent and relevant text.
']
```

@ArthurZucker  Thanks for the clarification. Now it's clear:

* The GLM pr was merged assuming THUDM will merge the hf/repo PR but they decided not to. Opting for new -hf repos. 
*  `chatglm` is the old model. `glm` is the new hf-ied model.
* [glm.md](https://github.com/huggingface/transformers/blob/a3d69a8994d673899608a7c17fbf4f953f50474e/docs/source/en/model_doc/glm.md?plain=1#L59) Example in transformers merged PR is pointing to wrong model due assumption THUDM will merge the repo PR. 
* THUDM developers never added `prominent ` extra model notes in their old models to guide users to new hf models in their old GLM-4-chat non-hf-ied hf repos as they promised in their own messages. https://huggingface.co/THUDM/glm-4-9b-chat/discussions/81#6718efcec4ec8112e7f57b79

> We [THUDM] ultimately decided to use a new repository, which is named glm-4-9b-hf. This old repository will cease maintenance and will have a prominent notice suggesting the switch to the new repository.

I guess hf should hold THUDM to their promises, before merging the model. Normal users are going to be confused beyond believe. 






We have observed that many open-source frameworks, such as [xinference](https://github.com/xorbitsai/inference) and [vLLM](https://github.com/vllm-project/vllm), currently do not support the `Glmmodel` syntax. We are submitting PRs to these frameworks one by one, but this process is very slow. If we switch to a new implementation midway, developers will not be able to use the GLM models in these application frameworks. Please give us a transition period. All subsequent GLM-4-Chat structured models will follow the `Glmmodel` instead of `ChatGLMModel`

Hi @zRzRzRzRzRzRzR, I also found the PR from @Cyrilvallez for GLM-1M does not work at the first place.

I have reported its issue on long-context tests, in this [discussion](https://huggingface.co/THUDM/glm-4-9b-chat-1m/discussions/17). Could you @Cyrilvallez please help review the this bug?


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.