Getting AttributeError: 'Mxfp4GptOssExperts' object has no attribute 'down_proj_scales'. When trying to load the GPT oss 20b model
Author: eltonjohnfanboy
Pull request: no
Labels: name: bug

### System Info

Hi guys! 
Recently I was trying to tune the GPT oss 20b model for some task I was interested in.
I tried to follow a strategy similar to the one indicated by the `gpt-oss-recipes` repositories for sft: https://github.com/huggingface/gpt-oss-recipes/blob/main/sft.py
But instead of full fine tuning I wanted to train a LoRA adapter on top of it.
Currently I've access to some A100s with 80 gb of VRAM so I though this should be doable easily.
But whenever I try to load the model for the training I get the following error:
```
[rank4]: AttributeError: 'Mxfp4GptOssExperts' object has no attribute 'down_proj_scales'. Did you mean: 'down_proj_bias'?
```
pd: I'm using the latest version of Transformers -> 4.56.0.dev0
Currently this is how I'm loading the model:
```python
model_config_parameters = {
        "pretrained_model_name_or_path": ckpt,
        "trust_remote_code": True,
        "revision": FLAGS.model_revision,
        "torch_dtype": _ACCELERATE_MIXED_PRECISION[accelerator.state.mixed_precision],
        "use_cache": not FLAGS.gradient_checkpointing,
        "token": os.getenv("HF_TOKEN"),
        "quantization_config": quantization_config
    }
arch_add_args = {"attn_implementation": "eager"}
model = AutoModelForCausalLM.from_pretrained(**(model_config_parameters | arch_add_args)).to(device)
```

Is this some bug or I'm getting wrong some part of the model loading process? Thanks!!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model_config_parameters = {
        "pretrained_model_name_or_path": ckpt,
        "trust_remote_code": True,
        "revision": FLAGS.model_revision,
        "torch_dtype": _ACCELERATE_MIXED_PRECISION[accelerator.state.mixed_precision],
        "use_cache": not FLAGS.gradient_checkpointing,
        "token": os.getenv("HF_TOKEN"),
        "quantization_config": quantization_config
    }
arch_add_args = {"attn_implementation": "eager"}
model = AutoModelForCausalLM.from_pretrained(**(model_config_parameters | arch_add_args)).to(device)

### Expected behavior

The should be loaded correctly and thus let the training continue.

Comments:
Hi, I am experiencing this issue too. Is there any progress on this?

cc @arthurzucker for text

+1

Same issue occurring for me on 4 H100s.

+1, exactly same issue on 4 H100s too

+1, got a similar one for the 120B variant with 4 A100s: `ValueError: GptOssExperts() does not have a parameter or a buffer named down_proj_blocks.`

Hey all! I'm unable to reproduce either on latest version (4.56.2) or on `main`, for both `device_map` and `tp_plan` options! Make sure to update on latest version and/or provide more infos on how to reproduce (the actual arguments passed to `from_pretrained` were not provided!)
I tested both with the native `quantization_config`, and with `quantization_config = Mxfp4Config(dequantize=True)`, both are fine

Hi Cyril, personally it only fails with `offload_state_dict=True` on MXFP4 MoE when using the BF16 (no-Triton) path

- `transformers==4.56.1`, Python 3.11, A100 (4×40 GiB)
- `triton`<3.4.0 and no `kernels` installed → MXFP4 weights are dequantized to BF16 and `GptOssExperts` is used.

I can't say i am 100% sure about the rest of the input arguments i used, i am pretty sure i was only getting this error with `offload_state_dict=True`.and I got the error after “Loading checkpoint shards … 100%”

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-120b",
    torch_dtype="bfloat16",
    low_cpu_mem_usage=True,         # also tried False
    offload_state_dict=True,        # <-- only when this is True it fails
    offload_folder="/mnt/ephemeral/offload_sd",
)
```

My hypothesis was: the offload index is built before MXFP4 dequantization/attr replacement, so at restore time it tries to set *_blocks tensors on `GptOssExperts`, but those attrs were removed (replaced by gate_up_proj/down_proj). Building the offload index after dequantization or translating keys should fix it.

Hey @kuantuna! I see, thanks for the added explanation! Note that `offload_state_dict` was a very old and unneeded argument (we never actually use more memory than needed during loading, we worked on a major refactor of the loading logic a few months ago), and I completely removed it on main in https://github.com/huggingface/transformers/pull/41103 recently! Also, `low_cpu_mem_usage` is ignored since a few releases, we ALWAYS load with as low memory usage as possible as explained (so no need for an argument)!
Similarly, `torch_dtype` has been renamed to a better/simpler `dtype` 🤗

Since it has probably always been failing with `offload_state_dict`, I don't think we want a new patch for this. And as explained, this is a no-issue on main as the arg was removed! 🤗

@Cyrilvallez said:

> offload_state_dict was a very old and unneeded argument

But removing the kwarg breaks compatibility with pretrained models and submodels. As an example, I want to run 

```
from diffusers import StableDiffusionXLControlNetInpaintPipeline

StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    torch_dtype=torch.float16,
)
```
from HF Hub on `transformers==4.57.0`. But I get:
```
Traceback (most recent call last):
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/inpaint.py", line 22, in <module>
    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/.venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/.venv/lib/python3.11/site-packages/diffusers/pipelines/pipeline_utils.py", line 961, in from_pretrained
    loaded_sub_model = load_sub_model(
                       ^^^^^^^^^^^^^^^
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/.venv/lib/python3.11/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 777, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/.venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/oliland/Lab/DiffusionLight-Turbo-Replicate/.venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4974, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CLIPTextModelWithProjection.__init__() got an unexpected keyword argument 'offload_state_dict'
```
So it would seem some checkpoints are stuck on `transformers==4.56.2` because the API has changed 🤷 

Oh wowww, you are totally right @oliland. If you look at the PR [here](https://github.com/huggingface/transformers/pull/41103/files#diff-6b72b98c4c2dcfc6cc606843917733f5d858374fbc22a735ff483bbc0c1e63eaL4634) which removed the usage of `offload_state_dict`, you will notice that handling of the kwarg `offload_state_dict` has not changed at all in `from_pretrained`, in order to avoid such issues. It is simply ignored downstream. However, for some reason on the v4.57 branch, it is not being removed from the `kwargs` as in the PR. I am very unsure of what happened here, probably a human error during cherry-picking the commits and fixing conflicts. We will add it again and patch!

cc @ArthurZucker for viz as well, if we patch soon!

Patch is out right? 

Yes, it was released and v4.57 was yanked anyway!