`tie_word_embeddings` not saved on customized model
Author: Tavish9
Pull request: no
Labels: name: bug

### System Info

transformers_version: 4.52.0.dev0
python: 3.10


### Who can help?

@ArthurZucker  @SunMarc @qubvel @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoConfig, AutoModelForCausalLM, PretrainedConfig, PreTrainedModel, Trainer, TrainingArguments


class MyConfig(PretrainedConfig):
    model_type = "test"
    sub_configs = {"text_config": AutoConfig}

    def __init__(
        self,
        text_config=None,
        **kwargs,
    ):
        self.text_config = text_config

        super().__init__(**kwargs)

class MyModel(PreTrainedModel):
    config_class = MyConfig

    def __init__(self, config, language_model=None) -> None:
        super().__init__(config)
        self.language_model = language_model

        self.post_init()

text_config = AutoConfig.from_pretrained("Qwen/Qwen3-8B")
config = MyConfig(text_config=text_config)
language_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = MyModel(config, language_model)

trainer = Trainer(
    model=model,
    args=TrainingArguments(),
)

trainer.save_model()

```

### Expected behavior

The saved `config.json` has no `tie_word_embeddings` key

Expected:
```json
{
  "architectures": [
    "MyModel"
  ],
  "model_type": "test",
  "text_config": {
    "_name_or_path": "Qwen/Qwen3-8B",
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 1024,
    "initializer_range": 0.02,
    "intermediate_size": 3072,
    "max_position_embeddings": 40960,
    "max_window_layers": 28,
    "model_type": "qwen3",
    "num_attention_heads": 16,
    "num_hidden_layers": 28,
    "num_key_value_heads": 8,
    "rms_norm_eps": 1e-06,
    "rope_scaling": null,
    "rope_theta": 1000000,
    "sliding_window": null,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936
  },
  "torch_dtype": "float32",
  "transformers_version": "4.52.0.dev0"
}
```

But:
```diff
{
  "architectures": [
    "MyModel"
  ],
  "model_type": "test",
  "text_config": {
    "_name_or_path": "Qwen/Qwen3-8B",
    "architectures": [
      "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "initializer_range": 0.02,
    "intermediate_size": 12288,
    "max_position_embeddings": 40960,
    "max_window_layers": 36,
    "model_type": "qwen3",
    "num_attention_heads": 32,
    "num_hidden_layers": 36,
    "num_key_value_heads": 8,
    "rms_norm_eps": 1e-06,
    "rope_scaling": null,
    "rope_theta": 1000000,
    "sliding_window": null,
-   "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936
  },
  "torch_dtype": "float32",
  "transformers_version": "4.52.0.dev0"
}
```

Comments:
More info:

If we use `Qwen/Qwen3-0.6B` or `Qwen/Qwen3-1.7B` (that has `tie_word_embeddings:True` in config.json), the saved config has `tie_word_embeddings`, which is the same as original config.

Hi @Tavish9, I don't think this is a bug! The base `PretrainedConfig` has a number of default config keys, including `tie_word_embeddings`, and when you inherit from it you'll need to set these in your `__init__` if you don't want them to be set to the default value. Take a look at `transformers/configuration_utils.py`. 

Hi, @Rocketknight1 thanks for replying.

However, you misunderstand my post. I don't change the original text config at all, but `tie_word_embeddings` is missing in the config.json file for `text_config`. 

```python
text_config = AutoConfig.from_pretrained("Qwen/Qwen3-8B")
config = MyConfig(text_config=text_config)
model = MyModel(config, language_model)
trainer.save_model()
```

Actually, we don't need to focus `config.tie_word_embeddings`, but `config.text_config.tie_word_embeddings`.

This happens for config that has `tie_word_embeddings:True` on the original config.

For MyConfig, it's `tie_word_embeddings` is True, which is the default value, I know this.



@Tavish9 hey, in composite configs we first compare if the defaults are different from the model's config, which will be [Qwen3Config](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3/configuration_qwen3.py) class. You can see that the config has a default value `False` and thus if `tie_word_embeddings=False` it will not be saved

As long as the loaded config has `onfig.text_config.tie_word_embeddings = False` it is expected. We had similar issues earlier and IIRC now it should not have weird edge cases

@zucchini-nlp , thanks for replying.

1. Could you please show me the saving logic?
> in composite configs we first compare if the defaults are different from the model's config

2. Besides, do you mean that even the `config.json` does not save `text_config.tie_word_embeddings`, the loaded config should have `config.text_config.tie_word_embeddings=False`? 

3. However, there might be a potential issue with `config.tie_word_embeddings`, as `AutoConfig` sets `tie_word_embeddings` to true if `tie_word_embeddings` is missing in `config.tie_word_embeddings` within `config.json`.
So, we might get loaded config: `config.tie_word_embeddings=True` while `config.text_config.tie_word_embeddings=False`.

Very Detailed Questions, thanks in advance.




@Tavish9 The snippet above for saving is correct. Since you are using a custom `MyConfig` the issue comes from the config definition itself. We usually do not simply assign the text config as is, but rather at load time initiate an LM config [as follows](https://github.com/huggingface/transformers/blob/04282a9ef5541e98a8e26eb33203ea58646c0f66/src/transformers/models/llava/configuration_llava.py#L127C9-L131). Thus all missing will default to `model_type`'s defaults. In this case it is Qwen3. The saving and loading back a composite config, will give `config.text_config.tie_word_embeddings=False`

https://github.com/huggingface/transformers/blob/04282a9ef5541e98a8e26eb33203ea58646c0f66/src/transformers/models/llava/configuration_llava.py#L127-L131


okay, that's aligned with my experiment.

what's about the Q3? There is a misalignment between `config.tie_word_embeddings` and `config.text_config.tie_word_embeddings` if `tie_word_embeddings` is missing and the `model_type`'s default is False.

I think it's a bug 🤔

We should not have `config.tie_word_embeddings` and all text model related keys should be available only in `config.text_config`. For some older models we have missed that and probably configs on repo contain duplicated `tie_word_embeddings`, for new models we don't allow it anymore


https://github.com/huggingface/transformers/blob/04282a9ef5541e98a8e26eb33203ea58646c0f66/src/transformers/configuration_utils.py#L223-L225

So, `config.tie_word_embeddings` is valid and would cause an issue in this case.

For composite configs, we don't use `config.tie_word_embeddings` but rather `config.get_text_config().tie_word_embeddings`, so yes. If config has it defined in the upper level, then we'll ignore it and use whatever is found in text config


Okay, things solved.

BTW, composite config is a new feature of transformers update. Some old models might not have composite configs so they still use `config.tie_word_embeddings`. 

If the model config is flat, i.e. no `text_config` exists then `config.get_text_config()` returns `self`. We didn't change anything for usual configs and only added a standardized way to handle composite ones :)

Feel free to close the issue, if it's resolved