Config rope_scaling and text_config.rope_scaling might be the same or different dict objects
Author: albertvillanova
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.0.dev0
- Platform: Linux-5.15.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.12.9
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.6.2
- Accelerate version: 1.4.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

After investigation of an issue in `trl`, I found a weird behavior of `transformers` config `rope_scaling`: the `config.rope_scaling` (at the root config level) and `config.text_config.rope_scaling` (under `text_config`) might be the same or different dict objects depending on whether we pass `text_config` param to `AutoConfig.from_pretrained`
- if we don't pass `text_config` param, the 2 `rope_scaling` point to the same dict object
- if we pass `text_config` param, the 2 `rope_scaling` are different dict objects
```python
In [1]: from transformers import AutoConfig

In [2]: model_id = "Qwen/Qwen2.5-VL-3B-Instruct"

In [3]: config1 = AutoConfig.from_pretrained(model_id)
In [4]: config1.text_config.rope_scaling
Out[4]: {'type': 'default', 'mrope_section': [16, 24, 24], 'rope_type': 'default'}
In [5]: config1.rope_scaling
Out[5]: {'type': 'default', 'mrope_section': [16, 24, 24], 'rope_type': 'default'}
In [6]: id(config1.text_config.rope_scaling)
Out[6]: 140211029392000
In [7]: id(config1.rope_scaling)
Out[7]: 140211029392000
# Both are the same dict object

In [8]: config2 = AutoConfig.from_pretrained(model_id, text_config={})
In [9]:config2.text_config.rope_scaling
Out[9]: {'type': 'default', 'mrope_section': [16, 24, 24], 'rope_type': 'default'}
In [10]: config2.rope_scaling
Out[10]: {'type': 'default', 'mrope_section': [16, 24, 24], 'rope_type': 'default'}
In [11]: id(config2.text_config.rope_scaling)
Out[11]: 140210801100608
In [12]: id(config2.rope_scaling)
Out[12]: 140211029786688
# Both are different dict objects
```

Is this expected?

We discovered this while investigating why changing (after initialization) the `config.text_config.rope_scaling` will or will not change the `config.rope_scaling` as well. See related comment in `trl` PR:
- https://github.com/huggingface/trl/pull/4101#issuecomment-3305590742

### Expected behavior

- Either they should be the same dict object in any case
- Or they should be different dict objects in any case

Comments:
Ideally `config.rope_scaling` should always be `None` because we don't use it anyway. When init the lm backbone, it uses the text config and thus the rope params from text config. In case of Qwen2.5VL, we changed the config layout to follow nested structure after model release, and the configs saved in the hub are plain flat dicts. This is the reason that the rope params point to the same object. Even though we don't use `config.rope_scaling` and discourage its usage, we are keeping it for BC

We can fix this by forcefully returning the text config params whenever anything text-related is obtained/set, though I am not sure if it is really needed. Does trl use directly `config.rope_scaling` attribute anywhere? 

Ah also, for nested config overwriting the last update was in https://github.com/huggingface/transformers/pull/40241. That would be the recommended way to update subconfigs. It will update the requested text config key while keeping the other values as in `model_id`

Thanks for your fast feedback, @zucchini-nlp! 🤗 

> That would be the recommended way to update subconfigs.

Indeed, that is what I did and how I discovered the issue:
```python
config = AutoConfig.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", 
    text_config={
        "rope_scaling": {"type": "default", "mrope_section": [2], "rope_type": "default"}
    }
) 
# BTW: not possible to update only the "mrope_section" param

config.text_config.rope_scaling
{'type': 'default', 'mrope_section': [2], 'rope_type': 'default'}

config.rope_scaling
{'type': 'default', 'mrope_section': [16, 24, 24], 'rope_type': 'default'}
```

Ah I see, do we still need to force the attributes to be poiting to the same object?
> Does trl use directly config.rope_scaling attribute anywhere?

> Does trl use directly config.rope_scaling attribute anywhere?

No, not the core `trl` library itself. We just use it in our scripts to generate the tiny model configs we use for testing:
https://github.com/huggingface/trl/blob/27f22ba5a1794a53e1de2cec887a46a381e9a37a/scripts/generate_tiny_models.py#L301-L303

Oke, then i think we can close the issue and adjust our test scripts accordingly

I opened a PR in `trl` to completely align to the recommended way of updating subconfigs; note we need to pass `rope_scaling` twice (as `text_config` and as root `kwargs`) because of the underlying issue:
```python
    if issubclass(model_class.config_class, (Qwen2VLConfig, Qwen2_5_VLConfig)):
        text_config["rope_scaling"] = {"type": "default", "mrope_section": [2], "rope_type": "default"}
        # Different dict object from text_config; see GH-4101 and transformers#41020
        kwargs["rope_scaling"] = {"type": "default", "mrope_section": [2], "rope_type": "default"}

...

    config = AutoConfig.from_pretrained(model_id, text_config=text_config, vision_config=vision_config, **kwargs)
```

I think we should not update the `kwargs["rope_scaling"]` and instead make sure to not access the root config for RoPE params

As far as I know we don't access it, we just pushed the model to the Hub.

If we don't update the kwargs as well, then the pushed model config shows different `mrope_section` values in text_config and at the root level, which could be considered as a malformed or ambiguous configuration?

See for example: https://huggingface.co/trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration/discussions/1/files

We had a more severe problem, related to this issue, this time for the parameters `num_hidden_layers`, `hidden_size` and `num_attention_heads`: a torch.AcceleratorError: CUDA error: device-side assert triggered.

We created our tiny model (`trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration`):
```python
config = AutoConfig.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", 
    text_config={
        "num_hidden_layers": 2,
        "hidden_size": 16,
        "num_attention_heads": 4,
        "layer_types": None,  # Set it automatically from num_hidden_layers to avoid ValueError
        "rope_scaling": {"type": "default", "mrope_section": [2], "rope_type": "default"},
    },
    vision_config={
        "num_hidden_layers": 2,
        "hidden_size": 16,
        "num_attention_heads": 4,
        "out_hidden_size": 16,
    },
    # Due to this issue GH-41020
    **{"rope_scaling": {"type": "default", "mrope_section": [2], "rope_type": "default"}}
)
model = Qwen2_5_VLForConditionalGeneration(config).to(dtype=torch.bfloat16)
model.push_to_hub(repo_id)
```

Then when trying to use it, we got an error: `torch.AcceleratorError: CUDA error: device-side assert triggered`
```python
>       trainer = RLOOTrainer(
            model="trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration",
            reward_funcs=reward_func,
            args=training_args,
            train_dataset=dataset,
        )

tests/test_rloo_trainer.py:1324: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
trl/trainer/rloo_trainer.py:499: in __init__
    super().__init__(
.venv/lib/python3.11/site-packages/transformers/utils/deprecation.py:172: in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.11/site-packages/transformers/trainer.py:460: in __init__
    enable_full_determinism(self.args.seed) if self.args.full_determinism else set_seed(self.args.seed)
                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.11/site-packages/transformers/trainer_utils.py:106: in set_seed
    torch.manual_seed(seed)
.venv/lib/python3.11/site-packages/torch/_compile.py:53: in inner
    return disable_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:929: in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.11/site-packages/torch/random.py:46: in manual_seed
    torch.cuda.manual_seed_all(seed)
.venv/lib/python3.11/site-packages/torch/cuda/random.py:131: in manual_seed_all
    _lazy_call(cb, seed_all=True)
.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:341: in _lazy_call
    callable()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def cb():
        for i in range(device_count()):
            default_generator = torch.cuda.default_generators[i]
>           default_generator.manual_seed(seed)
E           torch.AcceleratorError: CUDA error: device-side assert triggered
E           CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
E           For debugging consider passing CUDA_LAUNCH_BLOCKING=1
E           Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

.venv/lib/python3.11/site-packages/torch/cuda/random.py:129: AcceleratorError
```

After investigation, we discovered that the `config` only updates the passed params under `text_config` (and `vision_config`), but not at the root level. The problem is that the root values seem to be used afterwards and then the error raised:
```python
In [10]: print(config.text_config.num_hidden_layers, config.text_config.hidden_size, config.text_config.num_attention_heads)
2 16 4

In [11]: print(config.num_hidden_layers, config.hidden_size, config.num_attention_heads)
36 2048 16
```

CC: @qgallouedec 

Interesting, oke lemme see if we can force the root params to be get/set via text-config and vice-versa