AttributeError: 'dict' object has no attribute 'to_dict'; for Inferencing Lora Merged Qwen/Qwen2.5-VL-3B-Instruct
Author: shahdivax
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-6.8.0-1021-aws-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config:    not found
- DeepSpeed version: 0.16.1
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: Tesla T4

### Who can help?

@amyeroberts, @qubvel  @gante and @Rocketknight1

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

So i Finetuned `Qwen/Qwen2.5-VL-3B-Instruct` using axolotl and then I merged the adapter with base model, then I tried to do inference with the default code below 

```python
BASE_MODEL_PATH = "axolotl/outputs/out_qwen_25_3b_lora_3000_512_4/merged" 
config = AutoConfig.from_pretrained(BASE_MODEL_PATH)
model = AutoModelForVision2Seq.from_pretrained(
    BASE_MODEL_PATH, 
    config=config,
    torch_dtype=torch.float16, 
    device_map="auto"
)
```
which Gave me below error
```
/home/ubuntu/vision_poc/venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 35
     32 client = DocumentAnalysisClient(endpoint=endpoint, credential=AzureKeyCredential(api_key))
     34 config = AutoConfig.from_pretrained(BASE_MODEL_PATH)
---> 35 model = AutoModelForVision2Seq.from_pretrained(
     36     BASE_MODEL_PATH, 
     37     config=config,
     38     torch_dtype=torch.float16, 
     39     device_map="auto"
     40 )
     41 # print(f"Base Model Embeddings {model.get_input_embeddings().num_embeddings}")
     42 
     43 
   (...)
     47 
     48 # Load processor
     49 processor = AutoProcessor.from_pretrained(BASE_MODEL_PATH,use_fast=True)

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py:564, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    562 elif type(config) in cls._model_mapping.keys():
    563     model_class = _get_model_class(config, cls._model_mapping)
--> 564     return model_class.from_pretrained(
    565         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    566     )
    567 raise ValueError(
    568     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    569     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    570 )

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/modeling_utils.py:262, in restore_default_torch_dtype.<locals>._wrapper(*args, **kwargs)
    260 old_dtype = torch.get_default_dtype()
    261 try:
--> 262     return func(*args, **kwargs)
    263 finally:
    264     torch.set_default_dtype(old_dtype)

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/modeling_utils.py:4185, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)
   4179     config = cls._autoset_attn_implementation(
   4180         config, use_flash_attention_2=use_flash_attention_2, torch_dtype=torch_dtype, device_map=device_map
   4181     )
   4183 with ContextManagers(init_contexts):
   4184     # Let's make sure we don't run the init function of buffer modules
-> 4185     model = cls(config, *model_args, **model_kwargs)
   4187 # make sure we use the model's config since the __init__ call might have copied it
   4188 config = model.config

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:1518, in Qwen2_5_VLForConditionalGeneration.__init__(self, config)
   1517 def __init__(self, config):
-> 1518     super().__init__(config)
   1519     self.visual = Qwen2_5_VisionTransformerPretrainedModel._from_config(config.vision_config)
   1520     self.model = Qwen2_5_VLModel(config)

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1378, in PreTrainedModel.__init__(self, config, *inputs, **kwargs)
   1376 self.name_or_path = config.name_or_path
   1377 self.warnings_issued = {}
-> 1378 self.generation_config = GenerationConfig.from_model_config(config) if self.can_generate() else None
   1379 # Overwrite the class attribute to make it an instance attribute, so models like
   1380 # `InstructBlipForConditionalGeneration` can dynamically update it without modifying the class attribute
   1381 # when a different component (e.g. language_model) is used.
   1382 self._keep_in_fp32_modules = copy.copy(self.__class__._keep_in_fp32_modules)

File ~/vision_poc/venv/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:1288, in GenerationConfig.from_model_config(cls, model_config)
   1286 if decoder_config is not model_config:
   1287     default_generation_config = GenerationConfig()
-> 1288     decoder_config_dict = decoder_config.to_dict()
   1289     for attr in generation_config.to_dict().keys():
   1290         is_unset = getattr(generation_config, attr) == getattr(default_generation_config, attr)

AttributeError: 'dict' object has no attribute 'to_dict'
```



### Expected behavior

Checkpoint should load successfully: 
```
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.66s/it]
```
---
I was able to solve it with below things : 

1) Change inference code a bit with : 
```python
BASE_MODEL_PATH = "axolotl/outputs/out_qwen_25_3b_lora_3000_512_4/merged"
# Load config
config = AutoConfig.from_pretrained(BASE_MODEL_PATH, trust_remote_code=True)

# Ensure text_config is a proper config object
if isinstance(config.text_config, dict):
    from transformers import PretrainedConfig
    config.text_config = PretrainedConfig.from_dict(config.text_config)

# Same for vision_config
if isinstance(config.vision_config, dict):
    config.vision_config = PretrainedConfig.from_dict(config.vision_config)

# Load model with modified config
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    BASE_MODEL_PATH,
    config=config,
    torch_dtype=torch.float16,
    device_map="auto",
)
```
2) Copied [preprocessor_config.json](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/preprocessor_config.json), to the merged folder and run an inference

this worked

---

Suggested changes at : https://github.com/huggingface/transformers/blob/298b3f19303294293f7af075609481d64cb13de3/src/transformers/generation/configuration_utils.py#L1285

```diff
- decoder_config_dict = decoder_config.to_dict()
+ decoder_config_dict = decoder_config.to_dict() if isinstance(decoder_config, PretrainedConfig) else decoder_config
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @zucchini-nlp @gante since this seems like a VLM / generation config issue

@shahdivax hmm this works with official checkpoints so I guess smth is in the model you merged and saved. Do you have it in hub, so we can reproduce the issue?

@zucchini-nlp I used axolotl to fine-tune and merge,

I used this branch: https://github.com/axolotl-ai-cloud/axolotl/tree/pixtral_integration2

You can recreate it using that, then I used axolotl merge to merge the model,@NanoCode012 can confirm this issue.

@shahdivax I would like the reproducing script to be minimal and preferable without external libraries. Since the issue is raised when loading the model, I can help you faster by using the same model. Seems weird that the config is returned as dict, given that Qwen2-VL doesn't have a text-config, thus it is the general config itself

If the model you are using is confidential, you can share with dummy weights :)

@zucchini-nlp , sorry for late reply. This error was due to a bug on our end with config saving. I mistakenly thought it was due to some config loading edge case. We fixed the issue last week. This issue can be closed.

@shahdivax  , I believe I pinged you on discord about the fix as well, but if I haven't, the latest release axolotl has the fix included, so feel free to give that a go.

> @zucchini-nlp , sorry for late reply. This error was due to a bug on our end with config saving. I mistakenly thought it was due to some config loading edge case. We fixed the issue last week. This issue can be closed.
> 
> @shahdivax  , I believe I pinged you on discord about the fix as well, but if I haven't, the latest release axolotl has the fix included, so feel free to give that a go.

Ahh yes, thanks for checking it out, will surely check it out.

Closing this issue 🙌🏼👍🏼