Consider reverting CLIP refactor
Author: dxqb
Pull request: no
Labels: name: bug

### System Info

Please consider reverting https://github.com/huggingface/transformers/pull/44431

Here is detailed information what problems it causes:
https://github.com/Nerogar/OneTrainer/pull/1506

Besides the one linked, it prevents multiple projects from upgrading or requires workarounds/conversion to match existing LoRA file formats:
| Project | transformers pin | Relation to the CLIP flattening (#44431) |
  |---|---|---|
  | **ai-toolkit** | `==5.5.3` | Pinned one release below the boundary — avoids the flattening entirely. Architecturally closest to OneTrainer (manual loading, kohya-format LoRAs). |
  | **SimpleTuner** | base deps `>=5.6.0` | Past the boundary, so conversion is necessary — it relies on `from_pretrained` to convert old↔flat keys and on explicit `convert_sd*.py` tables for single-file export |
  | **sd-scripts**  | `==4.54.1` | Still on v4 — not affected yet. |
  | **musubi-tuner**  | `==4.57.6` | Still on v4 (same 4.57.6 OneTrainer is on now) — not affected yet. |


Feel free to close this issue if you had very good reasons to do this, but from the outside looking in, it looks like house-keeping that isn't worth it.

### Who can help?

text models: @ArthurZucker @Cyrilvallez


Comments:
Hey! thanks a lot for the ping. 
I don't think its something we can do now as new checkpoints were released, but also as it does not really make sense. 
It was house keeping + bug fixes + v5 upgrade of the code. 

What we can do however is help the libs make sure its easy to accomodate for. With v5 and the dynamic weight loader + the PRs to PEFT etc it should be alright. 

Is there something we can do to make the dynamic weight loader and the conversion mapping easier to use?



Thanks for the detailed explanation.

I understand the CLIP refactor was part of housekeeping + v5 upgrade, so reverting is not ideal.

However, I think the main pain point for downstream libraries (like OneTrainer / SimpleTuner) is the lack of a clear migration guide for:
- dynamic weight loader usage with flattened CLIP keys
- mapping between old LoRA formats and new flattened structure

Would it be helpful to add a small “compatibility guide” or example snippet showing:
- how to load legacy LoRA checkpoints after the refactor
- recommended conversion mapping workflow for external tools

This could reduce fragmentation without requiring a revert.

Happy to help test or document this if needed.


> 
> This could reduce fragmentation without requiring a revert.
> 
> Happy to help test or document this if needed.

Thanks!
Here are the changes we had to do to upgrade to transformers 5.5, but I'm sure many of them were self-inflicted because we relied on transformers 4 internals: https://github.com/Nerogar/OneTrainer/pull/1524/changes

I'll come back to your offer when we upgrade to >= 5.6, but it might be a while because 5.5 satisfies all requirements of newer models like Anima, Microsoft Lens and Ideogram 4, without requiring the CLIP refactor yet.


One issue to consider specifically, because it makes me rely on transformers v5 internals again:

Tied weights can be a problem, because in a trainer you work a lot with custom quantizations and custom linear layers, hooking LoRAs to linear layers, etc.

Below is code that *duplicates* instead of ties, because otherwise we dun into an issue that `lm_head.weight` is quantized (because it's a linear layer) and `model.embed_tokens.weight` is not quantized (because it's not a linear layer). But they are tied weights!

If you could make the tied weights a public API, as you did with the weights conversions, that would be helpful to keep this stable

```
        #tied weights (e.g. T5EncoderModel's encoder.embed_tokens.weight <-> shared.weight, or
        #Qwen3ForCausalLM's lm_head.weight <-> model.embed_tokens.weight) are saved only once in the checkpoint,
        #so the tied key above is never assigned and stays an empty meta tensor. populate it by cloning the
        #source weight that was actually loaded, rather than aliasing the same Parameter object: aliasing would
        #make a later in-place quantization of one side (e.g. quantize_layers() quantizing lm_head) silently
        #corrupt the other (e.g. the embedding table), since both attribute paths would refer to the same object.
        tied_weights_keys = getattr(sub_module, '_tied_weights_keys', None)
        if tied_weights_keys is not None:
            for target_key, source_key in tied_weights_keys.items():
                module = sub_module
                *parents, tensor_name = target_key.split(".")
                for p in parents:
                    module = getattr(module, p)
                if module._parameters[tensor_name].is_meta:
                    source = sub_module.get_parameter(source_key)
                    module._parameters[tensor_name] = type(module._parameters[tensor_name])(source)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.