Add Magma from Microsoft to Transformers
Author: VladOS95-cyber
Pull request: no
Labels: name: New model

### Model description

Magma is a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets.

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Paper: https://huggingface.co/papers/2502.13130
Original code: https://github.com/microsoft/Magma

Comments:
@ArthurZucker @qubvel Hi, if there any plans to add this model and nobody is working on it from HF team, I could take it when I finish with DeepSeekV2 that should be soon enough.

I think Microsoft team is already looking into it

cc @alvarobartt , afaik you were discussing it with the team

@zucchini-nlp Yes, we were looking into it but occupied with the release of training code and pretraining data. And thanks to @alvarobartt again for the help!

@VladOS95-cyber , your help would be highly appreciated if you could help with the integration!

Hi @jwyang! Sure, no problem

@VladOS95-cyber , thank you so much!

BTW, transformers lib does not take some special cases in the vision encoder into account. As Magma used convnext as the backbone, and it has a gamma parameter key for the layer scaling. Unfortunately, this will be automatically replaced by "weight" as for layernorm layer. Please refer this explanation:

https://github.com/microsoft/Magma?tab=readme-ov-file#customized-transformers

I am wondering can we fix the issue in transformers? We may not want to replace ALL 'gamma' with 'weight'.


Hi @jwyang! Ok, thanks for the info, I'll take a look

Hi, @VladOS95-cyber , any updates on your side?

If not, I can take this over. As mentioned above: 

https://github.com/huggingface/transformers/issues/36629#issuecomment-2716320175

Before integrating our Magma model, we need to fix an issue of model parameter loading. What is your suggstion about this?

Hi @jwyang! Sorry, not yet, I was stacked a little bit by finishing another models integration. I plan to start working on it this weekends. If it is too late, then of course you could take it over. 

great! Should I submit a PR first to fix the model loading issue, and then another PR for the Magma integration?

> great! Should I submit a PR first to fix the model loading issue, and then another PR for the Magma integration?

I think it's better to ask HF team @zucchini-nlp @alvarobartt 

@jwyang so to make sure I understand fully: the convNext you used when training has a `gamma` param and thus the ckpt has keys for `gamma`. However, in transformers the same model uses a different key: `weight`. If that is the case, we usually just rename all leys from the state dict to match with the transformers. It can be done in `convert_magma_weights.py` script

Hi, @zucchini-nlp, yes, it is a bit tricky here in that Magma used open_clip convnext constructor to build the vision tokenizer. When loding Magma ckpt, if the key containing "gamma" in the ckpt is replaced with "weight", the model parameters in open_clip convnext cannot be fully loaded into Magma model. 

Yes, but the new checkpoint can be saved in the hub under a new model, like `Magma-hf`. We don't have to preserve loading ability with the original repo, and the shipped model will be adapted for the core transformers naming.

The provided fix from Magma repo which replaced all keys when loading will be very breaking if we accept in transformers. Let's see what other ways we have. If you can make a draft PR, and we can discuss there to see more clearly the root issue and possible solutions

Hi, @zucchini-nlp, here is the supposed changes for the PR: 

https://github.com/jwyang/transformers/commit/1708d3ca3608b3c229d7ce6f0351e652436d7d33

It will not break the behavriour of transformers i guess, but would love to learn your thoughts.

actually, this is not a stand-alone issue for Magma, I did see some related ask about this in the past, e.g.:

https://github.com/huggingface/transformers/issues/35617
https://github.com/huggingface/transformers/issues/33190

Hmm, I just noticed this PR from Open_CLIP/timm author:

https://github.com/huggingface/transformers/pull/35615/files

Probably this has been resolved in some newest version of transformers? The answer seems to be Yes in v4.49!


@jwyang cool! Didn't know we were already replacing them when loading. Sorry misunderstood you and thought we need to add new replacement keys

If it is resolved, super nice! 

great, I will move forward to submit a PR for Magma.