Add support for BAGEL from ByteDance
Author: Shakib-IO
Pull request: no
Labels: name: New model

### Model description

### Model description
ByteDance recently released a multimodal understanding and generation model. While the code and weights have been made publicly available, the code requires significant formatting and cleaning to align with the standards of the Hugging Face Transformers library. 

**BAGEL**

BAGEL, an open‑source multimodal foundation model with 7B active parameters (14B total) trained on large‑scale interleaved multimodal data. BAGEL outperforms the current top‑tier open‑source VLMs like Qwen2.5-VL and InternVL-2.5 on standard multimodal understanding leaderboards, and delivers text‑to‑image quality that is competitive with strong specialist generators such as SD3. Moreover, BAGEL demonstrates superior qualitative results in classical image‑editing scenarios than the leading open-source models. More importantly, it extends to free-form visual manipulation, multiview synthesis, and world navigation, capabilities that constitute "world-modeling" tasks beyond the scope of previous image-editing models. The figure below showcases BAGEL's qualitative performance.


### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Research Paper: https://arxiv.org/pdf/2505.14683

Authors: Chaorui Deng, Deyao Zhu, Kunchang Li, et al.

Implementation: 
- [ByteDance-Seed/BAGLE](https://github.com/bytedance-seed/BAGEL)

Model Weights:
- [ByteDance-Seed/BAGEL](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT)

Comments:
I would like to implement these models in `transformers`. However, due to the large size of this model (with 7 billion active parameters), I focus on developing and testing smaller architectures initially. Once the core functionality is validated, I then evaluate the final outputs and consistency on larger machines using the full-scale architecture. Does that sound possible, or should I avoid this altogether?

Hey @Shakib-IO ! 

Yes, adding the model is planned and the authors are willing to contribute the model themselves, though probably it will be in diffusers. 

Hi @Shakib-IO ,
Thank you so much for offering to help add transformers support for BAGEL! As the authors, we really appreciate it.
Due to our limited time and experience with transformers, your help would save us a lot of effort. Starting with the 1.5B model makes sense, and we can work together to test the 7B version afterwards.
Thanks again for your support!

Hi! @zucchini-nlp @Andy1621  I’d love to help implement the model with my experience of implementing Janus, especially if @Shakib-IO hasn’t started yet or I can tag along. It looks really exciting, especially it's `any-to-any` capability. Could you confirm whether this should go in transformers or diffusers coz from skimming the paper it looks like transformers approach mostly even though they have diffusion module. From codebase I can see they have used CFG which transformers also supports 🤔 . 

Your help will be appreciated @yaswanth19 . I didn't really read the paper yet, we'll decide on which library the model belongs to and let you know under this issue

Hey all! I think we can host the model on transformers side, indeed it is similar to other models we already shipped. So as long as the authors are okey, feel free to work on a PR 🤗 

Thanks @zucchini-nlp! I’ll start working on this and raise a draft PR soon. I’ll also ping @Andy1621 if I run into any questions. Given the complexity of this model, it might be helpful to have a Slack channel to streamline communication — just a thought to make things more efficient! WDYT?

Cool! Just create a workspace~

Hi @yaswanth19 
Thanks for your interest! I’ve just finished reading the paper and would be glad to collaborate on the implementation. Once you draft PR, feel free to add me to the repository so we can move forward together.

I'm also happy to connect on Slack to discuss the implementation details. I hope that’s alright with @Andy1621  as well.

Hi @yaswanth19 
From an image generative models' view, the BAGEL model seems to more like a traditional diffusion/flow based model with some contextual improvements. Thus I wonder if it's possible and benificial to move part (specifically the flow based iterative generation process) of the model into the [diffuers](https://github.com/huggingface/diffusers) library, which already have a pipeline infrastructure for iterative (latent) image generation and latent decoding?

I'm looking forward to further discussion.

Hey @Seas0 , Yep BAGEL is a flow model for image generation and AFAIK this might be the first flow based model in transformers. IMO this model leans more towards transformers lib coz of the architecture (similar to most of the vlms in here) and capability like text generation, image generation and editing **with thinking** which I don't think a diffusers pipeline could support. WDYT? @zucchini-nlp @sayakpaul 

>if it's possible and benificial to move part (specifically the flow based iterative generation process) of the model into the diffusers library,

I don't thinking splitting the model is possible as both the ViT and LLM are required for any type of generation and generally do the heavy lifting 🤔 

We have a pipeline called UniDiffuser that might be worth referring to here:
https://huggingface.co/docs/diffusers/en/api/pipelines/unidiffuser

> I don't thinking splitting the model is possible as both the ViT and LLM are required for any type of generation and generally do the heavy lifting 🤔

+1, currently we have only Qwen-Omni which uses a DiT for audio generation module which has its whole pipeline in transformers. Since we don't have many such models yet, there is no clear standard as to how to use `diffusers as backbone`. IMO we should consider moving flow-based/denoising modules into `diffusers` similarly to how `TimmBackbone` is used for new VIT models

Let's keep BAGEL in `transformers` until then

@yaswanth19 I’m delighted to see your contributions to supporting the bagel model. Could you please let me know if the official Transformers support for BAGEL is expected to go live within the next week or two?

Hey @Dswn888 , Unfortunately it won't be possible due to my bandwidth constraints. A bit streached but a rough PR should be ready for review in 4-5 weeks if that helps.

> Hey [@Dswn888](https://github.com/Dswn888) , Unfortunately it won't be possible due to my bandwidth constraints. A bit streached but a rough PR should be ready for review in 4-5 weeks if that helps.

Got it. Thanks for reply.

> Hey [@Dswn888](https://github.com/Dswn888) , Unfortunately it won't be possible due to my bandwidth constraints. A bit streached but a rough PR should be ready for review in 4-5 weeks if that helps.

Hi @yaswanth19, I'm just writing to check if there are any updates.