Add Deepseek AI's Janus model
Author: ArthurZucker
Pull request: no
Labels: name: New model

name: Good Difficult Issue

### Model description

🚀 Janus-Series: Unified Multimodal Understanding and Generation Models
- https://github.com/deepseek-ai/Janus
- https://huggingface.co/deepseek-ai/Janus-Pro-1B 


### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
@ArthurZucker I will work on integrating this Model 🤩🤗 

Awesome! 🚀 

@yaswanth19 I am also interested in working on this. If you have something already it would be nice if you could open a PR. I can help code the model or review/test it.

Hi @ArthurZucker and @zucchini-nlp, do you plan to add "janus-flow" in transformers as well?

"janus" and "janus-pro" use the same architecture, but "janus-flow" is quite different and cannot be used under the same `JanusModel` that is being worked on.

Below, I've added some links and info for the 3 models in "janus-series".

**Janus**

- Weights: [deepseek-ai/Janus-1.3B](https://huggingface.co/deepseek-ai/Janus-1.3B) - Downloads 63,883
- Code: [deepseek-ai/Janus](https://github.com/deepseek-ai/Janus/tree/main/janus/models)
- Research Paper: [Janus](https://arxiv.org/abs/2410.13848) - Published: 17 Oct 2024 - Citations: 21

**Janus-Flow**

- Weights: [deepseek-ai/JanusFlow-1.3B](https://huggingface.co/deepseek-ai/JanusFlow-1.3B) - Downloads: 9,874
- Code: [deepseek-ai/Janus/janusflow](https://github.com/deepseek-ai/Janus/tree/main/janus/janusflow)
- Research Paper: [JanusFlow](https://arxiv.org/abs/2411.07975) - Published: 12 Nov 2024 - Citations: 2

**Janus-Pro**

- Weights: [deepseek-ai/Janus-Pro-1B](https://huggingface.co/deepseek-ai/Janus-Pro-1B) - Downloads: 64,859
- Weights: [deepseek-ai/Janus-Pro-7B](https://huggingface.co/deepseek-ai/Janus-Pro-7B) - Downloads: 223,437
- Code: [deepseek-ai/Janus](https://github.com/deepseek-ai/Janus/tree/main/janus/models) (same as Janus)
- Research Paper: [Janus-Pro](https://arxiv.org/abs/2501.17811) - Published: 29 Jan 2025 - Citations: 0

Janus-flow is a flow based model and we don't have any flow based models in transformers currently. I don't think it is worth adding it in the moment

> Janus-flow is a flow based model and we don't have any flow based models in transformers currently. I don't think it is worth adding it in the moment

@zucchini-nlp And what about the Deepseek-VL series? they are different from Janus-series
- Deepseek-VL: [paper](https://arxiv.org/abs/2403.05525) - [code](https://github.com/deepseek-ai/DeepSeek-VL/) - [weights](https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3)
- Deepseek-VL2: [paper](https://arxiv.org/abs/2412.10302) - [code](https://github.com/deepseek-ai/DeepSeek-VL2) - [weights](https://huggingface.co/collections/deepseek-ai/deepseek-vl2-675c22accc456d3beb4613ab)



@geetu040 yep, a very much needed model. I am not 100% sure if anyone is working on it, we have a PR for deepseek-v3 (https://github.com/huggingface/transformers/pull/35926) and for the v2 no-one is working currently as per last comment (https://github.com/huggingface/transformers/issues/35317).

CMIIW, Deepseek-VL2 uses v2 as backbone, so we'll have to first add the LM v2 model. If there isn't much difference between deepseek-v2 and v3, I think we can add support for v2 easily with modular after the above PR is merged. So feel free to comment under the above issue and work on v2 support :)

Thanks, I realised the constraints. I will start work on these.

Created #36110 and #36111 for tracking