Add support for DeepseekAI's DeepseekVL
Author: geetu040
Pull request: yes
Labels: name: New model

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36110

This PR adds [DeepseekAI](https://deepseek.ai/)'s DeepseekVL model to Hugging Face Transformers.

DeepseekVL is an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios.

Relevant Links
- Research Paper: [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://arxiv.org/abs/2403.05525)
- Authors: [Haoyu Lu](https://arxiv.org/search/cs?searchtype=author&query=Lu,+H), [Wen Liu](https://arxiv.org/search/cs?searchtype=author&query=Liu,+W), [Bo Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang,+B), et al.
- Implementation: [github.com/deepseek-ai/DeepSeek-VL](https://github.com/deepseek-ai/DeepSeek-VL/)
- Models Weights: [huggingface.co/collections/deepseek-ai/deepseek-vl](https://huggingface.co/collections/deepseek-ai/deepseek-vl-65f295948133d9cf92b706d3)

CC: @Benjamin-eecs, @RERV (github contributors of [deepseek-ai/DeepSeek-VL](https://github.com/deepseek-ai/DeepSeek-VL/))

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@ArthurZucker, @Rocketknight1, @Cyrilvallez, @zucchini-nlp 

## TODOs

- [x] Documentation
- [x] Import Statements and Auto Modeling
- [x] Modeling
  - [x] VisionModel
  - [x] TextModel
  - [x] AlignerModel
- [x] Tests
- [x] Integration Tests
- [x] Weight Conversion Script
- [x] Model Cards
- [x] Tokenizer/Processor
- [x] Fix CI/CD tests

Comments:
@zucchini-nlp , @Rocketknight1, @Cyrilvallez

The `Deepseek-VL` uses `Sam` as backbone for encoding high-resolution images.
And to be more specific, the backbone is `SamVisionEncoder` instead of `SamModel`, which is not available as a public class. By which I mean that, you can do following with `SamModel` but not with `SamVisionEncoder`
```py
from transformers import SamConfig, SamModel
config = SamConfig()
model = SamModel(config)
```
---
I think that we should rename `SamVisionEncoder` -> `SamVisionModel`, inherit it from `SamPreTrainedModel` and make it accessible to the user. I don't think it breaks backward compatibility in any way.

Otherwise, we would have to copy all the classes that build `SamVisionEncoder` for deepseek. There is nothing wrong with this either but having a `SamVisionModel` along with a `SamModel` makes sense, since it might benefit someone else as well.

If you think having a `SamVisionModel` makes sense, should that be done in a separate PR?

Btw, final results would look like this
```py
from transformers import SamVisionConfig, SamVisionModel
config = SamVisionConfig()
model = SamVisionModel(config)
```
and `SamVisionConfig` is already available publically.

@geetu040 we had similar situation with ideficsVision afair. Yes, in that case, we can just make it public and add in the docs. Renaming though would be breaking, imo we can leave name as is

@zucchini-nlp is it okay to do it in the same PR? or should I create a new one

@geetu040 imo a new PR will make it easier for us to iterate and review

Hi @zucchini-nlp, I am working on the `SamVisionEncoder` (going to create the PR soon) and I have a quick question.
I realized that `SamVisionAttention` and `SamVisionSdpaAttention` produce `attn_weights` of different shapes when `output_attentions=True`.

Can you please answer these 2 questions:
1. Is this allowed in `transformers` for the 2 attentions to produce outputs of different shapes?
2. And lets suppose we do something that changes the shape of `output_attentions`, does that break backward compatibility?

@geetu040 no, that is not expected to have different shapes. Usually using sdpa attention means that no `attn_weights` are returned, so it should be available only through 'eager' attention modules

I see that the weights are calculated on top of SDPA by manual matmul of key and query, which imo defeats the purpose of using SDPA in the first place. Can you remove the returned attention and raise warning similar to what is done in ViT?  

@zucchini-nlp sure I'll do that.

Hi @zucchini-nlp I have a question regarding the config and model hierarchy, considering how the sub_configs are handled in transformers by `PreTrainedModel` and the tests. Which one would you prefer? I have shown the configuration below, the same would be reflected for the models.

(1)
```
config: DeepseekVLConfig
    text_config: LlamaConfig
    vision_config: DeepseekVLVisionConfig
        use_high_res: bool
        low_res_config: SiglipVisionConfig
        high_res_config: SamVisionConfig
```
(2)
```
config: DeepseekVLConfig
    text_config: LlamaConfig
    use_high_res: bool
    low_res_vision_config: SiglipVisionConfig
    high_res_vision_config: SamVisionConfig
```
Or if there is a third better solution.

---

CC: @Shakib-IO

@geetu040 the second one is more aligned with transformers format. The main thing to follow, is to store all values used in a pre-trained backbone in its own config. Then all these backbone configs need to be under a general `MyModelConfig`. If there are values used outside of the backbone, like `mm_proj_dimension`, we add them under the big general config

@zucchini-nlp , @Rocketknight1, @Cyrilvallez

Hi everyone, I've moved this PR out of draft. Everything is complete except for the model cards. It's now ready for review. Thanks!

Thanks @geetu040 , I will take a look tomorrow

Hi @zucchini-nlp, Thanks for a review, I have updated accordingly.

> Also we could use modular, to copy parts of processing and image processing. Not required though

I wanted to apply modular but can't really find much code that is reusable. We can reuse the 2 ModelOutput classes, except for that I cannot really think of anything else that could benefit from modular. I was using the `preprocess` from `ViT`, but that's also changed now since https://github.com/huggingface/transformers/pull/36248#discussion_r2016197539. Do you still suggest that I use modular to keep everything under one file and to reuse the few lines we possibly can?

PS: Failing tests are unrelated

Feel free to ping me again once ready!

This PR is now 100% complete, covering modeling, processing, documentation, checkpoint conversion, tests, and model cards. It should be ready for review.

I have also transferred the checkpoints from my personal Hugging Face account [geetu040](https://huggingface.co/geetu040) to the [deepseek-community](https://huggingface.co/deepseek-community) account. You can find these updated checkpoints in this collection: [Deepseek-VL](https://huggingface.co/collections/deepseek-community/deepseek-vl-684d369d6e04c26577750485).
- [deepseek-community/deepseek-vl-1.3b-chat](https://huggingface.co/deepseek-community/deepseek-vl-1.3b-chat)
- [deepseek-community/deepseek-vl-1.3b-base](https://huggingface.co/deepseek-community/deepseek-vl-1.3b-base)
- [deepseek-community/deepseek-vl-7b-chat](https://huggingface.co/deepseek-community/deepseek-vl-7b-chat)
- [deepseek-community/deepseek-vl-7b-base](https://huggingface.co/deepseek-community/deepseek-vl-7b-base)

---

@zucchini-nlp previously reviewed and approved the modeling code in [https://github.com/huggingface/transformers/pull/36248#pullrequestreview-2725116254](https://github.com/huggingface/transformers/pull/36248#pullrequestreview-2725116254). Since then, I haven’t changed the core implementation, just mainly refactored by separating `DeepseekVLHybrid` from `DeepseekVL`, as suggested here: [https://github.com/huggingface/transformers/pull/36248#discussion\_r2018434624](https://github.com/huggingface/transformers/pull/36248#discussion_r2018434624).

CC: @ArthurZucker, @LysandreJik, @zucchini-nlp, @Rocketknight1

> This PR is now 100% complete, covering modeling, processing, documentation, checkpoint conversion, tests, and model cards. It should be ready for review.

A soft reminder for review: @ArthurZucker @zucchini-nlp 

Sorry sir! Reviewing

> A soft reminder for review: @ArthurZucker @zucchini-nlp

@ArthurZucker, another soft reminder for review!

Everything is complete as said in detail here [#36248-issuecomment](https://github.com/huggingface/transformers/pull/36248/#issuecomment-2972501705). CI is also green and up-to-date with `main`

#### Notes regarding SAM
I have changed the `_init_weights` for `SamPreTrainedModel` which also reflects changes in `SamHQPreTrainedModel`. This was intentional and required because of the failing tests at [jobs/1817956](https://app.circleci.com/pipelines/github/huggingface/transformers/137157/workflows/5d6721a9-c434-48dc-83cb-d32ea4a1fe4e/jobs/1817956)
```
FAILED tests/models/deepseek_vl_hybrid/test_modeling_deepseek_vl_hybrid.py::DeepseekVLHybridModelTest::test_can_init_all_missing_weights - AssertionError: False is not true : The following keys are not properly handled by `_init_weights()`:
['high_res_vision_model.vision_encoder.pos_embed']
```
Also, this makes sense, since `pos_embed` had already been explicitly set to zero for `SamHQPreTrainedModel`, so doing same for `SamPreTrainedModel` doesnot hurt anything.
If you think this should have been done in a separate PR, let me know, I'll create one!

Left a few tiny comments 😄 

@ArthurZucker @zucchini-nlp

Thanks for the reviews! I've addressed all the feedback, this should be ready for another look.

For checkpoints, see this: [#36248-issuecomment](https://github.com/huggingface/transformers/pull/36248/#issuecomment-2972501705)

On it!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_vl, deepseek_vl_hybrid, janus