Add MLCD model
Author: tanhuajie
Pull request: yes
Labels: name: New model

name: Vision

# What does this PR do?

This PR adds **MLCD** model from DeepGlint-AI Team.

Fixes #36181


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts @qubvel @ArthurZucker 

## Quick Test

```python
from transformers import AutoProcessor, MLCDVisionModel
from PIL import Image
import requests

# Load model and processor
model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-336")
processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-336")

# Process single image
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt")

# Get visual features
outputs = model(**inputs)
features = outputs.last_hidden_state

print(f"Extracted features shape: {features.shape}")
```

Comments:
![image](https://github.com/user-attachments/assets/0a61c221-f0ef-4b2f-9d59-c8fc7eede9b1)

Hi, Pavel @qubvel. Hope this message finds you well. It appears that our PR failed on the final step during CI. From the error messages, it looks like the issue stems from the Rt-Detr tests rather than problems within our code. Could you please guide us on how to disable or skip the tests for other models so we can successfully complete the CI process for our PR? Thanks!

Hi Pavel and Arthur, @qubvel @Rocketknight1 :

Could you please take a moment to review the pull request? Your insights would be immensely appreciated and would greatly contribute to ensuring the quality of the changes. We're truly grateful for your help!

Thank you so much!

Gentle ping @qubvel, but let me know if you want me to take any part of the review!

> Hey @tanaymeh! Sorry for the delay and thanks a lot for working on the model addition to `transformers`, great work, and already looks super clean!
> 
> I see the model is built on pretty standard modules, so it would be incredibly helpful if you could reuse library modules with inheritance using our new `modular` tool.
> 
> * https://huggingface.co/docs/transformers/modular_transformers
> * see other models such as gemma/ijepa/siglip2 (`modular_*.py` file)
> 
> See other comments below!

I think you tagged the wrong person here @qubvel 😅

Ohh, sorry! Fixed 😄

Hi @qubvel! Thanks for your suggestions. I've made necessary modifications based on your advice and think it's ready for your review again. Here's a short summary of what we've done so far by this moment:

1. The `modeling_mlcd.py` file has been refactored and regenerated using the new `modular` tool, and we've reused as many existing modules and functions from other modeling files as possible right now.
2. The conversion script `convert_mlcd_weights_to_hf.py` has been refactored, following your recommendation to refer to both `mllama` and `siglip2`.
3. The `output_attentions` argument in `MLCDVisionModel` is now supported in the newest commit, and some cycle processes have been also vectorized to improve GPU parallel computation.
4. Additional integration tests for `MLCD` have been added to `test_modeling_mlcd.py`.
5. Fixed some typos.

Looking forward to your feedback on these updates. Much thanks!!!

Hi @qubvel! Thank you for your further suggestions. Based on your latest comments, I have made additional modifications to `modular_mlcd.py`. These changes include improving the reusability of the modular components through class inheritance and refactoring the Attention interface to support different attention implementations. 🤗

Hi @qubvel, just a gentle reminder to review the latest commits. Your feedback is incredibly valuable, and I’d appreciate it if you could take a look when you have a moment. Thanks! 🤗

Hey @qubvel, Thanks for your comments! I have refactored the attention interface and adjusted the code based on your suggestions. I think it's ready for your review again. Please take a look when you have time! Thanks 🤗

Hi @qubvel ,

Thanks again for your valuable feedback earlier! We’ve refactored the attention interface and made adjustments to the code based on your suggestions. It should be ready for your review now. Whenever you have some time, we’d really appreciate it if you could take a look. Thank you so much! 

Hi @qubvel, hope you're doing well! Just a quick follow-up on the updated interface based on your feedback—would love to hear your thoughts when you have time.

Thanks again!

Hi @tanhuajie, thanks for the ping, I will review it today or tomorrow 🤗 

Hi @qubvel ，Just a gentle reminder—when you have a moment, please help us review the updated interface. Would love to hear your feedback!  🤗🤗🤗

run-slow: mlcd

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mlcd']
quantizations: [] ...

Also integraion test is failing

```
FAILED tests/models/mlcd/test_modeling_mlcd.py::MLCDVisionModelIntegrationTest::test_inference - RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor
```

please, make sure to take into account `torch_device`

run-slow: mlcd

run-slow: mlcd

Hi @qubvel, I think it ready to merge now. 🤗 Thanks again for your review!

run-slow: mlcd

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mlcd']
quantizations: [] ...

@tanhuajie please see above comment regarding positional_embeddings

Also there is failing test for multi-GPU setup, I suppose `_no_split_modules` attribute should be updated or we should add `.to(device)`

```
FAILED tests/models/mlcd/test_modeling_mlcd.py::MLCDVisionModelTest::test_model_parallelism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument tensors in method wrapper_CUDA_cat)
```

Stacktrace:

```
self = MLCDVisionTransformer(
  (embeddings): MLCDVisionEmbeddings(
    (patch_embedding): Conv2d(3, 32, kernel_size=(2, 2), ...t_layernorm): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
  (vision_rotary_embedding): MLCDRotaryEmbedding()
)
pixel_values = tensor([[[[7.4678e-01, 9.0169e-01, 5.9101e-01,  ..., 1.9737e-01,
           2.6010e-01, 3.8128e-01],
          [8.7451...         [2.5349e-01, 4.0613e-01, 3.7064e-01,  ..., 8.5400e-01,
           5.6[229](https://github.com/huggingface/transformers/actions/runs/14318436461/job/40129963836#step:12:230)e-01, 1.6790e-01]]]], device='cuda:0')
output_attentions = False, output_hidden_states = False, return_dict = True

    @add_start_docstrings_to_model_forward(MLCD_VISION_INPUTS_DOCSTRING)
    def forward(
        self,
        pixel_values: Optional[torch.FloatTensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""
        Returns:
    
        """
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    
        if pixel_values is None:
            raise ValueError("You have to specify pixel_values")
    
        num_patches_height = pixel_values.shape[-2] // self.config.patch_size
        num_patches_width = pixel_values.shape[-1] // self.config.patch_size
        rotary_pos_emb = self.vision_rotary_embedding(num_patches_height, num_patches_width)
>       rotary_pos_emb = torch.cat([self.class_pos_emb, rotary_pos_emb], dim=0)
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument tensors in method wrapper_CUDA_cat)

src/transformers/models/mlcd/modeling_mlcd.py:536: RuntimeError
```


> @tanhuajie please see above comment regarding positional_embeddings

Remove this unused args 'positional_embeddings' now 🤗 

> self.class_pos_emb

I tried to add `rotary_pos_emb = rotary_pos_emb.to(self.class_pos_emb.device)`, and hope it work.

run-slow: mlcd

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mlcd']
quantizations: [] ...

Re the change, the idea is to use `position_embeddings` rather than `rotary_pos_emb` argument.

```python
        if position_embeddings is None:
            logger.warning_once(
                "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
                "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
                "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
                "removed and `position_embeddings` will be mandatory."
            )
            emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
            cos = emb.cos()
            sin = emb.sin()
        else:
            cos, sin = position_embeddings
```

So, 

```python
emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
position_embeddings = (emb.cos(), emb.sin())
```

Should happen not inside the attention, but inside the model itself and then have them propagated across layers

> Re the change, the idea is to use `position_embeddings` rather than `rotary_pos_emb` argument.
> 
> ```python
>         if position_embeddings is None:
>             logger.warning_once(
>                 "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
>                 "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
>                 "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
>                 "removed and `position_embeddings` will be mandatory."
>             )
>             emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
>             cos = emb.cos()
>             sin = emb.sin()
>         else:
>             cos, sin = position_embeddings
> ```
> 
> So,
> 
> ```python
> emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
> position_embeddings = (emb.cos(), emb.sin())
> ```
> 
> Should happen not inside the attention, but inside the model itself and then have them propagated across layers

Okay, I'll make the revisions based on your suggestions later~

> > Re the change, the idea is to use `position_embeddings` rather than `rotary_pos_emb` argument.
> > ```python
> >         if position_embeddings is None:
> >             logger.warning_once(
> >                 "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
> >                 "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
> >                 "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
> >                 "removed and `position_embeddings` will be mandatory."
> >             )
> >             emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
> >             cos = emb.cos()
> >             sin = emb.sin()
> >         else:
> >             cos, sin = position_embeddings
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > So,
> > ```python
> > emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
> > position_embeddings = (emb.cos(), emb.sin())
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Should happen not inside the attention, but inside the model itself and then have them propagated across layers
> 
> Okay, I'll make the revisions based on your suggestions later~

Hey @qubvel. I think it's done now. 🤗 Let me know if you have any feedback when you have a moment.

run-slow: mlcd

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mlcd']
quantizations: [] ...

cc @ArthurZucker for review when you have bandwidth

Dear @ArthurZucker ,

I hope this message finds you well! I just wanted to kindly follow up and ask if you could help review this whenever you have the time. Please don’t hesitate to let me know if there’s anything I can provide to assist. I truly appreciate your time and support!

Hey, @tanhuajie let's resolve conflicts with main, and we are good to merge it!

> Hey, @tanhuajie let's resolve conflicts with main, and we are good to merge it!

Great, I'll resolve the conflicts right away. 🤗

Hi @qubvel, I've resolved the conflicts, removed the dummy `MLCDConfig`, and refined `_init_weights()`. Then, I think MLCD should now be ready to merge. 🤗cc @anxiangsir @ArthurZucker 

run-slow: mlcd

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/mlcd']
quantizations: [] ...

Thanks a lot for iterating on the model addition! Gerat work 🤗 
Merging 🚀