Fix Mask2Former Weight Initialization Issues #35877
Author: sambhavnoobcoder
Pull request: yes

## Issue Description
As reported in #35877  , the `_init_weights` method in Mask2Former has several initialization issues:

1. Embeddings are initialized with std=0.02 instead of PyTorch's default (std=1.0)
2. Mask MLP biases are incorrectly initialized to zero
3. Multi-scale Deformable Attention initialization is being overwritten by the Mask2FormerPixelDecoderEncoderOnly branch

## Problem Identification
By comparing with the original Mask2Former implementation and analyzing the weight conversion scripts, we identified that:
- The embedding initialization diverged from PyTorch defaults
- MLP bias terms were not properly initialized
- The order of initialization blocks caused the multi-scale attention parameters to be overwritten

## Solution
The fix involves:
1. Using PyTorch's default std=1.0 for embedding initialization
2. Adding proper non-zero bias initialization for MaskedAttentionDecoderLayer
3. Reordering initialization blocks to prevent overwriting of multi-scale attention parameters
4. Fixing dtype in theta initialization (float32 instead of int64)

## Testing Strategy
Added new test file `test_initialization_mask2former.py` with three specific test cases:

1. `test_embedding_initialization`: 
   - Verifies embeddings are initialized with std=1.0
   - Checks empirical standard deviation of weights

2. `test_mlp_bias_initialization`:
   - Ensures MLP biases are non-zero
   - Validates proper initialization of bias terms

3. `test_multiscale_deformable_attention`:
   - Confirms correct initialization of sampling offsets
   - Verifies attention weights initialization
   - Checks value projection initialization

## Test Results
<img width="1253" alt="Screenshot 2025-01-27 at 4 06 54 PM" src="https://github.com/user-attachments/assets/52fb04b6-d7d9-411d-b48f-33e109386348" />

## Backward Compatibility
This change maintains backward compatibility as it only affects the initialization of new models and does not change the model architecture or forward pass.

## Related Issues
Fixes #35877 

## Implementation Links
Original implementation references:
- [Mask2Former Official Repository](https://github.com/facebookresearch/Mask2Former)
- [Initialization Code](https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/transformer_decoder/maskformer_transformer_decoder.py)
- [Deformable Attention](https://github.com/facebookresearch/Mask2Former/blob/main/mask2former/modeling/transformer_decoder/position_encoding.py)


**cc:** @Rocketknight1 @qubvel 

Comments:
cc @qubvel, can you do the initial review here?

Yeah, sure, will review tomorrow, thanks for the ping :hugs: 

to address point 1 , i have added links in the Implementation Links section in the PR description itself . 

Regrading point 2 , about Dependent Models
After careful analysis of the initialization patterns between Mask2Former and MaskFormer, I believe the current changes maintain consistency while respecting architectural differences. However, I would absolutely welcome further discussion on this point.

My reasoning:
1. The initialization changes I made are specific to Mask2Former's unique components:
   - Deformable attention initialization
   - Masked attention decoder initialization
   - These components don't exist in MaskFormer's architecture

2. For shared components:
   - The embedding initialization change (std=1.0) actually brings Mask2Former in line with PyTorch's defaults, which MaskFormer already follows
   - Common layers (Linear, Conv2d, BatchNorm2d) maintain consistent initialization patterns between both models

3. The conversion scripts show that both models handle weight loading similarly where architectures overlap:
   ```python
   # Both models use identical patterns for attention weight splitting
   size = src_att_weight.shape[0]
   offset = size // 3
   dst_state_dict[...] = src_att_weight[:offset, :]  # Query
   dst_state_dict[...] = src_att_weight[offset : offset * 2, :]  # Key
   dst_state_dict[...] = src_att_weight[-offset:, :]  # Value
   ```

However, if you feel there are specific initialization patterns that should be aligned between the models, I'm happy to make additional changes to ensure complete consistency across the codebase.

Also , @qubvel would it be fine if i discussed this with you on discord ? after spending ample amount of time going to and fro between if this is required or not , i think it would have been much faster if we could have had a really short discussion on discord . however , i'm also not sure if i can even text you on discord , since i cant seem to join discord due to the verification system on hugging face discord . @Rocketknight1 a really gentle ping to you if you could help in any way as well . Apologies in advance if the ping was redundant .

Hey @sambhavnoobcoder! Thanks for addressing the comments, I will review it next week, thanks for your patience 🤗 
Can you please ensure `make repo-consistency` is passing on the repo? We should either update dependent models or remove `#Copied from` statement on top of the method/function

@qubvel i was trying to get this to pass the torch tests , but to no avail . i cant seem to figure out what to fix here . could you please look into this a bit and tell me what i am missing here or what to change to fix this ? i can see the issue going stale multiple times and its high time i want to close this quickly . your assistance would be appreciated , and i'll take the appropriate action at the earliest accordingly . 

To make it easier for you, I would recommend install dev env with transformers and run 2 following command locally to make sure everything is fixed

```
make modified_only_fixup
```

```
make fix-copies
```

\+ run tests to make sure everything is green

```
RUN_SLOW=1 pytest -v tests/models/mask2former/
```