Fix initialization of OneFormer
Author: bvantuan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Following #38061 and #38864, I noticed the same error for OneFormer due to duplicate initializations. I essentially applied the same fix as in #38864.

Code to reproduce the bug:
```python
from transformers import (
    AutoModelForImageClassification,
    OneFormerForUniversalSegmentation,
    OneFormerConfig,
)

backbone_name = "microsoft/resnet-18"
# backbone_name = "microsoft/swin-tiny-patch4-window7-224"

def params_match(params1, params2):
            return all((p1 == p2).all() for p1, p2 in zip(params1, params2))

# load pretrained backbone model
backbone_model = AutoModelForImageClassification.from_pretrained(backbone_name)

# load OneFormerConfig with a pretrained backbone
config = OneFormerConfig(
    backbone=backbone_name,
    use_pretrained_backbone=True,
)
oneformer = OneFormerForUniversalSegmentation(config)

# AssertionError: parameters don't match
assert params_match(
    backbone_model.base_model.encoder.parameters(),
    oneformer.model.pixel_level_module.encoder.encoder.parameters(),
)

```

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Cyrilvallez 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
It looks like there are still quite some blocks that are redundant here no? This one seem even worst then `mask2former` 🥲

BTW, if you're interested in model initializations, and want to keep contributing, here is a message I shared on internal slack some time ago 🤗 Most of the time, it's much easier than `mask2former` and this one (where we do have to manually follow all the graph to simplify all redundant blocks)

*Hey team! Just merged https://github.com/huggingface/transformers/pull/37070
TLDR, almost all model's _init_weightsmethods are missing some layers (usually the different Norms, or layer that are explicit nn.Parameter). It is in general not an issue, but might lead to weird behavior if using an old corrupted checkpoint on the hub, or initializing a model on meta, and then wanting to move back to cpu and initialize correctly. But as we reinforced the usage of initialization on meta device in from_pretrained for efficiency, it's a nice idea to fix it. Also, _init_weightswill now correctly dispatch for composite model, i.e. if a module in a model is itself a PreTrainedModel, when going though the layer graph to init all weights, it will use the correct model's init_weightsfor itself and all its children (which was not the case before and would sometime lead to very awkward implementation of _init_weightsfor the "outer-most" model.
I already fixed about between 100 and 150 models, but would be great to have as much as possible fixed before next release! A new test is enforcing this behavior for "most important" and newer models, so that we don't merge new models that do not respect it. Fixing it for older models is extremely easy but dull work, so this could be a nice way to still do something useful if you find yourself mentally tired at some point and don't want to think that much :grin: The PR in itself should provide a lot of different examples, and you can ask me if you want to participate but something is unclear! The easiest way to do that is to relax the skip here https://github.com/huggingface/transformers/pull/37070/files#diff-a6ebb0bccfe159ca855e9d9b5d6fbbeb62fced0c0296de0993ba39ef5c862cf0R522-R523, and fix the model that are not passing the test - it will tell you what layers are problematic for each model!*

> It looks like there are still quite some blocks that are redundant here no? This one seem even worst then `mask2former` 🥲

Yes, indeed, I felt the same :)). After looking again, I noticed some redundant initializations. I removed two blocks `if isinstance(module, OneFormerTransformerModule)` and `elif isinstance(module, OneFormerTextContextDecoder)` since they were never actually executed. Additionally, the three blocks `OneFormerTransformerDecoderSelfAttentionLayer`, `OneFormerTransformerDecoderCrossAttentionLayer` and `OneFormerTransformerDecoderFFNLayer` can be consolidated into a single block `OneFormerTransformerDecoderLayer`. I also enabled training mode, as `oneformer` includes an additional module that requires it.
https://github.com/huggingface/transformers/blob/2166b6b4ff09f6dd3867ab982f262f66482aa968/src/transformers/models/oneformer/modeling_oneformer.py#L2877-L2878

> TLDR, almost all model's _init_weightsmethods are missing some layers (usually the different Norms, or layer that are explicit nn.Parameter). It is in general not an issue, but might lead to weird behavior if using an old corrupted checkpoint on the hub, or initializing a model on meta, and then wanting to move back to cpu and initialize correctly. But as we reinforced the usage of initialization on meta device in from_pretrained for efficiency, it's a nice idea to fix it. Also, _init_weightswill now correctly dispatch for composite model, i.e. if a module in a model is itself a PreTrainedModel, when going though the layer graph to init all weights, it will use the correct model's init_weightsfor itself and all its children (which was not the case before and would sometime lead to very awkward implementation of _init_weightsfor the "outer-most" model.

Thank you @Cyrilvallez for the proposal! I'm definitely interested and would be happy to contribute 🤗! Could you please let me know how to proceed? For example, would you prefer one big PR covering many models, or a small PR for each model?

The best way to proceed would be to open a PR for each new year [here](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L529). For example, setting it to 2024 should give you about 15 new model errors that you can fix in a single PR!

Got it, thanks @Cyrilvallez for the guidance! I’ve opened a draft PR for 2024 #38987 and will let you know once it’s ready for review.



Thanks @Cyrilvallez for the review!