Fix missing initializations for models created in 2024
Author: bvantuan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes missing weight initializations for models created in 2024.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Cyrilvallez 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Cc @Cyrilvallez ! I’ve completed fixes for 10 models, but with one remaining `TimmWrapperPreTrainedModel`. I think `test_can_init_all_missing_weights` is failing because the `from_pretrained` method for `TimmWrapperPreTrainedModel` is either not supported or not yet implemented. Currently, the only available method for loading a `timm` pretrained model is through `timm.create_model`.

Ha yes! For `timm`, you can skip the test instead, as the weights are created by timm, and we assume the checkpoints are always present!

Hi @Cyrilvallez , thank you for your response! The PR is now ready for review.

For `TimmWrapper`, since its initialization is handled on the `timm` side, I don't think we need the `post_init` and `_init_weights` functions.

For parametrizations such as `weight_norm`, the method reparameterizes a given weight to new parameters. In this case, only the underlying weight needs to be initialized, while the reparameterized components can be ignored. So I added a block to skip the parametrization check in `test_can_init_all_missing_weights`:
```python
if "parametrizations" in k1:
       continue
```

That’s all on my end!😊

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: dab_detr, dac, encodec, falcon_mamba, grounding_dino, mamba, mamba2, musicgen, musicgen_melody, omdet_turbo, qwen2_audio, seggpt, superglue, timm_wrapper

With my pleasure! Thanks for reviewing, @Cyrilvallez ❤️!

Hi @bvantuan Thank you for this PR 🤗 

There is however a small issue 🙏 

```bash
pip install pytest-flakefinder

git checkout e355c0a1

RUN_SLOW=1 python3 -m pytest -v --flake-finder --flake-runs=20 tests/models/omdet_turbo/test_modeling_omdet_turbo.py::OmDetTurboModelTest::test_batching_equivalence
```

this will fails almost always , but italways pass in the previous commit 1125513a.

The difference after this PR is quite large :


```
FAILED tests/models/omdet_turbo/test_modeling_omdet_turbo.py::OmDetTurboModelTest::test_batching_equivalence - AssertionError: tensor(False, device='cuda:0') is not true : Batched and Single row outputs are not equal in OmDetTurboForObjectDetection for key=decoder_coord_logits. Difference=0.13529819250106812.
```

Could you check why this test are now failing with larger difference between single v.s. batch inference?

Hi @ydshieh ! 🤗 Glad to help out! We can move the initialization of `nn.BatchNorm2d` into `nn.LayerNorm`’s initialization as shown below.
```python
elif isinstance(module, (nn.LayerNorm, nn.BatchNorm2d)):
        module.weight.data.fill_(1.0)
        module.bias.data.zero_()
```
but it will break BC. @Cyrilvallez , a second opinion, please!


Hey @bvantuan, fine for me to break BC on this one, from what I can gather from the PR history it was just copied from some other file, and it does not really make sense to have the BatchNorm initiliazed randomly like this!

Hi @Cyrilvallez @ydshieh ! Thank you for confirming this. I’ve added the change in #39239.



Confirmed #39239 fixes the issue I linked 🎉