Bug fix for drop path decay rate in swin transformer
Author: abhi-glitchhg
Pull request: yes

# What does this PR do?

This PR fixes #33974 . 


As i had mentioned in the issue, I feel that swin transformer implementation has incorrect implementation of stochastic depth decay.


According to the official implementation, drop_prob for every SwinLayer is different. 

https://github.com/microsoft/Swin-Transformer/blob/f82860bfb5225915aca09c3227159ee9e1df874d/models/swin_transformer.py#L544

https://github.com/microsoft/Swin-Transformer/blob/f82860bfb5225915aca09c3227159ee9e1df874d/models/swin_transformer.py#L558

https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L397-#L408

But in transformers, we were using a constant value that is picked from the config file. I feel that implementations in transformers should be closer to the official ones.  This also applies for the SwinV2 model.  (and maybe swin2sr as well)

Please do look into this and let me know. Also i have changed the variable names as well. I am very bad at naming, so any suggestions for the argument names are welcome 😄 

 

- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. -> #33974 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@amyeroberts, @qubvel @ArthurZucker 


Comments:
cc @molbap 

Hi, thanks for the reply. I was waiting for confirmation if this was indeed a bug or intended behaviour. I will fix the lint and other errors.  
Thanks.
abhijit

In the #33974 , @ArthurZucker mentioned the messy initialisation of swinlayer class, i would not like to touch it in this pr. 

Personally i think  the initialisation looks good. But if you think we should make it simpler, i m happy to tackle it in another pr

```
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::DeiT2RobertaModelTest::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 21]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::ViT2BertModelTest::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 21]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::Swin2BartModelTest::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 21]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::ViT2TrOCR::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 21]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::LayoutLMv32TrOCR::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 28]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::VIT2GPT2Test::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 27]) != (13, 20)
FAILED tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::Donut2GPT2Test::test_encoder_decoder_model_generate - AssertionError: torch.Size([13, 27]) != (13, 20)
```

Hmm, these tests are failing in CI. 

Edit: I guess these were unrelated to my changes, merged the main branch and these errors have gone


The CI is green. Yey! ig now the pr is ready for review @molbap 

Btw i really loved the infra for CI/CD, tests run fast! Linting tools work amazingly!  I also recently watched @ArthurZucker's  pytorch conference talk and now i **really** understand the pain points mentioned in the video. 

Thank you guys for maintaining such a high-impact library!


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34291). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.