Add qwen2
Author: JustinLin610
Pull request: yes

# Adding Qwen2

This PR adds the support of codes for the coming Qwen2 models. For information about Qwen, please visit https://github.com/QwenLM/Qwen. @ArthurZucker 

Comments:
Feel free to rebase on main as well if some CIs are unrelated to your PR, and ping me whenever for a final review! 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28436). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks a lot for this pr and bearing with me!  🤗

Qwen1.5-0.5B-Chat lose output.weight after load from_pretrained and save_pretrained
![image](https://github.com/huggingface/transformers/assets/99069487/b0d7db55-d461-47c6-9fca-f32174578c7b)
![image](https://github.com/huggingface/transformers/assets/99069487/7d65f395-e44e-43a1-af7e-d2028fac1d49)


SDPA & eager implem don't seem to match on main (5c341d4555ba3e4b656053317e372ebed0c5af37), even when not using an attn_mask:

```
FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 2.188e+00, torch at...
FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 1.156e+00, torch at...
FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_2_float32 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 1.369e+00, torch at...
```

cc @JustinLin610 have you tested both code paths?

Hi, Can I use it on qwen1? If not, how can I adapt it to qwen1? Thank you!

@JustinLin610 In the tests, `Qwen/Qwen2-450m-beta` is used in several places, but there is no such repository on Huggingface Hub. Could you let us know which repository you used when adding those tests, please. Thank you in advance.


Currently, this kind implies there is no integration tests (being run) at all when this model was added to `transformers`.


I'll fix the code paths. Previously we have tested it and passed all the test. However, we had name changes and caused troubles. Sry about this.

> @JustinLin610 In the tests, `Qwen/Qwen2-450m-beta` is used in several places, but there is no such repository on Huggingface Hub. Could you let us know which repository you used when adding those tests, please. Thank you in advance.
> 
> Currently, this kind implies there is no integration tests (being run) at all when this model was added to `transformers`.

https://huggingface.co/Qwen/Qwen1.5-0.5B This is the one that corresponds to the original `Qwen/Qwen2-450m-beta`

> SDPA & eager implem don't seem to match on main ([5c341d4](https://github.com/huggingface/transformers/commit/5c341d4555ba3e4b656053317e372ebed0c5af37)), even when not using an attn_mask:
> 
> ```
> FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_0_float16 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 2.188e+00, torch at...
> FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 1.156e+00, torch at...
> FAILED tests/models/qwen2/test_modeling_qwen2.py::Qwen2ModelTest::test_eager_matches_sdpa_inference_2_float32 - AssertionError: False is not true : padding_side=left, use_mask=False, batch_size=1, enable_kernels=False: mean relative difference: 1.369e+00, torch at...
> ```

This is strange to me btw as this part of code is exactly copied from the mistral impl.

> Hi, Can I use it on qwen1? If not, how can I adapt it to qwen1? Thank you!

No, you can't use it directly btw. You need to transform the state dict for the adaptation