Remove unnecessary unsqueeze - squeeze in rotary positional embedding
Author: fxmarty
Pull request: yes

As per title, removes unnecessary operations in the model initialization and forward.

Fixes https://github.com/pytorch/pytorch/issues/109292

Comments:
_The documentation is not available anymore as the PR was closed or merged._

FYI @gante if there is anything we are missing here? 

There's the same unsqueeze - squeeze in falcon. Maybe I am misunderstanding something in rotary positional embedding.

---
Edit:

This test fails on this branch:
```
FAILED tests/models/llama/test_modeling_llama.py::CodeLlamaIntegrationTest::test_model_7b_logits - AssertionError: Lists differ: ['
▁<PRE> def remove_non_ascii(s: str) -> st[893 chars]ID>'] != ['
 <PRE> def remove_non_ascii(s: str) -> st[893 chars...
```

however it seems unrelated (also fails on 866df66fe4a3b3b948b926d849ef291675b8a24c & tokenizers==0.13.3)



Yep it's unrelated, seen this failing, the `fast` tokenizer is not properly splitting 

Updated other affected archs.

This PR https://github.com/huggingface/transformers/pull/22785 was great, but incomplete. cc @fpgaminer

Running slow tests (`RUN_SLOW=1 pytest tests/models/gpt_neox/test_modeling_gpt_neox.py -s -vvvvv`, `RUN_SLOW=1 pytest tests/models/idefics/ -s -vvvvv`,  `RUN_SLOW=1 pytest tests/models/falcon/ -s -vvvvv`), no more tests fail than on main.

Related PR: https://github.com/huggingface/transformers/pull/25830, maybe @ArthurZucker you want to merge that first?

---

Note: some slow tests do not pass, but they don't pass on main 0a55d9f7376f72ad3ff296d4249840021b03bcc4 either:

```
FAILED tests/models/falcon/test_modeling_falcon.py::FalconModelTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/falcon/test_modeling_falcon.py::FalconModelTest::test_disk_offload - AssertionError: False is not true
FAILED tests/models/falcon/test_modeling_falcon.py::FalconModelTest::test_feed_forward_chunking - AssertionError: False is not true
FAILED tests/models/falcon/test_modeling_falcon.py::FalconModelTest::test_left_padding_compatibility - AssertionError: False is not true

FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_disk_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_feed_forward_chunking - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_model_parallelism - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_disk_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_feed_forward_chunking - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_model_parallelism - AssertionError: False is not true
```

Do you think this can make it in the release?

Actually let's wait a bit in case this breaks things! 


sure

Should be in a good state now.

Summary of slow tests:

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/llama/ -s -vvvvv` errors on (same as main)

```
FAILED tests/models/llama/test_modeling_llama.py::CodeLlamaIntegrationTest::test_model_7b_logits
E       AssertionError: Lists differ: ['
▁<PRE> def remove_non_ascii(s: str) -> st[893 chars]ID>'] != ['
 <PRE> def remove_non_ascii(s: str) -> st[893 chars]ID>']
```

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/idefics/ -s -vvvvv` errors on (same as main)

```
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_disk_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_feed_forward_chunking - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_disk_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_feed_forward_chunking - AssertionError: False is not true
```

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/mistral/ -s -vvvvv` errors on (same as main)

```
FAILED tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_model_7b_generation - AssertionError: 'My f[17 chars]t is mayonnaise. I love it on sandwiches, in s[13 chars]gers' != 'My f[17 chars]t is 100% ketchup. I love it on everythin...
FAILED tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_model_7b_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
```

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/falcon/ -s -vvvvv` errors on (same as main)

```
FAILED tests/models/falcon/test_modeling_falcon.py::FalconModelTest::test_left_padding_compatibility - AssertionError: False is not true
```

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/gpt_neox/ -s -vvvvv` pass.

This PR fixes `tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_model_7b_logits` as well. 

The test `tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_model_7b_generation` (that uses using slow tokenizers) seem to have never worked (on 72958fcd3c98a7afdc61f953aa58c544ebda2f79) cc @Bam4d, so I will ignore it for now.

@fxmarty thank you for the fix! 

I suppose this redundant pattern got in `gpt_neox`, and we were copying it over to other new models with RoPE :)