Several fixes related to rotary position embeddings
Author: mseeger
Pull request: yes

# What does this PR do?

- Changes related to `position_embeddings` being a mandatory argument
- Remove `position_ids` argument of `apply_rotary_pos_emb`
- Replace `torch.stack` by `torch.cat`, former requires equal shapes
- `esm`: RoPE depends on `position_ids`, which was ignored.
  Fix changes behavior, but should improve.
- `gpt_neox`: Selection of attention compute type via class removed
- `gptj`, `codegen`: RoPE must be applied per head, and some shape issues.
  Probably changes behavior.
- `nemotron`: `config.partial_rotary_factor` was not implemented. This is
  why default changed to 1, so that behavior in default case does not change.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #35233. This is the first of two PRs providing the fix. I split it for easier reviewing.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
      https://github.com/huggingface/transformers/issues/35233
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?
      The second PR fixes the bug the issue is about, and I am adding new tests to confirm it is fixed.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @Rocketknight1 as you were on the original issue; please ping Arthur if you want to confirm the direction/for the very final review

Ping?

I am also fixing bugs, certainly with the second half.
In general, I’d recommend to use one solution for RoPE everywhere, the one from gptx-neo. But that is a major refactoring 

Maybe it is better if add my second PR to this one? Then, there'd only be a single one, which fixes the original issue, and only one has to be reviewed. It'd be slightly more complex, though. In particular, I added tests.

Hello @ArthurZucker : This is what I can do:

- Put in the fixes that resolve the issue, plus extra tests
- Remove anything that changes behavior, except that bugs are fixed. So, if a model does not support RoPE, but it has corresponding arguments in its config, then I'd remove the args from the config (so users do not set them and expect RoPE to be there!), instead of adding RoPE to the model

So, all changes in behavior will be bug fixes.

The reason you may not have seen these bugs, is because they occur under unusual conditions, for example `head_size` being odd, or (more likely) `int(partial_rotary_factor * head_size)` being odd -- this is what I ran into.

This is not forbidden, and there is also no reason to forbid it, so let us just make sure the code does not break when this happ[ens.

Sounds very good thanks for your understanding! 🤗 

@ArthurZucker I am running into some problems. There are a number of models (for example, `phi`) where RoPE is in some kind of limbo state:

- `*Config` has field `partial_rotary_factor`, and `*Attention` in `*_modeling.py` is implemented to support this (so, to support `partial_rotary_factor < 1`)
- This means that `*Attention` and `*RotaryEmbeddings` should actually be copied from `gpt_neox`, and not from `llama`, because `llama` does not support `partial_rotary_factor`
- BUT: `*_modular.py` inherits from the Llama model, and not from the GPTNeoX model

So, either `partial_rotary_factor` is supported and we copy and inherit from GPTNeoX, or it is not supported and we use Llama, but then it should not appear in the `*Config`.

 I have opened a new PR #35901, which does all fixes. I am closing this one here instead, and tag you on the new one.