Update modeling_llama4.py
Author: alfredo-etched
Pull request: yes

Something is off. The code shows Llama4TextL2Norm function name, but it computes RMS Norm. Which one is the correct one for LLAMA4? RMS or L2? If so, the function should be renamed (if RMS) or modified (if L2) accordingly.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @zach-huggingface @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker @S1ro1
- CIs: @ydshieh

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
Good point, this function is probably wrong! The main reason it wasn't visible as a bug is that `Llama4TextL2Norm` is only used under specific conditions:

```python
  if self.config.use_qk_norm and self.use_rope:
      self.qk_norm = Llama4TextL2Norm(config.rms_norm_eps)
```

I believe `use_qk_norm` is `False` for the released Llama-4 checkpoints, and so this code is never called unless a user makes their own Llama-4 model using `qk_norm`. Still, might be worth fixing! cc @arthurzucker do you know if this was in the original codebase?

@Rocketknight1 on the config.json for LLAMA4 Scout, use_qk_norm is set as true. But then if you look at the TextAttention function, it also requires that use_rope is true (line 310). But then use_rope is always true, because in config.json the noropelayers is an empty list. Therefore, qk_norm is always computed for llama 4 scout. My question though is: is the norm supposed to be actual L2 norm or RMS norm?

Hi @alfredo-etched, I didn't realize that `scout` uses that code path - I only checked `maverick`, my bad! 

If `scout` gives good output, though, then the code here is probably correct, just because of the large output difference we would have noticed if we'd swapped one norm for a different one. Therefore, I think the problem is just that the "L2Norm" class is mis-named. The difference is small, though - if they used `.sum(-1)` instead of `.mean(-1)` I believe this would be equivalent to L2 normalization.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llama4

> Hi @alfredo-etched, I didn't realize that `scout` uses that code path - I only checked `maverick`, my bad!
> 
> If `scout` gives good output, though, then the code here is probably correct, just because of the large output difference we would have noticed if we'd swapped one norm for a different one. Therefore, I think the problem is just that the "L2Norm" class is mis-named. The difference is small, though - if they used `.sum(-1)` instead of `.mean(-1)` I believe this would be equivalent to L2 normalization.

I hoped I could get a confirmation of which normalization is the correct one from someone at Meta who helped writing this code, rather than some guesswork. Exactly because the difference is small, it might be that the models behave ALMOST the same, but one is correct and the other one is not, and it might be complex to ascertain it by testing the model for correctness of inference.

Hi @alfredo-etched, although the difference is small in terms of code changes, it's very large numerically! The difference between ``sqrt(sum())` and `sqrt(mean())` is a factor of `sqrt(dim)` in the output, and I'm pretty sure the model would break completely if it was trained with one and inferenced with the other. 

Feel free to test it, though - change `L2Norm` to be a true L2 Norm and see what effect that has on `scout`'s output