layernorm_decay_fix
Author: Ryoo72
Pull request: yes

# What does this PR do?
This PR improves the weight decay parameter filtering mechanism to properly handle all normalization layers, especially RMSNorm variants. 

## Background and Motivation
Normalization layers (like LayerNorm and RMSNorm) should not have weight decay applied to their parameters. (See [discussion](https://github.com/karpathy/minGPT/pull/24#issuecomment-679316025))

The current implementation filters out instances of `nn.LayerNorm` through type checking. However, this approach has limitations:
1. Many recent models use RMSNorm instead of LayerNorm
2. These models often implement their own RMSNorm from scratch instead of using `nn.RMSNorm`
   - Example: [LLaMA's RMSNorm implementation](https://github.com/meta-llama/llama/blob/689c7f261b9c5514636ecc3c5fefefcbb3e6eed7/llama/model.py#L34)
   - Example: [Qwen2's RMSNorm implementation](https://github.com/huggingface/transformers/blob/14a9bb520e50723b7dbd0663521975d7174f7790/src/transformers/models/qwen2/modeling_qwen2.py#L209)

Simply adding `nn.RMSNorm` to `ALL_LAYERNORM_LAYERS` would not catch these custom implementations. Therefore, this PR adds name-based filtering alongside the existing type-based filtering.

## Changes
1. Added name-based parameter filtering to catch all normalization layers regardless of their implementation
2. Updated documentation to reflect the enhanced filtering approach
3. Made the filtering more robust by adding RMSNorm to excluded patterns

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Agree that all normalization layers should not have weight decay applied to their parameters, so this is a good change, but I'll need a review from @SunMarc @muellerzr for the trainer changes!

please also fix the CI with `make style`

> please also fix the CI with `make style`

Thank you for the review comments!

I ran
```
ruff format --check src/transformers/trainer_pt_utils.py
```
and confirmed it shows `1 file already formatted`.

I'll add the additional test for RMSNorm right away!

> The changes looks good to me. Can you add an additional test to check if rmsnorm is correctly removed, thanks !

Thank you for the review comments. I added test code to verify RMSNorm parameters are correctly excluded:

```
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.trainer_pt_utils import get_parameter_names
from transformers.trainer import ALL_LAYERNORM_LAYERS

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-7B-Instruct", device_map="auto")
forbidden_layer_names = ["bias", "layernorm", "rmsnorm"]

decay_parameters = get_parameter_names(model, ALL_LAYERNORM_LAYERS, forbidden_layer_names)

all_params = [name for name, _ in model.named_parameters()]

no_decay = set(all_params) - set(decay_parameters)
```

I confirmed that RMSNorm parameters are correctly filtered out to no_decay set. Here are some examples from no_decay:
`'model.layers.17.input_layernorm.weight', 'model.layers.24.input_layernorm.weight', 'model.layers.21.self_attn.k_proj.bias', ...`

These are the layers that were not filtered out in the previous code.

Nice ! Could you add a test in the PR with a smaller model like `hf-internal-testing/tiny-random-LlamaForCausalLM` or create a dummy models with rmsnorm layers and linear layers ? Thanks ! 

there is still an issue with the formatting, the ci is still red 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35927). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Nice ! Could you add a test in the PR with a smaller model like `hf-internal-testing/tiny-random-LlamaForCausalLM` or create a dummy models with rmsnorm layers and linear layers ? Thanks !

Sorry for my confusion earlier, and thank you for your patience and clear guidance!

I've added a test using `test_get_parameter_names_rmsnorm` in `tests/trainer/test_trainer_utils.py`. The test verifies that both RMSNorm parameters and bias terms are correctly filtered out from weight decay parameters :)