WavLM Model is not loaded properly
Author: MordehayM
Pull request: no
Labels: name: bug

### System Info

I dont have this commad

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just install the package (without specific version) for wavlm model and running this code:
from transformers import AutoProcessor, WavLMModel, AutoModel
import torch

model = AutoModel.from_pretrained("microsoft/wavlm-large")

I got the following warning, which should not be shown:
Some weights of the model checkpoint at microsoft/wavlm-large were not used when initializing WavLMModel: ['encoder.pos_conv_embed.conv.weight_g', 'encoder.pos_conv_embed.conv.weight_v']
- This IS expected if you are initializing WavLMModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing WavLMModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of WavLMModel were not initialized from the model checkpoint at microsoft/wavlm-large and are newly initialized: ['encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'encoder.pos_conv_embed.conv.parametrizations.weight.original1']


### Expected behavior

I should get the loaded model without this warning.
I suspect the warning comes from this line:
https://github.com/huggingface/transformers/blob/7e5d46ded433605a906fcab6be43ac85307cca9b/src/transformers/models/wavlm/modeling_wavlm.py#L282

Since the weights are from old version packages so it's familiar with this parameterizations module 

Comments:
Hi @MordehayM , thanks for the issue! looks like `weight_g` and  `weight_v` are only used in case you're using deepspeed ZeRO-3. I think what's happening is that the checkpoint contains these externally registered parameters, but in your environment you might not have deepspeed ZeRO-3 enabled. Can you confirm? If you have it enabled, that's unexpected behaviour.

You are correct but it still should not do that since when installing transformers, it should be installed too.
So the transformers package has to install this package.

You are correct but it still should not do that since when installing transformers, it should be installed too.
So the transformers package has to install this package.

Those parameters are from the old [weight norm](https://pytorch.org/docs/1.13/generated/torch.nn.utils.weight_norm.html?highlight=weight+norm#torch.nn.utils.weight_norm). Seems like in the checkpoint `weight_g` and `weight_v` should be renamed to `original0` and `original1` based on the new naming scheme in [weight norm](https://pytorch.org/docs/2.4/generated/torch.nn.utils.parametrizations.weight_norm.html#torch.nn.utils.parametrizations.weight_norm)

How can I enable the deepspeed?

Hi again @MordehayM  
From what I see: 
- either you have deepspeed enabled, which I recommend looking up online resources around it, it's disconnected from transformers, once enabled loading will work normally
- otherwise you can as suggested by @will-rice  rename the `state_dict` keys to match yours, and it should behave as expected

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I made a fixed [model](https://huggingface.co/wrice/wavlm-large-weight-norm-fix) if you still need it. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.