Huggingface wav2vec2-Conformer conversion script fails for Fairseq models
Author: Getmany1
Pull request: no
Labels: name: Audio

### System Info

- `transformers` version: 4.57.0.dev0
- Platform: Linux-4.18.0-553.74.1.el8_10.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- Huggingface_hub version: 1.0.0.rc2
- Safetensors version: 0.6.2
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.5.1+cu124 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: No
- GPU type: Tesla V100-SXM2-32GB

### Who can help?

@eustlb @amyeroberts 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The official conversion script (`transformers/models/wav2vec2_conformer/convert_wav2vec2_conformer_original_pytorch_checkpoint_to_pytorch.py`) fails to properly convert Fairseq wav2vec2-Conformer models. Note that standard Wav2Vec2 models with the Transformer encoder convert successfully - this issue is specific to models with the Conformer encoder.

**Test Model:** [GetmanY1/wav2vec2-conformer-rel-pos-base-fi-ft](https://huggingface.co/GetmanY1/wav2vec2-conformer-rel-pos-base-fi-ft) (includes both original Fairseq checkpoint and converted model for reproducibility)

**Fairseq Model Configuration:**
```yaml
model:
  _name: wav2vec2
  quantize_targets: true
  latent_temp: [2.0, 0.5, 0.999995]
  extractor_mode: default
  layer_norm_first: false
  final_dim: 256
  dropout_input: 0.1
  dropout_features: 0.1
  feature_grad_mult: 0.1
  encoder_embed_dim: 768
  encoder_layers: 12
  encoder_ffn_embed_dim: 3072
  encoder_attention_heads: 12
  activation_fn: gelu
  dropout: 0.1
  attention_dropout: 0.1
  activation_dropout: 0.0
  encoder_layerdrop: 0.0
  layer_type: conformer
  attn_type: espnet
  pos_enc_type: rel_pos
```

**Steps to reproduce the behavior**

1. Run the conversion script:

```bash
python3 transformers/models/wav2vec2_conformer/convert_wav2vec2_conformer_original_pytorch_checkpoint_to_pytorch.py \
--pytorch_dump_folder_path="/converted" \
--checkpoint_path="/path/to/checkpoint_best.pt" \
--dict_path="/path/to/dict.ltr.txt"
```

2. **Error 1:**` AttributeError: 'ParametrizedConv1d' object has no attribute 'weight_g'`
**Fix:** Update lines 99 and 101 in `convert_wav2vec2_conformer_original_pytorch_checkpoint_to_pytorch.py`:

```python
# Line 99: Change
hf_pointer.weight_g.data = value
# To:
hf_pointer.parametrizations.weight.original0.data = value
   
# Line 101: Change
hf_pointer.weight_v.data = value
# To:
hf_pointer.parametrizations.weight.original1.data = value
```

(Related: [Issue #26796](https://github.com/huggingface/transformers/issues/26796), which probably wasn't fixed in this particular conversion script. )

3. **Error 2:** Shape mismatch for pos_conv weights:
`ValueError: Shape of hf wav2vec2_conformer.encoder.pos_conv_embed.conv.weight_g is torch.Size([768, 48, 128]), but should be torch.Size([1, 1, 128]) for w2v_encoder.w2v_model.encoder.pos_conv.0.weight_g`

**Temporary workaround:** Comment out shape validation (lines 90-94) in `convert_wav2vec2_conformer_original_pytorch_checkpoint_to_pytorch.py`:
```python
    if hf_shape != value.shape:
        raise ValueError(
            f"Shape of hf {key + '.' + weight_type if weight_type is not None else ''} is {hf_shape}, but should be"
            f" {value.shape} for {full_name}"
        )
```
which results in the script running without errors, but the saved converted model doesn't work correctly and only outputs blanks.

---

**P.S. Common Fairseq Setup Issues:**

If you encounter:
`omegaconf.errors.ConfigKeyError: Key 'multi_corpus_keys' not in 'AudioFinetuningConfig'`
**Fix:** Install the latest Fairseq from source: `pip install --no-build-isolation git+https://github.com/pytorch/fairseq`

If you get:
`TypeError: ConformerEncoder.build_encoder_layer() got an unexpected keyword argument 'layer_idx'`
**Fix:** Add `**kwargs` to two methods in `fairseq/models/wav2vec/wav2vec2.py`:
```python
# Line 1182: Change
def build_encoder_layer(self, args):
# To:
def build_encoder_layer(self, args, **kwargs):

# Line 1225: Change
def extract_features(self, x, padding_mask=None, tgt_layer=None):
# To:
def extract_features(self, x, padding_mask=None, tgt_layer=None, **kwargs):
```
(Fix was proposed in https://github.com/facebookresearch/fairseq/issues/5386 , but still not merged with the main branch)

### Expected behavior

The converted HuggingFace model should produce outputs matching the Fairseq model.

The original Fairseq model runs correctly. Here's a minimal example:

```python
import torch
import fairseq
from datasets import load_dataset

# Paths (update these)
MODEL_PATH = "path/to/checkpoint_best.pt"
DICT_PATH = "path/to/dict.ltr.txt"

def load_dict(path):
    vocab = {0: '
', 1: '<pad>', 2: '
', 3: '<unk>'}
    with open(path) as f:
        for line in f:
            token = line.strip().split()[0]
            vocab[len(vocab)] = token
    return vocab

def decode_ctc(logits, vocab):
    preds = logits.argmax(dim=-1)[0]
    blank_idx = logits.shape[-1] - 1
    
    tokens = []
    prev = None
    for p in preds:
        p = p.item()
        if p != blank_idx and p != prev and p >= 4:
            tokens.append(vocab[p])
        prev = p
    return ''.join(tokens).replace('|', ' ').strip()

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

dataset = load_dataset("google/fleurs", "fi_fi", split="test", streaming=True)
sample = next(iter(dataset))
audio = sample["audio"]["array"]
print(f"Reference: {sample['transcription']}")

models, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([MODEL_PATH])
model = models[0].eval().to(device)
audio_tensor = torch.from_numpy(audio).float().unsqueeze(0).to(device)
with torch.no_grad():
    out = model(source=audio_tensor, padding_mask=None)
    logits = out["encoder_out"].transpose(0, 1) if isinstance(out, dict) else out[0]

vocab = load_dict(DICT_PATH)
transcript = decode_ctc(logits, vocab)
print(f"Fairseq transcription: {transcript}")
```

I also tried to convert a wav2vec2-conformer-base with rotary positional embeddings instead of relative embeddings, but the converted model still works incorrectly, yielding>80% WER.

### **Impact**

This issue affects anyone trying to convert Fairseq wav2vec2-Conformer models to HuggingFace format, making it difficult to share these models with the broader research and developer community through HuggingFace Hub.

---

I've tried to include all necessary information to verify and reproduce the issue. Please let me know if you need any additional details


Comments:
cc @eustlb @ebezzam

@Getmany1 thanks for opening the issue and the detailed comments!

_A few notes on conversion scripts first_: these scripts aren’t considered part of the Transformers library itself. They’re mainly used for converting/pushing checkpoints when a model is added, and aren’t actively maintained or tested afterwards.

The errors you’re seeing are not necessarily bugs in Transformers, they come from changes in PyTorch conventions over time and in this case with weight norm. With newer model additions, we prefer having a model definition without weight norm for less operations at inference. Which means that during conversion we:
1. Add weight norm
2. Convert weights
3. Remove weight norm

[Here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/xcodec/convert_xcodec_weights_to_hf.py#L233-L247) is an example.

That said, below are some thoughts and potential fixes without having to remove weight norm from modeling.

## Error1 `AttributeError: 'ParametrizedConv1d' object has no attribute 'weight_g'`

This is related to old vs new weight norm conventions in PyTorch:

Older models used `torch.nn.utils.weight_norm()`, which injects `weight_g` and `weight_v` parameters (as you've commented):
```python
hf_pointer.weight_g.data = value
hf_pointer.weight_v.data = value
````

Newer models use `torch.nn.utils.parametrizations.weight_norm()`, which injects `parametrizations.weight.original0/original1` as you did in your replacement :
```python
hf_pointer.parametrizations.weight.original0.data = value
hf_pointer.parametrizations.weight.original1.data = value
```

**Issue**: The original model uses the old convention [here](https://github.com/facebookresearch/fairseq/blob/fad2c4d1ebe14d974876de52dcb06db6d99b0b4a/fairseq/models/wav2vec/wav2vec2.py#L939) but the HF model used in the conversion script assumes the new convention, which was later added [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2_conformer/modeling_wav2vec2_conformer.py#L91-L109), hence the parameter naming mismatch.

**Potential fix**: Add a `legacy_weight_norm` flag to the script so users can choose old/new convention:
```python
elif weight_type == "weight_g":
    if legacy_weight_norm:
        hf_pointer.weight_g.data = value
    else:
        hf_pointer.parametrizations.weight.original0.data = value
elif weight_type == "weight_v":
    if legacy_weight_norm:
        hf_pointer.weight_v.data = value
    else:
        hf_pointer.parametrizations.weight.original1.data = value
```
It would be good to make the flag backward-compatible (namely defaulting to True).

## Error 2: Shape mismatch for pos_conv weights

Looking at your error:
```
ValueError: Shape of hf wav2vec2_conformer.encoder.pos_conv_embed.conv.weight_g is torch.Size([768, 48, 128]), but should be torch.Size([1, 1, 128]) for w2v_encoder.w2v_model.encoder.pos_conv.0.weight_g
```

While commenting out:
```python
    if hf_shape != value.shape:
        raise ValueError(
            f"Shape of hf {key + '.' + weight_type if weight_type is not None else ''} is {hf_shape}, but should be"
            f" {value.shape} for {full_name}"
        )
```
may let the script run, the conversion script may not be properly mapping the weights to the HF model, which would explain why the saved converted model doesn't work correctly. 

For the HF model, `pos_conv_embed` (again) has to do with the [`Wav2Vec2ConformerPositionalConvEmbedding`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2_conformer/modeling_wav2vec2_conformer.py#L646) module defined [here](https://github.com/huggingface/transformers/blob/2b5e4c0d13ffec43fea89523ac286409654eb9fe/src/transformers/models/wav2vec2_conformer/modeling_wav2vec2_conformer.py#L80). 

For the original model, `pos_conv` seems to be defined around [here](https://github.com/facebookresearch/fairseq/blob/fad2c4d1ebe14d974876de52dcb06db6d99b0b4a/fairseq/models/wav2vec/wav2vec2.py#L1045-L1050) in the original model. They apply weight norm in [this case](https://github.com/facebookresearch/fairseq/blob/fad2c4d1ebe14d974876de52dcb06db6d99b0b4a/fairseq/models/wav2vec/wav2vec2.py#L939).

And again the mismatch is because the original model is using the old convention `nn.utils.weight_norm`, while the HF version (likely) uses the newer `nn.utils.parametrizations.weight_norm` given Error1 you were seeing.

In other words, the HF version seems to expect a shape of torch.Size([768, 48, 128]) because it uses `nn.utils.parametrizations.weight_norm` and the original model has a tensor shape torch.Size([1, 1, 128]) because it uses `nn.utils.weight_norm`. 

One way to resolve this (with backward compatibility) would be to propagate the `legacy_weight_norm` parameter to the modeling so that `nn.utils.weight_norm` can be used in the HF modeling like in the original model, even if `hasattr(nn.utils.parametrizations, "weight_norm")`.

Or (an idea from ChatGPT so to double-check) you may be able to expand the tensor from the original checkpoint:
```python
# value = old checkpoint weight_g
out_ch, in_ch_per_group, k = hf_pointer.parametrizations.weight.original0.shape

# broadcast old weight_g to full shape
hf_pointer.parametrizations.weight.original0.data = value.expand(out_ch, in_ch_per_group, k).clone()

# copy weight_v directly
hf_pointer.parametrizations.weight.original1.data = old_weight_v.clone()
```

# Summing up

I think both errors are due to weight norm conventions. Maybe there is another issue with `layer_idx` in which case you would have to modify the modeling code. But the error messages you're getting don't seem to indicate that.

For testing, you could try [commenting out the new weight norm lines](https://github.com/huggingface/transformers/blob/2b5e4c0d13ffec43fea89523ac286409654eb9fe/src/transformers/models/wav2vec2_conformer/modeling_wav2vec2_conformer.py#L92C12-L93) in the HF modeling code to see if the conversion script works without your other fixes.

Hope that helps, and happy to support if you start a PR!

@ebezzam Thank you very much for the detailed analysis!

I followed your suggestion to comment out the new weight norm lines in the HuggingFace modeling code, and while this allowed the conversion script to run without errors or need for other fixes, the resulting converted model still produces incorrect outputs that diverge significantly from the original Fairseq checkpoint.

To rule out issues with my own checkpoint, I also tested with Meta's official wav2vec2-conformer-large checkpoint from the Fairseq repository:
`https://dl.fbaipublicfiles.com/fairseq/conformer/wav2vec2/librilight/LL_relpos_PT_960h_FT.pt`

Unfortunately, the conversion failed for this checkpoint as well: the converted model outputs nonsensical predictions.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The issue is still relevant, please keep it open.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.