---
license: mit
tags:
- generated_from_trainer
base_model: facebook/w2v-bert-2.0
datasets:
- generator
metrics:
- wer
model-index:
- name: wav2vec2-bert-fon
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: generator
      type: generator
      config: default
      split: train
      args: default
    metrics:
    - type: wer
      value: 0.13241653693132677
      name: Wer
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-bert-fon

This model is a fine-tuned version of [facebook/w2v-bert-2.0](https://huggingface.co/facebook/w2v-bert-2.0) on the generator dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1612
- Wer: 0.1324

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 4
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| No log        | 0.18  | 250  | 1.2212          | 0.8079 |
| 2.1756        | 0.35  | 500  | 0.6697          | 0.6058 |
| 2.1756        | 0.53  | 750  | 0.5137          | 0.4606 |
| 0.5041        | 0.7   | 1000 | 0.4337          | 0.4234 |
| 0.5041        | 0.88  | 1250 | 0.3452          | 0.3529 |
| 0.426         | 1.05  | 1500 | 0.2770          | 0.2910 |
| 0.426         | 1.23  | 1750 | 0.2681          | 0.2439 |
| 0.2916        | 1.4   | 2000 | 0.2423          | 0.2155 |
| 0.2916        | 1.58  | 2250 | 0.2342          | 0.2077 |
| 0.2591        | 1.75  | 2500 | 0.1986          | 0.1791 |
| 0.2591        | 1.93  | 2750 | 0.1864          | 0.1597 |
| 0.2261        | 2.1   | 3000 | 0.1712          | 0.1419 |
| 0.2261        | 2.28  | 3250 | 0.1786          | 0.1497 |
| 0.1564        | 2.45  | 3500 | 0.1612          | 0.1324 |
| 0.1564        | 2.63  | 3750 | 0.1730          | 0.1591 |
| 0.1542        | 2.8   | 4000 | 0.1558          | 0.1364 |
| 0.1542        | 2.98  | 4250 | 0.1493          | 0.1581 |
| 0.1559        | 3.15  | 4500 | 0.1489          | 0.1347 |
| 0.1559        | 3.33  | 4750 | 0.2036          | 0.1486 |
| 0.1992        | 3.5   | 5000 | 0.2644          | 0.1582 |
| 0.1992        | 3.68  | 5250 | 0.2401          | 0.1878 |
| 0.291         | 3.85  | 5500 | 0.2409          | 0.1749 |


### Framework versions

- Transformers 4.39.3
- Pytorch 2.1.2
- Datasets 2.18.0
- Tokenizers 0.15.2