File size: 19,091 Bytes
cc426f2
8190f9a
 
cc426f2
 
 
8190f9a
cc426f2
 
 
 
8190f9a
cc426f2
 
 
 
 
 
8190f9a
cc426f2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
---
language:
- he
license: apache-2.0
base_model: openai/whisper-small
tags:
- hf-asr-leaderboard
- generated_from_trainer
metrics:
- wer
model-index:
- name: he-cantillation
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# he-cantillation

This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1496
- Wer: 9.8776
- Avg Precision Exact: 0.9087
- Avg Recall Exact: 0.9107
- Avg F1 Exact: 0.9093
- Avg Precision Letter Shift: 0.9225
- Avg Recall Letter Shift: 0.9246
- Avg F1 Letter Shift: 0.9232
- Avg Precision Word Level: 0.9249
- Avg Recall Word Level: 0.9267
- Avg F1 Word Level: 0.9254
- Avg Precision Word Shift: 0.9697
- Avg Recall Word Shift: 0.9724
- Avg F1 Word Shift: 0.9706
- Precision Median Exact: 1.0
- Recall Median Exact: 1.0
- F1 Median Exact: 1.0
- Precision Max Exact: 1.0
- Recall Max Exact: 1.0
- F1 Max Exact: 1.0
- Precision Min Exact: 0.0
- Recall Min Exact: 0.0
- F1 Min Exact: 0.0
- Precision Min Letter Shift: 0.0
- Recall Min Letter Shift: 0.0
- F1 Min Letter Shift: 0.0
- Precision Min Word Level: 0.0
- Recall Min Word Level: 0.0
- F1 Min Word Level: 0.0
- Precision Min Word Shift: 0.1429
- Recall Min Word Shift: 0.1
- F1 Min Word Shift: 0.1176

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- training_steps: 200000
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Wer      | Avg Precision Exact | Avg Recall Exact | Avg F1 Exact | Avg Precision Letter Shift | Avg Recall Letter Shift | Avg F1 Letter Shift | Avg Precision Word Level | Avg Recall Word Level | Avg F1 Word Level | Avg Precision Word Shift | Avg Recall Word Shift | Avg F1 Word Shift | Precision Median Exact | Recall Median Exact | F1 Median Exact | Precision Max Exact | Recall Max Exact | F1 Max Exact | Precision Min Exact | Recall Min Exact | F1 Min Exact | Precision Min Letter Shift | Recall Min Letter Shift | F1 Min Letter Shift | Precision Min Word Level | Recall Min Word Level | F1 Min Word Level | Precision Min Word Shift | Recall Min Word Shift | F1 Min Word Shift |
|:-------------:|:-------:|:------:|:---------------:|:--------:|:-------------------:|:----------------:|:------------:|:--------------------------:|:-----------------------:|:-------------------:|:------------------------:|:---------------------:|:-----------------:|:------------------------:|:---------------------:|:-----------------:|:----------------------:|:-------------------:|:---------------:|:-------------------:|:----------------:|:------------:|:-------------------:|:----------------:|:------------:|:--------------------------:|:-----------------------:|:-------------------:|:------------------------:|:---------------------:|:-----------------:|:------------------------:|:---------------------:|:-----------------:|
| No log        | 0.0001  | 1      | 7.3549          | 108.0965 | 0.0006              | 0.0037           | 0.0009       | 0.0204                     | 0.0217                  | 0.0204              | 0.0078                   | 0.0760                | 0.0140            | 0.1364                   | 0.1552                | 0.1388            | 0.0                    | 0.0                 | 0.0             | 0.2                 | 1.0              | 0.3077       | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.0                      | 0.0                   | 0.0               |
| 0.0527        | 0.5167  | 10000  | 0.1247          | 17.8011  | 0.8390              | 0.8441           | 0.8410       | 0.8596                     | 0.8648                  | 0.8616              | 0.8632                   | 0.8682                | 0.8651            | 0.9344                   | 0.9409                | 0.9369            | 0.9167                 | 0.9167              | 0.9167          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.0769                   | 0.0667                | 0.0714            |
| 0.0181        | 1.0334  | 20000  | 0.1178          | 14.7550  | 0.8697              | 0.8697           | 0.8690       | 0.8884                     | 0.8887                  | 0.8878              | 0.8911                   | 0.8918                | 0.8907            | 0.9502                   | 0.9517                | 0.9501            | 0.9333                 | 0.9333              | 0.9474          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.0                      | 0.0                   | 0.0               |
| 0.0121        | 1.5501  | 30000  | 0.1202          | 13.1219  | 0.8866              | 0.8899           | 0.8878       | 0.9039                     | 0.9074                  | 0.9052              | 0.9067                   | 0.9103                | 0.9080            | 0.9551                   | 0.9595                | 0.9568            | 1.0                    | 1.0                 | 0.9630          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.125                 | 0.1333            |
| 0.0041        | 2.0668  | 40000  | 0.1301          | 12.2219  | 0.8897              | 0.8925           | 0.8907       | 0.9058                     | 0.9087                  | 0.9068              | 0.9089                   | 0.9114                | 0.9097            | 0.9579                   | 0.9612                | 0.9591            | 1.0                    | 1.0                 | 0.9655          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.125                    | 0.1                   | 0.1176            |
| 0.0053        | 2.5834  | 50000  | 0.1294          | 12.3635  | 0.8863              | 0.8873           | 0.8864       | 0.9030                     | 0.9042                  | 0.9031              | 0.9055                   | 0.9068                | 0.9057            | 0.9594                   | 0.9623                | 0.9603            | 1.0                    | 1.0                 | 0.9630          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.0909                   | 0.1                   | 0.0952            |
| 0.01          | 3.1001  | 60000  | 0.1385          | 12.5586  | 0.8824              | 0.8860           | 0.8838       | 0.8984                     | 0.9023                  | 0.8999              | 0.9010                   | 0.9049                | 0.9025            | 0.9557                   | 0.9609                | 0.9578            | 1.0                    | 1.0                 | 0.9630          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0037        | 3.6168  | 70000  | 0.1374          | 12.4076  | 0.8905              | 0.8936           | 0.8916       | 0.9062                     | 0.9093                  | 0.9073              | 0.9085                   | 0.9115                | 0.9095            | 0.9561                   | 0.9598                | 0.9574            | 1.0                    | 1.0                 | 0.9630          | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1111                | 0.125             |
| 0.0031        | 4.1335  | 80000  | 0.1370          | 11.7562  | 0.8959              | 0.8971           | 0.8961       | 0.9115                     | 0.9128                  | 0.9117              | 0.9137                   | 0.9150                | 0.9139            | 0.9610                   | 0.9639                | 0.9620            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.0                      | 0.0                   | 0.0               |
| 0.0012        | 4.6502  | 90000  | 0.1371          | 11.4573  | 0.8972              | 0.8963           | 0.8964       | 0.9126                     | 0.9119                  | 0.9118              | 0.9150                   | 0.9146                | 0.9144            | 0.9655                   | 0.9665                | 0.9655            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0021        | 5.1669  | 100000 | 0.1448          | 11.6492  | 0.8926              | 0.8952           | 0.8935       | 0.9075                     | 0.9103                  | 0.9085              | 0.9102                   | 0.9128                | 0.9111            | 0.9601                   | 0.9637                | 0.9614            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1                      | 0.1                   | 0.1053            |
| 0.0013        | 5.6836  | 110000 | 0.1440          | 11.0199  | 0.9003              | 0.9033           | 0.9014       | 0.9156                     | 0.9188                  | 0.9168              | 0.9179                   | 0.9211                | 0.9191            | 0.9645                   | 0.9685                | 0.9660            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0004        | 6.2003  | 120000 | 0.1427          | 10.7901  | 0.9033              | 0.9043           | 0.9035       | 0.9183                     | 0.9194                  | 0.9185              | 0.9205                   | 0.9215                | 0.9206            | 0.9667                   | 0.9686                | 0.9672            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0005        | 6.7170  | 130000 | 0.1434          | 10.7178  | 0.9012              | 0.9039           | 0.9021       | 0.9161                     | 0.9189                  | 0.9171              | 0.9185                   | 0.9212                | 0.9194            | 0.9648                   | 0.9684                | 0.9662            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1111                | 0.125             |
| 0.0006        | 7.2336  | 140000 | 0.1452          | 10.7776  | 0.9014              | 0.9041           | 0.9023       | 0.9162                     | 0.9190                  | 0.9172              | 0.9187                   | 0.9211                | 0.9195            | 0.9661                   | 0.9699                | 0.9676            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1111                | 0.125             |
| 0.0005        | 7.7503  | 150000 | 0.1471          | 10.6297  | 0.9037              | 0.9039           | 0.9034       | 0.9177                     | 0.9180                  | 0.9175              | 0.9198                   | 0.9199                | 0.9195            | 0.9656                   | 0.9673                | 0.9660            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0001        | 8.2670  | 160000 | 0.1484          | 10.3905  | 0.9035              | 0.9068           | 0.9048       | 0.9177                     | 0.9212                  | 0.9191              | 0.9200                   | 0.9231                | 0.9212            | 0.9667                   | 0.9703                | 0.9680            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0019        | 8.7837  | 170000 | 0.1468          | 10.1136  | 0.9078              | 0.9099           | 0.9085       | 0.9220                     | 0.9242                  | 0.9227              | 0.9244                   | 0.9262                | 0.9249            | 0.9679                   | 0.9704                | 0.9687            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0           | 9.3004  | 180000 | 0.1495          | 10.1167  | 0.9051              | 0.9073           | 0.9058       | 0.9194                     | 0.9217                  | 0.9202              | 0.9219                   | 0.9241                | 0.9226            | 0.9681                   | 0.9709                | 0.9691            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0           | 9.8171  | 190000 | 0.1497          | 10.0412  | 0.9078              | 0.9104           | 0.9087       | 0.9218                     | 0.9245                  | 0.9227              | 0.9243                   | 0.9266                | 0.9251            | 0.9680                   | 0.9713                | 0.9692            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |
| 0.0           | 10.3338 | 200000 | 0.1496          | 9.8776   | 0.9087              | 0.9107           | 0.9093       | 0.9225                     | 0.9246                  | 0.9232              | 0.9249                   | 0.9267                | 0.9254            | 0.9697                   | 0.9724                | 0.9706            | 1.0                    | 1.0                 | 1.0             | 1.0                 | 1.0              | 1.0          | 0.0                 | 0.0              | 0.0          | 0.0                        | 0.0                     | 0.0                 | 0.0                      | 0.0                   | 0.0               | 0.1429                   | 0.1                   | 0.1176            |


### Framework versions

- Transformers 4.41.2
- Pytorch 2.2.1
- Datasets 2.20.0
- Tokenizers 0.19.1