hubert-base-libri-demo-feature_extractor_not_frozen_v4_25epochs_weight_decay

This model is a fine-tuned version of facebook/hubert-base-ls960 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1189
• Wer: 0.1105

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00015
• train_batch_size: 64
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 3000
• num_epochs: 25
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
4.3661	1.12	500	3.4415	0.9837
2.8763	2.24	1000	3.1292	0.9837
1.6666	3.36	1500	0.5027	0.4804
0.4759	4.48	2000	0.2187	0.2536
0.2774	5.61	2500	0.1555	0.1898
0.2026	6.73	3000	0.1297	0.1543
0.1745	7.85	3500	0.1201	0.1419
0.1596	8.97	4000	0.1220	0.1339
0.1449	10.09	4500	0.1156	0.1280
0.1134	11.21	5000	0.1131	0.1244
0.1143	12.33	5500	0.1189	0.1226
0.0915	13.45	6000	0.1138	0.1196
0.0904	14.57	6500	0.1125	0.1195
0.0853	15.7	7000	0.1125	0.1168
0.0775	16.82	7500	0.1103	0.1155
0.0732	17.94	8000	0.1115	0.1138
0.0728	19.06	8500	0.1196	0.1142
0.0755	20.18	9000	0.1170	0.1122
0.0647	21.3	9500	0.1167	0.1117
0.064	22.42	10000	0.1177	0.1109
0.0591	23.54	10500	0.1182	0.1110
0.0566	24.66	11000	0.1189	0.1105

Framework versions

• Transformers 4.30.0.dev0
• Pytorch 2.0.1
• Datasets 2.12.1.dev0
• Tokenizers 0.13.3