pere commited on
Commit
848b408
1 Parent(s): aae3c1f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +286 -86
README.md CHANGED
@@ -1,96 +1,296 @@
1
  ---
 
2
  language:
3
  - 'no'
4
- license: apache-2.0
5
- base_model: NbAiLab/nb-whisper-small-RC1
 
 
 
 
 
 
6
  tags:
7
  - audio
8
  - asr
9
  - automatic-speech-recognition
10
  - hf-asr-leaderboard
11
- model-index:
12
- - name: nb-whisper-small-v0.8-vad3
13
- results: []
 
 
 
 
 
 
 
14
  ---
15
 
16
- <!-- This model card has been generated automatically according to the information Keras had access to. You should
17
- probably proofread and complete it, then remove this comment. -->
18
-
19
- # nb-whisper-small-v0.8-vad3
20
-
21
- This model is a fine-tuned version of [NbAiLab/nb-whisper-small-RC1](https://huggingface.co/NbAiLab/nb-whisper-small-RC1) on the NbAiLab/ncc_speech_styling_v2_vad3 dataset.
22
- It achieves the following results on the evaluation set:
23
- - step: 49999
24
- - validation_nst_loss: 0.4444
25
- - train_loss: 0.4400
26
- - validation_nst_wer: 3.0595
27
- - validation_nst_cer: 0.9443
28
- - validation_nst_exact_wer: 3.7237
29
- - validation_nst_exact_cer: 1.0431
30
- - validation_clean_stortinget_no_loss: 0.7056
31
- - validation_clean_stortinget_no_wer: 10.0663
32
- - validation_clean_stortinget_no_cer: 6.2768
33
- - validation_clean_stortinget_no_exact_wer: 13.4172
34
- - validation_clean_stortinget_no_exact_cer: 6.8042
35
-
36
- ## Model description
37
-
38
- More information needed
39
-
40
- ## Intended uses & limitations
41
-
42
- More information needed
43
-
44
- ## Training and evaluation data
45
-
46
- More information needed
47
-
48
- ## Training procedure
49
-
50
- ### Training hyperparameters
51
-
52
- The following hyperparameters were used during training:
53
- - learning_rate: 5e-05
54
- - lr_scheduler_type: linear
55
- - per_device_train_batch_size: 32
56
- - total_train_batch_size_per_node: 128
57
- - total_train_batch_size: 1024
58
- - total_optimization_steps: 50,000
59
- - starting_optimization_step: None
60
- - finishing_optimization_step: 50,000
61
- - num_train_dataset_workers: 32
62
- - num_hosts: 8
63
- - total_num_training_examples: 51,200,000
64
- - steps_per_epoch: 7455
65
- - num_beams: None
66
- - weight_decay: 0.01
67
- - adam_beta1: 0.9
68
- - adam_beta2: 0.98
69
- - adam_epsilon: 1e-06
70
- - dropout: True
71
- - bpe_dropout_probability: 0.2
72
- - activation_dropout_probability: 0.1
73
-
74
- ### Training results
75
-
76
- | step | validation_nst_loss | train_loss | validation_nst_wer | validation_nst_cer | validation_nst_exact_wer | validation_nst_exact_cer | validation_clean_stortinget_no_loss | validation_clean_stortinget_no_wer | validation_clean_stortinget_no_cer | validation_clean_stortinget_no_exact_wer | validation_clean_stortinget_no_exact_cer |
77
- |:-----:|:-------------------:|:----------:|:------------------:|:------------------:|:------------------------:|:------------------------:|:-----------------------------------:|:----------------------------------:|:----------------------------------:|:----------------------------------------:|:----------------------------------------:|
78
- | 0 | 0.4313 | 1.0396 | 2.8254 | 0.8865 | 3.5168 | 0.9900 | 0.5547 | 9.6092 | 5.9949 | 12.6794 | 6.4755 |
79
- | 5000 | 0.4484 | 0.5692 | 3.2010 | 1.0142 | 3.8870 | 1.1172 | 0.6138 | 10.1824 | 6.1896 | 13.4124 | 6.6954 |
80
- | 10000 | 0.4477 | 0.5317 | 3.3589 | 1.0347 | 4.0176 | 1.1337 | 0.6275 | 10.3316 | 6.4310 | 13.6022 | 6.9442 |
81
- | 15000 | 0.4493 | 0.5132 | 3.3099 | 1.0086 | 3.9904 | 1.1145 | 0.6599 | 10.2203 | 6.3042 | 13.4100 | 6.8175 |
82
- | 20000 | 0.4491 | 0.4911 | 3.2283 | 1.0226 | 3.8924 | 1.1227 | 0.6755 | 10.1421 | 6.3188 | 13.4409 | 6.8428 |
83
- | 25000 | 0.4441 | 0.4766 | 3.1575 | 0.9816 | 3.8924 | 1.0898 | 0.6763 | 10.2700 | 6.3383 | 13.5951 | 6.8658 |
84
- | 30000 | 0.4498 | 0.4632 | 3.1357 | 0.9741 | 3.8543 | 1.0797 | 0.6599 | 10.2274 | 6.3787 | 13.5144 | 6.8974 |
85
- | 35000 | 0.4480 | 0.4523 | 3.0432 | 0.9378 | 3.7727 | 1.0486 | 0.6948 | 10.2416 | 6.3617 | 13.5547 | 6.8912 |
86
- | 40000 | 0.4471 | 0.4606 | 3.0486 | 0.9080 | 3.7291 | 1.0101 | 0.6754 | 10.2155 | 6.3375 | 13.5097 | 6.8506 |
87
- | 45000 | 0.4442 | 0.4412 | 2.9778 | 0.9275 | 3.6366 | 1.0229 | 0.7021 | 10.1468 | 6.2994 | 13.5286 | 6.8358 |
88
- | 49999 | 0.4444 | 0.4400 | 3.0595 | 0.9443 | 3.7237 | 1.0431 |
89
- | 49999 | 0.7056 | 0.4400 | 10.0663 | 6.2768 | 13.4172 | 6.8042 |
90
-
91
-
92
- ### Framework versions
93
-
94
- - Transformers 4.34.1
95
- - Datasets 2.16.1
96
- - Tokenizers 0.14.1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: apache-2.0
3
  language:
4
  - 'no'
5
+ - nb
6
+ - nn
7
+ - en
8
+ datasets:
9
+ - NbAiLab/ncc_speech
10
+ - NbAiLab/NST
11
+ - NbAiLab/NPSC
12
+ base_model: openai/whisper-small
13
  tags:
14
  - audio
15
  - asr
16
  - automatic-speech-recognition
17
  - hf-asr-leaderboard
18
+ metrics:
19
+ - wer
20
+ - cer
21
+ library_name: transformers
22
+ pipeline_tag: automatic-speech-recognition
23
+ widget:
24
+ - src: https://datasets-server.huggingface.co/assets/google/fleurs/--/nb_no/train/1/audio/audio.mp3
25
+ example_title: FLEURS sample 1
26
+ - src: https://datasets-server.huggingface.co/assets/google/fleurs/--/nb_no/train/4/audio/audio.mp3
27
+ example_title: FLEURS sample 2
28
  ---
29
 
30
+
31
+ # NB-Whisper Small (Release Candidate)
32
+
33
+ **IMPORTANT:** These models are currently Release Candidates. We are in the final stages of testing. If everything proceeds smoothly, we plan to officially release the models later this month.
34
+
35
+ Introducing the **_Norwegian NB-Whisper Small model_**, proudly developed by the National Library of Norway. NB-Whisper is a cutting-edge series of models designed for automatic speech recognition (ASR) and speech translation. These models are based on the work of [OpenAI's Whisper](https://arxiv.org/abs/2212.04356). Each model in the series has been trained for 250,000 steps, utilizing a diverse dataset of 8 million samples. These samples consist of aligned audio clips, each 30 seconds long, culminating in a staggering 66,000 hours of speech. For an in-depth understanding of our training methodology and dataset composition, keep an eye out for our upcoming article.
36
+
37
+ | Model Size | Parameters | Model |
38
+ |------------|------------|------------|
39
+ | Tiny | 39M | [NB-Whisper Tiny](https://huggingface.co/NbAiLabBeta/nb-whisper-tiny) |
40
+ | Base | 74M | [NB-Whisper Base](https://huggingface.co/NbAiLabBeta/nb-whisper-base) |
41
+ | Small | 244M | [NB-Whisper Small](https://huggingface.co/NbAiLabBeta/nb-whisper-small) |
42
+ | Medium | 769M | [NB-Whisper Medium](https://huggingface.co/NbAiLabBeta/nb-whisper-medium) |
43
+ | Large | 1550M | [NB-Whisper Large](https://huggingface.co/NbAiLabBeta/nb-whisper-large) |
44
+
45
+
46
+
47
+ ### Specialised Models
48
+ While the main models are suitable for most transcription task, we demonstrate how easy it is to change the output of the main model. The following models are trained 250 additional steps from the main models above, and might be suitable for more targetted use cases:
49
+ - **Verbatim version**: This lower-cased variant is more literal and suitable for tasks requiring detailed transcription, such as linguistic analysis.
50
+ - **Semantic version**: This variant focuses less on verbatim accuracy but captures the essence of content, ideal for meeting minutes and subtitling.
51
+
52
+
53
+ | Model Size | Parameters | Verbatim version | Semantic version |
54
+ |------------|------------|------------|------------------|
55
+ | Tiny | 39M | [Tiny - verbatim](https://huggingface.co/NbAiLabBeta/nb-whisper-tiny-verbatim) | [Tiny - semantic](https://huggingface.co/NbAiLabBeta/nb-whisper-tiny-semantic) |
56
+ | Base | 74M | [Base - verbatim](https://huggingface.co/NbAiLabBeta/nb-whisper-base-verbatim) | [Base - semantic](https://huggingface.co/NbAiLabBeta/nb-whisper-base-semantic) |
57
+ | Small | 244M | [Small - verbatim](https://huggingface.co/NbAiLabBeta/nb-whisper-small-verbatim) | [Small - semantic](https://huggingface.co/NbAiLabBeta/nb-whisper-small-semantic) |
58
+ | Medium | 769M | [Medium - verbatim](https://huggingface.co/NbAiLabBeta/nb-whisper-medium-verbatim) | [Medium - semantic](https://huggingface.co/NbAiLabBeta/nb-whisper-medium-semantic) |
59
+ | Large | 1550M | [Large - verbatim](https://huggingface.co/NbAiLabBeta/nb-whisper-large-verbatim) | [Large - semantic](https://huggingface.co/NbAiLabBeta/nb-whisper-large-semantic) |
60
+
61
+
62
+ ### Model Description
63
+
64
+ - **Developed by:** [NB AI-Lab](https://ai.nb.no/)
65
+ - **Shared by:** [NB AI-Lab](https://ai.nb.no/)
66
+ - **Model type:** `whisper`
67
+ - **Language(s) (NLP):** Norwegian, Norwegian Bokmål, Norwegian Nynorsk, English
68
+ - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
69
+ - **Trained from model:** [openai/whisper-small](https://huggingface.co/openai/whisper-small)
70
+ - **Code Repository:** https://github.com/NbAiLab/nb-whisper/
71
+ - **Paper:** _Coming soon_
72
+ - **Demo:** _See Spaces on this page_
73
+
74
+
75
+ ## How to Use the Models
76
+
77
+ ### Online Demos
78
+ You can try the models directly through the HuggingFace Inference API, accessible on the right side of this page. Be aware that initially, the model needs to load and will run on limited CPU capacity, which might be slow. To enhance your experience, we are temporarily hosting some models on TPUs for a few days, significantly boosting their performance. Explore these under the **Spaces** section on the [Main Page](https://huggingface.co/NbAiLabBeta/).
79
+
80
+ ### Local Setup with HuggingFace
81
+ Alternatively, you can run the models locally. The Tiny, Base, and Small models are optimized for CPU execution. For the Medium and Large models, we recommend a system equipped with a GPU to ensure efficient processing. Setting up and using these models with HuggingFace's Transformers is straightforward, provided you have [Python](https://www.python.org/downloads/) installed on your machine. For practical demonstrations, refer to examples using this [sample mp3 file](https://github.com/NbAiLab/nb-whisper/raw/main/audio/king.mp3).
82
+
83
+ ```bash
84
+ # Download the sample file
85
+ $ wget -N https://github.com/NbAiLab/nb-whisper/raw/main/audio/king.mp3
86
+
87
+ # Install necessary libraries.
88
+ $ pip install transformers>=4.35.2
89
+ ```
90
+
91
+ After this is done, you should be able to run this in Python:
92
+
93
+ ```python
94
+ from transformers import pipeline
95
+
96
+ # Load the model
97
+ asr = pipeline("automatic-speech-recognition", "NbAiLabBeta/nb-whisper-small")
98
+
99
+ #transcribe
100
+ asr("king.mp3", generate_kwargs={'task': 'transcribe', 'language': 'no'})
101
+
102
+ ```
103
+
104
+ <details>
105
+ <summary>Expected output</summary>
106
+
107
+ ```json
108
+ {
109
+ {'text': ' Nordmenn er nordlendinger, trøndere, sørlendinger og folk fra alle andre regioner. Nordmenn er også innvandret fra Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria. Det er ikke alltid så lett å si hvor vi er fra, hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra.'}
110
+ }
111
+ ```
112
+ </details>
113
+
114
+ #### Extended HuggingFace
115
+ Examining the output above, we see that there are multiple repetitions at the end. This is because the video is longer than 30 seconds. By passing the ```chunk_lengt_s``` argument, we can transcribe longer file. Our experience is that we get slightly better result by setting that to 28 seconds instead of the default 30 seconds. We also recommend setting the beam size to 5 if possible. This greatly increases the accuracy but takes a bit longer and requires slightly more memory. The examples below also illustrates how to transcribe to English or Nynorsk, and how to get timestamps for sentences and words.
116
+
117
+ ```python
118
+ # Long Transcripts
119
+ asr("king.mp3", chunk_length_s=28, generate_kwargs={'task': 'transcribe', 'language': 'no'})
120
+
121
+ # Increase accuracy by setting beam size to 5
122
+ asr("king.mp3", chunk_length_s=28, return_timestamps=True, generate_kwargs={'num_beams': 5, 'task': 'transcribe', 'language': 'no'})
123
+
124
+ # Return Timestamps
125
+ asr("king.mp3", chunk_length_s=28, return_timestamps=True, generate_kwargs={'task': 'transcribe', 'language': 'no'})
126
+
127
+ # Return Word Level Timestamps
128
+ asr("king.mp3", chunk_length_s=28, return_timestamps="word", generate_kwargs={'task': 'transcribe', 'language': 'no'})
129
+
130
+ # Transcribe to Nynorsk
131
+ asr("king.mp3", chunk_length_s=28, generate_kwargs={'task': 'transcribe', 'language': 'nn'})
132
+
133
+ # Transcribe to English
134
+ asr("king.mp3", chunk_length_s=28, generate_kwargs={'task': 'transcribe', 'language': 'en'})
135
+
136
+ ```
137
+ <details>
138
+ <summary>Expected output</summary>
139
+
140
+ Long transcripts:
141
+ ```json
142
+ {
143
+ {'text': ' Nordmenn er nordlendinger, trøndere, sørlendinger og folk fra alle andre regioner. Nordmenn er også innvandret fra Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria. Det er ikke alltid så lett å si hvor vi er fra, hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra, hvilken nasjonalitet vi tilhører. Det vi kaller hjem, er der hjertet vårt er, og det kan ikke alltid plasseres innenfor landegrenser. Nordmenn er jenter som er glad i jenter, gutter som er glad i gutter, og jenter og gutter som er glad i hverandre. Nordmenn trommer på Gud, Allah, Altet og ingenting. Nordmenn liker Grieg, Kygo, Helbilis og Kari Bremnes. Med andre ord, Norge er dere. Norge er oss. Mitt største håp for Norge er at vi skal klare å ta vare på hverandre, at vi skal bygge dette landet videre på tillit, fellesskap og raushet.'}
144
+ }
145
+ ```
146
+
147
+ Timestamps:
148
+ ```json
149
+ {
150
+ {'text': ' Nordmenn er nordlendinger, trøndere, sørlendinger og folk fra alle andre regioner. Nordmenn er også innvandret fra Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria. Det er ikke alltid så lett å si hvor vi er fra, hvilken nasjonalitet vi er fra. Hvilken nasjonalitet vi er fra. hvilken nasjonalitet vi tilhører. Det vi kaller hjem, er der hjertet vårt er, og det kan ikke alltid plasseres innenfor landegrenser. Nordmenn er jenter som er glad i jenter, gutter som er glad i gutter, og jenter og gutter som er glad i hverandre. Nordmenn trommer på Gud, Allah, Altet og ingenting. Nordmenn liker Grieg, Kygo, Helbiles og Kari Bremnes. Med andre ord, Norge er dere. Norge er oss. Mitt største håp for Norge er at vi skal klare å ta vare på hverandre, at vi skal bygge dette landet videre på tillit, fellesskap og raushet.',
151
+ 'chunks': [{'timestamp': (0.0, 5.46),
152
+ 'text': ' Nordmenn er nordlendinger, trøndere, sørlendinger'},
153
+ {'timestamp': (5.52, 8.68), 'text': ' og folk fra alle andre regioner.'},
154
+ {'timestamp': (8.68, 16.64),
155
+ 'text': ' Nordmenn er også innvandret fra Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria.'},
156
+ {'timestamp': (16.64, 13.3),
157
+ 'text': ' Det er ikke alltid så lett å si hvor vi er fra, hvilken nasjonalitet vi er fra.'},
158
+ {'timestamp': (13.32, 30.28),
159
+ 'text': ' Hvilken nasjonalitet vi er fra. hvilken nasjonalitet vi tilhører.'},
160
+ {'timestamp': (32.52, 39.16),
161
+ 'text': ' Det vi kaller hjem, er der hjertet vårt er, og det kan ikke alltid plasseres'},
162
+ {'timestamp': (39.16, 42.0), 'text': ' innenfor landegrenser.'},
163
+ {'timestamp': (42.0, 46.74),
164
+ 'text': ' Nordmenn er jenter som er glad i jenter, gutter som er glad i gutter,'},
165
+ {'timestamp': (46.74, 51.12),
166
+ 'text': ' og jenter og gutter som er glad i hverandre.'},
167
+ {'timestamp': (51.16, 57.42),
168
+ 'text': ' Nordmenn trommer på Gud, Allah, Altet og ingenting.'},
169
+ {'timestamp': (57.42, 64.3),
170
+ 'text': ' Nordmenn liker Grieg, Kygo, Helbiles og Kari Bremnes.'},
171
+ {'timestamp': (64.34, 71.24),
172
+ 'text': ' Med andre ord, Norge er dere. Norge er oss.'},
173
+ {'timestamp': (71.24, 78.04),
174
+ 'text': ' Mitt største håp for Norge er at vi skal klare å ta vare på hverandre,'},
175
+ {'timestamp': (78.12, 84.68),
176
+ 'text': ' at vi skal bygge dette landet videre på tillit, fellesskap og raushet.'}]}
177
+ }
178
+ ```
179
+
180
+ Word Level Timestamps:
181
+ ```json
182
+ {
183
+ {"text": "Nordmenn er nordlendinger, trøndere, sørlendinger og folk fra alle andre regioner. Nordmenn er også innvandret fra Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria. Det er ikke alltid så lett å si hvor vi er fra, hvilken nasjonalitet vi tilhører. Det vi kaller hjem, er der hjertet vårt er, og det kan ikke alltid plasseres innenfor landegrenser. Nordmenn er jenter som er glad i jenter, gutter som er glad i gutter, og jenter og gutter som er glad i hverandre. Nordmenn trommer på Gud, Allah, Altet og ingenting. Nordmenn liker Grieg, Kygo, Helbilis og Kari Bremnes. Med andre ord, Norge er dere. Norge er oss. Mitt største håp for Norge er at vi skal klare å ta vare på hverandre, at vi skal bygge dette landet videre på tillit, fellesskap og raushet.",
184
+ "chunks": [
185
+ {"text": "Nordmenn", "timestamp": [0.72, 1.42]},
186
+ {"text": "er", "timestamp": [1.42, 1.74]},
187
+ // ... more chunks ...
188
+ {"text": "raushet.", "timestamp": [83.1, 84.88]}
189
+ ]
190
+ }
191
+ }
192
+ ```
193
+
194
+ Nynorsk:
195
+ ```json
196
+ {
197
+ {"text": "Nordmenn er nordlendingar, trøndarar, sørlendingar og folk frå alle andre regionar. Nordmenn er også innvandra frå Afghanistan, Pakistan, Polen, Sverige, Somalia og Syria. Det er ikkje alltid så lett å seie kvar vi er frå, kva nasjonalitet vi tilhøyrer. Det vi kallar heim, er der hjartet vårt er, og det kan ikkje alltid plasserast innanfor landegrenser. Nordmenn er jenter som er glad i jenter, gutar som erade i gutar, og jenter og gutar som er glade i kvarandre. Nordmenn trommar på Gud, Allah, Altet og ingenting. Nordmenn liker Grieg, Kygo, Helbiles og Kari Bremnes. Med andre ord, Noreg er dere! Noreg er oss. Mitt største håp for Noreg er at vi skal klare å ta vare på kvarandre, at vi skal byggje dette landet vidare på tillit, fellesskap og raushet."}
198
+ }
199
+ ```
200
+
201
+ English:
202
+ ```json
203
+ {
204
+ {"text": "Norwegians are Norwegians, trønders, southerners and people from all other regions. Norwegians are also invaded from Afghanistan, Pakistan, Poland, Sweden, Somalia and Suria. It is not always so easy to say where we are from, what nationality we belong to. What we call home is where our heart is, and it cannot always be placed within national borders. Norwegians are girls who like girls, boys who like boys, and girls and boys who like each other. Norwegians thrump on God, Allah, Altet and nothing. Norwegians like Grieg, Kygo, Helbilis and Kari Bremnes. In other words, Norway is you. Norway is us. My biggest hope for Norway is that we should be able to take care of each other, that we should build this country on trust, community and generosity."}
205
+ }
206
+ ```
207
+
208
+ </details>
209
+
210
+ ### Whisper CPP
211
+ Whisper CPP is a C++ implementation of the Whisper model, offering the same functionalities with the added benefits of C++ efficiency and performance optimizations. This allows embedding any Whisper model into a binary file, facilitating the development of real applications. However, it requires some familiarity with compiling C++ programs. Their [homepage](https://github.com/ggerganov/whisper.cpp) provides examples of how to build applications, including real-time transcription.
212
+
213
+ We have converted this model to the ggml-format model used by Whisper CPP binaries. The file can be downloaded [here](blob/main/ggml-model.bin), and a `q5_0` quantized version is also available [here](blob/main/ggml-model-q5_0.bin).
214
+
215
+ ```bash
216
+ # We can download and compile whisper.cpp
217
+ $ git clone --depth 1 https://github.com/ggerganov/whisper.cpp --branch v1.5.1
218
+ $ cd whisper.cpp/
219
+ $ make
220
+
221
+ # We also need to convert the audio to WAV as that is the only format supported by whisper.cpp
222
+ $ wget -N https://github.com/NbAiLab/nb-whisper/raw/main/audio/king.mp3
223
+ $ ffmpeg -i king.mp3 -ar 16000 -ac 1 -c:a pcm_s16le king.wav
224
+
225
+ # Lets download the two ggml-files from this site
226
+ wget -N https://huggingface.co/NbAiLabBeta/nb-whisper-small/resolve/main/ggml-model.bin -O models/nb-small-ggml-model.bin
227
+ wget -N https://huggingface.co/NbAiLabBeta/nb-whisper-small/resolve/main/ggml-model-q5_0.bin -O models/nb-small-ggml-model-q5_0.bin
228
+
229
+ # And run it with the f16 default model
230
+ $ ./main -l no -m models/nb-small-ggml-model.bin king.wav
231
+
232
+ # Or the quantized version
233
+ $ ./main -l no -m models/nb-small-ggml-model-q5_0.bin king.wav
234
+ ```
235
+
236
+ ### WhisperX and Speaker Diarization
237
+ Speaker diarization is a technique in natural language processing and automatic speech recognition that identifies and separates different speakers in an audio recording. It segments the audio into parts based on who is speaking, enhancing the quality of transcribing meetings or phone calls. We find that [WhisperX](https://github.com/m-bain/whisperX) is the easiest way to use our models for diarizing speech. In addition, WhisperX is using phoneme-based Wav2Vec-models for improving the alignment of the timestamps. As of December 2023 it also has native support for using the nb-wav2vec-models. It currently uses [PyAnnote-audio](https://github.com/pyannote/pyannote-audio) for doing the actual diarization. This package has a fairly strict licence where you have to agree to user terms. Follow the instructions below.
238
+
239
+ ```bash
240
+ # Follow the install instructions on https://github.com/m-bain/whisperX
241
+ # Make sure you have a HuggingFace account and have agreed to the pyannote terms
242
+
243
+ # Log in (or supply HF Token in command line)
244
+ huggingface-cli login
245
+
246
+ # Download a test file
247
+ wget -N https://github.com/NbAiLab/nb-whisper/raw/main/audio/knuthamsun.mp3
248
+
249
+ # Optional. If you get complains about not support for Norwegian, do:
250
+ pip uninstall whisperx && pip install git+https://github.com/m-bain/whisperx.git@8540ff5985fceee764acbed94f656063d7f56540
251
+
252
+ # Transcribe the test file. All transcripts will end up in the directory of the mp3-file
253
+ whisperx knuthamsun.mp3 --model NbAiLabBeta/nb-whisper-small --language no --diarize
254
+
255
+ ```
256
+
257
+ You can also run WhisperX from Python. Please take a look at the instructions on [WhisperX homepage](https://github.com/m-bain/whisperX).
258
+
259
+
260
+
261
+
262
+ ### API
263
+ Instructions for accessing the models via a simple API are included in the demos under Spaces. Note that these demos are temporary and will only be available for a few weeks.
264
+
265
+ ## Training Data
266
+ The training data originates from Språkbanken and the National Library of Norway's digital collection, including:
267
+
268
+ - NST Norwegian ASR Database (16 kHz) and its corresponding dataset
269
+ - Transcribed speeches from the Norwegian Parliament by Språkbanken
270
+ - TV broadcast (NRK) subtitles (NLN digital collection)
271
+ - Audiobooks (NLN digital collection)
272
+
273
+ ## Downstream Use
274
+
275
+ The models, especially the smaller ones, may exhibit occasional hallucinations and may drop parts of the transcript. They are designed to convert spoken language into grammatically correct written sentences, which might not always be word-for-word translations. We have made two extra model variant for users that want a different transcription style. We encourage users to try the models themselves to get a better understanding.
276
+
277
+ ## Bias, Risks, and Limitations
278
+
279
+ Using these models without adequate risk assessment and mitigation could be considered irresponsible. They may contain biases or other undesirable distortions. Users who deploy these models or integrate them into systems or services are responsible for mitigating risks and complying with applicable AI regulations. The National Library of Norway, as the model owner, disclaims liability for any outcomes resulting from third-party use of these models.
280
+
281
+ ### Software
282
+ The model was trained using Jax/Flax and converted to PyTorch, Tensorflow, whisper.cpp, and ONXX formats. These are available under `Files and versions`. We welcome requests for conversion to other formats. All training code and scripts are released under the Apache License 2.0 in the GitHub repository [nb-whisper](https://github.com/NbAiLab/nb-whisper/).
283
+
284
+ ## Citation & Contributors
285
+ The NB-Whisper Small model is a product of the NoSTram project led by Per Egil Kummervold ([@pere](https://huggingface.co/pere)) at the National Library of Norway. Key contributors include Javier de la Rosa ([@versae](https://huggingface.co/versae)), Freddy Wetjen ([@freddyw](https://huggingface.co/freddyw)), and Rolv-Arild Braaten ([@Rolv-Arild](https://huggingface.co/Rolv-Arild)). NB AI-Lab, under the direction of Svein Arne Brygfjeld ([@Brygfjeld](https://huggingface.co/Brygfjeld)), supported the project's successful completion. A detailed paper on our process and findings is forthcoming.
286
+
287
+ ## Disclaimer
288
+
289
+ The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions. When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence. In no event shall the owner of the models (The National Library of Norway) be liable for any results arising from the use made by third parties of these models.
290
+
291
+ ## Acknowledgements
292
+
293
+ Our gratitude extends to [Google TPU Research Cloud](https://sites.research.google/trc/about/) for training resources, Google Cloud for translation credits, and HuggingFace's Sanchit Ghandi for technical support. A special thank you to Per Erik Solberg at Språkbanken for the collaboration on the Stortinget corpus.
294
+
295
+ ## Contact
296
+ For feedback, technical concerns, or collaboration inquiries, please contact <a rel="noopener nofollow" href="mailto:ailab@nb.no">ailab@nb.no</a>. If you plan to include this model in your research, contact us for the latest information on our upcoming paper for citation purposes.