|
--- |
|
license: cc-by-nc-4.0 |
|
tags: |
|
- speech processing |
|
- self-supervision |
|
- african languages |
|
--- |
|
## Model description |
|
This self-supervised speech model (a.k.a. SSA-HuBERT-base-60k) is based on a HuBERT Base architecture (~95M params) [1]. |
|
It was trained on nearly 60 000 hours of speech segments and covers 21 languages and variants spoken in Sub-Saharan Africa. |
|
|
|
### Pretraining data |
|
- Dataset: The training dataset was composed of both studio recordings (controlled environment, prepared talks) and street interviews (noisy environment, spontaneous speech). |
|
|
|
- Languages: Bambara (bam), Dyula (dyu), French (fra), Fula (ful), Fulfulde (ffm), Fulfulde (fuh), Gulmancema (gux), Hausa (hau), Kinyarwanda (kin), Kituba (ktu), Lingala (lin), Luba-Lulua (lua), Mossi (mos), Maninkakan (mwk), Sango (sag), Songhai (son), Swahili (swc), Swahili (swh), Tamasheq (taq), Wolof (wol), Zarma (dje). |
|
|
|
## ASR fine-tuning |
|
The SpeechBrain toolkit (Ravanelli et al., 2021) is used to fine-tune the model. |
|
Fine-tuning is done for each language using the FLEURS dataset [2]. |
|
The pretrained model (SSA-HuBERT-base-60k) is considered as a speech encoder and is fully fine-tuned with two 1024 linear layers and a softmax output at the top. |
|
|
|
## License |
|
This model is released under the CC-by-NC 4.0 conditions. |
|
|
|
## Publication |
|
This model were presented at AfricaNLP 2024. |
|
The associated paper is available here: [Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context](https://openreview.net/forum?id=zLOhcft2E7) |
|
|
|
### Citation |
|
Please cite our paper when using SSA-HuBERT-base-60k model: |
|
|
|
Caubrière, A., & Gauthier, E. (2024). Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context. In 5th Workshop on African Natural Language Processing (AfricaNLP 2024). |
|
|
|
**Bibtex citation:** |
|
``` |
|
@inproceedings{caubri{\`e}re2024ssaspeechssl, |
|
title={Africa-Centric Self-Supervised Pretraining for Multilingual Speech Representation in a Sub-Saharan Context}, |
|
author={Antoine Caubri{\`e}re and Elodie Gauthier}, |
|
booktitle={5th Workshop on African Natural Language Processing}, |
|
year={2024}, |
|
url={https://openreview.net/forum?id=zLOhcft2E7}} |
|
``` |
|
|
|
## Results |
|
The following results are obtained in a greedy mode **(no language model rescoring)**. |
|
Character error rates (CERs) and Word error rates (WERs) are given in the table below, on the 20 languages of the SSA subpart of the FLEURS dataset. |
|
|
|
| **Language** | **CER** | **CER (joint finetuning)** | **WER** | **WER (joint finetuning)** | |
|
| :----------------- | :--------- | :--------- | :--------- | :--------- | |
|
| **Afrikaans** | 23.3 | 20.3 | 68.4 | 62.6 | |
|
| **Amharic** | 15.9 | 14.9 | 52.7 | 49.0 | |
|
| **Fula** | 21.2 | 17.8 | 61.9 | 56.4 | |
|
| **Ganda** | 11.5 | 10.7 | 52.8 | 50.3 | |
|
| **Hausa** | 10.5 | 9.0 | 32.5 | 29.4 | |
|
| **Igbo** | 19.7 | 17.2 | 57.5 | 52.9 | |
|
| **Kamba** | 16.1 | 15.6 | 53.9 | 53.7 | |
|
| **Lingala** | 8.7 | 6.9 | 24.7 | 20.9 | |
|
| **Luo** | 9.9 | 8.2 | 38.9 | 34.9 | |
|
| **Northen-Sotho** | 13.5 | 11.7 | 43.2 | 38.9 | |
|
| **Nyanja** | 13.3 | 10.9 | 54.2 | 48.3 | |
|
| **Oromo** | 22.8 | 20.1 | 78.1 | 74.8 | |
|
| **Shona** | 11.6 | 8.3 | 50.2 | 39.3 | |
|
| **Somali** | 21.6 | 19.7 | 64.9 | 60.3 | |
|
| **Swahili** | 7.1 | 5.5 | 23.8 | 20.3 | |
|
| **Umbundu** | 21.7 | 18.8 | 61.7 | 54.2 | |
|
| **Wolof** | 19.4 | 17.0 | 55.0 | 50.7 | |
|
| **Xhosa** | 11.9 | 9.9 | 51.6 | 45.9 | |
|
| **Yoruba** | 24.3 | 23.5 | 67.5 | 65.7 | |
|
| **Zulu** | 12.2 | 9.6 | 53.4 | 44.9 | |
|
| *Overall average* | *15.8* | *13.8* | *52.3* | *47.7* | |
|
|
|
|
|
## Reproductibilty |
|
We propose a notebook to reproduce the ASR experiments mentioned in our paper. See `SB_ASR_FLEURS_finetuning.ipynb`. |
|
By using the `ASR_FLEURS-swahili_hf.yaml` config file, you will be able to run the recipe on Swahili. |
|
|
|
## References |
|
[1] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. In 2021 IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp.3451–3460, 2021. doi: 10.1109/TASLP.2021.3122291. |
|
[2] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 798–805, 2022. doi: 10.1109/SLT54892.2023.10023141. |
