|
--- |
|
datasets: |
|
- librispeech_asr |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
library_name: speechbrain |
|
license: apache-2.0 |
|
--- |
|
<iframe src="https://ghbtns.com/github-btn.html?user=SupradeepDanturti&repo=ConvAIProject&type=star&count=true&size=large&v=2" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe> |
|
<br/><br/> |
|
|
|
# Speaker Counting with XVector and ECAPA-TDNN models |
|
|
|
This repository provides tools for detecting and counting the number of speakers in an audio recording using models like XVector and ECAPA-TDNN, trained on datasets such as LibriSpeech. |
|
|
|
The pre-trained system processes audio inputs to detect the presence and count of speakers, providing output in the following format: |
|
|
|
``` |
|
0.00-2.50 has 1 speaker |
|
2.50-4.20 has 2 speakers |
|
4.20-5:30 has no speakers |
|
``` |
|
|
|
The system expects input recordings sampled at 16kHz. If your signal has a different sample rate, resample it using torchaudio before using the interface. |
|
|
|
|
|
# Model Performance |
|
|
|
Error rates for different numbers of speakers detected by the models are provided below: |
|
|
|
## XVector Model Error Rates |
|
|
|
| Number of Speakers | Error Rate | |
|
|--------------------|------------------| |
|
| No Speakers | 0.00e+00 | |
|
| 1 Speaker | 4.06e-01 | |
|
| 2 Speakers | 3.20e-02 | |
|
| 3 Speakers | 2.15e-01 | |
|
| 4 Speakers | 4.67e-01 | |
|
| **Overall** | **2.29e-01** | |
|
|
|
## ECAPA-TDNN Model Error Rates |
|
|
|
| Number of Speakers | Error Rate | |
|
|--------------------|------------------| |
|
| No Speakers | 0.00e+00 | |
|
| 1 Speaker | 4.12e-01 | |
|
| 2 Speakers | 4.10e-01 | |
|
| 3 Speakers | 3.26e-01 | |
|
| 4 Speakers | 5.23e-01 | |
|
| **Overall** | **2.40e-01** | |
|
|
|
## Self-supervised MLP Model Error Rates |
|
|
|
| Number of Speakers | Error Rate | |
|
|--------------------|------------------| |
|
| No Speakers | 0.00e+00 | |
|
| 1 Speaker | 8.67e-03 | |
|
| 2 Speakers | 1.14e-01 | |
|
| 3 Speakers | 4.08e-01 | |
|
| 4 Speakers | 4.49e-01 | |
|
| **Overall** | **2.00e-01** | |
|
|
|
## Self-supervised XVector Model Error Rates |
|
|
|
| Number of Speakers | Error Rate | |
|
|--------------------|------------------| |
|
| No Speakers | 0.00e+00 | |
|
| 1 Speaker | 1.64e-02 | |
|
| 2 Speakers | 1.39e-01 | |
|
| 3 Speakers | 3.34e-01 | |
|
| 4 Speakers | 5.41e-01 | |
|
| **Overall** | **2.10e-01** | |
|
|
|
Error rates are crucial to understanding where the models excel and where they may need further improvement. In particular, they can reveal challenges in scenarios with higher numbers of speakers. |
|
|
|
### Limitations |
|
Please note that these models are evaluated under controlled conditions and their performance may vary when applied to different datasets or in real-world scenarios. |
|
|
|
|
|
## System Overview |
|
This project includes interfaces for processing audio files, detecting speech, and counting speakers. It leverages pre-trained models and custom scripts for refining and aggregating predictions. |
|
|
|
### Installation |
|
``` |
|
pip install speechbrain |
|
``` |
|
Please look at [SpeechBrain](https://speechbrain.github.io/) for tutorials or more information. |
|
|
|
## Using the Speaker Counter Interface |
|
### For XVector & ECAPA-TDNN: |
|
|
|
``` |
|
from interface.SpeakerCounter import SpeakerCounter |
|
|
|
wav_path = "path/to/your/audio.wav" |
|
model_path = "path/to/your/model" |
|
save_dir = "path/to/save/results" |
|
|
|
# Initialize the SpeakerCounter |
|
speaker_counter = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir) |
|
|
|
# Run inference |
|
speaker_counter.classify_file(wav_path) |
|
``` |
|
|
|
### For a Self-supervised model: |
|
``` |
|
from interface.SpeakerCounterSelfsupervisedMLP import SpeakerCounter |
|
|
|
wav_path = "path/to/your/audio.wav" |
|
model_path = "path/to/your/selfsupervised_model" |
|
save_dir = "path/to/save/results" |
|
|
|
# Initialize the SpeakerCounter |
|
audio_classifier = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir) |
|
|
|
# Run inference |
|
audio_classifier.classify_file(wav_path) |
|
``` |
|
|
|
|
|
## **Setup and Training Instructions** |
|
|
|
To reproduce the results of this project, follow these steps. |
|
|
|
``` |
|
!git clone https://github.com/SupradeepDanturti/ConvAIProject |
|
%cd ConvAIProject |
|
``` |
|
Download the project code from the GitHub repository and navigate into the project directory. |
|
|
|
|
|
``` |
|
!python prepare_dataset/download_required_data.py --output_folder <destination_folder_path> |
|
``` |
|
Download the necessary datasets (LibriSpeech etc.), specifying the desired destination folder. |
|
|
|
``` |
|
!python prepare_dataset/create_custom_dataset.py prepare_dataset/dataset.yaml |
|
``` |
|
Create custom dataset based on set parameters as shown in the sample below |
|
|
|
Sample of dataset.yaml: |
|
``` |
|
n_sessions: |
|
train: 1000 # Creates 1000 sessions per class |
|
dev: 200 # Creates 200 sessions per class |
|
eval: 200 # Creates 200 sessions per class |
|
n_speakers: 4 # max number of speakers. In this case the total classes will be 5 (0-4 speakers) |
|
max_length: 120 # max length in seconds for each session/utterance. |
|
``` |
|
<center>Sample of dataset.yaml</center> |
|
|
|
To train the XVector model run the following command. |
|
``` |
|
%cd xvector |
|
!python train_xvector_augmentation.py hparams_xvector_augmentation.yaml |
|
``` |
|
|
|
To train the ECAPA-TDNN model run the following command. |
|
|
|
``` |
|
%cd ecapa_tdnn |
|
!python train_ecapa_tdnn.py hparams_ecapa_tdnn_augmentation.yaml |
|
``` |
|
|
|
To train the SelfSupervised MLP model run the following command. |
|
``` |
|
%cd selfsupervised |
|
!python selfsupervised_mlp.py hparams_selfsupervised_mlp.yaml |
|
``` |
|
|
|
To train the SelfSupervised XVector model run the following command. |
|
``` |
|
%cd selfsupervised |
|
!python selfsupervised_xvector.py hparams_selfsupervised_xvector.yaml |
|
``` |
|
|
|
### **This Project was developed completely using [SpeechBrain](https://speechbrain.github.io/) ** |
|
|
|
## Reference |
|
``` |
|
@misc{speechbrain, |
|
title={{SpeechBrain}: A General-Purpose Speech Toolkit}, |
|
author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio}, |
|
year={2021}, |
|
eprint={2106.04624}, |
|
archivePrefix={arXiv}, |
|
primaryClass={eess.AS}, |
|
note={arXiv:2106.04624} |
|
} |
|
``` |
