File size: 6,480 Bytes
f0f5f5e d8050d4 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 06d5b0a cffe859 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 |
---
datasets:
- librispeech_asr
language:
- en
metrics:
- accuracy
library_name: speechbrain
license: apache-2.0
---
<iframe src="https://ghbtns.com/github-btn.html?user=SupradeepDanturti&repo=ConvAIProject&type=star&count=true&size=large&v=2" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe>
<br/><br/>
# Speaker Counting with XVector and ECAPA-TDNN models
This repository provides tools for detecting and counting the number of speakers in an audio recording using models like XVector and ECAPA-TDNN, trained on datasets such as LibriSpeech.
The pre-trained system processes audio inputs to detect the presence and count of speakers, providing output in the following format:
```
0.00-2.50 has 1 speaker
2.50-4.20 has 2 speakers
4.20-5:30 has no speakers
```
The system expects input recordings sampled at 16kHz. If your signal has a different sample rate, resample it using torchaudio before using the interface.
# Model Performance
Error rates for different numbers of speakers detected by the models are provided below:
## XVector Model Error Rates
| Number of Speakers | Error Rate |
|--------------------|------------------|
| No Speakers | 0.00e+00 |
| 1 Speaker | 4.06e-01 |
| 2 Speakers | 3.20e-02 |
| 3 Speakers | 2.15e-01 |
| 4 Speakers | 4.67e-01 |
| **Overall** | **2.29e-01** |
## ECAPA-TDNN Model Error Rates
| Number of Speakers | Error Rate |
|--------------------|------------------|
| No Speakers | 0.00e+00 |
| 1 Speaker | 4.12e-01 |
| 2 Speakers | 4.10e-01 |
| 3 Speakers | 3.26e-01 |
| 4 Speakers | 5.23e-01 |
| **Overall** | **2.40e-01** |
## Self-supervised MLP Model Error Rates
| Number of Speakers | Error Rate |
|--------------------|------------------|
| No Speakers | 0.00e+00 |
| 1 Speaker | 8.67e-03 |
| 2 Speakers | 1.14e-01 |
| 3 Speakers | 4.08e-01 |
| 4 Speakers | 4.49e-01 |
| **Overall** | **2.00e-01** |
## Self-supervised XVector Model Error Rates
| Number of Speakers | Error Rate |
|--------------------|------------------|
| No Speakers | 0.00e+00 |
| 1 Speaker | 1.64e-02 |
| 2 Speakers | 1.39e-01 |
| 3 Speakers | 3.34e-01 |
| 4 Speakers | 5.41e-01 |
| **Overall** | **2.10e-01** |
Error rates are crucial to understanding where the models excel and where they may need further improvement. In particular, they can reveal challenges in scenarios with higher numbers of speakers.
### Limitations
Please note that these models are evaluated under controlled conditions and their performance may vary when applied to different datasets or in real-world scenarios.
## System Overview
This project includes interfaces for processing audio files, detecting speech, and counting speakers. It leverages pre-trained models and custom scripts for refining and aggregating predictions.
### Installation
```
pip install speechbrain
```
Please look at [SpeechBrain](https://speechbrain.github.io/) for tutorials or more information.
## Using the Speaker Counter Interface
### For XVector & ECAPA-TDNN:
```
from interface.SpeakerCounter import SpeakerCounter
wav_path = "path/to/your/audio.wav"
model_path = "path/to/your/model"
save_dir = "path/to/save/results"
# Initialize the SpeakerCounter
speaker_counter = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir)
# Run inference
speaker_counter.classify_file(wav_path)
```
### For a Self-supervised model:
```
from interface.SpeakerCounterSelfsupervisedMLP import SpeakerCounter
wav_path = "path/to/your/audio.wav"
model_path = "path/to/your/selfsupervised_model"
save_dir = "path/to/save/results"
# Initialize the SpeakerCounter
audio_classifier = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir)
# Run inference
audio_classifier.classify_file(wav_path)
```
## **Setup and Training Instructions**
To reproduce the results of this project, follow these steps.
```
!git clone https://github.com/SupradeepDanturti/ConvAIProject
%cd ConvAIProject
```
Download the project code from the GitHub repository and navigate into the project directory.
```
!python prepare_dataset/download_required_data.py --output_folder <destination_folder_path>
```
Download the necessary datasets (LibriSpeech etc.), specifying the desired destination folder.
```
!python prepare_dataset/create_custom_dataset.py prepare_dataset/dataset.yaml
```
Create custom dataset based on set parameters as shown in the sample below
Sample of dataset.yaml:
```
n_sessions:
train: 1000 # Creates 1000 sessions per class
dev: 200 # Creates 200 sessions per class
eval: 200 # Creates 200 sessions per class
n_speakers: 4 # max number of speakers. In this case the total classes will be 5 (0-4 speakers)
max_length: 120 # max length in seconds for each session/utterance.
```
<center>Sample of dataset.yaml</center>
To train the XVector model run the following command.
```
%cd xvector
!python train_xvector_augmentation.py hparams_xvector_augmentation.yaml
```
To train the ECAPA-TDNN model run the following command.
```
%cd ecapa_tdnn
!python train_ecapa_tdnn.py hparams_ecapa_tdnn_augmentation.yaml
```
To train the SelfSupervised MLP model run the following command.
```
%cd selfsupervised
!python selfsupervised_mlp.py hparams_selfsupervised_mlp.yaml
```
To train the SelfSupervised XVector model run the following command.
```
%cd selfsupervised
!python selfsupervised_xvector.py hparams_selfsupervised_xvector.yaml
```
### **This Project was developed completely using [SpeechBrain](https://speechbrain.github.io/) **
## Reference
```
@misc{speechbrain,
title={{SpeechBrain}: A General-Purpose Speech Toolkit},
author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
year={2021},
eprint={2106.04624},
archivePrefix={arXiv},
primaryClass={eess.AS},
note={arXiv:2106.04624}
}
``` |