Package: bt4vt
Bias Tests for Voice Technologies (bt4vt)About this packagebt4vtis a python library to diagnose performance discrepancies (i.e. bias) in speaker verification models. The library provides evaluation measures and visualisations to interrogate model performance and can be integrated into development pipelines to test for bias. We plan to extend the library to other speech processing tasks in future. Speak to us if you're interested to help.Read the docsThe development of this open source library is part of theFair EVAproject and has been supported by the Mozilla Technology Fund.Setup instructionsYou needpython 3to use this library. The easiest way to use the library is to install it with pip.$ pip install bt4vtTo use the library in development mode, install it as follows:Clone this repository from github and navigate to the project's root directory (bt4vt\)$ git clone https://github.com/wiebket/bt4vt.gitInstall the project.$ pip install -e .UsageBelow is an example for usingbt4vt. All necessary files can be copied by usingcopy_example(). The example evaluates the fairness of models released with theClova AI VoxCeleb Trainer.Run Bias Tests for Speaker Verification1. Copy example resourcesAll files that are necessary to reproduce the example can be copied to a folder of your choice. Here, we copy the resources to~/bias_tests_4_voice_tech/example/.import bt4vt

    bt4vt.dataio.copy_example("~/bias_tests_4_voice_tech/example/")1. Create config fileA template for theconfig.yamlfile is now provided in the~/bias_tests_4_voice_tech/example/folder.
If you copied the files to a different folder you need to adjust the path to thespeaker_metadata_fileandresults_dir.speaker_metadata_file: "~/bias_tests_4_voice_tech/example/vox1_meta.csv"
    results_dir: "~/bias_tests_4_voice_tech/results/"

    # for metadata
    id_column: "VoxCeleb1 ID"
    select_columns: ["Gender", "Nationality"]
    speaker_groups: [["Gender"], ["Nationality"], ["Gender", "Nationality"]]

    # for scores
    reference_filepath_column: "ref_file"
    test_filepath_column: "com_file"
    label_column: "lab"
    scores_column: "sc"

    # for dataset evaluation
    dataset_evaluation: True

    # for run_tests
    dcf_costs: [[0.05, 1, 1]]2. Run the bias testsImportbt4vtand specify your score and config file. Pass the score and config file path to theSpeakerBiastTestclass and run therun_tests()function.score_file = "~/bias_tests_4_voice_tech/example/resnetse34v2_H-eval_scores.csv"
config_file = "~/bias_tests_4_voice_tech/example/config.yaml"

test = bt4vt.core.SpeakerBiasTest(score_file, config_file)

test.run_tests()Test results will be stored in~/bias_tests_4_voice_tech/results. The results file containsmetrics ratiosfor the metrics and speaker groups specified in the config file.Themetrics ratiois calculated asspeaker group metric / average metric.Under DevelopmentThe project is under continuous development and we appreciate contributions! Planned enhancements include:advanced plotting of test resultsimplementation of further metrics and fairness measuresinclusive evaluation dataset generatorsIf you'd like to get involved, have a look at:https://www.faireva.org/get-involvedResourcesAn early versions of this library was developed as part of the following research:Wiebke Toussaint Hutiri and Aaron Yi Ding. 2022. Bias in Automated Speaker Recognition. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT '22). Association for Computing Machinery, New York, NY, USA, 230–247.https://doi-org.tudelft.idm.oclc.org/10.1145/3531146.3533089LicenseThis code is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.This software is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for details.You should have received a copy of the GNU General Public License along with this source code. If not, go the following link:http://www.gnu.org/licenses/.