Package: tts-mos-test-mturk
tts-mos-test-mturkCommand-line interface (CLI) to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk).Featuresinit: initialize project from .json-filemaskscreate: create empty maskmask-workers-by-id: mask workers by their WorkerIdmask-workers-by-age-group: mask workers by their age groupmask-workers-by-gender: mask workers by their gendermask-workers-by-assignments-count: mask workers by their count of assignmentsmask-workers-by-masked-ratings-count: mask workers by their count of masked ratingsmask-workers-by-correlation: mask workers by their algorithm/sentence correlationmask-workers-by-correlation-percent: mask workers by their algorithm/sentence correlation (percentage-wise)mask-assignments-by-id: mask assignments by their AssignmentIdmask-assignments-by-device: mask assignments by their listening devicemask-assignments-by-status: mask assignments by their statusmask-assignments-by-time: mask assignments by their submit timemask-rating-outliers: mask outlying ratingsmerge-masks: merge masks togetherreverse-mask: reverse maskstatsprint-mos: print MOS and CI95print-masking-stats: print masking statisticsprint-worker-stats: print worker statistics for each algorithmprint-assignment-stats: print assignment statistics for each workerprint-sentence-stats: print sentence statistics for each algorithmprint-data: print all data pointsmturkprepare-approval: generate approval fileprepare-rejection: generate rejection fileprepare-bonus-payment: generate bonus payment fileInstallationpipinstalltts-mos-test-mturk--userUsageusage: mos-cli [-h] [-v] {init,masks,stats,mturk} ...

CLI to evaluate text-to-speech MOS studies done on MTurk.

positional arguments:
  {init,masks,stats,mturk}
                                        description
    init                                initialize project from .json-file
    masks                               masks commands
    stats                               stats commands
    mturk                               mturk commands

options:
  -h, --help                            show this help message and exit
  -v, --version                         show program's version number and exitProject JSON example{"algorithms":["alg1","alg2","alg3","alg4"],"files":["file1","file2","file3"],"workers":{"worker1":{"gender":"male","age_group":"18-29","assignments":{"assignment1":{"device":"headphone","state":"Approved","hit":"hit1","time":"13.07.23 05:08:04","ratings":[{"algorithm":"alg1","file":"file1","votes":{"naturalness":3,"intelligibility":5}},{"algorithm":"alg2","file":"file3","votes":{"naturalness":2,"intelligibility":4}}]}}}}}For a longer example seeetc/example.json. It contains 4 algorithms and 120 files which were rated by 36 dummy workers in batches of 10 files per assignment. An example parsing of that file is underetc/example.sh.Roadmapaddmasks mask-assignments-not-of-last-month/week/daymake device, state, hit and time optionalmake tax value optionalDependenciesnumpypandastqdmordered-set>=4.1.0mean-opinion-score==0.0.2ContributingIf you notice an error, please don't hesitate to open an issue.Development setup# updatesudoaptupdate# install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be runsudoaptinstallpython3-pip\python3.8python3.8-devpython3.8-distutilspython3.8-venv\python3.9python3.9-devpython3.9-distutilspython3.9-venv\python3.10python3.10-devpython3.10-distutilspython3.10-venv\python3.11python3.11-devpython3.11-distutilspython3.11-venv# install pipenv for creation of virtual environmentspython3.8-mpipinstallpipenv--user# check out repogitclonehttps://github.com/stefantaubert/tts-mos-test-mturk.gitcdtts-mos-test-mturk# create virtual environmentpython3.8-mpipenvinstall--devRunning the tests# first install the tool like in "Development setup"# then, navigate into the directory of the repo (if not already done)cdtts-mos-test-mturk# activate environmentpython3.8-mpipenvshell# run teststoxFinal lines of test result output:py38: commands succeeded
  py39: commands succeeded
  py310: commands succeeded
  py311: commands succeeded
  congratulations :)LicenseMIT LicenseAcknowledgmentsRibeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419.https://doi.org/10.1109/ICASSP.2011.5946971Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410CitationIf you want to cite this repo, you can use this BibTeX-entry generated by GitHub (seeAbout => Cite this repository).Taubert, S. (2024). tts-mos-test-mturk (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.10461539Changelogv0.0.2 (2024-01-05)Bugfix:mask mask-workers-by-correlation-percent: sorting was not always correctAdded:mask mask-workers-by-correlation-percent: added option to include masked workers in percentage calculationmturk prepare-bonus-payment: added logging of fees for Mechanical Turkmturk prepare-approval: added logging of fees for Mechanical Turkaddedmask mask-assignments-by-statusaddedmask mask-assignments-by-timeadded parsing ofHITIdadded option to mask assignments before preparing rejection CSVadded!before mask name reverses mask on inputaddedmasks mask-assignments-by-idaddedmasks mask-workers-by-idChanged:moved template creation and preparation to another repositoryremoved worktime parsingv0.0.1 (2023-02-23)Initial release