Package: commonvoice-utils
Common Voice UtilsThis repository collects together basic linguistic processing data for using dataset
dumps from theCommon Voiceproject. It aims to provide a one-stop-shop for
utilities and data useful in training ASR and TTS systems.ToolsPhonemiser:A rudimentary grapheme to phoneme (g2p) system based on either:a deterministic longest-match left-to-right replacement of orthographic units; ora weighted finite-state transducerValidator:A validation and normalisation script.It checks a sentence to see if it can be converted and if possible normalises the encoding, removes punctuation and returns itAlphabet:The relevant alphabet of the language, appropriate for use in training ASRSegmenter:A deterministic sentence segmentation algorithm tuned for segmenting paragraphs from WikipediaCorpora:Contains metadata for different corpora you may be interested in using with Common VoiceInstallationThe easiest way is withpip:$pipinstallgit+https://github.com/ftyers/commonvoice-utils.gitHow to use itCommand line toolThere is also a command line tool,covo/ˈkəʊvəʊ/ which aims to expose much of the functionality
through the command line. Some examples on the next lines:Process a Wikipedia dumpUse a Wikipedia dump to get text for a language mode in the right format:$covodumpmtwiki-latest-pages-articles.xml.bz2|covosegmentmt|covonormmt
x'inhi l-wikipedijail-wikipedija hi mmexxija mill-fondazzjoni wikimedia fondazzjoni mingħajr fini ta'lukrulitospitaproġettioħrab'kontenut ħieles u multilingwiil-malti huwa l-ilsien nazzjonali tar-repubblika ta'malta
huwal-ilsienuffiċjaliflimkienmal-ingliżkifukollwieħedmill-ilsnauffiċjalital-unjoniewropeaQuery the OPUS corpus collectionGet a list of URLs for a particular language from the OPUS corpus collection:$covoopusmt|sort-gr23859documents,69.4Mtokenshttps://object.pouta.csc.fi/OPUS-DGT/v2019/mono/mt.txt.gz8665documents,25.8Mtokenshttps://object.pouta.csc.fi/OPUS-JRC-Acquis/v3.0/mono/mt.txt.gz5388documents,8.9Mtokenshttps://object.pouta.csc.fi/OPUS-JW300/v1b/mono/mt.txt.gz
...Convert grapheme input to phonemesGet the grapheme to phoneme output for some arbitrary input:$echo"euskal herrian euskaraz"|covophoneu
eus̺kalerianeus̺kaɾas̻

$echo"قايتا نىشان بەلگىلەش ئورنى ئۇيغۇرچە ۋىكىپىدىيە"|covophonug
qɑjtɑnɪʃɑnbɛlɡɪlɛʃornɪujʁurtʃɛvɪkɪpɪdɪjɛExport data for use in Coqui STTDesigned for use withCoqui STT, converts
to 16kHz mono-channel PCM .wav files and runs the transcripts through the validation
step. In addition outputs.csvfiles for each of the input.tsvfiles. The
structure of the command is:$covoexportcoqui[languagecode][commonvoicedatasetdirectory]For example for Erzya,myv:$covoexportmyvcv-corpus-8.0-2022-01-19/myv/
LoadingTSVfile:cv-corpus-8.0-2022-01-19/myv/test.tsvImportingmp3files...Imported292samples.Skipped2samplesthatwerelongerthan10seconds.Finalamountofimportedaudio:0:27:03from0:27:23.SavingnewCoquiSTT-formattedCSVfileto:cv-corpus-8.0-2022-01-19/myv/clips/test.csvWritingCSVfilefortrain.pyas:cv-corpus-8.0-2022-01-19/myv/clips/test.csvExport data for use in NVIDIA NeMoDesigned for use withNVIDIA's Nemo, converts
to 16kHz mono-channel PCM .wav files and runs the transcripts through the validation
step. In addition outputs.jsonfiles for each of the input.tsvfiles. The
structure of the command is:$covoexportnemo[languagecode][commonvoicedatasetdirectory]For example for Sardinian,sc:INFO:root:Findexistingfolder/tmp/cv-corpus-10.0-2022-07-04/sc/
INFO:root:Convertingmp3towavfor/tmp/cv-corpus-10.0-2022-07-04/sc/test.tsv.100%|█████████████████████████████████████|98/98[00:00<00:00,466.73it/s]INFO:root:Creatingmanifests...100%|█████████████████████████████████████|98/98[00:00<00:00,94059.91it/s]INFO:root:Convertingmp3towavfor/tmp/cv-corpus-10.0-2022-07-04/sc/dev.tsv.100%|█████████████████████████████████████|79/79[00:00<00:00,494.77it/s]INFO:root:Creatingmanifests...100%|█████████████████████████████████████|79/79[00:00<00:00,100744.91it/s]INFO:root:Convertingmp3towavfor/tmp/cv-corpus-10.0-2022-07-04/sc/train.tsv.100%|█████████████████████████████████████|200/200[00:00<00:00,497.96it/s]INFO:root:Creatingmanifests...100%|█████████████████████████████████████|200/200[00:00<00:00,113836.45it/s]Python moduleThe code can also be used as a Python module, here are some examples:AlphabetReturns an alphabet appropriate for end-to-end speech recognition.>>>fromcvutilsimportAlphabet>>>a=Alphabet('cv')>>>a.get_alphabet()' -абвгдежзийклмнопрстуфхцчшщыэюяёҫӑӗӳ'CorporaSome miscellaneous tools for working with corpora:>>>fromcvutilsimportCorpora>>>c=Corpora('kpv')>>>c.dump_url()'https://dumps.wikimedia.org/kvwiki/latest/kvwiki-latest-pages-articles.xml.bz2'>>>c.target_segments()[]>>>c=Corpora('cv')>>>c.target_segments()['нуль','пӗрре','иккӗ','виҫҫӗ','тӑваттӑ','пиллӗк','улттӑ','ҫиччӗ','саккӑр','тӑххӑр','ҫапла','ҫук']>>>c.dump_url()'https://dumps.wikimedia.org/cvwiki/latest/cvwiki-latest-pages-articles.xml.bz2'Grapheme to phonemeFor a given token, return an approximate broad phonemised version of it.>>>fromcvutilsimportPhonemiser>>>p=Phonemiser('ab')>>>p.phonemise('гӏапынхъамыз')'ʕapənqaməz'>>>p=Phonemiser('br')>>>p.phonemise("implijout")'impliʒut'ValidatorFor a given input sentence/utterance, the validator returns either a validated and normalised
version of the string according to the validation rules, orNoneif the string cannot be
validated.>>>fromcvutilsimportValidator>>>v=Validator('ab')>>>v.validate('Аллаҳ хаҵеи-ԥҳәыси иеилыхны, аҭыԥҳацәа роума иалихыз?')'аллаҳ хаҵеи-ԥҳәыси иеилыхны аҭыԥҳацәа роума иалихыз'>>>v=Validator('br')>>>v.validate('Ha cʼhoant hocʼh eus da gendercʼhel da implijout ar servijer-mañ ?')"ha c'hoant hoc'h eus da genderc'hel da implijout ar servijer-mañ"Sentence segmentationMostly designed for use with Wikipedia, takes a paragraph and returns a list of the
sentences found within it.>>>fromcvutilsimportSegmenter>>>s=Segmenter('br')>>>para="Peurliesañ avat e kemm ar vogalennoù e c'hengerioù evit dont da vezañ heñvel ouzh ar vogalennoù en nominativ (d.l.e. ar stumm-meneg), da skouer e hungareg: Aour, tungsten, zink, uraniom, h.a., a vez kavet e kondon Bouryatia. A-bouez-bras evit armerzh ar vro eo al labour-douar ivez pa vez gounezet gwinizh ha legumaj dreist-holl. A-hend-all e vez gounezet arc'hant dre chaseal ha pesketa.">>>forsentins.segment(para):...print(sent)...Peurliesañavatekemmarvogalennoùec'hengerioù evit dont da vezañ heñvel ouzh ar vogalennoù en nominativ (d.l.e. ar stumm-meneg), da skouer e hungareg: Aour, tungsten, zink, uraniom, h.a., a vez kavet e kondon Bouryatia.A-bouez-brasevitarmerzharvroeoallabour-douarivezpavezgounezetgwinizhhalegumajdreist-holl.A-hend-allevezgounezetarc'hant dre chaseal ha pesketa.Language supportLanguageAutonymCode(CV)(WP)PhonValidAlphabetSegmentAbkhazАԥсуаabkab—✔✔✔Amharicአማርኛamh—am✔✔✔Arabicاَلْعَرَبِيَّةُaraarar—✔✔Assameseঅসমীয়াasmasas✔✔✔AsturianAsturianuastastast✔✔AzeriAzərbaycancaazeazaz✔✔BashkortБашҡортсаbakbaba✔✔✔BasaaBasaabasbas—✔✔✔BelarusianБеларуская моваbelbebe✔✔✔Bengaliবাংলাbenbnbn✔✔✔BretonBrezhonegbrebrbr✔✔✔✔BulgarianБългарскиbulbgbg✔✔✔CatalanCatalàcatcaca✔✔CzechČeštinacescscs✔✔✔ChukchiԒыгъоравэтԓьэнckt——✔✔✔ChuvashЧӑвашлаchvcvcv✔✔✔✔Hakha ChinHakha Laicnhcnh—✔✔Highland Chatino—ctp——✔✔WelshCymraegcymcycy✔✔✔Dhivehiދިވެހިdivdvdv✔✔✔✔GreekΕλληνικάellelel✔✔✔DanishDanskdandada✔✔GermanDeutschdeudede✔✔EnglishEnglishengenen—✔✔EsperantoEsperantoepoeoeo✔✔EweEʋegbeeweeeee✔✔✔SpanishEspañolspaeses✔✔✔ErzyaЭрзянь кельmyvmyvmyv✔✔EstonianEestiestetet✔✔✔BasqueEuskaraeuseueu✔✔✔✔Persianفارسیpesfafa—✔✔FinnishSuomifinfifi✔✔✔FrenchFrançaisfrafrfr—✔✔FrisianFryskfryfy-NLfy✔✔✔IgboÁsụ̀sụ́ Ìgbòiboigig✔✔✔IrishGaeilgeglega-IEga✔✔GalicianGalegoglgglgl✔✔✔GuaraníAvañeʼẽguggngn✔✔✔Hindiहिन्दीhinhihi✔✔✔HausaHarshen Hausahauhaha✔✔✔Upper SorbianHornjoserbšćinahsbhsbhsb✔✔✔HungarianMagyar nyelvhunhuhu✔✔✔ArmenianՀայերենhyehy-AMhy✔✔✔InterlinguaInterlinguainaiaia✔✔✔IndonesianBahasa indonesiaindidid✔✔✔IcelandicÍslenskaislisis✔✔ItalianItalianoitaitit✔✔✔Japanese日本語jpnjaja——Georgianქართული ენაkatkaka✔✔✔KabyleTaqbaylitkabkabkab✔✔✔KazakhҚазақшаkazkkkk✔✔✔KikuyuGĩgĩkũyũkikkiki✔✔✔KyrgyzКыргызчаkirkyky✔✔✔✔Kurmanji KurdishKurmancîkmrkuku✔✔✔Sorani Kurdishسۆرانیckbckbckb✔✔✔Komi-ZyrianКоми кывkpvkvkv✔✔LugandaLugandaluglglg✔✔✔LithuanianLietuvių kalbalitltlt✔✔✔LingalaLingálalinlnln✔✔LatvianLatviešu valodalvslvlv✔✔✔LuoDholuoluoluo—✔✔✔MacedonianМакедонскиmkdmkmk✔✔Malayalamമലയാളംmalmlml✔✔Marathiमराठीmarmrmr✔✔Hill MariМары йӹлмӹmrjmrjmrj✔✔Meadow MariОлык марийmhrmhrmhr✔✔MongolianМонгол хэлkhkmnmn✔✔✔✔MokshaМокшень кяльmdfmdfmdf✔✔✔MalteseMaltimltmtmt✔✔✔✔Yoloxóchitl Mixtecxty——✔✔DutchNederlandsnldnlnl✔✔✔ChewaChichewanyanyny✔✔✔Sierra Puebla Nahuatl—azz——✔✔✔Nepaliनेपालीnenene✔✔Norwegian NynorskNynorsknnonn-NOnn✔✔Oriyaଓଡ଼ିଆorioror✔✔✔Punjabiਪੰਜਾਬੀpanpa-INpa✔✔PolishPolskipolplpl✔✔✔PortuguesePortuguêsporptpt✔✔KʼicheʼKʼicheʼquc——✔✔✔Romansch (Sursilvan)Romontschrohrm-sursilvrm✔✔✔Romansch (Vallader)Rumantschrohrm-valladerrm✔✔✔RomanianRomâneșteronroro✔✔✔RussianРусскийrusruru✔✔KinyarwandaKinyarwandakinrwrw✔✔✔SakhaСаха тылаsahsahsah✔✔✔✔SardinianLimba sardasrdscsc✔✔Santaliᱥᱟᱱᱛᱟᱲᱤsatsatsat✔✔✔Saraikiskrskr—✔✔✔SerbianSrpskisrpsrsr✔✔✔SlovakSlovenčinaslksksk✔✔✔SlovenianSlovenščinaslvslsl✔✔✔SwahiliKiswahiliswa—sw✔✔✔SwedishSvenskaswesv-SEsv✔✔✔Tamilதமிழ்tamtata✔✔✔Thaiภาษาไทยthathth✔✔✔Tigreትግራይትtigtitig✔✔Tigrinyaትግርኛtititi✔✔TurkishTürkçeturtrtr✔✔✔✔TatarТатар телеtattttt✔✔✔✔Highland Totonac—tos——✔✔TwiTwitwtwtw✔✔✔UkrainianУкраїнська моваukrukuk✔✔✔Urduاُردُوurdurur✔✔✔Uyghurئۇيغۇر تىلىuigugug✔✔✔UzbekOʻzbekchauzbuzuz✔✔✔VietnameseTiếng Việtvievivi✔✔✔✔VoticVaďďa tšeelivotvot—✔✔WolofWolofwol—wo✔✔✔YorubaÈdè Yorùbáyor—yo✔✔✔Chinese (China)中文cmnzh-CNzh——Chinese (Hong Kong)中文cmnzh-HKzh——Chinese (Taiwan)中文cmnzh-TWzh——Frequently asked questionsWhy not use [insert better system] for [insert task here] ?There are potentially a lot of better language-specific systems for doing these tasks, but each one has
a slightly different API, so if you want to support all the Common Voice languages or even a reasonable
subset you have to learn and use the same number of language-specific APIs.The idea of these utilities is to provide adequate implementations of things are are likely to be useful
when working with all the languages in Common Voice. If you are working on a single language or have
a specific setup or are using more data than just Common Voice, maybe this isn't for you. But if you
want to just traincoqui-ai/STTon Common Voice, then maybe it is :)Why not just make the alphabet from the transcripts ?Depending on the language in Common Voice, the transcripts can contain a lot of random punctuation,
numerals, and incorrect character encodings (for example Latinçinstead of Cyrillicҫfor Chuvash). These
may look the same but will result in bigger sparsity for the model. Additionally some languages may
have several encodings of the same character, such as the apostrophe. These will ideally be normalised
before training.Also, if you are working with a single language you probably have time to look through all the transcripts
for the alphabetic symbols, but if you want to work with a large number of Common Voice languages at the
same time it's useful to have them all in one place.Hey aren't some of those languages not in Common Voice ?That's right, some of the languages are either not in Common Voice (yet!) or are in Common Voice but have
not been released yet. If I've been working with them I've included them anyway.See alsoepitran: Great grapheme to phoneme system that supports a wide
range of languages.LicenceAll the code, aside from that explicitly licensed under a different licence, is licensed under
theAGPL v 3.0.AcknowledgementsGrapheme to phoneme correspondences for the following languages fromepitran:vi,uk,kk,ky,taCode fortransducer lookupfrom Måns Huldén.Code forWikipedia extractionfrom Apertium.Code forCoqui exportfromCoqui.Code forNeMo exportfromNVIDIA