Package: lexikos
Lexikos - λεξικός /lek.si.kós/A collection of pronunciation dictionaries and neural grapheme-to-phoneme models.Install LexikosInstall from PyPIpipinstalllexikosEditable install from Sourcegitclonehttps://github.com/bookbot-hive/lexikos.git
pipinstall-elexikosUsageLexicon>>>fromlexikosimportLexicon>>>lexicon=Lexicon()>>>print(lexicon["added"]){'ˈæ d ɪ d','ˈæ ɾ ə d','æ ɾ ɪ d','a d ɪ d','ˈa d ɪ d','æ ɾ ə d','ˈa d ə d','a d ə d','ˈæ d ə d','æ d ə d','æ d ɪ d','ˈæ ɾ ɪ d'}>>>print(lexicon["runner"]){'ɹ ʌ n ɚ','ɹ ʌ n ə','ɹ ʌ n ɝ','ˈr ʌ n ɝ'}>>>print(lexicon["water"]){'ˈʋ aː ʈ ə r ɯ','ˈw oː t ə','w ɑ t ə ɹ','ˈw aː ʈ ə r ɯ','ˈw ɔ t ɝ','w ɔ t ə ɹ','ˈw ɑ t ə ɹ','w ɔ t ɝ','w ɑ ɾ ɚ','ˈw ɑ ɾ ɚ','ˈʋ ɔ ʈ ə r','w ɔ ɾ ɚ','w ɔː t ə','ˈw oː ɾ ə','ˈw ɔ ʈ ə r'}To get a lexicon where phonemes are normalized (diacritics removed, digraphs split):>>>fromlexikosimportLexicon>>>lexicon=Lexicon(normalize_phonemes=True)>>>print(lexicon["added"]){'æ ɾ ɪ d','a d ɪ d','a d ə d','æ ɾ ə d','æ d ə d','æ d ɪ d'}>>>print(lexicon["runner"]){'ɹ ʌ n ɚ','ɹ ʌ n ə','r ʌ n ɝ','ɹ ʌ n ɝ'}>>>print(lexicon["water"]){'w o ɾ ə','w ɔ t ə','ʋ ɔ ʈ ə r','w a ʈ ə r ɯ','w ɔ t ə ɹ','ʋ a ʈ ə r ɯ','w ɑ ɾ ɚ','w o t ə','w ɔ t ɝ','w ɔ ʈ ə r','w ɔ ɾ ɚ','w ɑ t ə ɹ'}To include synthetic (non-dictionary-based) pronunciations:>>>fromlexikosimportLexicon>>>lexicon=Lexicon(include_synthetic=True)>>>print(lexicon["athletic"]){'æ t l ɛ t ɪ k','æ θ ˈl ɛ t ɪ k','æ θ l ɛ t ɪ k'}Phonemization>>>fromlexikosimportG2p>>>g2p=G2p(lang="en-us")>>>g2p("Hello there! $100 is not a lot of money in 2023.")['h ɛ l o ʊ','ð ɛ ə ɹ','w ʌ n','h ʌ n d ɹ ɪ d','d ɑ l ɚ z','ɪ z','n ɒ t','ə','l ɑ t','ʌ v','m ʌ n i','ɪ n','t w ɛ n t i','t w ɛ n t i','θ ɹ iː']>>>g2p=G2p(lang="en-au")>>>g2p("Hi there mate! Have a g'day!")['h a ɪ','θ ɛ ə ɹ','m e ɪ t','h e ɪ v','ə','ɡ ə ˈd æ ɪ']Dictionaries & ModelsEnglish(en)LanguageDictionaryPhone SetCorpusG2P ModelenWikipronIPALinkbookbot/byt5-small-wikipron-eng-latnEnglish(en-US)LanguageDictionaryPhone SetCorpusG2P Modelen-USCMU DictARPAExternal Linkbookbot/byt5-small-cmudicten-USCMU Dict IPAIPAExternal Linken-USCharsiuG2PIPAExternal Linkcharsiu/g2p_multilingual_byT5_small_100en-US (Broad)WikipronIPAExternal Linkbookbot/byt5-small-wikipron-eng-latn-us-broaden-US (Narrow)WikipronIPAExternal Linken-USLibriSpeechIPALinkEnglish(en-UK)LanguageDictionaryPhone SetCorpusG2P Modelen-UKCharsiuG2PIPAExternal Linkcharsiu/g2p_multilingual_byT5_small_100en-UK (Broad)WikipronIPAExternal Linkbookbot/byt5-small-wikipron-eng-latn-uk-broaden-UK (Narrow)WikipronIPAExternal LinkEnglish(en-AU)LanguageDictionaryPhone SetCorpusG2P Modelen-AU (Broad)WikipronIPALinkbookbot/byt5-small-wikipron-eng-latn-au-broaden-AU (Narrow)WikipronIPALinken-AUAusTalkIPALinken-AUSC-CWIPALinkEnglish(en-CA)LanguageDictionaryPhone SetCorpusG2P Modelen-CA (Broad)WikipronIPALinkbookbot/byt5-small-wikipron-eng-latn-ca-broaden-CA (Narrow)WikipronIPALinkEnglish(en-NZ)LanguageDictionaryPhone SetCorpusG2P Modelen-NZ (Broad)WikipronIPALinkbookbot/byt5-small-wikipron-eng-latn-nz-broaden-NZ (Narrow)WikipronIPALinkEnglish(en-IN)LanguageDictionaryPhone SetCorpusG2P Modelen-IN (Broad)WikipronIPALinkbookbot/byt5-small-wikipron-eng-latn-in-broaden-IN (Narrow)WikipronIPALinkTraining G2P ModelWe modified the sequence-to-sequence training script of🤗 HuggingFacefor the purpose of training G2P models. Refer to theirinstallation requirementsfor more details.Training a new G2P model generally follow this recipe:python run_translation.py \+   --model_name_or_path $PRETRAINED_MODEL \+   --dataset_name $DATASET_NAME \--output_dir $OUTPUT_DIR \--per_device_train_batch_size 128 \--per_device_eval_batch_size 32 \--learning_rate 2e-4 \--lr_scheduler_type linear \--warmup_ratio 0.1 \--num_train_epochs 10 \--evaluation_strategy epoch \--save_strategy epoch \--logging_strategy epoch \--max_source_length 64 \--max_target_length 64 \--val_max_target_length 64 \--pad_to_max_length True \--overwrite_output_dir \--do_train --do_eval \--bf16 \--predict_with_generate \--report_to tensorboard \--push_to_hub \+   --hub_model_id $HUB_MODEL_ID \--use_auth_tokenExample: Fine-tune ByT5 on CMU Dictpythonrun_translation.py\--model_name_or_pathgoogle/byt5-small\--dataset_namebookbot/cmudict-0.7b\--output_dir./byt5-small-cmudict\--per_device_train_batch_size128\--per_device_eval_batch_size32\--learning_rate2e-4\--lr_scheduler_typelinear\--warmup_ratio0.1\--num_train_epochs10\--evaluation_strategyepoch\--save_strategyepoch\--logging_strategyepoch\--max_source_length64\--max_target_length64\--val_max_target_length64\--pad_to_max_lengthTrue\--overwrite_output_dir\--do_train--do_eval\--bf16\--predict_with_generate\--report_totensorboard\--push_to_hub\--hub_model_idbookbot/byt5-small-cmudict\--use_auth_tokenEvaluating G2P ModelThen to evaluate:python eval.py \+   --model $PRETRAINED_MODEL \+   --dataset_name $DATASET_NAME \--source_text_column_name source \--target_text_column_name target \--max_length 64 \--batch_size 64Example: Evaluate ByT5 on CMU Dictpythoneval.py\--modelbookbot/byt5-small-cmudict\--dataset_namebookbot/cmudict-0.7b\--source_text_column_namesource\--target_text_column_nametarget\--max_length64\--batch_size64Corpus RoadmapWikipronLanguage FamilyCodeRegionCorpusG2P ModelAfrican Englishen-ZASouth AfricaAustralian Englishen-AUAustralia✅✅East Asian Englishen-CN, en-HK, en-JP, en-KR, en-TWChina, Hong Kong, Japan, South Korea, TaiwanEuropean Englishen-UK, en-HU, en-IEUnited Kingdom, Hungary, Ireland🚧🚧Mexican Englishen-MXMexicoNew Zealand Englishen-NZNew Zealand✅✅North Americanen-CA, en-USCanada, United States✅✅Middle Eastern Englishen-EG, en-ILEgypt, IsraelSoutheast Asianen-TH, en-ID, en-MY, en-PH, en-SGThailand, Indonesia, Malaysia, Philippines, SingaporeSouth Asian Englishen-INIndia✅✅ResourcesCharsiuG2PMicrosoft CNTKCMU Pronouncing Dictionary - IPAWikipronReferences@inproceedings{lee-etal-2020-massively,title="Massively Multilingual Pronunciation Modeling with {W}iki{P}ron",author="Lee, Jackson L.  andAshby, Lucas F.E.  andGarza, M. Elizabeth  andLee-Sikka, Yeonju  andMiller, Sean  andWong, Alan  andMcCarthy, Arya D.  andGorman, Kyle",booktitle="Proceedings of LREC",year="2020",publisher="European Language Resources Association",pages="4223--4228",}@misc{zhu2022byt5,title={ByT5 model for massively multilingual grapheme-to-phoneme conversion},author={Jian Zhu and Cong Zhang and David Jurgens},year={2022},eprint={2204.03067},archivePrefix={arXiv},primaryClass={cs.CL}}