Package: vina2vi
vina2vivina2vistands forVietnamesenoaccenttoVietnamese,which is a Python package aiming at helping foreignersdecryptVietnamese messages.Among other things, we plan to makevina2vicapable ofRestoring Vietnamese diacriticsCorrecting spellingTranslating acronyms,đổi vần, etc.InstallationDuring development, I have used Python3.10.
But I think all Python versions >= 3.8 will be fine.Run the following command to installvina2vi:pipinstallvina2viAlternatively, one can also install the latest commit from GitLab as follows.pipinstallgit+https://gitlab.com/phunc20/vina2viUsageI only work on this project on my spare time,
and work slowly. This README is meant to
get changed fast and a lot. Therefore, please pay
attention to the versions and the corresponding README.
For the moment, there is not much in
the package that is super useful. As time goes by, I will add more.I roughly classify the code into the following categoriesmodelsmetricsutilvina2vi/models/For example, one can try to play with a Transformer modelIn[1]:fromvina2vi.models.tf.tuto_transformerimportTranslatorIn[2]:translator=Translator.from_pretrained()In[3]:translator.translate("Sang nay toi dan con toi di cong vien choi.")Out[3]:'sang này tôi đàn con tôi đi công viên chơi .'In[4]:translator.translate("Truong dai hoc xa hoi va nhan van")Out[4]:'trường đại học xã hội và nhân văn'Or with a bigram modelIn[1]:fromvina2vi.models.char_based.bigramimportBigramIn[2]:bigram=Bigram.from_pretrained()In[3]:bigram.translate("Sang nay toi dan con toi di cong vien choi.")Out[3]:'Sang này tôi đãn cón tôi đi cóng viện chôi.'In[4]:bigram.translate("Truong dai hoc xa hoi va nhan van")Out[4]:'Trường đãi hôc xã hôi và nhàn vàn'In particular, the CRF model invina2vi/models/crf.pyis a direct borrow fromtrungtv. The only reason I put a GitHub installation
link, like inrequirements.txt, is that the original repo has a small bug and that it seems
that that pacakge is no longer maintained. As a result, I forked the work and made
a few commits to fix the small bug.However, it seems that PyPi does not acceptpyproject.tomlcontaining any GitHub link
as dependency. Therefore, in order to use the CRF model, please install thepyvipackage
according torequirements.txt.vina2vi/util.pyFor example, there is an utility function to help tell whether a string contains
non-Vietnamese characters,is_foreign. As the name suggests,If the string contains characters other than the modern Vietnamese alphabets,
thenis_foreignreturnsTrueIf the string consists exclusively of characters of modern Vietnamese alphabets,
thenis_foreignreturnsFalseLanguages whose alphabets are a subset of Vietnamese's are thus considered as VietnameseCurrently, we do not consider chữ Nôm as Vietnamese; maybe we will in the futureIn[1]:fromvina2vi.utilimportVietnameseIn[2]:Vietnamese.is_foreign("Российская Федерация\tRossiyskaya Federatsiya")Out[2]:TrueIn[3]:Vietnamese.is_foreign("\n\tRossiyskaya Federatsiya")Out[3]:FalseIn[4]:Vietnamese.is_foreign("Tôi nói tiếng Việt Nam\t碎呐㗂越南")Out[4]:TrueIn[5]:Vietnamese.is_foreign("Tôi nói tiếng Việt Nam\t")Out[5]:FalseThere are also four useful normalizers (from Hugging Face'stokenizerslibrary)uncased_vi_normalizercased_vi_normalizeruncased_vina_normalizercased_vina_normalizerwhich helpMake sure that similar-looking characters are not only similar but also exactly the same(In the case of(un)cased_vina_normalizer) Remove diacriticsEvaluation and Datavina2vi/metrics/evaluate_models.pyhelps give a quick overview of the models' performances.$pythonevaluate_models.pymeanmedian
baseline0.78620.7813
bigram0.84050.8399
crf_trungtv0.85910.8561
tf_tuto_transformer(cased)0.86910.8758
tf_tuto_transformer(uncased)0.89180.9101where themeanandmedianstand for the mean and median similarities calculated on some test
dataset (to be described below).vina2vi/data/: Different types of text are collected with balance --
Modern literature, lyrics, classics, etc.$wc-cvina2vi/data/*.txt3576vina2vi/data/canh_dong_bat_tan.txt2261vina2vi/data/cho_toi_xin_mot_ve_di_tuoi_tho.txt1775vina2vi/data/chuyen_tinh_nguoi_trinh_nu_ten_thi.txt3174vina2vi/data/tron_tim.txt1613vina2vi/data/truyen_kieu.txt12399total