Package: gec-noise-generator-ko
Noise Generator for Korean Text Grammar Error Correction ModelThis model is using python-mecab-ko as the main tokenizer.If you need to know the token types check the website belowhttps://m.blog.naver.com/PostView.naver?isHttpsRedirect=true&blogId=aramjo&logNo=221404488280RequirementsPython >= 3.7pip install wget
pip install konlpy
pip install hangul_utils
pip install hangul_jamo
pip install inko
pip install g2pk
pip install gensimCurrently not available in Windows OS.Only available at Linux or ColabHow to usePreparationgit clone https://github.com/JKJIN1999/gec_noise_generator_ko.gitManualy Download Wiki.ko.vec from this linkhttps://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ko.vecPlace the wiki.ko.vec file to gec_noise_generator_ko/src/resourcesType this in your terminalpython3 gec_noise_generator_ko/src/gecnk/noise.py --e (Error type of your preference)The result will appear in results file.Requires a Text file with its directory as the First argument when calling the function.Text data should be in txt file and each sentence should be organized through each line.Functions (Error Type) (second argument)spacing_errorCreates Random spacing error according to rule written below(명사, 관형어) + (명사 , 관형어 의존명사) 인 경우 띄어쓰기를 만든다조사와 접사를 띄어써서 생기는 오류단어 가운대에 띄어쓰기를 해서 생기는 오류punctuation_errorConverts puntuation within the same type문장속의 토큰중 품사가 “S-” 로 시작하는 랜덤하게 고른 토큰을 같은 품사표 내에서 랜덤하게 변경한다.phonetic_first_error or phonetic_last_errorConverts the first or last character if it exists in phonetic data list여 > 녀 / 율 > 률remove_josa_errorRandomly remove josa from sentenceRemove token which type starts with "J"addition_errorRandomly add consonant to character which has "ㅇ" as its starting consonant or doens't have the last consonant아기 > 바기 / 다치 > 닫치separation_errorFrom the randomly selected word, decompose the letter to consonant and vowel할 > ㅎㅏㄹtyping_language_errorConvert Korean text to English text regarding to the same keyboard position한글 > gksrmf / 고양이 > rhdiddlpostposition_diff_josa_error or postposition_same_josa_errorConvert a Josa to either different or same type of Josa from the Josa dataset.를 > 을 / 에게 > 할busa_errorConvert busa "이" to "히" either way부단히 > 부단이 / 같이 > 같히middle_shiot_errorIf there is "ㅅ" as the last consonant in a word longer than 2 characters, erase the last consonant "ㅅ"숫자 > 수자 / 찻잔 > 차잔grapheme_to_phonem_errorConvert the word's textual form as it is pronounciated. If the textual outcome based on the pronounciation of that specific word is not same as the current textual form of the word행복하다 > 행보카다 / 같이 > 가치overlapping_sound_errorIf there are two continuous Tensed Consonant letter positioning in each first consonant letter, convert the second letter's first consonant into Basic Consonant.딱딱하다 > 딱닥하다 / 쌉쌀하다 > 쌉살하다final_suffix_errorConvert the randomly selected final suffix into a different final suffix that doesn't match the original suffix하겠습니다 > 하겠습네까 / 하고있다 > 하고있니mag_error & maj_errorBy using the similarity function in Gensim KeyedVector, convert the typical pumsa type (mag or maj) into a different busa.mag 얼마나 > 어떻게 / maj 하지만 > 그러나polite_speech_errorMisusage of two types of polite speech josa, nominative josa and adverbal josa이,가 > 께서 / 에게 > 께