Package: klega
Korean LExico-Grammatical Analyzer (KLEGA)A Korean lexical diversity analyzer with following features:Remove typos from the raw texts automaticallyTokenize using one of the six tokenizers (okt, komoran, mecab, kkma, hannanum fromkonlpy,stanza)Calculate lexical diversity using the packagetaaledAnalyze with and without function wordsSetupNote: The typo removal function is available only on Windows environment with Microsoft Office installed. To be able to execute the code on other OS environments, the typo removal function must be off by using the flagno-typo-removal. Refer below for the usage.Basic setupInstall the required packages using:pip install -r requirements.txtData (Input)The Korean texts (input data) for analyzing the lexical diversity must be in plain text files with the extension.txt.
You can process multiple files, which are stored in onedatadirectory.e.g.data/text1.txtdata/text2.txtdata/text3.txtUsageFor the basic usage, run:python src/klega/main.py -i [INPUT_DIR]This will process your texts in yourINPUT_DIRusing (default)okttokenizer and save the output in theresultdirectory, processing contents words only.[INPUT_DIR]must be a directory where all the text files to process are stored.TokenizersTo choose different tokenizers, use the argument-t. You can process the same input multiple times using different tokenizers. E.g. To useoktandmecab, run this command:python src/klega/main.py -i [INPUT_DIR] -t okt mecabCurrently available tokenizers are:oktmecabhannanumkomorankkmastanzaProcessing functional wordsTo process functional words as well (contents words and functional words), add-fflag in the command:python src/klega/main.py -i [INPUT_DIR] -fNote: Forstanzatokenizer,-fmust be always added.stanzadoes not have an option to extract content words only.Changing output directoryThe default output directory isresult. If you want to change the output directory, use the flag-o:python src/klega/main.py -i [INPUT_DIR] -o [OUTPUT_DIR]Turning off the typo removal function (for Mac and Linux)Currently, the typo removal function is available only on Windows environment with Microsoft Office installed. To be able to execute the code on other OS environments, the typo removal function must be off by using the flag-no-typo-removal:python src/klega/main.py -i [INPUT_DIR] -no-typo-removalExample usageIf you want to process the files in the directoryinputusing the tokenizer sethannanumandkomoran, processing contents words only, and save the output to the directoryoutput:python src/klega/main.py -i input -o output -t hannanum komoranResult (Output)Three kinds of output files are saved after a successful run.LogfileThe log filelog_[yymmdd]_[hhmm].logshows the configuration of your run, e.g. selected tokenizer, processed files, etc.Processed FilesThe tsv fileprocessed_data.tsvincludes raw texts (columnraw), list of typos removed (columntypo) and processed texts (columnprocessed) to tokenize and calculate lexical diversity. This file is useful when you want to reuse the processed texts for other text manipulation or evaluation. Note: Ifno-typo-removalset, this file is not generated.Lexical Diversity ValuesThe list of lexical diversity values are stored as a tsv format in the file[TOKENIZER]_[FUNCTION_WORD_OPTION].tsv.
E.g. the configuration of the output filehannanum_content_only.tsvishannanum, without function words.This is an example result file:Web DemoAweb demo versionof Korean LExico-Grammatical Analyzer KLEGA is available now.Currently optimized for the Chrome browser on PC. (under development)Please create a GitHub issue if you have any questions or bug-reports.Email to the writers:Sooyeon Cho&Hakyung Sung