Package: prenlp
PreNLPPreprocessing Library for Natural Language ProcessingInstallationRequirementsPython >= 3.6Mecab morphological analyzer for Koreansh scripts/install_mecab.sh
# Only for Mac OS users, run the code below before run install_mecab.sh script.
# export MACOSX_DEPLOYMENT_TARGET=10.10
# CFLAGS='-stdlib=libc++' pip install konlpyC++ Build tools for fastTextg++ >= 4.7.2 or clang >= 3.3ForWindows,Visual Studio C++is recommended.With pipprenlp can be installed using pip as follows:pip install prenlpUsageDataDataset LoadingPopular datasets for NLP tasks are provided in prenlp. All datasets is stored in/.datadirectory.Sentiment Analysis: IMDb, NSMCLanguage Modeling: WikiText-2, WikiText-103, WikiText-ko, NamuWiki-koDatasetLanguageArticlesSentencesTokensVocabSizeWikiText-2English720-2,551,84333,27813.3MBWikiText-103English28,595-103,690,236267,735517.4MBWikiText-koKorean477,9462,333,930131,184,780662,949667MBNamuWiki-koKorean661,03216,288,639715,535,7781,130,0083.3GBWikiText-ko+NamuWiki-koKorean1,138,97818,622,569846,720,5581,360,5383.95GBGeneral use cases are as follows:WikiText-2 / WikiText-103>>>wikitext2=prenlp.data.WikiText2()>>>len(wikitext2)3>>>train,valid,test=prenlp.data.WikiText2()>>>train[0]'= Valkyria Chronicles III ='IMDB>>>imdb_train,imdb_test=prenlp.data.IMDB()>>>imdb_train[0]["Minor Spoilers<br /><br />Alison Parker (Cristina Raines) is a successful top model, living with the lawyer Michael Lerman (Chris Sarandon) in his apartment. She tried to commit ...",'pos']NormalizationFrequently used normalization functions for text pre-processing are provided in prenlp.url, HTML tag, emoticon, email, phone number, etc.General use cases are as follows:>>>fromprenlp.dataimportNormalizer>>>normalizer=Normalizer(url_repl='[URL]',tag_repl='[TAG]',emoji_repl='[EMOJI]',email_repl='[EMAIL]',tel_repl='[TEL]',image_repl='[IMG]')>>>normalizer.normalize('Visit this link for more details: https://github.com/')'Visit this link for more details: [URL]'>>>normalizer.normalize('Use HTML with the desired attributes: <img src="cat.jpg" height="100" />')'Use HTML with the desired attributes: [TAG]'>>>normalizer.normalize('Hello 🤩, I love you 💓 !')'Hello [EMOJI], I love you [EMOJI] !'>>>normalizer.normalize('Contact me at lyeoni.g@gmail.com')'Contact me at [EMAIL]'>>>normalizer.normalize('Call +82 10-1234-5678')'Call [TEL]'>>>normalizer.normalize('Download our logo image, logo123.png, with transparent background.')'Download our logo image, [IMG], with transparent background.'TokenizerFrequently used (subword) tokenizers for text pre-processing are provided in prenlp.SentencePiece, NLTKMosesTokenizer, MecabSentencePiece>>>fromprenlp.tokenizerimportSentencePiece>>>SentencePiece.train(input='corpus.txt',model_prefix='sentencepiece',vocab_size=10000)>>>tokenizer=SentencePiece.load('sentencepiece.model')>>>tokenizer('Time is the most valuable thing a man can spend.')['▁Time','▁is','▁the','▁most','▁valuable','▁thing','▁a','▁man','▁can','▁spend','.']>>>tokenizer.tokenize('Time is the most valuable thing a man can spend.')['▁Time','▁is','▁the','▁most','▁valuable','▁thing','▁a','▁man','▁can','▁spend','.']>>>tokenizer.detokenize(['▁Time','▁is','▁the','▁most','▁valuable','▁thing','▁a','▁man','▁can','▁spend','.'])Timeisthemostvaluablethingamancanspend.Moses tokenizer>>>fromprenlp.tokenizerimportNLTKMosesTokenizer>>>tokenizer=NLTKMosesTokenizer()>>>tokenizer('Time is the most valuable thing a man can spend.')['Time','is','the','most','valuable','thing','a','man','can','spend','.']Comparisons with tokenizers on IMDbBelow figure shows the classification accuracy from various tokenizer.Code:NLTKMosesTokenizer,SentencePieceComparisons with tokenizers on NSMC (Korean IMDb)Below figure shows the classification accuracy from various tokenizer.Code:Mecab,SentencePieceAuthorHoyeon Lee @lyeoniemail :lyeoni.g@gmail.comfacebook :https://www.facebook.com/lyeoni.f