Package: toiro
toiroToiro is a comparison tool of Japanese tokenizers.Compare the processing speed of tokenizersCompare the words segmented in tokenizersCompare the performance of tokenizers by benchmarking application tasks (e.g., text classification)It also provides useful functions for natural language processing in Japanese.Data downloader for Japanese text corporaPreprocessor of these corporaText classifier for Japanese text (e.g., SVM, BERT)InstallationPython 3.6+ is required. You can install toiro with the following command.Janomeis included in the default installation.pipinstalltoiroAdding a tokenizer to toiroIf you want to add a tokenizer to toiro, please install it individually.
This is an example of addingSudachiPyandnagisato toiro.pipinstallsudachipysudachidict_core
pipinstallnagisaHow to install other tokenizersmecab-python3pip install mecab-python3GiNZApip install spacy ginzaspaCypip install spacy[ja]KyTeaYou need to install KyTea. Please refer tohere.pip install kyteaJuman++ v2You need to install Juman++ v2. Please refer tohere.pip install pyknpSentencePiecepip install sentencepiecefugashi-ipadicpip install fugashi ipadicfugashi-unidicpip install fugashi unidic-litetinysegmenterpip install tinysegmenter3If you want to install all the tokonizers at once, please use the following command.pipinstalltoiro[all_tokenizers]Getting startedYou can check the available tokonizers in your Python environment.fromtoiroimporttokenizersavailable_tokenizers=tokenizers.available_tokenizers()print(available_tokenizers)Toiro supports 12 different Japanese tokonizers. This is an example of adding SudachiPy and nagisa.{'nagisa':{'is_available':True,'version':'0.2.7'},'janome':{'is_available':True,'version':'0.3.10'},'mecab-python3':{'is_available':False,'version':False},'sudachipy':{'is_available':True,'version':'0.4.9'},'spacy':{'is_available':False,'version':False},'ginza':{'is_available':False,'version':False},'kytea':{'is_available':False,'version':False},'jumanpp':{'is_available':False,'version':False},'sentencepiece':{'is_available':False,'version':False},'fugashi-ipadic':{'is_available':False,'version':False},'fugashi-unidic':{'is_available':False,'version':False},'tinysegmenter':{'is_available':False,'version':False}}Download the livedoor news corpus and compare the processing speed of tokenizers.fromtoiroimporttokenizersfromtoiroimportdatadownloader# A list of avaliable corpora in toirocorpora=datadownloader.available_corpus()print(corpora)#=> ['livedoor_news_corpus', 'yahoo_movie_reviews', 'amazon_reviews']# Download the livedoor news corpus and load it as pandas.DataFramecorpus=corpora[0]datadownloader.download_corpus(corpus)train_df,dev_df,test_df=datadownloader.load_corpus(corpus)texts=train_df[1]# Compare the processing speed of tokenizersreport=tokenizers.compare(texts)#=> [1/3] Tokenizer: janome#=> 100%|███████████████████| 5900/5900 [00:07<00:00, 746.21it/s]#=> [2/3] Tokenizer: nagisa#=> 100%|███████████████████| 5900/5900 [00:15<00:00, 370.83it/s]#=> [3/3] Tokenizer: sudachipy#=> 100%|███████████████████| 5900/5900 [00:08<00:00, 696.68it/s]print(report){'execution_environment':{'python_version':'3.7.8.final.0 (64 bit)','arch':'X86_64','brand_raw':'Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz','count':8},'data':{'number_of_sentences':5900,'average_length':37.69593220338983},'janome':{'elapsed_time':9.114670515060425},'nagisa':{'elapsed_time':15.873093605041504},'sudachipy':{'elapsed_time':9.05256724357605}}# Compare the words segmented in tokenizerstext="都庁所在地は新宿区。"tokenizers.print_words(text,delimiter="|")#=>        janome: 都庁|所在地|は|新宿|区|。#=>        nagisa: 都庁|所在|地|は|新宿|区|。#=>     sudachipy: 都庁|所在地|は|新宿区|。Run toiro in DockerYou can use all tokenizers by building a docker container from Docker Hub.dockerrun--rm-ittaishii/toiro/bin/bashHow to run the Python interpreter in the Docker containerRun the Python interpreter.root@cdd2ad2d7092:/workspace# python3Compare the words segmented in tokenizers>>>fromtoiroimporttokenizers>>>text="都庁所在地は新宿区。">>>tokenizers.print_words(text,delimiter="|")mecab-python3:都庁|所在地|は|新宿|区|。janome:都庁|所在地|は|新宿|区|。nagisa:都庁|所在|地|は|新宿|区|。sudachipy:都庁|所在地|は|新宿区|。spacy:都庁|所在|地|は|新宿|区|。ginza:都庁|所在地|は|新宿区|。kytea:都庁|所在|地|は|新宿|区|。jumanpp:都庁|所在|地|は|新宿|区|。sentencepiece:▁|都|庁|所在地|は|新宿|区|。fugashi-ipadic:都庁|所在地|は|新宿|区|。fugashi-unidic:都庁|所在|地|は|新宿|区|。tinysegmenter:都庁所|在地|は|新宿|区|。Get more information about toiroThe slides at PyCon JP 2020Speaker DeckPyConJP2020_Online.ipynbTutorials in Japanese01_getting_started_ja.ipynb05_svm_vs_bert_benchmarking_application_tasks_ja.ipynbContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.