Package: natto-py
What is natto-py?A package leveraging FFI (foreign function interface),natto-pycombines
thePythonprogramming language withMeCab, the part-of-speech and
morphological analyzer for the Japanese language. No compiler is necessary, as
it isnota C extension.natto-pywill run on Mac OS, Windows and
*nix.You can learn more aboutnatto-py at GitHub.If you are still usingPython 2 after sunset, please stick with versionnatto-py==0.9.2.Requirementsnatto-pyrequires the following:An existing installation ofMeCab 0.996A system dictionary, likeIPA,JumanorUnidiccffi 0.8.6or greaterThe following Python 3 versions are supported:Python 3.7Python 3.8Python 3.9Python 3.10For Python 2, please use version0.9.2.InstallationInstallnatto-pyas you would any other Python package:$pipinstallnatto-pyThis will automatically install thecffipackage, whichnatto-pyuses
to bind to themecablibrary.Automatic ConfigurationAs long as themecab(andmecab-configfor *nix and Mac OS)
executables are on yourPATH,natto-pydoes not require any explicit
configuration.On *nix and Mac OS, it queriesmecab-configto discover the path to thelibmecab.soorlibmecab.dylib, respectively.On Windows, it queries the Windows Registry to locate the MeCab installation folder.In order to convert character encodings to/from Unicode,natto-pywill examine the charset of themecabsystem dictionary.Explicit configuration via MECAB_PATH and MECAB_CHARSETIfnatto-pyfor some reason cannot locate themecablibrary,
or if it cannot determine the correct charset used internally bymecab, then you will need to set theMECAB_PATHandMECAB_CHARSETenvironment variables.Set theMECAB_PATHenvironment variable to the exact name/path to yourmecablibrary.Set theMECAB_CHARSETenvironment variable to thecharsetcharacter encoding used by your system dictionary.e.g., for Mac OS:exportMECAB_PATH=/usr/local/Cellar/mecab/0.996/lib/libmecab.dylibexportMECAB_CHARSET=utf8e.g., for bash on UNIX/Linux:exportMECAB_PATH=/usr/local/lib/libmecab.soexportMECAB_CHARSET=euc-jpe.g., on Windows:setMECAB_PATH=C:\Program Files\MeCab\bin\libmecab.dllsetMECAB_CHARSET=shift-jise.g., from within a Python program:importosos.environ['MECAB_PATH']='/usr/local/lib/libmecab.so'os.environ['MECAB_CHARSET']='utf-16'UsageHere’s a very quick guide to usingnatto-py.Instantiate a reference to themecablibrary, and display some details:fromnattoimportMeCabnm=MeCab()print(nm)# displays details about the MeCab instance<natto.mecab.MeCabmodel=<cdata'mecab_model_t *'0x801c16300>,tagger=<cdata'mecab_t *'0x801c17470>,lattice=<cdata'mecab_lattice_t *'0x801c196c0>,libpath="/usr/local/lib/libmecab.so",options={},dicts=[<natto.dictionary.DictionaryInfodictionary='mecab_dictionary_info_t *'0x801c19540>,filepath="/usr/local/lib/mecab/dic/ipadic/sys.dic",charset=utf8,type=0],version=0.996>Display details about themecabsystem dictionary used:sysdic=nm.dicts[0]print(sysdic)# displays the MeCab system dictionary info<natto.dictionary.DictionaryInfodictionary='mecab_dictionary_info_t *'0x801c19540>,filepath="/usr/local/lib/mecab/dic/ipadic/sys.dic",charset=utf8,type=0>Parse Japanese text and send the MeCab result as a single string tostdout:print(nm.parse('ピンチの時には必ずヒーローが現れる。'))# MeCab result as a single stringピンチ名詞,一般,*,*,*,*,ピンチ,ピンチ,ピンチの助詞,連体化,*,*,*,*,の,ノ,ノ時名詞,非自立,副詞可能,*,*,*,時,トキ,トキに助詞,格助詞,一般,*,*,*,に,ニ,ニは助詞,係助詞,*,*,*,*,は,ハ,ワ必ず副詞,助詞類接続,*,*,*,*,必ず,カナラズ,カナラズヒーロー名詞,一般,*,*,*,*,ヒーロー,ヒーロー,ヒーローが助詞,格助詞,一般,*,*,*,が,ガ,ガ現れる動詞,自立,*,*,一段,基本形,現れる,アラワレル,アラワレル。記号,句点,*,*,*,*,。,。,。EOSNext, try parsing the text with MeCab node parsing. A generator yielding the
MeCabNode instances lets you efficiently iterate over the output without first
materializing each and every resulting MeCabNode instance. The MeCabNode
instances yielded allow access to more detailed information about each
morpheme.Here we use aPython with-statementto automatically clean up after we
finish node parsing with the MeCab tagger. This is the recommended approach
for usingnatto-pyin a production environment:# Use a Python with-statement to ensure mecab_destroy is invoked#withMeCab()asnm:forninnm.parse('ピンチの時には必ずヒーローが現れる。',as_nodes=True):...# ignore any end-of-sentence nodes...ifnotn.is_eos():...print('{}\t{}'.format(n.surface,n.cost))...ピンチ3348の3722時5176に5083は5305必ず7525ヒーロー11363が10508現れる10841。7127MeCab output formatting is extremely flexible and is highly recommended for
any serious natural language processing task. Rather than parsing the MeCab
output as a single, large string, use MeCab’s--node-formatoption
(short form-F) to customize the node’sfeatureattribute.morpheme surfacepart-of-speechpart-of-speech IDpronunciationIt is good practice when using--node-formatto also specify node
formatting in the case where the morpheme cannot be found in the dictionary,
by using--unk-format(short form-U).This example formats the nodefeatureto capture the items above as a
comma-separated value:# MeCab options used:## -F    ... short-form of --node-format# %m    ... morpheme surface# %f[0] ... part-of-speech# %h    ... part-of-speech id (ipadic)# %f[8] ... pronunciation## -U    ... short-form of --unk-format#           output ?,?,?,? for morphemes not in dictionary#withMeCab(r'-F%m,%f[0],%h,%f[8]\n -U?,?,?,?\n')asnm:forninnm.parse('ピンチの時には必ずヒーローが現れる。',as_nodes=True):...# only normal nodes, ignore any end-of-sentence and unknown nodes...ifn.is_nor():...print(n.feature)...ピンチ,名詞,38,ピンチの,助詞,24,ノ時,名詞,66,トキに,助詞,13,ニは,助詞,16,ワ必ず,副詞,35,カナラズヒーロー,名詞,38,ヒーローが,助詞,13,ガ現れる,動詞,31,アラワレル。,記号,7,。Partial parsing(制約付き解析), allows you to pass hints to MeCab on
how to tokenize morphemes when parsing. Most useful are boundary constraint
parsing and feature constraint parsing.With boundary constraint parsing, you can specify either a compiledreregular expression object or a string to tell MeCab where the boundaries of
a morpheme should be. Use theboundary_constraintskeyword. For hints on
tokenization, please seeRegular expression operationsandre.finditerin particular.This example uses the-Fnode-format option to customize the resultingMeCabNodefeature attribute to extract:%m- morpheme surface%f[0]- node part-of-speech%s- nodestatstatus value, 1 isunknownNote that any such morphemes captured will have nodestatstatus of 1 (unknown):importrewithMeCab(r'-F%m,\s%f[0],\s%s\n')asnm:text='俺は努力したよっ？ お前の10倍、いや100倍1000倍したよっ！'# capture 10倍, 100倍 and 1000倍 as single parts-of-speechpattern=re.compile('10+倍')forninnm.parse(text,boundary_constraints=pattern,as_nodes=True):...print(n.feature)...俺,名詞,0は,助詞,0努力,名詞,0し,動詞,0たよっ,動詞,0？,記号,0お前,名詞,0の,助詞,010倍,名詞,1、,記号,0いや,接続詞,0100倍,名詞,11000倍,名詞,1し,動詞,0たよっ,動詞,0！,記号,0EOSWith feature constraint parsing, you can provide instructions to MeCab
on what feature to use for a matching morpheme. Use thefeature_constraintskeyword to pass in atuplecontaining elements
that themselves aretupleinstances with a specific morpheme (str)
and a corresponding feature (str), in order of constraint precedence:withMeCab(r'-F%m,\s%f[0],\s%s\n')asnm:text='心の中で3回唱え、 ヒーロー見参！ヒーロー見参！ヒーロー見参！'features=(('ヒーロー見参','感動詞'),)forninnm.parse(text,feature_constraints=features,as_nodes=True):...print(n.feature)...心,名詞,0の,助詞,0中,名詞,0で,助詞,03,名詞,1回,名詞,0唱え,動詞,0、,記号,0ヒーロー見参,感動詞,1！,記号,0ヒーロー見参,感動詞,1！,記号,0ヒーロー見参,感動詞,1！,記号,0EOSLearn MoreExamples and more detailed information aboutnatto-pycan be found on theproject Wiki.Working code in Jupyter notebook form can be found under thisproject’s notebooks directory.API documentation on Read the Docs.Contributing to natto-pyUsegitandcheck out the latest code at GitHubto make sure the
feature hasn’t been implemented or the bug hasn’t been fixed yet.Browse the issue trackerto make sure someone already hasn’t requested it
and/or contributed it.Fork the project.Start a feature/bugfix branch.Commit and push until you are happy with your contribution.Make sure to add tests for it. This is important so I don’t break it in a
future version unintentionally.Please try not to mess with thesetup.py,CHANGELOG, or version
files. If you must have your own version, that is fine, but please isolate
to its own commit so I can cherry-pick around it.This project uses the following packages for development:Sphinxfor document generationtwinefor secure uploads during releaseunittestfor unit tests, as it is very natural and easy-to-usePyYAMLfor data loading during testsChangelogPlease see theCHANGELOGfor the release history.CopyrightCopyright © 2022, Brooke M. Fujita. All rights reserved. Please see
theLICENSEfile for further details.