Package: unidic2ud
UniDic2UDTokenizer, POS-tagger, lemmatizer, and dependency-parser for modern and contemporary Japanese, working onUniversal Dependencies.Basic usage>>>importunidic2ud>>>nlp=unidic2ud.load("kindai")>>>s=nlp("其國を治めんと欲する者は先づ其家を齊ふ")>>>print(s)# text = 其國を治めんと欲する者は先づ其家を齊ふ1其其のDET連体詞_2det_SpaceAfter=No|Translit=ソノ2國国NOUN名詞-普通名詞-一般_4obj_SpaceAfter=No|Translit=クニ3ををADP助詞-格助詞_2case_SpaceAfter=No|Translit=ヲ4治め収めるVERB動詞-一般_7advcl_SpaceAfter=No|Translit=オサメ5んむAUX助動詞_4aux_SpaceAfter=No|Translit=ン6ととADP助詞-格助詞_4case_SpaceAfter=No|Translit=ト7欲する欲するVERB動詞-一般_8acl_SpaceAfter=No|Translit=ホッスル8者者NOUN名詞-普通名詞-一般_14nsubj_SpaceAfter=No|Translit=モノ9ははADP助詞-係助詞_8case_SpaceAfter=No|Translit=ハ10先づ先ずADV副詞_14advmod_SpaceAfter=No|Translit=マヅ11其其のDET連体詞_12det_SpaceAfter=No|Translit=ソノ12家家NOUN名詞-普通名詞-一般_14obj_SpaceAfter=No|Translit=ウチ13ををADP助詞-格助詞_12case_SpaceAfter=No|Translit=ヲ14齊ふ整えるVERB動詞-一般_0root_SpaceAfter=No|Translit=トトノフ>>>t=s[7]>>>print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)7欲する欲するVERB動詞-一般_8acl_SpaceAfter=No|Translit=ホッスル>>>print(s.to_tree())其<══╗det(決定詞)國═╗═╝<╗obj(目的語)を<╝║case(格表示)治め═╗═╗═╝<╗advcl(連用修飾節)ん<╝║║aux(動詞補助成分)と<══╝║case(格表示)欲する═══════╝<╗acl(連体修飾節)者═╗═══════╝<╗nsubj(主語)は<╝║case(格表示)先づ<══════╗║advmod(連用修飾語)其<══╗║║det(決定詞)家═╗═╝<╗║║obj(目的語)を<╝║║║case(格表示)齊ふ═════╝═╝═══╝root(親)>>>f=open("trial.svg","w")>>>f.write(s.to_svg())>>>f.close()unidic2ud.load(UniDic,UDPipe)loads a natural language processor pipeline, which usesUniDicfor tokenizer POS-tagger and lemmatizer, then usesUDPipefor dependency-parser. The defaultUDPipeisUDPipe="japanese-modern". AvailableUniDicoptions are:UniDic="gendai": Use現代書き言葉UniDic.UniDic="spoken": Use現代話し言葉UniDic.UniDic="novel": Use近現代口語小説UniDic.UniDic="qkana": Use旧仮名口語UniDic.UniDic="kindai": Use近代文語UniDic.UniDic="kinsei": Use近世江戸口語UniDic.UniDic="kyogen": Use中世口語UniDic.UniDic="wakan": Use中世文語UniDic.UniDic="wabun": Use中古和文UniDic.UniDic="manyo": Use上代語UniDic.UniDic=None: UseUDPipefor tokenizer, POS-tagger, lemmatizer, and dependency-parser.unidic2ud.UniDic2UDEntry.to_tree()has an optionto_tree(BoxDrawingWidth=2)for old terminals, whose Box Drawing characters are "fullwidth".You can simply useunidic2udon the command line:echo其國を治めんと欲する者は先づ其家を齊ふ|unidic2ud-UkindaiCaboCha emulator usage>>>importunidic2ud.cabochaasCaboCha>>>c=CaboCha.Parser("kindai")>>>s=c.parse("其國を治めんと欲する者は先づ其家を齊ふ")>>>print(s.toString(CaboCha.FORMAT_TREE_LATTICE))其-D國を-D治めんと-D欲する-D者は-------D先づ-----D其-D|家を-D齊ふEOS*01D0/00.000000其連体詞,*,*,*,*,*,其の,ソノ,*,DETO1<-det-2*12D0/10.000000國名詞,普通名詞,一般,*,*,*,国,クニ,*,NOUNO2<-obj-4を助詞,格助詞,*,*,*,*,を,ヲ,*,ADPO3<-case-2*23D0/10.000000治め動詞,一般,*,*,*,*,収める,オサメ,*,VERBO4<-advcl-7ん助動詞,*,*,*,*,*,む,ン,*,AUXO5<-aux-4と助詞,格助詞,*,*,*,*,と,ト,*,ADPO6<-case-4*34D0/00.000000欲する動詞,一般,*,*,*,*,欲する,ホッスル,*,VERBO7<-acl-8*48D0/10.000000者名詞,普通名詞,一般,*,*,*,者,モノ,*,NOUNO8<-nsubj-14は助詞,係助詞,*,*,*,*,は,ハ,*,ADPO9<-case-8*58D0/00.000000先づ副詞,*,*,*,*,*,先ず,マヅ,*,ADVO10<-advmod-14*67D0/00.000000其連体詞,*,*,*,*,*,其の,ソノ,*,DETO11<-det-12*78D0/10.000000家名詞,普通名詞,一般,*,*,*,家,ウチ,*,NOUNO12<-obj-14を助詞,格助詞,*,*,*,*,を,ヲ,*,ADPO13<-case-12*8-1D0/00.000000齊ふ動詞,一般,*,*,*,*,整える,トトノフ,*,VERBO14<-rootEOS>>>forcin[s.chunk(i)foriinrange(s.chunk_size())]:...ifc.link>=0:...print(c,"->",s.chunk(c.link))...其->國を國を->治めんと治めんと->欲する欲する->者は者は->齊ふ先づ->齊ふ其->家を家を->齊ふCaboCha.Parser(UniDic)is an alias forunidic2ud.load(UniDic,UDPipe="japanese-modern"), and its default isUniDic=None.CaboCha.Tree.toString(format)has five available formats:CaboCha.FORMAT_TREE: tree (numbered as 0)CaboCha.FORMAT_LATTICE: lattice (numbered as 1)CaboCha.FORMAT_TREE_LATTICE: tree + lattice (numbered as 2)CaboCha.FORMAT_XML: XML (numbered as 3)CaboCha.FORMAT_CONLL: Universal Dependencies CoNLL-U (numbered as 4)You can simply useudcabochaon the command line:echo其國を治めんと欲する者は先づ其家を齊ふ|udcabocha-Ukindai-f2-U UniDicspecifiesUniDic.-f formatspecifies the output format in 0 to 4 above (default is-f 0) and in 5 to 8 below:-f 5:to_tree()-f 6:to_tree(BoxDrawingWidth=2)-f 7:to_svg()-f 8:raw DOTgraph throughImmediate Catena AnalysisTrynotebookfor Google Colaboratory.Usage via spaCyIf you have already installedspaCy2.1.0 or later, you can useUniDicvia spaCy Language pipeline.>>>importunidic2ud.spacy>>>nlp=unidic2ud.spacy.load("kindai")>>>d=nlp("其國を治めんと欲する者は先づ其家を齊ふ")>>>print(unidic2ud.spacy.to_conllu(d))# text = 其國を治めんと欲する者は先づ其家を齊ふ1其其のDET連体詞_2det_SpaceAfter=No|Translit=ソノ2國国NOUN名詞-普通名詞-一般_4obj_SpaceAfter=No|Translit=クニ3ををADP助詞-格助詞_2case_SpaceAfter=No|Translit=ヲ4治め収めるVERB動詞-一般_7advcl_SpaceAfter=No|Translit=オサメ5んむAUX助動詞_4aux_SpaceAfter=No|Translit=ン6ととADP助詞-格助詞_4case_SpaceAfter=No|Translit=ト7欲する欲するVERB動詞-一般_8acl_SpaceAfter=No|Translit=ホッスル8者者NOUN名詞-普通名詞-一般_14nsubj_SpaceAfter=No|Translit=モノ9ははADP助詞-係助詞_8case_SpaceAfter=No|Translit=ハ10先づ先ずADV副詞_14advmod_SpaceAfter=No|Translit=マヅ11其其のDET連体詞_12det_SpaceAfter=No|Translit=ソノ12家家NOUN名詞-普通名詞-一般_14obj_SpaceAfter=No|Translit=ウチ13ををADP助詞-格助詞_12case_SpaceAfter=No|Translit=ヲ14齊ふ整えるVERB動詞-一般_0root_SpaceAfter=No|Translit=トトノフ>>>t=d[6]>>>print(t.i+1,t.orth_,t.lemma_,t.pos_,t.tag_,t.head.i+1,t.dep_,t.whitespace_,t.norm_)7欲する欲するVERB動詞-一般8aclホッスル>>>fromdeplacy.depreljaimportdeprelja>>>forbinunidic2ud.spacy.bunsetu_spans(d):...fortinb.lefts:...print(unidic2ud.spacy.bunsetu_span(t),"->",b,"("+deprelja[t.dep_]+")")...其->國を(決定詞)國を->治めんと(目的語)治めんと->欲する(連用修飾節)欲する->者は(連体修飾節)其->家を(決定詞)者は->齊ふ(主語)先づ->齊ふ(連用修飾語)家を->齊ふ(目的語)unidic2ud.spacy.load(UniDic,parser)loads a spaCy pipeline, which usesUniDicfor tokenizer POS-tagger and lemmatizer (as shown above), then usesparserfor dependency-parser. The defaultparserisparser="japanese-modern"and available options are:parser="ja_core_news_sm": UsespaCy Japanese model(small).parser="ja_core_news_md": Use spaCy Japanese model (middle).parser="ja_core_news_lg": Use spaCy Japanese model (large).parser="ja_ginza": UseGiNZA.parser="japanese-gsd": UseUDPipe Japanese model.parser="stanza_ja": UseStanza Japanese model.Installation for LinuxTar-ball is available for Linux, and is installed by default when you usepip:pipinstallunidic2udBy default installation,UniDicis invoked through Web APIs. If you want to invoke them locally and faster, you can downloadUniDicwhich you use just as follows:python-munidic2uddownloadkindai
python-munidic2uddictlistLicenses of dictionaries and models are: GPL/LGPL/BSD forgendaiandspoken; CC BY-NC-SA 4.0 for others.Installation for CygwinMake sure to getgcc-g++python37-pippython37-develpackages, and then:pip3.7installunidic2udUsepython3.7command inCygwininstead ofpython.Installation for Jupyter Notebook (Google Colaboratory)!pipinstallunidic2udBenchmarksResults of舞姬/雪國/荒野より-Benchmarks舞姬LASMLASBLEXUniDic="kindai"81.1370.3777.78UniDic="qkana"79.2570.3777.78UniDic="kinsei"72.2260.7164.29雪國LASMLASBLEXUniDic="qkana"89.2985.7181.63UniDic="kinsei"89.2985.7177.55UniDic="kindai"84.9681.6377.55荒野よりLASMLASBLEXUniDic="kindai"76.4461.5453.85UniDic="qkana"75.3961.5453.85UniDic="kinsei"71.8858.9751.28AuthorKoichi Yasuoka (安岡孝一)References安岡孝一:形態素解析部の付け替えによる近代日本語(旧字旧仮名)の係り受け解析, 情報処理学会研究報告, Vol.2020-CH-124「人文科学とコンピュータ」, No.3 (2020年9月5日), pp.1-8.安岡孝一:漢日英Universal Dependencies平行コーパスとその差異, 人文科学とコンピュータシンポジウム「じんもんこん2019」論文集 (2019年12月), pp.43-50.