Package: naeval
Naeval — comparing quality and performance of NLP systems for Russian language. Naeval is used to evaluateproject Natashacomponents:Razdel,Navec,Slovnet:TokenizationSeeRazdel evalualtion sectionfor more info.corporasyntaggicryarncerrorstimeerrorstimeerrorstimeerrorstimere.findall(\w+|\d+|\p+)41610.526600.522770.476060.4spacy43886.221035.817404.140573.9nltk.word_tokenize142453.4608933.3134962.7414852.9mystem45145.031534.724973.720283.9mosestokenizer18862.113301.917961.621231.7segtok.word_tokenize27722.312882.317591.812291.8aatimofeev/spacy_russian_tokenizer293048.771951.167839.5268152.2koziev/rutokenizer26271.113861.028930.894110.9razdel.tokenize15102.914832.83222.021242.2Sentence segmentationcorporasyntaggicryarncerrorstimeerrorstimeerrorstimeerrorstimere.split([.?!…])204560.965760.6100840.7233561.0segtok.split_single1900817.8442213.41597381.11642182.8mosestokenizer416668.9220825.7126636.4505607.4nltk.sent_tokenize1642010.143505.370745.6325348.9deeppavlov/rusenttokenize1019210.912107.989106.8214107.0razdel.sentenize92746.18243.9114144.5105947.5Pretrained embeddingsSeeNavec evalualtion sectionfor more info.typeinit, sget, µsdisk, mbram, mbvocabruscorpora_upos_cbow_300_20_2019w2v12.11.6220.6236.1189Kruwikiruscorpora_upos_skipgram_300_2_2019w2v15.71.7290.0309.4248Ktayga_upos_skipgram_300_2_2019w2v15.71.2290.7310.9249Ktayga_none_fasttextcbow_300_10_2019fasttext11.314.32741.92746.9192Karaneum_none_fasttextcbow_300_5_2018fasttext7.815.42752.12754.7195Khudlit_12B_500K_300d_100qnavec1.019.950.695.3500Knews_1B_250K_300d_100qnavec0.520.325.447.7250Ktypesimlexhjrtaeae2lrwcruscorpora_upos_cbow_300_20_2019w2v0.3590.6850.8520.7580.8960.602ruwikiruscorpora_upos_skipgram_300_2_2019w2v0.3210.7230.8170.8010.8600.629tayga_upos_skipgram_300_2_2019w2v0.4290.7490.8710.7710.8990.639tayga_none_fasttextcbow_300_10_2019fasttext0.3690.6390.7930.6820.8130.536araneum_none_fasttextcbow_300_5_2018fasttext0.3490.6710.8010.7060.7930.579hudlit_12B_500K_300d_100qnavec0.3100.7070.8420.9310.9230.604news_1B_250K_300d_100qnavec0.2300.5900.7840.8660.8610.589Morphology taggersnewswikifictionsocialpoetryrupostagger0.6730.6450.6610.6410.636rnnmorph0.8960.8120.8900.8600.838maru0.8940.8080.8870.8610.840udpipe0.9180.8110.9570.8700.776spacy0.9190.8120.9380.8360.729deeppavlov0.9400.8410.9440.8700.857deeppavlov_bert0.9510.8680.9640.8920.865init, sdisk, mbram, mbspeed, it/srupostagger4.8311848.0rnnmorph8.71028916.6maru15.84437036.4udpipe6.94524256.2spacy10.98957930.6deeppavlov4.0321024090.0 (gpu)deeppavlov_bert20.01393870485.0 (gpu)Syntax parsernewswikifictionsocialpoetryuaslasuaslasuaslasuaslasuaslasudpipe0.8730.8230.6220.5310.9100.8760.7000.6240.6250.534spacy0.8760.8180.7700.6650.8800.8330.7570.6660.6570.544deeppavlov_bert0.9620.9100.8820.7860.9630.9290.8440.7610.7840.691init, sdisk, mbram, mbspeed, it/sudpipe6.94524256.2spacy10.98957931.6deeppavlov_bert34.01427870475.0 (gpu)NERSeeSlovnet evalualtion sectionfor more info.factrugareevne5bsnlpf1PERLOCORGPERORGPERLOCORGPERLOCORGdeeppavlov0.9100.8860.7420.9440.7980.9420.9190.8810.8660.7670.624deeppavlov_bert0.9710.9280.8250.9800.9160.9970.9900.9760.9540.8400.741pullenti0.9050.8140.6860.9390.6390.9520.8620.6830.9000.7690.566texterra0.9000.8000.5970.8880.5610.9010.7770.5940.8580.7830.548tomita0.9290.9210.9450.881natasha0.8670.7530.2970.8730.3470.8520.7090.3940.8360.7550.350mitie0.8880.8610.5320.8490.4520.7530.6420.4320.7360.8010.524init, sdisk, mbram, mbspeed, articles/sdeeppavlov5.91024307224.3 (gpu)deeppavlov_bert34.52048614413.1 (gpu)pullenti2.9162536.0texterra47.619333794.0tomita2.0646329.8natasha2.011608.8mitie28.332726132.8