Package: razdel
razdel— rule-based system for Russian sentence and word tokenization..Usage>>>fromrazdelimporttokenize>>>tokens=list(tokenize('Кружка-термос на 0.5л (50/64 см³, 516;...)'))>>>tokens[Substring(0,13,'Кружка-термос'),Substring(14,16,'на'),Substring(17,20,'0.5'),Substring(20,21,'л'),Substring(22,23,'(')...]>>>[_.textfor_intokens]['Кружка-термос','на','0.5','л','(','50/64','см³',',','516',';','...',')']>>>fromrazdelimportsentenize>>>text='''... - "Так в чем же дело?" - "Не ра-ду-ют".... И т. д. и т. п. В общем, вся газета... '''>>>list(sentenize(text))[Substring(1,23,'- "Так в чем же дело?"'),Substring(24,40,'- "Не ра-ду-ют".'),Substring(41,56,'И т. д. и т. п.'),Substring(57,76,'В общем, вся газета')]Installationrazdelsupports Python 3.5+ and PyPy 3.$pipinstallrazdelQuality, performanceUnfortunately, there is no single correct way to split text into sentences and tokens. For example, one may split«Как же так?! Захар...» — воскликнут Пронин.into three sentences["«Как же так?!",  "Захар...»", "— воскликнут Пронин."]whilerazdelsplits it into two["«Как же так?!", "Захар...» — воскликнут Пронин."]. What would be the correct way to tokenizerт.е.? One may split in intoт.|е.,razdelsplits intoт|.|е|..razdeltries to mimic segmentation of these 4 datasets :SynTagRus,OpenCorpora,GICRYAandRNC. These datasets mainly consist of news and fiction.razdelrules are optimized for these kinds of texts. Library may perform worse on other domains like social media, scientific articles, legal documents.We measure absolute number of errors. There are a lot of trivial cases in the tokenization task. For example, textчуть-чуть?!is not non-trivial, one may split it intoчуть|-|чуть|?|!while the correct tokenization isчуть-чуть|?!, such examples are rare. Vast majority of cases are trivial, for example textв 5 часов ...is correctly tokenized even via Python nativestr.splitintoв| |5| |часов| |.... Due to the large number of trivial case overall quality of all segmenators is high, it is hard to compare differentiate between for examlpe 99.33%, 99.95% and 99.88%, so we report the absolute number of errors.errors— number of errors. For example, consider etalon segmentation isчто-то|?, prediction isчто|-|то?, then the number of errors is 3: 1 for missing splitто?+ 2 for extra splitsчто|-|то.time— total seconds taken.spacy_tokenize,aatimofeevand others a defined innaeval/segment/models.py. Tables are computed insegment/main.ipynb.Tokenscorporasyntaggicryarncerrorstimeerrorstimeerrorstimeerrorstimere.findall(\w+|\d+|\p+)41610.526600.522770.476060.4spacy43886.221035.817404.140573.9nltk.word_tokenize142453.4608933.3134962.7414852.9mystem45145.031534.724973.720283.9mosestokenizer18862.113301.917961.621231.7segtok.word_tokenize27722.312882.317591.812291.8aatimofeev/spacy_russian_tokenizer293048.771951.167839.5268152.2koziev/rutokenizer26271.113861.028930.894110.9razdel.tokenize15102.914832.83222.021242.2Sentenciescorporasyntaggicryarncerrorstimeerrorstimeerrorstimeerrorstimere.split([.?!…])204560.965760.6100840.7233561.0segtok.split_single1900817.8442213.41597381.11642182.8mosestokenizer416668.9220825.7126636.4505607.4nltk.sent_tokenize1642010.143505.370745.6325348.9deeppavlov/rusenttokenize1019210.912107.989106.8214107.0razdel.sentenize92746.18243.9114144.5105947.5SupportChat —https://telegram.me/natural_language_processingIssues —https://github.com/natasha/razdel/issuesDevelopmentTest:pipinstall-e.
pipinstall-rrequirements/ci.txt
maketestmakeint# 2000 integration testsPackage:makeversion
gitpush
gitpush--tags

makecleanwheeluploadmystemerrors onsyntag:# see naeval/datacatsyntag_tokens.txt|razdel-ctlsample1000|razdel-ctlgen|razdel-ctldiff--showmoses_tokenize|lessNon-trivial token tests:pvdata/*_tokens.txt|razdel-ctlgen--recall|razdel-ctldiffspace_tokenize>tests.txt
pvdata/*_tokens.txt|razdel-ctlgen--precision|razdel-ctldiffre_tokenize>>tests.txtUpdate integration tests:cdrazdel/tests/data/
pvsents.txt|razdel-ctlupsentenize>t;mvtsents.txtrazdelandmosesdiff:catdata/*_tokens.txt|razdel-ctlsample1000|razdel-ctlgen|razdel-ctluptokenize|razdel-ctldiffmoses_tokenize|lessrazdelperformance:catdata/*_tokens.txt|razdel-ctlsample10000|pv-l|razdel-ctlgen|razdel-ctldifftokenize|wc-l