Package: corus
Links to publicly available Russian corpora + code for loading and parsing.20+ datasets, 350Gb+ of text.UsageFor example lets usedump of lenta.ru by @yutkin. Manually download the archive (link in theReferencesection):wgethttps://github.com/yutkin/Lenta.Ru-News-Dataset/releases/download/v1.0/lenta-ru-news.csv.gzUsecorusto load the data:>>>fromcorusimportload_lenta>>>path='lenta-ru-news.csv.gz'>>>records=load_lenta(path)>>>next(records)LentaRecord(url='https://lenta.ru/news/2018/12/14/cancer/',title='Названы регионы России с\xa0самой высокой смертностью от\xa0рака',text='Вице-премьер по социальным вопросам Татьяна Голикова рассказала, в каких регионах России зафиксирована наиболее высокая смертность от рака, сооб...',topic='Россия',tags='Общество')Iterate over texts:>>>records=load_lenta(path)>>>forrecordinrecords:...text=record.text......For links to other datasets and their loaders see theReferencesection.DocumentationMaterials are in Russian:Corus page on natasha.github.ioCorus section of Datafest 2020 talkInstallcorussupports Python 3.5+, PyPy 3.$pipinstallcorusReferenceDatasetAPIfrom corus importTagsTextsUncompressedDescriptionLenta.ruLenta.ru v1.0load_lenta#news739 3511.66 Gbwget https://github.com/yutkin/Lenta.Ru-News-Dataset/releases/download/v1.0/lenta-ru-news.csv.gzLenta.ru v1.1+load_lenta2#news800 9751.94 Gbwget https://github.com/yutkin/Lenta.Ru-News-Dataset/releases/download/v1.1/lenta-ru-news.csv.bz2Lib.rus.ecload_librusec#fiction301 871144.92 GbDump of lib.rus.ec prepared for RUSSE workshopwget http://panchenko.me/data/russe/librusec_fb2.plain.gzRossiya Segodnyaload_ria_raw#load_ria#news1 003 8693.70 Gbwget https://github.com/RossiyaSegodnya/ria_news_dataset/raw/master/ria.json.gzMokoron Russian Twitter Corpusload_mokoron#socialsentiment17 633 4171.86 GbRussian Twitter sentiment markupManually download https://www.dropbox.com/s/9egqjszeicki4ho/db.sqlWikipediaload_wiki#1 541 40112.94 GbRussian Wiki dumpwget https://dumps.wikimedia.org/ruwiki/latest/ruwiki-latest-pages-articles.xml.bz2GramEval2020load_gramru#162 37230.04 Mbwget https://github.com/dialogue-evaluation/GramEval2020/archive/master.zipunzip master.zipmv GramEval2020-master/dataTrain trainmv GramEval2020-master/dataOpenTest devrm -r master.zip GramEval2020-masterwget https://github.com/AlexeySorokin/GramEval2020/raw/master/data/GramEval_private_test.conlluOpenCorporaload_corpora#morph4 03020.21 Mbwget http://opencorpora.org/files/export/annot/annot.opcorpora.xml.zipRusVectores SimLex-965load_simlex#embsimwget https://rusvectores.org/static/testsets/ru_simlex965_tagged.tsvwget https://rusvectores.org/static/testsets/ru_simlex965.tsvOmnia Russicaload_omnia#morphwebfiction489.62 GbTaiga + Wiki + Araneum. Read "Even larger Russian corpus" https://events.spbu.ru/eventsContent/events/2019/corpora/corp_sborn.pdfManually download http://bit.ly/2ZT4BY9factRuEval-2016load_factru#nernews254969.27 KbManual PER, LOC, ORG markup prepared for 2016 Dialog competitionwget https://github.com/dialogue-evaluation/factRuEval-2016/archive/master.zipunzip master.ziprm master.zipGareevload_gareev#nernews97455.02 KbManual PER, ORG markup (no LOC)Email Rinat Gareev (gareev-rm@yandex.ru) ask for datasettar -xvf rus-ner-news-corpus.iob.tar.gzrm rus-ner-news-corpus.iob.tar.gzCollection5load_ne5#nernews1 0002.96 MbNews articles with manual PER, LOC, ORG markupwget http://www.labinform.ru/pub/named_entities/collection5.zipunzip collection5.ziprm collection5.zipWiNERload_wikiner#ner203 28736.15 MbSentences from Wiki auto annotated with PER, LOC, ORG tagswget https://github.com/dice-group/FOX/raw/master/input/Wikiner/aij-wikiner-ru-wp3.bz2BSNLP-2019load_bsnlp#ner4641.16 MbMarkup prepared for 2019 BSNLP Shared Taskwget http://bsnlp.cs.helsinki.fi/TRAININGDATA_BSNLP_2019_shared_task.zipwget http://bsnlp.cs.helsinki.fi/TESTDATA_BSNLP_2019_shared_task.zipunzip TRAININGDATA_BSNLP_2019_shared_task.zipunzip TESTDATA_BSNLP_2019_shared_task.zip -d test_pl_cs_ru_bgrm TRAININGDATA_BSNLP_2019_shared_task.zip TESTDATA_BSNLP_2019_shared_task.zipPersons-1000load_persons#nernews1 0002.96 MbSame as Collection5, only PER markup + normalized nameswget http://ai-center.botik.ru/Airec/ai-resources/Persons-1000.zipThe Russian Drug Reaction Corpus (RuDReC)load_rudrec#ner4 8091.73 KbRuDReC is a new partially annotated corpus of consumer reviews in Russian about pharmaceutical products for the detection of health-related named entities and the effectiveness of pharmaceutical products. Here you can download and work with the annotated part, to get the raw part (1.4M reviews) please refer to https://github.com/cimm-kzn/RuDReC.wget https://github.com/cimm-kzn/RuDReC/raw/master/data/rudrec_annotated.jsonTaigaLarge collection of Russian texts from various sources: news sites, magazines, literacy, social networkswget https://linghub.ru/static/Taiga/retagged_taiga.tar.gztar -xzvf retagged_taiga.tar.gzArzamasload_taiga_arzamas#news3114.50 MbFontankaload_taiga_fontanka#news342 683786.23 MbInterfaxload_taiga_interfax#news46 42977.55 MbKPload_taiga_kp#news45 50361.79 MbLentaload_taiga_lenta#news36 44695.15 MbTaiga/N+1load_taiga_nplus1#news7 69624.96 MbMagazinesload_taiga_magazines#39 8902.19 GbSubtitlesload_taiga_subtitles#19 011909.08 MbSocialload_taiga_social#social1 876 442648.18 MbProzaload_taiga_proza#fiction1 732 43438.25 GbStihiload_taiga_stihi#9 157 68612.80 GbRussian NLP DatasetsSeveral Russian news datasets from webhose.io, lenta.ru and other news sites.Newsload_buriy_news#news2 154 8016.84 GbDump of top 40 news + 20 fashion news sites.wget https://github.com/buriy/russian-nlp-datasets/releases/download/r4/news-articles-2014.tar.bz2wget https://github.com/buriy/russian-nlp-datasets/releases/download/r4/news-articles-2015-part1.tar.bz2wget https://github.com/buriy/russian-nlp-datasets/releases/download/r4/news-articles-2015-part2.tar.bz2Webhoseload_buriy_webhose#news285 965859.32 MbDump from webhose.io, 300 sources for one month.wget https://github.com/buriy/russian-nlp-datasets/releases/download/r4/webhose-2016.tar.bz2ODS #proj_news_vizSeveral news sites scraped by members of #proj_news_viz ODS project.Interfaxload_ods_interfax#news543 9611.22 Gbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/interfax.csv.gzGazetaload_ods_gazeta#news865 8471.63 Gbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/gazeta.csv.gzIzvestiaload_ods_izvestia#news86 601307.19 Mbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/iz.csv.gzMeduzaload_ods_meduza#news71 806270.11 Mbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/meduza.csv.gzRIAload_ods_ria#news101 543233.88 Mbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/ria.csv.gzRussia Todayload_ods_rt#news106 644187.12 Mbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/rt.csv.gzTASSload_ods_tass#news1 135 6353.27 Gbwget https://github.com/ods-ai-ml4sg/proj_news_viz/releases/download/data/tass-001.csv.gzUniversal DependenciesGSDload_ud_gsd#morphsyntax5 0301.01 Mbwget https://github.com/UniversalDependencies/UD_Russian-GSD/raw/master/ru_gsd-ud-dev.conlluwget https://github.com/UniversalDependencies/UD_Russian-GSD/raw/master/ru_gsd-ud-test.conlluwget https://github.com/UniversalDependencies/UD_Russian-GSD/raw/master/ru_gsd-ud-train.conlluTaigaload_ud_taiga#morphsyntax3 264353.80 Kbwget https://github.com/UniversalDependencies/UD_Russian-Taiga/raw/master/ru_taiga-ud-dev.conlluwget https://github.com/UniversalDependencies/UD_Russian-Taiga/raw/master/ru_taiga-ud-test.conlluwget https://github.com/UniversalDependencies/UD_Russian-Taiga/raw/master/ru_taiga-ud-train.conlluPUDload_ud_pud#morphsyntax1 000207.78 Kbwget https://github.com/UniversalDependencies/UD_Russian-PUD/raw/master/ru_pud-ud-test.conlluSynTagRusload_ud_syntag#morphsyntax61 88911.33 Mbwget https://github.com/UniversalDependencies/UD_Russian-SynTagRus/raw/master/ru_syntagrus-ud-dev.conlluwget https://github.com/UniversalDependencies/UD_Russian-SynTagRus/raw/master/ru_syntagrus-ud-test.conlluwget https://github.com/UniversalDependencies/UD_Russian-SynTagRus/raw/master/ru_syntagrus-ud-train.conllumorphoRuEval-2017General Internet-Corpusload_morphoru_gicrya#morph83 14810.58 Mbwget https://github.com/dialogue-evaluation/morphoRuEval-2017/raw/master/GIKRYA_texts_new.zipunzip GIKRYA_texts_new.ziprm GIKRYA_texts_new.zipRussian National Corpusload_morphoru_rnc#morph98 89212.71 Mbwget https://github.com/dialogue-evaluation/morphoRuEval-2017/raw/master/RNC_texts.rarunrar x RNC_texts.rarrm RNC_texts.rarOpenCorporaload_morphoru_corpora#morph38 5104.80 Mbwget https://github.com/dialogue-evaluation/morphoRuEval-2017/raw/master/OpenCorpora_Texts.rarunrar x OpenCorpora_Texts.rarrm OpenCorpora_Texts.rarRUSSE Russian Semantic RelatednessHJ: Human Judgements of Word Pairsload_russe_hj#embsimwget https://github.com/nlpub/russe-evaluation/raw/master/russe/evaluation/hj.csvRT: Synonyms and Hypernyms from the Thesaurus RuThesload_russe_rt#embsimwget https://raw.githubusercontent.com/nlpub/russe-evaluation/master/russe/evaluation/rt.csvAE: Cognitive Associations from the Sociation.org Experimentload_russe_ae#embsimwget https://github.com/nlpub/russe-evaluation/raw/master/russe/evaluation/ae-train.csvwget https://github.com/nlpub/russe-evaluation/raw/master/russe/evaluation/ae-test.csvwget https://raw.githubusercontent.com/nlpub/russe-evaluation/master/russe/evaluation/ae2.csvToloka DatasetsLexical Relations from the Wisdom of the Crowd (LRWC)load_toloka_lrwc#embsimwget https://tlk.s3.yandex.net/dataset/LRWC.zipunzip LRWC.ziprm LRWC.zipThe Russian Adverse Drug Reaction Corpus of Tweets (RuADReCT)load_ruadrect#social9 5152.09 MbThis corpus was developed for the Social Media Mining for Health Applications (#SMM4H) Shared Task 2020wget https://github.com/cimm-kzn/RuDReC/raw/master/data/RuADReCT.zipunzip RuADReCT.ziprm RuADReCT.zipSupportChat —https://t.me/natural_language_processingIssues —https://github.com/natasha/corus/issuesCommercial support —https://lab.alexkuk.ruAdd new sourceImplementcorus/sources/<source>.pyAdd import intocorus/sources/__init__.pyAdd meta intocorus/source/meta.pyAdd example intodocs.ipynb(check meta table is correct)Run tests (readme is updated)DevelopmentDev envpython-mvenv~/.venvs/natasha-corussource~/.venvs/natasha-corus/bin/activate

pipinstall-rrequirements/dev.txt
pipinstall-e.

python-mipykernelinstall--user--namenatasha-corusLint + update docsmakelint
makeexec-docsRelease# Update setup.py versiongitcommit-am'Up version'gittagv0.10.0

gitpush
gitpush--tags