Package: navec
Navec is a library of pretrained word embeddings for russian language. It shows competitive or better results thanRusVectores, loads ~10 times faster (~1 sec), takes ~10 times less space (~50 MB). Seenatasha.github.io articlefor more info.Navec = large russian text datasets + vanila GloVe + quantizationDownloadsHow to read model filename:navec_hudlit_v1_12B_500K_300d_100q.tar
                 |    |    |    |
                 |    |    |     ---- 100 dimentions after quantization
                 |    |     --------- original vectors have 300 dimentions
                 |     -------------- vocab size is 500 000 words + 2 for <unk>, <pad>
                  ------------------- dataset of 12 billion tokens was usedCurrently two models are published:ModelSizeDescriptionSourcesnavec_hudlit_v1_12B_500K_300d_100q.tar#50MBShould be used by default. Shows best results onintrinsic evaluations. Model was trained on large corpus of russian literature (~150GB).librusecnavec_news_v1_1B_250K_300d_100q.tar#25MBTry to use this model to news texts. It is two times smaller than `hudlit` but covers same 98% of words in news articles.lentariataiga_fontankaburiy_newsburiy_webhoseods_gazetaods_interfaxInstallationNavec supports Pyton 3.5+ and PyPy 3.$pipinstallnavecUsageFirst downloadhudlitemdeddings (see the table above):wgethttps://storage.yandexcloud.net/natasha-navec/packs/navec_hudlit_v1_12B_500K_300d_100q.tarLoad tar-archive withNavec.load, it takes ~1s and ~100MB of RAM:>>>fromnavecimportNavec>>>path='hudlit_12B_500K_300d_100q.tar'>>>navec=Navec.load(path)Thennaveccan be used as a dict object:>>>navec['навек']array([0.3955571,0.11600914,0.24605067,-0.35206917,-0.08932345,0.3382279,-0.5457616,0.07472657,-0.4753835,-0.3330848,...>>>'нааавееек'innavecFalse>>>navec.get('нааавееек')NoneTo get an index of word, usenavec.vocab:>>>navec.vocab['навек']225823>>>navec.vocab.get('наааавеeeк',navec.vocab.unk_id)500000# == navec.vocab['<unk>']There are two special words in vocab,<unk>and<pad>:>>>navec['<unk>']array([3.69125791e-02,9.32818875e-02,2.01917738e-02,...>>>navec['<pad>']array([0.,0.,0.,0.,0.,0.,...To use Navec in PyTorch model there is a Slovnet module:>>>importtorch>>>fromslovnet.model.embimportNavecEmbedding>>>emb=NavecEmbedding(navec)>>>input=torch.tensor([1,2,0])>>>output=emb(input)>>>output.shapetorch.Size([3,300])>>>outputtensor([[4.2000e-01,3.6666e-01,1.7728e-01,-3.8719e-01,-1.0762e-01,1.6954e-01,-4.6063e-01,5.4519e-01,-2.1212e-01,2.0965e-01,1.9658e-01,2.7807e-01,-2.3802e-01,3.5155e-01,1.4491e-02,...EvaluationLet's compore Navec to top 5 RusVectores models (based onsimlexandhjeval datasets). In each column top 3 results are highlighted.init— time it takes to load model file to RAM.tayga_upos_skipgram_300_2_2019word2vec binary file takes 5 seconds to load withgensim.KeyedVectors.load_word2vec_format.tayga_none_fasttextcbow_300_10_2019fastText large ~2.7 GB file takes 8 seconds. Navechudlitwith vocab 2 times larger than previous two takes 1 second.get— time is takes to query embedding vector for a single word. Word2vec models win here, to fetch a vector they basically dodict.get. FastText and Navec for every query do extra computation. FastText extracts and sums word ngrams, Navec unpacks vector from quantization table. In practice query to embeddings table is small compared to all other computation in network.disk— model file size. It is convenient for deployment and distribution to have small models. Notice thathudlitmodel is 4-20 times smaller with vocab size 2 times bigger.ram— space model takes in RAM after loading. It is convenient to have small memory footprint to fit more computation on single server.vocab— number of words in vocab, number of embedding vectors. Since Navec vectors table takes less space we can have larger vocab. With 500K vocabhudlitmodel has ~2% OVV rate on average.typeinit, sget, µsdisk, mbram, mbvocabhudlit_12B_500K_300d_100qnavec1.121.650.695.3500Knews_1B_250K_300d_100qnavec0.820.725.447.7250Kruscorpora_upos_cbow_300_20_2019w2v3.31.4220.6236.1189Kruwikiruscorpora_upos_skipgram_300_2_2019w2v5.01.5290.0309.4248Ktayga_upos_skipgram_300_2_2019w2v5.21.4290.7310.9249Ktayga_none_fasttextcbow_300_10_2019fasttext8.013.42741.92746.9192Karaneum_none_fasttextcbow_300_5_2018fasttext16.410.62752.12754.7195KNow let's look at intrinsic evaluation scores. Navechudlitmodel does not show best results on all datasets but it is usually in top 3. We'll use 6 datasets:simlex965,hj— two small datasets (965 and 398 tests respectively) used by RusVectores, see thetheir paperfor more info. Metric is spearman correlation, other datasets use average precision.rt,ae,ae2— large datasets (114066, 22919, 86772 tests) from RUSSE workshop, seeproject descriptionfor more.lrwc— relatively new dataset by Yandex.Toloka, seetheir page.typesimlexhjrtaeae2lrwchudlit_12B_500K_300d_100qnavec0.3100.7070.8420.9310.9230.604news_1B_250K_300d_100qnavec0.2300.5900.7840.8660.8610.589ruscorpora_upos_cbow_300_20_2019w2v0.3590.6850.8520.7580.8960.602ruwikiruscorpora_upos_skipgram_300_2_2019w2v0.3210.7230.8170.8010.8600.629tayga_upos_skipgram_300_2_2019w2v0.4290.7490.8710.7710.8990.639tayga_none_fasttextcbow_300_10_2019fasttext0.3690.6390.7930.6820.8130.536araneum_none_fasttextcbow_300_5_2018fasttext0.3490.6710.8010.7060.7930.579SupportChat —https://telegram.me/natural_language_processingIssues —https://github.com/natasha/navec/issuesCommercial support —https://lab.alexkuk.ruDevelopmentTestmaketestPackagemakeversion
gitpush
gitpush--tags

makecleanwheeluploadNotice! All commands belows use code fromnavec/train, it is not under CI, it works only with Python 3, it is expected user is familiar with source code. We use Yandex Cloud Compute and Object Storage.Create remote workerTo compute cooc (large HDD, 1Tb for librusec).yccomputeinstancecreate\--nameworker\--zoneru-central1-a\--network-interfacesubnet-name=default,nat-ip-version=ipv4\--create-boot-diskimage-folder-id=standard-images,image-family=ubuntu-1804,type=network-hdd,size=1000\--memory8\--cores2\--core-fraction100\--ssh-key~/.ssh/id_rsa.pub\--folder-namedefault\--preemptible# in case computation takes <24hTo fit embedings (multiple cores). HDD should be > cooc.bin * 3 (for suffle + tmp)yccomputeinstancecreate\--nameworker\--zoneru-central1-a\--network-interfacesubnet-name=default,nat-ip-version=ipv4\--create-boot-diskimage-folder-id=standard-images,image-family=ubuntu-1804,type=network-hdd,size=700\--memory16\--cores16\--core-fraction100\--ssh-key~/.ssh/id_rsa.pub\--folder-namedefault\--preemptibleSetup machineyccomputeinstancelist--folder-namedefault
sshyc-user@123.123.123.123

sudolocale-genen_US.UTF-8
sudotimedatectlset-timezoneEurope/Moscow
sudoapt-getupdate
sudoDEBIAN_FRONTEND=noninteractiveapt-getinstall-ylanguage-pack-rupython3-pipscreenunzipgitpvcmake

wgethttps://nlp.stanford.edu/software/GloVe-1.2.zip
unzipGloVe-1.2.zip
rmGloVe-1.2.zip
mvGloVe-1.2glovecdglove
makecd..exportGLOVE_DIR=~/glove/build

gitclonehttps://github.com/natasha/navec.git
sudo-Hpip3install-enavec
sudo-Hpip3install-rnavec/requirements/train.txt

screen
ctrladRemove instanceyccomputeinstancelist--folder-namedefault
yccomputeinstancedeletexxxxxxxxxEnv, used bynavec-train s3|vocab|cooc|embexportS3_KEY=_XxXXXxxx_XXXxxxxXxxxexportS3_SECRET=XXxxx_XXXXXXxxxxxxXXXXxxXXx-XxxXXxxxXexportS3_BUCKET=XXXXXXXexportGLOVE_DIR=~/path/to/glove/buildShare text data (see corus)navec-trains3uploadlibrusec_fb2.plain.gzsources/librusec.gz
navec-trains3uploadtaiga/proza_ru.zipsources/taiga_proza.zip

navec-trains3uploadruwiki-latest-pages-articles.xml.bz2sources/wiki.xml.bz2

navec-trains3uploadlenta-ru-news.csv.gzsources/lenta.csv.gz
navec-trains3uploadria.json.gzsources/ria.json.gz
navec-trains3uploadtaiga/Fontanka.tar.gzsources/taiga_fontanka.tar.gz
navec-trains3uploadburiy/news-articles-2014.tar.bz2sources/buriy_news1.tar.bz2
navec-trains3uploadburiy/news-articles-2015-part1.tar.bz2sources/buriy_news2.tar.bz2
navec-trains3uploadburiy/news-articles-2015-part2.tar.bz2sources/buriy_news3.tar.bz2
navec-trains3uploadburiy/webhose-2016.tar.bz2sources/buriy_webhose.tar.bz2
navec-trains3uploadods/gazeta_v1.csv.zipsources/ods_gazeta.csv.zip
navec-trains3uploadods/interfax_v1.csv.zipsources/ods_interfax.csv.zip

navec-trains3downloadsources/librusec.gz
navec-trains3downloadsources/taiga_proza.zip

navec-trains3downloadsources/wiki.xml.bz2

navec-trains3downloadsources/lenta.csv.gz
navec-trains3downloadsources/ria.json.gz
navec-trains3downloadsources/taiga_fontanka.tar.gz
navec-trains3downloadsources/buriy_news1.tar.bz2
navec-trains3downloadsources/buriy_news2.tar.bz2
navec-trains3downloadsources/buriy_news3.tar.bz2
navec-trains3downloadsources/buriy_webhose.tar.bz2
navec-trains3downloadsources/ods_gazeta.csv.zip
navec-trains3downloadsources/ods_interfax.csv.zipText to tokensnavec-traincorpuslibruseclibrusec.gz|pv|navec-traintokenize>tokens.txt# ~12B wordsnavec-traincorpustaiga_prozataiga_proza.zip|pv|navec-traintokenize>tokens.txt# ~3Bnavec-traincorpuswikiwiki.xml.bz2|pv|navec-traintokenize>tokens.txt# ~0.5Bnavec-traincorpuslentalenta.csv.gz|pv|navec-traintokenize>>tokens.txt
navec-traincorpusriaria.json.gz|pv|navec-traintokenize>>tokens.txt
navec-traincorpustaiga_fontankataiga_fontanka.tar.gz|pv|navec-traintokenize>>tokens.txt
navec-traincorpusburiy_newsburiy_news1.tar.bz2|pv|navec-traintokenize>>tokens.txt
navec-traincorpusburiy_newsburiy_news2.tar.bz2|pv|navec-traintokenize>>tokens.txt
navec-traincorpusburiy_newsburiy_news3.tar.bz2|pv|navec-traintokenize>>tokens.txt
navec-traincorpusburiy_webhoseburiy_webhose.tar.bz2|pv|navec-traintokenize>>tokens.txt
navec-traincorpusods_gazetaods_gazeta.csv.zip|pv|navec-traintokenize>>tokens.txt
navec-traincorpusods_interfaxods_interfax.csv.zip|pv|navec-traintokenize>>tokens.txt# ~1Bpvtokens.txt|gzip>tokens.txt.gz
navec-trains3uploadtokens.txt.gzlibrusec_tokens.txt.gz

navec-trains3uploadtokens.txttaiga_proza_tokens.txt
navec-trains3uploadtokens.txtnews_tokens.txt
navec-trains3uploadtokens.txtwiki_tokens.txtTokens to vocabpvtokens.txt\|navec-trainvocabcount\>full_vocab.txt

pvfull_vocab.txt\|navec-trainvocabquantile# librusec# ...# 0.970      325 882# 0.980      511 542# 0.990    1 122 624# 1.000   22 129 654# taiga_proza# ...# 0.960      229 906# 0.970      321 810# 0.980      517 647# 0.990    1 224 277# 1.000   14 302 409# wiki# ...# 0.950     380 134# 0.960     519 817# 0.970     757 561# 0.980   1 223 201# 0.990   2 422 265# 1.000   6 664 630# news# ...# 0.970    163 833# 0.980    243 903# 0.990    462 361# 1.000  3 744 070# threashold at ~0.98# librusec 500000# taiga_proza 500000# wiki 750000# news 250000catfull_vocab.txt\|head-500000\|LC_ALL=Csort\>vocab.txt

navec-trains3uploadfull_vocab.txtlibrusec_full_vocab.txt
navec-trains3uploadvocab.txtlibrusec_vocab.txt

navec-trains3uploadfull_vocab.txttaiga_proza_full_vocab.txt
navec-trains3uploadvocab.txttaiga_proza_vocab.txt

navec-trains3uploadfull_vocab.txtwiki_full_vocab.txt
navec-trains3uploadvocab.txtwiki_vocab.txt

navec-trains3uploadfull_vocab.txtnews_full_vocab.txt
navec-trains3uploadvocab.txtnews_vocab.txtCompute coocurence pairs# Default limit on max number of open files is 1024, merge fails if# number of chunks is largeulimit-n# 1024sudonano/etc/security/limits.conf

*softnofile65535*hardnofile65535# reloginulimit-n# 65535pvtokens.txt\|navec-traincooccountvocab.txt--memory7--window10\>cooc.bin# Monitorls/tmp/cooc_*
tail-c16cooc.bin|navec-traincoocparse

navec-trains3uploadcooc.binlibrusec_cooc.bin
navec-trains3uploadcooc.bintaiga_proza_cooc.bin
navec-trains3uploadcooc.binwiki_cooc.bin
navec-trains3uploadcooc.binnews_cooc.binMerge (did not give much boost compared to plain librusec, so all_vocab.txt, all_cooc.bin not used below)foriinlibrusectaiga_prozawikinews;donavec-trains3download$i_vocab.txt;navec-trains3download$i_cooc.bin;donenavec-trainmergevocabs\librusec_vocab.txt\taiga_proza_vocab.txt\wiki_vocab.txt\news_vocab.txt\|pv>vocab.txt

navec-trainmergecoocsvocab.txt\librusec_cooc.bin:librusec_vocab.txt\taiga_proza_cooc.bin:taiga_proza_vocab.txt\wiki_cooc.bin:wiki_vocab.txt\news_cooc.bin:news_vocab.txt\|pv>cooc.bin

navec-trains3uploadvocab.txtall_vocab.txt
navec-trains3uploadcooc.binall_cooc.binCompute embedingsnavec-trains3downloadlibrusec_vocab.txtvocab.txt
navec-trains3downloadlibrusec_cooc.bincooc.bin

navec-trains3downloadwiki_vocab.txtvocab.txt
navec-trains3downloadwiki_cooc.bincooc.bin

navec-trains3downloadnews_vocab.txtvocab.txt
navec-trains3downloadnews_cooc.bincooc.bin

pvcooc.bin\|navec-traincoocshuffle--memory15\>shuf_cooc.bin# Search dim with best scoreforiin100200300400500600;donavec-trainembshuf_cooc.binvocab.txtemb_${i}d.txt--dim$i--threads10--iterations2;done# 300 has ok score. 400, 500 are a bit better, but too heavynavec-trainembshuf_cooc.binvocab.txtemb.txt--dim300--threads16--iterations15navec-trains3uploademb.txtlibrusec_emb.txt
navec-trains3uploademb.txtwiki_emb.txt
navec-trains3uploademb.txtnews_emb.txtQuantizenavec-trains3downloadlibrusec_emb.txtemb.txt
navec-trains3downloadwiki_emb.txtemb.txt
navec-trains3downloadnews_emb.txtemb.txt# Search for best compression that has still ok scoreforiin150100756050;dopvemb.txt|navec-trainpqfit$i--sample100000--iterations15>pq_${i}q.bin;done# 100 is <1% worse on eval but much lighterpvemb.txt|navec-trainpqfit100--sample100000--iterations20>pq.bin

navec-trainpqpad<pq.bin>t;mvtpq.bin

navec-trains3uploadpq.binlibrusec_pq.bin
navec-trains3uploadpq.binwiki_pq.bin
navec-trains3uploadpq.binnews_pq.binPacknavec-train s3 download librusec_pq.bin pq.bin
navec-train s3 download librusec_vocab.txt vocab.txt

navec-train s3 download news_pq.bin pq.bin
navec-train s3 download news_vocab.txt vocab.txt

navec-train vocab pack < vocab.txt > vocab.bin

navec-train pack vocab.bin pq.bin hudlit_v1_12B_500K_300d_100q
navec-train s3 upload navec_hudlit_v1_12B_500K_300d_100q.tar packs/navec_hudlit_v1_12B_500K_300d_100q.tar

navec-train pack vocab.bin pq.bin news_v1_1B_250K_300d_100q
navec-train s3 upload navec_news_v1_1B_250K_300d_100q.tar packs/navec_news_v1_1B_250K_300d_100q.tarPublishnavec-train s3 download packs/navec_hudlit_v1_12B_500K_300d_100q.tar
navec-train s3 download packs/navec_news_v1_1B_250K_300d_100q.tar