Package: nonce2vec
nonce2vecWelcome to Nonce2Vec!The main branch of this repository now refers to the Kabbach et al. (2019) ACL SRW 2019 paperTowards incremental learning of word embeddings using context informativeness.If you are looking for the Herbelot and Baroni (2017) repository, check out theemnlp2017branch.If you use this code, please cite:@inproceedings{kabbach-etal-2019-towards,
    title = "Towards Incremental Learning of Word Embeddings Using Context Informativeness",
    author = "Kabbach, Alexandre  and
      Gulordava, Kristina  and
      Herbelot, Aur{\'e}lie",
    booktitle = "Proceedings of the 57th Conference of the Association for Computational Linguistics: Student Research Workshop",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-2022",
    pages = "162--168"}AbstractIn this paper, we investigate the task of learning word embeddings from very sparse data in an incremental, cognitively-plausible way. We focus on the notion of informativeness, that is, the idea that some content is more valuable to the learning process than other. We further highlight the challenges of online learning and argue that previous systems fall short of implementing incrementality. Concretely, we incorporate informativeness in a previously proposed model of nonce learning, using it for context selection and learning rate modulation. We test our system on the task of learning new words from definitions, as well as on the task of learning new words from potentially uninformative contexts. We demonstrate that informativeness is crucial to obtaining state-of-the-art performance in a truly incremental setup.A note on the codeWe have significantly refactored the original Nonce2Vec code in order to make replication easier and to make it work with gensim v3.x. You can use Nonce2Vec v2.x to replicate the results of the SRW paper. However, to replicate results of the original ENMLP paper, refer to Nonce2Vec v1.x found under theemnlp2017 branchas wecannotguarantee fair replication between v1.x and v2.x.InstallYou can install Nonce2Vec via pip:pip3installnonce2vecor, after a git clone, via:python3setup.pyinstallPre-requisitesTo run Nonce2Vec, you need two gensim Word2Vec models (a skipgram model and a cbow model to compute informativeness-metrics). You can download the skipgram model from:wgetbackup.3azouz.net/gensim.w2v.skipgram.model.7zand the cbow model from:wgetbackup.3azouz.net/gensim.w2v.cbow.model.7zor generate both yourself following the instructions below.Generating a Word2Vec model from a Wikipedia dumpYou can download our English Wikipedia dump of January 2019 here:wgetbackup.3azouz.net/enwiki.20190120.7zIf you want to generate a completely new (tokenized-one-sentence-per-line) dump
of Wikipedia, for English or any other language, check outWiToKit.Once you have a Wikipedia txt dump, you can generate a gensim Word2Vec skipgram model via:n2vtrain\--data/absolute/path/to/wikipedia/tokenized/text/dump\--outputdir/absolute/path/to/dir/where/to/store/w2v/model\--alpha0.025\--neg5\--window5\--sample1e-3\--epochs5\--min-count50\--size400\--num-threadsnumber_of_cpu_threads_to_use\--train-modeskipgramand a gensim Word2Vec cbow model via:n2vtrain\--data/absolute/path/to/wikipedia/tokenized/text/dump\--outputdir/absolute/path/to/dir/where/to/store/w2v/model\--alpha0.025\--neg5\--window5\--sample1e-3\--epochs5\--min-count50\--size400\--num-threadsnumber_of_cpu_threads_to_use\--train-modecbowTo check the correlation of your word2vec model(s) with the MEN dataset, run:n2vcheck-men\---model/absolute/path/to/gensim/w2v/modelRunning the codeRunning Nonce2Vec on the definitional of chimeras datasets is done via then2v testcommand. You can pass in the--reloadparameter to run inone-shotmode, without it the code runs in incremental model by default. You can further pass in the--shuffleparameter to shuffle the test set before running n2v.You will find below a list of commands corresponding to the experiments reported in the SRW 2019 paper. For example, to test the SUM CWI model (a basic sum model with context-word-informativeness-based filtering), which provides a rather robust baseline on all datasets in incremental setup, run, for the definitional dataset:n2vtest\--ondef\--model/absolute/path/to/gensim/w2v/skipgram/model\--info-model/absolute/path/to/gensim/w2v/cbow/model\--sum-only\--sum-filtercwi\--sum-threshold0To run the N2V CWI alpha model on the chimera L4 test set, with shuffling and in
one-shot evaluation setup (which provides SOTA performance), do:n2vtest\--onl4\--model/absolute/path/to/gensim/w2v/skipgram/model\--info-model/absolute/path/to/gensim/w2v/cbow/model\--sum-filtercwi\--sum-threshold0\--train-withcwi_alpha\--alpha1.0\--beta1000\--kappa1\--neg3\--epochs1\--reloadTo test N2V as-is (the original N2V code without background freezing), in incremental setup on the definitional dataset, do:n2vtest\--ondef\--model/absolute/path/to/gensim/w2v/skipgram/model\--sum-filterrandom\--sample10000\--alpha1.0\--neg3\--window15\--epochs1\--lambda70\--sample-decay1.9\--window-decay5\--replicationTo test N2V CWI init (the original N2V with CWI-based sum initialization) on the definitional dataset in one-shot evaluation setup, do:n2vtest\--ondef\--model/absolute/path/to/gensim/w2v/skipgram/model\--info-model/absolute/path/to/gensim/w2v/cbow/model\--sum-filtercwi\--sum-threshold0\--alpha1.0\--neg3\--window15\--epochs1\--lambda70\--sample-decay1.9\--window-decay5\--replication\--reload