Package: pytorch-nce
An NCE implementation in pytorchAbout NCENoise Contrastive Estimation (NCE) is an approximation method that is used to work around
the huge computational cost of large softmax layer. The basic idea is to convert the prediction
problem into classification problem at training stage. It has been proved that these two criterions
converges to the same minimal point as long as noise distribution is close enough to real one.NCE bridges the gap between generative models and discriminative models, rather than simply speedup
the softmax layer. With NCE, you can turn almost anything into posterior with less effort (I think).Refs:NCE:http://www.cs.helsinki.fi/u/ahyvarin/papers/Gutmann10AISTATS.pdfNCE on rnnlm:https://pdfs.semanticscholar.org/144e/357b1339c27cce7a1e69f0899c21d8140c1f.pdfComparison with other methodsA review of softmax speedup methods:http://ruder.io/word-embeddings-softmax/NCE vs. IS (Importance Sampling): Nce is a binary classification while IS is sort of multi-class
classification problem.http://demo.clab.cs.cmu.edu/cdyer/nce_notes.pdfNCE vs. GAN (Generative Adversarial Network):https://arxiv.org/abs/1412.6515On improving NCESampling methodsIn NCE, unigram distribution is usually used to approximate the noise distribution because it's fast to
sample from. Sampling from a unigram is equal to multinomial sampling, which is of complexity $O(\log(N))$
via binary search tree. The cost of sampling becomes significant when noise ratio increases.Since the unigram distribution can be obtained before training and remains unchanged across training,
some works are proposed to make use of this property to speedup the sampling procedure. Alias method is
one of them.By constructing data structures, alias method can reduce the sampling complexity from $O(log(N))$ to $O(1)$,
and it's easy to parallelize.Refs:alias method:https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/Generic NCE (full-NCE)Conventional NCE only perform the contrasting on linear(softmax) layer, that is, given an input of a
linear layer, the model outputs are $p(noise|input)$ and $p(target|input)$. In fact NCE can be applied
to more general situations where models are capable to output likelihood values for both real data and
noise data.In this code base, I use a variant of generic NCE named full-NCE (f-NCE) to clarify. Unlike normal NCE,
f-NCE samples the noises at input embedding.Refs:whole sentence language model by IBM (ICASSP2018)Bi-LSTM language model by speechlab,SJTU (ICSLP2016?)Batched NCEConventional NCE requires different noise samples per data token. Such computational pattern is not fully
GPU-efficient because it needs batched matrix multiplication. A trick is to share the noise samples across
the whole mini-batch, thus sparse batched matrix multiplication is converted to more efficient
dense matrix multiplication. The batched NCE is already supported by Tensorflow.A more aggressive approach is to called self contrasting (named by myself). Instead of sampling from noise
distribution, the noises are simply the other training tokens the within the same mini-batch.Ref:batched NCEhttps://arxiv.org/pdf/1708.05997.pdfself contrasting:https://www.isi.edu/natural-language/mt/simple-fast-noise.pdfRun the word language model exampleThere's an example illustrating how to use the NCE module inexamplefolder.
This example is forked from the pytorch/examples repo.RequirementsPlease runpip install -r requirementsfirst to see if you have the required python lib.tqdmis used for process bar during trainingdillis a more flexible replacement for pickleNCE related Arguments--nce: whether to use NCE as approximation--noise-ratio <50>: numbers of noise samples per batch, the noise is shared among the
tokens in a single batch, for training speed.--norm-term <9>: the constant normalization termLn(z)--index-module <linear>: index module to use for NCE module (currently
 and  available,  does not support PPL calculating )--train: train or just evaluation existing model--vocab <None>: use vocabulary file if specified, otherwise use the words in train.txt--loss [full, nce, sampled, mix]: choose one of the loss type for training, the loss is
converted tofullfor PPL evaluation automatically.ExamplesRun NCE criterion with linear module:pythonmain.py--cuda--noise-ratio10--norm-term9--nce--trainRun NCE criterion with gru module:pythonmain.py--cuda--noise-ratio10--norm-term9--nce--train--index-modulegruRun conventional CE criterion:pythonmain.py--cuda--trainA small benchmark in swbd+fisher datasetIt's a performance showcase. The dataset is not bundled in this repo however.
The model is trained on concatenated sentences,but the hidden states are not
passed across batches. An
is inserted between sentences. The model is
evaluated on
padded sentences separately.Generally a model trained on concatenated sentences performs slightly worse than
the one trained on separate sentences. But we saves 50% of training time by reducing
the sentence padding operation.dataset statisticstraining samples: 2200000 sentences, 22403872 wordsbuilt vocabulary size: ~30Ktestbed1080 Tii7 7700Kpytorch-0.4.0cuda-8.0cudnn-6.0.1how to run:pythonmain.py--train--batch-size96--cuda--lossnce--noise-ratio500--nhid300\--emsize300--log-interval1000--nlayers1--dropout0--weight-decay1e-8\--datadata/swb--min-freq3--lr2--savence-500-swb--concatRunning timecrossentropy: 6.5 mins/epoch (56K tokens/sec)nce: 2 mins/epoch (187K tokens/sec)performanceThe rescore is performed on swbd 50-best, thanks to HexLee.training loss typeevaluation typePPLWER3gramnormed??19.4CE(no concat)normed(full)5313.1CEnormed(full)5513.3NCEunnormed(NCE)invalid13.4NCEnormed(full)5513.4importance samplenormed(full)5513.4importance samplesampled(500)invalid19.0(worse than w/o rescore)File structureexample/log/: some log files of this scriptsnce/: the NCE module wrappernce/nce_loss.py: the NCE lossnce/alias_multinomial.py: alias method samplingnce/index_linear.py: an index module used by NCE, as a replacement for normal Linear modulence/index_gru.py: an index module used by NCE, as a replacement for the whole language model modulesample.py: a simple script for NCE linear.example: a word langauge model sample to use NCE as loss.example/vocab.py: a wrapper for vocabulary objectexample/model.py: the wrapper of allnn.Modules.example/generic_model.py: the model wrapper for index_gru NCE moduleexample/main.py: entry pointexample/utils.py: some util functions for better code structureModified README from Pytorch/examplesThis example trains a multi-layer LSTM on a language modeling task.
By default, the training script uses the PTB dataset, provided.pythonmain.py--train--cuda--epochs6# Train a LSTM on PTB with CUDAThe model will automatically use the cuDNN backend if run on CUDA with
cuDNN installed.During training, if a keyboard interrupt (Ctrl-C) is received,
training is stopped and the current model is evaluated against the test dataset.Themain.pyscript accepts the following arguments:optionalarguments:-h,--helpshowthishelpmessageandexit--dataDATAlocationofthedatacorpus--emsizeEMSIZEsizeofwordembeddings--nhidNHIDhumberofhiddenunitsperlayer--nlayersNLAYERSnumberoflayers--lrLRinitiallearningrate--lr-decaylearningratedecaywhennoprogressisobservedonvalidationset--weight-decayweightdecay(L2normalization)--clipCLIPgradientclipping--epochsEPOCHSupperepochlimit--batch-sizeNbatchsize--dropoutDROPOUTdropoutappliedtolayers(0=nodropout)--seedSEEDrandomseed--cudauseCUDA--log-intervalNreportinterval--saveSAVEpathtosavethefinalmodel--bpttmaxlengthoftruncatedbptt--concatuseconcatenatedsentenceinsteadofindividualsentenceCHANGELOG2019.09.09: Improve numeric stability by directly calculation on logits