clementfarabet/lua---nnx — pull request #8 by nicholas-leonard (2014-05-20 03:52:16+00:00)
Title: SoftMaxTree

This PR adds an experimental module for doing a hierarchical gated softmax. Instead of doing a Linear + LogSoftmax with, say, a vocabulary of 800k words, we can do a product of, say, 6 Linear +SoftMax of 10 words each to get the likelihood of the target words.

The module is initialized with a hierarchy of words. I have tested it with a hierarchy of 800k words divided into 6 levels of clusters of about 10 items. Its about 300 times faster than the alternative of just doing one big Linear + LogSoftMax. And about only five times slower than a Linear with only 100 outputs + LogSoftMax.

This PR includes unit tests.