factorie/factorie — pull request #133 by davidBelanger (2014-04-05 15:46:31+00:00)
Title: adding WordEmbedding.scala

This pull request adds stuff for loading word embeddings in the format produced by Jeevan's new factorie word embedding training code, which is the same format as the Google word2vec files. This is designed to be used when plugging word embeddings into other NLP tasks in Factorie.

There are two things I've added, a class called WordEmbedding, which essentially wraps a map from string to DenseTensor1, and WordEmbeddingOptions which provides a mixin for command line options for other nlp tasks. (note that this is different from command line arguments used when training word embeddings).

**the Google embedding files and the ones Factorie produces are one-word-per-line, sorted by frequency. This is useful because you can take the first N lines from the file if you don't want to use all of the embeddings. The WordEmbedding constructor takes an optional final argument for the number of lines you want to take from the file**

**Also note that the WordEmbedding class does not provide a default value for out-of-vocabulary words That needs to be handling outside, but checkout wordEmbedding.contains(str). **

Here is an example of how to use the mix in the command line arguments:

class TransitionBasedParserWithEmbeddingsArgs extends TransitionBasedParserArgs with WordEmbeddingOptions

Here is an example of how to instantiate a WordEmbedding object
val useEmbeddings = opts.useEmbeddings.value
if(useEmbeddings) println("using embeddings") else println("not using embeddings")
val embedding = if(useEmbeddings) new WordEmbedding(() => new FileInputStream(opts.embeddingFile.value),opts.embeddingDim.value,opts.numEmbeddingsToTake.value) else null