Factorial Hidden Markov Models for Learning Representations of Natural Language
Authors: Anjan Nepal, Alexander Yates
ICLR 2014

Abstract: Most representation learning algorithms for language and image processing are local, in that they identify features for a data point based on surrounding points. Yet in language processing, the correct meaning of a word often depends on its global context. As a step toward incorporating global context into representation learning, we develop a representation learning algorithm that incorporates joint prediction into its technique for producing features for a word. We develop efficient variational methods for learning Factorial Hidden Markov Models from large texts, and use variational distributions to produce features for each word that are sensitive to the entire input sequence, not just to a local context window. Experiments on part-of-speech tagging and chunking indicate that the features are competitive with or better than existing state-of-the-art representation learning methods.


Peer review:
[Review]:
The paper looks into representing words as (approximate) posterior distributions of states in a Factorial HMM estimated on a unlabeled dataset of a moderate size. Unlike standard word representations (e.g., obtained with the neural probabilistic model of Bengio et al. (2003)), these representations encode not only properties of a word but also properties of the word context (in theory unbounded) . For example, this may result in a form of word sense disambiguation.  Of course, this come at the cost of performing inference at test time.  The authors evaluate their approach on PoS tagging and chunking tasks. 

I find the paper quite interesting, as using some form of disambiguation does seem like a good idea to me. 

Comments:

1) The authors use a variational approximation which can be regarded as a form of the structured variational method for Factorial HMMs introduced in Ghahramani & Jordan (MLJ 1997) (not cited).  It would be good to explain which parts of the inference algorithm (e.g., relaxation for softmax) are novel, and which are borrowed from previous work.
2)  I do not quite understand how exactly the variational parameters phi are computed -- something like Newton-Raphson? (In the paper: 'these expectations are used to find the new set of variational parameters', par 1, page 6)
3) I wonder if Brown clusters are used as atomic variables or the features are paths in the binary tree (as in Koo et al. (ACL 2008)). Using paths may help substantially. 
4) Actually, factorial HMMs induce other types of word representations as well: parameters associated with emission distributions can perhaps be regarded as such representations. I am wondering if the authors considered using them as an additional baseline. (Of course, this representation would not be affected by the context.)
5) It would be nice to see methods using word embeddings produced by more conventional methods as additional baselines. This would make the paper more convincing.
6) Stochastic neural network models (namely sigmoid belief networks) for syntactic parsing of Titov & Henderson (ACL 2007) and Henderson & Titov (JMLR 2010) may also be relevant. They learn representations of parsing states (and some of their states corresponds to word emissions). The posterior distribution of the state variables are affected by global context. They also use variational approximation methods somewhat similar to the ones considered in this submission.