Factorial Hidden Markov Models for Learning Representations of Natural Language
Authors: Anjan Nepal, Alexander Yates
ICLR 2014

Abstract: Most representation learning algorithms for language and image processing are local, in that they identify features for a data point based on surrounding points. Yet in language processing, the correct meaning of a word often depends on its global context. As a step toward incorporating global context into representation learning, we develop a representation learning algorithm that incorporates joint prediction into its technique for producing features for a word. We develop efficient variational methods for learning Factorial Hidden Markov Models from large texts, and use variational distributions to produce features for each word that are sensitive to the entire input sequence, not just to a local context window. Experiments on part-of-speech tagging and chunking indicate that the features are competitive with or better than existing state-of-the-art representation learning methods.


Peer review:
[Review]:
The authors propose using a factorial hidden Markov model (FHMM), which is an HMM with multiple latent Markov chains cooperating to produce observations, to induce context-dependent word representations from text. They derive a variational EM algorithm for training these models efficiently and use a structured variational posterior consisting of independent Markov chains to capture the sequential structure of the exact posterior. Given a sequence of words, the authors obtain their representations by computing the variational posterior for the sequence and representing each word with the resulting distribution over the states of the latent variables associated with the timestep that generated the word.

Although HMMs have been used to induce word representations before, using factorial HMMs for this task is novel. The variational algorithm for training FHMMs on multinomial observations, such as words, derived in the paper appears to be new as well.

The idea of using FHMMs to obtain context-dependent word representations is a sensible one. Unfortunately, the paper is not particularly well written and the experimental evaluation is not sufficiently convincing. The variational EM algorithm derived by the authors is a relatively minor modification of the algorithm of Ghahramani and Jordan (1997), something the paper does not state clearly enough. The form of the variational posterior comes from the same paper. The bound in Eq. 15 has already been used in a very similar setting by Blei and Lafferty (2007). The presentation of the derivation is unclear and contains a number of small mistakes. For example:
-The terms corresponding to S_1 are missing from P({S_t}|{Y_t}) and Q({S_t}|{Y_t}).
-In equations (3), (12), and (13), the denominator contains S^m_{t-1,j} instead of S^m_{t,k'}.
-It would be cleaner to parameterize the multinomial distributions directly in terms probabilities (as was done by Ghahramani and Jordan) instead of log-unnormalized-probabilities.
-The parameterization of the variational posterior in equation 13 is odd. Why are the transition terms explicitly normalized but the observation terms are not?

The experimental section does have some interesting results but suffers from a non-standard experimental protocol and absence of some obvious baselines. The representations produced by FHMM are interesting because they are distributed and context-dependent. Since distributed but not context-dependent representations are not included in the evaluation, it is not possible to disentangle the effects of those two factors on the performance of the FHMM representations. Representations learned by neural language models as well as the 'non-temporal FHMM' (effectively a distributed mixture model) would be the natural baselines for this. Until this issue is addressed, it will remain unclear how much context dependence contributes and whether using FHMMs instead of simpler models is worth the effort. Finally, it is unfortunate that the authors chose not to use exactly the same training data as in [25] and [6], making it impossible to compare their results to those in the literature.