Factorial Hidden Markov Models for Learning Representations of Natural Language
Authors: Anjan Nepal, Alexander Yates
ICLR 2014

Abstract: Most representation learning algorithms for language and image processing are local, in that they identify features for a data point based on surrounding points. Yet in language processing, the correct meaning of a word often depends on its global context. As a step toward incorporating global context into representation learning, we develop a representation learning algorithm that incorporates joint prediction into its technique for producing features for a word. We develop efficient variational methods for learning Factorial Hidden Markov Models from large texts, and use variational distributions to produce features for each word that are sensitive to the entire input sequence, not just to a local context window. Experiments on part-of-speech tagging and chunking indicate that the features are competitive with or better than existing state-of-the-art representation learning methods.


Peer review:
[Review]:
The author use a Factorial Hidden Markov Model (FHMM) on two Natural Language Processing tasks, namely POS tagging and Chunking.  Such a model associates a factorial state (a tuple of states) with each word position in the corpus. This can be interpreted as a left-context dependent word representation. In order to evaluate the quality of this representation, the authors train the representation on WSJ data from the Penn repository and test on biomedical text. This domain adaptation performance is compared with that of a variety of systems (no learned features, HMM features, Brown clusters).   Although the approach makes sense, the empirical evaluation leaves questions open.  For instance, before seeing domain adaptation performance results, I would have liked to know whether the FHMM representation leads to competitive performance when testing on similar data. Also the domain adaptation performance is not compared with the performance afforded by some of the readily available word embeddings (other than Brown clusters.)   In conclusion, I do not really know what to think of the performance of FHMMs on such tasks.