Probabilistic Language Modeling with Hidden Stochastic Automata

Probabilistic Language Modeling with Hidden Stochastic Automata Mark Andrews Division of Psychology Nottingham-Trent University and Division of Psychology and Language Sciences University College London Abstract In this paper, we introduce a novel dynamical Bayesian network model for probabilistic language modeling. We refer to this as the Hidden Stochastic Automaton. This model, while based on a generalization of the Hid- den Markov model, has qualitatively greater generative power than either the Hidden Markov model itself or any of its existing variants and generalizations. This al- lows the Hidden Stochastic Automaton to be used as a probabilistic model of natural languages in a way that is not possible with existing dynamical Bayesian networks. Its relevance to Cognitive Science is primarily as a com- putational — in the Marr (1982) sense of the term — model of cognition, but potentially also as a model of resource bounded cognitive processing, and as a model of the implementation of computation in physical dy- namical systems. A probabilistic language model is a hypothetical gen- erative model of a language, where a language is defined most generally as a set of strings concatenated out of a finite set of symbols. By far the most widely used formalisms for specifying probabilistic language mod- els are stochastic grammars, which are symbol rewrit- ing rules with accompanying probabilities. The use of grammars is motivated by the fact that human languages are structurally complex, with properties that place them between the so-called context-free and context- sensitive formal languages (see, e.g., Chomsky, 1956, 1963; Shieber, 1985), and formal grammars are compu- tationally universal in the sense that they can generate any recursively enumerable set (see, e.g., Hopcroft, Mot- wani, & Ullman, 2001). By contrast to the case of language modeling, in probabilistic modeling more generally, the most widely used formalism for specifying probabilistic models is the graphical model (see, e.g., Koller & Friedman, 2009; Jor- dan, 2004). Graphical models are directed or undirected graphs whose vertices are identified with random vari- ables and whose edges indicate conditional dependen- cies. The appeal of graphical models is their flexibility to represent complex relationships between large num- bers of variables, and their graph-theoretic properties that afford general and computationally efficient algo- rithms for probabilistic inference, whether exactly or approximately by, for example, Monte Carlo methods. As a result, graphical models have effectively become a graph-based modeling language for developing and ex- tending probabilistic models. They have had widespread application in fields such as bioinformatics (e.g., Fried- man, 2004), computer vision (e.g., Oliver, Rosario, & Pentland, 2000), machine learning (e.g., Bishop, 2006, 2013), expert systems (e.g., Lauritzen & Spiegelhalter, 1988; Pearl, 1988), information retrieval (e.g., Salakhut- dinov & Hinton, 2009), and in cognitive science (see, e.g., Chater & Oaksford, 2008; Griffiths, Chater, Kemp, Perfors, & Tenenbaum, 2010, for overviews). Despite their breadth of appeal, graphical models have had a rather limited role as language models, if by lan- guage models we specifically mean generative models of language. There are at least two important reasons for this. On the one hand, stochastic grammars can not, in general, be represented as graphical models. (In some cases, notably stochastic regular grammars, the termi- nal and nonterminal variables of the grammar can be identified with vertices of a directed Markovian graph. For the super-regular grammars, however, this is not the case and the variables of the grammar can not be identified with the vertices of any fixed graph). On the other hand, the most widely used graphical models for sequential probabilisitic modeling, including the Hidden Markov model and its extensions, are limited in their generative power to the regular languages (i.e. the Type- 3 languages in the Chomsky hierarchy). In other words, graphical models have had a relatively limited role as language models because the most widely used proba- bilistic models that have sufficient generative power to model human languages can not be represented as graph- ical models, and the most widely used graphical models for sequential structures do not have sufficient generative power to model natural languages. There is, however, no inherent limitation to the gen- erative power of graphical models. In this paper, we introduce a graphical model, specifically a dynamical Bayesian network, whose generative power is equivalent to that of an arbitrary stochastic grammar. This model, that we will refer to as the Hidden Stochastic Automa- ton, is based on a novel generalization of the widely used Hidden Markov model. As such, it retains many of the appealing characteristics of the Hidden Markov model while extending its generative power. Hidden Stochastic Automata To introduce the Hidden Stochastic Automaton (HSA), it is necessary to first briefly describe the Hidden Markov model (HMM). Given a set of J independent discrete