Finite discrete Markov process clustering

In this paper I describe a probabilistic method for clustering discrete Markov processes with a pre-specified number of clusters. I derive two algorithms for estimating the model parameter; one based on a Gibbs sampler and the other based on an EM algorithm. The Gibbs sampling algorithm is accurate at the expense of speed and a constrained EM algorithm is optimized for speed at the expense of accuracy. A hybrid algorithm is also formulated. 1 Problem description Consider an s-state discrete Markov process where the transition matrix for the process, T, is unknown. Now assume that we believe that the process came from one of m distinct transition matrices so that, ∑ = = m l l l P T 1 δ (1) where δ =1 if the process came P and otherwise δ =0. For example we may observe a set of N processes with states enumerated by 1, . , s. Process 1: 4 7 3 4 2 5 4 3 2 1 5 Process 2: 5 3 7 9 1 2 Process 3: 3 4 1 3 2 1 2 3 : Process N: 4 1 8 7 4 3 2 1 From the start we do not know which of the m transition matrices generated each process, nor do we know the initial state distribution, nor the transition probabilities in the transition matrices. Despite our lack of knowledge, estimation is reasonably tractable. In the Bayesian clustering framework the vector δ is the unobserved class variable. If we have a collection of processes generated from T then we can compute joint posterior densities for the parameters in this model. Of particular interest are the elements of the P and the class variable. This discrete Markov process clustering methodology is a potentially useful method for clustering graph traversals, discovering task-related sequences, and next-step prediction. 1.1 Notation The following notation will be used throughout this report. i index for the origin states j index for the destination states k index for the observed processes index for the clusters N the number of observed processes i0 the initial state of the process nij the number of times a process transitioned from state i to state j m the number of clusters s the number of states p a probability vector of length s specifying cluster ’s initial state distribution P an s×s matrix of transition probabilities for cluster . →k indicates the event that cluster generated process k. This implies that P is the transition matrix underlying this process. Pr( →k|δ) is the th element of δ (k) and Pr( →k)=α . α a probability vector of length m that contains the proportion of processes coming from any particular cluster. δ the probability vector containing the probability that cluster generated process k. The Gibbs sampling algorithm (section 2.1) and the constrained EM algorithm (section 3.2) force this vector so that the th element is 1 if process k was generated from P and all other elements are 0. 2 Gibbs sampling algorithm for process clustering Under a Bayesian formulation where a prior distribution of the parameters can be specified the likelihood density can be inverted by Bayes’ Theorem in order to obtain a posterior density of the model parameters. Although the posterior distribution can be quite complex, Gibbs sampling provides a means for exploring the posterior distribution. 2.1 Likelihood function P is a Markov transition matrix.           = mm l m l m l l l P P P P P ) ( 1 ) ( 1 ) ( 11 ) ( (2) N is the number of observed processes and nij’s indicate the observed number of transitions that process k made from state i to state j. Therefore the likelihood is as follows ) ( ) ( ) ( 0 1 1 1 1 ) ( ) ( ) , , | ( k l k ij k N