Understanding web traffic on a web site is challenging but fundamental task in improving link structure and simplifying navigation through a large number of individual web pages. The useful information about the traffic on the site is buried in a large server log file with millions of lines. To gain insight into type of traffic that takes place on the server one must model the data and understand it by understanding the underlying model. However, models involved in such large data sets are often themselves quite complex and cannot be understood or visualized easily. If the web site contains hundreds or thousands of different web pages, individual pages are grouped into semantically similar groups.
Different users have different behavior and different patterns of web page requests.
The second step in analysis is selecting a suitable model to describe the data. To model heterogenuity of users we use a mixture of Markov chains, where each component in a mixture represents a prototypical behavior described by a single Markov chain. A model is fitted on a subsample of sequences and then scored on the remaining data. This way we get an objective measure of how well each model fits the data. We also consider a Markov Chain where each cluster is constrained to begin with the same state.