Hierarchical Bayesian Modeling for Clustering Sparse Sequences in the Context of Group Profiling

1 INTRODUCTION . Most of the existing sequence classification or clustering techniques make some assumptions which are not applicable for sequences that are generated from real-world human actions and that can be represented as a chain of events . Some examples of such sequences are following : the sequence of counts of weekly user-visits to a food-joint collected for a several weeks period , click-stream data of user navigation , the order in which the items are added to a shopping cart , the sequence of weekly updates of the Wikipedia users . All these sequences show a common property ; they are a sequence of non-negative integers . However , most of the existing sequence modeling techniques assume that the entries of the sequences are real numbers . For example , in the worlds largest collection of time series data-sets , the UCR classification archive , all forty-five time-series data-sets are only real number sequences . Other than this real number sequence assumption , most of the existing sequence classification techniques make some more assumptions implicitly or explicitly as described in Bing Hu ’ s paper ( ? ) ” To summarize , much of the progress in time series classification from streams in the last decade is almost certainly optimistic , given that most of the literature implicitly or explicitly assumes one or more of the following : 1 . Copious amounts of perfectly aligned atomic patterns can be obtained [ 11 ] [ 35 ] [ 37 ] . 2 . The patterns are all of equal length [ 11 ] [ 13 ] [ 16 ] [ 23 ] [ 31 ] . 3 . Every item that we attempt to classify belongs to exactly one of our well-defined classes [ 10 ] [ 13 ] [ 23 ] [ 30 ] . ” ( et.al. , 2009 ) ( E.Keogh ) ( Koch , 2010 ) ( Prkk , 2006 ) ( Ratanamahatana & Keogh , 2004 ) ( Reiss & Stricker , 2011 ) ( D.GafurovandE.Snekkenes , 2008 ) These three assumptions are very common in existing works about sequence modeling . However , in this paper , we consciously avoid making these assumptions , since these assumptions are not applicable in real world count data . For example , in this work , the data that has been used are sequences of counts of weekly user-visits to a food-joint collected for a two year period . But the beginning of the sequence of each user is not at the user ’ s beginning of her lifetime in the business and the end of the sequence is not the end of the user ’ s lifetime . Each user ’ s sequence starts and ends at different phases of their lifetime in the business . Thus the sequences that have been used in this work are not aligned . Also , the sequences are not of equal length in this study , but their maximum length is 104 ( weeks ) . There is no way to classify these users into well-defined classes other than hand annotating them , which is an enormous task and impractical to do in a business environment . Moreover , there are no well-accepted criteria to define the classes of the users which can be taken to be a gold standard for evaluation . One more feature of the real world sequences that are generated from human actions and more specifically , user behavior , is sparseness . For example in this weekly user-visits data that has been used for the work , most of the weekly visit entries of most of the users are zero , because most of the users of this business are infrequent visitors . This is another challenge that has been addressed in this work . Since the data is so sparse , or in other words , the sequences mostly contain zeros and it is difficult to be certain about a structure in the data . Thus we did not model the sequence as a time series model like Markov chain where each entry is dependent on its previous entry . We modeled the sequence as a bag of words model being generated from a mixture of distribution because it can provide us with much more flexibility of modeling . Daily human activities are determined by real-life motivators such as schedules , locations , tastes , and expectations . The similarity in the motivators can give rise to the similarity in behavior . Thus , the similarity in motivators can create latent groups among users who visit a shop or a restaurant or a website . In this work , an algorithm is proposed to identify such latent groups and to generate a group profile for the latent group . An individual user is perceived as partially affiliated to every latent group and from the data , the proportion of the user affiliation to every latent group is inferred . As we model the users to be soft assigned to every latent group with a proportion of affiliation to the groups , our mixture model can accommodate users that do not belong to a well-defined cluster . It is assumed that this mixing proportion or affiliation proportion is constant for the whole user sequence . User-group profiles are generated through clustering the user behavior sequence data using Hierarchical Bayesian Modeling and then by inference of the model parameters . A function of each of the cluster-specific parameters in the model constitutes each entry of each latent group profile . In this work , we assume a generative model where each individual sequence is generated by a Poisson distribution whose parameter is a convolution of K parameters drawn from K distributions associated with each of the K clusters . The entries of the sequence are assumed to be independent of the entries before or after it given the parameters of the generating distributions . Thus , even if the data is sequential in time , it has not been modeled as a time series sequence , which could be done using something like HMM . The reason for doing this is following . Primarily , each user sequence does not span the whole lifetime of that user in the loyalty system but is arbitrarily selected twoyear span in the users lifetime . So unlike RNA sequencing problems , we do not know the whole sequence and so we do not have enough data for assuming a structure in the part of the sequence that we have . Secondly , the data that has been used for this work has been aggregated into weeks from the daily user visit data in a restaurant for two years . If every daily entry is treated as a feature of the sequence , the number of the feature of each sequence is 730 . Moreover , the data is very sparse . So we have a large number of features whose values are mostly zero . Thus the data do not provide enough support for making any assumptions about the dependence structure among the features . The assumption of independence is often made for high-dimensional continuous data ( T Speed , 2001 ) , ( Tibshirani , 2003 ) , ( Witten , 2011 ) since for high dimensional data there are too few observations available to be able to effectively estimate the dependence structure among the features . Thus we assume that the sequence of each user is generated by a mixture of distribution , but not assuming the direct dependence of a later entry in a sequence on a former entry in the sequence . In this work , we use a Bayesian Finite mixture model where the number of mixture components or the number of clusters we provide beforehand , the number of clusters being determined empirically . Bayesian finite mixture models require the specification of the number of mixture components , or groups , a priori . Existing examples assume the number of groups to be unknown and therefore most commonly infer this prior specification based on marginal likelihoods or Bayes factors ( J Lau ) . For example , ( Fahey ) used the Bayesian Information Criterion to determine the optimal number of groups when analyzing dietary patterns based on the UK National Diet and Nutrition Survey . Again , this approach often leads to complex group narratives , for example , ( Fahey ) identified six groups , with narratives differing intricately in their consumption of multiple different food types . Conversely , Bayesian infinite mixture models assume an infinite number of mixture components , with the total number of occupied groups inferred from the data during model fitting ( S Kim & Vannucci ) . This approach often leads to a large number of groups containing a small percentage of individuals , complicating group narratives further . For example , using this approach , ( A Crpet , 2011 ) ) identified seventeen groups within the Individual and National Study on Food Consumption , differing in dietary consumption behaviour , with only three of the groups accounting for 98 percent of the surveyed individuals . Similarly , ( Swartz ) employed this approach to group students based on surveyed course satisfaction , identifying 10 groups within just 75 surveyed students , with many groups differing only slightly in their satisfaction response profiles . The existing Sequence Clustering Techniques follow variations of three basic ideas . In several bioinformatics papers on gene-sequence clustering is based on non sequence attributes , such as the length of the sequence and similarity of the entries are considered to be the main features based on which the clustering is done . This technique is not a model based technique . Also as variable length sequences are not feature vectors , the ’ distance ’ between sequences is an ambiguous term , which should be defined clearly in context.Based on the idea of distance the ’ similarity ’ between sequences is measured , thus this measurement needs to be defined too . The second technique is based on modeling sequences as Markov chains where the order of the chain determines number of past states affecting current state.As the order increases the cost of modeling also scales up in this model and also this model does not support the idea of an unobserved variable which might control the generation of the sequence , which is often the case in practical scenario . The third technique is based on hidden Markov model where the observed outcome is a resultant of hidden latent state.This papers presents a hierarchical Bayesian model for soft clustering of the sequences . The proposed technique is ideologically closest to an HMM , since HMM can be considered a generalization of a mixture model where the hidden variables ( or latent states ) , which control the mixture component to be selected for each observation , are related through a Markov process . This model however is more interpretable in terms of its parameters associated to each sequence and the clusters/groups of sequences.Thus the parameters and some quantities derived from the parameters can be mapped to real physical quantities and can be used to create group profiles . 1.1 BUSINESS SCENARIO . The data used in this paper is real industrial data from a loyalty program where by using an enrolled service people can earn points or rewards.In a loyalty program users enroll , use the service through the loyalty program for a period and then at some point become disinterested in using the program and thus churn out of the pool of the users of the loyalty program.It is important to study user behavior to predict his next visit , predict probability of churning out and study his buying behavior . Not only in case of individual users , but identifying the latent groups and studying user-group behavior for next-visit , churn-probability and buying behavior is also important for user segment targeting in a campaign . In this work , we have modelled user visit behavior . The user visit-behavior is recorded as a sequence of number of visits to a restaurant every week . Hence all the users are represented by variable length sequences where every entry of a sequence is the number of visits of the corresponding user in a particular week .

Summary: This paper propose a hierarchical Bayesian model to cluster sparse sequences data. The observations are modeled as Poisson distributions, whose rate parameter \lambda_i is written as the summation of \lambda_{ik}, a Gamma distribution with rate equal to the mixture proportion \alpha_{ik}. The model is implemented in Pystan. Experimental results on a real-world user visit dataset were presented.