Advanced mixtures for complex high dimensional data : from model-based to Bayesian non-parametric inference Advanced mixtures for complex high dimensional data : from model-based to Bayesian non-parametric inference

Cluster analysis of complex data is an essential task in statistics and machine learning. One of the most popular approaches in cluster analysis is the one based on mixture models. It includes mixture-model based clustering to partition individuals or possibly variables into groups, block mixture-model based clustering to simultaneously associate individuals and variables to clusters, that is co-clustering, generally for independent multivariate data, and regression mixture-model based clustering approaches for conditionally dependent data, including functional data. These modelbased (co)-clustering approaches rely on density estimation, possibly Bayesian, of parametric finite (block) mixtures. These parametric models may be limited in some situations namely to represent complex and realistic data sets. Another issue in the finite mixture model-based approach is the one of model selection, which is in general equivalent to selecting the number of mixture components. The Bayesian non-parametric alternative based on Dirichlet Process mixture (DPM) or infinite mixtures, provides a principled way to overcome these issues. The aim of this project is to propose new Bayesian non-parametric mixtures and block mixtures for model-based (co-)clustering of high dimensional multivariate data, and new regression mixture model-based approaches for clustering complex functional data. I will perform my research within the Probability and Statistics team of “Laboratoire Paul-Painlevé” UMR CNRS 8524, one of the first teams over the world in the domain. 1 Scientific context and objectives 1.1 Scientific context The problem of complex data analysis is a central topic of modern statistical and computer and information sciences, and is connected to both theoretical and applied parts of theses sciences, as well as to several application domains including pattern recognition, signal processing, bio-informatics data mining, complex systems modeling, etc. The analysis of complex data, in general, implies the development of statistical models and autonomous learning algorithms that aim at acquiring knowledge from raw data for analysis, interpretation and to make accurate decisions and predictions for future data. The development of models therefore have to rely on well-established statistical background with efficient algorithmic tools to deal namely with the problem of data complexity, including heterogeneity, missing information, high dimensionality, big volume, dynamical structure, etc. To ensure such reliability of models and algorithms for decision from raw data, it is important to understand the processes generating such data. From a statistical learning prospective, this in general leads to generative learning approaches, which will be in the core of this project. Generative model-based approaches are indeed very useful and well-established statistical models that explicit the processes generating the data. They are very suitable in many contexts, in particular the unsupervised context when the supervision (e.g., expert information) is missing, hidden or difficult to obtain, etc, and are useful for many applications, including clustering and classification of grouped data. Latent data models, including (Bayesian) mixture model-based approaches are one of the most popular and successful generative Faicel Chamroukhi Projet de délégation CNRS, année 2015/2016 1/5 Projet de délégation CNRS, 2015/2016 Advanced mixtures for complex high dimensional data: from model-based to Bayesian non-parametric inference unsupervised learning approaches. They are very used in particular in cluster analysis for automatically finding clusters, in discriminant analysis when the classes are dispersed, as well as in regression when the predictor is governed by a hidden process, etc. Such models are naturally tailored for an exploratory data analysis and rely on efficient algorithmic tools such as the expectation-maximization (EM) algorithms or Markov Chain Monte Carlo (MCMC) sampling techniques. These approaches have been mainly developed for multivariate data analysis. On the other hand, in many areas of application, including signal and image processing, functional imaging, handwritten text recognition, bio-informatcis, etc., the analyzed data are often available in the form of discretized values of functions or curves (e.g., times series, waveforms, etc) and surfaces (2D-images, spatio-temporal data, etc) which makes them very structured. This “functional” aspect of the data adds additional difficulties compared to the case of a classical multivariate analysis. Since the standard multivariate latent data model, including mixture-model based approaches rely on a sound statistical background, and thus are likely to be very beneficial for functional data analysis, there is a growing investigation of adapting them to the context of functional data, including curves and surfaces. But this is still less well investigated as compared to the case of multivariate (non-functional) data. 1.2 Scientific objectives The scientific objectives of this project are therefore two-fold. The first main objective is to develop flexible mixture model-based cluster analysis approaches for high dimensional multivariate data. This includes Bayesian parametric and non-parametric mixtures for clustering, as well as for co-clustering (block mixture-model based approaches). The second main objective is to develop mixture-model based model-based cluster and discriminant analysis approaches for functional data, including curves and surfaces. This mainly involves the development of regression mixture models for curves and surfaces. In particular, I will focus on the unsupervised learning context. The problem of building fully unsupervised approaches will be central to the project. Hence, the problem of model selection will be central in order to develop fully unsupervised techniques, that automatically learn, compare and select the models. I will consider the problem of model selection from two prospectives: the first one relying on model selection criteria such as BIC or the Bayes Factors. While the second one considers the Bayesian non-parametric alternative, that is the Dirichlet Process Mixture (DPM) or by equivalence the Chinese Restaurant Process mixture (CRPM). The project is structured according to the following tasks: 1. Bayesian non-parametric mixture model-based clustering 2. Bayesian non-parametric block mixture model-based co-clustering 3. Bayesian mixture model-Based functional data clustering 2 Bayesian non-parametric mixture model-based clustering 2.1 Model-based clustering Model-based clustering (McLachlan and Basford, 1988; Celeux and Govaert, 1992, 1993; Banfield and Raftery, 1993; Fraley and Raftery, 2002; Biernacki et al., 2006), generally used in a multivariate data analysis context, is based on the finite mixture model formulation (McLachlan and Peel., 2000; Titterington et al., 1985). The data probability density function is assumed to be a finite mixture density, each component density being associated with a cluster. The problem of clustering therefore becomes the one of estimating the parameters of the supposed mixture model. The model estimation can be performed by maximizing the observed-data likelihood. This approach is referred to as the mixture (or estimation) approach and in which likelihood maximization is generally achieved via the expectation-maximization (EM) algorithm (Dempster et al., 1977; McLachlan and Krishnan, 1997) or extensions, namely stochastic extensions Celeux et al. (1996), such as stochastic EM (Celeux and Diebolt, 1985). Another approach is the classification approach, also referred to as the maximum classification likelihood approach, maximizes a classification likelihood function, which is the complete-data likelihood by using the classification version of the EM algorithm, that is the classification EM (CEM) algorithm (Celeux and Govaert, 1992). Banfield and Raftery (1993) and Celeux and Govaert (1995) developed a parsimonious GMM clustering by exploiting an eigenvalue decomposition of the group covariance matrices for the GMM, which provides a wide range of very flexible models with different clustering criteria. This decomposition leads to numerous flexible models (with different volumes, shapes and orientations) going from simplest spherical models to the complex general one ans hence is adapted to various clustering situations. The parameters of the parsimonious Gaussian mixture models may be estimated in a MLE framework by using the EM algorithm (see details in Celeux and Govaert (1995)). Fraley and Raftery (2002) also demonstrate that the parsimonious mixture models-based clustering framework provide very good results in density estimation cluster and discriminant analyses. Recently, Biernacki and Lourme (2014) proposed new parsimonious Gaussian mixture models for Faicel Chamroukhi Projet de délégation CNRS, année 2015/2016 2/5 Projet de délégation CNRS, 2015/2016 Advanced mixtures for complex high dimensional data: from model-based to Bayesian non-parametric inference clustering based on a variance-correlation decomposition of the group covariance matrices, which are stable and visualizable and have more desirable properties compared to the parsimonious GMMs of Banfield and Raftery (1993) and Celeux and Govaert (1995). The mode parameters are estimated by a Generalized EM. The MLE approach using the EM algorithm for normal mixtures may fail due to singularities or degeneracies (Stephens, 1997; Fraley and Raftery, 2007). The Bayesian approach of mixture models (Stephens, 1997; Robert, 1994; Jean-Michel Marin and Robert, 2005; Fraley and Raftery, 2007; Bensmail et al., 1997; Richardson and Green, 1997) avoids the problems associated with the maximum likelihood described previously. by replacing the MLE by a maximum a posterior (MAP) estimation. This is namely achieved by adding regularization over the model parameters via pr