Package: mixes
Mixes- repository of mixture modelsThis repository was created as part of Research paper "Estimation of Gaussian Mixture Networks"
required as part of Master of Mathematics degree in Computational Mathematics at the University of Waterloo authored
by Andriy Dmytruk and supervised by Ryan Browne.The repository includes implementation of the following mixture models:Gaussian Mixture Model (GMM)Skew Gaussian Mixture Model (Skew GMM)Deep Gaussian Misture Modle (DGMM)Gaussian Mixture Network (GMN)UsageThe implementation is present in themixes/folder.
You can see an example of usage in theexperiments/example.ipynbjupyter notebook.All the experiments that were performed as part of the research paper can also be found inside theexperiments/folder.Models descriptionSkew Gaussian Mixture ModelSkew GMM was implemented based on paper"Maximum likelihood estimation for multivariate skew normal mixture models"by Tsung I. Lin (2006).Gaussian Mixture NetworkGMN was proposed in the author's research paper. The model creates a network of gaussian distributions
where next layers in the model have conditional probability distribution based on the previous layer.
Each layer is a mixture of components, therefore the whole model creates a network of gaussian nodes.The most important parameters are:layer_sizes- these are the sizes of layers. The first layer will be used for clusterization
and therefore its size should correspond to the desired number of clusters.layer_dims- the input dimensions of each layer. Each layer has an input and output dimension.
The output of the first layer is automatically set to the dimensionality of the data, and output of prevous layer
is considered the input of the next one. By reducing dimensionality of the deeper layers have fewer parameters,
and each layer becomes similar to Mixtures of Factor Analyzers. You would probably want to set the
dimensions in a non-increasing order.init- determines how the model is initialized. Usekmeans(default) for initialization byK-Meansand
factor analysis on each layer. Userandomfor a completely random initialization.Deep Gaussian Mixture ModelDGMM is based on papers"Deep Gaussian mixture models"by Cinzia Viroli, Geoffrey J. McLachlan (2019) and"Factoring Variations in Natural Images with Deep Gaussian Mixture Models"by Aaron van den Oord, Benjamin Schrauwen (2014).The parameters are similar to GMN model, as is the implementation in this repository.The difference between DGMM and GMN is that GMN gives probabilities to layer's components conditional
on the previous layer, while DGMM has them independent.AnnealingWe implemented deterministic annealing for mixture models as described in the paper"On the Bumpy Road to the Dominant Mode"by Hua Zhou, Kenneth L. Lange (2010).Since the log-likelihood functions is frequently non-concave, the EM algorithm can end up
in suboptimal modes. The idea of annealing is to flatten the objective function and therefore
increase the chances of terminating in a dominant mode.The parameteruse_annealingdetermines whether to use annealing, while the parameterannealing_start_vdetermines the intial value for annealing. The value must be between0and1.
Lower values correspond to a more flattened objective function, while1corresponds to no
annealing. Starting for theannealing_start_v, the annealing value will be increased to1during model fitting
ifuse_annealingis set to true.RegularizatoinGMM, GMN and DGMM models have the variance regularization parametervar_regularization.
Regularization makes the covariances larger on each step. This keeps the covariance matrix from becoming
close to singular, which would greatly degrade optimization for it. The parameter can also be used
for restricting the model to larger covariances and avoid overfitting.Stopping criterionUse thestopping_criterionparameter in models to specify a stopping criterion. Specified function
must have the same signature as functions in themixes/stopping_criterion.pyfile.