Statistical modeling with finite mixtures of skew-t distributions and their application in the life sciences

There is a need for flexible distribution and error models considering multivariate data in the life sciences. This thesis explores a flexible class of multivariate models, especially suited for data encountered in real-life situations, where outliers and skewness are prevalent and provides an algorithm for fitting such models to use them efficiently in practice.We introduce univariate and multivariate mixtures of skew-t distributions, beginning with their mathematical definitions and followed by their hierarchical representation, which facilitates the implementation of the Expectation-Maximization (EM) algorithm for parameter estimation. A new proposal for classifying multivariate clinical data is proposed, based on fitting multivariate skew-t mixtures separately to patients from diseased case and non-diseased control groups. The ratio of multivariate densities for cases to controls forms a likelihood ratio, which multiplied by the ratio of prior probabilities of being a case versus control, leads to the posterior odds of being a case. The posterior odds of cases are back-transformed to the probability scale, resulting in individual predictions. The form of the density ratio is discussed for different situations, such as for the constraint of equal variance. We construct efficient EM algorithms that can accommodate collapsed clusters. A collapsed cluster can be viewed as a distribution that places all of its mass on a lower-dimensional space with no variance. Our approach for the applications targeted in this thesis is that the same underlying data generating process applies to collapsed and non-collapsed clusters. We develop and publish a novel R package fitmixst4, which implements the EM algorithm for fitting multivariate mixtures of skew-t distributions and differentiates collapsed clusters from non-collapsed ones by restricting the variance of the latter to be above a specific bound.We implement the algorithm in two applications. The first application concerns the update of a leading online clinical risk prediction model for prostate cancer on biopsy to incorporate two novel serum markers. We fit the multivariate skew-t mixtures to the bivariate distribution of the two markers in a sample of cancer cases and controls, respectively, thus forming the likelihood ratio. Prior odds of prostate cancer for individual patients are formed based on their standard clinical risk factor profiles from an existing online risk prediction tool. Multiplication by the likelihood ratio leads to updated individualized posterior probabilities of prostate cancer that combines information from standard risk factors with the new markers. We implement the resulting risk tool with the R package shiny and post it online at the Cleveland Clinic Risk Library to make it accessible to patients and clinicians worldwide.For the second application, we fit mixtures of multivariate skew-t distributions with collapsed clusters to describe and classify trees that experience mortality versus not from a network of European Beech trees. We model up to five individual tree characteristics and competition indices to form risk prediction models for tree mortality. We visualize two-dimensional contour plots of predictive characteristics for trees experiencing versus not experiencing mortality in order to facilitate communication with forest researchers concerning indicators for mortality. Using separate training and test sets, we show that skew-t based methods slightly outperform traditional logistic regression.This thesis provides means for life science researchers to implement an intricate modeling framework in order to maximize prediction of outcomes, as well as an understanding of underlying complex nonlinear associations among risk factors. The published R package facilitates implementation, bringing the impact of the models to applications in many fields beyond those shown in this thesis.