Applications of Geometry in Optimization and Statistical Estimation

Geometric properties of statistical models and their influence on statistical inference and asymptotic theory reveal the profound relationship between geometry and statistics. This thesis studies applications of convex and differential geometry to statistical inference, optimization and modeling. We, particularly, investigate how geometric understanding assists statisticians in dealing with non-standard inferential problems by developing novel theory and designing efficient computational algorithms. The thesis is organized in six chapters as it follows. Chapter 1 provides an abstract overview to a wide range of geometric tools, including affine, convex and differential geometry. It also provides the reader with a short literature review on the applications of geometry in statistical inference and exposes the geometric structure of commonly used statistical models. The contributions of this thesis are organized in the following four chapters, each of which is the focus of a submitted paper which is either accepted or under revision. Chapter 2 introduces a new parametrization to general family of mixture models of the exponential family. Despite the flexibility and popularity of mixture models, their associated parameter spaces are often difficult to represent due to fundamental identification problems. Other related problems include the difficulty of estimating the number of components, possible unboundedness and non-concavity of the log-likelihood function, non-finite Fisher information, and boundary problems giving rise to non-standard analysis. For instance, the order of a finite mixture is not well defined and often can not be estimated from a finite sample when components are not well separated, or some are not observed in the sample. We introduce a novel family of models, called the discrete mixture of local mixture models, which reparametrizes the space of general mixtures of the exponential family, in a way that the parameters are identifiable, interpretable, and, due to a tractable geometric structure, the space allows fast computational algorithms. This family also gives a well-defined characterization to the number of components problem. The component densities are flexible enough for fitting mixture models with unidentifiable components, and our proposed algorithm only includes the components for which there is iii enough information in the sample. This paper is under revision in Statistics and Computing Journal. Chapter 3 uses geometric concepts to characterize the parameter space of local mixture models (LMM), introduced in Marriott (2002) as a local approximation to continuous mixture models. Although LMMs are shown to satisfy nice inferential properties, their parameter space is restricted by two types of boundaries, called the hard boundary and the soft boundary. The hard boundary guarantees that an LMM is a density function, while the soft boundary ensures that it behaves locally in a similar way to a mixture model. The boundaries are shown to have particular geometric structures that can be characterized by geometry of polytopes, ruled surface and developable surfaces. As working examples the LMM of a normal model and the LMM of a Poisson distribution are considered. The boundaries described in this chapter have both discrete aspects, (i.e. the ability to be approximated by polytopes), and smooth aspects (i.e. regions where the boundaries are exactly or approximately smooth). A version of this chapter has been published in Geometric Science of Information, Lecture Notes in Computer Science, 9389. p 577-585. Chapter 4 uses the model space introduced in Chapter 2 for extending a prior model and defining a perturbation space in the Bayesian sensitivity analysis. This perturbation space is well-defined, tractable, and consistent with the elicited prior knowledge, the three properties that improve the methodology in Gustafson (1996). We study both local and global sensitivity in conjugate Bayesian models. In the local analysis the worst direction of sensitivity is obtained by maximizing the directional derivative of a functional between the perturbation space and the space of posterior expectations. For finding the maximum global sensitivity, however, two criteria are used; the divergence between posterior predictive distributions and the difference between posterior expectations. Both local and global analyses lead to optimization problems with a smooth boundary restriction. Work from this chapter is in a paper under revision in Statistics and Computing Journal. Chapter 5 studies Cox’s proportional hazard model with an unobserved frailty for which no specific distribution is assumed. The likelihood function, which has a mixture structure with an unknown mixing distribution, is approximated by the model introduced in Chapter 2, which is always identifiable and estimable. The nuisance parameters in the approximativ ing model, which represent the frailty distribution through its moments, lie in a convex space with a smooth boundary, characterized as a smooth manifold. Using differential geometric tools, a new algorithm is proposed for maximizing the likelihood function restricted by the smooth yet non-trivial boundary. The regression coefficients, the parameters of interest, are estimated in a two step optimization process, unlike the existed methodology in Klein (1992) which assumes a gamma assumption and uses Expectation-Maximization approach. Simulation studies and data examples are also included, illustrating that the new methodology is promising as it returns small estimation bias; however, it produces larger standard deviation compared to the EM method. The larger standard deviation can be the result of using no information about the shape of the frailty model, while the EM model assumes the gamma model in advance; however, there are still ways to improve this methodology. Also, the simulation section and data analysis in this chapter is rather incomplete and more work needs to be done. Chapter 6 outlines a few topics as future directions and possible extensions to the methodologies developed in this thesis. v Acknowledgements Although my name is alone on the front cover of this dissertation, I am by no means its sole contributor. Rather, there are a number of people behind this piece of work who deserve my greatest thanks and appreciation. First and foremost, my sincere gratitude goes to my supervisor, Prof. Paul Marriott, for his inspiring guidance, kindness, and generosity. Without him this work would never have been possible. His support went far beyond technical insights into geometry and statistics and financial support; he lead me to a more rational and less stressful academic life. I learned from him how to work in a group with courage, patience and mutual respect, how to stay on my feet after disappointments, and how to learn from mistakes. I would also like to extend my gratitude to Prof. Christopher Small, as my presence at the University of Waterloo was due to his kindness, generosity and trust in me. To my thesis committee, Prof. Shinto Eguchi at Institute of Statistical Mathematics, Tokyo, Prof. Pengfei Li, Prof. Martin Lysy, Prof. Tony Wirjanto and Prof. Dinghai Xu, at University of Waterloo, for their insightful comments on the earlier version of this thesis. Thank you to all the faculty members and staff of the department of Statistics and Actuarial science for accepting me as a member of such a wonderful and friendly family. Special mention to Mary Lou Dufton, Lucy Simpson, Marg Feeney, Karen Richardson, and Anthea Dunne, who had answers to all my administrative inquiries and solutions to all the problems. I would also like to thank my best friends Reza Ramezan, Gwyneth Ramezan, Kadir Mirza, Amirhosein Vakili and Sina Hajitaheri for being my true family in Canadafor being there with me at all the tough moments and for all the fun and memorable times we had during these years. Finally, the biggest sacrifice I made to make this happen was to move away from my closest loved ones: my loving, strong, and protective parents Rahim Maroufy and Soda Ahmadzadeh, and my funny, generous and supportive brothers Ahad, Farhad, and Mohammad. I am forever indebted to them and I want to thank them for all the encouragement, love, and emotional support they provided me throughout my life. vi To my parents Rahim and Soda who taught me the first lessons about life. I could never have done this without your support and encouragement. and my three cherished brothers Ahad, Farhad and Mohammad, without you my life would be boring and incomplete. vii