Role of Majorization in Learning the Kernel within a Gaussian Process Regression Framework

Over the recent years, machine learning techniques have breathed a new life in to the classical regression framework. The primary focus in these techniques has often been the predictive performance of the estimated models and the models themselves have developed in to sophisticated non-linear predictive machines. In this development, the ubiquitous “kernel-trick” has played a very important role by providing a means to compute the inner products in the unwieldy high-dimensional spaces via simple and easily computable functions on the low-dimensional covariate domains, called as kernels. The domain knowledge of data dictates the collection of kernels suitable for the specific application. In “learning the kernel” paradigm, current state of the art is to use some optimization method to select the best kernel for the data at hand from this collection. The work in this dissertation assumes the existence of a “true” underlying process, a Gaussian Process, (defined by a fully specified covariance kernel) for the given data. The Gaussian Process itself is considered as a prior on the reproducing kernel Hilbert space of functions characterized by the associated kernel. The goal is to make suggestions towards developing some diagnostic tools which can be used to hasten the kernel learning process. In particular, the setup for computational experimentation such a collection is either an ad hoc class of finitely many kernels or a convex collection of kernels which may include one of more parametric family of kernels, for example, Gaussian kernels with bandwidth as the parameter ii is restricted to a Gaussian Process Regression framework with some “mild stationarity” and “closure” type of assumptions on the possible family of kernels. Tools are developed based on the generalized cross validation and the functional norm of the estimated functions. The sign-change behaviors of these tools are exploited for diagnostic purposes. For the tool based on generalized cross validation, a result is conjectured based on computational evidence, and partially proved, which attempts to justify the observed sign-change patterns. Complete proofs for the said result are given under some special classes of kernels. These sign-change behaviors are intended to be a “guiding stick” for reducing the computational effort and search space for “learning the kernel. ” the kernel value of a point with itself does not depend on the point Gram matrices closed under the Schür product iii To my dearest mother, Mrs. Tapati Kapat, my father, Mr. Sibapada Kapat, and my grandfather, Late. Mr. Naryandas Ghose. iv Acknowledgments First and foremost I would like to express my earnest gratitude towards Prof. Prem Goel for his continuous support and endless patience. Be the late evenings in his office, the unending meetings on the weekends, or the occasional sessions at his home, his availability has been invaluable to me. Most importantly, his eternal optimism in this problem and perseverance towards it has been, at times, my only source of encouragement, and has kept me going. This dissertation would never have seen the light of day without his help and for it, I cannot be thankful enough. I am also grateful to Prof. R. B. Bapat at the Indian Statistical Institute, Delhi, for guiding us towards the tool of majorization among Hadamard product of matrices which was crucial for this research. Along with Prof. Goel, I am also indebted to Prof. Mark McCord from the Civil Engineering Department at OSU for supporting me over the past few years through various GRA positions. I am thankful to my colleagues at J.P.Morgan Chase for giving me the flexibility over the past few months to work on my thesis. I would also like to give a special shout-out to Brian Smith from the computer support team at the Statistics Department. He has always gone out of his way to help me with my idiosyncratic needs. And I am certainly grateful to the Statistics Department for giving me the opportunity to pursue a doctoral degree. This Ph.D. has been an arduous journey. Yet, I owe a small but wonderful group of friends to make it immensely memorable. I have been rather lucky to have the “quality v over quantity! ” Two of them, Niranjan Balachandran and Indrajit Bhattacharya, deserve a special mention. Niranjan, from my very first day here, has had an indelible influence in shaping my outlook and personality: from dinners, movies, and sitcoms to mathematics, music, and philosophy; he has impressed upon me in every thinkable way. Indrajit has been more than a friend, he has been a shadow to me in these last few years. His unpretentious opinions and objective deliberations has helped me in making better decisions. I’ll always cherish the countless hours spent with him at various coffee shops. I am also fortunate to have had the opportunity to share many laughs and unforgettable few trips with Arun Kumar, Mallik Rettiganti, and Aritra Sengupta. Nathan, has been the perfect office mate and has made the days and nights spent writing the thesis in the office more bearable. Among friends from my alma mater, Indian Statistical Institute, I particularly thank Suvajit Samanta for the many encouraging and supportive phone conversations; Sasikiran Goteti and Pritam Ranjan for the wonderful memories; they all made the first few years of my graduate school feel much more homely. Last but not the least, I would like to acknowledge Deepayan Sarkar for introducing and getting me hooked on to Debian and R for life. He has painstakingly guided me through the many hurdles in the initial stages. Finally, for my parents, Maa and Bapi, I truly cannot articulate my gratitude towards them in a few terse sentences. Not being able to be beside them through all these years and my growingly sparse habit of calling them has been considerably difficult on them. Yet, their understanding and acceptance of my whims and fancies has given me tremendous freedom and strength. What I am today is a tribute to their years of hardships; I can never imagine to repay that. I would also like to thank vi my kith and kins at Amulyadham for being ever so considerate. Their encouragement and affection has always been truly comforting.