Robust and effective prediction for large and complex data with conformal prediction method
Creators: name: Zheng, Zheshi
Publisher: name: No Publisher Supplied
Year: 2023

description: Real-world data often have complex genesis and structures that cannot be perfectly described by a mathematical model or standard assumptions. Furthermore, the underlying randomness mechanisms are often unknown and elusive, raising doubts on the reliability of estimates and results for decision-making. In this context, it is crucial to develop methods that are robust to model misspecification. Building on a recently developed machine learning prediction framework known as conformal prediction, this dissertation aims to provide a widely applicable approach for conducting robust and effective statistical analyses in three application domains -- survival analysis, time series forecasting, and outlier detection. The research on survival analysis is motivated by the need to make a risk prediction for seniors to develop Alzheimer's disease (AD) in a project sponsored by the OneFlorida Consortium. Existing methods for predictive inference in survival analysis have several limitations: Bayesian and model-based frequentist methods rely heavily on the accuracy of model assumptions; while non-parametric methods like Kaplan Meier curve do not provide individualized results. To address these limitations, we adopt conformal prediction method to develop robust and individualized lower predictive bounds on patient's time to AD with both unconditional and conditional coverage. A challenge is that the survival data are subject to potential censoring and existing methods have strong assumptions on the censoring mechanism that do not apply to our OneFlorida dataset. We develop a novel data-driven imputing method and provide an effective conformal predictive inference. This development can handle both the dependent and independent censoring mechanism. We validate the proposed method through theoretical and numerical results and show its superiority over existing prediction methods. In time series forecasting, traditional model-and-distribution-based methods require correct working models and data distribution assumptions to obtain valid results. While recent machine learning methods such as random forest and neural networks have good empirical results, they lack uncertainty quantification in the form of prediction intervals. To address these issues, we propose to apply conformal prediction to time series forecasting and obtain robust and effective prediction intervals for future outcomes. The major challenge for conformal prediction in time series data is the dependence structure in the data, and existing methods bypass this issue by assuming they have a correct working model and treating the residuals as iid. In this thesis, however, we delve into a detailed examination of dependent structure of time series data by imposing stationary and mixing conditions for both the observed time series and the working model. We demonstrate that obtaining a correct working model is not necessary for a valid prediction interval with conformal prediction methods. We further derive prediction intervals with both conditional and unconditional coverage and prove their validity both theoretically and numerically. For outlier detection problem, existing methods can be classified into two categories: model-based methods, which rely on strong assumptions on the underlying distribution; and clustering-based methods, which utilize artificial thresholds on the distance metrics. In this project, we propose a conformal prediction based framework for outlier detection with uncertainty quantification. We define an "outlyingness score" based on the conformity score, which enables us to detect outliers with uncertainty quantification and model robustness. Under the multiple hypothesis testing setting, we aim to detect outliers in any testing set with a given reference set that contains no outliers. However, we found that the standard conformity outlier detection (COD) method has low power when the reference set is small compared to the testing set. To address this limitation, we further propose an extreme value theorem enhanced COD method (EVT-enhanced COD) that improves the performance of our algorithm. Numerical results show significant improvement by adopting the EVT-enhanced COD method.