Survival Modeling — Accelerated Failure Time — XGBoost | by Avinash Barnwal, Ph.D. | Towards Data Science

Survival analysis is a “censored regression” where the goal is to learn time-to-event function. This is similar to the common regression analysis where data-points are uncensored. Survival modeling is not as equally famous as regression and classification. Therefore, I would explain it more in detail with example. This is a modeling task that has censored data. Censored data are the data where the event of interest doesn’t happen during the time of study or we are not able to observe the event of interest due to some reasons.
Censoring Types and Examples
Left Censoring occurs when we start the study/campaign and the event has already happened leading to higher Example — We want to study about users buying new iPhone given offers, but some users might have already bought it is not useful to throw this data out.
Interval Censoring is used when we don’t know the exact time of the event and it could happen between time interval. Example — Users buying Insurance where users not sure about the exact date of purchase.
Right Censoring is used when an event has not happened during the time of the study. Example — Treatment received to the cancer patients.
Time-to-event modeling is critical for understanding users/companies' behaviors not limited to credit, cancer, and attrition risks.
Cox-Proportional Hazard model is a semi-parametric model where we model hazard ratio using predictors while in accelerated Failure time log of survival time is modeled using predictors. It is parametric survival modeling as we are assuming the distribution of response data.
Below is the Flow-Chart of Survival Modeling-
Tree-based models have improved the supervised learning to lead to many prizes on Kaggle and performed better in the industry as well. We have gradient boosting models implemented in R and python both for Cox-Proportional Hazard Function and Accelerated Failure Time. It is natural to develop more tree-based models for survival modeling as well. For Example — GBM, mboost, Scikit-survival and etc. Currently, XGBoost supports the Cox-Ph model without baseline prediction. Therefore, we have included Accelerated Failure Time in Xgboost to improve survival modeling support.
Most of the survival models support only the right censoring data types. We have generally 4 types of data — Right, Left, Interval, and Uncensored. Right censored is the most used censored data type. It is also important to understand other censoring types.
Accelerated Failure Time Modeling is parametric survival modeling where survival time follows certain distribution, unlike Cox-Proportional Hazard Model. We support 3 types of distribution — Normal, Logistic, and Extreme. The normal distribution is the most common distribution type but it has a thin tail which might not be the right choice in case of the fat tail events or of extreme events.
It's better to visualize it.
import numpy as npimport seaborn as snssns.distplot(np.random.normal(0,1,200),hist=False,label=”Normal”)sns.distplot(np.random.logistic(0,1,200),hist=False,label=”Logistic”)sns.distplot(np.random.gumbel(0,1,200),hist=False,label=”Extreme”)
When data have fat tails, Logistic/Extreme would lead to better results.
Hyperparameter Tuning
Hyperparameter optimization is one of the important tasks for machine learning. We have kept the error of the response(\sigma) and base score as hyperparameters. There are many methods to find the best hyperparameter.
Grid Search is one of those. We have used GridSearch to find the best hyperparameters.
We can also start with sigma equals to an average of sigma of y lower and sigma of y higher for interval-censored datasets and accordingly for different censored datasets and base score equals to average of y lower and y higher.
We have used ChIP-seq data sets available here. It has multiple datasets. The output/response of the dataset is log-transformed. We will be using 5 datasets — ATAC_JV_adipose, CTCF_TDH_ENCODE, H3K27ac-H3K4me3_TDHAM_BP, H3K27ac_TDH_some, and H3K36me3_AM_immune. In all datasets, folds are already defined.
Base-Score and Sigma are the key hyper-parameters.
Below is the code for generating the performance of Accelerated Failure Time with different distributions and sigma.
R Code
Parametric Censored Regression — Accelerated Failure Time is a powerful alternative to Cox-Proportional Hazard Model. We need more support of the above model in different tree-based models like gradient boosting, XGBoost, and LightGBM. We have implemented it in XGBoost now. Most of the datasets have been flat while varying the sigma meaning we might need more iteration. For Normal, sigma plays a crucial role as it gets flattered when sigma is increased.
Hyperparameter tuning is an important aspect here as we might find that there is no movement of training/test loss. It is soon going to be available for all in the XGBoost package.
[1] Project Description
[2] More Details
[3] Derivation
P.S. — We have submitted the paper related to this to NeurIPS — 2020 and Arxiv Version of the paper is here — https://arxiv.org/abs/2006.04920.