Introduction to Survival Analysis: the Kaplan-Meier estimator | by Eryk Lewinson | Towards Data Science

In my previous article, I described the potential use-cases of survival analysis and introduced all the building blocks required to understand the techniques used for analyzing the time-to-event data.
I continue the series by explaining perhaps the simplest, yet very insightful approach to survival analysis — the Kaplan-Meier estimator. After a theoretical introduction, I will show you how to carry out the analysis in Python using the popular lifetimes library.
The Kaplan-Meier estimator (also known as the product-limit estimator, you will see why later on) is a non-parametric technique of estimating and plotting the survival probability as a function of time. It is often the first step in carrying out the survival analysis, as it is the simplest approach and requires the least assumptions. To carry out the analysis using the Kaplan-Meier approach, we assume the following:
The event of interest is unambiguous and happens at a clearly specified time.
The survival probability of all observations is the same, it does not matter exactly when they have entered the study.
Censored observations have the same survival prospects as observations that continue to be followed.
In real-life cases, we never know the true survival function. That is why with the Kaplan-Meier estimator, we approximate the true survival function from the collected data. The estimator is defined as the fraction of observations who survived for a certain amount of time under the same circumstances and is given by the following formula:
where:
t_i is a time when at least one event happened,
d_i is the number of events that happened at time t_i,
n_i represents the number of individuals known to have survived up to time t_i (they have not yet had the death event or have been censored). Or to put it differently, the number of observations at risk at time t_i.
From the product symbol in the formula, we can see the connection to the other name of the method, the product-limit estimator. The survival probability at time t is equal to the product of the percentage chance of surviving at time t and each prior time.
What we most often associate with this approach to survival analysis and what we generally see in practice are the Kaplan-Meier curves — a plot of the Kaplan-Meier estimator over time. We can use those curves as an exploratory tool — to compare the survival function between cohorts, groups that received some kind of treatment or not, behavioral clusters, etc.
The survival line is actually a series of decreasing horizontal steps, which approach the shape of the population’s true survival function given a large enough sample size. In practice, the plot is often accompanied by confidence intervals, to show how uncertain we are about the point estimates — wide confidence intervals indicate high uncertainty, probably due to the study containing only a few participants — caused by both observations dying and being censored. For more details on the calculation of the confidence intervals using the Greenwood method, please see [2].
The interpretation of the survival curve is quite simple, the y-axis represents the probability that the subject still has not experienced the event of interest after surviving up to time t, represented on the x-axis. Each drop in the survival function (approximated by the Kaplan-Meier estimator) is caused by the event of interest happening for at least one observation.
The actual length of the vertical line represents the fraction of observations at risk that experienced the event at time t. This means that a single observation (not actually the same one, but simply singular) experiencing the event at two different times can result in a drop of difference size — depending on the number of observations at risk. This way, the height of the drop can also inform us about the number of observations at risk (even when unreported and/or there are no confidence intervals).
When no observations experienced the event of interest or some observations were censored, there is no drop in the survival curve.
We have learned how to use the Kaplan-Meier estimator to approximate the true survival function of a population. And we know we can plot multiple curves to compare their shapes, for example, by the OS the users of our mobile app use. However, we still do not have a tool that will actually allow for comparison. Well, at least a more rigorous one than eyeballing the curves.
That is when the log-rank test comes into play. It is a statistical test that compares the survival probabilities between two groups (or more, for that please see the Python implementation). The null hypothesis of the test states that there is no difference between the survival functions of the considered groups.
The log-rank test uses the same assumptions as of the Kaplan-Meier estimator. Additionally, there is the proportional hazards assumption — the hazard ratio (please see the previous article for a reminder about the hazard rate) should be constant throughout the study period. In practice, this means that the log-rank test might not be an appropriate test if the survival curves cross. However, this is still a topic of active debate, please see [4] and [5].
For brevity, we do not cover the maths behind the test. If you are interested, please see this article or [3].
In this part, I wanted to mention some of the common mistakes that can occur while working with the Kaplan-Meier estimator.
It might be tempting to remove censored data as it can significantly alter the shape of the Kaplan-Meier curve, however, this can lead to severe biases so we should always include it while fitting the model.
Pay special attention when interpreting the end of the survival curves, as any big drops close to the end of the study can be explained by only a few observations reaching this point of time (this should also be indicated by wider confidence intervals)
By dichotomizing I mean using the median or “optimal” cut-off point to create groups such as “low” and “high” regarding any continuous metric. This approach can create multiple problems:
Finding an “optimal“ cut-off point can be very dataset-dependent and impossible to replicate in different studies. Also, by doing multiple comparisons, we risk increasing the chances of false positives (finding a difference in the survival functions, when actually there is none).
Dichotomizing decreases the power of the statistical test by forcing all measurements to a binary value, which in turn can lead to the need for a much larger sample size required to detect an effect. It is also worth mentioning that with survival analysis, the required sample size refers to the number of observations with the event of interest.
When dichotomizing, we make poor assumptions about the distribution of risk among observations. Let’s assume we use the age of 50 as the split between young and old patients. If we do so, we assume that an 18-year-old is in the same risk group as a 49-year-old, which is not true in most of the cases.
The Kaplan-Meier estimator is a univariable method, as it approximates the survival function using at most one variable/predictor. As a result, the results can be easily biased — either exaggerating or missing the signal. That is caused by the so-called omitted-variable bias, which causes the analysis to assume that the potential effects of multiple predictors should be attributed only to the single one, which we take into account. Because of that, multivariable methods such as the Cox regression should be used instead.
It is time to implement what we have learned in practice. We start by importing all the required libraries.
Then, we load the dataset and do some small wrangling to make it work nicely with the lifelines library. For the analysis, we use the popular Telco Customer Churn dataset (available here or on my GitHub). The dataset contains client information of a telephone/internet provider, including their tenure, what kind of services they use, some demographical data, and ultimately the flag indicating churn.
For this analysis, we use the following columns:
tenure — the number of months the customer has stayed with the company,
churn — information whether the customer churned (binary encoded: 1 if the event happened, 0 otherwise),
PaymentMethod— what kind of payment method the customers used.
For the most basic scenario, we actually only need the time-to-event and the flag indicating if the event of interest happened.
The KaplanMeierFitter works similarly to the classes known from scikit-learn: we first instantiate the object of the class and then use the fit method to fit the model to our data. While plotting, we specify at_risk_counts=True to additionally display information about the number of observations at risk at certain points of time.
Normally, we would be interested in the median survival time, that is, the point in time in which on average 50% of the population has already died, or in this case, churned. We can access it using the following line:
kmf.median_survival_time_
However, in this case, the command returns inf, as we can see from the survival curve that we actually do not observe that point in our data.
We have seen the basic use-case, now let’s complicate the analysis and plot the survival curves for each variant of the payment method. We can do so by running the following code:
Running the block of code generates the following plot:
We can see that the probability of survival is definitely the lowest for the electronic check, while the curves for automatic bank transfer/credit card are very similar. This is a perfect time to use the log-rank test to see if they are actually different.
The following table presents the results.
By looking at the p-value of 0.35, we can see that there are no reasons to reject the null hypothesis stating that the survival functions are identical. For this example, we only compared two methods of payment. However, there are definitely more combinations we could test. There is a handy function called pairwise_logrank_test, which makes the comparison very easy.
In the table, we see the previous comparison we did, as well as all the other combinations. The bank transfer vs. credit card is the only case in which we should not reject the null hypothesis. Also, we should be cautious about interpreting the results of the log-rank test, as we can see in the plot above that the curves for the bank transfer and credit card payments actually cross, so the assumption of proportional hazards is violated.
There are two more things we can easily test using the lifelines library. The first one is the multivariate log-rank test, in which the null hypothesis states that all the groups have the same “death” generating process, so their survival curves are identical.
The results of the test indicate that we should reject the null hypothesis, so the survival curves are not identical, which we have already seen in the plot.
Lastly, we can test the survival difference at a specific point in time. Coming back to the example, in the plot, we can see that the curves are furthest apart around t = 60. Let’s see if that difference is statistically significant.
By looking at the test’s p-value, there is no reason to reject the null hypothesis stating that there is no difference between the survival at that point of time.
In this article, I described a very popular tool for conducting survival analysis — the Kaplan-Meier estimator. We also covered the log-rank test for comparing two/multiple survival functions. The described approach is a very popular one, however, not without flaws. Before concluding, let’s take a look at the pros and cons of the Kaplan-Meier estimator/curves.
Advantages:
Gives the average view of the population, also per groups.
Does not require a lot of features — only the information about the time-to-event and if the event actually occurred. Additionally, we can use any categorical features describing groups.
Automatically handles class imbalance, as virtually any proportion of death to censored events is acceptable.
As it is a non-parametric method, few assumptions are made about the underlying distribution of the data.
Disadvantages:
We cannot evaluate the magnitude of the predictor’s impact on survival probability.
We cannot simultaneously account for multiple factors for observations, for example, the country of origin and the phone’s operating system.
The assumption of independence between censoring and survival (at time t, censored observations should have the same prognosis as the ones without censoring) can be inapplicable/unrealistic.
When the underlying data distribution is (to some extent) known, the approach is not as accurate as some competing techniques.
Summing up, even with a few disadvantages the Kaplan-Meier survival curves are a great place to start off while conducting survival analysis. While doing so, we can get valuable insights about the potential predictors of survival and accelerate our progress with some more advanced techniques (which I will describe in future articles).
You can find the code used for this article on my GitHub. As always, any constructive feedback is welcome. You can reach out to me on Twitter or in the comments.
In case you found this article interesting, you might also like the other ones in the series:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1] Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282), 457–481. — available here
[2] S. Sawyer (2003). The Greenwood and Exponential Greenwood Confidence Intervals in Survival Analysis — available here
[3] Kaplan-Meier Survival Curves and the Log-Rank Test — available here
[4] Non-proportional hazards — so what? — available here
[5] Bouliotis, G., & Billingham, L. (2011). Crossing survival curves: alternatives to the log-rank test. Trials, 12(S1), A137.