Schoenfeld Residuals: The idea that turned regression modeling on its head
By: Sachin Date
Tags: Regression Analysis, Survival Analysis, Getting Started, Data Science, Schoenfeld Residuals

How to Calculate the Schoenfeld Residuals?

The calculation of Schoenfeld residuals is best described by fitting the Cox Proportional Hazards model on a sample data set.

We’ll use the Stanford heart transplant data set which is a data set of 103 heart patients who have been voluntarily admitted into a study after it was determined that a transplant was the only option left for them. The study collected various variables related to each individual such as their age, evidence of prior open heart surgery, their genetic makeup etc. If they received a transplant during the study, this event was noted down. It was also noted down how many days elapsed before an individual died irrespective of whether they received a transplant. Some individuals left the study for various reasons or they were still alive when the study ended. These ‘lost-to-observation’ cases constituted what are known as ‘right-censored’ observations.

I have uploaded the CSV version of this data set at this location.

Using Python and Pandas, let’s start by loading the data into memory:

import pandas as pd

df = pd.read_csv('stanford_heart_transplant_dataset_full.csv')

Let’s print out the columns in the data set:

df.columns

We see the following list:

Index(['PATIENT_ID', 'YR_OF_ACCEPTANCE', 'AGE', 'SURVIVAL_STATUS',

'SURVIVAL_TIME', 'PRIOR_SURGERY', 'TRANSPLANT_STATUS',

'WAITING_TIME_FOR_TRANSPLANT', 'MISMATCH_ON_ALLELES',

'MISMATCH_ON_ANTIGEN', 'MISMATCH_SCORE'],

dtype='object')

The columns of immediate interest to us are the following ones:

SURVIVAL_TIME: The number of days the patient survived after induction into the study. This is our response variable y.

SURVIVAL_STATUS: 1=dead, 0=alive at SURVIVAL_TIME days after induction.

We’ll consider the following three regression variables which will form our regression variables matrix X:

AGE: The patient’s age when they were inducted into the study.

PRIOR_SURGERY: Whether the patient had at least one open-heart surgery prior to entry into the study.1=Yes, 0=No

TRANSPLANT_STATUS: Whether the patient received a heart transplant while in the study. 1=Yes, 0=No

Let’s carve out a vertical slice of the data set containing only columns of our interest:

df2 = df[['AGE', 'PRIOR_SURGERY', 'TRANSPLANT_STATUS', 'SURVIVAL_TIME', 'SURVIVAL_STATUS']].copy() df2.head()

We see the following output:

Top 5 rows of the data set (Image by Author)

Let’s fit the Cox PH model from the Lifelines library on this data set.

Let’s import the Cox model:

from lifelines import CoxPHFitter

Create and train the Cox model on the training set:

cph_model = CoxPHFitter() cph_model.fit(df=df2, duration_col='SURVIVAL_TIME', event_col='SURVIVAL_STATUS')

Display the model training summary:

cph_model.print_summary()

We get the following output:

Model summary of the Cox model (Image by Author)

Here are the fitted coefficients and their exponents of the three regression variables:

Fitted coefficients of the Cox model (Image by Author)

These three coefficients form our β vector:

β=[0.06, -0.74, -1.66]

The Schoenfeld residuals are calculated for each regression variable to see if each variable independently satisfies the assumptions of the Cox model. In our case those would be AGE, PRIOR_SURGERY and TRANSPLANT_STATUS.

We’ll show how the Schoenfeld residuals can be calculated for the AGE variable.

To illustrate the calculation for AGE, let’s focus our attention on what happens at row number # 23 in the data set. We see that one death has occurred at T=30 days. The set of patients who were at ‘at-risk’ of dying just before T=30 are shown in the red box below:

The at-risk set R_30 at t=30 days (Image by Author)

The set of indices [23, 24, 25,…,102] form our ‘at-risk’ set R_30 corresponding to the event occurring at T=30 days. Out of this at-risk set, the patient with ID=23 is the one who died at T=30 days.

Let’s carve out the X matrix consisting of only the patients in R_30:

import numpy as np X30 = np.array(df2[['AGE', 'PRIOR_SURGERY', 'TRANSPLANT_STATUS']])[23:,]

We get the following X matrix that was shown inside the red box in the earlier figure:

X30 (Image by Author)

Let’s focus on the first column (column index 0) of X30. This is the AGE column and it contains the ages of the volunteers at risk at T=30. We’ll denote it as X30[…][0] where the three dots denote all rows in X30.

What we want to do next is estimate the expected value of the AGE column.

In other words, we want to estimate the expected age of the study volunteers who are at risk of dying at T=30 days.

Notice that this strategy effectively fixes the value of response variable y to a known value (30 days) and it makes X30[…][0] i.e. the age of the volunteer as the random variable having an expected value and a variance!

The expected age of at-risk volunteers in R_30 can be calculated by the usual formula for expectation namely the value times the probability summed over all values:

Expected age of volunteers at risk of dying at T=30 (Image by Author)

In the above equation, the summation is over all indices in the ‘at-risk’ set R30.

From the earlier discussion about the Cox model, we know that the probability of the jth individual in R30 dying at T=30 is given by:

Probability of the jth individual in R30 dying at T=30 (Image by Author)

We plug this probability into the earlier equation for E(X30[…][0]) to get the following formula for the expected age of individuals who were at risk of dying at T=30 days:

Expected age of volunteers at risk of dying at T=30 (Image by Author)

Similarly, we can get the expected values for PRIOR_SURGERY and TRANSPLANT_STATUS regression variables by replacing the index 0 in the above equation with 1 and 2 respectively.

The above equation for E(X30[…][0]) can be generalized for the ith time instant at which a significant event (such as death) occurs. For T=t_i, the ‘at-risk’ set is R_i and expected value of the mth regression variable i.e. E(Xi[…][m]) can be estimated as follows:

Expected value of the mth regression variable of volunteers at risk of dying at T=t_i (Image by Author)

Let’s put these equations to work by calculating the expected age of patients in R30 for our sample data set.

We’ll use a little bit of very simple matrix algebra to make the computation more efficient.

#The regression coefficients vector β of shape (3 x 1)

Beta = np.array(cph_model.params_).reshape(-1,1) #exp(X30.Beta). A vector of size (80 x 1). Note that X30 has a shape (80 x 1)

exp_X30_Beta = np.exp(np.matmul(X30, Beta)) #The summation in the denominator (a scaler quantity)

sum_exp_X30_Beta = np.sum(exp_X30_Beta) #The Cox probability of the kth individual in R30 dying0at T=30. A vector of shape (80 x 1)

p_k = exp_X30_Beta/sum_exp_X30_Beta #Column 0 (Age) in X30, transposed to shape (1 x 80)

X30_0 = X30[:,0].reshape(1,-1) #Expected Age of volunteers in R30

expected_age_at_R30 = np.matmul(X30_0, p_k) print('Expected age of volunteers in R30='+str(expected_age_at_R30))

We get the following output:

Expected age of volunteers in R30=[[47.83197965]]

Next, we subtract the observed age from the expected value of age to get the vector of Schoenfeld residuals r_i_0 corresponding to T=t_i and risk set R_i.

r_i_0 is a vector of shape (1 x 80).

r_i_0 = X30_0-expected_age_at_R30

The value of the Schoenfeld residual for Age at T=30 days is the mean value (actually a weighted mean) of r_i_0:

mean_r_i_0 = np.mean(r_i_0) print(mean_r_i_0)

Which prints the following:

-3.46947965294184

In practice, one would repeat the above procedure for each regression variable and at each time instant T=t_i at which the event of interest such as death occurs. That results in a time series of Schoenfeld residuals for each regression variable.

It is also common practice to scale the Schoenfeld residuals using their variance.

Thankfully, you don’t have to hand crank out the residuals like we did! All major statistical regression libraries will do all the hard work for you. We’ll soon see how to generate the residuals using the Lifelines Python library.