Machine Learning with Datetime Feature Engineering: Predicting Healthcare Appointment No-Shows
By: Andrew Long
Tags: Data Science, Machine Learning, Feature Engineering, Healthcare

Machine Learning with Datetime Feature Engineering: Predicting Healthcare Appointment No-Shows

Dates and times are rich sources of information that can be used with machine learning models. However, these datetime variables do require some feature engineering to turn them into numerical data. In this post, I will demonstrate how to create datetime features with built in pandas functions for your machine learning models.

When I was teaching a machine learning course at Northeastern University, a few of my students used a really interesting dataset (medical appointment no-shows) which I am going to explore in this blog post. Medical appointment no-shows is a large problem in healthcare as approximately 1 in 5 patients will miss their appointment (source). This is an issue for everyone involved: 1) the scheduled patient presumably needs to be seen otherwise they wouldn’t have had an appointment, 2) other patients would have liked to have that spot but couldn’t, 3) healthcare providers must spend extra time to contact and re-schedule the patient as well as wasted any time they used to prepare for the visit.

Project Definition

Given the dates and times of scheduling day and appointment day, predict if a patient will miss their medical appointment.

Dataset

Here we will use the medical appointment no-show hosted on Kaggle (https://www.kaggle.com/joniarroba/noshowappointments). This dataset consists of over 110,000 medical appointments. The main three columns we will use for this project are the ScheduledDay (date and time when the appointment was scheduled), AppointmentDay(date of the appointment, no time included), No-Show (binary flag indicating if they no-showed). For the purposes of this post we will ignore the rest of the numerical features (although to be honest they added no value to the AUC).

Data Preparation

Let’s begin by loading our dataset, creating an output column (1 = no-show, 0= showed up), and converting our datetimes (currently strings) into python datetimes.

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

Here I assume that you downloaded the data from Kaggle and placed it in a ‘data’ folder:

df = pd.read_csv(‘data/KaggleV2-May-2016.csv’)

We can investigate the No-Show column with value_counts

Let’s define a binary column OUTPUT_LABEL to indicate Yes = 1, No = 0.

df[‘OUTPUT_LABEL’] = (df[‘No-show’] == ‘Yes’).astype(‘int’)

We can check the prevalence of our OUTPUT_LABEL:

def calc_prevalence(y):

return (sum(y)/len(y))

which means that 1 in 5 patients will miss their scheduled appointment.

Now let’s work with the datetime columns by looking at the first 5 rows of ScheduledDay and AppointmentDay

As you can see, the dtype for both columns is object which means pandas currently views these values as strings. Another thing to point out is that ScheduledDay has times where as Appointment Day has all the times as 00:00:00. We should probably deal with this, but I find it kind of odd that they didn’t included the time of the appointment in the dataset. Presumably, the time of the appointment would be predictive too.

To convert these strings into internal datetimes, we can use the pandas function to_datetime . I like to use the format parameter to specifically indicate the format. If you use the format parameter, you have to specify what to do with errors. Here I will make any errors turn into not a datetime (NaT). In general (although not the case here), using the format parameter will speed up the execution of this line.

df[‘ScheduledDay’] = pd.to_datetime(df[‘ScheduledDay’],

format = ‘%Y-%m-%dT%H:%M:%SZ’,

errors = ‘coerce’)

df[‘AppointmentDay’] = pd.to_datetime(df[‘AppointmentDay’],

format = ‘%Y-%m-%dT%H:%M:%SZ’,

errors = ‘coerce’)

I haven’t seen T and Z in standard formats before so I just put them in the format string. If someone knows how to handle these better, let me know.

Whenever I use the format parameter and if I assume all rows should have dates, I like to put an assertion statement to verify I didn’t get the format incorrect.

assert df.ScheduledDay.isnull().sum() == 0, ‘missing ScheduledDay dates’

assert df.AppointmentDay.isnull().sum() == 0, ‘missing AppointmentDay dates’

If you check the dtype now, you will see it is datetime64, which is what we want because it exposes us to all the datetime properties in pandas.

One thing I noticed was that currently there are ~40k appointments that were scheduled after the appointment datetime.

I think this is due to the fact that all the appointment times were set at the earliest time (00:00:00) whereas the times are included in ScheduledDay. To adjust for this, lets just shift all the appointment times to the end of the day. If I was doing this project for work, I would actually go and get the appointment times.

df[‘AppointmentDay’] = df[‘AppointmentDay’] +pd.Timedelta(‘1d’) — pd.Timedelta(‘1s’)

With this change there are only 5 rows where scheduledDay time is after appointmentDay. Let’s just drop those rows.

Engineer Datetime Features

By converting the strings into datetimes, this exposes all the pandas dt properties.

Basically you can break apart the date and get the year, month, week of year, day of month, hour, minute, second, etc. You can also get the day of the week (Monday = 0, Sunday = 6). Note be careful with week of year because the first few days of the year may be 53 if that week begins in the prior year. Let’s apply some of these properties to both of our datetime columns.

df[‘ScheduledDay_year’] = df[‘ScheduledDay’].dt.year

df[‘ScheduledDay_month’] = df[‘ScheduledDay’].dt.month

df[‘ScheduledDay_week’] = df[‘ScheduledDay’].dt.week

df[‘ScheduledDay_day’] = df[‘ScheduledDay’].dt.day

df[‘ScheduledDay_hour’] = df[‘ScheduledDay’].dt.hour

df[‘ScheduledDay_minute’] = df[‘ScheduledDay’].dt.minute

df[‘ScheduledDay_dayofweek’] = df[‘ScheduledDay’].dt.dayofweek df[‘AppointmentDay_year’] = df[‘AppointmentDay’].dt.year

df[‘AppointmentDay_month’] = df[‘AppointmentDay’].dt.month

df[‘AppointmentDay_week’] = df[‘AppointmentDay’].dt.week

df[‘AppointmentDay_day’] = df[‘AppointmentDay’].dt.day

df[‘AppointmentDay_hour’] = df[‘AppointmentDay’].dt.hour

df[‘AppointmentDay_minute’] = df[‘AppointmentDay’].dt.minute

df[‘AppointmentDay_dayofweek’] = df[‘AppointmentDay’].dt.dayofweek

You can verify this works:

At this point it would be good to explore our dates a bit.

As you can see here, the appointments are in April, May and June of 2016 and range from Monday through Saturday with no appointments on Sunday. I would never use Year as a feature (but showed it anyways) because presumably we want to use this predictive model in the future and those future years wouldn’t be included in the dataset. However, I am kind of disappointed that the months are just for a few months of the year. This means month (and consequently week of year) probably shouldn’t be used as a feature either. If I was doing this for work, I would go back to the database and get a whole year’s (or many years) worth of data. I would imagine that certain times of the year (ie around holidays) would impact the rate of no show.

Let’s quickly check if dayofweek is predictive of no-show:

Looks like more people skip their appointments on Friday and Saturday, although the effect is modest.

Another nice thing with pandas datetime representation is that you can calculate the ‘time’ between datetimes. Let’s create a new feature that is the number of days between the scheduled date and the appointment date.

df[‘delta_days’] = (df[‘AppointmentDay’]-df[‘ScheduledDay’]).dt.total_seconds()/(60*60*24)

Note that here I use total_seconds . There is a function dt.days but I got in the habit of using total_seconds because 1) dt.days rounds to the nearest day, 2) dt.days used to take much longer than total_seconds. The second point seems to have been fixed in later versions of pandas.

We can plot the histogram of our two classes on this variable:

plt.hist(df.loc[df.OUTPUT_LABEL == 1,’delta_days’],

label = ‘Missed’,bins = range(0,60,1), normed = True)

plt.hist(df.loc[df.OUTPUT_LABEL == 0,’delta_days’],

label = ‘Not Missed’,bins = range(0,60,1), normed = True,alpha =0.5)

plt.legend()

plt.xlabel(‘days until appointment’)

plt.ylabel(‘normed distribution’)

plt.xlim(0,40)

plt.show()

This distribution is a bit odd to me since most of the patients who did not miss their appointment scheduled the appointment on the same day. I kind of wonder if walk-in appointments are included in this data set. My guess is that this model will just draw a line at 1 day and say not-missed if you scheduled it the same day.

We are now ready to split our samples and train a model!

Split Samples

For simplicity, I’ll just split into two datasets: train (70%) and validation (30%). It is important to shuffle your samples because you may have been given the data in order of dates.

# shuffle the samples

df = df.sample(n = len(df), random_state = 42)

df = df.reset_index(drop = True) df_valid = df.sample(frac = 0.3, random_state = 42)

df_train = df.drop(df_valid.index)

We can check the prevalence is about 20% in each:

print(‘Valid prevalence(n = %d):%.3f’%(len(df_valid),calc_prevalence(df_valid.OUTPUT_LABEL.values)))

print(‘Train prevalence(n = %d):%.3f’%(len(df_train), calc_prevalence(df_train.OUTPUT_LABEL.values)))

Given this data comes from just Apr-Jun 2016 and their are no appointment times, we will just use these columns:

col2use = [‘ScheduledDay_day’, ‘ScheduledDay_hour’,

‘ScheduledDay_minute’, ‘ScheduledDay_dayofweek’,

‘AppointmentDay_day’,

‘AppointmentDay_dayofweek’, ‘delta_days’]

The ‘day’ features may even be suspect, but let’s leave in for now.

This could be expanded if we had:

appointments from the entire calendar year

appointment times

We can now build our X (inputs) and Y(output) for training and validation:

X_train = df_train[col2use].values

X_valid = df_valid[col2use].values y_train = df_train[‘OUTPUT_LABEL’].values

y_valid = df_valid[‘OUTPUT_LABEL’].values print(‘Training shapes:’,X_train.shape, y_train.shape)

print(‘Validation shapes:’,X_valid.shape, y_valid.shape)

Train a Machine Learning Model

Since the focus of this post is about the datetime features, we will just train a random forest model here. Note if you want to use other types of models, you may need to scale or normalize your data. Another thing you may want to do is convert the dayofweek into a categorical variable via one-hot encoding. We don’t need to do these things for a tree-based method though.

from sklearn.ensemble import RandomForestClassifier

rf=RandomForestClassifier(max_depth = 5, n_estimators=100, random_state = 42)

rf.fit(X_train, y_train)

We can then get our predictions with:

y_train_preds = rf.predict_proba(X_train)[:,1]

y_valid_preds = rf.predict_proba(X_valid)[:,1]

Evaluate performance

Here we will evaluate performance of the model. If you are new to classification metrics, I recommend checking out my posts about these metrics (technical post or non-technical post) .

from sklearn.metrics import roc_auc_score, accuracy_score, precision_score, recall_score

def calc_specificity(y_actual, y_pred, thresh):

# calculates specificity

return sum((y_pred < thresh) & (y_actual == 0)) /sum(y_actual ==0) def print_report(y_actual, y_pred, thresh):



auc = roc_auc_score(y_actual, y_pred)

accuracy = accuracy_score(y_actual, (y_pred > thresh))

recall = recall_score(y_actual, (y_pred > thresh))

precision = precision_score(y_actual, (y_pred > thresh))

specificity = calc_specificity(y_actual, y_pred, thresh)

print(‘AUC:%.3f’%auc)

print(‘accuracy:%.3f’%accuracy)

print(‘recall:%.3f’%recall)

print(‘precision:%.3f’%precision)

print(‘specificity:%.3f’%specificity)

print(‘prevalence:%.3f’%calc_prevalence(y_actual))

print(‘ ‘)

return auc, accuracy, recall, precision, specificity

Using this print_report function we can evaluate the performance for training and validation. Here I set the threshold at the prevalence of 0.201

We can plot the ROC with

from sklearn.metrics import roc_curve fpr_train, tpr_train, thresholds_train = roc_curve(y_train, y_train_preds)

auc_train = roc_auc_score(y_train, y_train_preds) fpr_valid, tpr_valid, thresholds_valid = roc_curve(y_valid, y_valid_preds)

auc_valid = roc_auc_score(y_valid, y_valid_preds) plt.plot(fpr_train, tpr_train, ‘r-’,label =’Train AUC:%.3f’%auc_train)

plt.plot(fpr_valid, tpr_valid, ‘b-’,label =’Valid AUC:%.3f’%auc_valid)

plt.plot([0,1],[0,1],’k — ‘)

plt.xlabel(‘False Positive Rate’)

plt.ylabel(‘True Positive Rate’)

plt.legend()

plt.show()

This indicates that we can get an AUC of 0.71 with just using datetime features. This ROC curve is kind of odd since it has that elbow.

We can investigate this a bit by looking at the top features

feature_importances = pd.DataFrame(rf.feature_importances_,

index = col2use,

columns=[‘importance’]).sort_values(‘importance’,

ascending=False) num = min([50,len(col2use)])

ylocs = np.arange(num)

# get the feature importance for top num and sort in reverse order

values_to_plot = feature_importances.iloc[:num].values.ravel()[::-1]

feature_labels = list(feature_importances.iloc[:num].index)[::-1] plt.figure(num=None, figsize=(6, 6), dpi=80, facecolor=’w’, edgecolor=’k’);

plt.barh(ylocs, values_to_plot, align = ‘center’)

plt.ylabel(‘Features’)

plt.xlabel(‘Importance Score’)

plt.title(‘Feature Importance Score — Random Forest’)

plt.yticks(ylocs, feature_labels)

plt.show()

Which shows that delta_days is basically the only features used in the model. This confirms our suspicion above that the model will likely struggle due to those same day appointments.

Since our training and validation scores are very similar, this means that we are in the case of high bias. In order to improve this model, we will need additional features, so I will end this project here.

Conclusion

In this project, we created many new features by unpacking the datetime variables to predict if a patient will no-show. In my experience, datetime features can have a big impact on healthcare machine learning models. I highly recommend trying this in your next project. Please reach out if you have any questions or concerns about this post.