Predicting Flight Delays
By: Stacy Shingleton
Tags: Xgboost, Supervised Learning, Classification Models, Confusion Matrix, Feature Importance

In order to test my knowledge of supervised machine learning, I was assigned a project to create a classification model. For this project, I was tasked with choosing my own dataset, one with sufficient complexity, a minimum of 1,000 rows and a minimum of 10 predictor columns. With this dataset, I would ask a question for which it could answer. After searching some of the large database resources, I landed on 2015 Flight Delays and Cancellations, from kaggle.com.

This dataset was published by the U.S. Department of Transportation’s (DOT) Bureau of Transportation Statistics and contains flight data for 14 major airlines during 2015. Here are some examples of what the data contains:

dates

airlines

origin and destination airports

departure/arrival times

departure delays

etc…

For this project, I built a classification model to predict whether a flight would be delayed by 15 minutes or more. In order to do so, I would need to filter down the dataset to a more reasonable size for modeling, as the full data set contained ~5.8 million entries. I chose to model flights only departing from George Bush International Airport (IAH) in Houston, Texas. This brought the dataset down to a more reasonable ~150K entries.

One thing the dataset did not contain, which I believed would be important for predicting delays, is weather data. In order to take weather into account, I conducting an API call from Dark Sky API, and collected weather data for every hour of 2015 at the IAH GPS coordinates.

I joined the weather dataset onto the IAH dataset using a date/time column where minutes were binned into every hour of every day. From here, I started cleaning the data, which consisted of identifying and correcting nan values, as well as identifying and deleting data leakage values. Data leakage occurs when you train your model using values that have the exact information you are trying to predict. For example, the flights data set contained 5 columns describing why the flight was delayed. If any of these columns contained a True value, then indeed, the flight was delayed. Consequently, these columns were deleted for the classification model, as well as any columns containing only one unique value, such as year, origin airport, etc.

The next step was to scale the continuous data. Before doing so, I needed to separate continuous data from categorical data, as categorical data should not be scaled, but instead, encoded, to either 0 or 1 (true or false) values. I scaled the continuous variables using StandardScaler() and encoded the categorical variables using OneHotEncoder(), both found in the scikit-learn preprocessing library. For this project, I decided to run a random forest model using RandomModelClassifier(), an XGBoost model with default hyperparameters, and a XGBoost model with a hyperparameter grid search, using XGBClassifier(). Each of the models would be run on the continuous data alone, as well as on the continuous plus categorical data. I wanted to run the continuous values alone in order to create a feature importance graph, which would not be possible once all of the categorical columns are encoded. An F1 score would be calculated for each model for comparison. The hyperparameter grid for XGBoost was optimized using a subset of only 10,000 values in order to save time on runs. The XGBoost models with a hyperparameter grid search provided the highest F1 score for each set of data, beating out the XGBoost with default hyperparamters and Random Forest model.

A feature importance graph was created from the XGBoost model with grid search and random forest model on the continuous data. The importance of each feature is decided while the model is constructing boosted trees, giving each attribute an importance score that represents how useful the attribute was when building the tree. The feature importance averages the scores from each individual tree to create the feature importance graph. The sum of all features will equal one. If one feature trumps all others, it may indicate a case of data leakage. The feature importance graph for my XGBoost model with grid search can be seen below. Data pulled from the weather API call is in lowercase, while the flight data is in capital letters.

From this graph, precipitation intensity, scheduled time for the entire flight, scheduled time of departure, and actual time of departure, have the highest importance scores for the model. It would be interesting to run a model on a origin airport with more extreme weather than Houston, for instance, Chicago O’hare, to see if weather features have higher importance than IAH.

Another visualization that describes the performance of a model is a confusion matrix. A confusion matrix numerically describes the true positives, true negative, false positives and false negatives for a model. For our model, a true positive means the model predicted correctly that the flight was delayed. A true negative means the model predicted correctly that the flight was not delayed. A false positive means the model predicted the flight would be delayed, when in fact, it was not delayed. A false negative means the model predicted the flight was not delayed, when in fact, it was delayed. The confusion matrix for the XGBoost, with grid search, is displayed below.

As mentioned, the XGBoost model with grid search yielded the highest F1 score for the classification model on both continuous data and continuous plus categorical data. XGBoost is a gradient boosting algorithm that provides parallel tree boosting. It has become the most popular and powerful library in machine learning. You are able to run XGBoost with default parameters, or you can create a grid search and optimize your hyperparameters. For this project, I ran both of these options, running the grid search optimization on a subset of only 10,000 values to save time. With the grid search, I optimized learning_rate, max_depth, min_child_weight and subsample. Learning_rate is the step size used in order to prevent overfitting. Max_depth is the depth of the tree — a large max_depth will increase complexity, computing needs, and will make the model more likely to overfit. Min_child_rate is the minimum sum of instance weight a leaf node must have before further partitioning. Subsample is the size of data XGBoost will sample prior to growing trees. These parameters and definitions can be seen in more detail in the XGBoost documentation linked below.

For each model, I calculated the recall, precision, accuracy and F1 score. Recall is the number of true positives divided by the number of actual total positives. You can get a 100% recall score by simply assuming every case is true, which wouldn’t necessarily be a better model. Precision is the number of true positives divided by the number of predicted positives. Precision can also be misleading if the model is very conservative about which cases they predict as true. Accuracy is simply the number of correct predictions divided by the number of total observations, very intuitive. But, accuracy can be very misleading with unbalanced datasets (when the true values are not 50/50 to false values — for instance, this dataset has only ~21% of flights delayed). F1 score represents the combination of recall and precision — an F1 score can only be high when both recall and precision are both high. For this reason, the F1 score is the best score for evaluating this particular model. With an F1 sore of 98.3% for my test values, the XGBoost with grid search parameters performed very well. Below are all the scores from this model.