Ultimate Function for Classification
By: Darius Fuller
Tags: Data Science, Classification, Scikit Learn, Functional Programming, UFC

Background

I was given a simple project: use a premade data set to create a use case for a classification model, then create that model. Essentially with a blank slate, I faced an entire internet’s worth of possible data sets to take as the one. Eventually I settled on something sports related, and out of all of the sports I expected to come across, I found one on UFC fights.

If you aren’t familiar with the sport, the Ultimate Fighting Championship (UFC) is a mixed martial arts promoter. Mixed martial arts, frequently referred to as MMA, is a fighting style that is a non-specific blend of multiple traditional fighting disciplines. A great example of blending fighting styles is the fighter Jon Jones. He went from a national junior college wrestling champion to obtaining a purple belt in Brazilian Jiu-Jitsu (BJJ) and a black belt in Gaidojutsu; Gaidojutsu itself being a fighting style developed in 1992 by Greg Jackson using techniques from catch wrestling, BJJ, kickboxing, and judo.

Beyond the greater freedom in how one is allowed to fight in MMA matches, the structure generally follows that of others combat sports such as boxing or wrestling. If you are curious about how it all works on a finer level, I recommend reading the “Rules” section on this Wikipedia page.

I was a big fan during the early 2010s; I had the video games, watched the shows, knew the fighters, etc. If there was one thing I remember about watching these fights, it’s how unpredictable they felt. It seemed that until someone gets knocked down or put in a submission that either fighter could win, no matter the situation. No matter how much of an underdog a particular fighter was statistically, they still had a chance to win and occasionally would shock the MMA community in doing so. It was those unpredictable moments that inspired me to use this data set for my project. The big question I wanted to answer was:

Can I predict the winner of a MMA fight based upon their prior performance?

Data Set

As mentioned before, one of my few restrictions in this project was that the data set needed to be already compiled before I start working with it. I started by going to Kaggle and a lot of time looking up data sets. While using the “sports” tag I found “UFC-Fight historical data from 1993 to 2019” posted by Rajeev Warrier (link). This data set had an extensive amount of information on over 5,000 fights, containing 145 features on various aspects of the fights.

There is information about:

A fighter’s Win/Loss/Draw record prior to fight

A fighter’s method of winning a number of fights

Preferred fighting Stance

Physical Attributes

Fight related averages

**These averages dealt with advanced stats such as: significant strikes, submissions, passing on the ground, etc.**

What to Bring

You will need to have these packages installed in order to follow along:

Sci-Kit Learn for the classifiers and most metrics

Matplotlib for the plotting methods

Pandas for the plotting method

Yellowbrick for the ROCAUC class

I will be walking through the creation of this function with respect to my project on the UFC, although this function should work on any cleaned data set prepped for machine learning. If you need any information on how to do that and some good strategies, check out this data cleaning guide by Omar Elgabry (link).

Otherwise, for brevity I will skip directly into the point where I can create my classifier evaluation function. At this point with the data set I have:

Filled and/or removed any missing data

One-hot encoded categorical features

Min-max scaling on numerical features

Train-test split into X and y data

Used SMOTE to synthetically resample training data (fixes target class imbalance)

Here’s a small peek into what the DataFrame looks like:

Columns detailing red corner fighter’s scaled information

Fitting and Predicting

This is the easy part! Sci-Kit Learn really makes it simple to fit a model to training data and predict on new inputs. All you need to do is first instantiate your Sci-Kit Learn machine learning model of choice and use it’s .fit() and .predict() methods.

Setup, fitting, and predicting using multiclass logistic regression model

One unique aspect about this code is the use of the multi_class= parameter, which will only be relevant if you are trying to predict more the two classes. In my project, for each fighter I was looking to predict if their match will end (for them) in a:

Win

Lose

Draw

This important detail is why we will need to use the Yellowbrick package later when visualizing the performance of our model when predicting outcomes, but for now let’s look at how to functionize this process.

Fitting and predicting on a Sci-Kit Learn classifier generalized

If you take a closer look, there really is not much of a change here. The variables (including the classifier itself) needed to be generalized and placed as required parameters. But besides the addition of a “return” statement for the predictions, that is all I needed to do!

Getting Results

So now that I have a model that can predict the winner of a UFC fight, I need a way to confirm it’s accuracy. I can do this by using the data fed in as inputs for those same predictions as the ground truth, since we already had the correct outcomes for the model to learn from. This is why this type of task is called supervised learning, as we will be able to adjust (or supervise) how our model learns in hopes of improving it’s performance when predicting on unseen data.

Once again, Sci-Kit Learn makes this super easy to do once a model has been trained upon some data. The classification report function found in the metrics module can generate the four commonly used classification measures in a convenient manner for the user. All that one needs to do is feed in the true and predicted labels for the data to get results!

Here’s an example from my project:

Example classification report from K-Neighbors classifier

For such a simple implementation, the classification report from Sci-Kit Learn brings a lot to the table when trying to evaluate the performance of a classifier. It provides the main classification metrics on an aggregate and individual label level, which can help to identify strategies for tuning your model or explaining it’s performance. In a general sense, these metrics are:

Precision — The ratio of true positives to all predicted positives; Models with high precision rarely predict the incorrect label but often have many false negatives

Recall — The ratio of true positives to all possible positives; Models with high recall often capture many false positives

F1-Score — Computed using precision and recall scores; Used in an effort to minimize both false negatives and false positives

Accuracy — The ratio of true positives to all predictions; Simply the percentage of how many predictions were correct

If this stuff seems a bit off and you’d like to know some more, I recommend checking this post out Clare Liu (link).

Showing Off

Although it may seem as if the classification report has everything already (it kind of does), there is actually couple more things that can be easily added into this function to help round out the information displayed. I will next show how to create a confusion matrix and a ROC curve before putting it all together into one function.

Confusion matrix

A confusion matrix is a great tool for visualizing the performance of a classification model. In a nutshell, it is a square divided into smaller squares that represent the values for true and false positives in addition to the true and false negatives. It’s a bit easier to understand visually:

Example confusion matrix for binary classifier

Starting from the top left and going diagonally downwards, these values represent true values; alternatively they are the recall values for that specific class. Additional information about them can be found in this post by Sarang Narkhede (link).

Confusion matrices are also easy to implement thanks to Sci-Kit Learn, here’s how to do it:

ROC curve

The second plot that I added into my function is a bit advanced, but can offer some interesting insights when evaluating a model’s performance. The ROC (Receiver Operating Characteristic) curve, in simple terms, illustrates the tradeoff between the true positive rate (TPR) and false positive rate (FPR). If you want to get more into the math behind these rates, Wikipedia does a dense, but great job detailing the formulas.

The ideal ROC curve will look like a smoother, sloped version of an upside-down “L”. Hopefully this will make more sense after looking at one:

Example ROC curve (bad performance)

The above example is from one of the poorer performing models I tested during my project. The two lines representing the TPRs and FPRs for each class stay relatively close to the black dotted line; this black line represents the “random chance” line, where the chance of being correct is essentially 50%. Thus, this model was not much better than flipping a coin and choosing which fighter would win. An ideal ROC curve would have a TPR that increases faster than the FPR, resulting in that upside-down “L” shape I was referring to.

Normally, if one is training a binary classification model, this plot can be produced using Sci-Kit Learn. However, since I was predicting three different classes, I will show how I created a ROC curve using the Yellowbrick package that allows for multi-class curves (no need to specify either).

Here’s how I did it:

Single implementation of Yellowbrick ROC curve

Wrapping It Up

Now its time to put all the code from above together into one, reusable function that can produce everything previously discussed in one line of code!

I’ll break it down in steps:

Fit the model and create predictions Generate classification report using predictions Create figure and axes for plots Generate and stylize confusion matrix Generate and stylize ROC curve Display plots Return predictions

Here we go! (Mario voice):

Functionized version of previous sections

Side note: In line 48 I use a custom function named plot_importance(). It’s purpose as an add-on is to display, as an option, an additional plot that is only relevant to a decision tree classification model. More on feature importances; code for plot_importance().

Here’s an example of what this outputs:

Example of vanilla logistic regression trained on UFC data

What Now?

So if you’ve been following along, you should have a fully functioning, function that will easily display a general report and two plots concerning the performance of a supervised classification model. Through the use of this function, one can quickly assess how well a model is learning the training data; doing so on a deeper level than each of the three tools could provide alone. I intended this function to be used as a judge for multiple classifiers when selecting one to train on a specific data set and also as a way to give insight during the tuning process. I would be interested to see what other uses may come of it as-is or altered (let me know!).

Granted in order to create this function, there does need to be some preparation, but overall using it saved me a bunch of time during this project. I hope it will do the same for others!

Link to UFC classification project