End-to-End Project of Game Prediction Based on LeBron’s Stats Using Three Machine Learning Models | by Yufeng | Towards Data Science

I am a huge fan of machine learning and basketball, so I like generating some mini-projects by combining these two. In this post, I would like to share with you one of these projects.
No matter whether you are a basketball fan or not, you must know LeBron James. As a core player, his performance is essential to the game result. So, the question I try to answer in this project is “Can we predict the game result based on LeBron’s game stats?”
I framed it into a binary classification problem with “Win” or “Lose” of the team as the output labels. The features are the basic statistics of LeBron James in each game.
I implemented three classifiers in the project, Logistic Regression, Random Forest Classifier, and Deep Learning Classifier, by using two popular Python libraries in machine learning, sklearn, and keras.
I am going through the project step by step and I hope it is helpful to you.
The libraries used in the code are listed here.
import pandas as pdimport numpy as npfrom sklearn.model_selection import train_test_splitfrom sklearn.model_selection import StratifiedKFoldfrom sklearn.impute import SimpleImputerfrom sklearn.preprocessing import StandardScalerfrom sklearn.pipeline import Pipelinefrom sklearn.linear_model import LogisticRegressionfrom sklearn.ensemble import RandomForestClassifierfrom keras.layers import Dense, Dropoutfrom keras.models import Model, Sequentialfrom keras.wrappers.scikit_learn import KerasClassifierfrom sklearn.model_selection import GridSearchCV
I manually compiled the basic statistics of LeBron’s games from season 2003–2004 to season 2019–2020 (until NBA suspension in March). In total, there are 1,258 games. The import code is as below:
df = pd.read_csv("lebron_2003_2020_career_gamelog.csv",index_col=0)df.head()
From the figure above, you can see the basic stats and the game results (“Win” and “Winby”).
Then, I want to make sure the data type is ‘float32’ so that I can directly feed them to the neuron network model in keras. The data type transform code is as below:
df = df.astype('float32')
Next, I need to specify the feature space and the label in the dataset using the code as below:
y = df['Win']X = df.drop(columns=['Win','Winby'])
Column “Win” is the recorded game result, where 1 means win and 0 means lose. And column “Winby” is the game score difference with the component, where a positive number means win and a negative number means lose. Therefore, it is necessary to remove both of them from the feature space.
Next, the data is split into the training and the testing set, where the testing set will never be touched before the model evaluation step. The code is as below:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,stratify=y, random_state=42)
You may notice that I use a stratified split here which avoids the situation where the game result is biased to one category in the training data. “stratify=y” means the “stratified split” is done based on y which is our output label.
Till now, the steak is ready to cook, we need to start to preheat the oven.
As I mentioned, three models are going to be used, Logistic Regression, Random Forest Classifier, and Deep Learning Classifier. To make them all fit the same Scikit-Learn workflow, we need to define the deep learning model first in a Scikit-Learn style as below:
def my_DL(epochs=6,batchsize=512):    model = Sequential()    model.add(Dense(32,activation='relu'))    model.add(Dense(16,activation='relu'))    model.add(Dense(1,activation='sigmoid'))    model.compile(loss='binary_crossentropy',optimizer='rmsprop', metrics=['accuracy'])    return model
Specifically, this neural network has two hidden layers with 32 and 16 nodes. The loss function, optimizer, and the metric of the network are fixed as ‘binary_crossentropy’, ‘rmsprop’ and ‘accuracy’, respectively.
There are two adjustable parameters for this compiled model, epochs is the number of epochs and batchsize is the number of samples for each batch. Both parameters have their default values, whose formats are similar to those of a classifier in sklearn.
The parameters of a model that cannot be trained from the data but need to be assigned before the training process are called hyperparameters. These hyperparameters are always related to the complexity of the model, which needs to be selected properly to avoid underfitting or overfitting problems.
To select the best set of hyperparameters, we can do it in two ways. First, we can further split the training dataset into two parts, namely the training and the validation dataset. Then, we need to evaluate the trained model from the training dataset on the validation set. The best set of hyperparameters is the one with the best performance on the validation set.
However, when the sample size is small, only one split of data can be biased. So, cross-validation is another way of training hyperparameters, which is more popular. I, therefore, use cross-validation in this project.
I list the whole function of the hyperparameter tunning as below and will go through it in detail.
def train_hyper_tune(X,y):    # create the pre-processing component    my_scaler = StandardScaler()    my_imputer = SimpleImputer(strategy="median")        # define classifiers    ## Classifier 1: Logistic Regression    clf_LR = LogisticRegression(random_state=0,penalty='elasticnet',solver='saga')    ## Classifier 2: Random Forest Classifier    clf_RF = RandomForestClassifier(random_state=0)    ## Classifier 3: Deep Learning Binary Classifier    clf_DL = KerasClassifier(build_fn=my_DL)        # define pipeline for three classifiers    ## clf_LR    pipe1 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('lr_model',clf_LR)])    ## clf_RF    pipe2 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('rf_model',clf_RF)])    ## clf_DL    pipe3 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('dl_model',clf_DL)])        # create hyperparameter space of the three models    ## clf_LR    param_grid1 = {        'lr_model__C' : [1e-1,1,10],        'lr_model__l1_ratio' : [0,0.5,1]    }    ## clf_RF    param_grid2 = {        'rf_model__n_estimators' : [50,100],        'rf_model__max_features' : [0.8,"auto"],        'rf_model__max_depth' : [4,5]    }    ## clf_DL    param_grid3 = {        'dl_model__epochs' : [6,12,18,24],        'dl_model__batchsize' : [256,512]    }        # set GridSearch via 5-fold cross-validation    ## clf_LR    grid1 = GridSearchCV(pipe1, cv=5, param_grid=param_grid1)    ## clf_RF    grid2 = GridSearchCV(pipe2, cv=5, param_grid=param_grid2)    ## clf_DL    grid3 = GridSearchCV(pipe3, cv=5, param_grid=param_grid3)        # run the hyperparameter tunning process    grid1.fit(X,y)    grid2.fit(X,y)    grid3.fit(X,y)        # return results of the tunning process    return grid1,grid2,grid3,pipe1,pipe2,pipe3
As shown in the code, there are mainly six steps inside the function:
Step 1. Create the pre-processing functions.
    # create the pre-processing component    my_scaler = StandardScaler()    my_imputer = SimpleImputer(strategy="median")
I use the feature’s median to impute the missing values and the standard scaler to normalize the data. This step is the same for all three models.
Step 2. Define all three classifiers.
    # define classifiers    ## Classifier 1: Logistic Regression    clf_LR = LogisticRegression(random_state=0,penalty='elasticnet',solver='saga')    ## Classifier 2: Random Forest Classifier    clf_RF = RandomForestClassifier(random_state=0)    ## Classifier 3: Deep Learning Binary Classifier    clf_DL = KerasClassifier(build_fn=my_DL)
First, the logistic regression classifier is usually used as the “Hello world!” model in machine learning books. Here, it is used together with a penalty function to avoid overfitting. The model with this penalty term is called ‘Elastic Net’, which is a combination of l1 and l2 norm in the regularization.
For those who are interested in why we chose Elastic Net as the penalty term, please read my another post as below:
towardsdatascience.com
Second, the Random Forest Classifier is defined in a more freestyle without fixing any hyperparameters. Three of its hyperparameters are going to be tuned in the following steps, which I will go over in detail later.
Third, the deep learning classifier used here is based on the Scikit-Learn style model as aforementioned, my_DL. Thankfully Keras provides the wonderful Wrappers for the Scikit-Learn API. I directly call the function my_DL by passing it to the function KerasClassifier().
Step 3. Define a pipeline for each model that combines the pre-processing and modeling together.
    # define pipeline for three classifiers    ## clf_LR    pipe1 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('lr_model',clf_LR)])    ## clf_RF    pipe2 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('rf_model',clf_RF)])    ## clf_DL    pipe3 = Pipeline([('imputer', my_imputer), ('scaler', my_scaler), ('dl_model',clf_DL)])
For each of the three models, I combine the pre-processing and the classifier together into a pipeline with the Pipeline function in sklearn. For each step of processing, a name should be given. For example, I name my logistic regression model as “lr_model” and call it via clf_LR in the pipeline.
The aim of combining everything into a pipeline is to make sure the exact same processing of the training data is used to the testing data in the cross-validation. This is essential to avoid data leaking.
Step 4. Create the hyperparameter space for each of the models.
    # create hyperparameter space of the three models    ## clf_LR    param_grid1 = {        'lr_model__C' : [1e-1,1,10],        'lr_model__l1_ratio' : [0,0.5,1]    }    ## clf_RF    param_grid2 = {        'rf_model__n_estimators' : [50,100],        'rf_model__max_features' : [0.8,"auto"],        'rf_model__max_depth' : [4,5]    }    ## clf_DL    param_grid3 = {        'dl_model__epochs' : [6,12,18,24],        'dl_model__batchsize' : [256,512]    }
This part is more flexible because there is plenty number of parameters in these three models. It’s important to select the parameters that are closely related to the complexity of the model. For example, the maximum depth of the trees in the random forest model is a must-tune hyperparameter. For those who are interested, please refer to this post below.
towardsdatascience.com
To note, the name of the step in a pipeline needs to be specified in the hyperparameter space. For example, the number of epochs in the deep learning model is named as “dl_model__epochs”, where “dl_model” is the name of the deep learning model in my pipeline and “epochs” is the name of a parameter that can be passed to my deep learning model. They are connected in a string format by “__” in the hyperparameter space.
Step 5. Set the grid search function across the hyperparameter space via cross-validation.
    # set GridSearch via 5-fold cross-validation    ## clf_LR    grid1 = GridSearchCV(pipe1, cv=5, param_grid=param_grid1)    ## clf_RF    grid2 = GridSearchCV(pipe2, cv=5, param_grid=param_grid2)    ## clf_DL    grid3 = GridSearchCV(pipe3, cv=5, param_grid=param_grid3)
Comparing to the randomized search, the grid search is more computationally costly because it spans the entire hyperparameter space. In this project, I use the grid search because the hyperparameter space is relatively small.
For each grid search, I use 5-fold cross-validation to evaluate the average performance of the combinations of hyperparameters.
Step 6. Run the tunning process.
    # run the hyperparameter tunning process    grid1.fit(X,y)    grid2.fit(X,y)    grid3.fit(X,y)
This step is pretty straight forward, which execute the grid search on the three defined pipelines.
Lastly, we just need to run the function as below:
my_grid1,my_grid2,my_grid3,my_pipe1,my_pipe2,my_pipe3 = train_hyper_tune(X_train, y_train)
We can check the training performance by pulling out the best score in the grid search result.
It seems the random forest has the best performance on the training dataset. But all three models are pretty comparable to each other.
After the hyperparameters are selected in the previous step, I use them to re-train the models on the entire training data. The code is shown below:
def train_on_entire(X,y,pipe,grid_res):    # fit pipeline    pipe.set_params(**grid_res.best_params_).fit(X, y)    # return the newly trained pipeline    return pipe
Here, **grid_res.best_params_ is used to pass the best parameters from the grid search to the pipeline for the hyperparameter setting.
After refit with X and y, the returned pipeline, pipe, is a completely trained model on the entire training dataset.
We then need to evaluate this trained model on the test dataset.
train_on_entire(X_train,y_train,my_pipe1,my_grid1).score(X_test,y_test)train_on_entire(X_train,y_train,my_pipe2,my_grid2).score(X_test,y_test)train_on_entire(X_train,y_train,my_pipe3,my_grid3).score(X_test,y_test)
The performance of logistic regression, random forest classifier, and deep learning classifier in terms of accuracy is 0.869, 0.901, and 0.877, respectively.
We can draw several conclusions from the results. First, the random forest classifier seems to outperform the other two methods in this prediction. Second, the deep learning method doesn’t show advantages in dealing with the tabular dataset like this. Third, all three methods show that LeBron’s game stats have the prediction ability on the game result, which shows his dominating status in his team.
That’s it, the end-to-end machine learning project. I hope you’ve learned something from it.