Machine Learning in Baseball
By: Paul Torres
Tags: Python, Data Science, Linear Regression, Data, Machine Learning

This is an interesting problem to undertake because of the renaissance that has been taking place in baseball over the last two decades or so. Data has come to play a huge role in baseball and that means that patterns and statistics that were once considered fringe are now mainstream metrics. The home run is no longer king. The era of Moneyball has supplanted Longball.

My source was Sean Lahman’s baseball data set, which ranged from the late 1800s to 2019. Since I wanted to walk this path, I needed to make sure that I chose players from the current era. Only players that made their debut after the year 2000 were included. Nineteen years is long enough for a good portion of the players in this dataset to have played a full career.

The first step was to clean the data. There is a lot of information that couldn’t be of good use and were dropped — including birthplace, death place, and birth year.

The data came in several different files. The main dataset contained the biographical information while the salary, pitching, batting, post-season batting, post-season pitching, fielding, and awards were all appropriately named for their contents. The goal was to average the first three years of statistics and isolate the fourth season salary. This fourth season would presumably be the product of arbitration.

The process was done in a series of steps listed below:

Data Cleaning EDA (Exploratory Data Analysis) Modeling

1. Data Cleaning

First, I want to talk about the helper functions that I will be using through import. Importing my own functions has been a pet project of mine. These are specialized but I am working more and more on creating generalized functions that I can use over and over again.

The first function was to gather the mean for the first three years of statistics. The second function is the isolation of the fourth year salary. I use them when processing the data. You can find them here.

I found the fourth year salary by player. This was going to be the target variable I modeled for later on.

When using this groupby command, if the player does not play a fourth season then the salary will be entered in as a zero. We can easily filter out players that would not have made it to arbitration that way.

The next step was to create features for both pitchers and batters based on their statistics. This included things like slugging percentage, walks + hits over how many innings pitched (whip), etc. These are much more comprehensive statistics that are popular in baseball to explain a player's importance. Since they are based on other statistics, there may be correlation issues in the future that we may have to fix.

Then I aggregated the players by statistics. I had both pitchers and batters so I did both. Here is an example of the batters.

The code above looks at each of the continuous variables and creates a mean of the first three years of a rookie’s career. This cuts down the observations to a third. This was repeated with all of the continuous variables for pitchers as well.

Next, we start on the categorical variables. This includes awards, height bins, position, and handedness. We use dummy variables and one-hot encoding in order to accomplish it. This process just means that we assign an integer value to the positive case in each variable. A ‘1’ if the person is left-handed vs right-handed or for the position — this also means having an explosion of variables.

After this action, the data set has an entry for each player that debuted after 2000, played at least four seasons. It also has the average number for each of their first three year statistics as well as an indicator that they won an award and other vitals like batting and throwing hand.

2. EDA

During the data analysis portion of the project, it is now a prerogative to look through the data for trends, patterns, or a deeper understanding of the target variable. The first step can be to take a look at the distribution for the salaries of players by position. Since this has nine possible categories, we will use a box plot.

Salary Distribution by Position

According to the plot above, the salaries are very similar. The outliers tell a tale on their own — outfielders and pitchers are much more likely to have prolific arbitration outcomes. We can do more analysis and see if hits per game play any part in this outcome.

Hits Per Game Distribution byPosition

From this graph, we can see that catchers rank towards the bottom in hits per game on average. However, the rest of the positions are much less clear, so we need to continue looking. Let’s take a look at the side of the plate batters hit from.

Count of Left vs Right-Handed Batters

From this graph, we can see that left-handed batters are available at a premium. This graph does not tell us much more than that so let’s delve deeper and take a look at the mean salary for left vs right-handed players.

Mean Salary by Handedness

From here we can see that the salary average is pretty close to the same for the two categories. Now, we can conduct some statistical tests. Let’s first do an overall correlation test. This graph will tell us which of the features are telling us the same information and will throw off the linear regression models.

Correlation Heatmap

As we thought, there are major sources of correlation in the features. This is to be expected because some of the features are based on the original data set columns. We are going to use a Ridge or Lasso model for feature selection so let’s move on.

Next, we begin the statistical tests. Firstly, we have to make a change in our data set. We do this because of the distribution of the salary. We need to get the distribution as close to a normal distribution as we can. For this technique, we log transform the salary feature. After the transformation, the distribution looks like this:

Log Transformed Salary Distribution by Handedness

3. Modeling

Since we are going to be using the model to try and find the features that have the best explanation of the target variable, we will be using R² as our metric. This means that the features explain a certain percentage of the variance of the target variable. The modeling process is mostly a standard procedure of preparation for this project. Our first step is to separate the pitchers from the batters. The intrinsically different positions and statistics would only throw each other off during a model.

The first step is the test, train, split for both pitchers and batters. After that, I ran a scaler on the different portions. It is important that this be done after the split so that the train and test sets do not have any interactions or affect each other in any way.

Batters

In the end, we ran four models for both pitchers and batters. The final scores for the batters look like this.

Final Batter Model R² Scores

The best performing model was the Lasso CV (cross-validation) model that helped with feature selection. The coefficients that survived the punishing test of importance are shown in the following graph.

Feature Coefficients for Batters

Next were the pitchers. The Lasso CV model did not work as well for the pitchers. Instead, it required a standard linear regression model with a little bit of tuning. The scores are shown here.

Final Pitcher Model R² Scores

The feature importance is described in the graph below.

Feature Coefficients for Pitchers

Looking Forward

The basis of this project was to identify the most important features of a salary arbitration for a rookie’s fourth year. This project was done very early in my data science career and as such, I would change a few of the routes I took.

Firstly, I would increase the data set to include more of the baseball statistics seen in-game. These include but are not limited to pitch velocity, strike zone location, and arm strength. Another perspective to look at the data is to look at trends over time vs averages.

This would mean looking at how a player improved or declined over their first three seasons rather than taking all of the seasons at equal weight.