Evaluating Recruiting and Player Development in College Football
By: Ashwin Sakhare
Tags: Data Science, College Football, Football Coaching

PC: CollegeFootballNews.com

Background

I’m an avid college football fan with a keen interest in data science. For my first dive into data science, I will use some basic statistics and data manipulation to evaluate coaching in college football.

Defining the Problem

Football is the crown jewel and breadwinner in any collegiate athletic department. A successful football program is not only a source of revenue for the university, it is also a powerful marketing tool, as studies have shown winning to be associated with increased donations, academic reputation, and lower acceptance rates. Therefore, it’s no surprise that coaches wield significant power and lofty salaries.

Coaches also carry the burden of high expectations, poor job security, and an average tenure of only 3.8 years. While the decision to hire or fire a coach is a multi-factorial process, it is primarily driven by their record. Consequently, a coach’s value, particularly with donors and fans, is inextricably tied to their performance on the field.

However, wins and losses are driven by several factors, many of which are outside a coach’s control, including conference affiliation, prestige, location, historical success, and financial investment from the university. As such, a coach’s record is not the most objective way to assess their value. In this project we will set out to define an approach for evaluating coaches while minimizing the impact of these confounding factors. Let’s get started!

Performance Metrics

There are two principal factors coaches control that drive winning in college football: (1) recruiting and (2) player development. We’ll use this as a starting point for evaluating a coach’s performance.

Datasets Used

To assess recruiting ability and player development, we will combine data from four websites: Pro Football Reference, 247Sports, Sports Reference, and The Football Database.

247Sports provides composite recruiting rankings for high school players and schools dating back to 2000. We’ll extract each school’s name, recruiting ranking, and recruiting year, and each player’s national rank, school, position, and enrollment year.

Sports Reference and The Football Database provide game logs on all college players. Both websites will be used to increase the likelihood of locating a player’s game logs. We’ll extract the yearly game logs for each player to determine whether they transferred to another school during their collegiate career. Any players that transferred will be excluded from our analysis.

Pro Football Reference provides draft information on all college players. We’ll extract the draft position of each player.

Pre-Processing

Creating datasets

We’ll create two datasets for our analysis: (1) player and (2) school.

The player dataset will contain the following measures:

(1) Player name

(2) School

(3) National rank

(4) Enrollment year

(5) Playing position

(6) Draft position (if drafted)

(7) Transfer status

(8) Star rating

The school dataset will contain the following measures:

(1) School name

(2) National rank

(3) Recruiting year

(4) Coach

We’ll use player data obtained from 247Sports to look up game logs and draft position from Sports Reference / The Football Database and Pro Football Reference, respectively. One concern with utilizing several websites to extract a player’s data is ensuring it is associated with the correct player. For example, there are 11 players named Brandon Jones in the 247Sports database. To match each unique Brandon Jones with his correct game logs and draft position, we will first check that the school name matches between websites. To protect against schools having multiple players with the same name, we will also check that the player’s game logs and draft year are within 5 years of their college enrollment year listed on 247Sports.

Another concern with matching player data is that there may be differences in how a player’s name is spelled across websites. This is particularly likely for players with special characters in their name, such as Ronald Jones II, Donovan Peoples-Jones, A.J. Epenesa and D’Andre Swift. To correct for this, we will remove all special characters from a player’s name when locating player data.

Note: Information regarding coaches and their associated schools was found in CSV format on a college football website and was not extracted from any of the websites mentioned above.

Checking datasets

First, let’s get sense of the number of players we extracted from 247Sports and what percentage of those players were drafted.

We can see that 247Sports started ranking players in 2000 but didn’t start ranking a significant number until 2002. The total number of players in our dataset from 2000–2020 was 56,447, with 2,624 being drafted. The NFL drafts approximately 256 players each year, suggesting that 5,120 players should be drafted over a 20-year period.

There are a few reasons for this discrepancy. First, the number players drafted from 2003–2006 is lower than expected because those players were high school seniors between 1998 and 2003, either prior to the inception of 247Sports or in the earlier years when fewer players were ranked. The number of players drafted between 2007 and 2020 ranges from 150 to 200, which is closer to the expected number of 256. The remaining drafted players that are unaccounted for may be due to 247Sports not identifying and ranking these players as high school seniors.

Data Analysis

Recruiting Ability

A coach’s recruiting ability will be assessed using their respective school’s recruiting rankings. However, school rankings can be impacted by conference affiliation, location, and historical success. To account for these potential confounding factors, we will create a standardized recruiting score.

First, we’ll apply a box-cox transformation to each school’s distribution of recruiting rankings. Briefly, a box-cox transformation is a statistical technique that can be used to transform a skewed distribution into a normal shape. While standardize scores can be created from skewed distributions, a more meaningful interpretation can be made if the underlying data is normally distributed.

Next, for each year of a coach’s tenure, we will subtract the school’s recruiting ranking for that year from the school’s mean recruiting ranking and then divide by the standard deviation of the school’s recruiting rankings.

We will then take the median to create a single standardized recruiting score for each coach. We will use median instead of mean to preserve directionality. A positive recruiting score will indicate the coach outperformed the school’s recruiting expectations, while a negative score will indicate the coach under-performed expectations.

Player development

We will use draft position and the likelihood of being drafted as a proxy measure for player development. First, we will need to account for two potential confounding factors: (1) star rating and (2) position group.

Briefly, 247Sports assigns each player a star rating between 2 and 5 based on their projected potential at the collegiate and NFL level. According to 247Sports, 5-star players are projected 1st round draft picks, while 2-star players have limited NFL potential. While the ratings system isn’t perfect, schools with higher ranked players tend to win at a higher rate than schools with lower ranked players. Therefore, we would expect to see differences in draft position based on star rating. This is problematic as a coach’s player development score will be directly influenced by their recruiting ability. Position group is the other potential confounding factor, as NFL teams may value certain position groups over others.

Let’s see if there are differences in draft position based on star rating and position group. For star rating, we will split 3-star players into equal groups of 3-, 3, and 3+, as this group represents approximately 50% of all players and splitting it will provide additional resolution to identify group differences. We’ll use non-parametric tests, Kruskal-Wallis and Dunn, for this analysis as the underlying data for star rating and position group is not normally distributed.

We can see that star rating is significantly associated with draft position (χ²(5) =186, p < 0.001). Pairwise comparisons show 5-star and 4-star players are drafted significantly higher than all lower rated players. 3+ star players are drafted significantly higher than 3- and 2 stars, but not 3 stars.

Position group is also significantly associated with draft position (χ²(10) =51, p < 0.001). However, this difference is primarily driven by special teams, who are drafted significantly lower than all position groups except for fullback and tight end. Therefore, we will not account for position group in our analysis of player development.

Draft Position

First, we will apply a box-cox transformation to the draft positions associated with each star rating. Next, we’ll create a standardized draft position score for each player by subtracting the player’s draft position from the mean draft position for all players with the same star rating and dividing by the standard deviation of the draft positions for those players.

For each coach, we will take the median value of the standardized draft position scores associated with the players they coached. Once again, we will use median instead of the mean to preserve directionality. A positive player development score will indicate a coach’s players were drafted higher (better) than expected, while a negative score will indicate a coach’s players were drafted lower (worse) than expected.

Likelihood of being drafted

First, let’s check if the likelihood of being drafted is associated with star rating.

We can see that the percentage of players drafted is positively associated with a player’s star rating. To account for star rating as a confounding factor, let’s first calculate the relative difference in drafted probability for each star rating by subtracting the percentage of players drafted under a coach by the percentage of all players drafted and dividing by the percentage of all players drafted.

We will then take the median to create a single draft probability score for each coach. Next, we will apply a box-cox transformation to normalize the distribution of draft probabilities for all coaches. Finally, we will create a standardized draft probability score for each coach by subtracting the coach’s draft probability by the mean draft probability for all coaches and dividing by the standard deviation of the draft probability for all coaches.

A positive player development score will indicate a player has a higher likelihood of being drafted with that coach compared to other coaches, while a negative score will indicate a player has a lower likelihood of being drafted with that coach compared to other coaches.

Our Findings

Let’s first look at player development (via draft position) and recruiting ability for active coaches from the power-5 conferences. All outcome measure values are reported as number of standard deviations from the mean. As a reminder, a positive recruiting score indicates the coach outperformed the school’s mean recruiting rankings. A positive draft position score indicates a coach’s players were drafted higher (better) than their expected draft position based on their star rating. Coaches that did not have at least 5 players drafted were excluded from these figures as it is difficult to meaningfully interpret player development with small sample sizes.

We can see two notable clusters of coaches that are distinct from the rest. For player development, Ryan Day, Ed Orgeron, Chris Petersen, and Nick Saban lead the pack, having coached players that were drafted at least .4 standard deviations higher (better) than their expected their draft position. For recruiting, Kirby Smart, Mario Cristobal, and Jimbo Fisher rate the highest, having recruited at least 1.4 standard deviations better than their respective school’s historical recruiting ranking.

Next, let’s take a closer look at player development by looking at both draft position and draft probability. As a reminder, a positive draft probability score indicates a player has a higher likelihood of being drafted with that coach compared to other coaches.

In the upper right quadrant we can see the coaches who perform better than average in not only getting players drafted but getting them drafted higher (better) than expected. We can see that Ryan Day and Chris Petersen stand out from the rest in player development. In the lower left quadrant are coaches who perform poorly in both, such that a lower percentage of their players are drafted and the ones that do get drafted are drafted lower (worse) than expected.

In the bottom right quadrant are coaches who perform better than average in getting players drafted but worse than average in getting them drafted at or above their expected draft position. Surprisingly, James Franklin, Jim Harbaugh, and Dabo Swinney, coaches with an established track record of winning at an elite level fall into this category. Conversely, Nick Saban falls into the upper left quadrant, such that he rates well with respect to a player’s draft position but slightly below average in getting a player drafted.

When looking at both figures, we notice that several data points at the extremes are associated with relatively new coaches. For example, Ryan Day rates highly in player development but is only a first-year coach at Ohio State. It is likely that players who were drafted under Ryan Day were also developed by his predecessor, Urban Meyer, for 2–3 years prior to his arrival. This can lead to a positively biased player development score if the drafted player’s development is primarily based on the teachings of the preceding coach. Conversely, Jimbo Fisher’s poor rating in player development may be a result of inheriting poorly developed players in his short 2-year tenure at Texas A&M.

Recruiting is not as influenced by coaching tenure as player development, but it is possible that new coaches bring a renewed excitement to the school that can temporarily and artificially boost recruiting. For example, Mario Cristobal is a first-year coach whose high recruiting score may be due to a combination of exceptional recruiting ability and renewed excitement within the program.

To mitigate the influence of new coaches on our data, we will remove coaches with less than 5 years of experience, such that all remaining coaches have had at least 1 full class they recruited, developed, and graduated.

We can see that among all coaches with at least 5 years at their respective schools, Chris Peterson rates the highest in overall player development. When player development and recruiting ability are taken together, Nick Saban rates the highest among coaches. Finally, let’s look at how this group of active coaches compares to all coaches from 2000–2020 with a coaching tenure of at least 5 years. Note: Coaches may be listed multiple times if they coached several teams during this period.

We can see that historically, Jim Tressel, Ron Zook, and Houston Nutt rate the highest when using draft position as a measure of player development. Ron Zook rates highest overall when taking player development and recruiting ability together. Chris Petersen rates the highest among all coaches when both draft position and draft probability are taken together.

Conclusions

Our analysis on coaching performance in college football provides some interesting insights and many of our findings correlate with media and fan perceptions. For example, Nick Saban is widely considered by many to be the best coach in college football, which our analysis agrees with as he rates highest overall among active coaches when taking player development and recruiting ability into account. Chris Petersen is viewed as a coach who can win at an elite level with lower ranked players, which is reflected by his success at Boise State and Washington. Our findings agree with this perception as well as he rates highest in draft likelihood and draft position, our two proxy measures of player development.

Overall, we have developed a new approach for evaluating a coach’s performance that doesn’t rely on simply looking at wins and losses, which is often impacted by factors outside a coach’s control such as conference affiliation, prestige, location, historical success, and financial investment from the university. However, there are several significant limitations to keep in mind. First, the accuracy of our player development and recruiting scores are highly sensitive to coaching tenure. Newer coaches may recruit better or worse than their actual ability depending on the school’s record prior to their arrival or the amount of buzz their arrival generated. Newer coaches may also have player development scores that are influenced by how well their players were developed by the preceding coach. Second, coaches at schools that historically recruit at an elite level will be limited in how much they can outperform the school’s average recruiting ranking. Finally, the turnstile nature of the coaching profession means that most coaches have limited data, making it difficult to create an accurate interpretation of their recruiting and player development abilities.

However, while these findings should be interpreted with caution, they do provide basic insight into how a coach compares to his peers in recruiting ability and player development. I’m hoping this article will provoke further discussion and be used as a springboard for exploring new ways to evaluate a coach’s performance. My code is available on GitHub and I happily welcome any feedback or suggestions for improvement on the statistical techniques used in this article.