Using Python to Scrape NFL Stats and Compare Quarterback Efficiencies
By: Naveen Venkatesan
Tags: Python, Data Science, Statistics, Data Visualization, NFL

I have always been apprehensive about trying to scrape my own data, and the fact that websites like Kaggle aggregate such high quality datasets has made learning this skill less of a need. However, the abundance of educational articles on data science on this platform have helped me make progress towards collecting my own datasets. A lot of the inspiration and methods for scraping data came from here.

In this article, I will pull quarterback stats from the 2019–20 NFL season from Pro Football Reference, and use them to create radar charts to assess QB efficiency.

Load Packages

To open the webpage and scrape the data, we will use two modules, urllib.request to open the URL, and BeautifulSoup to parse through the HTML.

# Import scraping modules

from urllib.request import urlopen

from bs4 import BeautifulSoup

In addition to these packages, we will need some packages to manipulate data, numpy and pandas , and plot our data, matplotlib .

# Import data manipulation modules

import pandas as pd

import numpy as np # Import data visualization modules

import matplotlib as mpl

import matplotlib.pyplot as plt

Scrape Data

The data we are going to import is the NFL passing data from the 2019 season, which can be found here. We open the site and pass it to BeautifulSoup with the following:

# URL of page

url = 'https://www.pro-football-reference.com/years/2019/passing.htm' # Open URL and pass to BeautifulSoup

html = urlopen(url)

stats_page = BeautifulSoup(html)

Note that we can easily adapt all of this analysis to previous years by changing the 2019 in URL to the year of your choosing.

The two BeautifulSoup functions we will use to scrape the page are findAll() and getText() , which return values based on the HTML of the page we are scraping. I present simplified use cases below — for all possibilities you should refer to the documentation.

findAll(name) Parameters

name -- HTML tags to use to parse webpage Returns array of all matches to name tag

getText() Returns text from HTML

For these to be effective, we have to determine patterns in the page source. In our, case the data is nicely formatted in a table, so we can find all the table rows ( tr ) and columns ( td ) and extract the text directly from the cells.

First, we need to collect the column headers so we can use them later in our DataFrame. To do this, we find the first tr element in page and collect the text from all the table headers ( th ):

# Collect table headers column_headers = stats_page.findAll('tr')[0]

column_headers = [i.getText() for i in column_headers.findAll('th')]

We index the first element because this is the row that contains the column headers. We can view our result:

print(column_headers) >>> ['Rk', 'Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD']

To collect the actual data, we will first collect all the table rows ( tr ) and store them in an array. Then, we iterate through each row and collect the text in each column ( td ) with getText() :

# Collect table rows

rows = stats_page.findAll('tr')[1:] # Get stats from each row

qb_stats = []

for i in range(len(rows)):

qb_stats.append([col.getText() for col in rows[i].findAll('td')])

We skip the first row because these are the column headers that we just collected previously. When we examine the first row of our qb_stats list:

print(qb_stats[0]) >>> ['Jared Goff', 'LAR', '25', 'QB', '16', '16', '9-7-0', '394', '626', '62.9', '4638', '22', '3.5', '16', '2.6', '220', '66', '7.4', '7.0', '11.8', '289.9', '86.5', '', '22', '170', '6.90', '6.46', '3.4', '1', '2']

Now we can combine our headers and stats into a pandas DataFrame. If you haven’t used pandas before, it’s essentially an Excel spreadsheet that you can manipulate programmatically, so it’s great for handling larger datasets. We notice above that the Rk column does not have corresponding data in our qb_stats list — this is because those values were all table headers, so our findAll() function did not pull their data. Regardless — this data is frivolous for our analysis so we will omit it when creating the DataFrame.

# Create DataFrame from our scraped data

data = pd.DataFrame(qb_stats, columns=column_headers[1:])

columns — the column titles for our DataFrame (we are omitting Rk )

Now, let’s take a look at the first five rows of our DataFrame:

# Examine first five rows of data

data.head()

Success!

Manipulating and Cleaning Data

Now, we can manipulate the data in this DataFrame to get what we need to make our radar charts. First, let’s look at all the columns in the set:

# View columns in data

data.columns >>> Index(['Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD'], dtype='object')

We see one issue immediately — there are two columns titled Yds ; one is for passing yards, and the other is yards lost due to being sacked. We can easily remedy this by renaming the latter:

# Rename sack yards column to `Yds_Sack`

new_columns = data.columns.values

new_columns[-6] = 'Yds_Sack'

data.columns = new_columns

Now, let’s view our columns again:

# View columns in data

data.columns >>> Index(['Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds_Sack', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD'], dtype='object')

We have successfully renamed the column to Yds_Sack and no longer have a conflict in our column names! Next, let’s identify which statistical categories we are interested in for our visualization. We will choose:

(1) Completion percentage — Cmp%

(2) Passing yards — Yds

(3) Passing touchdowns — TD

(4) Interceptions — Int

(5) Yards per attempt — Y/A

(6) Passer rating — Rate

# Select stat categories

categories = ['Cmp%', 'Yds', 'TD', 'Int', 'Y/A', 'Rate']

Now let’s create a new DataFrame as a subset of our original data, only with the data from our chosen categories. Additionally, we will add the player name and team.

# Create data subset for radar chart

data_radar = data[['Player', 'Tm'] + categories]

data_radar.head()

Now that we have our subset of data, let’s check the data type, because we scraped the values as text from the URL:

# Check data types

data_radar.dtypes >>> Player object

Tm object

Cmp% object

Yds object

TD object

Int object

Y/A object

Rate object

dtype: object

All of our numerical data have been stored as objects, so we can’t manipulate them. So before we proceed further, we must convert all this data to numerical values. To do so, we use a function called pandas.to_numeric .

# Convert data to numerical values

for i in categories:

data_radar[i] = pd.to_numeric(data[i])

Now, let’s check our data types once again:

# Check data types

data_radar.dtypes >>> Player object

Tm object

Cmp% float64

Yds float64

TD float64

Int float64

Y/A float64

Rate float64

dtype: object

We have one last piece of data cleaning to do. On the original website, they put ornamental characters next to players that had end-of-season achievements, such as a Pro Bowl (*) or All-Pro (+) selection. We will remove these using str.replace() :

# Remove ornamental characters for achievements

data_radar['Player'] = data_radar['Player'].str.replace('*', '')

data_radar['Player'] = data_radar['Player'].str.replace('+', '')

Let’s filter our data down to only the quarterbacks who threw for more than 1500 yards:

# Filter by passing yards

data_radar_filtered = data_radar[data_radar['Yds'] > 1500]

Now, for our radar chart, we want to calculate the each quarterback’s statistical rank by percentile, which is easily done in pandas with DataFrame.rank(pct=True) . The rank() function can take other arguments to rank based on other parameters for which you can look up the online documentation.

Additionally, we want to flip our interceptions rank, as we don’t want the QB with the most interceptions to have the highest percentile!

# Create columns with percentile rank

for i in categories:

data_radar_filtered[i + '_Rank'] =

data_radar_filtered[i].rank(pct=True) # We need to flip the rank for interceptions

data_radar_filtered['Int_Rank'] =

1 - data_radar_filtered['Int_Rank']

Now we examine our new data:

# Examine data

data_radar_filtered.head()

Great! We can now make our visualizations!

Generating Radar Charts

We will start by editing some general plot parameters. We are going to be generating a polar plot, so the x-ticks correspond to the angle around the circle — we are increasing the padding between the axis and tick labels:

# General plot parameters

mpl.rcParams['font.family'] = 'Avenir'

mpl.rcParams['font.size'] = 16

mpl.rcParams['axes.linewidth'] = 0

mpl.rcParams['xtick.major.pad'] = 15

For colors in our chart, we will use the HEX codes of the NFL team colors, which were collected from this link.

team_colors = {'ARI':'#97233f', 'ATL':'#a71930', 'BAL':'#241773', 'BUF':'#00338d', 'CAR':'#0085ca', 'CHI':'#0b162a', 'CIN':'#fb4f14', 'CLE':'#311d00', 'DAL':'#041e42', 'DEN':'#002244', 'DET':'#0076b6', 'GNB':'#203731', 'HOU':'#03202f', 'IND':'#002c5f', 'JAX':'#006778', 'KAN':'#e31837', 'LAC':'#002a5e', 'LAR':'#003594', 'MIA':'#008e97', 'MIN':'#4f2683', 'NWE':'#002244', 'NOR':'#d3bc8d', 'NYG':'#0b2265', 'NYJ':'#125740', 'OAK':'#000000', 'PHI':'#004c54', 'PIT':'#ffb612', 'SFO':'#aa0000', 'SEA':'#002244', 'TAM':'#d50a0a', 'TEN':'#0c2340', 'WAS':'#773141'}

The angles at which we plot all our points will be dependent on the number of statistical categories we have. In our example, we have 6 categories, so we will plot our points every 2π/6 radians, or 60 degrees. We can calculate this as follows, using numpy.linspace() :

# Calculate angles for radar chart

offset = np.pi/6

angles = np.linspace(0, 2*np.pi, len(categories) + 1) + offset

We are adding the offset term to adjust where the labels appear on the circle (instead of the first label at 0 radians, it now appears at π/6).

When we plot the data, we need to actually duplicate the first category, so that the shape closes itself. If you notice when we calculated the angles, we have 7 data points, and the first and last angles correspond to the same point on the circle. So when we plot the data, we use the following (we will turn this into a function, so player_data represents a row of player-specific data that we will pass):

# Plot data and fill with team color

ax.plot(angles, np.append(player_data[-(len(angles)-1):], player_data[-(len(angles)-1)]), color=color, linewidth=2) ax.fill(angles, np.append(player_data[-(len(angles)-1):], player_data[-(len(angles)-1)]), color=color, alpha=0.2)

The above code works because our categories are the last 6 columns of our DataFrame, so len(angles) — 1 corresponds to the first of these categories (since angles has an extra element). We then append the value of the first category onto the end of this array so that we can close the shape.

Now we can set the labels for the category names (since we have one less category than angles, we omit the last element):

# Set category labels

ax.set_xticks(angles[:-1])

ax.set_xticklabels(categories)

Finally, we will add the player name on top of the radar chart — we place the text object at (π/2, 1.7), in absolute plot coordinates so that it appears above the axis:

# Add player name

ax.text(np.pi/2, 1.7, player_data[0], ha='center', va='center', size=18, color=color)

We can put this all together to make a helper function to generate our radar charts as follows:

Let’s also make another helper function to return a numpy array of QB data when given a team input:

# Function to get QB data

def get_qb_data(data, team):

return np.asarray(data[data['Tm'] == team])[0]

Visualizing Data

NFC West

Since I’m a huge 49ers fan, let’s start by looking at the radar charts for QBs in the NFC West:

# Create figure

fig = plt.figure(figsize=(8, 8), facecolor='white') # Add subplots

ax1 = fig.add_subplot(221, projection='polar', facecolor='#ededed')

ax2 = fig.add_subplot(222, projection='polar', facecolor='#ededed')

ax3 = fig.add_subplot(223, projection='polar', facecolor='#ededed')

ax4 = fig.add_subplot(224, projection='polar', facecolor='#ededed') # Adjust space between subplots

plt.subplots_adjust(hspace=0.8, wspace=0.5) # Get QB data

sf_data = get_qb_data(data_radar_filtered, 'SFO')

sea_data = get_qb_data(data_radar_filtered, 'SEA')

ari_data = get_qb_data(data_radar_filtered, 'ARI')

lar_data = get_qb_data(data_radar_filtered, 'LAR') # Plot QB data

ax1 = create_radar_chart(ax1, angles, lar_data, team_colors['LAR'])

ax2 = create_radar_chart(ax2, angles, ari_data, team_colors['ARI'])

ax3 = create_radar_chart(ax3, angles, sea_data, team_colors['SEA'])

ax4 = create_radar_chart(ax4, angles, sf_data, team_colors['SFO']) plt.show()

The data seems to support what we see from the eye test :

Russell Wilson is on an elite level, reaching at least 75th percentiles in all statistical categories. This analysis also doesn’t take into account what he brings with his scrambling ability. Jimmy Garoppolo is generally a solid QB, but throws a few boneheaded interceptions from time-to-time. Jared Goff gets a lot of yards, but it does not seem to translate to a lot of production (i.e. “empty yards”). Kyler Murray is hard to judge because he was a rookie, but his efficiency looks to be the lowest of the four. Like Wilson, his mobility is not accounted for in this plot.

MVP Race

The MVP race (at QB) came down Russell Wilson and Lamar Jackson (who eventually won):

# MVP Race

# Create figure

fig = plt.figure(figsize=(8, 4), facecolor='white') # Add subplots

ax1 = fig.add_subplot(121, projection='polar', facecolor='#ededed')

ax2 = fig.add_subplot(122, projection='polar', facecolor='#ededed') # Adjust space between subplots

plt.subplots_adjust(hspace=0.8, wspace=0.5) # Get QB data

bal_data = get_qb_data(data_radar_filtered, 'BAL')

sea_data = get_qb_data(data_radar_filtered, 'SEA') # Plot QB data

ax1 = create_radar_chart(ax1, angles, sea_data, team_colors['SEA'])

ax2 = create_radar_chart(ax2, angles, bal_data, team_colors['BAL']) plt.show()

From a pure passer standpoint, Russell Wilson has more balanced stats, but Lamar Jackson clearly had the more “bang-for-buck” passing yards, as shown by his number of passing TDs and quarterback rating. Additionally, Lamar Jackson set the QB rushing record for a season, which is not captured in this chart.

Teams who drafted 1st round QBs

Four quarterbacks were taken in the 1st round of the 2020 NFL Draft: Joe Burrow (CIN), Tua Tagovailoa (MIA), Justin Herbert (LAC), and (surprisingly) Jordan Love (GB). Let’s visualize the stats of the quarterbacks that these draft picks would replace.

# 1st Round Draft Picks

# Create figure

fig = plt.figure(figsize=(8, 8), facecolor='white') # Add subplots

ax1 = fig.add_subplot(221, projection='polar', facecolor='#ededed')

ax2 = fig.add_subplot(222, projection='polar', facecolor='#ededed')

ax3 = fig.add_subplot(223, projection='polar', facecolor='#ededed')

ax4 = fig.add_subplot(224, projection='polar', facecolor='#ededed') # Adjust space between subplots

plt.subplots_adjust(hspace=0.8, wspace=0.5) # Get QB data

cin_data = get_qb_data(data_radar_filtered, 'CIN')

mia_data = get_qb_data(data_radar_filtered, 'MIA')

lac_data = get_qb_data(data_radar_filtered, 'LAC')

gnb_data = get_qb_data(data_radar_filtered, 'GNB') # Plot QB data

ax1 = create_radar_chart(ax1, angles, cin_data, team_colors['CIN'])

ax2 = create_radar_chart(ax2, angles, mia_data, team_colors['MIA'])

ax3 = create_radar_chart(ax3, angles, lac_data, team_colors['LAC'])

ax4 = create_radar_chart(ax4, angles, gnb_data, team_colors['GNB']) plt.show()

Andy Dalton had a terrible season by all metrics — it clearly appears that it was time to move on Ryan Fitzpatrick was below average in all six statistical metrics and had a fairly mediocre season. Considering Josh Rosen was benched for Fitzpatrick, Miami needed to invest in Tua for the future. Philip Rivers actually performed average to above average in most categories, but clearly had some bad decision-making, judging by the number of interceptions he threw. Aaron Rodgers had what appears to be an above average season, but still below average by the lofty expectations we have for him. The Jordan Love pick was a shock for everyone.

Other Divisions

NFC North

NFC East

NFC South

AFC West

AFC North

AFC East

AFC South

Final Remarks

As you can see, the possibilities are endless — and this framework can be easily adapted for other stats such as rushing and receiving metrics. The Jupyter notebook used for this article can be found at this Github repository.

Thank you for reading! I appreciate any feedback, and you can find me on Twitter and connect with me on LinkedIn for more updates and articles.