Using Python to Scrape NFL Stats and Compare Quarterback Efficiencies | by Naveen Venkatesan | Towards Data Science

I have always been apprehensive about trying to scrape my own data, and the fact that websites like Kaggle aggregate such high quality datasets has made learning this skill less of a need. However, the abundance of educational articles on data science on this platform have helped me make progress towards collecting my own datasets. A lot of the inspiration and methods for scraping data came from here.
In this article, I will pull quarterback stats from the 2019–20 NFL season from Pro Football Reference, and use them to create radar charts to assess QB efficiency.
To open the webpage and scrape the data, we will use two modules, urllib.request to open the URL, and BeautifulSoup to parse through the HTML.
# Import scraping modulesfrom urllib.request import urlopenfrom bs4 import BeautifulSoup
In addition to these packages, we will need some packages to manipulate data, numpy and pandas, and plot our data, matplotlib.
# Import data manipulation modulesimport pandas as pdimport numpy as np# Import data visualization modulesimport matplotlib as mplimport matplotlib.pyplot as plt
The data we are going to import is the NFL passing data from the 2019 season, which can be found here. We open the site and pass it to BeautifulSoup with the following:
# URL of pageurl = 'https://www.pro-football-reference.com/years/2019/passing.htm'# Open URL and pass to BeautifulSouphtml = urlopen(url)stats_page = BeautifulSoup(html)
Note that we can easily adapt all of this analysis to previous years by changing the 2019 in URL to the year of your choosing.
The two BeautifulSoup functions we will use to scrape the page are findAll() and getText(), which return values based on the HTML of the page we are scraping. I present simplified use cases below — for all possibilities you should refer to the documentation.
findAll(name)Parametersname -- HTML tags to use to parse webpageReturns array of all matches to name taggetText()Returns text from HTML
For these to be effective, we have to determine patterns in the page source. In our, case the data is nicely formatted in a table, so we can find all the table rows (tr) and columns (td) and extract the text directly from the cells.
First, we need to collect the column headers so we can use them later in our DataFrame. To do this, we find the first tr element in page and collect the text from all the table headers (th):
# Collect table headerscolumn_headers = stats_page.findAll('tr')[0]column_headers = [i.getText() for i in column_headers.findAll('th')]
We index the first element because this is the row that contains the column headers. We can view our result:
print(column_headers)>>> ['Rk', 'Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD']
To collect the actual data, we will first collect all the table rows (tr) and store them in an array. Then, we iterate through each row and collect the text in each column (td) with getText() :
# Collect table rowsrows = stats_page.findAll('tr')[1:]# Get stats from each rowqb_stats = []for i in range(len(rows)):  qb_stats.append([col.getText() for col in rows[i].findAll('td')])
We skip the first row because these are the column headers that we just collected previously. When we examine the first row of our qb_stats list:
print(qb_stats[0])>>> ['Jared Goff', 'LAR', '25', 'QB', '16', '16', '9-7-0', '394', '626', '62.9', '4638', '22', '3.5', '16', '2.6', '220', '66', '7.4', '7.0', '11.8', '289.9', '86.5', '', '22', '170', '6.90', '6.46', '3.4', '1', '2']
Now we can combine our headers and stats into a pandas DataFrame. If you haven’t used pandas before, it’s essentially an Excel spreadsheet that you can manipulate programmatically, so it’s great for handling larger datasets. We notice above that the Rk column does not have corresponding data in our qb_stats list — this is because those values were all table headers, so our findAll() function did not pull their data. Regardless — this data is frivolous for our analysis so we will omit it when creating the DataFrame.
# Create DataFrame from our scraped datadata = pd.DataFrame(qb_stats, columns=column_headers[1:])
columns — the column titles for our DataFrame (we are omitting Rk)
Now, let’s take a look at the first five rows of our DataFrame:
# Examine first five rows of datadata.head()
Success!
Now, we can manipulate the data in this DataFrame to get what we need to make our radar charts. First, let’s look at all the columns in the set:
# View columns in datadata.columns>>> Index(['Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD'], dtype='object')
We see one issue immediately — there are two columns titled Yds; one is for passing yards, and the other is yards lost due to being sacked. We can easily remedy this by renaming the latter:
# Rename sack yards column to `Yds_Sack`new_columns = data.columns.valuesnew_columns[-6] = 'Yds_Sack'data.columns = new_columns
Now, let’s view our columns again:
# View columns in datadata.columns>>> Index(['Player', 'Tm', 'Age', 'Pos', 'G', 'GS', 'QBrec', 'Cmp', 'Att', 'Cmp%', 'Yds', 'TD', 'TD%', 'Int', 'Int%', '1D', 'Lng', 'Y/A', 'AY/A', 'Y/C', 'Y/G', 'Rate', 'QBR', 'Sk', 'Yds_Sack', 'NY/A', 'ANY/A', 'Sk%', '4QC', 'GWD'], dtype='object')
We have successfully renamed the column to Yds_Sack and no longer have a conflict in our column names! Next, let’s identify which statistical categories we are interested in for our visualization. We will choose:
(1) Completion percentage — Cmp%
(2) Passing yards — Yds
(3) Passing touchdowns — TD
(4) Interceptions — Int
(5) Yards per attempt — Y/A
(6) Passer rating — Rate
# Select stat categoriescategories = ['Cmp%', 'Yds', 'TD', 'Int', 'Y/A', 'Rate']
Now let’s create a new DataFrame as a subset of our original data, only with the data from our chosen categories. Additionally, we will add the player name and team.
# Create data subset for radar chartdata_radar = data[['Player', 'Tm'] + categories]data_radar.head()
Now that we have our subset of data, let’s check the data type, because we scraped the values as text from the URL:
# Check data typesdata_radar.dtypes>>> Player    object    Tm        object    Cmp%      object    Yds       object    TD        object    Int       object    Y/A       object    Rate      object    dtype: object
All of our numerical data have been stored as objects, so we can’t manipulate them. So before we proceed further, we must convert all this data to numerical values. To do so, we use a function called pandas.to_numeric.
# Convert data to numerical valuesfor i in categories:  data_radar[i] = pd.to_numeric(data[i])
Now, let’s check our data types once again:
# Check data typesdata_radar.dtypes>>> Player     object    Tm         object    Cmp%      float64    Yds       float64    TD        float64    Int       float64    Y/A       float64    Rate      float64    dtype: object
We have one last piece of data cleaning to do. On the original website, they put ornamental characters next to players that had end-of-season achievements, such as a Pro Bowl (*) or All-Pro (+) selection. We will remove these using str.replace():
# Remove ornamental characters for achievementsdata_radar['Player'] = data_radar['Player'].str.replace('*', '')data_radar['Player'] = data_radar['Player'].str.replace('+', '')
Let’s filter our data down to only the quarterbacks who threw for more than 1500 yards:
# Filter by passing yardsdata_radar_filtered = data_radar[data_radar['Yds'] > 1500]
Now, for our radar chart, we want to calculate the each quarterback’s statistical rank by percentile, which is easily done in pandas with DataFrame.rank(pct=True). The rank() function can take other arguments to rank based on other parameters for which you can look up the online documentation.
Additionally, we want to flip our interceptions rank, as we don’t want the QB with the most interceptions to have the highest percentile!
# Create columns with percentile rankfor i in categories:  data_radar_filtered[i + '_Rank'] =   data_radar_filtered[i].rank(pct=True)# We need to flip the rank for interceptionsdata_radar_filtered['Int_Rank'] = 1 - data_radar_filtered['Int_Rank']
Now we examine our new data:
# Examine datadata_radar_filtered.head()
Great! We can now make our visualizations!
We will start by editing some general plot parameters. We are going to be generating a polar plot, so the x-ticks correspond to the angle around the circle — we are increasing the padding between the axis and tick labels:
# General plot parametersmpl.rcParams['font.family'] = 'Avenir'mpl.rcParams['font.size'] = 16mpl.rcParams['axes.linewidth'] = 0mpl.rcParams['xtick.major.pad'] = 15
For colors in our chart, we will use the HEX codes of the NFL team colors, which were collected from this link.
team_colors = {'ARI':'#97233f', 'ATL':'#a71930', 'BAL':'#241773', 'BUF':'#00338d', 'CAR':'#0085ca', 'CHI':'#0b162a', 'CIN':'#fb4f14', 'CLE':'#311d00', 'DAL':'#041e42', 'DEN':'#002244', 'DET':'#0076b6', 'GNB':'#203731', 'HOU':'#03202f', 'IND':'#002c5f', 'JAX':'#006778', 'KAN':'#e31837', 'LAC':'#002a5e', 'LAR':'#003594', 'MIA':'#008e97', 'MIN':'#4f2683', 'NWE':'#002244', 'NOR':'#d3bc8d', 'NYG':'#0b2265', 'NYJ':'#125740', 'OAK':'#000000', 'PHI':'#004c54', 'PIT':'#ffb612', 'SFO':'#aa0000', 'SEA':'#002244', 'TAM':'#d50a0a', 'TEN':'#0c2340', 'WAS':'#773141'}
The angles at which we plot all our points will be dependent on the number of statistical categories we have. In our example, we have 6 categories, so we will plot our points every 2π/6 radians, or 60 degrees. We can calculate this as follows, using numpy.linspace() :
# Calculate angles for radar chartoffset = np.pi/6angles = np.linspace(0, 2*np.pi, len(categories) + 1) + offset
We are adding the offset term to adjust where the labels appear on the circle (instead of the first label at 0 radians, it now appears at π/6).
When we plot the data, we need to actually duplicate the first category, so that the shape closes itself. If you notice when we calculated the angles, we have 7 data points, and the first and last angles correspond to the same point on the circle. So when we plot the data, we use the following (we will turn this into a function, so player_data represents a row of player-specific data that we will pass):
# Plot data and fill with team colorax.plot(angles, np.append(player_data[-(len(angles)-1):], player_data[-(len(angles)-1)]), color=color, linewidth=2)ax.fill(angles, np.append(player_data[-(len(angles)-1):], player_data[-(len(angles)-1)]), color=color, alpha=0.2)
The above code works because our categories are the last 6 columns of our DataFrame, so len(angles) — 1 corresponds to the first of these categories (since angles has an extra element). We then append the value of the first category onto the end of this array so that we can close the shape.
Now we can set the labels for the category names (since we have one less category than angles, we omit the last element):
# Set category labelsax.set_xticks(angles[:-1])ax.set_xticklabels(categories)
Finally, we will add the player name on top of the radar chart — we place the text object at (π/2, 1.7), in absolute plot coordinates so that it appears above the axis:
# Add player nameax.text(np.pi/2, 1.7, player_data[0], ha='center', va='center', size=18, color=color)
We can put this all together to make a helper function to generate our radar charts as follows:
Let’s also make another helper function to return a numpy array of QB data when given a team input:
# Function to get QB datadef get_qb_data(data, team):  return np.asarray(data[data['Tm'] == team])[0]
NFC West
Since I’m a huge 49ers fan, let’s start by looking at the radar charts for QBs in the NFC West:
# Create figurefig = plt.figure(figsize=(8, 8), facecolor='white')# Add subplotsax1 = fig.add_subplot(221, projection='polar', facecolor='#ededed')ax2 = fig.add_subplot(222, projection='polar', facecolor='#ededed')ax3 = fig.add_subplot(223, projection='polar', facecolor='#ededed')ax4 = fig.add_subplot(224, projection='polar', facecolor='#ededed')# Adjust space between subplotsplt.subplots_adjust(hspace=0.8, wspace=0.5)# Get QB datasf_data = get_qb_data(data_radar_filtered, 'SFO')sea_data = get_qb_data(data_radar_filtered, 'SEA')ari_data = get_qb_data(data_radar_filtered, 'ARI')lar_data = get_qb_data(data_radar_filtered, 'LAR')# Plot QB dataax1 = create_radar_chart(ax1, angles, lar_data, team_colors['LAR'])ax2 = create_radar_chart(ax2, angles, ari_data, team_colors['ARI'])ax3 = create_radar_chart(ax3, angles, sea_data, team_colors['SEA'])ax4 = create_radar_chart(ax4, angles, sf_data, team_colors['SFO'])plt.show()
The data seems to support what we see from the eye test :
Russell Wilson is on an elite level, reaching at least 75th percentiles in all statistical categories. This analysis also doesn’t take into account what he brings with his scrambling ability.Jimmy Garoppolo is generally a solid QB, but throws a few boneheaded interceptions from time-to-time.Jared Goff gets a lot of yards, but it does not seem to translate to a lot of production (i.e. “empty yards”).Kyler Murray is hard to judge because he was a rookie, but his efficiency looks to be the lowest of the four. Like Wilson, his mobility is not accounted for in this plot.
Russell Wilson is on an elite level, reaching at least 75th percentiles in all statistical categories. This analysis also doesn’t take into account what he brings with his scrambling ability.
Jimmy Garoppolo is generally a solid QB, but throws a few boneheaded interceptions from time-to-time.
Jared Goff gets a lot of yards, but it does not seem to translate to a lot of production (i.e. “empty yards”).
Kyler Murray is hard to judge because he was a rookie, but his efficiency looks to be the lowest of the four. Like Wilson, his mobility is not accounted for in this plot.
MVP Race
The MVP race (at QB) came down Russell Wilson and Lamar Jackson (who eventually won):
# MVP Race# Create figurefig = plt.figure(figsize=(8, 4), facecolor='white')# Add subplotsax1 = fig.add_subplot(121, projection='polar', facecolor='#ededed')ax2 = fig.add_subplot(122, projection='polar', facecolor='#ededed')# Adjust space between subplotsplt.subplots_adjust(hspace=0.8, wspace=0.5)# Get QB databal_data = get_qb_data(data_radar_filtered, 'BAL')sea_data = get_qb_data(data_radar_filtered, 'SEA')# Plot QB dataax1 = create_radar_chart(ax1, angles, sea_data, team_colors['SEA'])ax2 = create_radar_chart(ax2, angles, bal_data, team_colors['BAL'])plt.show()
From a pure passer standpoint, Russell Wilson has more balanced stats, but Lamar Jackson clearly had the more “bang-for-buck” passing yards, as shown by his number of passing TDs and quarterback rating. Additionally, Lamar Jackson set the QB rushing record for a season, which is not captured in this chart.
Teams who drafted 1st round QBs
Four quarterbacks were taken in the 1st round of the 2020 NFL Draft: Joe Burrow (CIN), Tua Tagovailoa (MIA), Justin Herbert (LAC), and (surprisingly) Jordan Love (GB). Let’s visualize the stats of the quarterbacks that these draft picks would replace.
# 1st Round Draft Picks# Create figurefig = plt.figure(figsize=(8, 8), facecolor='white')# Add subplotsax1 = fig.add_subplot(221, projection='polar', facecolor='#ededed')ax2 = fig.add_subplot(222, projection='polar', facecolor='#ededed')ax3 = fig.add_subplot(223, projection='polar', facecolor='#ededed')ax4 = fig.add_subplot(224, projection='polar', facecolor='#ededed')# Adjust space between subplotsplt.subplots_adjust(hspace=0.8, wspace=0.5)# Get QB datacin_data = get_qb_data(data_radar_filtered, 'CIN')mia_data = get_qb_data(data_radar_filtered, 'MIA')lac_data = get_qb_data(data_radar_filtered, 'LAC')gnb_data = get_qb_data(data_radar_filtered, 'GNB')# Plot QB dataax1 = create_radar_chart(ax1, angles, cin_data, team_colors['CIN'])ax2 = create_radar_chart(ax2, angles, mia_data, team_colors['MIA'])ax3 = create_radar_chart(ax3, angles, lac_data, team_colors['LAC'])ax4 = create_radar_chart(ax4, angles, gnb_data, team_colors['GNB'])plt.show()
Andy Dalton had a terrible season by all metrics — it clearly appears that it was time to move onRyan Fitzpatrick was below average in all six statistical metrics and had a fairly mediocre season. Considering Josh Rosen was benched for Fitzpatrick, Miami needed to invest in Tua for the future.Philip Rivers actually performed average to above average in most categories, but clearly had some bad decision-making, judging by the number of interceptions he threw.Aaron Rodgers had what appears to be an above average season, but still below average by the lofty expectations we have for him. The Jordan Love pick was a shock for everyone.
Andy Dalton had a terrible season by all metrics — it clearly appears that it was time to move on
Ryan Fitzpatrick was below average in all six statistical metrics and had a fairly mediocre season. Considering Josh Rosen was benched for Fitzpatrick, Miami needed to invest in Tua for the future.
Philip Rivers actually performed average to above average in most categories, but clearly had some bad decision-making, judging by the number of interceptions he threw.
Aaron Rodgers had what appears to be an above average season, but still below average by the lofty expectations we have for him. The Jordan Love pick was a shock for everyone.
As you can see, the possibilities are endless — and this framework can be easily adapted for other stats such as rushing and receiving metrics. The Jupyter notebook used for this article can be found at this Github repository.
Thank you for reading! I appreciate any feedback, and you can find me on Twitter and connect with me on LinkedIn for more updates and articles.