How to Scrape Predicted Lineups from Fantasy Football Scout with Python
By: Liam Hartley
Tags: Football, Programming, Python, Data Science, Technology

1. Scrape ALL the HTML

To scrape the HTML and to prepare it for parsing we’re going to use two popular Python libraries: Requests and Beautiful Soup:

import requests

from bs4 import BeautifulSoup

We want to take all of the HTML from from our target page. We can do this with the ‘requests’ and load the plain plain text element of the resulting object:

html = requests.get(url).text

Now in order to use our parsing library (Beautiful Soup) we have to load this into a ‘soup object’ to parse the HTML more easily.

soup = BeautifulSoup(html, "html.parser")

We’re now ready to being parsing the HTML to pull out the data that we’re interested in!

2. Find the lineups

There are all sorts of ways to parse through this BeautifulSoup object, I’ve decided to use ‘find_all’ to parse Fantasy Football Scout’s HTML.

lineups_raw = soup.find_all("div", {"class": re.compile('formation.*')})

This line of code looks for all classes inside the HTML within a ‘div’ that match the regex expression: “formation.*”.

The regex is required as Fantasy Football Scout splits its tags for different formations. The HTML tags look like:

< div class ="formation formation-3–4–3>

or

< div class ="formation formation-4–3–3>

The lineups_raw object now contains twenty elements that contain the lineups for each respective team. In order to iterate through the raw data we require a simple loop:

for team in raw_data:

lineup_dictionary[key] = team.text.split(' ')[1:]

logger.info(f'Lineup {key}: {lineup_dictionary[key]}')

assert len(lineup_dictionary[key]) == 11

key += 1

This will loop through each element and add add the list of players to a dictionary with an integer key.

3. Find the team names

Now to find the bits of HTML that identify each of the teams by their respective names.

team_names_raw = soup.find_all('header')

Apart from the first two objects in this list, we get a complete collection of the teams names. In order to iterate through this data we’re going to need another simple loop:

for team in range(2, 22):

team_map[team-2] = raw_data[team].text.split('Next')[0]

logger.info(f'{team_map[team-2]} mapped to {team-2}')

The range spans from 2 to 22 to account for the first two objects that we’re not interested in, this is also why team_map has the team-2 key assigned to it instead of simply team .

Note that this dictionary has keys which will match up to the previous dictionary that we have created.

4. Match the lineups to the team names

These matching keys are important as we’re now going to create a dictionary which brings the values of both dictionaries into a combined dictionary:

for key in range(0, 20):

formatted_dictionary[mapping_dictionary[key]] = lineups_data[key]

logger.info(f'Team: {mapping_dictionary[key]}')

logger.info(f'Lineup: {formatted_dictionary[mapping_dictionary[key]]}')

The resulting dictionary will be of the format:

{team1: [player1, player2... player11], ... team20: [...] }

And that’s it! You can now play with this data and use it for a larger project. Let me know if you have any questions and I’d be happy to help you out!

You can find the full solution and contribute additional features here.