I Used to Pay $180/yr for a Profitable Betting Tool. This Year I Built One in Python
By: Frank Andrade
Tags: Technology, Data Science, Programming, Python, Web Scraping

First, we accept the cookies pop-up, then we set the website language to English, so we can navigate through multiple competitions and get the bookie's odds available for the next 7 days.

If you feel you can do that on your own, skip section 1, but make sure the data you obtain is similar to the one we’ll get in section 1. Let’s start coding! The full code of the scraper is available at the end of section 1.

Section 1: Scraping the odds

1. Scraping pre-match odds with Selenium

Import libraries and Change Chromedriver default options

We need to import Options to change chromedriver’s default options; By , WebDriverWait , EC , time to wait for a certain condition to occur (explicit wait) and time to wait for a specific amount of time (implicit wait).

The default options we’re going to edit are “headless mode” and “windows size.” You can disable headless mode by writing option.headless=False and use driver.maximize_window() instead of options.add_argument(‘window-size=1920x1080’) as shown in the full code.

Breaking down the code:

options = Options() creates an instance of the Options class

creates an instance of the Options class options.headless = True turns on the headless mode

turns on the headless mode options.add_argument(‘window-size=1920x1080’) opens the windows in a customized size

Initialize storage and loop through each league

In the next steps, we will create 1 dataframe for 1 football league. That’s why we need to store each dataframe in a dictionary I’m calling dict_frames . We also need to create another dictionary to help our scraper recognize the leagues we want to get data from.

The dictionary’s keys are the name of the country (as written on the website) , while the values are in the form of a list. Each list contains the leagues of each country. You can remove or add any other country/league, but remember to write them as shown in the English version of the website (I’ll show how to pick the names in the “Clicking on specific leagues” subsection below).

Once we created our dictionaries, we need to loop through each country and league. With this, we’re going to open a chrome window for each league and scrape the odds and other important data.

Breaking down the code:

webdriver.Chrome(path, options = options) applies the changes in the chromedriver we made before

applies the changes in the chromedriver we made before driver.get opens the browser

opens the browser web represents the betting site’s URL, while path represents the chromedriver’s path in your computer

represents the betting site’s URL, while represents the chromedriver’s path in your computer driver.maximize_window() maximizes the chrome window. I usually need it when I don’t work in headless mode

maximizes the chrome window. I usually need it when I don’t work in headless mode time.sleep(2) is an implicit wait. You can also use an explicit wait, as I show in the “option 1” comment

is an implicit wait. You can also use an explicit wait, as I show in the “option 1” comment accept is the "accept all cookies" button that pops up every time we visit this website

Screenshot of the accept button

To get the XPath of the accept button, right-click and “inspect” the element. After doing this, you’ll see a line highlighted. Right-click, then copy and paste the XPath.

Switch languages from a dropdown menu

Before we scrape the data, we need to switch the website language to English. To do so, we write the following code.

Breaking down the code:

language_box is the box that contains all the languages available for the website. We locate it with the class name ssc-hlsw You can verify this class name by inspecting the language box in the top right corner.

Screenshot of language_box

WebDriverWait(language_box,5).until(EC.element_to_be_clickable((By.CLASS_NAME, ‘ssc-en_GB’))).click() clicks on the English language icon.

Finally, we need to make selenium wait for the website to load all the content in English. To do so, we need to pick a reference to make an explicit wait. In this case, I’ll use “Over/Under 2.5 Goals” as a reference, so we’re going to wait until the dropdown contains the text element Over/Under 2.5 Goals in English within the span tag. This is shown in the code below.

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//span[contains(text(), "Over/Under 2.5 Goals")]')))

Clicking on specific leagues (inside the loop)

Once all the content is in English, we have to follow this sequence to get every league's odds.

Click Competition → Click country name → Click league name. To do so, we write this code.

Breaking down the code:

header is the box that contains the “competitions” button. Inspect the following box, and you’ll get the class name of it.

Screenshot of the header

competition is the "competitions" button we need to click on, so the list of countries is displayed. We locate that button with the contains option to match an element that contains COMPETITIONS inside title within a tag.

is the "competitions" button we need to click on, so the list of countries is displayed. We locate that button with the option to match an element that contains inside within tag. Before we click country_button and league_button is better to locate first the competition_table shown in the picture below.

Screenshot — Competition table

To locate competition_table right-click the competition table and look for the id. It should look something like this mod-multipickazmenu-1061-container

country_button and league_button display all the leagues available on the website. The XPath of a country has the following form.

'.//div[contains(@data-category, "australian football")]'

This locates an element that contains “australian football” inside data-category within a div tag. To find the data-category, inspect the element like the picture below.

That being said, in our code, we replace the name of a specific data league (e.g. “australian football”) for the country variable so we can loop through each country in the dict_countries dictionary. The final XPath looks like this.

WebDriverWait(competitions_table, 5).until(EC.element_to_be_clickable((By.XPATH, './/div[contains(@data-category,' +'"' + country + '"' + ')]')))

We follow the same steps for the league_button.

Scraping team_names, odds, and dates

Finally, we’re ready to scrape the data we want. First, we choose any betting market available in the dropdown. In this example, I’ll choose to scrape odds from “Over/Under 2.5 Goals” and “Both teams to Score?”. We also need a dictionary dict_odds to store the list_odds , teams and list_dates lists that will contain the data we’re about to scrape.

This is the scheme to scrape the data of each league.

As you can see, it has a sequence similar to those we followed to get pre-match odds or live odds in previous articles. First, we loop through each betting market and then select them from the dropdown menu with the chooser variable. Then we loop through each section to split the data for each date. In the end, we loop through each row , which represents a football match.

In this case, I’ll explain only those details that aren’t included in previous articles. If you have any questions about the rest of the code, just ask me.

Breaking down the code:

section_date = section_date * len(rows) helps us create a list of dates with the length of the rows inside each section. This is stored in list_dates

helps us create a list of dates with the length of the inside each This is stored in list_dates = [element for section in list_dates for element in section] unpacks the nested lists stored in list_dates

unpacks the nested lists stored in dict_odds stores the data scraped. %s % i helps us change the name of the keys according to the enumerate(markets)

stores the data scraped. helps us change the name of the keys according to the driver.quit() closes the browser

2. Clean and transform pre-match odds data

Once we finished the scraping part, we need to format the data properly. Keep in mind that we’ll do this inside the for loop started in subsection 1.

Creating dataframes for each league

We have to create one dataframe for each betting market odds we scraped. In this example, I scraped “Over/Under 2.5 Goals” and “Both Teams to Score,” so I created df_over_under and df_btts frames.

Breaking down the code:

pd.DataFrame creates a dataframe from the lists we created before

creates a dataframe from the lists we created before pd.concat concatenates the two dataframes in one dataframe I’m calling df_betfair

concatenates the two dataframes in one dataframe I’m calling .fillna('') fills any NaN value with an empty value, while .replace() replaces any data that contains the words "SUSPENDED" with an empty value.

fills any value with an empty value, while .replace() replaces any data that contains the words "SUSPENDED" with an empty value. .applymap() applies the formula .strip() to each string element of the df_betfair dataframe

Managing Time in the pandas dataframe

We need to convert the time format used in the website to a standard format YYYY–MM–DD.

Breaking down the code:

datetime.date.today() creates a today variable that we’ll use to identify in which date we’re scraping.

creates a variable that we’ll use to identify in which date we’re scraping. datetime.timedelta(days=1) adds 1 day to get the tomorrow variable

The format used in the website looks like this “Saturday, 13 February” (%A, %d %B) except for matches played on the present day (“Today”) or the next day (“Tomorrow”). We need to format everything to YYYY–MM–DD (%Y %d %B)

re.sub() replaces words such as “In-Play,” “Today” and “Tomorrow” that are displayed on the website for the standard time format used in the website (%A, %d %B)

replaces words such as “In-Play,” “Today” and “Tomorrow” that are displayed on the website for the standard time format used in the website (%A, %d %B) x.split(‘,’)[1] gets everything that is after the comma in “Saturday, 13 February ”

gets everything that is after the comma in “Saturday, 13 February datetime.datetime.strptime() converts the date format to YYYY–MM–DD (%Y %d %B)

Finally, we store the df_betfair dataframe containing each league inside the dict_frames dictionary with dict_frames[dict_countries[country][league]] = df_betfair and save the file with pickle.dump() We’ll save the dict_frames with the name dict_betfair that we’ll use later.

Congratulation! We finished working with script 1 (Selenium). You can find the code of our first script on my Github. Now it’s time to start working with script 2 (Pandas)

Remember that the data scraped contains only the odds for matches in the next 4 days or so. So I highly recommend you automate this first script, so it can run, let’s say, every Wednesday and Sunday to never run out of data. In the article below, I show how to do this in 3 simple steps.