Make Money With Python — The Sports Arbitrage Project
By: Frank Andrade
Tags: Technology, Python, Web Scraping, Programming, Data Science

Time to Code

We’ll be scraping live data from 3 betting sites. We’ll start with live data from matches in the betting site ‘Tipico.’ For those who followed my previous web scraping tutorial, this will be a similar approach, but with advanced tools necessary to get live odds data.

Among the 3 betting sites, I considered this one the hardest one to get data from, so once you’re able to get live odds from this one, you’d be able to understand the code for the other 2 sites easily and also create your own betting site scraper.

Let’s start coding! The full code is available at the end of section 4.

Update March 21st, 2021:

I uploaded CSV output examples of the Tipico, Betfair, and Bwin scrapers to my Github, so you have an idea of how the output of section 1 (Scraping live betting data with Selenium) looks like. However, this data still needs to be preprocessed in sections 2 and 3 before finding surebets. There was a slight change of names in the dropdowns of the Tipico betting site. Initially, the name was, “Both teams to score?” but now it is “Both Teams to Score” The code below considers the new names.

1. Scraping live betting data with Selenium

Importing libraries

We need to import Options to change chromedriver’s default options; Select for selecting within dropdown menus and By , WebDriverWait , EC , time to wait for a certain condition to occur.

Changing Chromedriver default options

To keep scraping while doing something else on the computer, we need to use the headless mode. To do so, we write the following code:

Breaking down the code:

options = Options() creates an instance of the Options class

creates an instance of the Options class options.headless = True turns on the headless mode

turns on the headless mode options.add_argument(‘window-size=1920x1080’) opens the windows in a customized size on the background

opens the windows in a customized size on the background webdriver.Chrome(path, options = options) applies the changes we made in the chromedriver

applies the changes we made in the chromedriver driver.get opens the browser

opens the browser web represents the betting site’s URL, while path represents the chromedriver’s path in your computer

If you don’t want to work in headless mode, then write option.headless=False and use driver.maximize_window() instead of options.add_argument(‘window-size=1920x1080’) as shown in the full code.

Select values from dropdown menus

We need to select the betting markets we want to get data from. We do this with Select and the Xpath of the dropdown. To get the Xpath, do this:

Image by author

But first, keep in mind that every time you open some betting sites, a cookies banner is displayed. We need to get rid of them by clicking the ‘accept button.’

To click on the ‘accept button’ and then select on the dropdown menus, we write the following code:

Breaking down the code:

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="_evidon-accept-button"]'))) makes the driver wait until the ‘ok button’ of the cookie banner is clickable. If this throws an error, then use ‘option 2’ instead. To get the Xpath, right-click on the ‘ok button’ and inspect as we did before for the dropdown menu.

makes the driver wait until the ‘ok button’ of the cookie banner is clickable. If this throws an error, then use ‘option 2’ instead. To get the Xpath, right-click on the ‘ok button’ and inspect as we did before for the dropdown menu. WebDriverWait(driver,5).until(EC.presence_of_element_located((By.XPATH, '...'))) makes the driver wait some seconds until the dropdown menu is located

makes the driver wait some seconds until the dropdown menu is located Select('...') selects dropdown menus

selects dropdown menus first_dropdown one of the three dropdowns that contain betting markets

one of the three dropdowns that contain betting markets first_dropdown.select_by_visible_text() selects an element inside the dropdown menu with the help of the betting market’s name.

I’m choosing ‘Both Teams to score’ and ‘Over/Under’ in particular because they represent betting markets that I consider easy to find surebets.

Looking for ‘live events’ and ‘sports titles’

Before getting the data, we need first to look for live events. Also, to simplify the analysis, we’ll choose only ‘football’ in sports’ names. To do so, we write the following code:

Breaking down the code:

box represents the box that contains sports events. The site also contains upcoming events, which we don’t need for this analysis.

represents the box that contains sports events. The site also contains upcoming events, which we don’t need for this analysis. driver.find_element_by_xpath() helps us find an element within the website through the ‘Xpath’ of that element.

Unlike Xpaths from most elements we worked with before, the ‘live events’ element is trickier to get. For that reason, we need to be very specific. Then we use the contains option to match an element that contains Program_LIVE inside the testid within a div tag. Check the picture below to find Program_LIVE

Program_live — Image by author

sport_title represents the sport name of each section.

represents the sport name of each section. driver.find_elements_by_class_name() helps us find an element within the website through the ‘class_name’ of that element. To obtain the class name, just follow the same steps we used before for dropdown menus, but in this case, click on the ‘Football’ header and copy the class name ‘SportTitle-styles-sport’. Since we use ‘elements,’ we’ll obtain a list of sports names that we’ll loop through later.

Before we go on with the code, we need to understand the scheme that will help us scrape live data.

Scheme for scraping live games

The scheme for scraping live games is similar to the one used for scraping pre-match games. However, live games in the betting site Tipico include ‘double row events’ and ‘empty events’ that we need to exclude.

Sports title: Represents the sports section. The website has many sports available, but we’ll focus only on football since it’s the most popular sport.

Double-row event: Football live events in the first half (0'-45') have 2 rows. The first row contains odds for the full game and the second contains odds for only the first half. We’ll only work with the first row.

Single-row event: Football live events in the second half (45'-90') have only 1 row that contains odds for the rest of the game.

odds_event: Represents the odds available within a row. Each row has 1 ‘odds_event’ and each ‘odds_event’ has 3 boxes with the markets ‘3-way,’ ‘Over/Under’ and ‘Handicap’ by default.

Empty events: Live odds might be suspended for a couple of seconds because of many reasons. We won’t consider rows with empty events to simplify the analysis.

Find ‘empty events’

Unlike pre-match games, odds in live games change quickly and could be suspended for a couple of seconds at any time. When betting odds are momentarily suspended, the odds are either locked or empty. Locked odds are easy to scrape, but empty odds are harder. That’s why we need to locate those empty odds with the following code:

Breaking down the code:

for sport in sport_title loops through the sports list previously obtained

loops through the sports list previously obtained parent represents the ‘parent node’ of the sport element, while grandparent is its ‘grandparent node’ that represents the whole ‘football’ container inside live events

represents the ‘parent node’ of the element, while is its ‘grandparent node’ that represents the whole ‘football’ container inside live events empty_groups represents each football game with suspended or empty odds

represents each football game with suspended or empty odds granparent.find_elements_by_class_name() gives a list of all the empty odds with class name ‘EventOddGroup-styles-empty-group’ inside the football live section. You can get the class name by doing this:

empty_events represents each row or football game with at least 1 empty odds market (3-way, over/under, etc.)

represents each row or football game with at least 1 empty odds market (3-way, over/under, etc.) [empty_group.find_element_by_xpath('./..') for empty_group in empty_groups] is a list comprehension that loops through the empty_groups list to obtain the ‘parent node’ of each element, that is, the empty_events

is a list comprehension that loops through the list to obtain the ‘parent node’ of each element, that is, the except handles errors that could be encountered in the try block. We need this to handle errors when there isn’t any game with empty odds

Remove empty_events from single_row_events

We find single row events and remove all empty events inside with the following code:

Breaking down the code:

single_row_events represent the first row of double-row events or the unique row of single-row events. As we explained before, both represent odds for the full game and we focus on them because it’s more common to find surebets in odds calculated for the whole game. Find the class name of single-row events by inspecting any single row. You should find this class name 'EventRow-styles-event-row'

represent the first row of double-row events or the unique row of single-row events. As we explained before, both represent odds for the full game and we focus on them because it’s more common to find surebets in odds calculated for the whole game. Find the class name of single-row events by inspecting any single row. You should find this class name [single_row_event for single_row_event in single_row_events if single_row_event not in empty_events] list comprehension that excludes any empty games inside the single_row_events

list comprehension that excludes any empty games inside the try/except handles errors that could be raised when there isn’t any game with empty odds

Getting live odds

Each dropdown selected before will give odds for 3 markets that are extracted with the following code:

Breaking down the code:

for match in single_row_events loops through all the matches inside the ‘single_row_events’ list

loops through all the matches inside the list odd_events represent each event with odds available

represent each event with odds available match.find_elements_by_class_name(‘EventOddGroup-styles-odd-groups’) helps us find all the ‘odds_event’ within every match. To find the class name ‘EventOddGroup-styles-odd-groups’ just right-click and inspect the code behind ‘odds box’ as we did before for empty odds

helps us find all the ‘odds_event’ within every match. To find the class name just right-click and inspect the code behind ‘odds box’ as we did before for empty odds for team in match.find_elements_by_class_name(‘EventTeams-styles-titles’) loops through the elements with the class name 'EventTeams-styles-titles' within the ‘match’ node. Matches have 2 teams (home and away team); that’s why we need to loop through them and obtain their name with team.text. Then they’re stored with the append method in the teams list

loops through the elements with the class name within the ‘match’ node. Matches have 2 teams (home and away team); that’s why we need to loop through them and obtain their name with Then they’re stored with the method in the list for odd_event in odds_events loops over the total number of live matches on the betting site.

loops over the total number of live matches on the betting site. for n, box in enumerate(odds_events) loops through all ‘odds boxes’ inside a match. In the beginning, we changed the dropdowns to ‘3-way,’ ‘Over/Under’ and ‘Both Teams to Score,’ so they’re the odds we’ll scrape

loops through all ‘odds boxes’ inside a match. In the beginning, we changed the dropdowns to ‘3-way,’ ‘Over/Under’ and ‘Both Teams to Score,’ so they’re the odds we’ll scrape rows = box.find_elements_by_xpath('.//*') gives all the children nodes (odds) inside the box element.

gives all the children nodes (odds) inside the box element. n==0 means ‘only take values from the first box.’ In this case, the first box is the ‘3-way’ box and is stored in the x12 list

means ‘only take values from the first box.’ In this case, the first box is the ‘3-way’ box and is stored in the list rows[0] tells Python ‘only pick the first row on each odds box.’ With this, we ignored the second row in double-row events.

Goal-line Image by author

The same process is followed for n==1 (Over/Under) and n==2 (BTTS), but in n==1 we also need to store the ‘goals line.’ This represents the total number of goals that you need to win the bet.

For example, Over 2.5 goals mean at least 3 goals to win the bet. Let’s break down the code for n==1.

box.find_element_by_xpath('./..') gives the ‘parent node’ of the box element that contains the odds.

gives the ‘parent node’ of the box element that contains the odds. goals = parent.find_element_by_class_name('EventOddGroup-styles-fixed-param-text').text finds the goal line by giving its class name. Just right click and inspect the goal line (shown in the picture above) to obtain its class name. Then we obtain the data with .text

finds the goal line by giving its class name. Just right click and inspect the goal line (shown in the picture above) to obtain its class name. Then we obtain the data with over_under.append(goals+'

'+rows[0].text) appends the goals line data in a standard format. The over/under odds format will look like this 2.5

2.4

1.5

appends the goals line data in a standard format. The over/under odds format will look like this driver.quit() closes the Chrome browser

At this point, the scraping part is done! Now we have to make the data readable and do some pre-cleaning with Pandas. Then we’ll save the data with Pickle.

Breaking down the code:

dict_gambling is the dictionary that stores all the lists that contain the odds scraped

is the dictionary that stores all the lists that contain the odds scraped pd.DataFrame.from_dict(dict_gambling) turn the dictionary into a dataframe df_tipico, so we can read it and work with it easily

turn the dictionary into a dataframe so we can read it and work with it easily df_tipico.applymap(lambda x: x.strip() if isinstance(x, str) else x) cleans all the leading and trailing white spaces that the odds might have with the strip method.

cleans all the leading and trailing white spaces that the odds might have with the method. open('...', 'wb') opens a file named df_tipico in the mode ‘write bytes’ (wb). We save this in the variable output

opens a file named in the mode ‘write bytes’ (wb). We save this in the variable pickle.dump(df_tipico, output) saves the dataframe created in the file named df_tipico. We used the same name for both the file and the dataframe, but they could be named differently

saves the dataframe created in the file named We used the same name for both the file and the dataframe, but they could be named differently output.close() closes the file

Great! We finished scraping the first bookmaker. Bookmaker 2 and 3 are easier to scrape. The full code of bookmakers 1, 2 and 3 is at the end of this article.

Now let’s work with the data obtained to find a surebet automatically!