Scraping and Exploring Sports Betting Data — Is Arbitrage Possible? A Hands-On Analysis with Code. | by Steven Eulig | Towards Data Science

Who wouldn’t like to own a risk-free profit machine? I guess everyone would like to have one. The practice of taking advantage of price differences (in this case betting rate differences) between different markets or bookmakers is also known as arbitrage. The idea is to place bets on every outcome of a sample space and generate a profit in every case. In soccer, you could bet which player wins a game, i.e. the sample space would consist of the three outcomes ‘Player 1 wins’, ‘Tie’ and ‘Player 1 loses’.
Before we dive deeper into the analysis, we should first look at the data collection and further topics which we will cover along the way:
Web scraping live betting data (BeautifulSoup)Storing results in dataframes (pandas)Automation of the scraping process with a functionVisualization of the results with (matplotlib-pyplot)First arbitrage analysis and profit calculations
Web scraping live betting data (BeautifulSoup)
Storing results in dataframes (pandas)
Automation of the scraping process with a function
Visualization of the results with (matplotlib-pyplot)
First arbitrage analysis and profit calculations
I have come up with many other exciting topics on this data, but I will leave them out of scope for now. (Might follow-up on them in the next articles.)
Web scraping from multiple websites and data consolidation
Derivation of rules for a betting / trading strategy incl. simulation and evaluation of the performance of the automated strategy
Driver analysis: What drives the evolution of betting rates (potentially including text analysis from a live conference ticker)
Probably there are many more interesting topics; happy to take your input
Legal disclaimer: I am not a legal expert and do not provide any recommendations for action here, but a pure technical explanation. Massive scraping of websites causes high traffic and could thereby burden them. I believe if you are accessing websites, you should always consider their terms of service and ideally obtain prior permission for projects like scraping. Moreover, I do not promote gambling, no matter what kind of betting it is.
To gather the data, we will use the library BeautifulSoup. I have downloaded and installed it with the command line via:
pip install beautifulsoup4
Once you have installed it, we can start by importing all relevant libraries.
from bs4 import BeautifulSoupimport urllib.requestimport re
We can now use them to retrieve the source code of any page and parse it. I have selected the live bets page from the German betting website tipico.de.
url = “https://www.tipico.de/de/live-wetten/"try: page = urllib.request.urlopen(url)except: print(“An error occured.”)soup = BeautifulSoup(page, ‘html.parser’)print(soup)
The above code checks if it can access the page and then print out the entire html source code of your page. This is then assigned to a variable called ‘soup’ by convention and will be about 200–300 A4 pages long, depending on how many bets are currently online.
As a next step, we want to extract only the relevant information in a well-structured manner. Let’s look at (a screenshot from) the website and check which information we care about.
To start with, we could try to extract the seven values marked in green: The time, the name of the players, the current score and the rates for each possible outcome. Here Rate 1 corresponds to Player 1 wins, Rate 2 corresponds to tie and Rate 3 corresponds to Player 1 loses. To do so, it is convenient to inspect the element with your browser (right-click and ‘Inspect’ with Chrome).
We see that the rates are stored in buttons of class “c_but_base c_but”. To extract the rates, we will use soup.find_all to get all buttons of this class.
regex = re.compile(‘c_but_base c_but’)content_lis = soup.find_all(‘button’, attrs={‘class’: regex})print(content_lis)
To get rid of the html code, we use the .getText() function, then store the values of all buttons in a list and remove line breaks and tabulators.
content = []for li in content_lis: content.append(li.getText().replace(“\n”,””).replace(“\t”,””))print(content)
The other variables can be queried analogously. You can find the details in my notebook on Github. Don’t hesitate to ask questions if something is unclear.
Having the raw and parsed data, we now want to structure it in a practical way. We understand that there is one time of the game, two players, one score and eleven rates for each row. As for the rates, we only want to store the three rates of “who wins the game” and three rates of “who scores the next goal”, because these are available on most of the betting sides and will later allow us a high degree of comparability.
N_games = 10 # number of games observed, say we want the first 10N_players = 2 # number of players per gameN_outcomes = 11 # number of possible outcomes (Win, lose, tie, Next goal etc.)df = []for i in range(N_games):    df.append([datetime.now(), content_names[i*N_players], content_names[1+i*N_players], content_minute[i],        content_score[i], content[i*N_outcomes], content[1+i*N_outcomes], content[2+i*N_outcomes],        content[6+i*N_outcomes], content[7+i*N_outcomes], content[8+i*N_outcomes]])pdf = pd.DataFrame(df, columns = ['Time', 'Player_1', 'Player_2', 'MinuteOfGame', 'Score', 'Win_1', 'Win_X', 'Win_2', 'NextGoal_1' , 'NextGoal_X' , 'NextGoal_2' ])pdf.head()
(Note: In my notebook, there are two more variables, j and k. They are used to get rid of bugs, when there is an additional row per game for ‘results of the first half of the game’. For sake of simplicity, I have excluded it from this description.)
To repeat the scraping process many times, it is convenient to write it into a function:
def get_soccer_rates_tipico():    """    This function creates a table with the live betting information,    this includes a timestamp, the players, the score and the rates    for each party winning and scoring the next goal.        Arguments:    None        Returns:    pdf -- pandas dataframe with the results of shape (N_games, 11)    """       ... FULL CODE ON GITHUBreturn pdf
The shape of the returned table depends on the number of games that are currently live. I have implemented a function which finds the first tennis game entry to figure of the total number of live soccer games and get all of them.
Now we can write it into a loop to repeat the scraping function at fixed time intervals.
def repeat_scraping(timedelay, number_of_scrapes, filename = 'bet_rates_scraping_tipico.csv'):    """    This function repeadetly calls the scraping function to create a timeseries of scraping data. The time interval between scrapes and number of scrapes in total are taken as argument. The result is saved in a csv-file.        Arguments:    timedelay -- delay between each scrape request in seconds (min. 15 sec recommended due to processing time) number_of_scrapes --  number of scrape requests         Returns:    Void    """                   ... FULL CODE ON GITHUB        dataframe = pdf.to_csv(filename, index=False)           # Check processing time and add sleeping time to fit the timedelay        time_run = time.time() - start_time        time.sleep(timedelay - time_run)
Note that the loading and parsing of the page might take up to 10 seconds. We can use the time library to track how long our function takes to subtract it from the sleeping time to align the time delays (assuming that all queries take approximately the same time).
We can call this function to scrape an entire game, for example with a time delay of 15 seconds and 500 queries, i.e. covering 125 minutes.
repeat_scraping(15, 500, 'scraping_500x15s.csv')
As a first step, after importing the data, we need to do some data cleaning. This includes filling the NaNs with zeros for all rates, replacing the commas by dots and converting them into float types.
dataframe = pd.read_csv('scraping_500x15s.csv', encoding = 'unicode_escape')dataframe = dataframe.fillna(0)ratecols = ['Win_1','Win_X','Win_2','NextGoal_1','NextGoal_X','NextGoal_2']dataframe[ratecols] = dataframe[ratecols].apply(lambda x: x.str.replace(',','.')).astype(float)
Because we want to query multiple times per minute, the ‘minute of the game’ is not precise enough, therefore, we add the query timestamp.
dataframe['Time_parsed'] = 0# Check for dates without milliseconds and add .000 to have a consistent formattingdataframe.Time.iloc[np.where(dataframe.Time.apply(lambda x: True if len(x) == 19 else False))] \= dataframe.Time.iloc[np.where(dataframe.Time.apply(lambda x: True if len(x) == 19 else False))].apply(lambda t: t + ".000")dataframe.Time_parsed = dataframe.Time.apply(lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f').time())dataframe = dataframe.drop(['Time'], axis=1)
Now we can explore the data of a first game and visualize the results. Let’s look at the very first Player of our table:
df1 = dataframe[dataframe[‘Player_1’] == dataframe[‘Player_1’][0]]df1
We now have data on the Bayern München vs 1. FC Köln game starting from minute 5 for every 15 seconds. The following plot visualizes the evolution of the rates for who scores the next goal. I used a log-scale on the y-axis to account for the rates over 30. Additionally, I included a horizontal line at a rate of three, signalling a simple form of arbitrage: If all quotes are greater than three, you could generate a risk-less profit by placing uniformly distributed bets (e.g. 100 € each) of on all three outcomes.
# Data for plottingt = df1.Time_parsed.valuesw1 = df1.NextGoal_1.valuesw2 = df1.NextGoal_X.valuesw3 = df1.NextGoal_2.values# Plot setupfig, ax = plt.subplots(figsize=(15, 6))        ax.plot(t, w1, marker ='', label = 'Player 1 scores next' ,color = 'green', linewidth = 2)ax.plot(t, w2, marker ='', label = 'No more goals', color = 'orange', linewidth = 2)ax.plot(t, w3, marker ='', label = 'Player 2 scores next', color = 'red', linewidth = 2)plt.axhline(y=3., label = 'Critical line', color='grey', linewidth = 2, linestyle='--') # Line for arbitrage detectionax.set(xlabel='Time', ylabel='Betting rates', title=str(np.unique(df1.Player_1)[0]) + ' vs ' + \               str(np.unique(df1.Player_2)[0]) + ': Rates for "Who scores next?"')ax.grid()plt.legend()ax.set_yscale('log')
In my notebook, you will find a function that automatically creates the plots for all games and saves them as images.
Coming back to our idea of finding arbitrage deals, it is probably unlikely to find them on one single website. I believe it is more likely to achieve arbitrage betting by using information differences between multiple bookmakers. However, we will now look at the analysis only for Tipico data. This analysis could easily be scaled to multiple websites from there on.
A brief excursion into mathematics:Be n the number of possible outcomes of an event and q_i the rate for each outcome, then arbitrage is possible if the sum of all 1/q_i is smaller than one.
df2['Check_a_1'] = df2.Win_1.apply(lambda x: 1/x)df2['Check_a_2'] = df2.Win_X.apply(lambda x: 1/x)df2['Check_a_3'] = df2.Win_2.apply(lambda x: 1/x)df2['Check_a_sum']=0df2['Check_b_1'] = df2.NextGoal_1.apply(lambda x: 1/x)df2['Check_b_2'] = df2.NextGoal_X.apply(lambda x: 1/x)df2['Check_b_3'] = df2.NextGoal_2.apply(lambda x: 1/x)df2['Check_b_sum']=0df2['Check_a_sum'] = df2.Check_a_1 + df2.Check_a_2 + df2.Check_a_3df2['Check_b_sum'] = df2.Check_b_1 + df2.Check_b_2 + df2.Check_b_3df2['Arbitrage_flag']=0arb_idx = np.unique(np.append(np.where(df2.Check_a_sum <= 1)[0],np.where(df2.Check_b_sum <= 1)[0]))df2.Arbitrage_flag[arb_idx] = 1
If we want to obtain an outcome-independent profit, we need to distribute the betting amounts according to the rates. When you consider rate q multiplied by amount s equals profit p to fulfill p_i = p_j, you attain s_j = q_i/q_j * s_i.
# Give the first bet the weight of 1 and adjust the other two bets accordinglydf2['Win_1_betting_fraction'] = 1df2['NextGoal_1_betting_fraction'] = 1... FULL CODE ON GITHUBdf2['Win_profit_percentage'] = df2.Win_1 * df2.Win_1_betting_amount * 100 - 100
In the data that I collected, I found the following example:
Rates for scoring the next goal:Player 1: 1.70No more goals: 4.70Player 2: 7.00
This leads to the following betting amount distribution:Bet 1: 62.32%Bet 2: 22.54%Bet 3: 15.14%
This results in a sure profit of 5.9%.
(It seems strange to me that I found an arbitrage chance on a single website. Maybe there was an error in scraping the numbers, I will investigate this.)
As demonstrated, this procedure enables you to calculate the sure profit of a combination of bets. Now it is up to you to find positive values!
In this article, you could learn how to load and parse web pages with Python using BeautifulSoup, extract only desired information and load it into structured data tables. To extract time series, we have looked at how we can implement it as functions and repeat our query at fixed time intervals. Afterwards, we wrote code to automatically clean the data, followed by visualizations and additional calculations to make the data interpretable.
We have seen that arbitrage betting is probably possible — if not on one website, then by combining the bets with multiple bookmakers. (Please note that I am not promoting gambling and/or web scraping and that even arbitrage bets may also be subject to risks, e.g. delays or cancellations of bets.)
Thank you very much for taking your time and reading this article!
I highly appreciate and welcome your feedback and remarks. You can reach out to me via LinkedIn: https://www.linkedin.com/in/steven-eulig-8b2450110/...and you can find my github including the Jupyter notebook here: https://github.com/Phisteven/scraping-bets