Easy Web Scraping With Python
By: Geoffrey Mariette
Tags: Python, Tutorial, Web Scraping, Data Science, DevOps

Let’s first talk about the module we will use are, defined from line 1 to 5:

requests : We need it to retrieve the HTML web page and it’s content

: We need it to retrieve the HTML web page and it’s content logging : always useful for debugging purpose

: always useful for debugging purpose BeautifulSoup : our HTML parser — we will talk about it in detail after

: our HTML parser — we will talk about it in detail after argparse : used to implement a verbose level using -v flag (line 7 to 14)

: used to implement a verbose level using -v flag (line 7 to 14) datetime: powerful to calculate our day +1 (see line 17 for easy calculation, and line 18 for typical Python var assignment)

This is easy. We lookup for tomorrow values, and pass them to a GET request (l25 / l29). Below is what the output will be like if run in DEBUG mode



2020–11–13 21:09:25,079 — root — DEBUG — day: 14, month: 11, year: 2020

2020–11–13 21:09:25,079 — root — DEBUG — URL :

2020–11–13 21:09:25,208 — urllib3.connectionpool — DEBUG — Starting new HTTPS connection (1):

2020–11–13 21:09:25,485 — urllib3.connectionpool — DEBUG —

2020–11–13 21:09:25,531 — root — DEBUG — Response content: b’<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “ python tennis.py -vv2020–11–13 21:09:25,079 — root — DEBUG — day: 14, month: 11, year: 20202020–11–13 21:09:25,079 — root — DEBUG — URL : https://www.tennisexplorer.com/matches/?type=atp-single&year=2020&month=11&day=14 2020–11–13 21:09:25,208 — urllib3.connectionpool — DEBUG — Starting new HTTPS connection (1): www.tennisexplorer.com:443 2020–11–13 21:09:25,485 — urllib3.connectionpool — DEBUG — https://www.tennisexplorer.com:443 “GET /matches/?type=atp-single&year=2020&month=11&day=14 HTTP/1.1” 200 167902020–11–13 21:09:25,531 — root — DEBUG — Response content: b’-//W3C//DTD XHTML 1.0 Transitional//EN” “ http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd ">



That’s looks good. Next, we need to analyse our HTML content. One thing that can help you before parsing the HTML with a script, is to use the Firefox / Chrome dev mode to identify clearly how are represented the informations you are looking for inside the web page.

In my example, I would like to grab all the informations inside the blue rectangle while discarting the one inside the red one.

If you use the inspector tool, you will quickly see that all the wanted informations are in a table with a class attribute = result.

Well, that’s a good start for an extract. Then, we can also filter on the tbody (table body) element just to reduce a litte bit more.

Then, we need to identify two things:

Are the tournament in a specific class / id or anything else ? yes, it’s always a tr element with a class set as “ head flags ”

element with a class set as “ ” Is the same thing applied to the players ? Well, not exactly. Each face to face is identified by a dedicated id. For example, player1 will have id=r1 while player 2 will have id=r1b. Maybe we can do easier here, we will see after.

First, let’s grab all the tournament position into our tbody, to be able to parse every blocs one by one:

Line 2, we init our BeautifulSoup parser with the response.content as a parameter.

Then, we apply a filter on the table element (l4), with a reduce on the tbody ligne 5.

Next, we use enumerate to get both the tournament name (t_name) — and the tournament line (i) which will be append in our tournament index list (tournament_idx_lst).

python tennis.py -vv

2020-11-13 21:38:05,436 - root - INFO - Sofia is located at line 0 of the tbody

2020-11-13 21:38:05,437 - root - INFO - Cary challenger is located at line 3 of the tbody

2020-11-13 21:38:05,438 - root - INFO - Bratislava challenger is located at line 6 of the tbody

2020-11-13 21:38:05,440 - root - INFO - Futures 2020 is located at line 11 of the tbody

2020-11-13 21:38:05,443 - root - INFO - [0, 3, 6, 11, 24]

Below is an output of what our loop will produce. That’s a more than a good start as we are now able to use our tournament_idx_lst to extract the content between two bounds.

If you didn’t notice, player’s names are truncated. I would like to have the whole name. We need to extract the href link for each player, go to the player’s web page, and get the full name.

An href link exists for each player, on a <td> element, class “t-name”

I will dedicated a function to this task as it will be use as many time as we have players.

This one is also super easy. On a player page, Player’s name is located in a table with class plDetail, between two <h3> tags. Note the trick that player_name is a list, and the first item [0] is always our player’s name.

Below example will show you how it works:

print(getPlayersFullName("/player/sinner-8b8e8/"))

Jannik Sinner # Check the content of player_name var

print(player_name)

[<h3>Sinner Jannik</h3>] # Check the content of item n°1 of the list

print(player_name[0])

<h3>Sinner Jannik</h3> # Convert it to text (beautiful soup method)

print(player_name[0].text)

Sinner Jannik # Reverse the content (I need first name - last name)

# Convert it to a list to easily reverse, then join it back to have a string

print(' '.join(player_name[0].text.split()[::-1]))

Jannik Sinner

This looks good. We just need now to parse the whole page with some for loops. What I would like to generate here is a dictionary organised this way:

tournament_dict = {

"tournament1": {

"current_date": ["x vs y", "z vs p"]

},

"tournament2": {

"current_date": ["a vs b", "c vs d"]

}

}

Let me explain what we do here

First, we parse our tournament_idx_list which contains all the tournaments lines numbers of our tbody item

which contains all the tournaments lines numbers of our tbody item The tournament name is extracted from the <td> tag, class t-name , from rows list (which is all the <tr> from the tbody table)

tag, , from (which is all the <tr> from the tbody table) If the tournament name does not exists into our dict, we create the above tree structure

Then the loop from line 8 to 9 parse all the record between two tournaments (based on the tournament positions — present into tournament_idx_list)

If you run the code, you will get something like this:



{'Sofia': {'2020-11-15': ['Jannik Sinner vs Vasek Pospisil']}} # Output with url = https://www.tennisexplorer.com/matches/?type=atp-single&year=2020&month=11&day=15 {'Sofia': {'2020-11-15': ['Jannik Sinner vs Vasek Pospisil']}}

{'Masters Cup ATP': {'2020-11-15': ['Dominic Thiem vs Stefanos Tsitsipas', 'Rafael Nadal vs Andrey Rublev']}} # Output with url = https://www.tennisexplorer.com/matches/?type=atp-single&year=2020&month=11&day=15 {'Masters Cup ATP': {'2020-11-15': ['Dominic Thiem vs Stefanos Tsitsipas', 'Rafael Nadal vs Andrey Rublev']}}

# Multiple tournaments

{'Cordoba': {'2020-11-15': ['Pablo Cuevas vs Federico Delbonis', 'Gianluca Mager vs Pablo Juan Ficovich', 'Attila Balazs vs Lorenzo Sonego']}, 'Pune': {'2020-11-15': ['Salvatore Caruso vs Ramkumar Ramanathan', 'Viktor Troicki vs Sumit Nagal', 'Yuichi Sugita vs Thomas Fabbiano', 'Cedrik-Marcel Stebe vs Ivo Karlovic']}, 'Montpellier': {'2020-11-15': ['Norbert Gombos vs Pablo Carreno-Busta', 'Gregoire Barrere vs Joao Sousa', 'Vasek Pospisil vs Aljaz Bedene', 'Damir Dzumhur vs Elliot Benchetrit', 'Sergiy Stakhovsky vs Danilo Petrovic', 'Enzo Couacaud vs Guillermo Garcia-Lopez', 'Emil Ruusuvuori vs Nicolas Mahut']}} # Output with url = https://www.tennisexplorer.com/matches/?type=atp-single&year=2020&month=02&day=03 # Multiple tournaments{'Cordoba': {'2020-11-15': ['Pablo Cuevas vs Federico Delbonis', 'Gianluca Mager vs Pablo Juan Ficovich', 'Attila Balazs vs Lorenzo Sonego']}, 'Pune': {'2020-11-15': ['Salvatore Caruso vs Ramkumar Ramanathan', 'Viktor Troicki vs Sumit Nagal', 'Yuichi Sugita vs Thomas Fabbiano', 'Cedrik-Marcel Stebe vs Ivo Karlovic']}, 'Montpellier': {'2020-11-15': ['Norbert Gombos vs Pablo Carreno-Busta', 'Gregoire Barrere vs Joao Sousa', 'Vasek Pospisil vs Aljaz Bedene', 'Damir Dzumhur vs Elliot Benchetrit', 'Sergiy Stakhovsky vs Danilo Petrovic', 'Enzo Couacaud vs Guillermo Garcia-Lopez', 'Emil Ruusuvuori vs Nicolas Mahut']}}

It was exactly what I wanted! Of course, you can run the script with a loop to have more complete picture of the future matches.

Hope this tutorial made web scraping a little bit less dark for you. Do not hesitate if you have any suggestions, please use the comment section below.

Thank you for reading