Python Web Scraping Tutorial
By: Bilguun Batbold
Tags: Web, Python, Crawling, Scraping

<Response [200]>

This will be output we get. Great, response 200 means that the page was fetched successfully. Let’s now use our Beautiful Soup module to create an object. Add the below into the file.

# Create a BeautifulSoup object soup = BeautifulSoup(page.text, 'html.parser') print(soup)

Output when running this new file

When we run the file, we can get the entire html page of the GitHub trending page! Let’s now explore how we can extract the useful data.

Extracting data

Highlighted shows ‘repo-list’

Head over to your browser (Chrome in this case) and open up the GitHub Trending Page. Click inspect anywhere, and you can see that the entire body of our wanted data is in the tag <div class="repo-list"> so the class repo-list should be our initial focus.

Each individual repository information

Next, we can see that each of the repositories are defined in the <li class='col-12 d-block width-full py-4 border-bottom'> This is what we will retrieve next

Your code should now look like this. If you run this script now, the output should show 25

Next we will iterate through each of the list to retrieve the desired information.

Repository Name

Highlighted shows the tag that displays full repository name

The above snip shows that the full repository name occurs under the very first <a> tag. We can extract the text from. Since the it returns a string with / in between them, we can split the string using / to get an array of string. First index will have the developer name and the next index will have the repository name.

Number of Stars

Stars are defined using <svg> tag with class <octagon>

Since not all repository contain the number of stars as the first element, we cannot use the position to retrieve the number of stars. However, we can see that the <svg> that defines the star and the number of stars itself are under the same parent. So if we get the <svg> by using the class octicon octicon-star we can get the parent and then extract the text (which will be the number of stars).

For loop

I have already implemented the loop as shown above. For each item in our repo_list (which contains 25 items), let’s find the developer, repo name and the stars.

Run the above code and the output should be something like this:

Output showing the 3 field information requested

Great! We can print what we have set out to achieve. Printing is good on its own, but it would be even better if we can store it somewhere, such as on a csv file. So let’s save this information there.

Saving it as CSV

First we need to import the built-in csv module as such:

import csv

Then we need to open a file and write the headers into our csv file:

# Open writer with name file_name = "github_trending_today.csv" # set newline to be '' so that that new rows are appended without skipping any f = csv.writer(open(file_name, 'w', newline='')) # write a new row as a header f.writerow(['Developer', 'Repo Name', 'Number of Stars'])

Next, in the for loop, we need to write a new row into our csv file

f.writerow([developer, repo_name, stars])

That is all you need to save the trending information onto our csv file!

This is what our script looks like finally. Once you run in, you will a new file github_trending_today.csv appear in our folder. If you open it it will look like this:

Scraped Information

Great! You have completed a simple tutorial to extract website information using python!