How to scrape ANY website with python and beautiful soup (static web) | by Patrick Collins | Towards Data Science

Note: This is a purely technical tutorial. Please check with the policies of the website before engaging in any scraping.
For those who want to see it done in front of your eyes, check out my YouTube video at the bottom of the page.
Scraping the web can be done for a TON of reasons.
Do you want to get stats on your football team so you can algorithmically manage your fantasy team? Boom, make a web scraper that scrapes ESPN. Track your competitor's activity on different social media? Great, that’s covered here too.
Or maybe you’re a Developer Advocate who is looking for good ways to measure his OKR of hackathon involvement and there is no current good tool out there so you want to build your own.
That last one was oddly specific, and is what we are going to be looking for! This tutorial shows how you can get all the hackathons from devpost that are ending in the next 50 days, based on the keyword blockchain .
Anyway, let’s jump right into how we can scrape anything with python. I’m going to assume you have space where you can code, and are familiar with how to work with python.
The documentation for this is very strong, so be sure to check it out after this tutorial!
Beautiful soup works great for static web pages. If you follow this and get weird/bad results, you’ll probably need a web driver to scrape the site. I published an ADVANCED version of doing this, but for 95% of cases, the following will do the trick.
pip install requests
pip install beautifulsoup
Run those two so you can work with the packages.
This one isn’t as cut-and-dry. If you’re looking to scrape through multiple web sites, you’ll need multiple URLs. This tutorial is focused on just scraping a single site. Once you understand how scraping a single page works, you can move to more pages.
For our tutorial, we are going to be using:
https://devpost.com/hackathons?utf8=%E2%9C%93&search=blockchain&challenge_type=all&sort_by=Submission+Deadline
Since it gives us all of our parameters; the blockchain keyword and time till the hackathon is over.
Every page is made of HTML/CSS/javascript (well... for the most part), and every bit of data that shows up on your screen shows up as text. You can every inspect this page! Just right click, and hit “inspect”.
This will bring up all the code that the pages uses to render. This is the key to web scraping. Do you see the “Elements” tab? That has all the HTML/CSS code you need.
Now you don’t need to know how HTML/CSS works (although, it can be really helpful if you do). The only thing that’s important to know is that you can think of every HTML tag as an object. These HTML tags have attributes that you can query, and each one is different.
Each line of code in that image that starts with <footer> ,<script> , or <div> , these are the start of the tags, and they end with </footer> , </script>, or </div> respectfully. Everything that is in between these tags, are also queryable, and count as part of that tag. Once you have a tag, you can get anything inside that tag.
So we start the scraping by pulling the website we want with the requests object:
import requestsfrom bs4 import BeautifulSoupresult = requests.get("https://devpost.com/hackathons?utf8=%E2%9C%93&search=blockchain&challenge_type=all&sort_by=Submission+Deadline")src = result.contentsoup = BeautifulSoup(src, 'lxml')
And we store the result in a BeautifulSoup object called soup above.
This is just the boiler plate to any soup scraping, the next is the customizable part.
You can now start to find out what tag you want, this is where you need to get a little creative, since you can generally approach the problem a number of different ways. For our example, we want to find all the hackathon listings, which we found they were all wrapped in an a tag, and had a featured_challenge attribute. Here is what their HTML code looked like:
<a class=”clearfix” data-role=”featured_challenge” href=”https://utonhack.devpost.com/?ref_content=default&amp;ref_feature=challenge&amp;ref_medium=discover">...</a>
The 3 . ‘s represent other tags inside this tag. We are going to ignore those for now, since the data we were looking for was right inside this tag. We want that URL. As you can see, this is an a tag since it starts with <a , and it has an attribute of data-role="featured_challenge" and that’s what we are going to use to identify it.
We can use this to find a list of every single one of these by using the find_all function.
featured_challenges = soup.find_all('a', attrs={'data-role': 'featured_challenge'})
The featured_challenges now is a list of a tag objects that we can get that URL from.
If we loop through that list we can do something like:
print(featured_challenge.attrs['href'])
The attrs is a map of attributes each tag has. If you look back up at the a tag we pulled from, you saw there was an href attribute that holds the URL of the hackathon we are looking for, hooray!
Each one of these tag objects counts as another HTML object, so you could do find_all on each one of the objects too! If you only want the first result, you can use the find function instead.
Also, if you want to just get the text of the object, you can just look for the text attribute of the tag object, like so:
featured_challenge.text
And that’s it!
If you want the code for a really simple scraper used in this demo, check it out here.
Full code for multi-hackathon scraper with web driver