How to scrape any website with Python and Beautiful Soup (dynamic web) | by Patrick Collins | Towards Data Science

Note: This is a purely technical tutorial. Please check with the policies of the website before engaging in any scraping. In the example I use, it’s not legal to scrape the site: https://gitcoin.co//legal/terms. Please do not use.
For those who want to see it done in front of your eyes, check out my YouTube video at the bottom of the page.
Now if you didn’t read my part one, go back and read it! This is building on the concepts learnt there. And remember, the documentation for this is very strong, so be sure to check it out after this tutorial!
We are now going to learn how to deal with dynamic web pages or web pages that have a lot going on and are more than just HTML/CSS. These pages are tricker to scrape with beautifulsoup, because sometimes, code happens on the server side, and beautifulsoup needs to let a browser run it.
Wouldn’t it be nice if there was a way for us to code and have our code pretend it was a browser?... Well it just so happens that’s exactly what this article is about!
Now, I first read about this in this article, and I’ve kept this article more focused on the exact implementation, so for more details, be sure to check it out!
These are the only two I know of at the moment that have this functionality, so be sure to use one of them. For this article, we are using Firefox.
The web driver we will be using is geckodriver, and you can find all the downloads at the bottom of that link. After you download it, extract the file from your downloads (just double click the zip file).
Web drivers are a little finicky, and don’t play so well with operating systems, so we need to:Move geckodriver to where a path executable is. Please check this article for how to do it with windows.
We can find out where our executable path is by running:
echo $PATH
And then we can either:
cp the geckodriver file from the zip into one of the locations your $PATH variable tells you (after running echo $PATH)Add a new $PATH location to your $PATH environment variable, and put the executable geckodriver in there. You can see how to do this here.
cp the geckodriver file from the zip into one of the locations your $PATH variable tells you (after running echo $PATH)
Add a new $PATH location to your $PATH environment variable, and put the executable geckodriver in there. You can see how to do this here.
Then restart your terminal for the changes to take affect. You may also need to notarize the application if you’re using MacOS (which I am). You can do that by running:
xattr -r -d com.apple.quarantine geckodriver
After running:pip install selenium , you can now start using the web driver! Instead of pulling the dynamic web page URL with requests.get you’ll pull it with driver.get , like this:
from selenium import webdriverfrom selenium.webdriver.firefox.options import Optionsfrom bs4 import BeautifulSoupoptions = Options()# Make it go faster by running headless# This will remove the web browser GUI from popping upoptions.headless = Truedriver = webdriver.Firefox(options=options)# Enter whatever URL you likedriver.get("https://gitcoin.co/{}".format(link.attrs['href']))# Let the code on their end runtime.sleep(20)# Save it to a variablehtml = driver.page_sourcedriver.quit()# And then just paste it right back into beautifulsoup!projects_soup = BeautifulSoup(html, 'lxml')
You can see a full code example located at the Alpha Chain demos repo, a full scale example at my hackathon-scraper project, and a video that shows you how to do it all right here.
And remember, you can always fast forward :)