Image Scraping with Python. A code-along guide to learn how to... | by Fabian Bosler | Towards Data Science

To train a model, you need images. You can most certainly download them by hand, possibly even somewhere in batch, but I think there is a much more enjoyable way. Let’s use Python and some web scraping techniques to download images.
Update 2 (Feb 25, 2020): One of the problems with scraping webpages is that the target elements depend on the a selector of some sort. We use css-selectors to get the relevant elements from the page. Google seemed to have changed its site layout sometime in the past, which made it necessary to update the relevant selectors. The provided script should be working again.
Update: Since writing this article on image scraping, I have published the article on building an image-recognizing convolutional neural network. If you want to put the scraped images to good use, check out the following article!
towardsdatascience.com
Scraping static pagesScraping interactive pagesScraping images from GoogleAfterword on legality
Scraping static pages
Scraping interactive pages
Scraping images from Google
Afterword on legality
A Python environment (I suggest Jupyter Notebook). If you haven’t set this up, don’t worry. It is effortless and takes less than 10 minutes.
towardsdatascience.com
Scraping static pages (i.e., pages that don’t utilize JavaScript to create a high degree of interaction on the page) is extremely simple. A static webpage is pretty much just a large file written in a markup language that defines how the content should be presented to the user. You can very quickly get the raw content without the markup being applied. Say that we want to get the following table from this Wikipedia page:
We could do this by utilizing an essential Python library called requests like this:
As you can see, this is not very useful. We don’t want all that noise but instead would like to extract only specific elements of the page (the table to be precise). Cases like this are where Beautiful Soup comes in extremely handy.
Beautiful Soup allows us to navigate, search, or modify the parse tree easily. After running the raw content through the appropriate parser, we get a lovely clean parse tree. In this tree, we can search for an element of the type “table”, with the class “wikitable sortable.” You can get the information about class and type by right-clicking on the table and clicking inspect to see the source code.We then loop through that table and extract the data row by row, ultimately getting this result:
Neat Trick:Pandas has a built-in read_html method that becomes available after installing lxml (a powerful XML and HTML parser) by running pip install lxml. read_html allows you to do the following:
As you can see, we call res[2] as pd.read_html() will dump everything it finds that even loosely resembles a table into an individual DataFrame. You will have to check which of the resulting DataFrames contains the desired data. It is worth to give read_html a try for nicely structured data.
However, most modern web pages are quite interactive. The concept of “single-page application” means that the web page itself will change without the user having to reload or getting redirected from page to page all the time. Because this happens only after specific user interactions, there are few options when it comes to scraping the data (as those actions do have to take place).
Sometimes the user action might trigger a call to an exposed backend API. In which case, it might be possible to directly access the API and fetch the resulting data without having to go through the unnecessary steps in-between. Most of the time, however, you will have to go through the steps of clicking buttons, scrolling pages, waiting for loads and all of that ... or at least you have to make the webpage think you are doing all of that. Selenium to the rescue!
Selenium can be used to automate web browser interaction with Python (also other languages). In layman’s term, selenium pretends to be a real user, it opens the browser, “moves” the cursor around and clicks buttons if you tell it to do so. The initial idea behind Selenium, as far as I know, is automated testing. However, Selenium is equally powerful when it comes to automating repetitive web-based tasks.
Let’s look at an example to illustrate the usage of Selenium. Unfortunately, a little bit of preparation is required beforehand. I will outline the installation and usage of Selenium with Google Chrome. In case you want to use another Browser (e.g., Headless) you will have to download the respective WebDriver. You can find more information here.
Steps:
Install Google Chrome (skip if its already installed)Identify your Chrome version. Typically found by clicking About Google Chrome. I currently have version 77.0.3865.90 (my main version is thus 77, the number before the first dot).Download der corresponding ChromeDriver from here for your main version and put the executable into an accessible location (I use Desktop/Scraping)Install the Python Selenium package viapip install selenium
Install Google Chrome (skip if its already installed)
Identify your Chrome version. Typically found by clicking About Google Chrome. I currently have version 77.0.3865.90 (my main version is thus 77, the number before the first dot).
Download der corresponding ChromeDriver from here for your main version and put the executable into an accessible location (I use Desktop/Scraping)
Install the Python Selenium package viapip install selenium
Starting a WebDriver
Run the following snippet (for ease of demonstration do it in a Jupyter Notebook) and see how a ghostly browser opens up.
import selenium# This is the path I use# DRIVER_PATH = '.../Desktop/Scraping/chromedriver 2'# Put the path for your ChromeDriver hereDRIVER_PATH = <YOUR PATH>wd = webdriver.Chrome(executable_path=DRIVER_PATH)
If all went according to plan you should see something like this now:
Now run (in a new cell):
wd.get('https://google.com')
Your browser should navigate to — not surprisingly — google.com. Now run:
search_box = wd.find_element_by_css_selector('input.gLFyf')search_box.send_keys('Dogs')
You’ll see the result as your Browser will type Dogs into the search box.
Alright, let’s close the driver down:
wd.quit()
Perfect! You’ve got the basics covered. Selenium is extremely powerful, and pretty much every interaction can be simulated. Some actions are even accessible via abstract methods, like clicking buttons or hovering over things. Also, if worst comes to worst, you can always fall back on mimicking human behavior by moving the cursor to where you want it and then performing a click action.
Since you now understand the basics, we can piece everything together. Let’s do the browser do our bidding by:
Searching for a specific term & get image links
Downloading the images
Searching for a particular phrase & get the image links
The function fetch_image_urls expects three input parameters:
query : Search term, like Dogmax_links_to_fetch : Number of links the scraper is supposed to collectwebdriver : instantiated Webdriver
query : Search term, like Dog
max_links_to_fetch : Number of links the scraper is supposed to collect
webdriver : instantiated Webdriver
Downloading the imagesFor the following snippet to work, we will first have to install PIL by running pip install Pillow.
The persist_image function grabs an image URL url and downloads it into the folder_path. The function will assign the image a random 10-digit id.
Putting it all togetherThe following function search_and_download combines the previous two functions and adds some resiliency to how we use the ChromeDriver. More precisely, we are using the ChromeDriver within a with context, which guarantees that the browser closes down ordinarily, even if something within the with context raises an error. search_and_download allows you to specify number_images, which by default is set to 5, but can be set to whatever number of images you want to download.
Now we can do the following:
and will get:
Congratulations! You have built your very own image scraper. Use the scraper with consideration and enjoy the cup of coffee that you are having instead of downloading 100 images by hand.
If you want to learn how to automate the scraping process and run it continuously, check out the following article:
towardsdatascience.com
Ohh and if you enjoy reading stories like these and want to support me as a writer consider signing up to become a Medium member. It’s $5 a month, giving you unlimited access to stories on Medium. If you sign up using my link, I’ll even make some 🍩.
medium.com
I am not a lawyer, so nothing I say should be taken as legal advice. Having said that, the question around the legality of web scraping most likely has to be evaluated on a case-by-case basis. It seems to be a consensus that you are in the clear as long as you do not violate any terms of services or negatively affect the web pages you are scraping. The act of web scraping itself can’t be illegal. You could scrape your page without repercussions whatsoever and also the Google bot is scraping the entire web every day after all. My advice:
Make sure that you are not breaking any laws, terms of services, or otherwise have a negative impact on your target.