A Tutorial on Scraping Images from the Web Using BeautifulSoup | by Acusio Bivona | Towards Data Science

In the real world of data science, it’ll often be a task that we have to obtain some or all of our data. As much as we love to work with clean, organized datasets from Kaggle, that perfection is not always replicated in day-to-day tasks at work. That’s why knowing how to scrape data is a very valuable skill to possess, and today I’m going to demonstrate how to do just that with images, along with eventually displaying your image results in a Pandas DataFrame.
To start, I’m going to scrape from the website that I first learned to scrape images from, which is books.toscrape.com. This is a great site to practice all of your scraping skills on, not just image scraping. Now, the first thing you’ll want to do is import some necessary packages — BeautifulSoup and requests.
from bs4 import BeautifulSoupimport requests
Next, you’ll want to make a get request to retrieve your webpage and then pass the contents of the page through BeautifulSoup so that it can be parsed.
html_page = requests.get('http://books.toscrape.com/')soup = BeautifulSoup(html_page.content, 'html.parser')warning = soup.find('div', class_="alert alert-warning")book_container = warning.nextSibling.nextSibling
Awesome! Now, we need our images. Being efficient with BeautifulSoup means having a little bit of experience and/or understanding of HTML tags. But if you don’t, using Google to find out which tags you need in order to scrape the data you want is pretty easy. Since we want image data, we’ll use the img tag with BeautifulSoup.
images = book_container.findAll('img')example = images[0]example
And the output of this is the following:
<img alt="A Light in the Attic" class="thumbnail" src="media/cache/2c/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg"/>
Good job! You successfully pulled the first image from this website. But, the part that is most important for us in pulling the actual image is in the beginning of line 2 where it says src=... because this is the URL. Well, it’s not quite fully a URL, but a URL extension. We want to pull that out as a separate variable, which will be done shortly. To breakdown the output a little bit more, we can use the .attrs method to pull out the URL found in src=... Look below:
example.attrs['src']
Bam. That’s it! If done correctly, you should receive an output that looks like...
'media/cache/2c/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg'
Now that this has been pulled out, we can download the image locally! To do this, you’ll want to import another package called shutil. Then, instead of kind of rambling on about what’s going on with the code here, I’m going to paste the code block below and comment on the most important lines so that my brain nor your brain doesn’t explode, because it’s kind of dense :).
url_base = "http://books.toscrape.com/" #Original websiteurl_ext = example.attrs['src'] #The extension you pulled earlierfull_url = url_base + url_ext #Combining first 2 variables to create       a complete URLr = requests.get(full_url, stream=True) #Get request on full_urlif r.status_code == 200:                     #200 status code = OK   with open("images/book1.jpg", 'wb') as f:       r.raw.decode_content = True      shutil.copyfileobj(r.raw, f)
The last block of code where the if statement starts essentially is saying that if you get the OK status code (200), it is going to open the image from book 1 (“A Light In The Attic” — the same book from you URL scrape), decode the raw content, and then save the image onto your local machine.
You’re doing great! Now that you have your first image downloaded, you’ll want to preview that image to make sure that it was downloaded correctly. To do this, you’ll need to import two more packages — matplotlib.pyplot and matplotlib.image.
import matplotlib.pyplot as pltimport matplotlib.image as mpim
To preview your image, run the following code:
img = mpimg.imread('images/book1.jpg')imgplot = plt.imshow(img)plt.show()
And your output will be.. *drumroll*
Your image! Congratulations! From here, the final thing you’ll want to do is display this in a Pandas DataFrame. We have a couple more packages that need to be imported, and I’m going to display the code right underneath that.
import pandas as pdfrom IPython.display import Image, HTMLrow_1 = [example.attrs['alt'], '<img src="images/book1.jpg"/>']df = pd.DataFrame(row_1).transpose()df.columns = ['Title', 'Cover']HTML(df.to_html(escape=False))
And voila! You have now converted an image you found on the Internet into a Pandas DataFrame! Obviously, you can write your code so that you can scrape an entire webpage at one time by writing a function instead of doing this process over and over again for each entry. But, this concludes my blog post. Thank you for reading and I hope this helped you learn something new or helped better your scraping skills!