“Hey Selenium, scrape me a shopping list”
By: Andre Violante
Tags: Artificial Intelligence, Retail, Python, Web Scraping, Data Science

“Hey Selenium, scrape me a shopping list”

I recently worked on a project that involved scraping some data from a website. I was new to scraping. Luckily, python makes it pretty simple to do basic web scraping using a library called Selenium. I’m going to go through a very simple project, using Selenium, of navigating the Zappos website and creating a holiday shopping list for myself. I’ll go step-by-step showing both python code and inspecting website elements while showing a handful of useful functions within Selenium that can get you up and running on your own web scraping project in no time.

Introduction to Selenium

Let’s do a quick intro into what Selenium is before getting into the “fun stuff”. Per the PyPI page, Selenium is a package “to automate web browser interaction from Python”. Pretty much anything that you can do as a human (actions like: browse, search, scroll, click, login, etc), you can write a python script using the Selenium library to do those tasks. Here are the basic steps of a scrape project that we’ll go through:

Open web browser Go to desired web page Perform desired action(s) Scrape information from rendered web page

As you can see even though there are only 4 steps mentioned, this could include A LOT of navigation within a website before arriving to the desired web page to scrape. Here is the link to the Git repo with all the step-by-step code and the final CSV file if you want to follow along or just jump ahead. Now, let’s jump into the project and start scraping!

Holiday Shopping List Project

1. Open Web Browser

The first step is to import selenium web driver and open a blank browser or “artistic canvas”. I found Firefox to be a little easier to work with as a browser. We’ll open and call our new browser driver so we can continue to “drive” or maneuver the browser through this driver object. Figure 1 illustrates the exciting and newly opened Firefox browser window.

from selenium import webdriver # open blank browser

driver = webdriver.Firefox()

Figure 1 - Empty Firefox web browser

2. Go to desired web page

The next step is to go to your desired web page. I’m going to navigate to the retail website Zappos. Full disclosure, I use to work at Zappos, but I’m still a HUGE fan. As you do more and more web scraping, you’ll quickly find out that just because a website looks pretty doesn’t mean its well coded. This leads to the point that ALL websites are unique so 1 project may not translate exactly to another.

Below you see the web page specified, url , and then we use that driver object from Step 1 to open the desired web page. Congrats! Through python you just opened a new Firefox browser and pointed that browser to a specific URL.

# navigate to a url

url = 'https://www.zappos.com/'

driver.get(url)

Figure 2 - Navigate to web page

3. Perform desired action(s)

Now its time for the bulk of the work where we’ll traverse through the website through specific actions. For this simple project we’re going to perform a search for “holiday gifts”, click buttons to get to a desired web page, and scrape the info from that page to create our shopping list.

Before jumping into these actions, I first want to discuss the main function we’re going to be using in this project. We’ll be using find_element_by_xpath() . The term XPath refers to XML Path which helps you look at the HTML structure of a webpage to find elements. I found this function to be the most reliable way to find the desired elements on a webpage. When using XPath you’ll notice we’ll be using a relative path which just means we’re starting somewhere in the middle of our HTML structure. I’m not going to go deep into HTML / CSS, but I want to show you what it looks like so you can make the connection between HTML and python code. You’ll see shortly what I mean. Let’s look at how it work!

Let’s look at the search bar and search button from Figure 2. When you right-click and do “inspect element” from your browser you will see HTML (Figure 3). There is a div class called “ef”. Within that div is a form class called “ff” and then an input with an id called “searchAll”. There is also a button within this class with type named “submit”. With XPath we can locate all these different elements an interact with them or as you’ll see later extract the information from them.

Figure 3 - Inspect element HTML Code for search bar and button

If you did inspect element on the form class “ff” and did a right-click → copy → XPath, you would see something like this. This is the full XPath, not the relative path so we don’t need this.

/html/body/div[1]/div[1]/header/div[3]/div/div[1]/form

The function, find_element_by_xpath() , uses the relative XPath. When you look at Figure 3 with the associated code it will make a lot more sense. What we want to do in terms of actions is insert some text into the search bar and click the search button. Easy enough right? Let’s do it!

The code below finds the search bar, clears out any default text, and then enters the search term “holiday gifts”. Notice how we found the search bar element using the function find_element_by_xpath() with the HTML in Figure 3. That’s how XPath works! Inspect the HTML in browser, find the element you want, and use find_element_by_xpath() to point to a specific element given its path.

# find search bar, clear, and enter search text for item

search_bar = driver.find_element_by_xpath("

//form[@class='ff']

//input[@id='searchAll']")

search_bar.clear()

search_bar.send_keys('holiday gifts')

You should now see the search bar in your browser contain the new search term, but we still need to actually click the search button. Again using XPath, find the click button, which we already did in Figure 3, and then we simply .click() the button to perform the action.

# find and click 'search' button

search_btn = driver.find_element_by_xpath("

//form[@class='ff']

//button[@type='submit']")

search_btn.click()

You should now find yourself on the page below (Figure 4)

Figure 4 - After searching “holiday gifts” from Zappos home page

At this point you already know enough to browse a website using selenium! I’m going to click on the “$100 & Under” tile here in Figure 4 using what we’ve already learned.

For this web page specifically, all 6 of the tiles in Figure 4 are in divs within divs of exact names. Sometimes that can be a problem if you’re looking for a specific element. In Figure 5 below you can see tile 1- “$50 & Under”, and tile 2 - “$100 & under” have the same div , p , and a class names. However, there are attributes within the a class like id , aria-label , etc. that are unique. I’m going to use aria-label and use a kind of “and” clause within my find_element_by_xpath() function to point to the specific element.

Figure 5 - Same HTML for 2 clickable tiles

Below is the code for finding and clicking the tile “$100 & Under”. We use the same .click() function from above once we find the element we want. One thing to pay attention to here is that when I search for a[@class='hatch-element...' I don’t have a continuation of // like in the other code blocks. Instead, I add on [@aria-label='image shop $100 & Under'] . This means to look for the specific class AND attribute mentioned.

gifts_under_100 = driver.find_element_by_xpath("

//div[@class='h_markdown']

//p[@class='h_hatch-element--p h_hatch-element--body-1 h_markdown__p']

//a[@class='h_hatch-element--a ez_image-link ez_image-link'][@aria-label='image Shop $100 & Under']")



gifts_under_100.click()

You should now be on a search result page that looks like Figure 6 below. This is the web page I want to start scraping aka my shopping list.

Figure 6 - Product result page for “$100 & Under” Tile

4. Scrape information from rendered web page

From this search result page, we need to loop through all the products and scrape the information like name, brand, price, image, product url, etc. This specific page doesn’t load the products until you scroll over them which introduces another, yet common, issue.

Sites like Facebook, in their feed, have this “infinite scroll” feature. Luckily, our product page is not infinite, but we do need our driver to scroll through in order for products to render. Remember, if it doesn’t render then it can’t be found and scraped.

For this particular page, scrolling directly to the bottom did not render everything in between so I needed to write a simple loop to scroll our driver section-by-section. The code below scrolls by a given pixel section size specified.

# step scroll to bottom by pixel size to load all images

for i in range(35):

driver.execute_script("window.scrollBy(0, 900)")

time.sleep(1)

You may need to adjust the pixel scroll in the first code block from 900 to whatever works with your browser window. I also used the time.sleep() function just to “pause” for 1 second before moving on to the next section to allow the products to load completely. I used 35 iterations in my loop, but you could set it higher without any issue even if you hit the bottom before your specified range limit.

You should now be at the bottom of your result page. Let’s inspect and find all the product elements needed. This is where having duplicate names can actually be to our advantage. Each individual product tile has the same div , article , and a class name (Figure 7). However, within each a class there is unique and valuable information in the label attribute. Here we can see product name, brand, and price. Also, in the href attribute we find the product URL. Thats what we’re going to scrape!

Figure 7 - Duplicate HTML tag names across different products

# get all product info: brand, product name, price, and product url

product_info = driver.find_elements_by_xpath("

//div[@class='h_quick-shop-style-tile']

//article[@class='h_hatch-style-tile']

//a[@class='h_hatch-element--a h_hatch-style-tile__link']")

Lets find the tags in the HTML that will be common for all products. From the above code, Selenium will return a list of elements named product_info . Until now, all of our returned objects have only been a single unique item that we interacted with(clicked, added text, etc.). However, since the XPath we passed to our driver exists many times over, 1 for each product in the results page, we’ll receive a list of elements back. To be exact, len(product_info) will be 190 elements that looks like this:

[<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="f0d90584-efa8-3e4d-b44a-3c6b3ba88e23", element="34c2a959-f3ff-a042-ad7c-909aad9b7438")>, <selenium.webdriver.firefox.webelement.FirefoxWebElement (session="f0d90584-efa8-3e4d-b44a-3c6b3ba88e23", element="2e0b1bcb-9f69-0c43-a281-8a0281059bcc")>...]

This doesn’t really provide us helpful information. We need to loop through this list of elements to extract the information. I’m going to create 2 lists: 1 for product description info (price, brand, and name) and 1 for the product urls.

product_desc = []

product_urls = []



for info in product_info:

product_desc.append(info.text.split('

'))

product_urls.append(info.get_attribute("href"))

For the product description I’m going to extract all text from each item ( info ) in our list ( product_info ). I used the split function here to split at each line break as the delimiter. The new list product_desc will look like this:

[['Blank NYC', 'Jumpsuit w/ Ruffle Detail', '$89.99MSRP $148.00'],['Blue by Betsey Johnson', 'Kassie', '$89.00'], ['Matt & Nat', 'Brave Micro - Dwell', '$94.95'], ['Swatch', 'Redvremya - SO29R700', '$80.00'],...]]

Looks pretty good! You can see, thanks to a nice uniform naming convention by Zappos, that it follows the pattern of brand, product name, and price. We have a little bit of a messy price feature but that can be cleaned up.

To get the product urls we just use the get_attribute() function and look for all href items. Lets take a look at that list:

Looks great! I double checked that both are the same length of 190 elements so we’re on a good track!

Now I want to get all the image urls so I can view the products. This will look very similar to the above code. We had to go a little deeper down our XPath to find the image items, but thats not hard now that we know what we’re looking for. Figure 8 below shows the HTML for the image src and alt attributes which we’re going to scrape. You can see from Figure 8 and the code block how we follow that path down.

Figure 8 - Inspected image element to show path

# get product image url and name

product_image = driver.find_elements_by_xpath("

//div[@class='h_quick-shop-style-tile']

//article[@class='h_hatch-style-tile']

//a[@class='h_hatch-element--a h_hatch-style-tile__link']

//div[@class='h_hatch-style-tile__image']

//div[@class='h_media h_media--3x4']

//span[@class='h_media__container']

//span[@class='h_media__image-container']

//img[@class='h_media__image h_media__image--loaded']")

Once our list ( product_image ) is complete, we again use the get_attribute() function in a loop to get the product image urls ( src )and the name of the products ( alt ).

product_image_url = []

product_image_alt = []



for image in product_image:

product_image_url.append(image.get_attribute("src"))

product_image_alt.append(image.get_attribute("alt"))

The alt tags match the product names from above exactly so thats not needed information. Maybe, in a more complex project you could use this to join data.



[‘

‘https://m.media-amazon.com/images/I/814o2ReEo5L.AC_SX600.jpg',

‘ # image urls[‘ https://m.media-amazon.com/images/I/717EZ8bvbsL.AC_SX1200.jpg' https://m.media-amazon.com/images/I/711GstLp0gL.AC_SX600.jpg' ,...] # image alt tags

['Jumpsuit w/ Ruffle Detail', 'Kassie', 'Brave Micro - Dwell',...]

Final Shopping List

Now that we have several lists, we can actually just join this data together easily. Since all the product information was scraped in the same order and page it appends easily in the correct order. I’ve added all those additional steps in the Git repo, but in short I create a pandas dataframe, clean up columns, and add columns that I thought would be good for quick sorting and filtering of a shopping list.

You can see below a snap shot of the final CSV that I “prettied up” just a bit. We have brand, product name, product url, image url, a flag if its on sale, sale price, MSRP price, and the sale percent off. You’re now ready to take this shopping list and filter / sort however you desire to get your holiday shopping done!