XPath For Python | The RegEx of Web  | Towards Data Science

XML path language (XPath) is a massively underappreciated tool in the world of web scraping and automation. Imagine RegEx, but for webpages — that is XPath.
Every element of a webpage is organized by the Document Object Model (DOM). The DOM is a tree-like structure, where each element represents a node, with paths to parent and child nodes.
XPath offers us a language for quickly traversing across this tree. And, like RegEx, we can add logic to our node selection to make our queries more powerful.
In this article, we will cover:
> XPath Essentials  - Testing Our Queries  - The Root  - Paths in XPath> Navigating the Tree  - Node Indexing  - Extracting XPaths from the Browser> XPath Logic> Example with Python
First, before we do anything else, we need to understand how we can test our XPath strings. Fortunately, we can do that right here in the web browser.
I’ll be using Chrome throughout this article, but the procedure is very similar across all modern browsers.
On our webpage, we open Developer Tools — either by clicking Fn+12 on Windows or opening it from your browser options menu (see for Chrome above).
Next, we click ctrl+F to open the search bar within our Elements window. Here we can search by string, selector, or XPath.
This method is the easiest way to test our XPath queries quickly.
If our query matches something, that element will be highlighted yellow. Where our query matches multiple elements, we can cycle through them using the arrows to the right of the search bar!
Let’s start with the very beginning of our query. In the case of //div — what does the // mean?
Every XPath query begins at the root of our XML tree — the very top element. For HTML documents, this is the <html> tag.
Now, if we write html//div we are saying “look for any descendant node of html that is a div”.
The result is that both html/div and html/body/div/div/article/div/div will be found with our query. This is because in both cases, we see a div which is a descendant of html.
Because our XPath query always begins at the root (html), we don’t need to write html//div. Instead, we write //div.
The // example we just used is called a path expression. There’s a few of these, and they’re super useful:
// — match any descendent node
/ — match only the child nodes (nodes directly following another):
. — matches the currently active node (more on this later)
.. — matches the parent node of the currently active node
@ — select an attribute of the current node (href for example):
With a combination of these path expressions, we can traverse across the XML tree with ease.
For example, back to our example HTML section — we can select the a tag by finding the span tag with attribute class="u-textScreenReader", and traversing up the tree to its parent node, like so:
Every node in our DOM is numbered. If we have a list ul which contains five li items, we can access each of those items specifically by indexing from 1 to 5:
<ul>    <li>London</li>    <li>Miami</li>    <li>New Dehli</li></li>
If we query //ul/li[1] we will return <li>London</li> — note that values are not zero-indexed. XPath indexing begins at 1.
An incredibly convenient feature in Chrome (and likely most modern browsers) is the ability to get the XPath of a node directly from the Elements window.
To do this, we right-click on the element we want the XPath for and click Copy > Copy XPath and paste the XPath for that element. Our span element returns:
//*[@id="_obv.shell._surface_1600536527994"]/div/div[1]/div[2] /div[1]/div[1]/a/span[1]
Alternatively, we can copy the full XPath, which provides the full path from the root to our element. For our span element this looks like:
/html/body/div[1]/div[2]/div/div[1]/div[2]/div[1]/div[1]/a/span[1]
Now, whether you want to call the following methods ‘advanced’ or not is debatable.
However, I justify this because, by using the previous Path Expressions alone, we can traverse the DOM very easily. In many cases, we won’t need anything more. But now and then, these ‘advanced’ methods can be incredibly useful.
There are many XPath functions that we won’t cover here, but a few of the most common ones are:
contains — searches for string A in string B, where contains(B, A):
not — we use this to negate parts of our query, like if we would like all span elements that don’t contain the class svgIcon-use:
boolean — the equal and opposite of not, if we would like all span elements that do contain an svg child node:
starts-with — similar to contains, but rather than containing string A, string B must start with string A.
ends-with — I’m sure you can figure this one out
These functions are just a scratch on the surface of XPath. A few items we haven’t even touched upon (I’ve added links to articles on them though):
More XPath axes
XPath operators
Selecting unknown nodes
Full list of XPath functions
Selenium is the best way to get familiar with XPath in Python (it is available for many other languages too). If you are not familiar with it, I wrote this article covering the setup and basics of the framework — it’s very easy to work with!
Once we have Selenium setup, we can select all elements of a webpage that satisfy an XPath query using the find_elements_by_xpath method.
We’ll try this out on webscraper.io. There, we can see that every item is contained in a div element with class="thumbnail". Here, we’ve highlighted the Acer Aspire product.
So, back to Selenium and XPath. First, we need to initialize our web driver and navigate to the Web Scraper training site:
from selenium import webdriverdriver = webdriver.Chrome('chromedriver.exe')driver.get('https://webscraper.io/test-sites/e-commerce/scroll')
Now, we need to select all div elements with class thumbnail:
shop_elems = driver.find_elements_by_xpath(    "//div[@class='thumbnail']")
If we print out the value of shop_elems we will return a list of WebElement objects:
Great, we have our shop container WebElements — now what? Well, we can treat these objects as individual XML trees.
We do this using the WebElement get_element method coupled with By.XPATH — which is a new import:
from selenium.webdriver.common.by import Byshop_elems[0].get_element(By.XPATH, <XPath Query Here>)
Here, we are setting the first instance of div[@class="thumbnail"] as active, and we can use . to select the currently active node in an XPath query.
We can use this new method to loop through every item container on the page and extract details for each one iteratively!
Let’s try extracting the item name for each of our WebElements in shop_elems.
Back in the browser, we can find item names in descendant <a> tags. As there are no other descendant <a> tags, we can select this element with just //a. This gives us //div[@class='thumbnail']//a.
In our code, the WebElements contained in shop_elems have set div[@class='thumbnail'] as their active node, which we select with .:
shop_elems[0].find_element(By.XPATH, ".//a")
To get the item name, we simply access the text value of our object. We can integrate that into a for-loop to pull every item name from our WebScraper.io e-commerce page:
And that’s it; we’ve extracted the item name of product on our e-commerce front-page with XPath and Selenium. We can couple this with more XPath queries to extract more information, like the price, rating, number of reviews, etc.
That’s it for this introduction to XPath and Selenium. We’ve covered:
DOM, nodes, and branches
Testing XPath queries — Fn+12, Ctrl+F
Navigating the tree
Pulling XPaths from the browser
Some more advanced XPath logic
As I said before, this just scratches the surface of XPath, and I would definitely recommend working on your own little web scraping/automation projects and learning more!
I hope you enjoyed the article. If you have any ideas or questions, let me know via Twitter or in the comments below! If you’d like more content like this, I post on YouTube too.
Thanks for reading!
*All images are by the author unless stated otherwise.