How to Handle Common Selenium Issues Using Python (Part 2)
By: Amal Hasni
Tags: Python, Web Scraping, Automation, Selenium, Programming

How to Deal With Dynamic Web Elements

Copy-pasting the XPath of an element from the browser’s dev tools is surely the easiest way to select elements. However, this might not work in cases where the element you are using has a selector that changes dynamically when you reload the page.

Demonstration of dynamic XPath in Firefox’s dev tools (Image source: Kayak.com)

There are multiple solutions to deal with this. Let’s explore them using a simple example:

In our example, we want to select the span with the dynamic id ( 'A{some_number}-my-span-element' ). Here are different solutions to this:

Use absolute XPath

In the example above, we can select the span by using its absolute XPath like this:

absolute_xpath = '/html/body/div[2]/span'

Some tips you might need:

You can select the nth child of a certain element using the brackets [n] as we did in the example.

as we did in the example. You can omit to mention the tag name of your element by using the asterisk (*) like this: /html/body/*[2]/span

If you’re too lazy to write the absolute XPath:

1. You can find them in the dev tools of newer versions of Chrome or Microsoft Edge. You just need to click on an element and copy the full XPath, like so.

2. If you’re on an older version or using Firefox like me, you can use your browser’s console to select it. All you need to do is:

Click on F12 to open the dev tools.

Select the Console ( and write “allow pasting” if you are using Firefox.)

Copy and paste this JavaScript function (adapted from Stack Overflow)

Select your element using the currently displayed dynamic id:

element = $("#A154897-my-span-element") getPathTo(element)

Your absolute XPath should appear in the console.

Use relative XPath

Using absolute_xpath works most of the time, but it can be tricky to write since you have to follow the full path to the root. Luckily, a more convenient solution exists, and it’s to use a relative XPath.

In our example, we can see that the div tag has a static ID. We can, therefore, use an XPath that is relative to that element:

relative_xpath = '//*[@id="my-div"]/span'

The character “/” is used to suggest a direct child (descendant) element, while the “//” is used to select deeper descendants (direct or undirect children).

In the example, we can see that selecting an element by id amounts to filtering the elements by using the brackets syntax. Inside the brackets, we specify that the id (preceded by “@” to imply it is an attribute) is equal to a certain value (“my-div”).

Using XPath functions

As we’ve seen in the relative path, we can use filters like *[@id="my-div"] to select our elements. We are, however, not limited to the standard operators (=, !=, <, <=, >, >=). We can use functions to manipulate strings such as contains . (You can find the full list of functions in the documentation.)

Knowing this, we can directly select our element using the following XPath:

element_xpath = '//span[contains(@id, "my-span-element")]'

We can also use the and and not keywords to combine multiple functions (filters) to select our element: