Top 25 Selenium Functions That Will Make You Pro | Towards Data Science

Web Scraping is a term for getting the data from webpages and making it so that you do not have to move a finger!
Get the data and then process it any way you want.
That is why today I want to show you some of the top functions that Selenium, which is a library for Web Scraping, has to offer.
I have written articles about Selenium and Web Scraping before, so before you begin with these, I would recommend you read this article “Everything About Web Scraping”, because of the setup process. And if you are already more advanced with Web Scraping, try my advanced scripts like “How to Save Money with Python” and “How to Make an Analysis Tool with Python”.
Let’s just jump right into it!
I have written about this one before, but get function is crucial and you can not do anything without it.
The get command launches a new browser and opens the given URL in your Webdriver. It simply takes the string as your specified URL and opens it for testing purposes.
If you are using Selenium IDE, it is similar to open command.
Example:
driver.get(“https://google.com");
The ‘driver’ is your Webdriver on which you are going to perform all actions and it looks like this after executing the command above:
This function is crucial when you want to access elements on the page. Let’s say we want to access the “Google search” button in order to perform a search.
There are many ways to access elements, but my preferred one is to find XPath of the element. XPath is the element’s ultimate location on the web page.
By clicking F12 you will inspect the page and get the background information about the page you are at.
By clicking the selection tool, you will be able to select elements.
After finding the button, click on the right at the blue marked section and Copy element’s “Full Xpath”.
self.driver.find_element_by_xpath('/html/body/div/div[4]/form/div[2]/div[1]/div[3]/center/input[1]')
That is the full command in order to find the specific element. I prefer full XPath over the regular XPath because regular one can be changed if the element in a new session changes and then next time you perform your script it does not work.
Overview of other find_element functions. (There is also find_elements ones)
Send_keys function is used for typing the text into a field that you selected by using the find_element function.
Let’s say we want to type “plate” into google, that is why we use send_keys function.
google_tray = self.driver.find_element_by_xpath('/html/body/div/div[4]/form/div[2]/div[1]/div[1]/div/div[2]/input')google_tray.send_keys("plate")google_search = self.driver.find_element_by_xpath('/html/body/div/div[4]/form/div[2]/div[1]/div[3]/center/input[1]')google_search.click()
I saved the elements in their respective variables and then performed functions on them for more clarity.
At the end you end up with this:
This function uses the found element and performs a “click” on it. Simple stuff.
This is a code example for clicking the “Google Search” button.
google_search = self.driver.find_element_by_xpath('/html/body/div/div[4]/form/div[2]/div[1]/div[3]/center/input[1]')google_search.click()
This command is used for retrieving the Class object that represents the runtime class of this object. That means you perform this function to get the class of a certain element.
driver.getClass()
When you are already on a certain page with this command you can retrieve the URL you are at.
There are no parameters required for this command and it returns the string.
driver.current_url()
With this command, you will be able to retrieve the Source of the page.
There are no parameters required for this command and it returns the string.
driver.getPageSource()
This command can be combined with other commands like contains() to check if that string exists under that filter.
driver.getPageSource().contains("Example");
In this case, this will be a boolean value so either True or False.
Retrieve the Title of the website you are currently at.
There are no parameters required for this command and it returns the string. If the title is not found, null will be returned.
driver.getTitle()
With this command, you will get the text part of the Web Element.
There are no parameters required for this command and it returns the string.
driver.findElement(By.id("Price")).getText()
With this command, you will get the attribute of the Web Element.
We pass a single string parameter the is supposed to be an attribute that we want to know and returns a string value.
driver.findElement(By.id("Price")).getAttribute("val");
This command is used when we have multiple windows to handle.
The command helps us switch to the newly opened window and performs actions on the new window.
You can also switch back to the previous window if he/she desires.
String winHandleBefore = driver.getWindowHandle()driver.switchTo().window(winHandleBefore)
This command is similar to getWindowHandles(), but the basic difference is that we are dealing with multiple windows here. So more than two.
driver.getWindowHandles()
This command is used for finding elements on the page, based on the hyperlinks on the page and of the text that it consists of.
Let’s say we want to find the first link out of the Google search.
driver.findElement(By.linkText(“Car - Wikipedia”))
There is also another option for this link searching:
Instead of typing the whole string you are looking for, you can also search for the partial or substrings on the page.
driver.findElement(By.partialLinkText(“BMW”))
This command will find the first link out of Top Stories on Google.
After you input the text into the fields for a form, you can use the submit function to click that button and finish the form.
driver.findElement(By.<em>id</em>("submit")).submit();
This method closes the current window the user is working on.
The command does not require any parameter and it does not return any value.
driver.close()
It might seem very similar to close command, but the main difference is that it closes all windows that the user is keeping open at the moment.
The command does not require any parameter and it does not return any value.
driver.quit()
When you want to check whether the element is enabled in Web Driver.
First, you have to find the element and then check it.
It has a return value of boolean so either True or False.
driver.findElement(By.id("Price")).isEnabled()
With this function, you can get the size of a selected Web Element.
driver.findElements(By.id("Price")).size()
It returns the size of the element and output looks like this:
{'width': 77, 'height': 22}
This function is used to pausing the thread of execution for a certain amount of seconds.
For example, you type in the login credentials and before the next page loads, there is a time delay. That is why we use sleep to overcome those loading times.
This function is something useful in general and it is not from Selenium, but there are functions built-in for Selenium.
sleep(1)
This will pause the execution for 1 second.
With sleep function, you have to basically guess the loading time that you want to overcome but with pageLoadTimeout() you can set it to be specific.
driver.manage().timeouts().pageLoadTimeout(3, SECONDS);
This command will wait for 3 seconds for the page to load.
This is one more iteration of those sleeping/waiting functions that will help you achieve that perfect multithreading.
In order to avoid errors and Exceptions being thrown, we add a command to for a specific amount of time before locating the element on the page.
driver.manage().timeouts().implicitlyWait(3, TimeUnit.SECONDS);
Wait 3 seconds before executing the next command.
These are being combined in order to switch between URLs and go back and forward as you please.
So you want to open another page, but instead of using get(), you want to use navigate here and easily manipulate between those pages.
driver.navigate().to("https://www.facebook.com");driver.navigate().back();driver.navigate().forward();
As mentioned before, this is not my first time writing about Selenium and Web Scraping in general. There are many more functions I would love to cover and many more to come. I hope you liked this tutorial and in order to keep up, follow me for more!
Thanks for reading!