How to Run JavaScript in Python for Web Scraping or Website Testing
By: Mahmud Ahsan
Tags: Selenium, JavaScript, Python, Chrome, Scraping

When we develop web applications sometimes we need to test the UX. Most of the time we do it manually. In this tutorial, I will show you how to use Selenium library, JavaScript, and Python to solve the problem. Using the technique you can also scrap data from a webpage.

We will be doing the following steps:

Installing Selenium library in Mac and Windows Install Headless Google Chrome driver in Mac and Windows Write and run a Python script to load github.com website in a headless browser Using Python selenium library to run JavaScript code Scraping github.com webpage data after it loaded Filling the github.com search form and submit by code Finally, taking the invisible browsers screenshot programmatically

Setup

We need pipenv to install the Selenium library for this project. If you don’t know how to install Pipenv then please check my other tutorial

1. Installing Selenium library in Mac and Windows

First, in terminal go to a directory. In my case I am in this directory:

/Users/mahmud/Desktop/demo/sel1

Now open the Terminal in Mac or PowerShell in Windows and run the following commands:

pipenv install selenium

It will create 2 files, Pipfile and Pipfile.lock

Now run the following command to activate the sel1 project’s virtualenv .

pipenv shell

2. Install Headless Google Chrome driver

To automate a web browser, we need to install a Google Chrome driver. Please visit the following website and download the latest released driver for your mac or windows or Linux operating system.

Now unzip the downloaded file and copy the chromedriver.exe file in our project directory sel1 .

3. Run the python script

Now in the sel1 directory, create a python script named chapter9.py or whatever and paste the following codes.

4. Run the program

In macOS terminal run the following command:

python3 chapter9.py

In windows, 10 power shell run the following command. Just use Python instead of Python3

python chapter9.py

After successfully run the program, you will get a png file named python-github.png

5. Python script analysis

It is a very simple script. At first, we import python selenium libraries in our script. Then we create a web driver object based on some options we provided also we mentioned the google chrome browser driver location via chrome_driver object.

Then by driver.get() method, we load github.com website.

In the #scrap info section, we scrap HTML h1 tag data and print it in the console. This is how we scrap via selenium and headless web driver.

In the #scrap info section, we scrap HTML h1 tag data and print it in the console. This is how we scrap via selenium and headless web driver.

# scrap info

h1_elem = driver.find_element_by_tag_name("h1")

print(h1_elem.text)

We see “Built for developers” is printed in the terminal.

Finally, we fill and submit the form by code. To select the search form in the webpage by javascript, we use Google Chrome Browser’s Inspect code option to check the form element name.

This is the code that automates search and submit the form:

# fill and submit form

elem = driver.find_element_by_name("q")

elem.clear()

elem.send_keys("python")

elem.send_keys(Keys.RETURN)

To take a screenshot of the final page we write the following code:

# screenshot capture

driver.get_screenshot_as_file("python-github.png")

driver.close()

Reference:

2. Selenium Python Docs: http://selenium-python.readthedocs.io/getting-started.html

3. Google Chrome Web Driver: https://sites.google.com/a/chromium.org/chromedriver/downloads

Check out my other web scraping tutorials: