HOW TO: Save browser sessions in Selenium using Redis and Flask
By: Abir Tawfique
Tags: Selenium, Automation, Flask, Python, Redis

While working on a recent full-stack Selenium project, I discovered that there are no tutorials for creating a Selenium web-driver instance that can be reused for tasks. Using an API, we’re going to be able to write code, and test code without closing down the web instance unless you want to make changes to the web-driver settings. Furthermore, this tutorial will show how you can use Redis to queue tasks in Selenium, which allows the ability to send off hundreds of API hits to Flask, and the tasks will be done in order.

Let’s start by familiarizing the libraries and framework we will be using today. For those who don’t know, Selenium is a framework that simply opens a browser such as Chrome or Firefox and can be controlled using code.

Next, Redis (AKA python-rq) is an asynchronous task queuing framework. Redis workers receive tasks, put them into a queue, and the task will be processed when the worker isn’t busy. It applies the basic understanding of first in, first out but can be customized to handle certain tasks with a priority, which is not something we will be getting into in this article but you can read more about it here.

And finally, Flask is a python library allowing users to build web applications rapidly. In this tutorial, we’ll be using Flask as a light back-end server to take HTML requests and send them for processing in the Redis queue.

So, why would you want to do this?

You can save login sessions

You can save Captcha or ReCaptcha session

Selenium apps with many purposes can use one web-driver, saving system resources

You can use Selenium as an API, doing tasks with the hit of an API end-point

For those using Selenium as a test suite, you can update your codebase with Flask reloader turned on, and those changes will reflect the next time you send an HTTP request. All without restarting Selenium, Flask, or Redis.

Saving and reusing sessions is a big topic in the Selenium community. The vast majority of solutions on the web recommend saving HTML/JavaScript cookies and upon creating a new web driver instance, writing those cookies into the driver. However, this is not a bulletproof solution and will not work for all websites. This solution works vastly better as the webdriver instance is never closed, preserving the cookies and data natively.

Note: There is a slightly more effective method at saving ChromeDriver sessions using Chrome Profile. There are some good resources out there on how to do this, but I plan to create a detailed article in the future regarding this topic.

The Workflow

The workflow between Selenium, Flask, and Redis.

The workflow is quite simple. HTTP Requests are sent to the backend API, which is Flask in this end, and Flask enqueues the job to the Redis queue who will perform jobs as they come.

For example, say you want to start the Selenium instance. You send a GET HTTP request to Flask with the following sample request:

http://127.0.0.1:5000/run

Flask will receive this, send the respective function to Redis who will then start selenium whenever it is not busy.

Let’s start coding

For the purposes of this tutorial, we will be making a Selenium instance that has the ability to go to https://github.com/ and https://stackoverflow.com/. These are very, very basic functions but way more complex functions can be implemented and anything you can normally do with Selenium, such as test websites and scrape data, you can do with this while saving your current session! For the full source code, please visit the SeleniumRedis repo. The repo has clear instructions on how to run the example that we will be coding below on your local machine.

Install Redis

First, start by installing Redis (https://redis.io/topics/quickstart). Once you are able to run redis-cli ping in terminal and get PONG returned, you are good to go. Since the Redis server must be online, let’s start by running the following command in a terminal window:

redis-server

Write Selenium code

Now, let’s create a new file called tester.py and put the following code in there:

import time # needed to sleep driver as example

from selenium import webdriver

from webdriver_manager.chrome import ChromeDriverManager

from selenium.webdriver.chrome.options import Options

from selenium.webdriver.remote.webdriver import WebDriver

from selenium.webdriver.remote.command import Command

import http.client

import socket





def run():

chromeOptions = Options()

global driver

driver = webdriver.Chrome(ChromeDriverManager().install(), options=chromeOptions)

driver.implicitly_wait(15)

driver.get('https://google.com/')

writeSession(driver)

print('Finished Initiating Chrome')



def writeSession(driver):

url = driver.command_executor._url

session_id = driver.session_id

f = open("sessioninfo.txt", "w")

f.write(f"{url}

")

f.write(f"{session_id}")

f.close()

print('Wrote webdriver session details')



class Scraper():

def __init__(self):

self.driver = attachToSession()



def getGithub(self):

try:

driver = self.driver

driver.get('https://github.com/')

print('Sleeping..')

time.sleep(10)

print('Sleep done..')

return 200

except Exception as e:

print(e)

raise e



def getStackOverflow(self):

try:

driver = self.driver

driver.get('https://stackoverflow.com')

print('Sleeping..')

time.sleep(10)

print('Sleep done..')

return 200

except Exception as e:

print(e)

raise e







def attachToSession():

# Code Reference : https://stackoverflow.com/a/48194907/11217153

# The stackover flow answer was adapted.

f = open("sessioninfo.txt", "r")

lines = f.readlines()

url = lines[0]

session_id = lines[1]

session_id.strip()

original_execute = WebDriver.execute

def new_command_execute(self, command, params=None):

if command == "newSession":

# Mock the response

return {'success': 0, 'value': None, 'sessionId': session_id}

else:

return original_execute(self, command, params)

# Patch the function before creating the driver object

WebDriver.execute = new_command_execute

driver = webdriver.Remote(command_executor=url, desired_capabilities={})

driver.session_id = session_id

# Replace the patched function with original function

WebDriver.execute = original_execute

return driver

The Scraper class includes two main functions, getGithub() and getStackOverflow() . Each function navigates to the respective website, and will sleep for ten seconds. The run() function creates the web-driver instance using ChromeDriverManager and writes the session details to a file named sessioninfo.txt

In normal Selenium practice, you would run getGithub() or getStackOverflow() at the end of the file, and it would do just that, but then the session could never be used again.

In contrary, now, the writeSession() and attachToSession() functions are the ones to focus. After countless hours of research, testing, these two functions are able to work together to write the details of the latest web-driver session and attach to it remotely so the session can be reused.

Create Flask Backend

Create a file called app.py that will house the infrastructure for a basic, lightweight API using Flask. Put the following code into that file:

from flask import Flask, request, jsonify

import requests

import os

import tester

from tester import Scraper

from rq import Queue

from rq.job import Job

from red import conn





# Initialize App

app = Flask(__name__)

basedir = os.path.abspath(os.path.dirname(__file__))

q = Queue(connection=conn)



# Run Downloader

@app.route('/run')

def run():

job = q.enqueue(tester.run)

print(job.get_id())

return 'OK'



@app.route('/run2')

def run_stack():

job = q.enqueue(Scraper().getStackOverflow)

print(job.get_id())

return 'OK'



@app.route('/run3')

def run_TA():

job = q.enqueue(Scraper().getTA)

print(job.get_id())

return 'OK'



if __name__ == '__main__':

app.run(debug=True, use_reloader=True)

The flask backend is actually quite simple. You define routes using @app.route() and we import and connect to a redis-queue, which we will define next. On a separate terminal tab, start this file by running python app.py

Setup Redis Worker

The Redis worker, as mentioned before, will queue and process tasks one by one. Put the following code into a new file, called red.py (short for Redis)

import os

import redis

from rq import Worker, Queue, Connection



listen = ['default']



redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')



conn = redis.from_url(redis_url)



if __name__ == '__main__':

with Connection(conn):

worker = Worker(list(map(Queue, listen)))

worker.work()

As you can see, Redis is actually quite easy to set up, and once familiarized with Redis, this queue can be implemented with just about any application. In a new terminal tab apart from the one running redis-server and the one running python app.py , let’s run this one by calling python red.py

Let’s start testing!

Now that all of our services are online and listening for requests, we can use either our web browser to hit API endpoints or use a tool like Postman.

Let’s start by running the Selenium instance!

Once you run this in either a web browser or an HTTP request tool, in a few seconds you will see that an automated Chrome instance has started!

Now, we can start navigating to webpages by sending the following HTTP requests:



- for StackOverflow http://127.0.0.1:5000/run2 - for Github http://127.0.0.1:5000/run3 - for StackOverflow

As soon as you send a request, the web driver will attach to the already open session and navigate to the webpage. And again, these functions can be customized to do any number of operations using Selenium.

Detailed Use-Cases

Now that we’ve built the program and you were able to see what the basic capabilities of it are, here are a few great ways to utilize this for Selenium projects:

Building a Scraper? Testing various XPATHs and scraping methods can be a strenuous process. With this method, Selenium doesn’t have to be restarted every time you are testing a new XPATH or attempting to find the most reliable and efficient way to grab data. On top of this, you can send infinite requests to Selenium functions which allows you to stress test individual functions to ensure reliability and add code for edge-cases and test quicker than ever before. In addition, if you are manually solving Captchas for Selenium, your web-driver instance can be reused infinitely, saving countless hours in solving Captchas. Building a Selenium test suite? Stress-test your websites at lightning-fast speed using this method. For example: if one of the features of the test suite tests all forms by entering data and submitting, changes can be made to the form real-time and Selenium does not have to be restarted to test those changes made to the form. This also works the other way around as new Selenium test features you code can be tested instantaneously. Building a cron task with Selenium? Selenium can be easily deployed to a DigitalOcean $5 Ubuntu server, and via this method, the web-driver never needs to be restarted so login tokens, captcha tokens, and cache can be preserved for easy fulfillment of cron tasks. I recommend scheduling Selenium tasks with Schedule.

In conclusion

This is a revolutionary way to use Selenium for testing, automation, scraping and more. Code can be changed and tested on the fly without restarting Selenium, crawlers and scrapers can test data-gathering methods and XPATHs in lightning-fast speeds and there are many more uses for reusable Selenium sessions.

I plan to turn this code into a python-package in the near future, and I will create a second article discussing that, along with updating this article for ease-of-access. And again, all of this code can be pulled down from the GitHub repo I’ve created and started within a minute or two.