Faking Redis/PostgreSQL connections while testing with Pytest in Python
By: Szymon Palucha
Tags: Postgresql, Redis, Testing, Pytest, Python

PostgreSQL example

Faking a PostgreSQL connection turned out to be a bit more challenging as expected. Firstly, a little reminder of how to connect and use psql within Python. The package I use for that is PyGreSQL. To make a connection we could do the following

import pgdb CONNECTION = pgdb.connect(user=info['user'], host=info['host'], database=info['database'], port=info['port'])

where info is a dictionary to be specified, which could be loaded from a config file. (Note I think there is another pgdb Python package and so you need to install the correct one otherwise you might get an error like “pgdb has no connect method”)

To then query the database we could do something like

cursor = CONNECTION.cursor()

records = cursor.execute("""SELECT age FROM students""").fetchall()

cursor.close()

Now to make a fake psql connection I used the testing.postgresql package (https://pypi.org/project/testing.postgresql/). Note that this package requires a local PostgresSQL database installed (as opposed to the birdisle package which I believe doesn’t require the Redis database). The package provides a fake database/ user/ host/ port which can be used to make a fake connection with the PyGreSQL package as follows

import pgdb

import testing.postgresql psql = testing.postgresql.Postgresql()

info = psql.dsn()

CONNECTION = pgdb.connect(user=info['user'], host=info['host'], database=info['database'], port=info['port'])

As an example let’s say we want to test a function which sums the ages for all students from the “students” relation (table) in psql. The function could look like

def sum_ages():

records = query_database(operation="""SELECT age FROM students""")

ages = [r.age for r in records]

return sum(ages)

where query_database() is another function which does all the cursor stuff shown previously.

To create the students relation in the fake psql database the Pandas package turns out to be very useful. It has a handy to_sql() method which converts a Pandas dataframe into a psql relation. It requires an engine object to be passed in which tells Pandas which psql database to insert into. To get this object we need to use yet another package called sqlalchemy. It makes sense to save the data needed to make the students table as a csv file so it can be easily loaded into a Pandas dataframe with the read_csv() method. Putting this all together we need to do something like

import testing.postgresql

import pandas as pd

from sqlalchemy import create_engine psql = testing.postgresql.Postgresql()

engine = create_engine(psql.url()) students_df = pd.read_csv(file)

students_df.to_sql('students', engine, if_exists='replace')

The if_exists = “replace" tells Pandas to replace an existing relation with the same name. There are other useful things that can be passed in to the to_sql() method like dtype which specifies the data type for the columns.

When dealing with psql connections we need to rollback the connection after each test case (rollback is used to undo transactions and it seems easier than trying to clear the database — I also ran into some problems without doing this) and close the connection after the last test case. As a result we also need to keep track of the total test cases and the test case ids. Below is the full test file

Full test file that uses a fake PostgreSQL connection