Package: pytest-postgresql
pytest-postgresqlWhat is this?This is a pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database.
It allows you to specify fixtures for PostgreSQL process and client.How to useWarningTested on PostgreSQL versions >= 10. See tests for more details.Install with:pipinstallpytest-postgresqlYou will also need to installpsycopg. Seeits installation instructions.
Note that this plugin requirespsycopgversion 3. It is possible to simultaneously install version 3
and version 2 for libraries that require the latter (seethose instructions).Plugin contains three fixtures:postgresql- it’s a client fixture that has functional scope.
After each test it ends all leftover connections, and drops test database
from PostgreSQL ensuring repeatability.
This fixture returns already connected psycopg connection.postgresql_proc- session scoped fixture, that starts PostgreSQL instance
at it’s first use and stops at the end of the tests.postgresql_noproc- a noprocess fixture, that’s connecting to already
running postgresql instance.
For example on dockerized test environments, or CI providing postgresql servicesSimply include one of these fixtures into your tests fixture list.You can also create additional postgresql client and process fixtures if you’d need to:frompytest_postgresqlimportfactoriespostgresql_my_proc=factories.postgresql_proc(port=None,unixsocketdir='/var/run')postgresql_my=factories.postgresql('postgresql_my_proc')NoteEach PostgreSQL process fixture can be configured in a different way than the others through the fixture factory arguments.Sample testdeftest_example_postgres(postgresql):"""Check main postgresql fixture."""cur=postgresql.cursor()cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")postgresql.commit()cur.close()If you want the database fixture to be automatically populated with your schema there are two ways:client fixture specificprocess fixture specificBoth are accepting same set of possible loaders:sql file pathloading function import path (string)actual loading functionThat function will receivehost,port,user,dbnameandpasswordkwargs and will have to perform
connection to the database inside. However, you’ll be able to run SQL files or even trigger programmatically database
migrations you have.Client specific loads the database each testpostgresql_my_with_schema=factories.postgresql('postgresql_my_proc',load=["schemafile.sql","otherschema.sql","import.path.to.function","import.path.to:otherfunction",load_this])WarningThis way, the database will still be dropped each time.The process fixture performs the load once per test session, and loads the data into the template database.
Client fixture then creates test database out of the template database each test, which significantly speeds up the tests.postgresql_my_proc=factories.postgresql_proc(load=["schemafile.sql","otherschema.sql","import.path.to.function","import.path.to:otherfunction",load_this])pytest--postgresql-populate-template=path.to.loading_function--postgresql-populate-template=path.to.other:loading_function--postgresql-populate-template=path/to/file.sqlThe loading_function from example will receive , and have to commit that.Connecting to already existing postgresql databaseSome projects are using already running postgresql servers (ie on docker instances).
In order to connect to them, one would be using thepostgresql_noprocfixture.postgresql_external=factories.postgresql('postgresql_noproc')By default thepostgresql_noprocfixture would connect to postgresql instance using5432port. Standard configuration options apply to it.These are the configuration options that are working on all levels with thepostgresql_noprocfixture:ConfigurationYou can define your settings in three ways, it’s fixture factory argument, command line option and pytest.ini configuration option.
You can pick which you prefer, but remember that these settings are handled in the following order:Fixture factory argumentCommand line optionConfiguration option in your pytest.ini fileConfiguration optionsPostgreSQL optionFixture factory argumentCommand line optionpytest.ini optionNoop process fixtureDefaultPath to executableexecutable–postgresql-execpostgresql_exec/usr/lib/postgresql/13/bin/pg_ctlhosthost–postgresql-hostpostgresql_hostyes127.0.0.1portport–postgresql-portpostgresql_portyes (5432)randompostgresql useruser–postgresql-userpostgresql_useryespostgrespasswordpassword–postgresql-passwordpostgresql_passwordyesStarting parameters (extra pg_ctl arguments)startparams–postgresql-startparamspostgresql_startparams-wPostgres exe extra arguments (passed via pg_ctl’s -o argument)postgres_options–postgresql-postgres-optionspostgresql_postgres_optionsLocation for unixsocketsunixsocket–postgresql-unixsocketdirpostgresql_unixsocketdir$TMPDIRDatabase name which will be created by the fixturesdbname–postgresql-dbnamepostgresql_dbnameyes, however with xdist an index is being added to name, resulting in test0, test1 for each worker.testDefault Schema either in sql files or import path to function that will load it (list of values for each)load–postgresql-loadpostgresql_loadyesPostgreSQL connection optionsoptions–postgresql-optionspostgresql_optionsyesExample usage:pass it as an argument in your own fixturepostgresql_proc=factories.postgresql_proc(port=8888)use--postgresql-portcommand line option when you run your testspy.testtests--postgresql-port=8888specify your port aspostgresql_portin yourpytest.inifile.To do so, put a line like the following under the[pytest]section of yourpytest.ini:[pytest]postgresql_port=8888ExamplesPopulating database for testsWith SQLAlchemyThis example shows how to populate database and create an SQLAlchemy’s ORM connection:Sample below is simplified session fixture frompyramid_fullauthtests:fromsqlalchemyimportcreate_enginefromsqlalchemy.ormimportscoped_session,sessionmakerfromsqlalchemy.poolimportNullPoolfromzope.sqlalchemyimportregister@pytest.fixturedefdb_session(postgresql):"""Session for SQLAlchemy."""frompyramid_fullauth.modelsimportBaseconnection=f'postgresql+psycopg2://{postgresql.info.user}:@{postgresql.info.host}:{postgresql.info.port}/{postgresql.info.dbname}'engine=create_engine(connection,echo=False,poolclass=NullPool)pyramid_basemodel.Session=scoped_session(sessionmaker(extension=ZopeTransactionExtension()))pyramid_basemodel.bind_engine(engine,pyramid_basemodel.Session,should_create=True,should_drop=True)yieldpyramid_basemodel.Sessiontransaction.commit()Base.metadata.drop_all(engine)@pytest.fixturedefuser(db_session):"""Test user fixture."""frompyramid_fullauth.modelsimportUserfromtests.toolsimportDEFAULT_USERnew_user=User(**DEFAULT_USER)db_session.add(new_user)transaction.commit()returnnew_userdeftest_remove_last_admin(db_session,user):"""
    Sample test checks internal login, but shows usage in tests with SQLAlchemy
    """user=db_session.merge(user)user.is_admin=Truetransaction.commit()user=db_session.merge(user)withpytest.raises(AttributeError):user.is_admin=FalseNoteSee the original code atpyramid_fullauth’s conftest file.
Depending on your needs, that in between code can fire alembic migrations in case of sqlalchemy stack or any other codeMaintaining database state outside of the fixturesIt is possible and appears it’s used in other libraries for tests,
to maintain database state with the use of thepytest-postgresqldatabase
managing functionality:For this import DatabaseJanitor and use its init and drop methods:importpytestfrompytest_postgresql.janitorimportDatabaseJanitor@pytest.fixturedefdatabase(postgresql_proc):# variable definitionjanitor=DatabaseJanitor(postgresql_proc.user,postgresql_proc.host,postgresql_proc.port,"my_test_database",postgresql_proc.version,password="secret_password",)janitor.init()yieldpsycopg2.connect(dbname="my_test_database",user=postgresql_proc.user,password="secret_password",host=postgresql_proc.host,port=postgresql_proc.port,)janitor.drop()or use it as a context manager:importpytestfrompytest_postgresql.janitorimportDatabaseJanitor@pytest.fixturedefdatabase(postgresql_proc):# variable definitionwithDatabaseJanitor(postgresql_proc.user,postgresql_proc.host,postgresql_proc.port,"my_test_database",postgresql_proc.version,password="secret_password",):yieldpsycopg2.connect(dbname="my_test_database",user=postgresql_proc.user,password="secret_password",host=postgresql_proc.host,port=postgresql_proc.port,)NoteDatabaseJanitor manages the state of the database, but you’ll have to create
connection to use in test code yourself.You can optionally pass in a recognized postgresql ISOLATION_LEVEL for
additional control.NoteSee DatabaseJanitor usage in python’s warehouse test codehttps://github.com/pypa/warehouse/blob/5d15bfe/tests/conftest.py#L127Connecting to Postgresql (in a docker)To connect to a docker run postgresql and run test on it, use noproc fixtures.dockerrun--namesome-postgres-ePOSTGRES_PASSWORD=mysecretpassword-dpostgresThis will start postgresql in a docker container, however using a postgresql installed locally is not much different.In tests, make sure that all your tests are usingpostgresql_noprocfixture like that:frompytest_postgresqlimportfactoriespostgresql_in_docker=factories.postgresql_noproc()postgresql=factories.postgresql("postgresql_in_docker",dbname="test")deftest_postgres_docker(postgresql):"""Run test."""cur=postgresql.cursor()cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")postgresql.commit()cur.close()And run tests:pytest--postgresql-host=172.17.0.2--postgresql-password=mysecretpasswordBasic database state for all testsIf you’ve got several tests that require common initialisation, you can to define aloadand pass it to
your custom postgresql process fixture:importpytest_postgresql.factoriesdefload_database(**kwargs):db_connection:connection=psycopg2.connect(**kwargs)withdb_connection.cursor()ascur:cur.execute("CREATE TABLE stories (id serial PRIMARY KEY, name varchar);")cur.execute("INSERT INTO stories (name) VALUES""('Silmarillion'), ('Star Wars'), ('The Expanse'), ('Battlestar Galactica')")db_connection.commit()postgresql_proc=factories.postgresql_proc(load=[load_database],)postgresql=factories.postgresql("postgresql_proc",)The way this will work is that the process fixture will populate template database,
which in turn will be used automatically by client fixture to create a test database from scratch.
Fast, clean and no dangling transactions, that could be accidentally rolled back.Same approach will work with noproces fixture, while connecting to already running postgresql instance whether
it’ll be on a docker machine or running remotely or locally.Using SQLAlchemy to initialise basic database stateHow to use SQLAlchemy for common initalisation:defload_database(**kwargs):connection=f"postgresql+psycopg2://{kwargs['user']}:@{kwargs['host']}:{kwargs['port']}/{kwargs['dbname']}"engine=create_engine(connection)Base.metadata.create_all(engine)session=scoped_session(sessionmaker(bind=engine))# add things to sessionsession.commit()postgresql_proc=factories.postgresql_proc(load=[load_database])postgresql=factories.postgresql('postgresql_proc')# still need to check if this is actually needed or not@pytest.fixturedefdbsession(postgresql):connection=f'postgresql+psycopg2://{postgresql.info.user}:@{postgresql.info.host}:{postgresql.info.port}/{postgresql.info.dbname}'engine=create_engine(connection)session=scoped_session(sessionmaker(bind=engine))yieldsession# 'Base.metadata.drop_all(engine)' here specifically does not work. It is also not needed. If you leave out the session.close()# all the tests still run, but you get a warning/error at the end of the tests.session.close()ReleaseInstall pipenv and –dev dependencies first, Then run:pipenvruntbump[NEW_VERSION]