Package: django-plpy
READMEDjango utilities for Postgres PL/Python. WIPWhat is django-plpyPostgreSQL's PL/Python plugin allows you to write stored procedures in Python. Django-plpy provides utilities and commands for using python functions from your project within Django ORM and more.RequirementsDjango 2.2 or later, Python 3.6 or higher, Postgres 10 or higher.InstallationPL/Python, therefore django-plpy requires Postgres plugin for plpython3u. Most of the distributions provide it in their repositories, here is how you install it on Ubuntu:apt-get -y install postgresql-plpython3-10Mind the PostgreSQL version at the end.Install django-plpy with pippip install django-plpyAdd it to INSTALLED_APPSINSTALLED_APPS=[...,"django_plpy",...,]Migrate./manage.py migrateCheck if your local python environment is compatible with your Postgres python interpreter../manage.pycheckenvDjango-plpy is ready to be used.FeaturesInstalling of python functionsThe main workflow for bringing python functions to the database is to decorate them with@plpythonand call manage.py commandsyncfunctionsto install them. Full annotation is needed for the proper arguments mapping to the corresponding Postgres type function signature.Imagine a function like this:fromdjango_plpy.installerimportplfunction@plfunctiondefpl_max(a:int,b:int)->int:ifa>b:returnareturnbFinding a maximum of two values. @plfunction decorator registers it for installation, if any function with that name already exists it will be overwritten. Callsyncfunctionscommand to install it into the database:./manage.py syncfynctionsPython functions in SQL queriesfromdjango.dbimportconnectionwithconnection.cursor()ascursor:cursor.execute("select pl_max(10, 20)")row=cursor.fetchone()assertrow[0]==20Python functions in annotationsfromdjango.db.modelsimportF,Funcfromtests.books.modelsimportBookBook.objects.annotate(max_value=Func(F("amount_sold"),F("amount_stock"),function="pl_max"))Using python functions for custom ORM lookupsfromdjango_plpy.installerimportplfunctionfromdjango.db.modelsimportTransformfromdjango.db.modelsimportIntegerFieldfromtests.books.modelsimportBook@plfunctiondefplsquare(a:int)->int:returna*aclassPySquare(Transform):lookup_name="plsquare"function="plsquare"IntegerField.register_lookup(PySquare)assertBook.objects.filter(amount_stock__plsquare=400).exists()Installing of python triggersTriggers are a very mighty mechanism, django-plpy allows you to easily mark a python function as a trigger, so some logic from your project is directly associated with the data changing events in the database.Here is an example of a python trigger using the@pltriggerdecorator.fromdjango_plpy.installerimportpltrigger@pltrigger(event="INSERT",when="BEFORE",table="books_book")defpl_trigger(td,plpy):# mind triggers don't return anythingtd["new"]["name"]=td["new"]["name"]+"test"td["new"]["amount_sold"]=plpy.execute("SELECT count(*) FROM books_book")[0]["count"]Using Django models in triggersThe parameters of@pltriggerdecorator declare the trigger parameters like event the trigger will bind to and table name. You can replacetable_namewith a model name, the table name will looked up automatically:fromdjango_plpy.installerimportpltriggerfromdjango.db.modelsimportModel,CharField,IntegerFieldclassBook(Model):name=CharField(max_length=10)amount_stock=IntegerField(default=20)amount_sold=IntegerField(default=10)@pltrigger(event="INSERT",when="BEFORE",model=Book)defpl_update_amount(new:Book,old:Book,td,plpy):# don't use save method here, it will kill the database because of recursionnew.amount_stock+=10Read more about plpy triggers in the official Postgres documentation:https://www.postgresql.org/docs/13/plpython-database.html.Using Django models in triggers comes at a price, read about the details of implementation below.Bulk operations and triggers, migrationsPython triggers are fully featured Postgres triggers, meaning they will be created for every row, unlike Django signals. So if you define a trigger with event="UPDATE" and call a bulk update on a model, the trigger will be called for all affected by the operation:fromdjango_plpy.installerimportpltriggerfromtests.books.modelsimportBook@pltrigger(event="UPDATE",when="BEFORE",model=Book)defpl_update_amount(new:Book,old:Book,td,plpy):# don't use save method here, it will kill the database because of recursionnew.amount_stock+=10Update results a trigger call on every line:fromtests.books.modelsimportBookBook.objects.values('amount_stock')# <QuerySet [{'amount_stock': 30}, {'amount_stock': 30}, {'amount_stock': 30}]>Book.objects.all().update(name="test")# 3Book.objects.values('amount_stock')# <QuerySet [{'amount_stock': 40}, {'amount_stock': 40}, {'amount_stock': 40}]>Unlike the code of Django models or signals, triggers will also be called while migrations.Turning Django signals to python triggersAlthough Django signals are neither asynchronous nor have any ability to be executed in another thread or process, many developers mistakenly expect them to behave this way. Often it leads to a callback hell and complex execution flow. Django signals implement a dispatcher-receiver pattern and only make an impression of asynchronous execution.With django-plpy, you can quickly turn your signals into triggers and make them truly asynchronous.Before:fromdjango.dispatchimportreceiverfromdjango.db.models.signalsimportpost_savefromdjango.contrib.auth.modelsimportUser@receiver(post_save,sender=User)defsend_mail(sender,instance,**kwargs):instance.send_mail()After:fromdjango_plpy.installerimportpltriggerfromdjango.contrib.auth.modelsimportUser@pltrigger(event="INSERT",when="AFTER",model=User)defpl_send_mail(new:User,old:User,td,plpy):new.send_mail()Manage.py commandssyncfunctionsinstalls functions and triggers decorated with@plfunctionand@pltriggerto the database.(venv)thorin@thorin-N141CU:~/PycharmProjects/django-plpy$./manage.pysyncfunctions
Synced4functionsand1triggerscheckenvchecks if your local python and database's python versions are compatible.(venv)thorin@thorin-N141CU:~/PycharmProjects/django-plpy$./manage.pycheckenv
Database'sPythonversion:3.7.3
Minorversionsmatch,localversion:3.7.12.Django-plpyDjangoORMcanbeusedintriggers.If your local python and database's python versions have different minor releases, psycopg won't work, so Django ORM cannot be used in triggers. This is what you will see in this case:(venv)thorin@thorin-N141CU:~/PycharmProjects/django-plpy$./manage.pycheckenv
Database's Python version: 3.6.9Postgres python and this python'sversionsdon'tmatch,localversion:3.7.12.Django-plpyDjangoORMcannotbeusedintriggers.Under the hoodSupported argument typesCurrently, supported types are:int: "integer",
str: "varchar",
Dict[str, str]: "JSONB",
List[str]: "varchar[]",
List[int]: "int[]",
bool: "boolean",
float: "real",Using Django in PL functions and triggersWhile installing with,syncfunctionsthe source code of the function will be copied to a corresponding stored procedure and installed in Postgres. This makes your local context not available to the functions, which means that no models or libraries can be used within the transferred functions.To solve this problem, you need to set up your python project and environment within a Postgres python interpreter. Django-plpy supports the following two scenarios of how you use your database.Database and application are on the same hostRarely used nowadays, but still out there, this scenario is the simplest for the environment sharing. Django-plpy creates stored procedures and transfers the necessary configuration to the database:secrets and database access credentialspath to the python env (defaults todistutils.sysconfig.get_python_lib(), for more config see below)loads Django applications the way manage.py does itDatabase is in a separate docker containerA more common production scenario is that the database is on a separate docker container.Couple of words about docker and plpython or django-plpyThe official Postgres image doesn't support plpython plugin out of the box, so if you want to use plpython as such you would need to create your image or use one of those provided by the maintainer of this package (thorinschiffer/postgres-plpython).All the images provide python 3.7 because Postgres uses the default python environment from the OS the image is based on and 3.7 is the standard for Debian Buster.Using django-plpy with dockerized PostgresTo make the code available to the Postgres python interpreter, it has to somehow appear within the docker container. You can either provision the image with it while building if you decided to write your docker image / dockerfile, or you can share the code using volumes.Once the code and environment exist somewhere within the Docker container, django-plpy can be told to use them: So if your environment lives under/env(copy site-packages folder to this path) and your app within/app, add following settings to yoursettings.pyPLPY_ENV_PATHS=["/env"]PLPY_PROJECT_PATH="/app"AWS RDS and other managed databasesIn the times of Saas there databases are rarely connected in a docker image but much
more frequently in a managed database like AWS RDS.
Django-plpy can be only install simple functions and triggers in such instances, because
there is no access to the database's filesystem in such setup.Besides that some managed databases won't give you superuser rights, meaning installing
extensions in such a scenario will be troublesome.How the code is installedDjango-plpy copies the function code, wraps it in a PL/Python stored procedure or trigger
and then installs it withmanage.py syncfunctions.
If you use Django models, database has to have access to your project file and virtualenv (see above),
or if you create your own database docker image, it has to be provisioned correspondingly.
This scenario seems quite exotic, and you do it on your own risk.TroubleshootingIf you seeError loading psycopg2 module: No module named 'psycopg2._psycopg', your local python and db's versions don't match.
Check your python versions withmanage.py checkenvIf you see this:django.db.utils.ProgrammingError: language "plpython3u" does not exist
HINT:  Use CREATE LANGUAGE to load the language into the database.you haven't migrated:manage.py migrateCaveatsCustom functions and triggers are not a part of Django ORM and can get out of hand, name collisions are not checked explicitely for now.For now enabling orm stores the os.environ in json in plaintext in the trigger function text, so this feature is
considered quite experimental.There were no real performance tests made, so this is for you to find out.
Currently only python3.7 is supported with Django ORM, as it is a current version in the Debian based python images,
that are default. There is an opportunity to build an image with a custom python verson and postgres version, but
it seems a total overkill assuming all the problems with Django ORM, see above.Prepend database functions with plpy_ prefix to be sure they are not executed locally.
Django-plpy won't remove any triggers, you will have to take care of the stale triggers yourself.Installation for developmentInstall project locally:pip install -e .Django-plpydjango-environfor passing the necessary env over dotenv, database for creds in particular. See .env_template for possible env variables.Changelog0.1.0 Initial releasepl functionspl triggers