Package: django-query-preparer
Django Query PreparerDjango Query Preparer (dqp) can prepare your queries in the database on application start so they don't need to be parsed and compiled at execution time! This is a postgres specific implementation which targets python 3.7+ and Django 2.2+.UsageRaw SQLQuickstartPreparing and executing raw SQL statements is straight forward:fromdqpimportexecute_stmt,prepare_sql@prepare_sqldefmyquery():return"select count(*) from django_migrations;"execute_stmt(myquery())# <- note that we call myquery() which gives us the name of the prepared statement.The steps to preparing and executing an SQL query on the fly are:Write a function that takes no arguments and returns a string which is the query to prepare.Decorate this function with the@prepare_sqldecorator. Note that@prepare_sqlevaluates the function and prepares the SQL in the database immediately.Callexecute_stmtand pass the function as the prepared statement name. The decorator changes the output of the function in your scope to return the name of the prepared statement.You can use placeholder parameters as normal too:fromdqpimportexecute_stmt,prepare_sql@prepare_sqldefmyquery():return"select count(*) from django_migrations where id <%s;"execute_stmt(myquery(),[10])orfromdqpimportexecute_stmt,prepare_sql@prepare_sqldefmyquery():return"select count(*) from django_migrations where id <%(migration_id)s;"execute_stmt(myquery(),{"migration_id":10})In DjangoIt's almost as easy to use prepared SQL statements in a Django app:Add"dqp.apps.DQPConfig"to your list ofINSTALLED_APPS.Again, write a function that takes no arguments and returns one string which is the query to prepare.Decorate this function with the@register_prepared_sqldecorator.  The decorated functions will be evaluated and prepared in the database when the app receives theon_readysignal.Callexecute_stmtand pass the function as the prepared statement name.e.g:fromdjango.httpimportJsonResponsefromdqpimportexecute_stmt,register_prepared_sql@register_prepared_sqldefmyquery():return"select count(*) from django_migrations;"defview_migrations(request):# migrations will be a list of dictsmigrations=execute_stmt(myquery())returnJsonResponse(migrations)Django ORM queriesBasic UsageYou can also prepare queries built using the Django ORM methods.Add"dqp.apps.DQPConfig"to your list ofINSTALLED_APPS.For the model that you want to prepare the query against, add thePreparedStatementManager:fromdjango.dbimportmodelsfromdqp.managerimportPreparedStatementManagerclassMyModel(models.Model):name=models.CharField(max_length=50,blank=True,null=True)alias=models.CharField(max_length=50,blank=True,null=True)objects=models.Manager()prepare=PreparedStatementManager()# <-- Here is the PreparedStatementManager!Write a function that takes no arguments and returns a query set using thePreparedStatementManager. Decorate it with theregister_prepared_qsdecorator:fromdqpimportregister_prepared_qs@register_prepared_qsdefget_all_from_my_model():returnMyModel.prepare.all()Callexecute_stmt(). It will return aPreparedStatementQuerySet:fromdqpimportexecute_stmtqs=execute_stmt(get_all_from_my_model())fromdqp.querysetimportPreparedStatementQuerySetisinstance(qs,PreparedStatementQuerySet)# -> TrueYou can use a placeholder when preparing query sets with input arguments:fromdqpimportregister_prepared_qs,Placeholder@register_prepared_qsdefget_my_model_lt():returnMyModel.prepare.filter(id__lt=Placeholder("id"))execute_stmt(get_my_model_lt(),id=4)You have to name your placeholders and then use those names as keyword arguments toexecute_stmt. This is because Django can re-order the filters so there's no guarantee that the order in which you specify the filters in the ORM functions will be the order they appear in the executed SQL.If you want to use lists as inputs you can useListPlaceholder:fromdqpimportregister_prepared_qs,ListPlaceholder@register_prepared_qsdefget_my_model_in():returnMyModel.prepare.filter(id__in=ListPlaceholder("ids"))execute_stmt(get_my_model_in(),ids=[4,5])Each placeholder must have a unique name within the same query. You can mix and match constant and passed in parameters:fromdqpimportregister_prepared_qs,ListPlaceholder@register_prepared_qsdefget_active_in_range():returnMyModel.prepare.filter(id__in=ListPlaceholder("ids"),active=True)execute_stmt(get_active_in_range(),ids=range(10))LimitationsThe following query set methods are not supported when preparing a query nor on aPreparedStatementQuerySet:aggregate()in_bulk()create()bulk_create()bulk_update()get_or_create()update_or_create()delete()update()exists()explain()Prefetch RelatedYou cannot useprefetch_relatedwhen preparing a query:@register_prepared_qsdefget_my_model_lt():returnMyModel.prepare.filter(id__lt=Placeholder).prefetch_related('related_field')>PreparedQueryNotSupported:Cannotuseprefetch_relatedwhenpreparingqueysets.AddtheprefetchrelatedtothereturnedquerysetonstatementexecutionAs the error says, you must use the prefetch related AFTER executing the prepared statement:@register_prepared_qsdefget_my_model_lt():returnMyModel.prepare.filter(id__lt=Placeholder)qs=execute_stmt(get_my_model_lt(),[4])qs=qs.prefetch_related('related_field')The query for theprefetch_relatedwill not be a prepared statement. If you need the related data to be prepared and executed as prepared statement then you'll have to write two separate queries and map one to the other manually.Differences with ordinary Django usage:In normal Django usageget(),count(),first(),last(),latest()andearliest()all execute immediately. This doesn't happen for prepared queries: they do not execute immediately. But they will still produce what you expect:# Normal django usageMyModel.objects.count()# > 12# Prepared statement usage@prepare_qsdefmy_model_count():returnMyModel.prepare.count()# > no result - not executed yetexecute_stmt(my_model_count())# > 12A query prepared usingget(),first(),last(),latest()andearliest()follows the Django behaviour when executed and will return a model instance, not a query set, or it will raise the same exceptions if zero or more than one rows match the filter.@prepare_qsdefmy_model_get():returnMyModel.prepare.get(id=1)# > no result - not executed yetqs=execute_stmt(my_model_get())isinstance(qs,MyModel)# > TrueYou can usecount(),first()andlast()on aPreparedStatementQuerySetbut you can't useget(),filter(),latest()orearliest(). You'll need to use python's built-infilterandsortedmethods to further filter or order the results of an executed prepared statement.Start-up BehaviourDjango-query-preparer will attempt to prepare all registered queries on app start-up when it receives the on-ready signal from Django.  Sometimes this will fail, e.g. because a table or column in one of your prepared queries hasn't been created by a migration yet, or because you're running tests and there's no schema!  In these cases,dqpwill catch the error and log it as awarning. Then when the prepared statement is executed for the first time (for each process) it will try again to prepare the query before execution. If it fails again then the error will be raised.  We feel that this offers the best compromise between performance and pragmatism.ConfigurationAs noted, add"dqp.apps.DQPConfig"to your list ofINSTALLED_APPS.Prepared statements in postgres are only valid for the duration of the database session. So if you start a new database session then any prepared statements that have been previously prepared will be lost.  To make use of prepared statements in Django you'll need to make sure that your Django app doesn't create a new database connection for every request. You can use theCONN_MAX_AGEparameter to force Django to re-use database connections.  Alternatively you can usepgbouncer(see below) to pool your db connections.Using with CeleryCelery creates a new database connection for every job.  This means that you cannot use prepared statements without some sort of connection pooling, e.g.pgbouncer.Using withpgbouncerWhilepgbouncerwill re-use a database connection between, by default it will runDISCARD ALLbefore returning a session back to the pool to be re-used. This will deallocate all prepared queries. To get around this, set theserver_reset_queryconfig item to be"SET SESSION AUTHORIZATION DEFAULT; RESET ALL; CLOSE ALL; UNLISTEN *; SELECT pg_advisory_unlock_all(); DISCARD SEQUENCES; DISCARD TEMP;"in yourpgbouncerconfig.Common errorsProgrammingError: can't adapt type 'PreparedStatementQuerySet'This is caused by passing a query set intoexecute_stmtrather than a primitive type, or more likely a list of primitive types. If you have a set of values from another query set then you need to calllist()on them to convert form a query set to a list.Usingdqpin testsUnittestPrepared queries are stored per database session in postgres. But database sessions are restarted between each test by Django. To make sure that all you prepared statements are re-prepared before every test add thePrepStmtTestMixinto your test class:fromdjango.testimportTestCasefromdqp.testingimportPrepStmtTestMixinclassMyTests(TestCase,PrepStmtTestMixin):@classmethoddefsetUp(cls):# If you have a setUp method in your test class then you'll need to call super().setUp()# to make sure the setUp method of PrepStmtTestMixin is called.super().setUp()deftest_stuff(self):# ...passAs re-preparing the statements does take a small amount of time you should only use thePrepStmtTestMixinin the tests that use prepared queries.PytestYou can use theprepare_allfunction to prepare statements at the start of any test that requires it:importpytestfromdqp.testingimportprepare_all@pytest.mark.django_db(transaction=True)deftest_stuff():prepare_all()# testing code...