Package: django-auto-prefetch
Automatically prefetch foreign key values as needed.PurposeWhen accessing aForeignKeyorOneToOneField(including in reverse) on
a model instance, if the field’s value has not yet been loaded then
auto-prefetch will prefetch the field for all model instances loaded by
the sameQuerySetas the current model instance. This is enabled at the
model level and totally automatic and transparent for users of the
model.RequirementsPython 3.8 to 3.12 supported.Django 3.2 to 5.0 supported.UsageInstall withpython-mpip installdjango-auto-prefetch.Change all these imports fromdjango.db.modelstoauto_prefetch:ForeignKeyManagerModel- including inheritingMetafromauto_prefetch.Model.MetaOneToOneFieldQuerySetIf you use custom subclasses of any of these classes, you should be able to swap for theauto_prefetchversions in your subclasses’ bases.For example, if you had:fromdjango.dbimportmodelsclassBook(models.Model):author=models.ForeignKey("Author",on_delete=models.CASCADE)classMeta:verbose_name="Book"…swap to:importauto_prefetchfromdjango.dbimportmodelsclassBook(auto_prefetch.Model):author=auto_prefetch.ForeignKey("Author",on_delete=models.CASCADE)classMeta(auto_prefetch.Model.Meta):verbose_name="Book"Runpython manage.py makemigrationsto generate migrations for all the models you modified.
These migrations will set theMeta.base_manager_nameoptiontoprefetch_managerfor every model that you’ve converted.
This change ensures that auto-prefetching happens on related managers.
Such migrations do not change anything in the database.(If you instead setMeta.base_manager_nameon your models, make sure it inherits fromauto_prefetch.Manager.)Background and RationaleCurrently when accessing an uncached foreign key field, Django will
automatically fetch the missing value from the database. When this
occurs in a loop it creates 1+N query problems. Consider the following
snippet:forchoiceinChoice.objects.all():print(choice.question.question_text,":",choice.choice_text)This will do one query for the choices and then one query per choice to
get that choice’s question.This behavior can be avoided with correct application ofprefetch_related()like this:forchoiceinChoice.objects.prefetch_related("question"):print(choice.question.question_text,":",choice.choice_text)This has several usability issues, notably:Less experienced users are generally not aware that it’s necessary.Cosmetic seeming changes to things like templates can change the fields that
should be prefetched.Related to that, the code that requires theprefetch_related()(e.g. the
template) may be quite removed from where theprefetch_related()needs to
be applied (e.g. the view).Subsequently finding whereprefetch_related()/select_related()calls are missing is non-trivial and needs to be done on an ongoing
basis.Excess entries inprefetch_related()calls are even harder
to find and result in unnecessary database queries.It is very difficult for libraries like the admin and Django Rest Framework
to automatically generate correctprefetch_related()clauses.On the first iteration of the loop in the example above, when we first
access a choice’s question field, instead of fetching the question for
just that choice, auto-prefetch will speculatively fetch the questions
for all the choices returned by theQuerySet. This change results in
the first snippet having the same database behavior as the second while
reducing or eliminating all of the noted usability issues.Some important points:ManyToManyFields are not changed at all.Because these areForeignKeyandOneToOneFields, the
generated queries can’t have more result rows than the original query
and may have less. This eliminates any concern about a multiplicative
query size explosion.This feature will never result in more database queries as a prefetch will
only be issued where the ORM was already going to fetch a single related
object.Because it is triggered by fetching missing related objects it will not at
all change the DB behavior of code which is fully covered byprefetch_related()and/orselect_related()calls.This will inherently chain across relations likechoice.question.author.
The conditions above still hold under such chaining.In some rare situations it may result in larger data transfer between the
database and Django (see below).An example of that last point is:qs=Choice.objects.all()list(qs)[0].questionSuch examples generally seem to be rarer and more likely to be visible
during code inspection (vs{{ choice.question }}in a template). And
larger queries are usually a better failure mode than producing hundreds
of queries. For this to actually produce inferior behavior in practice
you need to:
* fetch a large number of choices
* filter out basically all of them
* …in a way that prevents garbage collection of the unfiltered onesIf any of those aren’t true then automatic prefetching will still
produce equivalent or better database behavior than without.See AlsoThe phabricator guide to the N+1 queries problem:https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/The django-developers mailing list discussion of adding the feature
to core Django:https://groups.google.com/forum/m/#!topic/django-developers/EplZGj-ejvgThe nplus package, useful for detecting the N+1 queries problem in
your application (but not solving it):https://pypi.org/project/nplusone/P.S.If you have concerns go look at the code, it’s all inauto_prefetch/__init__.pyand is fairly short.