Package: nplus1loader
An SqlAlchemy loader that solves the N+1 problemTL;DRWhat happens if you touch an unloaded column or relationship while looping over results from the DB?users=ssn.query(User).all()foruserinusers:user.articles# load a relationshipRight. If you have 1000 users, you'll end up with 1000 queries.Here's a simple solution for you that will lazy-load an attribute for all those users with just one query:fromnplus1loaderimportdefault_columns,nplus1loaderusers=ssn.query(User).options(default_columns(User),nplus1loader('*')).all()foruserinusers:user.articlesIt will only make 1 query to load all the users, then when it sees that you want articles,
it will only make 1 additional query to load all articles¹ for those users.¹: it will actually make 1 query per 500 users.The N+1 ProblemFirst of all, what's the N+1 problem?
Suppose you have the following code which loads a number of objects from the DB,
but some fields are deferred:users=ssn.query(User).options(load_only('id')# worst case scenario)And then somewhere in the code somejsonify()function decides to check one of
the unloaded properties:foruserinusers:user.state# load the attributeBecause the attribute hasn't been loaded on the whole collection...
it will trigger N queries to load that attribute on every iteration of the loop.
Crying shame.In many cases we can tell SqlAlchemy to load exactly the attributes we are going
to use (by usingundefer()/joinedload()/selectinload()), but in other
cases that's utterly impossible. We just don't know the names of those attributes
in advance.The SolutionThis module offers a solution: when your codetouchesone unloaded attribute
on a collection, it assumes that you're going to iterate through them, and loads
all of them for you.The solution is, however, not all-encompassing: you only enable it where you see fit.
It's implemented as a load option, like this:fromnplus1loaderimportnplus1loaderusers=ssn.query(User).options(load_only('id'),# worst case scenarionplus1loader('*')# include every column and every relationship)Now, whenever you touch any unloaded column or relationship on any of those loaded users,
the attribute will be loaded for all User instance in that session.
So now, you can safely iterate through them.Warning: do not rely on this mechanism; only use it as a safeguard for production apps.
It is recommended that you do something like this instead:fromnplus1loaderimportdefault_columns,raiseload_all,nplus1loaderusers=ssn.query(User).options(default_columns(User),raiseload_all('*')ifin_testingelsenplus1loader('*'))Make sure you import the module first in order for those load options to get registered :)importnplus1loaderOne minor inconvenienceDid you notice thedefault_columns()load option?
You'll have to insert it early into your query, I'm sorry.This is becausenplus1loader('*')is a catch-all column loader option.
In other words, if no column strategies are given by you, there are no loaded columns,
and thisnplus1loader('*')will handle all columns!This is not what we want. We would instead want to apply the defaults... then override them...
and put thisnplus1loader('*')in the end. But that's not possible with SqlAlchemy.So we have to live with thisdefault_columns(Model): it reads the default loading strategies
from your Model and applies it early. In other words, it knows which columns are deferred by default.If you don't put it there,nplus1loader()will remove all columns from the query, and you'll get
this weird error:sqlalchemy.exc.InvalidRequestError: Query contains no columns with which to SELECT from.This is effectively the same asdefer('*').Now... why can't we justundefer('*')everything?Because that's not right with someColumn(deferred=True), andColumn(deferred_group='password'),
which have other preferences about being loaded.
Sodefault_columns(User)will respect those defaults.LoggingAll logging is done to the 'lib.db.nplus1loader.NPlusOneLazyColumnLoader' logger:05:09:37 [W] nplus1loader.NPlusOneLazyColumnLoader: Number.es: N+1 loading of 4 instances
emitted by: _nplusone_lazy_loading() nplus1loader/strategies.py:76Other solutionsoperator/sqlalchemy_bulk_lazy_loaderonly supports relationships and not columns, has code copy-pasted from an old SqlAlchemy version,
requires re-configuring your relationships on the model (lazy='bulk')operator/sqlalchemy_bulk_lazy_loaderis a monitoring tool; it does not solve the N+1 problem at allBoth projects served as a source of inspiration.Database supportTested with:PostgreSQLMySQLSQLiteOther Included ToolsHandle specific attributesThenplus1loaderpackage has other tools that you might find useful.First of all, you can be more specific about which attributes you want to have handled by the N+1 loader.
You can opt to handle only relationships, all, or specific ones:nplus1loader_rels('*')nplus1loader_rels(relationship_name,...)or you can opt to handle only columns:nplus1loader_cols('*')nplus1loader_cols(column_name,...)nplus1loader('*', nested=True)You may have noticed that the N+1 loader has thenested=Truekeyword argument.
It controls which options should be put on those relationships that have been lazy-loaded in bulk.By default, loader options in SqlAlchemy would only affect your top-level entities.
For instance:query(User).options(default_columns(User).nplus1loader('*'))would only apply the N+1 loader on the immediate attributes of theUser, but not to its related models.
So, if the N+1 loader happens to load User'sArticles, the attributes ofArticlewould use the plain
SqlAlchemy lazy-loading one by one.Withnested=True, the N+1 loader would also put annplus1loader('*')onArticlewhen those are loaded.Thisnested=Trueis the default, and you don't have to bother about it.
Moreover, there probably is no sane reason for disabling it, ever. So it's a good candidate for removal.default_columns(Model)This loader option takes the defaultdefer()/undefer()settings from a model and sets them on a Query.
In itself, this option is useless, but it enables you to alter the defaults by using other loading options.In particular,nplus1loader('*')won't work unless you applydefault_columns()first.Note thatnplus1loader('*')can be chained fromdefault_columns(), or it can be written after a comma.
Both will have the same effect.raiseload_all('*')Raiseload for both columns and relationships.Example:ssn.query(User).options(default_columns('id'),# Raise an error if any attribute at all is lazy-loadedraiseload_all('*'))The error that it throws isnplus1loader.LazyLoadingAttributeError(), so it's easy to catch in your code ;)raiseload_col(*columns)Raiseload for columns. Use to fine-tune.Example:ssn.query(User).options(load_only('id'),# Raise an error if any other column is lazy-loadedraiseload_col('*'))You can also specify individual columns as a list.The error that it throws isnplus1loader.LazyLoadingAttributeError().raiseload_rel(*relations)Raiseload for relatioships. Use to fine-tune.It's essentially the same assqlalchemy.orm.raiseload(), but throws thenplus1loader.LazyLoadingAttributeError()error instead.bulk_load_attribute_for_instance_states()This is the heart of N+1 loader.You give it a list of instances, name one unlaoded attribute,
and it makes just one query to load this attribute's value from the database, and efficiently
augments all the existing instances with the loaded value of this attribute.This is an advanced feature.
See the code:nplus1loader/bulk_load.py