Package: sa-filters
Filter, sort and paginate SQLAlchemy query objects. Ideal for
exposing these actions over a REST API.This project is fork ofsqlalchemy-filterswith sqlalchemy 1.4 and 2.0 support
and other improvements.FilteringAssuming that we have aSQLAlchemyqueryobject:fromsqlalchemyimportColumn,Integer,Stringfromsqlalchemy.ext.declarativeimportdeclarative_baseclassBase(object):id=Column(Integer,primary_key=True)name=Column(String(50),nullable=False)count=Column(Integer,nullable=True)@hybrid_propertydefcount_square(self):returnself.count*self.count@hybrid_methoddefthree_times_count(self):returnself.count*3Base=declarative_base(cls=Base)classFoo(Base):__tablename__='foo'# ...query=session.query(Foo)Then we can apply filters to thatqueryobject (multiple times):fromsa_filtersimportapply_filters# `query` should be a SQLAlchemy query or Select objectfilter_spec=[{'field':'name','op':'==','value':'name_1'}]filtered_query=apply_filters(query,filter_spec)more_filters=[{'field':'foo_id','op':'is_not_null'}]filtered_query=apply_filters(filtered_query,more_filters)result=filtered_query.all()It is also possible to filter queries that contain multiple models,
including joins:classBar(Base):__tablename__='bar'foo_id=Column(Integer,ForeignKey('foo.id'))query=session.query(Foo).join(Bar)filter_spec=[{'model':'Foo','field':'name','op':'==','value':'name_1'},{'model':'Bar','field':'count','op':'>=','value':5},]filtered_query=apply_filters(query,filter_spec)result=filtered_query.all()apply_filterswill attempt to automatically join models toqueryif they’re not already present and a model-specific filter is supplied.
For example, the value offiltered_queryin the following two code
blocks is identical:query=session.query(Foo).join(Bar)# join pre-applied to queryfilter_spec=[{'model':'Foo','field':'name','op':'==','value':'name_1'},{'model':'Bar','field':'count','op':'>=','value':5},]filtered_query=apply_filters(query,filter_spec)query=session.query(Foo)# join to Bar will be automatically appliedfilter_spec=[{'field':'name','op':'==','value':'name_1'},{'model':'Bar','field':'count','op':'>=','value':5},]filtered_query=apply_filters(query,filter_spec)The automatic join is only possible ifSQLAlchemycan implictly
determine the condition for the join, for example because of a foreign
key relationship.Automatic joins allow flexibility for clients to filter and sort by related
objects without specifying all possible joins on the server beforehand. Feature
can be explicitly disabled by passingdo_auto_join=Falseargument to theapply_filterscall.Note that first filter of the second block does not specify a model.
It is implictly applied to theFoomodel because that is the only
model in the original query passed toapply_filters.It is also possible to apply filters to queries defined by fields, functions orselect_fromclause:query_alt_1=session.query(Foo.id,Foo.name)query_alt_2=session.query(func.count(Foo.id))query_alt_3=session.query().select_from(Foo).add_column(Foo.id)Hybrid attributesYou can filter by ahybrid attribute: ahybrid propertyor ahybrid method.query=session.query(Foo)filter_spec=[{'field':'count_square','op':'>=','value':25}]filter_spec=[{'field':'three_times_count','op':'>=','value':15}]filtered_query=apply_filters(query,filter_spec)result=filtered_query.all()Restricted LoadsYou can restrict the fields thatSQLAlchemyloads from the database by
using theapply_loadsfunction:query=session.query(Foo,Bar).join(Bar)load_spec=[{'model':'Foo','fields':['name']},{'model':'Bar','fields':['count']}]query=apply_loads(query,load_spec)# will load only Foo.name and Bar.countThe effect of theapply_loadsfunction is to_defer_the load
of any other fields to when/if they’re accessed, rather than loading
them when the query is executed. It only applies to fields that would be
loaded during normal query execution.Effect on joined queriesThe defaultSQLAlchemyjoin is lazy, meaning that columns from the
joined table are loaded only when required. Thereforeapply_loadshas limited effect in the following scenario:query=session.query(Foo).join(Bar)load_spec=[{'model':'Foo','fields':['name']}{'model':'Bar','fields':['count']}# ignored]query=apply_loads(query,load_spec)# will load only Foo.nameapply_loadscannot be applied to columns that are loaded asjoined eager loads.
This is because a joined eager load does not add the joined model to the
original query, as explainedhereThe following would not prevent all columns fromBarbeing eagerly
loaded:query=session.query(Foo).options(joinedload(Foo.bar))load_spec=[{'model':'Foo','fields':['name']}{'model':'Bar','fields':['count']}]query=apply_loads(query,load_spec)Automatic JoinIn fact, what happens here is thatBaris automatically joined
toquery, because it is determined thatBaris not part of
the original query. Theload_spectherefore has no effect
because the automatic join results in lazy evaluation.If you wish to perform a joined load with restricted columns, you must
specify the columns as part of the joined load, rather than withapply_loads:query=session.query(Foo).options(joinedload(Bar).load_only('count'))load_spec=[{'model':'Foo','fields':['name']}]query=apply_loads(query.load_spec)# will load ony Foo.name and Bar.countSortfromsa_filtersimportapply_sort# `query` should be a SQLAlchemy query or Select objectsort_spec=[{'model':'Foo','field':'name','direction':'asc'},{'model':'Bar','field':'id','direction':'desc'},]sorted_query=apply_sort(query,sort_spec)result=sorted_query.all()apply_sortwill attempt to automatically join models toqueryif
they’re not already present and a model-specific sort is supplied.
The behaviour is the same as inapply_filters.This allows flexibility for clients to sort by fields on related objects
without specifying all possible joins on the server beforehand.Hybrid attributesYou can sort by ahybrid attribute: ahybrid propertyor ahybrid method.Paginationfromsa_filtersimportapply_pagination# `query` should be a SQLAlchemy query or Select objectquery,pagination=apply_pagination(query,page_number=1,page_size=10)page_size,page_number,num_pages,total_results=paginationassert10==len(query)assert10==page_size==pagination.page_sizeassert1==page_number==pagination.page_numberassert3==num_pages==pagination.num_pagesassert22==total_results==pagination.total_resultsSelect objectYou can useapply_filters,apply_loads,apply_sortandapply_paginationwith SQLAlchemySelectobject:stmt=select(Foo)filter_spec=[{'field':'name','op':'==','value':'name_1'}]filtered_stmt=apply_filters(stmt,filter_spec)result=session.execute(filtered_stmt).scalars().all()Note that when usingapply_paginationwithSelectobject,
you must pass session argument:stmt=select(Foo)query,pagination=apply_pagination(stmt,page_number=1,page_size=10,session=session)Filters formatFilters must be provided in a list and will be applied sequentially.
Each filter will be a dictionary element in that list, using the
following format:filter_spec=[{'model':'model_name','field':'field_name','op':'==','value':'field_value'},{'model':'model_name','field':'field_2_name','op':'!=','value':'field_2_value'},# ...]Themodelkey is optional if the original query being filtered only
applies to one model.If there is only one filter, the containing list may be omitted:filter_spec={'field':'field_name','op':'==','value':'field_value'}Wherefieldis the name of the field that will be filtered using the
operator provided inop(optional, defaults to==) and the
providedvalue(optional, depending on the operator).You can also specify the table name instead of the model name by specifying
the optional keytable:filter_spec=[{'table':'table_name','field':'field_name','op':'==','value':'field_value'},{'table':'table_name','field':'field_2_name','op':'!=','value':'field_2_value'},# ...]This is the list of operators that can be used:is_nullis_not_null==,eq!=,ne>,gt<,lt>=,ge<=,lelikeilikenot_ilikeinnot_inanynot_anyany / not_anyPostgreSQL specific operators allow to filter queries on columns of typeARRAY.
Useanyto filter if a value is present in an array andnot_anyif it’s not.Boolean Functionsand,or, andnotfunctions can be used and nested within the
filter specification:filter_spec=[{'or':[{'and':[{'field':'field_name','op':'==','value':'field_value'},{'field':'field_2_name','op':'!=','value':'field_2_value'},]},{'not':[{'field':'field_3_name','op':'==','value':'field_3_value'}]},],}]Note:orandandmust reference a list of at least one element.notmust reference a list of exactly one element.Sort formatSort elements must be provided as dictionaries in a list and will be
applied sequentially:sort_spec=[{'model':'Foo','field':'name','direction':'asc'},{'model':'Bar','field':'id','direction':'desc'},# ...]Wherefieldis the name of the field that will be sorted using the
provideddirection.Themodelkey is optional if the original query being sorted only
applies to one model.nullsfirst / nullslastsort_spec=[{'model':'Baz','field':'count','direction':'asc','nullsfirst':True},{'model':'Qux','field':'city','direction':'desc','nullslast':True},# ...]nullsfirstis an optional attribute that will placeNULLvalues first
if set toTrue, according to theSQLAlchemy documentation.nullslastis an optional attribute that will placeNULLvalues last
if set toTrue, according to theSQLAlchemy documentation.If none of them are provided, thenNULLvalues will be sorted according
to the RDBMS being used. SQL defines thatNULLvalues should be placed
together when sorting, but it does not specify whether they should be placed
first or last.Even though bothnullsfirstandnullslastare part ofSQLAlchemy,
they will raise an unexpected exception if the RDBMS that is being used does
not support them.At the moment they aresupported by PostgreSQL,
but they arenotsupported by SQLite and MySQL.Running testsThe default configuration usesSQLite,MySQL(if the driver is
installed, which is the case whentoxis used) andPostgreSQL(if the driver is installed, which is the case whentoxis used) to
run the tests, with the following URIs:sqlite+pysqlite:///test_sa_filters.dbmysql+mysqlconnector://root:@localhost:3306/test_sa_filterspostgresql+psycopg2://postgres:@localhost:5432/test_sa_filters?client_encoding=utf8'A test database will be created, used during the tests and destroyed
afterwards for each RDBMS configured.There are Makefile targets to run docker containers locally for bothMySQLandPostgreSQL, using the default ports and configuration:$makemysql-container$makepostgres-containerTo run the tests locally:$# Create/activate a virtual environment$pipinstalltox$toxThere are some other Makefile targets that can be used to run the tests:There are other Makefile targets to run the tests, but extra
dependencies will have to be installed:$pipinstall-U--editable".[dev,mysql,postgresql]"$# using default settings$maketest$makecoverage$# overriding DB parameters$ARGS='--mysql-test-db-uri mysql+mysqlconnector://root:@192.168.99.100:3340/test_sa_filters'maketest$ARGS='--sqlite-test-db-uri sqlite+pysqlite:///test_sa_filters.db'maketest$ARGS='--mysql-test-db-uri mysql+mysqlconnector://root:@192.168.99.100:3340/test_sa_filters'makecoverage$ARGS='--sqlite-test-db-uri sqlite+pysqlite:///test_sa_filters.db'makecoverageDatabase management systemsThe following RDBMS are supported (tested):SQLiteMySQLPostgreSQLSQLAlchemy supportThe followingSQLAlchemyversions are supported:1.4,2.0.ChangelogConsult theCHANGELOGdocument for fixes and enhancements of each version.LicenseApache 2.0. SeeLICENSEfor details.