Package: apiql
ApiQL - a Simple API query language for RESTful servicesApiQL is a simple domain language for API consumers to express REST resource filtering criteria in a concise, powerful, URL friendly way.InstallingApiQL can be installed and updated with pip:pip install apiqlAPI query languageApi query language provides set of predicates and expressions allowing API providers and consumers to build complex resource queries. While ApiQL syntax is in largely SQL inspired, it is designed to be technology agnostic. ApiQL is translated into abstract criteria tree and can be transformed into backend specific functionality.ApiQL query syntaxApiQL query is set of basic predicates which may be then composed using conjunctions (and) or disjunctions (or) into expressions.For example, let's assume we have REST API exposing movie information:wget -q -O - 'http://awso.me/api/movies'[{"title":"Monty Python and the Holy Grail","release_year":1975,"original_title":"Monty Python and the Holy Grail""created_datetime":"2019-05-01T09:17:06.527181+00:00","external_id":"762","genres":[{"id":1,"name":"Comedy"},{"id":2,"name":"Adventure"},{"id":2,"name":"Fantasy"}],"id":1,"plot":"King Arthur, accompanied by his squire, recruits his Knights of the Round Table [...]","rating":6.0,"source":"tmdb","ignored":false},...]ApiQL query basicsIn its most basic form, ApiQL query is just single predicate:wget -q -O - 'http://awso.me/api/movies?filter=title=="Monty Python and the Holy Grail"'which will filter only "Monty Python and the Holy Grails" resources. (whole ApiQL query is contained within single URL query param; note that ApiQL uses==operator for equality, not=).Note: ApiQL was designed to be as much URL friendly as possible, however all ApiQL queries, should be URL-encoded. In this document we usewgetas it URL-encodes all URLs by default;curlhas little bit more arcane syntax in this case:curl -G   --data-urlencode "filter=[my ApiQL query]" http://awso.me/api/moviesApiQL predicatesIn reality API consumers require much more than simple==predicate. ApiQL supports following set of predicates:==,!=,>,>=,<,<=,like- equivalent SQL LIKE operator, however You don't need explicitly add%,ilike- case insensitive version oflike,notlike- equivalent SQL NOT LIKE operator,notilike- case insensitive version onnotlike,startswith- equivalent to SQL STARTS WITH operator,istartswith- case insensitive version ofstartswith,endswith- equivalent to SQL ENDS WITH,iendswith- case insensitive version ofiendswith,contains- alias tolike,notcontains- alias tonotlike,icontains- case insensitive version ofcontains,inotcontains- case insensitive version ofnotcontains,in- equivalent to SQL IN operator,notinequivalent to SQL NOT IN operator.For example, query:wget -q -O - 'http://awso.me/api/movies?filter=title ilike "Holy"'will return all movies matching "Holy": "Monty Python and the Holy Grails", "Holy Money" and possibly bunch of other filcks matching "Holy".and query:wget -q -O - 'http://awso.me/api/movies?filter=release_year>=1975'will return all movies released in 1975 or later.Query literalsLiterals are the values, things that can be on the right hand side of predicate. So far we have seen strings ("Holy") and numeric literals (1975). ApiQL support bunch of other literals too:Strings - all string literals are unicode and are following the same rules like JSON string literals. ApiQL strings are always double-quoted (for example, this is a string: "This is a string", this however: 'not a string'is not), and escaped ("The movie: \"The Movie\"").Numbers - are basically integers and floats:release_year != 2003orrating > 3.3or evenrating > -1.6E-35.Boolean -trueandfalseare translated into platform specific booleans. Example usage:ingored != falseNil - specialnullliteral is translated into platform specific literal, for example:genres != nullDatetime - literal representing datetime:created_datetime >= datetime("2019-05-01T08:00:00.527181+00:00"). Out of the box ApiQL supports ISO-8601 datetime formats (however, this behavior can be customized).Tuples - represents series of values ininandnotinclauses:release_year notin (1975, 2011). Tuples can contain coma separated list of other literals:release_date in (flase, null, datetime("datetime("1975-01-01T00:00:00.000000+00:00"))Combining queriesQueries can be combined into more complicated expressions by grouping atomic predicates (separated by;).For example:wget -q -O - 'http://awso.me/api/movies?filter=title ilike "Holy";release_year>1975;ignored!=null'predicates in this query are interpreted asconjunction(and), returning all movies withtitlematching "Holy"andreleased after 1975 which are not marked asignored.Logical expressionsApiQL supports logicalconjunctions(and) anddisjunctions(or); both of them can nest predicates:and(criteria(;criteria)*)andor(criteria(;criteria)*)Query below, is equivalent to the previous one:wget -q -O - 'http://awso.me/api/movies?filter=and(title ilike "Holy";release_year > 1975;ignored != true)'This one, however:wget -q -O -'http://awso.me/api/movies?filter=or(title ilike "Holy";release_year>1975;ignored!=true)'will return all movies with titles matching "Holy"orreleased after 1975ornot ignored.Conjunctions and Disjunctions can be nested. Let's say we want to filter movies with rating greater than 7 or source is "IMDB", however we would like to filter only not-ignored resources:wget -q -O - 'http://awso.me/api/movies?filter=and(or(rating>7;source="IMDB");ignored!=flase)'Parsing ApiQL queriesSo far, so good. Now, how ApiQL Queries can actually be interpreted by API data store? ApiQL queries are internally translated into python data structure (syntax tree) represented byCrtieriaclass.Criteriaalong withPredicate,ConjunctionandDisjunctionfully represents parsed query tree.Criteriaclass aggregates list ofCriterion.Criterionjust abstract atomic criteria element; it's either:simplePredicatean atomic logical expression (for examplePredicate('title', '==', 'Apocalypse Now')for querytitle=="Apocalypse Now")Conjunctionwhich again is just logicalandoperator with group of predicatesConjunction([Predicate('title','==', 'Apocalypse Now'),Predicate('release_year','>', 1975)])for queryand(title=="Apocalypse Now";release_year>1975)Disjunction- logicaloroperatorParsing ApiQL query with python:importapiql.parserasparserfromapiql.criteriaimportCriteria,Conjunction,Disjunction,Predicate# ...parsed_criteria=parser.parse('and(title like "Monty";genres == null;ignored!=false;release_year<=1975)')syntax_tree=Criteria([Conjunction([Predicate('title','like','Monty'),Predicate('genres','==',None),Predicate('ignored','!=',False),Predicate('release_year','<=',1975)])])# parsed_criteria is equal to syntax_treeassertEqual(syntax_tree,parsed_criteria)A Tour of queriesNow we can actually use ApiQL to filter resources. Out of the box ApiQL provides SQLAlchemy ORM integration (it can be, however integrated with other backends). This section showcases all basic query examples. Complete list of query capabilities can be found in the test suite.All examples will use this sample data model:Base=declarative_base()classGenre(Base):__tablename__='genre'id=Column(Integer,primary_key=True)name=Column(String)genre_id=Column(Integer)movie_id=Column(Integer,ForeignKey('movie.id'),nullable=True)classMovie(Base):__tablename__='movie'id=Column(Integer,primary_key=True)title=Column(String)original_title=Column(String)release_year=Column(Integer)source=Column(String)rating=Column(String)created_datetime=Column(DateTime,default=datetime.utcnow)genres=relationship('Genre',cascade="all",backref="movie",lazy=True)drama=Genre(name="Drama",genre_id=1)scifi=Genre(name="Sci-Fi",genre_id=2)war=Genre(name="War",genre_id=3)adventure=Genre(name="Adventure",genre_id=4)comedy=Genre(name="Comedy",genre_id=5)monty_python=Movie(title="Monty Python and the Holy Grail",release_year=1975,source="IMDB",rating="8",genres=[comedy,adventure])jurassic_park=Movie(title="Jurassic Park",release_year=1993,source='IMDB',rating="9",genres=[adventure,scifi])apocalypse_now=Movie(title="Apocalypse Now",release_year=1979,source="TMDB",rating="9",original_title="Apocalypse Now, The",genres=[drama,war])gosford_park=Movie(title="Gosford Park",release_year=2001,source='IMDB',rating="7",genres=[drama])session.add(monty_python)session.add(jurassic_park)session.add(apocalypse_now)session.add(gosford_park)A main entry point to SQLAlchemy integration iswith_criteriaextension method, which extends plain SQLAlchemyQueryobject with ApiQL capabilities.with_criteriais following basic SQLAlchemy conventions, so it can be freely used with nativefilter_byorfilterfunctions.Following examples show ApiQL queries and their native SQLAlchemy representations.Simple conjunction criteriafromapiql.backends.sqlalchemy.ormimportwith_criteriaactual=session.query(Movie).with_criteria('and(rating=="8";release_year==1975)')# is equivalent toexpected=session.query(Movie).filter(and_(Movie.rating==8,Movie.release_year==1975))Simple disjunction criteriafromapiql.backends.sqlalchemy.ormimportwith_criteriaactual=session.query(Movie).with_criteria('or(rating=="8";release_year==1993;source=="TMDB")')# is equivalent toexpected=session.query(Movie).filter(or_(Movie.rating==8,Movie.release_year==1993,Movie.source=='TMDB'))<and>predicatesactual=session.query(Movie).with_criteria('and(release_year > 1975; release_year < 2001)')# is equivalent toexpected=session.query(Movie).filter(and_(Movie.release_year>1975,Movie.release_year<2001))likeandilikepredicatesactual=session.query(Movie).with_criteria('or(title like "THE"; original_title ilike "THE")')# is equivalent toexpected=session.query(Movie).filter(or_(Movie.title.like('%THE%'),Movie.original_title.ilike("%THE%")))notlikepredicateactual=session.query(Movie).with_criteria('and(title notlike "the"; release_year > 1990)')# is equivalent toexpected=session.query(Movie).filter(and_(Movie.title.notlike('%the%'),Movie.release_year>1990))inandnotinpredicateactual=session.query(Movie).with_criteria('release_year in (1979, 2001))')# is equivalent toexpected=session.query(Movie).filter(Movie.release_year.in_((1979,2001)))andactual=session.query(Movie).with_criteria('release_year notin (1979, 2001))')# is equivalent toexpected=session.query(Movie).filter(Movie.release_year.notin_((1979,2001)))IS NULLandIS NOT NULLpredicatesactual=session.query(Movie).with_criteria('original_title == null)')# is equivalent toexpected=session.query(Movie).filter(Movie.original_title.is_(None))andactual=session.query(Movie).with_criteria('original_title != null)')# is equivalent toexpected=session.query(Movie).filter(Movie.original_title.isnot(None))startswithandendswithpredicatesFollowing queries are equivalentsactual=session.query(Movie).with_criteria('title startswith "The")')# is equivalent toexpected=session.query(Movie).filter(Movie.title.startswith("The"))andactual=session.query(Movie).with_criteria('title endswith "Park")')# is equivalent toexpected=session.query(Movie).filter(Movie.title.endswith("Park"))containspredicateactual=session.query(Movie).with_criteria('or(original_title contains "The";title contains "the")')# is equivalent toexpected=session.query(Movie).filter(or_(Movie.original_title.contains('The'),Movie.title.contains('the')))datetimeliteralsFollowing queries are equivalentsnow=datetime.datetime.now().isoformat()actual=session.query(Movie).with_criteria('created_datetime<datetime("{}")'.format(now))# is equivalent toexpected=session.query(Movie).filter(Movie.created_datetime<datetime.datetime.fromisoformat(now))Joins and aliased entitiesJoins are supported as well. Following queries are equivalents:actual=session.query(Movie).join(Genre).with_criteria('name=="War"')# is equivalent toexpected=session.query(Movie).join(Genre).filter(Genre.name=='War')ApiQL supportsaliasedentities:kind=aliased(Genre,name='kind')actual=session.query(Movie).join(kind).with_criteria('kind.name=="War"')# is equivalent toexpected=session.query(Movie).join(kind).filter(kind.name=='War')Complete API example with Flask-SQLAlchemyApiQL is technology agnostic whenever possible and can be used with all popular python web frameworks (Flask, Bottle, Django etc.).(side note: ApiQL reference implementation fully supports Flask-SQLAlchemy extension as well).Here's is a simple Flask API with ApiQL filtering (assumingMovieandGenreclasses arejsonserializable)fromapiql.backends.sqlalchemy.ormimportwith_criteria# ...@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')returnjsonify(Movie.query.join(Genre).with_criteria(criteria).all())Now we can filter resources with ApiQL:wget -q -O - 'localhost:5000/api/movies?filter=and(or(title like "Python";original_title like "Python");source=="TMDB")'Note: that we can use empty string when API consumer do not specify filtering query.This query:wget -q -O - 'localhost:5000/api/movies'will return all, unfiltered resources.Whitelisting API attributesIn most cases API providers gives only limited access to attributes consumers can use. ApiQL supports this capability bywhitelistingwhich attributes can be accessed in queries (this does not change however what attributes are exposed in resources).Whitelists can be enabled withwhitelistedQueryextension method.By default all resource attributes are are whitelisted.Whitelisting only specific attributesApiQLjustbuilder will only whitelist explicitly specified resource attributes.fromapiql.backends.sqlalchemy.ormimportwith_criteria,whitelistedfromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixed@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')query=Movie.query.join(Genre).whitelisted(just((Movie.title,Movie.release_year))).with_criteria(criteria)returnjsonify(query.all())Now, this query:wget -q -O - 'http://localhost:5000/api/movies?filter=release_year==2001'will work just fine, as Movie.release_year is whitelisted,however, this call:wget -q -O - 'http://localhost:5000/api/movies?filter=rating=="8.0"'will fail with:ValueError: Invalid query attribute: 'rating'.Whitelisting all attributeseverythingbuilder whitelistsallentity (or entities) attributes. This is default behavior when whitelist is not specified. ApiQL engine will scan allQueryentities, and whitelist all attributes.fromapiql.backends.sqlalchemy.ormimportwith_criteria,whitelistedfromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixed@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')query=Movie.query.join(Genre).whitelisted(everything(Movie)).with_criteria(criteria)returnjsonify(query.all())Note, that in this case, onlyMovieattributes are whitelisted, while allGenreattributes are not.In this case:wget -q -O - 'http://localhost:5000/api/movies?filter=rating=="8.0"'will work as expected. However, this query:wget -q -O - 'http://localhost:5000/api/movies?filter=name=="Sci-Fi"'will fail again.Whitelisting all attributes, except specified set of attributeseverything_butbuilder whitelists all attributes, except those specified inbutclause.fromapiql.backends.sqlalchemy.ormimportwith_criteria,whitelistedfromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixed@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')query=Movie.query.join(Genre).whitelisted(everything_but(entities=Movie,but=Movie.id)).with_criteria(criteria)returnjsonify(query.all())Now, this call will fail asMovie.idis not whitelisted:wget -q -O - 'http://localhost:5000/api/movies?filter=id==1'Merging whitelistsFinallymergedwhitelist builder can combine (merge) two whitelists. Following example:fromapiql.backends.sqlalchemy.ormimportwith_criteria,whitelistedfromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixed@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')query=Movie.query.join(Genre).whitelisted(merged([everything(Movie),just(Genre.name)])).with_criteria(criteria)returnjsonify(query.all())will whitelist all attributes fromMovieand justGenre.namefromGenre.Prefixed attributesQuery attributes can be prefixed to be more consumer friendly. For instance, in above examples,nameattribute will resolve toGenre.name(we don't have column name collision here betweenMovieandGenre). However from consumer perspective it would be much elegant to map this attribute to something more obvious;prefixeddoes exactly this. Here's an example:fromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixedactual=session.query(Movie).join(Genre)\.whitelisted(everything(Movie,prefixed('genre',Genre)))\.with_criteria('rating=="9";genre.name=="War"')# is equivalent toexpected=session.query(Movie).filter(Movie.rating=="9")\.join(Genre).filter(Genre.name=='War')Mapped attributesSometimes we would like to expose query attribute under different name (for example we would like to keep backward contract compatibility).mappedfunction is does just for this.Let's say we would like to mapGenre.nameto query attributekind, so we can use nicer queries likekind=="War"fromapiql.backends.sqlalchemy.ormimportwith_criteria,whitelistedfromapiql.backends.sqlalchemy.orm.whitelistimporteverything,everything_but,just,mapped,prefixed@app.route("/api/movies",methods=["GET"])defmovies():criteria=request.args.get('filter','')query=Movie.query.join(Genre).whitelisted(just(mapped('kind',Genre.name))).with_criteria(criteria)returnjsonify(query.all())