Package: pygeofilter
pygeofilterpygeofilter is a pure Python parser implementation of OGC filtering standardsFeaturesParsing of several filter encoding standardsCQL as defined in CSW 2.0CQL JSON as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)JSON Filter Expressions (JFE)Soon:CQL Text as defined in OGC API - Features - Part 3: Filtering and the Common Query Language (CQL)FESSeveral backends includedDjangoSQLAlchemy(Geo)PandasNative Python objectsInstallationThe package can be installed via PIP:pipinstallpygeofilterSome features require additional dependencies. This currently only affects the backends. To install these, the features have to be listed:# for the Django backendpipinstallpygeofilter[backend-django]# for the sqlalchemy backendpipinstallpygeofilter[backend-sqlalchemy]# for the native backendpipinstallpygeofilter[backend-native]Usagepygeofilter can be used on various levels. It provides parsers for various filtering languages, such as ECQL or CQL-JSON. Each parser lives in its own sub-package:>>>frompygeofilter.parsers.ecqlimportparseasparse_ecql>>>filters=parse_ecql(filter_expression)>>>frompygeofilter.parsers.cql_jsonimportparseasparse_json>>>filters=parse_json(filter_expression)Each parser creates an abstract syntax tree (AST) representation of that filter expression and thus unifies all possible languages to a single common denominator. All possible nodes are defined as classes in thepygeofilter.astmodule.InspectionThe easiest way to inspect the resulting AST is to use theget_reprfunction, which returns a
nice string representation of what was parsed:>>>filters=pygeofilter.parsers.ecql.parse('id = 10')>>>print(pygeofilter.get_repr(ast))ATTRIBUTEid=LITERAL10.0>>>>>>>>>filter_expr='(number BETWEEN 5 AND 10 AND string NOT LIKE\'%B\') OR INTERSECTS(geometry, LINESTRING(0 0, 1 1))'>>>print(pygeofilter.ast.get_repr(pygeofilter.parse(filter_expr)))((ATTRIBUTEnumberBETWEEN5AND10)AND(ATTRIBUTEstringNOTLIKE'%B'))OR(INTERSECTS(ATTRIBUTEgeometry,Geometry(geometry={'type':'LineString','coordinates':((0.0,0.0),(1.0,1.0))})))EvaluationA parsed AST can then be evaluated and transformed into filtering mechanisms in the required context. Usually this is a language such as SQL or an object-relational mapper (ORM) interfacing a data store of some kind.There are a number of pre-defined backends available, where parsed expressions can be applied. For the moment this includes:Djangosqlalchemy(Geo)PandasPure Python object filteringThe usage of those are described in their own documentation.pygeofilter provides mechanisms to help building such an evaluator (the included backends use them as well). TheEvaluatorclass allows to conveniently walk through an AST depth-first and build the filters for the API in question. Only handled node classes are evaluated, unsupported ones will raise an exception.Consider this example:frompygeofilterimportastfrompygeofilter.backends.evaluatorimportEvaluator,handlefrommyapiimportfilters# <- this is where the filters are created.# of course, this can also be done in the# evaluator itself# Evaluators must derive from the base class `Evaluator` to workclassMyAPIEvaluator(Evaluator):# you can use any constructor as you needdef__init__(self,field_mapping=None,mapping_choices=None):self.field_mapping=field_mappingself.mapping_choices=mapping_choices# specify the handled classes in the `handle` decorator to mark# this function as the handler for that node class(es)@handle(ast.Not)defnot_(self,node,sub):returnfilters.negate(sub)# multiple classes can be declared for the same handler function@handle(ast.And,ast.Or)defcombination(self,node,lhs,rhs):returnfilters.combine((lhs,rhs),node.op.value)# handle all sub-classes, like ast.Equal, ast.NotEqual,# ast.LessThan, ast.GreaterThan, ...@handle(ast.Comparison,subclasses=True)defcomparison(self,node,lhs,rhs):returnfilters.compare(lhs,rhs,node.op.value,self.mapping_choices)@handle(ast.Between)defbetween(self,node,lhs,low,high):returnfilters.between(lhs,low,high,node.not_)@handle(ast.Like)deflike(self,node,lhs):returnfilters.like(lhs,node.pattern,node.nocase,node.not_,self.mapping_choices)@handle(ast.In)defin_(self,node,lhs,*options):returnfilters.contains(lhs,options,node.not_,self.mapping_choices)defadopt(self,node,*sub_args):# a "catch-all" function for node classes that are not# handled elsewhere. Use with caution and raise exceptions# yourself when a node class is not supported....# ...further ast handlings removed for brevityTestingFor testing, several requirements must be satisfied. These can be installed, via pip:pipinstall-rrequirements-dev.txt
pipinstall-rrequirements-test.txtThe functionality can be tested usingpytest.python-mpytestDockerTo execute tests in Docker:docker build -t pygeofilter/test -f Dockerfile-3.9 .
docker run --rm pygeofilter/testBackendsThe following backends are shipped withpygeofilter. Some require additional dependencies, refer to theinstallationsection for further details.DjangoFor Django there is a default backend implementation, where all the filters are translated to the
Django ORM. In order to use this integration, we need two dictionaries, one mapping the available
fields to the Django model fields, and one to map the fields that usechoices. Consider the
following example models:fromdjango.contrib.gis.dbimportmodelsoptional=dict(null=True,blank=True)classRecord(models.Model):identifier=models.CharField(max_length=256,unique=True,null=False)geometry=models.GeometryField()float_attribute=models.FloatField(**optional)int_attribute=models.IntegerField(**optional)str_attribute=models.CharField(max_length=256,**optional)datetime_attribute=models.DateTimeField(**optional)choice_attribute=models.PositiveSmallIntegerField(choices=[(1,'ASCENDING'),(2,'DESCENDING'),],**optional)classRecordMeta(models.Model):record=models.ForeignKey(Record,on_delete=models.CASCADE,related_name='record_metas')float_meta_attribute=models.FloatField(**optional)int_meta_attribute=models.IntegerField(**optional)str_meta_attribute=models.CharField(max_length=256,**optional)datetime_meta_attribute=models.DateTimeField(**optional)choice_meta_attribute=models.PositiveSmallIntegerField(choices=[(1,'X'),(2,'Y'),(3,'Z')],**optional)Now we can specify the field mappings and mapping choices to be used when applying the filters:FIELD_MAPPING={'identifier':'identifier','geometry':'geometry','floatAttribute':'float_attribute','intAttribute':'int_attribute','strAttribute':'str_attribute','datetimeAttribute':'datetime_attribute','choiceAttribute':'choice_attribute',# meta fields'floatMetaAttribute':'record_metas__float_meta_attribute','intMetaAttribute':'record_metas__int_meta_attribute','strMetaAttribute':'record_metas__str_meta_attribute','datetimeMetaAttribute':'record_metas__datetime_meta_attribute','choiceMetaAttribute':'record_metas__choice_meta_attribute',}MAPPING_CHOICES={'choiceAttribute':dict(Record._meta.get_field('choice_attribute').choices),'choiceMetaAttribute':dict(RecordMeta._meta.get_field('choice_meta_attribute').choices),}Finally we are able to connect the CQL AST to the Django database models. We also provide factory
functions to parse the timestamps, durations, geometries and envelopes, so that they can be used
with the ORM layer:frompygeofilter.backends.djangoimportto_filterfrompygeofilter.parsers.ecqlimportparsecql_expr='strMetaAttribute LIKE\'%parent%\'AND datetimeAttribute BEFORE 2000-01-01T00:00:01Z'ast=parse(cql_expr)filters=to_filter(ast,mapping,mapping_choices)qs=Record.objects.filter(**filters)SQLpygeofilterprovides a rudimentary way to create an SQLWHEREclause from an AST. The following example shows this usage in conjunction with the OGRExecuteSQLfunction:fromosgeoimportogrfrompygeofilter.backends.sqlimportto_sql_wherefrompygeofilter.parsers.ecqlimportparseFIELD_MAPPING={'str_attr':'str_attr','maybe_str_attr':'maybe_str_attr','int_attr':'int_attr','float_attr':'float_attr','date_attr':'date_attr','datetime_attr':'datetime_attr','point_attr':'GEOMETRY',}FUNCTION_MAP={'sin':'sin'}# parse the expressionast=parse('int_attr > 6')# open an OGR DataSourcedata=ogr.Open(...)# create the WHERE clause, field and function mappings must be providedwhere=to_sql_where(ast,FIELD_MAPPING,FUNCTION_MAP)# filter the DataSource to get a result Layerlayer=data.ExecuteSQL(f"""SELECT id, str_attr, maybe_str_attr, int_attr, float_attr, date_attr, datetime_attr, GEOMETRYFROM layerWHERE{where}""",None,"SQLite")Note that it is vital to specify theSQLitedialect as this is the one used internally.:warning: Input values arenotsanitized/separated from the generated SQL text. This is due to the compatibility with the OGR API not allowing to separate the SQL from the arguments.OptimizationThis is a special kind of backend, as the result of the AST evaluation is actually a new AST. The purpose of this backend is to eliminate static branches of the AST, potentially reducing the cost of an actual evaluation for filtering values.What parts of an AST can be optimized:Arithmetic operations of purely static operandsAll predicates (spatial, temporal, array,like,between,in) if all of the operands are already staticFunctions, when passed in a special lookup table and all arguments are staticAndandOrcombinators can be eliminated if either branch can be predictedWhat cannot be optimized are branches that contain references to attributes or functions not passed in the dictionary.The following example shows how a static computation can be optimized to a static value, replacing the whole branch of the AST:>>>importmath>>>frompygeofilterimportast>>>frompygeofilter.parsers.ecqlimportparse>>>frompygeofilter.backends.optimizeimportoptimize>>>>>>root=parse('attr < sin(3.7) - 5')>>>optimized_root=optimize(root,{'sin':math.sin})>>>print(ast.get_repr(root))ATTRIBUTEattr<((sin(3.7))-5)>>>print(ast.get_repr(optimized_root))ATTRIBUTEattr<-5.529836140908493