Package: iguala
Iguala: non-linear pattern matching for Python's objects, or a regexp-like for objectsIguala - from spanish "igual a", "equivalent to" or "iguala", "make it equivalent" - is a non-linear pattern matcher for Python's objects.
It is heavily inpired by query in graphs, term rewriting (tomor pattern matching inrascal) and structural pattern matching.
The goal of this project is to provide an easy to use, but powerful, library for matching complex objects.
Patterns are defined in a declarative way and can match complex objects and extract data from them.
They support:logical composition ("or"),wildcards and variables,the reuse of variable accross patterns to express equality,wildcards and variables in collections,negative patterns,pattern/matchers compositions,composite and recursive paths,yield all combinations found when a pattern matches against an object,matching over dictionary the same way as for classes/objects (more or less),conditional matchers (you can capture variable and/or the tested object to test a condition),matcher generators (you can capture variables and/or the tested object to produce a new matcher),regex matchers,compatibility with thematchof Python (using few tricks, documented below).More operators/matchers will arriveThis implementation is derived fromMoTion, implemented in Smalltalk and based on work initiated withAless Hosryduring her PhD.Quick exampleHere is the example of a pattern that will check if an object is an instance ofMyObjectwith the same value foratt1andatt2.NOTE: in this document, the termmatcherandpatternare used to refer to the same thing, but a matcher matches a single small entity while a pattern defines a composition of matchers (or a single matcher).# we consider that "MyObject" is an already known type# and "instance" and instance of it with att1 = att2 = 4 and name = 'foo'fromigualaimportmatch,is_notpattern=match(MyObject)[# must be an instance of MyObject"att1":"@value",# with att1 stored in "value""att2":"@value",# and att2 with the same value as the one from att1"name":is_not("bar")@"name"# and name is not "bar", store the name in "name"]print(pattern.match(instance))# displays: <True: [{'value': 4, 'name': 'foo'}]># An equivalent Python code would be:iftype(instance)==MyObjectandinstance.att1==instance.att2andinstance.name!="bar":value=instance.att1name=instance.name# As an alternative syntax, here is how you can write the same pattern# You can choose the one you prefer, there is pros and cons with bothpattern=match(MyObject)%{# must be an instance of MyObject"att1":"@value",# with att1 stored in "value""att2":"@value",# and att2 with the same value as the one from att1"name":is_not("bar")@"name"# and name is not "bar", store the name in "name"}InstallationpipinstalligualaSyntax, operators and special charactersFor each pattern/matcher, there is a set of operators and special characters that controls the way data are matched.
For a most comprehensive way of how to use all of this, please refer to the sections belows that present small tutorials about how to match values against objects and dicts.
Here is a list of all currently supported operators.Paths operators for objects and dictionariesPatterns over objects and dictionary supports the sub-patterns.
Each sub-patterns are connected by a path.
A path represent the way to navigate from a pattern to others.There is different kind of paths:named/direct paths: they express a direct navigation through a relationship,creating a direct path is done by using directly the name of the relation, e.g:fooorbar.composed paths: they express the navigation of different relationships one after each other,creating a composed path is done by using the>operator between two names, e.g:foo>barto express, 'first navigatefoothenbar.named recursive paths: they express the recursive navigation of a relationship (unbound depth),creating a named rec. path is done by using the*or+operator after a name, e.g:foo*expresses thatfooneeds to be followed 0 or many times andfoo+expresses thatfooneeds to be followed 1 or many times.children recursive paths: they express the recursive navigation of all "instance variable" of an object.creating a children rec. path is done by using*alone, e.g:*means all the "children" (the instance variable of the object/keys of the dict) and their children.Those operators can be composed with>.
For examples:bar>foo>*means "barthenfoothen all the children recursively"*>foomeans "all the children recursively thenfoo" (iffooexists for each object)child*>namemeans "followchildrecursively and getnameeach time"...Wildcards/variablesWildcards/variables stores information and checks if the same information appears many times.variable are strings starting by@, e.g:@x,@name...single@alone is considered as an anonymous variable, it is useful to ensure the presence of something without storing any data.variable starting by*represent a collection of elements in sequences: e.g:*xor*y, they are used in list/sequence matching (see sections below)....is used as an anonymous variable for collections of elements in sequences.Matchers/pattern operatorsThere is few pattern operators.match(...)with a type as parameter matches exactly a type, e.g:match(A) % {}means, match an instance ofA(but not subclasses).~used in front of an object matcher expresses "and all its subclasses", e.g:~match(object) % {}means, match an instance ofobjector from its subclasses.matcher[...]with slices as indexes which expresses the properties of an object, e.g:match(A)['name': 'foo']means, match an instance ofAwhere thenameequalsfoo.%with a dictionnary on its right expresses the properties of an object, e.g:match(A) % {'name': 'foo'}means, match an instance ofAwhere thenameequalsfoo.@stores a data into a dedicated name, e.g:match(A)['name': 'foo'] @ 'inst'means, match an instance ofAwhere thenameequalsfooand store the instance ofAininstfor further usage.is_not(...)expresses a negation (needs to be importedfrom iguala import is_not), e.g:match(A)['name': is_not('foo')], means match an instance ofAwhere thenameis not equal tofoo,regex(...)expresses a regular expression matcher (needs to be importedfrom iguala import regex). The regex to match needs to be passed as a string, e.g:match(A)['name': regex('[A-Z].*')], means match an instance ofAwhere thenamematches the regex[A-Z].*.This matcher supports an additional operator>>that is used to store the matching result for further usage. This mecomes really handy to get matched groups (especially if named match group are used), e.g:match(A)['name': regex('[A-Z].*') >> 'match_result']will store the "match" object obtained during the regex matching operation under the labelmatch_result. This variable will be accessible as all variables, in the result procuded byiguala.The same behavior as describe above can be achieved without using the>>by passing an extra argument toregex(...), e.g:match(A)['name': regex('[A-Z].*', label='match_result')]. Using the operator or not is a matter of taste, the effect is exactly the same.range(...), if you use therange(...)constructor (from builtins), a special "range matcher" is created, e.g:match(A)['x': range(0, 5)]means, match an instance ofAwherexis in the range[0..4].|expresses a logical "or" between two patterns, e.g:match(A)['name': is_not(m('foo') | 'bar')], means match an instance ofAwherenameis neitherfoonorbar. In this example,mis a renaming of theas_matcherfunction made this way:from iguala import as_matcher as m.Collections patternsHere is a list of some examples of patterns that can be applied to collections:[], means empty collection[3]means a collection with only one value:3[3, 'r']means a collection with only two values:3and'r'[3, ...]means3is first element[..., 3]means3is last element[..., '@x']means a last element stored inx[..., '@x', ...]means an element stored inx(will match all the element of a collection one after each others)[..., '@x', '@y']means the two last element stored inxandy[..., '@x', ..., '@y']means an element stored inxwith the last element stored iny[..., '@x', ..., '@x']means a collection that have an element that is equal to the last element['@x', ..., '@x']means a collection where the first and the last element are the same[..., '@x', '@x', ..]means a collection that have two times the same element that follow each other[..., '@x', ..., is_not('@x'), ..]means a collection where two elements that are not the same (a collection where all elements are different)is_not([..., '@x', ..., is_not('@x'), ...])means a collection where there is no elements that are not the same (a collection where all elements are the same)Lambda based matchersLambdas are used to express patterns over captured variables:lambda VAR1, VAR2, ....: SOMETHINGS WITHS VARSis a matcher generator.
Matcher generators uses captured variable to generate new matchers that are executed when all necessary variables have been captured, e.g:match(A)['x': '@x', 'y': lambda x: x + 1]means, match an instance ofAthat have an attributexand an attributeythat is equals tox + 1.cond(lambda ....)is a condition matcher (needs to be importedfrom iguala import cond).
Condition matchers uses captured variable to execute a function and use the result as matching result. Consequently, the return type of the function must be a boolean, e.g:match(A)['x': '@x', 'y': cond(lambda x, __self__: x ==  __self__ + 1)]means, match an instance ofAthat have an attributexand an attributeythat is equals tox + 1.__self__is a meta-variable that can be passed as arguments of the matcher generator or conditional matcher. This variable resolves to the object currently matched.Matcher generators and conditional matchers also works with sequence matchers, negative matchers, range matcher, regex matcher...etc.
Here is some examples:[..., '@x', lambda x: x + 1, ...]means a collection where one element is followed by its successor.[..., '@x', is_not(lambda x: x + 1), ...]means a collection where one element is not followed by its successor.is_not([..., '@x', is_not(lambda x: x + 1), ...])means a collection where there is no element that is not followed by its successor (a collection that is sorted).match(A)['x': '@x', 'y': lambda x: range(0, x + 1)]means match an instance ofAwhich has anxvalue and ayvalue contained in the[0..x]interval.NOTE: Argument names of the function used for the matcher generator or the conditional matcher have to match the name of variables defined in the pattern.
If other names are used,igualawill ignore the matcher, but will generate a warning message stating what are the missing variables and their positions in the pattern.Walkthrough - Draw me a pattern on an ObjectTo see how to match information from an object, let's create three of two kind.
We will consider that they already exists somewhere in memory.
We will then express patterns and see if those data (a1,a2,b) are matched by the patterns.fromdataclassesimportdataclass@dataclassclassA(object):x:inty:intl:list[int]@dataclassclassB(object):x:inta1=A(x=3,y=4,l=[1,2,2,3,4,3,5])a2=A(x=4,y=4,l=[2])b=B(x=4)Now let's define a first pattern.fromigualaimportmatch# Simple patternpattern=match(A)%{}# answers the question it's an instance of A?This pattern will check if the data is actually an instance ofA.
The%operator with the empty dictionnary after expresses the fact that we don't want to express any properties overA.And now, we see if it matches:print(pattern.match(a1))# displays: <True - [{}]>print(pattern.match(a2))# displays: <True - [{}]>print(pattern.match(b))# displays: <False - []>No suprises,a1anda2are instances ofAwhilebis not.Let's now see if those objects are instances ofAwith an attributezthat equals 4, then if they own an attributexequals to 4.pattern=match(A)['z':4]# is it an instance of A with 'z' == 4?print(pattern.match(a1))# and a2, b# displays: <False - []>, they don't have 'z' propertiespattern=match(A)['x':4]# is it an instance of A with 'z' == 4?print(pattern.match(a1))# displays: <False - []>print(pattern.match(a2))# displays: <True - [{}]>Now, let's deconstruct to get the value ofx.
We will express in our pattern that the data should be an instance ofAwith an attributexand that we want to store the value ofxin a variable namedvalue.pattern=match(A)[# is it an instance of A?'x':'@value'# with an attribute "x"? (stored in "value")]result=pattern.match(a1)print(result)# displays: <True - [{'value': 3}]>print(result.bindings[0]['value'])# displays: 3print(pattern.match(a2))# displays: <True - [{'value': 4}]>print(result.bindings[0]['value'])# displays: 4Each right side of the property dictionnary that starts with an@means that it's a variable.
If the name is found again in the pattern, then, it means that the data needs to have the same value for those variable in those positions.Let's use that to check now if each data is an instance ofAwherexandyhave the same value.pattern=match(A)[# is it an instance of A?'x':'@value'# with an attribute "x"? (stored in "value")'y':'@value'# with an attribute "y" with the same value as "x"]result=pattern.match(a1)print(result)# displays: <False - []>print(pattern.match(a2))# displays: <True - [{'value': 4}]>Let's check now if we have an instance ofAwith a2in the collectionl.pattern=match(A)[# is it an instance of A?'l':2# does "l" contains a 2?]print(pattern.match(a1))# displays: <True - [{}, {}]>, there is two 2 in "l", so two combination, this information is returned by the matcherprint(pattern.match(a2))# displays: <True - [{}]>, here there is only one 2 in "l"Let's try more precise questions over the collectionl:## Patterns over collections#pattern=match(A)[# is it an instance of A?'l':[2]# does "l" contains only a 2?]print(pattern.match(a1))# displays: <False: []>print(pattern.match(a2))# displays: <True: [{}]>pattern=match(A)[# is it an instance of A?'l':[...,2]# does the last element of "l" is 2?]print(pattern.match(a1))# displays: <False: []>print(pattern.match(a2))# displays: <True: [{}]>pattern=match(A)[# is it an instance of A?'l':[...,'@value']# does "l" has a last element?]print(pattern.match(a1))# displays: <True: [{'value': 5}]>print(pattern.match(a2))# displays: <True: [{'value': 2}]>pattern=match(A)[# is it an instance of A?'l':[...,'@value','@value',...]# does "l" has two times the same element that follow each other?]print(pattern.match(a1))# displays: <True: [{'value': 2}]>print(pattern.match(a2))# displays: <False: []>pattern=match(A)[# is it an instance of A?'l':[...,'@value',...,'@value',...]# does "l" has at least two times the same element?]print(pattern.match(a1))# displays: <True - [{'value': 2}, {'value': 3}]>print(pattern.match(a2))# displays: <False: []>pattern=match(A)[# is it an instance of A?'l':["*l1",'@value',"*l2",'@value',"*l3"]# does "l" has at least two times the same element? and store the collections around]print(pattern.match(a1))# input for l was: [1, 2, 2, 3, 4, 3, 5]# displays: <True - [#    {'l1': [1], 'value': 2, 'l2': [], 'l3': [3, 4, 3, 5]},#    {'l1': [1, 2, 2], 'value': 3, 'l2': [4], 'l3': [5]}# ]>Those are only few examples of what you can express over collections.
Obviously, it's possible to compose patterns and to express patterns elements in collections.fromigualaimportas_matchera3=A(x=3,y=4,l=[])col=[a1,a2,b,a3,b]# does the collection contains two instances of A that have the same value for x# and store the nodes in 'inst1' and 'inst2'matcher=as_matcher([...,match(A)['x':'@x']@'inst1',...,match(A)['x':'@x']@'inst2',...])print(matcher.match(col))# displays: <True - [{'inst1': A(x=3, y=4, l=[1, 2, 2, 3, 4, 3, 5]), 'x': 3, 'inst2': A(x=3, y=4, l=[])}]>The@operator stores the results when a combination is found.
The example above shows us that there is only one combination where the data matches the pattern.Walkthrough - Draw me a pattern for an ASTThis small tutorial shows how to match information from the Python's AST.
Please note that the library is not limited to Pyton's AST, but is supposed to work with any Python object.
This tutorial also suppose that you are used to the Python's AST.In a first place, let's get a beautiful simple AST treefromastimport*# What we parsetree=parse("""class A(object):Z = 0def __init__(self, x, y=3):self.x = xself.y = yself.z = 4class InnerCls(object):def __init__(self):self.inner_x = 1class B(object):def __init__(self, w):if w > 0:self.w = welse:self. w = -1""")Do we match something that has this shape?Let's try to see if this module contains classes, which means: does thetreevariable have the shape of a Module that contains Classes in this body?
Let's define the pattern to check that:# Pattern example 1pattern=match(Module)[# we want a Module'body':match(ClassDef)# that contains a ClassDef in 'body']This pattern expresses that we want to match aModuleusingmatch(Module), and in thebodyrelation of thisModule, must contain aClassDef.
To express the content of a class instance, we use the collection acces notation where each index is a slice with the start of the slice that is the name of the relation and the value is against what the value needs to be matched (the stop of the slice).
The key (start) is considered as a "path", we will see later what it means, while the value is a matcher.
In this case,bodywill yield a collection of ast nodes forModule.
Even if the path will resolve as a collection, as we use a simple matcher here, the semantic is equivalent to "contains".
Consequently, we can read this as doesbodycontains aClassDef.Now that we have our matcher, we "execute it" against thetreeobject.result=pattern.match(tree)print(result)# displays: <True - [{}, {}]>print(result.is_match)# displays: Trueprint(result.contexts)# displays: [<...Context object at 0x7f3736ae0280>, <...Context object at 0x7f3736ae0310>]TheTrue - [{}, {}]output means that yes, the pattern matches the data (True) and that it found exactly two combinations.
The matcher will always gives as result all theContextwhich are the equivalent of a matching combination.
We can conclude here that ourtreevariable is aModulethat owns classes.Deconstruct/extract information from a patternKnowing if an object matches against a pattern is nice, but it's even better to be able to gather some data.
To have more details about what was matched by the pattern, we will introduce variables/wildcards.# Pattern example 2pattern=match(Module)[# we want a Module'body':match(ClassDef)["name":"@name"]# that contains a ClassDef in 'body' and has a "name"]# Let's match the "tree"result=pattern.match(tree)print(result.is_match)# Trueprint(result.bindings)# displays: [{'name': 'A'}, {'name': 'B'}]Obviously, we only have the classes that are directly underbodyin the module.
If we want all the modules, whereever there are defined, we will use instead the children recursive path operator.NOTE: if you execute this code in IPython, it could be long,parse(...)seems to behave a little bit different in interactive environement.# Pattern exmaple 3pattern=match(Module)[# we want a Module'*':match(ClassDef)["name":"@name"]# that contains a ClassDef in 'body' and has a "name"]# Let's match the "tree"result=pattern.match(tree)print(result.is_match)# Trueprint(result.bindings)# displays: [{'name': 'A'}, {'name': 'B'}, {'name': 'InnerCls'}]The*performs a "wide first" traversal of all the data it encounters to give a feeling of "levels".
Consequently,Awill always be followed byBhere because they appear at the "same level".Now, we will go further by trying to match a module that contains classes that contains a__init__method and the__init__method should have assignment with the formself.XXX = SOMETHING.fromigualaimportmatch# defines the pattern we want to look for# we want a Modulepattern=match(Module)[# that owns in all its children, recursively (i.e: somewhere, at any depth)# a ClassDef instance'*':match(ClassDef)['name':'@name',# where name is equivalent to "name" (store the node)'body':match(FunctionDef)[# that has a FunctionDef in body'name':'__init__',# that is named "__init__"'body>*':match(Assign)[# and that has an Assign in it's body, somewhere (at any depth)'targets>value>id':'self',# where the target is "self"'targets>attr':'@attr',# where the attr is equivalent to "attr" (store the node)]]]]result=matcher.match(tree)# Find all combinationsprint(result)print(result.is_match)# displays Trueprint(result.bindings)# displays# [{'name': 'A', 'attr': 'x'},#  {'name': 'A', 'attr': 'y'},#  {'name': 'A', 'attr': 'z'},#  {'name': 'B', 'attr': 'w'},#  {'name': 'B', 'attr': 'w'},#  {'name': 'InnerCls', 'attr': 'inner_x'}]And finally, as last pattern for this tutorial, we will try to match modules that have classes (at any level) which own an__init__method and that have instance variables assigned to a value that have the same name than the instance variable they are assigned to and which is coming from an argument of__init__.fromigualaimportmatch# defines the pattern we want to look for# we want a Modulepattern=match(Module)[# that owns in all its children, recursively (i.e: somewhere, at any depth)# a ClassDef instance'*':match(ClassDef)['name':'@name',# where name is equivalent to "name" (store the node)'body':match(FunctionDef)[# that has a FunctionDef in body'name':'__init__',# that is named "__init__"'args>*':match(arg)['*':'@attr'],# among the args, there is one that a field equals to "attr"'body>*':match(Assign)[# and that has an Assign in it's body, somewhere (at any depth)'targets>value>id':'self',# where the target is "self"'targets>attr':'@attr',# where the attr is equivalent to "attr" (store the node)'value>id':"@attr",# and the "id" of the "value" of the assignment as is equivalent to "attr"]]]]result=matcher.match(tree)# Find all combinationsprint(result)print(result.is_match)# displays Trueprint(result.bindings)# displays# [{"name": "A", "attr": "x"},#  {"name": "A", "attr": "y"},#  {"name": "B", "attr": "w"}]Makeigualaworks with Python'smatchsyntaxSince version 3.9, Python owns a specific syntax for structural matching.
In this context, a pattern-matching in Python is a set of different pattern against which an object is tested.
The first pattern that answers "yes" to the question "do you match this object" triggers the execution of an indented block.
There is a kind of overlap between Python's pattern-matching andiguala.
Python's pattern-matching defines a syntax for: (1) defining structural patterns and (2) orchestring those patterns and test them one after the other on an input object (using differentcase).igualais an internal DSL to define structural and deep patterns only.
The patterns you can define withigualaare sometimes equivalent, but differs as they allow you to define patterns that returns many results.
Also, currently, there is no way of mixing definition of patterns between Python andiguala.
However, it's possible to useigualapatterns with thecasesyntax, but it requires a little trick.Defining a set ofigualapatternsThe "problem" with Python's pattern-matching syntax is that you cannot reference an external variable.x=3match4:casex:print("The variable x capture all values")To overcome this "limitation" (it's not a limitation, but in our case, it can feel like it), we will define a class with all the patterns we want as class variable.fromigualaimportas_matcherasmclassPatterns(object):case1=match(A)["x":"@x"]case2=match(B)["y":4]case3=m([1,"*value",3,...])# we want that the list starts with 1, then there is a bunch of 3 inside at any positionIntegratingigualapatterns with PythoncasesyntaxThen, from here, we can pass them to thecasesyntax:instance=...# an existing objectmatchinstance:casePatterns.case1asx:print("I matched the first case",x)casePatterns.case2asx:print("I matched the second case",x)casePatterns.case3asx:print("I matched the third case",x)case4:print("I matched the number 4")case_:print("I'm something else")Thematchoperation will try to see which of thecaseactually matches and, if it does, will storeinstanceinxfor the first 3 cases.Accessing the results/bindings of anigualapatternWith the current version, there is no way of accessing all the bindings and results produced by anigualamatcher.
To do so, we need to cheat a little bit more and to ask for anextendedmatch ofinstance.fromigualaimportextendedinstance=...# an existing objectmatchextended(instance):casePatterns.case1asx:o,result=xprint("I matched the first case",result)casePatterns.case2asx:o,result=xprint("I matched the second case",result)casePatterns.case3asx:o,result=xprint("I matched the third case",result)case4:print("I matched the number 4")case_:print("I'm something else")This time, stored inxwill not be theinstanceobject, but a wrapper giving access toothe object that matched (x.o) andresultthe result of the match (x.result).
This wrapper that can be spread.
With that, inresultwe can access now all the combination that made that pattern match.We can see that it is easy to integrateigualapatterns with Python's pattern-matching mecanism, providing thus a way to extend it for not so expensive.
The counterpart of this current solution is that ifextended(...)is used, it's important to remember that the object that will be captured is a wrapper over the tested instance.NOTE: theextendedfunction is still a prototype and can be changed in further versions.
NOTE2: some more syntactic sugar could be added to ease the patterns set definition, but for a first PoC, it's sufficient.