Package: pycond
author: gk
version: 20230212pycond: Lightweight Declarative Condition ExpressionsTable Of ContentsWhatWhyAlternativesMechanicsParsingBuildingStructured ConditionsEvaluationDefault LookupPassing StateDeep Lookup / Nested State / ListsLookup Performance: Prebuilt Deep GettersBest PracticesPrefixed DataAttributes AccessCustom Lookup And Value PassingLazy EvaluationCondition Operators (Comparators)Using Symbolic OperatorsExtending Condition OperatorsNegationnotReversalrevWrapping Condition OperatorsGlobal WrappingCondition Local WrappingCombining OperationsDetailsDebugging LookupsEnabling/Disabling of BranchesBuilding Conditions From TextGrammarAtomic ConditionsNestingTokenizing DetailsFunctioningSeparatorsepApostrophesEscapingBuildingAutoconv: Casting of values into python simple typesContext On DemandLookup ProvidersAccepted SignaturesParametrized Lookup FunctionsNamespaceCachingExtensionsNamed Conditions: QualificationOptionsPartial EvaluationStreaming DataFilteringStreaming ClassificationSelective ClassificationTreating of Booleans (Conditions, Not Names)Asyncronous OperationsAsyncronous FilterWhatYou have a bunch of data, possibly streaming...id,first_name,last_name,email,gender,ip_address
1,Rufe,Morstatt,rmorstatt0@newsvine.de,Male,216.70.69.120
2,Kaela,Scott,scott@opera.com,Female,73.248.145.44,2
(...)... and you need to filter. For now lets say we have them already as list of dicts.You can do it imperatively:foo_users=[uforuinusersif(u['gender']=='Male'oru['last_name']=='Scott')and'@'inu['email']]or you have this module assemble a condition function from a declaration like:frompycondimportmake_filtercond='email contains .de and gender eq Male or last_name eq Scott'is_foo=make_filter(cond)# the built filter function is firstand then apply as often as you need, against varying state / facts / models (...):foo_users = filter(is_foo, users)with roughly the same performance (factor 2-3) than the handcrafted python.In real life performance is oftenbetterthen using imperative code, due topycond'slazy evaluationfeature.WhyWhen the developer can decide upon the filters to apply on data he'll certainly
use Python's excellent expressive possibilities directly, e.g. as shown above
through list comprehensions.But what if the filtering conditions are based on decisions outside of the program's
control? I.e. from an end user, hitting the program via the network, in a somehow serialized form, which is rarely directly evaluatable Python.This is the main use case for this module.AlternativesBut why yet another tool for such a standard job?There is a list of great tools and frameworks where condition parsing is a (small) part of them, e.g.pykeordurableand many in the django world or from SQL statement parsers.1.I just needed a veryslimtool for only the parsing into functions - but this pretty transparent and customizablepycond allows to customizethe list of condition operatorsthe list of combination operatorsthe general behavior of condition operators via global or condition local wrapperstheir namesthe tokenizerthe value lookup functionand ships as zero dependency single module.All evaluation is done viapartialsand not lambdas, i.e. operations can be introspected and debugged very simply, through breakpoints or custom logging operator or lookup wrappers.2.Simplicity of the grammar: Easy to type directly, readable by non
programmers but also synthesisable from structured data, e.g. from a web framework.3.Performance: Good enough to have "pyconditions" used withinstream filters.
With the current feature set we are sometimes a factor 2-3 worse but (due to lazy eval) often better,
compared with handcrafted list comprehensions.MechanicsParsingpycond parses the condition expressions according to a set of constraints given to the parser in thetokenizerfunction.The result of the tokenizer is given to the builder.importpycondaspcexpr='[a eq b and [c lt 42 or foo eq bar]]'cond=pc.to_struct(pc.tokenize(expr,sep=' ',brkts='[]'))print('filter:',cond)# test:data=[{'a':'b','c':1,'foo':42},{'a':'not b','c':1},]filtered=list(filter(pc.make_filter(expr),data))print('matching:',filtered)returncond,len(filtered)Output:filter: [['a', 'eq', 'b', 'and', ['c', 'lt', '42', 'or', 'foo', 'eq', 'bar']]]
matching: [{'a': 'b', 'c': 1, 'foo': 42}]BuildingAfter parsing, the builder is assembling a nested set of operator functions,
combined via combining operators. The functions are partials, i.e. not yet
evaluated - but information about the necessary keys is already available:f,meta=pc.parse_cond('foo eq bar')assertmeta['keys']==['foo']assertf(state={'foo':'bar'})==TrueNote: Themake_filterfunction is actually a convencience function forparse_cond, ignoring that meta information and calling withstate=<filter val>Structured ConditionsOther processes may deliver condition structures via serializable formats (e.g.
json). If you pass such already tokenized constructs to thepycondfunction,
then the tokenizer is bypassed:cond=[['a','eq','b'],'or',['c','in',['foo','bar']]]assertpc.pycond(cond)(state={'a':'b'})==True# json support is built in:cond_as_json=json.dumps(cond)assertpc.pycond(cond_as_json)(state={'a':'b'})==TrueEvaluationThe result of the builder is a 'pycondition', i.e. a function which can be run many times against varying state of the system.
How state is evaluated is customizable at build and run time.Default Lookup"Lookup" denotes the process of deriving the actual values to evaluate, from a given state. Can be simple gets, getattrs, walks into the structure - or arbitrary, via custom lookup functions.The default is togetlookup keys within expressions from an initially emptyStatedict within the module. This isnotthread safe, i.e. not to be used in async or non cooperative multitasking environments.f=pc.pycond('foo eq bar')assertf()==Falsepc.State['foo']='bar'# not thread safe!assertf()==True(pycondis a shortcut forparse_cond, when meta infos are not required).Passing StateUsing a state argument at evaluationisthread safe:assertpc.pycond('a gt 2')(state={'a':42})==Trueassertpc.pycond('a gt 2')(state={'a':-2})==FalseDeep Lookup / Nested State / ListsYou may supply a path seperator for diving into nested structures like so:m={'a':{'b':[{'c':1}]}}assertpc.pycond('a.b.0.c',deep='.')(state=m)==Trueassertpc.pycond('a.b.1.c',deep='.')(state=m)==Falseassertpc.pycond('a.b.0.c eq 1',deep='.')(state=m)==True# convencience argument for string conditions:assertpc.pycond('deep: a.b.0.c')(state=m)==True# This is how you express deep access via structured conditions:assertpc.pycond([('a','b',0,'c'),'eq',1])(state=m)==True# Since tuples are not transferrable in json, we also allow deep paths as list:# We apply heuristics to exclude expressions or conditions:c=[[['a','b',0,'c'],'eq',1],'and','a']f,nfos=pc.parse_cond(c)# sorting order for keys: tuples at end, sorted by len, rest default py sorted:assertf(state=m)==Trueandnfos['keys']==['a',('a','b',0,'c')]The structure may also contain objects, then we use getattribute to get to the next value.deep="."is actually just convience notation for supplying the following "lookup function" (see below):m={'a':{'b':[{'c':1}]}}assertpc.pycond('a.b.0.c',lookup=pc.state_get_deep)(state=m)==TrueLookup Performance: Prebuilt Deep GettersThe value lookup within nested structures can be stored into item and attribute getters (or , alternatively, an evaluated synthesized lookup function), built, when the first item has a matching structure.Upside:Performanceis a few times better compared to when the structure of items is explored each time, as with the 'deep' parameter.Downside: The lookup remains as built for the first structurely matching item. Schematic changes like from a key within a dict to an attribute will not except but deliver always False for the
actual condition value matching.pycond.Getters.state_get_deep2: A list of item and attribute getters is built at first successfull lookup evaluation.pycond.Getters.state_get_evl: An expression like "lambda state=state['a'].b[0]['c']" is built and evaluated, then applied to the items.Fastest way to get to the values at evaluation time.Security: Round brackets within key names are forbidden and deliver always false - but an eval is an eval i.e. potentially evil.These two additional "deep" lookup functions are conveniently made accessible by supplying adeep2ordeep3argument:m={'a':{'b':[{'c':1}]}}# 3 times faster than deep. Safe.assertpc.pycond('a.b.0.c',deep2='.')(state=m)==True# 4 times faster than deep. Eval involved.assertpc.pycond('a.b.0.c',deep3='.')(state=m)==TrueThe evaluation results for the keys are cached. The cache is cleared after 1Mio entries but can be cleared manually viapc.clear_caches()any time before that.Best PracticesLookup keys change all the time, not many items checked for specific key: UsedeepMany items to be checked with same keys, input from untrusted users: Usedeep2Many items to be checked with same keys, input from trusted users: Usedeep3Prefixed DataWhen data is passed through processing pipelines, it often is passed with headers. So it may be useful to pass a global prefix to access the payload like so:m={'payload':{'b':[{'c':1}],'id':123}}assertpc.pycond('b.0.c',deep='.',prefix='payload')(state=m)==TrueAttributes AccessSince version 20210221 we try attributes when objects are not dicts:classMyObj:val={'a':'b'}m={'payload':{'obj':MyObj()}}cond=[['obj.val.a','eq','b']]assertpc.pycond(cond,deep='.',prefix='payload')(state=m)==TrueCustom Lookup And Value PassingYou can supply your own function for value acquisition.Signature: See example.Returns: The value for the key from the current state plus the
compare value for the operator function.# must return a (key, value) tuple:model={'eve':{'last_host':'somehost'}}defmy_lu(k,v,req,user,model=model):print('user check. locals:',dict(locals()))return(model.get(user)or{}).get(k),req[v]f=pc.pycond('last_host eq host',lookup=my_lu)req={'host':'somehost'}assertf(req=req,user='joe')==Falseassertf(req=req,user='eve')==TrueOutput:user check. locals: {'k': 'last_host', 'v': 'host', 'req': {'host': 'somehost'}, 'user': 'joe', 'model': {'eve': {'last_host': 'somehost'}}}
user check. locals: {'k': 'last_host', 'v': 'host', 'req': {'host': 'somehost'}, 'user': 'eve', 'model': {'eve': {'last_host': 'somehost'}}}as you can see in the example, the state parameter is just a convention
forpyconds'[title: default lookup function, fmatch:pycond.py, lmatch:def state_get] < SRC > .Lazy EvaluationThis is avoiding unnecessary calculations in many cases:When an evaluation branch contains an "and" or "and_not" combinator, then
at runtime we evaluate the first expression - and stop if it is already
False.
Same when first expression is True, followed by "or" or "or_not".That way expensive deep branch evaluations are omitted or, when
the lookup is done lazy, the values won't be even fetched:evaluated=[]defmyget(key,val,cfg,state=None,**kw):evaluated.append(key)returnpc.state_get(key,val,cfg,state,**kw)f=pc.pycond('[a eq b] or foo eq bar and baz eq bar',lookup=myget)assertf(state={'foo':42})==False# the value for "baz" is not even fetched and the whole (possibly# deep) branch after the last and is ignored:assertevaluated==['a','foo']print(evaluated)evaluated.clear()f=pc.pycond('[[a eq b] or foo eq bar] and baz eq bar',lookup=myget)assertf(state={'a':'b','baz':'bar'})==True# the value for "baz" is not even fetched and the whole (possibly# deep) branch after the last and is ignored:assertevaluated==['a','baz']print(evaluated)Output:['a', 'foo']
['a', 'baz']Remember that all keys occurring in a condition(which may be provided by the user at runtime) are returned by the condition parser. Means that building of evaluation contexts[can be done](  # context-on-demand-and-lazy-evaluation), based on the data actually needed and not more.Condition Operators (Comparators)All booleanstandardlib operatorsare available by default:frompytest2mdimporthtml_tableastbl# just a table gen.frompycondimportget_opsforkin'nr','str':s='Default supported '+k+' operators...(click to extend)'print(tbl(get_ops()[k],[k+' operator','alias'],summary=s))Default supported nr operators...(click to extend)nr operatoraliasadd+and_&eq==floordiv//ge>=gt>iadd+=iand&=ifloordiv//=ilshift<<=imod%=imul*=ior|=ipow**=irshift>>=is_isis_notisisub-=itruediv/=ixor^=le<=lshift<<lt<mod%mul*ne!=or_|pow**rshift>>sub-truediv/xor^itemgetterlength_hintDefault supported str operators...(click to extend)str operatoraliasattrgetterconcat+containscountOficoncat+=indexOfmethodcallerUsing Symbolic OperatorsBy default pycond uses text style operators.ops_use_symbolicswitches processwide to symbolic style only.ops_use_symbolic_and_txtswitches processwide to both notations allowed.pc.ops_use_symbolic()pc.State['foo']='bar'assertpc.pycond('foo == bar')()==Truetry:# this raises now, text ops not known anymore:pc.pycond('foo eq bar')except:pc.ops_use_symbolic_and_txt(allow_single_eq=True)assertpc.pycond('foo = bar')()==Trueassertpc.pycond('foo == bar')()==Trueassertpc.pycond('foo eq bar')()==Trueassertpc.pycond('foo != baz')()==TrueOperator namespace(s) should be assigned at process start, they are global.Extending Condition Operatorspc.OPS['maybe']=lambdaa,b:int(time.time())%2# valid expression now:assertpc.pycond('a maybe b')()in(True,False)NegationnotNegates the result of the condition operator:pc.State['foo']='abc'assertpc.pycond('foo eq abc')()==Trueassertpc.pycond('foo not eq abc')()==FalseReversalrevReverses the arguments before calling the operatorpc.State['foo']='abc'assertpc.pycond('foo contains a')()==Trueassertpc.pycond('foo rev contains abc')()==Truerevandnotcan be combined in any order.Wrapping Condition OperatorsGlobal WrappingYou may globally wrap all evaluation time condition operations through a custom function:l=[]defhk(f_op,a,b,l=l):l.append((getattr(f_op,'__name__',''),a,b))returnf_op(a,b)pc.run_all_ops_thru(hk)# globally wrap the operatorspc.State.update({'a':1,'b':2,'c':3})f=pc.pycond('a gt 0 and b lt 3 and not c gt 4')assertl==[]f()expected_log=[('gt',1,0.0),('lt',2,3.0),('gt',3,4.0)]assertl==expected_logpc.ops_use_symbolic_and_txt()You may compose such wrappers via repeated application of therun_all_ops_thruAPI function.Condition Local WrappingThis is done through theops_thruparameter as shown:defmyhk(f_op,a,b):returnTruepc.State['a']=1f=pc.pycond('a eq 2')assertf()==Falsef=pc.pycond('a eq 2',ops_thru=myhk)assertf()==TrueUsingops_thruis a good way to debug unexpected results, since you
can add breakpoints or loggers there.Combining OperationsYou can combine single conditions withandand notoror notxorby default.The combining functions are stored inpycond.COMB_OPSdict and may be extended.Do not use spaces for the names of combining operators. The user may use them but they are replaced at before tokenizing time, likeand not->and_not.DetailsDebugging Lookupspycond provides a key getter which prints out every lookup.f=pc.pycond('[[a eq b] or foo eq bar] or [baz eq bar]',lookup=pc.dbg_get)assertf(state={'foo':'bar'})==TrueOutput:Lookup: a b -> None
Lookup: foo bar -> barEnabling/Disabling of BranchesInsert booleans like shown:f=pc.pycond(['foo','and',['bar','eq',1]])assertf(state={'foo':1})==Falsef=pc.pycond(['foo','and',[True,'or',['bar','eq',1]]])assertf(state={'foo':1})==TrueBuilding Conditions From TextCondition functions are created internally from structured expressions -
but those are[hard to type](  # lazy-dynamic-context-assembly),
involving many apostropies.The text based condition syntax is intended for situations when end users
type them into text boxes directly.GrammarCombine atomic conditions with boolean operators and nesting brackets like:[< atom1 > < and | or | and not|... > <atom2 > ] < and|or... > [ [ < atom3 > ....Atomic Conditions[not] < lookup_key > [[rev] [not] < condition operator (co) > <value > ]When justlookup_keyis given, thencois set to thetruthyfunction:deftruthy(key,val=None):returnoperatur.truth(k)so such an expression is valid and True:pc.State.update({'foo':1,'bar':'a','baz':[]})assertpc.pycond('[ foo and bar and not baz]')()==TrueWhennot lookup_keyis given, thencois set to thefalsyfunction:m={'x':'y','falsy_val':{}}# normal wayassertpc.pycond(['foo','eq',None])(state=m)==True# using "not" as prefix:assertpc.pycond('not foo')(state=m)==Trueassertpc.pycond(['not','foo'])(state=m)==Trueassertpc.pycond('not falsy_val')(state=m)==Trueassertpc.pycond('x and not foo')(state=m)==Trueassertpc.pycond('y and not falsy_val')(state=m)==FalseNestingCombined conditions may be arbitrarily nested using brackets "[" and "]".Via thebrktsconfig parameter you may change those to other separators at build time.Tokenizing DetailsBrackets as strings in this flat list form, e.g.['[', 'a', 'and' 'b', ']'...]FunctioningThe tokenizers job is to take apart expression strings for the builder.SeparatorsepSeparates the different parts of an expression. Default is ' '.pc.State['a']=42assertpc.pycond('a.eq.42',sep='.')()==Truesep can be a any single character including binary.Bracket characters do not need to be separated, the tokenizer will do:# equal:assert(pc.pycond('[[a eq 42] and b]')()==pc.pycond('[ [ a eq 42 ] and b ]')())The condition functions themselves do not evaluate equal - those
had been assembled two times.ApostrophesBy putting strings into Apostrophes you can tell the tokenizer to not further inspect them, e.g. for the seperator:pc.State['a']='Hello World'assertpc.pycond('a eq "Hello World"')()==TrueEscapingTell the tokenizer to not interpret the next character:pc.State['b']='Hello World'assertpc.pycond('b eq Hello\ World')()==TrueBuildingAutoconv: Casting of values into python simple typesExpression string values are automatically cast into bools and numbers via the publicpycond.py_typefunction.This can be prevented by setting theautoconvparameter toFalseor by using Apostrophes:pc.State['a']='42'assertpc.pycond('a eq 42')()==False# compared as string nowassertpc.pycond('a eq "42"')()==True# compared as string nowassertpc.pycond('a eq 42',autoconv=False)()==TrueIf you do not want to provide a custom lookup function(where you can do what you want)
but want to have looked up keys autoconverted then use:foridin'1',1:pc.State['id']=idassertpc.pycond('id lt 42',autoconv_lookups=True)Context On DemandOften the conditions are in user space, applied on data streams under
the developer's control only at development time.The end user might pick only a few keys from many offered within an API.pycond'sctx_builderallows to only calculate those keys at runtime,
the user decided to base conditions upon:
At condition build time hand over a namespace for *all * functions which
are available to build the ctx.pyconwill return a context builder function for you, calling only those functions
which the condition actually requires.pc.ops_use_symbolic_and_txt(allow_single_eq=True)# Condition the end user configured, e.g. at program run time:cond=[['group_type','in',['lab','first1k','friendly','auto']],'and',[[[[['cur_q','<',0.5],'and',['delta_q','>=',0.15],],'and',['dt_last_enforce','>',28800],],'and',['cur_hour','in',[3,4,5]],],'or',[[[['cur_q','<',0.5],'and',['delta_q','>=',0.15],],'and',['dt_last_enforce','>',28800],],'and',['clients','=',0],],],]# Getters for API keys offered to the user, involving potentially# expensive to fetch context delivery functions:# Signature must provide minimum a positional for the current# state:classApiCtxFuncs:defexpensive_but_not_needed_here(ctx):raiseException("Won't run with cond. from above")defcur_q(ctx):print('Calculating cur_q')return0.1defcur_hour(ctx):print('Calculating cur_hour')return4defdt_last_enforce(ctx):print('Calculating dt_last_enforce')return10000000defdelta_q(ctx):print('Calculating (expensive) delta_q')time.sleep(0.1)return1defclients(ctx):print('Calculating clients')return0ifsys.version_info[0]<3:# we don't think it is a good idea to make the getter API stateful ;-)p2m.convert_to_staticmethods(ApiCtxFuncs)f,nfos=pc.parse_cond(cond,ctx_provider=ApiCtxFuncs)# now we create (incomplete) data..data1={'group_type':'xxx'},Falsedata2={'group_type':'lab'},True# this key stores a context builder function, calculating the complete data:make_ctx=nfos['complete_ctx']t0=time.time()forevent,expectedindata1,data2:assertf(state=make_ctx(event))==expectedprint('Calc.Time (delta_q was called twice):',round(time.time()-t0,4)),returncond,ApiCtxFuncsOutput:Calculating clients
Calculating cur_hour
Calculating cur_q
Calculating (expensive) delta_q
Calculating dt_last_enforce
Calculating clients
Calculating cur_hour
Calculating cur_q
Calculating (expensive) delta_q
Calculating dt_last_enforce
Calc.Time (delta_q was called twice): 0.2006Lookup ProvidersContextBuilders are interesting but we can do better.We still calculated values for keys which might(dependent on the data) be not needed in dead ends of a lazily evaluated condition.Lets avoid calculating these values, remembering thecustom lookup functionfeature.This is where lookup providers come in, providing namespaces for functions to be called conditionally.Pycondtreats the condition keys as function nameswithin that namespace and calls them, when needed.Accepted SignaturesLookup provider functions may have the following signatures:classF:# simple data passingdeff1(data):"""simple return a value being compared, getting passed the state/data"""returndata['a']# simple, with ctxdeff2(data,**kw):"""simple return a value being compared, getting passed the state/dataAll context information within kw, compare value not modifiable"""returndata['b']# full pycond compliant signature,deff3(key,val,cfg,data,**kw):"""full pycond signature.val is the value as defined by the condition, and which you could return modifiedkw holds the cache, cfg holds the setupv has to be returned:"""returndata['c'],100# not 45!# applied aldeff4(*a,**kw):"""Full variant(always when varargs are involved)"""returna[3]['d'],'foo'_='and'f=pc.pycond([[':f1','eq',42],_,[':f2','eq',43,_,':f3','eq',45],_,[':f4','eq','foo'],],lookup_provider=F,)assertf(state={'a':42,'b':43,'c':100,'d':'foo'})==TrueParametrized Lookup FunctionsVia the 'params' parameter you may supply keyword args to lookup functions:classF:defhello(k,v,cfg,data,count,**kw):returndata['foo']==count,0m=pc.pycond([':hello'],lookup_provider=F,params={'hello':{'count':2}})(state={'foo':2})assertm==TrueNamespaceLookup functions can be found in nested class hirarchies or dicts. Separator is colon(':')As shown above, if they are flat within a toplevel class or dict you should still prefix with ':', to get build time exception(MissingLookupFunction) when not presentYou can switch that behaviour off per condition build as config arg, as shown belowYou can switch that behaviour off globally viapc.prefixed_lookup_funcs=FalseWarning: This is a breaking API change with pre-20200610 versions, where the prefix was not required to find functions in, back then, only flat namespaces. Use the global switch after import to get the old behaviour.classF:defa(data):returndata['foo']classinner:defb(data):returndata['bar']m={'c':{'d':{'func':lambdadata:data['baz']}}}# for the inner lookup the first prefix may be omitted:_='and'cond=[[':a','eq','foo1'],_,['inner:b','eq','bar1'],_,['c:d','eq','baz1',],]c=pc.pycond(cond,lookup_provider=F,lookup_provider_dict=m)assertc(state={'foo':'foo1','bar':'bar1','baz':'baz1'})==True# Prefix checking on / off:try:pc.pycond([':xx','and',cond])i=9/0# above will raise this:exceptpc.MissingLookupFunction:passtry:pc.pycond([':xx','and',cond],prefixed_lookup_funcs=False)i=9/0# above will raise this:exceptpc.MissingLookupFunction:passcond[0]='a'# remove prefix, will still be foundc=pc.pycond(['xx','or',cond],lookup_provider=F,lookup_provider_dict=m,prefixed_lookup_funcs=False,)assertc(state={'foo':'foo1','bar':'bar1','baz':'baz1'})==TrueYou can switch that prefix needs off - and pycond will then check the state for key presence:# we let pycond generate the lookup function (we use the simple signature type):f=pc.pycond(cond,lookup_provider=ApiCtxFuncs,prefixed_lookup_funcs=False)# Same events as above:data1={'group_type':'xxx'},Falsedata2={'group_type':'lab'},Truet0=time.time()forevent,expectedindata1,data2:# we will lookup only once:assertf(state=event)==expectedprint('Calc.Time (delta_q was called just once):',round(time.time()-t0,4),)# The deep switch keeps working:cond2=[cond,'or',['a-0-b','eq',42]]f=pc.pycond(cond2,lookup_provider=ApiCtxFuncs,deep='-',prefixed_lookup_funcs=False,)data2[0]['a']=[{'b':42}]print('sample:',data2[0])assertf(state=data2[0])==TrueOutput:Calculating cur_q
Calculating (expensive) delta_q
Calculating dt_last_enforce
Calculating cur_hour
Calc.Time (delta_q was called just once): 0.1004
sample: {'group_type': 'lab', 'a': [{'b': 42}]}
Calculating cur_q
Calculating (expensive) delta_q
Calculating dt_last_enforce
Calculating cur_hourThe output demonstrates that we did not even call the value provider functions for the dead branches of the condition.NOTE: Instead of providing a class tree you may also provide a dict of functions aslookup_provider_dictargument, seequalifyexamples below.CachingNote: Currently you cannot override these defaults. Drop an issue if you need to.Builtin state lookups: Not cachedCustomlookupfunctions: Not cached(you can implement caching within those functions)Lookup provider return values: Cached, i.e. called only once, per data setNamed condition sets(see below): CachedExtensionsWe deliver a few lookup functionextensionsfor time checksfor os.environ checks(re-evaluated at runtime)fromdatetimeimportdatetimeasdtfromosimportenvironasenvthis_sec=dt.now().secondthis_utc_hour=dt.utcnow().hourf=pc.pycond([['env:foo','eq','bar'],'and',# not breaking the build when the sec just jumps:['dt:second','in',[this_sec,this_sec+1,0]],'and',['utc:hour','eq',this_utc_hour],])env['foo']='bar'assertf(state={'a':1})==TrueNamed Conditions: QualificationInstead of just delivering booleans, pycond can be used to determine a whole set of
information about data declaratively, like so:# We accept different forms of delivery.# The first full text is restricted to simple flat dicts only:forcin['one: a gt 10, two: a gt 10 or foo eq bar',{'one':'a gt 10','two':'a gt 10 or foo eq bar'},{'one':['a','gt',10],'two':['a','gt',10,'or','foo','eq','bar'],},]:f=pc.qualify(c)r=f({'foo':'bar','a':0})assertr=={'one':False,'two':True}We may refer to results of other named conditions and also can pass named condition sets as lists instead of dicts:defrun(q):print('Running',q)classF:defcustom(data):returndata.get('a')f=pc.qualify(q,lookup_provider=F)assertf({'a':'b'})=={'first':True,'listed':[False,False],'thrd':True,'zero':True,'last':True,}res=f({'c':'foo','x':1})assertres=={'first':False,'listed':[False,True],'thrd':False,'zero':True,'last':True,}q={'thrd':['k','or',':first'],'listed':[['foo'],['c','eq','foo']],'zero':[['x','eq',1],'or',':thrd'],'first':[':custom','eq','b'],'last':True,# you might want to do this to always get at least one matcher, e.g. for data streaming}# as list of conditions:run(q)# as dict:q=dict([[k,v]fork,vinq.items()])run(q)Output:Running {'thrd': ['k', 'or', ':first'], 'listed': [['foo'], ['c', 'eq', 'foo']], 'zero': [['x', 'eq', 1], 'or', ':thrd'], 'first': [':custom', 'eq', 'b'], 'last': True}
Running {'thrd': ['k', 'or', ':first'], 'listed': [['foo'], ['c', 'eq', 'foo']], 'zero': [['x', 'eq', 1], 'or', ':thrd'], 'first': [':custom', 'eq', 'b'], 'last': True}WARNING: For performance reasons there is no built in circular reference check. You'll run into python's built in recursion checker!Optionsinto: Put the matched named conditions into the original dataprefix: Work from a prefix nested in the rootadd_cached: Return also the data from function result cacheHere a few variants to parametrize behaviour, by example:conds={0:['foo'],1:['bar'],2:['func'],3:['n'],'n':['bar'],}classF:deffunc(*a,**kw):returnTrue,0q=lambdad,**kw:pc.qualify(conds,lookup_provider=F,prefixed_lookup_funcs=False,**kw)(d)m=q({'bar':1})assertm=={0:False,1:True,2:True,3:True,'n':True}# return data, with matched conds in:m=q({'bar':1},into='conds')assertm=={'bar':1,'conds':{0:False,1:True,2:True,3:True,'n':True},}defmsg():return{'bar':1,'pl':{'a':1}}# add_cached == True -> it's put into the cond results:m=q(msg(),into='conds',add_cached=True)assertm=={'bar':1,'conds':{0:False,1:True,2:True,3:True,'n':True,'func':True},'pl':{'a':1},}m=q(msg(),into='conds',add_cached='pl')assertm=={'bar':1,'conds':{0:False,1:True,2:True,3:True,'n':True},# n had been put into the cache, was not evaled twice:'pl':{'a':1,'func':True,'n':True},}m=q({'bar':1},add_cached='pl')assertm=={0:False,1:True,2:True,3:True,'n':True,'func':True}# prefix -> Nr 1, bar,  should NOT be True, since not in pl now:m=q(msg(),prefix='pl',into='conds',add_cached='pl',)assertm=={'bar':1,'conds':{0:False,1:False,2:True,3:False,'n':False},'pl':{'a':1,'func':True,'n':False},}Partial EvaluationIf you either supply a key called 'root' OR supply it as argument toqualify, pycond will only evaluate named conditions required to calculate the root key:called=[]defexpensive_func(k,v,cfg,data,**kw):called.append(data)return1,vdefxx(k,v,cfg,data,**kw):called.append(data)returndata.get('a'),vfuncs={'exp':{'func':expensive_func},'xx':{'func':xx}}q={'root':['foo','and',':bar'],'bar':[['somecond'],'or',[[':exp','eq',1],'and',':baz'],],'x':[':xx'],'baz':[':exp','lt',10],}qualifier=pc.qualify(q,lookup_provider_dict=funcs,add_cached=True)d={'foo':1}r=qualifier(d)# root, bar, baz had been calculated, not xassertr=={'root':True,'bar':True,'baz':True,'exp':1}# expensive_func result, which was cached, is also returned.# expensive_func only called once allthough result evaluated for bar and baz:assertlen(called)==1called.clear()f=pc.qualify(q,lookup_provider_dict=funcs,root='x',add_cached=True)assertf({'a':1})=={'x':True,'xx':1}assertf({'b':1})=={'x':False,'xx':None}assertcalled==[{'a':1},{'b':1}]This means pycond can be used as a lightweight declarative function dispatching framework.Streaming DataSince version 20200601 and Python 3.x versions, pycond can deliverReactiveXcompliant stream operators.Lets first set up a test data stream, by defining a functionrx_setuplike so:# simply `import rx as Rx and rx = rx.operators`:# import pycond as pc, like always:Rx,rx,GS=pc.import_rx('GS')defpush_through(*test_pipe,items=4):"""Function which takes a set of operators and runs an 'rx.interval' stream, until count items are through"""# stream sink result holder plus a stream completer:l,compl=[],rx.take(items)l.clear()# clear any previous resultsdefnext_(x):# simply remember what went through in a list:l.append(x)deferr(*a):# should never happen:print('exception',a)stream=Rx.interval(0.01)# numbers, each on its own thread# turns the ints into dicts: {'i': 1}, then {'i': 2} and so on:# (we start from 1, the first 0 we filter out)stream=stream.pipe(rx.filter(lambdai:i>0),rx.map(lambdai:{'i':i}))# defines the stream through the tested operators:test_pipe=test_pipe+(compl,)s=stream.pipe(*test_pipe)# runs the stream:d=s.subscribe(on_error=err,on_next=next_,on_completed=lambda:l.append('completed'),)# blocks until completed:whilenot(landl[-1]=='completed'):time.sleep(0.001)l.pop()# removes completed indicatorreturnl# returns all processed messagesreturnRx,rx,push_throughLets test the setup by having some messages streamed through:Rx,rx,push_through=rx_setup()# test test setup:r=push_through(items=3)assertr==[{'i':1},{'i':2},{'i':3}]-> test setup works.FilteringThis is the most simple operation: A simple stream filter.Rx,rx,push_through=rx_setup()# ask pycond for a stream filter based on a condition:pcfilter=partial(pc.rxop,['i','mod',2])r=push_through(pcfilter())odds=[{'i':1},{'i':3},{'i':5},{'i':7}]assertr==odds# try the stream filter with message headered data:pl='payload'r=push_through(rx.map(lambdai:{pl:i}),pcfilter(prefix=pl))print('Full messages passed:',r)r=[m[pl]forminr]assertlen(r)==4assertr==oddsOutput:Full messages passed: [{'payload': {'i': 1}}, {'payload': {'i': 3}}, {'payload': {'i': 5}}, {'payload': {'i': 7}}]Streaming ClassificationUsing named condition dicts we can classify data, i.e. tag it, in order to process subsequently:Rx,rx,push_through=rx_setup()# generate a set of classifiers:conds=[['i','mod',i]foriinrange(2,4)]defrun(offs=0):# and get a classifying operator from pycond, adding the results in place, at key 'mod':r=push_through(pc.rxop(conds,into='mod'))i,j=0+offs,1+offsassertr==[{'i':1,'mod':{i:1,j:1}},{'i':2,'mod':{i:0,j:2}},{'i':3,'mod':{i:1,j:0}},{'i':4,'mod':{i:0,j:1}},]# this will automatically number the classifiers, from 0:run()# we can also provide the names of the classifiers by passing a dict:# here we pass 2 and 3 as those names:conds=dict([(i,['i','mod',i])foriinrange(2,4)])run(2)Normally the data has headers, so thats a good place to keep the classification tags.Selective ClassificationWe fall back to an alternative condition evaluation(which could be a function call) * only * when a previous condition evaluation returns something falsy - by providing a * root condition*.
When it evaluated, possibly requiring evaluation of other conditions, we return:Rx,rx,push_through=rx_setup()# using the list style:conds=[[i,[['i','mod',i],'or',':alt']]foriinrange(2,4)]conds.append(['alt',['i','gt',1]])# provide the root condition. Only when it evals falsy, the named "alt" condiction will be evaluated:r=push_through(pc.rxop(conds,into='mod',root=2,add_cached=True))assertr==[# evaluation of alt was not required:{'i':1,'mod':{2:True}},# evaluation of alt was required:{'i':2,'mod':{2:True,'alt':True}},{'i':3,'mod':{2:True}},{'i':4,'mod':{2:True,'alt':True}},]Treating of Booleans (Conditions, Not Names)For the special case of booleans in a condition list we do not treat them as names.# 2 unnamed conditions -> keys will be positionalqs=pc.qualify([True,False])res=qs({'a':1})assertres=={0:True,1:False}# and not {True: False}# 2 named condsqs=pc.qualify([[1,['a','eq',1]],[2,['b','eq',42]]])res=qs({'a':1})assertres=={1:True,2:False}Asyncronous OperationsWARNING: Early Version. Only for the gevent platform.Selective classification allows to call condition functions only when other criteria are met.
That makes it possible to read e.g. from a database only when data is really required - and not always, "just in case".pycond allows to define, that blocking operations should be run * async* within the stream, possibly giving up order.Asyncronous FilterFirst a simple filter, which gives up order but does not block:Rx,rx,push_through=rx_setup()classF:defcheck(k,v,cfg,data,t0=[],**kw):# will be on different thread:i,pointer=data['i'],''ifnott0:t0.append(now())ifi==1:# ints are fired at 0.01, i.e. the 1 will land 4 after 1:time.sleep(0.048)pointer='   <----- not in order, blocked'# demonstrate that item 1 is not blocking anything - just order is disturbed:print('item%s:%.3fs%s'%(i,now()-t0[0],pointer))returni%2,v# have the operator built for us - with a single condition filter:rxop=pc.rxop([':check'],into='mod',lookup_provider=F,asyn=['check'],)r=push_through(rxop,items=5)assert[m['i']forminr]==[3,5,1,7,9]Output:item 2: 0.011s 
item 3: 0.022s 
item 4: 0.034s 
item 5: 0.045s 
item 1: 0.049s    <----- not in order, blocked
item 6: 0.057s 
item 7: 0.068s 
item 8: 0.079s 
item 9: 0.090sFinally asyncronous classification, i.e. evaluation of multiple conditions:def_thn(msg,data):returnprint('thread:',cur_thread().name,msg,data)# push_through just runs a stream of {'i': <nr>} through a given operator:Rx,rx,push_through=rx_setup()# Defining a simple 'set' of classifiers, here as list, with one single key: 42:conds=[[42,[['i','lt',100],'and',[[':odd','eq',1],'or',['i','eq',2]],'and_not',[':blocking','eq',3],],]]classF:"""Namespace for condition lookup functions.You may also pass a dict(lookup_provider_dict)We provide the functions for 'odd' and 'blocking'."""defodd(k,v,cfg,data,**kw):# just print the threadname.# will go up, interval stream has each nr on its own thread:_thn('odd',data)# fullfill condition only for odd numbers# -> even nrs won't even run func 'blocking':returndata['i']%2,vdefblocking(k,v,cfg,data,**kw):i=data['i']# will be on different thread:_thn('blocking',data)ifi==1:# two others will "overtake the i=1 item,# since the interval stream is firing every 0.01 secs:time.sleep(0.028)elifi==2:# Exceptions, incl. timeouts, will simply be forwarded to cfg['err_handler']# i.e. also timeout mgmt have to be done here, in the custom functions themselves.# Rationale for not providing a timeout monitoring within pycond itself:# Async ops are done with libs, which ship with their own timeout params.# No need to re-invent / overlay with our own monitoring of that.# In the err handler, then further arrangements can be done.raiseTimeoutError('ups')elifi==5:1/0returndata['i'],verrors=[]defhandle_err(item,cfg,ctx,exc,t=errors,**kw):# args are: [item, cfg]if'ups'instr(exc):assertitem['i']==2assertexc.__class__==TimeoutErrort.append(item)else:assertitem['i']==5assertexc.__class__==ZeroDivisionErrort.append(item)# have the operator built for us:rxop=pc.rxop(conds,into='mod',lookup_provider=F,err_handler=handle_err,asyn=['blocking'],)r=push_through(rxop,items=5)assert[m['i']forminr]==[3,1,4,6,7]assert[m['mod'][42]forminr]==[False,True,False,False,True]# item 2 caused a timeout:assert[t['i']fortinerrors]==[2,5]Output:thread: Thread-55 odd {'i': 1}
thread: Dummy-57 blocking {'i': 1}
thread: Thread-56 odd {'i': 2}
thread: Dummy-59 blocking {'i': 2}
thread: Thread-58 odd {'i': 3}
thread: Dummy-61 blocking {'i': 3}
thread: Thread-60 odd {'i': 4}
thread: Thread-62 odd {'i': 5}
thread: Dummy-64 blocking {'i': 5}
thread: Thread-63 odd {'i': 6}
thread: Thread-65 odd {'i': 7}
thread: Dummy-67 blocking {'i': 7}*Auto generated bypytest2md, running./tests/test_tutorial.py