Package: yo-fluq
yo_fluqSummaryyo_fluqfocuses on a fluent, lazy, expandable way of writing data processing pipelines. The typical example is(Query.en(orders).take(1000).where(lambdaorder:order['is_shipped']).select(lambdaorder:order['payment_information']).group_by(lambdapayment:payment['customer_id']).to_dictionary(lambdagroup:group.key,lambdagroup:Query.en(group.value).select(lambdapayment:payment['value']).sum()))This way of writing code is typical for C# Linq and Spark, and this project makes it available for Python as well.Unlikepandas,yo_fluqis:Lazy, so it does not require to keep the whole collection in memoryExtendable, so you can define your own filters and use them in pipelines.Unlikeasqorpy_linq, well-known ports of C# LINQ to Python,yo_fluqfully supports data annotations, even in case of user-defined extensions. Therefore, in IDE like PyCharm you will see the available methods in the hints.plinqsupports annotations, but does not offer extendability technique.The library has been developed since 2017, is extensively tested and is currently available under MIT licence, at Beta development stage.Pull-pipelinesyo_fluqis a port ofC# LINQ-to-objects, which is a set of methods for data processing, focused on:Fluent interfaceLazy evaluationExtendabilityType hintsThis port maintains this focus.MotivationConsider the task:You have two collections ofarticlesandcustomersYou have afunctionthat predicts how well the article fits the customerYou want to get the top-10 tuples of(article, customer, value), but only if the function’s value is greater than 0These tuples have to be sorted byvaluein descending orderNaive PythonThe easiest way to solve it is to just write it in Python:result=[]forarticleinarticles:forcustomerincustomers:value=function(article,customer)ifvalue>0:result.append((article,customer,value))result.sort(key=lambdaz:z[2],reverse=True)returnresult[:10]The problems are:The code gets more and more nested with every step, which contributes to poor code qualityEverything is done in a different way: we use operators for filtering, function with and without returning values for different cases. It's not uniformIf you want to restore the task from this code, it requires an effort. The code is not self-explanatoryList comprehensionList comprehension is another way of doing it:result=[(article,customer,function(article,customer))forarticleinarticlesforcustomerincustomersiffunction(article,customer)>0]result.sort(key=lambdaz:z[2],reverse=True)returnresult[:10]The problems are:The functionality is limited and non-extendable. In order to put filtration inside list comprehension, I had to computefunctiontwice. Ordering and slicing are impossible to fitI usearticleandcustomerbefore they are defined. Hence, no type hints will be available in IDE(subjective) The syntax is rather exoticpandasIf we convert the whole data in pandas, it's easy to do:df=pd.DataFrame([(article,customer)forarticleinarticlesforcustomerincustomers],columns=['article','customer'])df=(df.assign(value=df.apply(lambdas:function(s.article,s.customer),axis=1)))return(df.loc[df.value>0].sort_values('value',ascending=False).iloc[:10])Here we haveFluentInterface, and it brings:We can apply more and more filters, just adding the linesEverything is done in the sameuniformfashion: all the filters are just methods ofpandasCode is readable: each line of code basically can be traced to the line in problem's descriptionHowever, there are still problems:It's not always easy to convert data from existing format to pandas, you need to write some (ugly) code for thatIn some cases it's not possible. Pandas requires the whole dataframe to be in memory, which is sometimes not possibleNot all the data are two-dimensionalpandasis still not extendable: I had to break a fluent style once to filter out byvalue.itertoolsThe survey of different methods won't be complete without the fourth method -itertools:returnislice(sorted(filter(lambdaz:z[2]>0,map(lambdaart_cust:(art_cust[0],art_cust[1],function(art_cust[0],art_cust[1])),product(articles,customers))),key=lambdares:res[2],reverse=True),0,10)This method solves all the problems of pandas:By default, all the itertools arelazy: that means, you don't need to store all the data in memory at onceIt's extendable: you can always write your own method to do precisely what you need. This whole code is written completely within one approachApplicable to any data in any formatHowever:itertoolshas horrible inside-out syntax which is almost unreadable.fluqThis is the solution, offered by this module:(Query.combinatorics.cartesian(articles,customers).select(lambdaart_cust:art_cust+(function(art_cust[0],art_cust[1]),)).where(lambdares:res[2]>0).order_by_descending(lambdares:res[2]).take(10).to_list())This solution is combining the best parts ofpandasanditertools. It is fluent, lazy,FluentLazyExtendable: you can add your own filterAfter each dot.in IDE you can see the list of methods applicable, so half of this query will be written by IDE.Related worksOf course, I didn't invent this solution. I learned it fromC#, which employed ideas fromScala(the same concepts are behind Spark for instance).What I did is port to Python.I know about these analogues:asq,py-linq. The key difference is thatyo_fluqhas annotations and it's expansion technique preserves type hintsplinqhas type annotations, but does not have an extendability mechanism.RxPycontains LINQ port, which is non-extendable and is not a main focus of the library anyway.In some sensefluentpy. However the approach of this module is more fundamental, and that leads to the side effects described in the repo.Fluent data processingThe typicalfluqpipeline looks like this:(Query.en(orders).take(1000).where(lambdaorder:order['is_shipped']).select(lambdaorder:order['payment_information']).group_by(lambdapayment:payment['customer_id']).to_dictionary(lambdagroup:group.key,lambdagroup:Query.en(group.value).select(lambdapayment:payment['value']).sum()))The typicalfluqpipeline consists of:The sourceQuery.en(orders). This creates theQueryableobject which can be perceived as a flow of objectsThe set of filterswhere,select,take,group_byThe final aggreagationto_dictionary, that initiates all other components and produces the result.SourcesCodeArgumentsOutput typeCommentsQuery.en(list)list: Iterable[T]Q[T]Query.args(*args)*args: AnyQ[Any]Mostly used in unit testsQuery.dict(dict)dict: Dict[TKey,TValue]Q[KeyValuePair[TKey,TValue]]Use Query.en(dict.values()) or Query.en(dict) to access only keys or valuesQuery.loop(begin,delta,end,endtype)Q[Any]Produces begin, begin+delta, begin+delta+delta, etc. until the end. Also works when delta is negative.endTypedetermines when exactly process ends.Notes:Queryis a singleton instance ofQueryFactoryclass that contains all the sources.Qis used as a shortening forQueryablethrough all this documentationThe annotations in code may be weaker than in documentation. Currently, annotations are messy in the library, basically I only worried that type hints work in PyCharmFiltersMain filtersFor these filters it is important to understand what is the type of the incoming and outgoing objects.MethodInputArgumentOutputCommentsselect(selector)Q[TIn]selector:TIn->TOutQ[TOut]Maps elements of the flow. Preserves length. Analog to itertools.mapwhere(filter)Q[T]filter:T->boolQ[T]Stops the elements where filter's value is False. Analog to itertools.filterdistinct(selector)Q[T]Opt.selector:T->AnyQ[T]Distinct elements. By some value ifselectoris providedselect_many(selector)Q[TIn]selector:TIn->Iterable[TOut]Q[TOut]Likeselect, but flattens the result, so returnsQ[TOut]instead of e.g.Q[List[TOut]].with_indices()Q[T]Q[KeyValuePair[int,T]]Enumerates objects in the incoming flowgroup_by(selector)Q[T]selector:T->TKeyQ[KeyValuePair[TKey,List[T]]Groups elements by the key, provided by selector, and propagates the groups.Flow control and set operationsFlow control and set operations:skip(N)skips N objects in the flowtake(N)takes N objects from the flow, then stops processing.skip_while(condition)skips objects while condition is true, then start processing them ignoring the conditiontake_while(condition)process objects while condition is true, then stops processingprepend(*args)places*argsin front of the flowappend(*args)places*argsafter the flowQuery.en(e1).concat(e2)concatenates collections e1 and e2 in the flowQuery.en(e1).intersect(e2)intersects the collection e1 and e2Orderingorder_by(key_selector)andorder_by_descending(key_selector)will order the flow by ascending/descending values of thekey_selectorIf ordering by several keys is needed, add them withthen_by/then_by_descending.Example:(Query.en(['B1','A0','A1','B1']).order_by(lambdaelement:element[0]).then_by_descending(lambdaelement:element[1]).to_list())will order the collection by the ascending first letter, and then by the descending second letter, producing['A1','A0','B1','B0']AggregatorsEach aggregator exists in two forms: as a method in Queryable, and as a class inaggmodule. The reason is, that aggregators are reused in both pull- and push-queries. The names are self-explanatory:Getters:first,lastandsingle(works likefirstbut throws exception if more than element occur)Basic math:sum,count,mean,std,max,minarg_maxandarg_min(acceptsselector, return the element for which its value is highest/lowest)all,anyOthersThe following methods are also implemented:foreach(processor)acts as aggregator, executesprocessoron each element in the flowforeach_and_continue(processor)execute processor on each element in the flow, but let flow further, thus acting like a filterparallel_selectrunsselectin parallel, thus simplifying multiprocessing usage in Python, if you want to speed-up some really time-consumingselect.Execution orderConsider the query(Query.en([0,1,2]).where(lambdaz:z%2==0).select(str).to_list())You might thinkthat it first filters out the odd numbers, then translates what is left into strings, and then puts the numbers to list.
The actual process you might see in the debugger is different, but lead to the same result.EnumerableQueryEffective python codeen1Query.en([0,1,2])for z0 in [0,1,2]: yield een2e1.where(lambda z: z%2==0)for z1 in en1: if z1%2==0: yield z1en3e2.select(str)for z2 in en2: yield str(e2)e3.to_list()for z3 in en3: lst.append(z3)The sequence of actions is:to_listrequests element fromen3en3is the result ofselect, soselectrequests element fromen2en2is the result ofwhere, sowhererequests element fromen1en1iterates over[0,1,2]. Firstz0is0, this is yielded asen1elementwherechecksz1=0, it satisfies the condition,whereyields0selectconverts0to'0'and yields itto_listplacesz3='0'intolstto_listrequests next element fromen3selectrequests next element fromen2whererequests next element fromen11is the next element in[0,1,2]. So it is yieldedwherechecks this element, it does not satisfy the conditionwhererequests another element fromen12is the next element in[0,1,2], so2is yielded as the next element inen1wherechecks2, it satisfies the condition and is yielded asen2secondelementselectconverts2to'2'to_listplaces'2'intolstto_listrequests next element fromen2selectrequests next element fromen2whererequests next element fromen1There is no more elements in[], so loop inen1terminateswhere's loop terminatesselect's loop terminatesto_list's loop terminatesExtendabilityIn C#, there areextension methods, the syntax which "add" a method to compiled object.
In Python, monkey-patching produces the similar effect, but unfortunately neither PyCharm nor Jupyter Notebook can infer the annotations for monkey-patched methods.
Therefore, extendability and type hints come into conflict in Python, and this section describes how the conflict is resolved.feed-methodfeedmethod inQueryableis defined as:deffeed(self,processor:Callable[[Queryable],T])->T:returnprocessor(self)Assume we want to extend withTQDMprogress bar:def_with_tqdm_iter(queryable):forelementintqdm(queryable):yieldelementdefwith_tqdm(queryable:Queryable)->Queryable:returnQueryable(_with_tqdm_iter(queryable))Now, the code:Query.en([1,2,3]).feed(with_tqdm).to_list()will push the flow through the tqdm and produce a list [1,2,3]. Due to type annotation forfeedandwith_tqdmmethods, the type of the expressionQuery.en([1,2,3]).feed(with_tqdm)can be inferred asQueryable, thus the type hint will appear.What if you want to pass some arguments to thiswith_tqdmmethod? Then it's a little more complicated:classwith_tqdm(Callable[[Queryable],Queryable]):def__init__(self,**kwargs):self.kwargs=kwargsdef_with_tqdm_iter(self,queryable):forelementintqdm(queryable,**self.kwargs):yieldelementdef__call__(self,queryable:Queryable)->Queryable:returnQueryable(self._with_tqdm_iter(queryable))And then:Query.en([1,2,3]).feed(with_tqdm(total=3)).to_list()So if you want to develop several extension methods, just follow these templates and you will keep the type hints.Updating Query and QueryableAt some point, there are simply too much of the extension methods used too often, so the pipeline looks like a sequence offeedmethods.In order to resolve that, there is a need to re-defineQueryable, so:the oldQueryablemethods produce instances of newQueryablethe annotation is bound to the newQueryablein both old and new cases.Fortunately, this is possible, and done in theyo_fluq_dspackage. I refer you to the source code for details, because this is definitely not the most frequently needed operation.Push-queriesPush-queries are important for data aggregation. Assume you want to execute this pull-query:(Query.en(orders_from_huge_file()).where(lambdaorder:order['is_shipped']).group_by(lambdaorder:order['shipping_country']).to_dictionary(lambdagroup:group.key,lambdagroup:Query.en(group.value).select(lambdaorder:order['billing_total']).mean()))The execution of the pull-query starts from the end.to_dictionaryaggregation requests data from the filters before.
The previous filter isgroup_by, and it can only provide data when all the file is read. Therefore, this whole file will be stored in memory, which is not possible if the file is huge.
Therefore, this particular operation cannot really be performed by pull-queries.Push-queries are introduced to solve this problem.pipeline=(Query.push().where(lambdaorder:order['is_shipped']).split_by_groups(lambdaorder:order['shipping_country']).mean())report=pipeline(orders_from_huge_file())The idea behind the this implementation is pretty much alikeRxPy. The differences are:It is optimized for data processing, so pipeline has "two ways": data comes in and in the end the report comes out.It has the same interface as pull-queries and reuse some of their codePush-queries architecturePushQueryis a class that followsBuilder pattern. It's methods, likewhereorselect, creates instances ofPushQueryElement-s and stores them inside the class. Thus,PushQueryis a sequence ofPushQueryElement, orPQE.PushQueryElementis notthe entity that processes data. It is the factory that creates such entities:PushQueryElementInstance, orPQEI.PQEI implements__enter__and__exit__function. It must be entered to before processing data, and exited from after processing. E.g., PQEI that writes to files will close the file on exit.So when we try to feed data toPushQuery, we actually:Take the first PQE-factory in thePushQuerylist.Create PQEI with this factory and enter to itFeed data to first PQEI. If there are more than one PQE in query, subsequent PQEI will be created and data will be forwarded to them.After data is over, collect the report from PQEI. If there is more that one PQE in the query, PQEI requests report from subsequent PQEI and may transform it.Exit the PQEIDepending on its type and purpose, PQEI processes data in following fashions:PQEI forsum,meanetc. compute reports. IfPushQueryconsists of only one suchPQE, it's behavior is straightforward: process data and return a report.PQEI forselecttransforms data and feeds it to the subsequent PQEIPQEI forwherechecks the condition and depending on the result, forwards it to the subsequent PQEI or discards.PQEI forgroup_bychecks the group key and:If this key is seen for the first time, creates a new instance of subsequet PQEI and forwards data to it, keeping a link to this PQEIIf the key was seen before, forwards data to the kept PQEIThus,group_byexpands the original pipeline of PQE intotreeof PQEI.sum,meanand others are actually the very sameagg.Sum,agg.Meanetc. that are used in the pull-queries.ExpandabilitySince push-queries are just sequences of factories, you can always write your own PQE and add it to the sequence withappendmethod ofPushQuery. No special magic is required.The base class for PQE isPushQueryElement. As instances, it creates_PushQueryElementInstanceobjects, but these objects redirect all their methods to the factory. It is very important tonevermodify anything that belongs to the factory in these methods, otherwise it's hard to predict the system's behavior.Comparing pull- and push-queriesNew filters for pull-queries are very easy to write withyield. However, push-queries are harder to write: in complex cases, when the processing of element is affected by processing the previous, you must write state machine yourself, while in pull-queries it is done by Python.Pull-queries can be easily continued with push-queries, and actually this happens always, because aggregators in pull-query are implemented as atomic push-queries. However, push-query can be continued with pull-query only in a separate thread. There is no other means to do that.In general, for data processing, the pull-queries are the weapon of choice. Push-queries should only be used for the cases, when the data flow "splits" and forms trees, like ingroup_by-based statistics.