Package: pyscanprev
Would you like to replace this:reduce(lambdaprev,x:abs(prev-x),[2,3,4,5])by this?last(abs(prev-x)forxin[2,3,4,5])Why not?>>>fromfunctoolsimportreduce>>>frompyscanprevimportenable_scan,last>>>@enable_scan("prev")...defevaluate(data):...returnreduce(lambdaprev,x:abs(prev-x),data),\...last(abs(prev-x)forxindata)>>>evaluate([2,3,4,5])(2,2)ExamplesExampleDescriptionComparisonSimple scan/accumulate and fold/reduce examples using the
PyScanPrev resources, the Python standard library and
the 3-for-section comprehensions for comparing readability
and expressiveness.Conditional TogglingSometimes feedback isn’t really required, but you should at
least store some state about what’s going on in the input.
That’s the case in this example, which toggles/updates the
state only for certain inputs, aLtErNaTiNg CaSeS, iGnOrInG oThEr
ChArAcTeRs. Historically, PyScanPrev was created after thinking
on how would be the simplest way to solve the problem described
in this example.Examples from the itertools.accumulate documentationThis is a copy and an adaptation of theitertools.accumulateexamples in the Python documentation to use the PyScanPrev
comprehensions instead of that standard library function. The
examples include:running productrunning maximumamortization tableschaotic logistic mapFactorial and MultiplicationFold/reduce factorial and multiplication / product of a sequence
implementation using PyScanPrev.FibonacciFibonacci sequence with PyScanPrev.Gray CodeGenerating Gray codes using the definition is slower than using
bitwise operations, but the recursive definition can be written
as a scan/fold expression and is useful for testing, as this
example shows.Single pole lowpass IIR FilterDSP (Digital Signal Processing) applications ofter requires
feedback, i.e., accessing some previous output value in a
process. PyScanPrev can be used to write simple signal
processing models like IIR (Infinite Impuse Response) linear
filters. This is an example with a single pole lowpass digital
filter. For testing and displaying the results, this example
usesAudioLazyandhipsterplot.State-space modelSimulation of linear discrete time modeling in control
engineering. The example gives an implementation of both time
varying and time invariant state-space models. These models are
then used to simulate:an accumulatora LTI (Linear Time Invariant) continuous time
mass-spring-damper SHM (Simple Harmonic Motion) modela linear time varying “leaking bucket”-spring-damper modelanother LTI IIR filter, designed from the difference equation
and simulated as a state-space model, compared with theAudioLazyresultsThe discretization process is included in the example, and the
simulations usehipsterplotto plot the motion
path/trajectory. This example includes an explanation/proof to
the conversion from difference equations to a state-space model
(via Z Transform).Why “Scan”?Scanis a high order function that can be seen as something between
the map and the reduce (fold) functions, returning all steps of a
fold. Since Python 3.3, it’s available in the functionitertools.accumulate.The goal here is to find an easy way to read the previous value in a
generator expression and anything alike, so that the scan/fold
(accumulate/reduce) code can be written using them.Readability counts!Package contentsenable_scan(name)Decorator that allows functions to have generator expressions
and list/set comprehensions with a variable (the one with the
given name) in its body for accessing the previous resulting
value.last(iterable)Gets the last value from an iterable, making it straightforward
to write a reduce/fold from the scan result.prepend(value,iterable)A version of[value] + some_listfor general iterables,
returning a generator. This function was created to allow
PyScanPrev-enabled generator expressions and list/set
comprehensions to include an explicit start value, but it can be
used to prepend a value in any context, e.g. to force a start
value onitertools.accumulate.scan(func,iterable,[start],*,echo_start=True)It’s an implementation of the scan higher order function with
more features thanitertools.accumulate(thestartand
the keyword-onlyecho_startparameters) and consistent to
thefunctools.reducefunction signature.Tell me, how is that possible at all?Magic! Some people say that’s bytecode manipulation, but isn’t that all the
same?InstallingYou can either use pip:# From PyPIpipinstallpyscanprev# From GitHub master branchpipinstall--upgradegit+https://github.com/danilobellini/pyscanprevOr setup.py directly:python3 setup.py installThis software depends onbytecode, which requires Python 3.4+.The world without this package (rationale)It’s not usual nor widely known that the cross/cartesian product applied on
multiple “for” sections in a generator expression or a list/set/dict
comprehension allows more than one section to have the same target variable
name. But that provides the means to do something akin to a scan, for example
this cumulative sum (Tested in Python 2.6+ and 3.2+):>>>[prevforprevin[0]forelinrange(5)forprevin[prev+el]][0,1,3,6,10]Whose parts are:[prevforprevin[start]fortargetiniterableforprevin[func(prev,target)]]But that’s a kludge, it’s hard to grasp, hard to change/update/maintain,
fixed/locked in that “for” section order, and its behavior has some minor
details whose control would need to be external (e.g. using the first value
from theiterableas thestart). Theprevvariable appears at
least 4x in such structure and twice as a target. The firstprevvalue isstart, which is just seen/used by the last “for” section in its firstfunccall, whose result is assigned toprevbefore the whole list
comprehension appends/”yields” any output/result, since it’s also the target
variable name in that “for” section. Sostartis never an output,
although everything starts withprev for prev in [start].It’s not only about code aesthetics or readability, but also about
pattern memorization: knowledge about the scan abstraction
and about the Python language is probably
not enough for one to remember that structure.Asfuncin the previous example was essentiallyoperator.add, let’s do
the same cumulative sum withitertools.accumulate(Python 3.2+):>>>fromitertoolsimportaccumulate>>>list(accumulate(range(5)))[0,1,3,6,10]It seems the same, but here the first zero output is thenext(iter(range(5))),
not the result of a sum or any otherfuncfor that matter (i.e., it
doesn’t depend onfuncat all). To be really equivalent to the
3-for-sections list comprehension above, it would need to be something like:>>>list(accumulate([0,0,1,2,3,4]))[1:][0,1,3,6,10]We had to prepend0torange(5). What’s going on here is thataccumulatereturns a generator that yields the values:[i0, i0+i1, i0+i1+i2, i0+i1+i2+i3, i0+i1+i2+i3+i4, ...]Where “in” is the n-th value from theiterableinput.
Every step obviously re-uses the previous step result instead
of summing all the previous inputs again,
and that’s what the scan is all about. On the other hand, the 3-for-sections
list comprehension does this whenfuncis the sum/add:[s+i0, s+i0+i1, s+i0+i1+i2, s+i0+i1+i2+i3, s+i0+i1+i2+i3+i4, ...]Where “s” is thestart. Since Python 3.3, itertools.accumulate has an
optional second parameter, which should be a binary
operator/function/callable. For a givenfunc, the resulting generator
would yield, in order:next(iterable),# result[0]func(result[0],next(iterable)),# result[1]func(result[1],next(iterable)),# result[2]func(result[2],next(iterable)),# result[3]...Wherestartis implicit as the first value fromiterable, andresultis that output iterable itself seen as a sequence. To grasp the
difference, let’s see a cumulative sum of squares starting with 3 in the
accumulator/register.>>>list(accumulate([3,5,1,1,2],lambdax,y:x+y**2))[3,28,29,30,34]To get the same result with a list comprehension, one would do:>>>[3]+[xforxin[3]...foryin[5,1,1,2]...forxin[x+y**2]][3,28,29,30,34]There’s also a really old package in PyPI calledfunctional,
whose last update was in 2006. Besides the distinction between
non-strict and “prime”/strict counterparts, it mimics all the4 scan and 4 fold Haskell functions, including their names
and their parameter order. Thefunctional.scanl1and
theitertools.accumulatefunctions are almost the
same, the difference is thatscanl1needs the function to be the first
argument and it isn’t optional. On the other hand,functional.scanlneeds
an extra “start” parameter. Both functions return a generator:>>>importfunctional,operator>>># scanl (+) 0 [0..4]>>>list(functional.scanl(operator.add,0,range(5)))[0,0,1,3,6,10]>>># scanl1 (+) [0..4]>>>list(functional.scanl1(operator.add,range(5)))[0,1,3,6,10]>>># scanl1 (\x y -> x + y^2) [3, 5, 1, 1, 2]>>>list(functional.scanl1(lambdax,y:x+y**2,[3,5,1,1,2]))[3,28,29,30,34]Bothscanlandscanl1have a behavior different from that
3-for-sections list comprehension.Pythonfunctools.reduce,functional.foldlandfunctional.foldl1, as fold/reduce implementations,
share a core idea: they return the last value of the scan
resulting from the same given inputs tofunctional.scanlandfunctional.scanl1. Thereducefunction can have an optionalstartas the 3rd and last argument, which gives to it both the behavior of bothfoldl, that requires thestartas the 2nd parameter, andfoldl1,
which uses the first iterable value as the start value. If there’s a way to
modify generator expressions so thatscanl/scanl1/accumulatecan be
implemented with them with a good readability, the same would apply to reduce.But, even for developers who like to think on these concepts as ready to use
abstractions stored in first class objects, here we got a parameter hell!
Their order is a mess:(iterable, func) ->itertools.accumulate(func, start, iterable) ->functional.scanl(func, iterable) ->functional.scanl1,map,filter(func, iterable, [start]) ->functools.reduceThe higher-order functions scan and fold appears respectively initertools.accumulateandfunctools.reducefirst-class objects
(functions are first-class objects in Python), which are quite easy for people
coming from a functional programming background to grasp, and far easier to
read/remember than the 3-for-sections list comprehension. One just neet to
know these two have their 2 parameters reversed, and that accumulate doesn’t
have an optional external start value. It would be great to have an optional
start parameter onitertools.accumulate, as well as a function signature
standardization, but the main purpose of this is just to get a cleaner
alternative to that 3-for-sections list comprehension.Copyright (C) 2016 Danilo de Jesus da Silva Bellini