Package: pipe
Pipe — Infix programming toolkitModule enabling a sh like infix syntax (using pipes).IntroductionAs an example, here is the solution for the2nd Euler Project
problem:Find the sum of all the even-valued terms in Fibonacci which do not
exceed four million.Givenfiba generator of Fibonacci numbers:sum(fib()|where(lambdax:x%2==0)|take_while(lambdax:x<4000000))Each pipes is lazy evalatated, can be aliased, and partially
initialized, so it could be rewritten as:is_even=where(lambdax:x%2==0)sum(fib()|is_even|take_while(lambdax:x<4000000)InstallingTo install the library, you can just run the following command:# Linux/macOSpython3-mpipinstallpipe# Windowspy-3-mpipinstallpipeUsingThe basic syntax is to use a|like in a shell:>>>fromitertoolsimportcount>>>frompipeimportselect,take>>>sum(count()|select(lambdax:x**2)|take(10))285>>>Some pipes take an argument:>>>frompipeimportwhere>>>sum([1,2,3,4]|where(lambdax:x%2==0))6>>>Some do not need one:>>>frompipeimporttraverse>>>foriin[1,[2,3],4]|traverse:...print(i)1234>>>In which case it's allowed to use the calling parenthesis:>>>frompipeimporttraverse>>>foriin[1,[2,3],4]|traverse():...print(i)1234>>>Existing Pipes in this moduleAlphabetical list of available pipes; when several names are listed
for a given pipe, these are aliases.batchedLike Python 3.12itertool.batched:>>>frompipeimportbatched>>>list("ABCDEFG"|batched(3))[('A','B','C'),('D','E','F'),('G',)]>>>chainChain a sequence of iterables:>>>frompipeimportchain>>>list([[1,2],[3,4],[5]]|chain)[1,2,3,4,5]>>>Warning : chain only unfold iterable containing ONLY iterables:[1,2,[3]]|chainGives aTypeError: chain argument #1 must support iterationConsider using traverse.chain_with(other)Like itertools.chain, yields elements of the given iterable,
then yields elements of its parameters>>>frompipeimportchain_with>>>list((1,2,3)|chain_with([4,5],[6]))[1,2,3,4,5,6]>>>dedup(key=None)Deduplicate values, using the givenkeyfunction if provided.>>>frompipeimportdedup>>>list([-1,0,0,0,1,2,3]|dedup)[-1,0,1,2,3]>>>list([-1,0,0,0,1,2,3]|dedup(key=abs))[-1,0,2,3]>>>enumerate(start=0)The builtinenumerate()as a Pipe:>>>frompipeimportenumerate>>>list(['apple','banana','citron']|enumerate)[(0,'apple'),(1,'banana'),(2,'citron')]>>>list(['car','truck','motorcycle','bus','train']|enumerate(start=6))[(6,'car'),(7,'truck'),(8,'motorcycle'),(9,'bus'),(10,'train')]>>>filter(predicate)Alias forwhere(predicate), seewhere(predicate).groupby(key=None)Likeitertools.groupby(sorted(iterable, key = keyfunc), keyfunc)>>>frompipeimportgroupby,map>>>items=range(10)>>>' / '.join(items|groupby(lambdax:"Odd"ifx%2else"Even")...|select(lambdax:"{}:{}".format(x[0],', '.join(x[1]|map(str)))))'Even: 0, 2, 4, 6, 8 / Odd: 1, 3, 5, 7, 9'>>>islice()Just theitertools.islicefunction as a Pipe:>>>frompipeimportislice>>>list((1,2,3,4,5,6,7,8,9)|islice(2,8,2))[3,5,7]>>>izip()Just theitertools.izipfunction as a Pipe:>>>frompipeimportizip>>>list(range(0,10)|izip(range(1,11)))[(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),(9,10)]>>>map(),select()Apply a conversion expression given as parameter
to each element of the given iterable>>>list([1,2,3]|map(lambdax:x*x))[1,4,9]>>>list([1,2,3]|select(lambdax:x*x))[1,4,9]>>>netcatThe netcat Pipe sends and receive bytes over TCP:data=[b"HEAD / HTTP/1.0\r\n",b"Host: python.org\r\n",b"\r\n",]forpacketindata|netcat("python.org",80):print(packet.decode("UTF-8"))Gives:HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://python.org/
Connection: closepermutations(r=None)Returns all possible permutations:>>>frompipeimportpermutations>>>foritemin'ABC'|permutations(2):...print(item)('A','B')('A','C')('B','A')('B','C')('C','A')('C','B')>>>>>>foriteminrange(3)|permutations:...print(item)(0,1,2)(0,2,1)(1,0,2)(1,2,0)(2,0,1)(2,1,0)>>>reverseLike Python's built-inreversedfunction.>>>frompipeimportreverse>>>list([1,2,3]|reverse)[3,2,1]>>>select(fct)Alias formap(fct), seemap(fct).skip()Skips the given quantity of elements from the given iterable, then yields>>>frompipeimportskip>>>list((1,2,3,4,5)|skip(2))[3,4,5]>>>skip_while(predicate)Like itertools.dropwhile, skips elements of the given iterable
while the predicate is true, then yields others:>>>frompipeimportskip_while>>>list([1,2,3,4]|skip_while(lambdax:x<3))[3,4]>>>sort(key=None, reverse=False)Like Python's built-in "sorted" primitive.>>>frompipeimportsort>>>''.join("python"|sort)'hnopty'>>>[5,-4,3,-2,1]|sort(key=abs)[1,-2,3,-4,5]>>>tLike Haskell's operator ":":>>>frompipeimportt>>>foriin0|t(1)|t(2):...print(i)012>>>tail(n)Yields the given quantity of the last elements of the given iterable.>>>frompipeimporttail>>>foriin(1,2,3,4,5)|tail(3):...print(i)345>>>take(n)Yields the given quantity of elements from the given iterable, likeheadin shell script.>>>frompipeimporttake>>>foriincount()|take(5):...print(i)01234>>>take_while(predicate)Likeitertools.takewhile, yields elements of the
given iterable while the predicate is true:>>>frompipeimporttake_while>>>foriincount()|take_while(lambdax:x**2<100):...print(i)0123456789>>>teetee outputs to the standard output and yield unchanged items, useful for
debugging a pipe stage by stage:>>>frompipeimporttee>>>sum(["1","2","3","4","5"]|tee|map(int)|tee)'1'1'2'2'3'3'4'4'5'515>>>The15at the end is thesumreturning.transpose()Transposes the rows and columns of a matrix.>>>frompipeimporttranspose>>>[[1,2,3],[4,5,6],[7,8,9]]|transpose[(1,4,7),(2,5,8),(3,6,9)]>>>traverseRecursively unfold iterables:>>>list([[1,2],[[[3],[[4]]],[5]]]|traverse)[1,2,3,4,5]>>>squares=(i*iforiinrange(3))>>>list([[0,1,2],squares]|traverse)[0,1,2,0,1,4]>>>uniq(key=None)Like dedup() but only deduplicate consecutive values, using the givenkeyfunction if provided (or else the identity).>>>frompipeimportuniq>>>list([1,1,2,2,3,3,1,2,3]|uniq)[1,2,3,1,2,3]>>>list([1,-1,1,2,-2,2,3,3,1,2,3]|uniq(key=abs))[1,2,3,1,2,3]>>>where(predicate),filter(predicate)Only yields the matching items of the given iterable:>>>list([1,2,3]|where(lambdax:x%2==0))[2]>>>Don't forget they can be aliased:>>>positive=where(lambdax:x>0)>>>negative=where(lambdax:x<0)>>>sum([-10,-5,0,5,10]|positive)15>>>sum([-10,-5,0,5,10]|negative)-15>>>Constructing your ownYou can construct your pipes using thePipeclass like:frompipeimportPipesquare=Pipe(lambdaiterable:(x**2forxiniterable))map=Pipe(lambdaiterable,fct:builtins.map(fct,iterable)>>>As you can see it's often very short to write, and with a bit of luck
the function you're wrapping already takes an iterable as the first
argument, making the wrapping straight forward:>>>fromcollectionsimportdeque>>>frompipeimportPipe>>>end=Pipe(deque)>>>and that's ititrable | end(3)isdeque(iterable, 3):>>>list(range(100)|end(3))[97,98,99]>>>In case it gets more complicated one can usePipeas a decorator to
a function taking an iterable as the first argument, and any other
optional arguments after:>>>fromstatisticsimportmean>>>@Pipe...defrunning_average(iterable,width):...items=deque(maxlen=width)...foriteminiterable:...items.append(item)...yieldmean(items)>>>list(range(20)|running_average(width=2))[0,0.5,1.5,2.5,3.5,4.5,5.5,6.5,7.5,8.5,9.5,10.5,11.5,12.5,13.5,14.5,15.5,16.5,17.5,18.5]>>>list(range(20)|running_average(width=10))[0,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5.5,6.5,7.5,8.5,9.5,10.5,11.5,12.5,13.5,14.5]>>>Euler project samplesFind the sum of all the multiples of 3 or 5 below 1000.>>>sum(count()|where(lambdax:x%3==0orx%5==0)|take_while(lambdax:x<1000))233168>>>Find the sum of all the even-valued terms in Fibonacci which do not
exceed four million.sum(fib()|where(lambdax:x%2==0)|take_while(lambdax:x<4000000))Find the difference between the sum of the squares of the first one
hundred natural numbers and the square of the sum.>>>square=map(lambdax:x**2)>>>sum(range(101))**2-sum(range(101)|square)25164150>>>Going deeperPartial PipesApipecan be parametrized without being evaluated:>>>running_average_of_two=running_average(2)>>>list(range(20)|running_average_of_two)[0,0.5,1.5,2.5,3.5,4.5,5.5,6.5,7.5,8.5,9.5,10.5,11.5,12.5,13.5,14.5,15.5,16.5,17.5,18.5]>>>For multi-argument pipes then can be partially initialized, you can think of curying:some_iterable|some_pipe(1,2,3)is strictly equivalent to:some_iterable|some_pipe(1)(2)(3)So it can be used to specialize pipes, first a dummy example:>>>@Pipe...defaddmul(iterable,to_add,to_mul):..."""Computes (x + to_add) * to_mul to every items of the input."""...foriiniterable:...yield(i+to_add)*to_mul>>>mul=addmul(0)# This partially initialize addmul with to_add=0>>>list(range(10)|mul(10))[0,10,20,30,40,50,60,70,80,90]Which also works with keyword arguments:>>>add=addmul(to_mul=1)# This partially initialize addmul with `to_mul=1`>>>list(range(10)|add(10))[10,11,12,13,14,15,16,17,18,19]>>>But now for something interesting:>>>importre>>>@Pipe...defgrep(iterable,pattern,flags=0):...forlineiniterable:...ifre.match(pattern,line,flags=flags):...yieldline...>>>lines=["Hello","hello","World","world"]>>>forlineinlines|grep("H"):...print(line)Hello>>>Now let's reuse it in two ways, first with a pattern:>>>lowercase_only=grep("[a-z]+$")>>>forlineinlines|lowercase_only:...print(line)helloworld>>>Or now with a flag:>>>igrep=grep(flags=re.IGNORECASE)>>>forlineinlines|igrep("hello"):...print(line)...Hellohello>>>Lazy evaluationPipe uses generators all the way down, so it is naturally lazy.In the following examples we'll useitertools.count:
an infinite generator of integers.We'll make use of theteepipe too, which prints every values that
passe through it.The following example does nothing, nothing is printed byteeso no
value passed through it. It's nice because generating an infinite
sequence of squares is "slow".>>>result=count()|tee|select(lambdax:x**2)>>>Chaining more pipes still won't make previous ones start generating
values, in the following example not a single value is pulled out ofcount:>>>result=count()|tee|select(lambdax:x**2)>>>first_results=result|take(10)>>>only_odd_ones=first_results|where(lambdax:x%2)>>>Same without variables:>>>result=(count()|tee...|select(lambdax:x**2)...|take(10)...|where(lambdax:x%2))>>>Only when values are actually needed, the generators starts to work.In the following example only two values will be extracted out ofcount:0which is squared (to0), passes thetake(10)eaily,
but is dropped bywhere1which is squared (to1), also easily passes thetake(10),
passes thewhere, and passes thetake(1).At this pointtake(1)is satisfied so no other computations need to
be done. Noticeteeprinting0and1passing through it:>>>result=(count()|tee...|select(lambdax:x**2)...|take(10)...|where(lambdax:x%2))>>>print(list(result|take(1)))01[1]>>>DeprecationsIn pipe 1.x a lot of functions were returning iterables and a lot
other functions were returning non-iterables, causing confusion. The
one returning non-iterables could only be used as the last function of
a pipe expression, so they are in fact useless:range(100)|where(lambdax:x%2==0)|addcan be rewritten with no less readability as:sum(range(100)|where(lambdax:x%2==0))so all pipes returning non-iterables were deprecated (raising
warnings), and finally removed in pipe 2.0.What should I do?Oh, you just upgraded pipe, got an exception, and landed here? You
have three solutions:Stop using closing-pipes, replace...|...|...|...|as_listtolist(...|...|...|), that's it, it's even shorter.If "closing pipes" are not an issue for you, and you really like
them, just reimplement the few you really need, it often take a very
few lines of code, or copy them fromhere.If you still rely on a lot of them and are in a hurry, justpip install pipe<2.And start testing your project using thePython Development
Modeso you catch
those warnings before they bite you.But I like them, pleassssse, reintroduce them!This has already been discussed in#74.An@Pipeis often easily implemented in a 1 to 3 lines of code
function, and thepipemodule does not aim at giving all
possibilities, it aims at giving thePipedecorator.So if you need more pipes, closing pipes, weird pipes, you-name-it,
feel free to implement them on your project, and consider the
already-implemented ones as examples on how to do it.See theConstructing your ownparagraph below.