Package: pipetools
Complete documentationpipetoolsenables function composition similar to using Unix pipes.It allows forward-composition and piping of arbitrary functions - no need to decorate them or do anything extra.It also packs a bunch of utils that make common operations more convenient and readable.Source is ongithub.Why?Piping and function composition are some of the most natural operations there are for
plenty of programming tasks. Yet Python doesn’t have a built-in way of performing them.
That forces you to either deep nesting of function calls or adding extraglue code.ExampleSay you want to create a list of python files in a given directory, ordered by
filename length, as a string, each file on one line and also with line numbers:>>>print(pyfiles_by_length('../pipetools'))1. ds_builder.py
2. __init__.py
3. compat.py
4. utils.py
5. main.pyAll the ingredients are already there, you just have to glue them together. You might write it like this:defpyfiles_by_length(directory):all_files=os.listdir(directory)py_files=[fforfinall_filesiff.endswith('.py')]sorted_files=sorted(py_files,key=len,reverse=True)numbered=enumerate(py_files,1)rows=("{0}.{1}".format(i,f)fori,finnumbered)return'\n'.join(rows)Or perhaps like this:defpyfiles_by_length(directory):return'\n'.join('{0}.{1}'.format(*x)forxinenumerate(reversed(sorted([fforfinos.listdir(directory)iff.endswith('.py')],key=len)),1))Or, if you’re a mad scientist, you would probably do it like this:pyfiles_by_length=lambdad:(reduce('{0}\n{1}'.format,map(lambdax:'%d.%s'%x,enumerate(reversed(sorted(filter(lambdaf:f.endswith('.py'),os.listdir(d)),key=len))))))Butthere should be one – and preferably only one – obvious way to do it.So which one is it? Well, to redeem the situation,pipetoolsgive you yet
another possibility!pyfiles_by_length=(pipe|os.listdir|where(X.endswith('.py'))|sort_by(len).descending|(enumerate,X,1)|foreach("{0}.{1}")|'\n'.join)Why would I do that, you ask? Comparing to thenativePython code, it’sEasier to read– minimal extra clutterEasier to understand– one-way data flow from one step to the next, nothing else to keep track ofEasier to change– want more processing? just add a step to the pipelineRemoves some bug opportunities– did you spot the bug in the first example?Of course it won’t solve all your problems, but a great deal of codecanbe expressed as a pipeline, giving you the above benefits. Read on to see how it works!Installation$pipinstallpipetoolsUh, what’s that?UsageThe pipeThepipeobject can be used to pipe functions together to
form new functions, and it works like this:frompipetoolsimportpipef=pipe|a|b|c# is the same as:deff(x):returnc(b(a(x)))A real example, sum of odd numbers from 0 tox:fromfunctoolsimportpartialfrompipetoolsimportpipeodd_sum=pipe|range|partial(filter,lambdax:x%2)|sumodd_sum(10)# -> 25Note that the chain up to thesumis lazy.Automatic partial application in the pipeAs partial application is often useful when piping things together, it is done
automatically when thepipeencounters a tuple, so this produces the same
result as the previous example:odd_sum=pipe|range|(filter,lambdax:x%2)|sumAs of0.1.9, this is even more powerful, seeX-partial.Built-in toolsPipetools contain a set ofpipe-utilsthat solve some common tasks. For
example there is a shortcut for the filter class from our example, calledwhere():frompipetoolsimportpipe,whereodd_sum=pipe|range|where(lambdax:x%2)|sumWell that might be a bit more readable, but not really a huge improvement, but
wait!If apipe-utilis used as first or second item in the pipe (which happens
quite often) thepipeat the beginning can be omitted:odd_sum=range|where(lambdax:x%2)|sumSeepipe-utils’ documentation.OK, but what about the ugly lambda?where(), but alsoforeach(),sort_by()and otherpipe-utilscan be
quite useful, but require a function as an argument, which can either be a named
function – which is OK if it does something complicated – but often it’s
something simple, so it’s appropriate to use alambda. Except Python’s
lambdas are quite verbose for simple tasks and the code gets cluttered…X objectto the rescue!frompipetoolsimportwhere,Xodd_sum=range|where(X%2)|sumHow ‘bout that.Read more about the X object and it’s limitations.Automatic string formattingSince it doesn’t make sense to compose functions with strings, when a pipe (or apipe-util) encounters a string, it attempts to use it for(advanced) formatting:>>>countdown=pipe|(range,1)|reversed|foreach('{}...')|' '.join|'{}boom'>>>countdown(5)'4... 3... 2... 1... boom'Feeding the pipeSometimes it’s useful to create a one-off pipe and immediately run some input
through it. And since this is somewhat awkward (and not very readable,
especially when the pipe spans multiple lines):result=(pipe|foo|bar|boo)(some_input)It can also be done using the>operator:result=some_input>pipe|foo|bar|booNoteNote that the above method of input won’t work if the input object
defines__gt__foranyobject - including the pipe. This can be the case for example with
some objects from math libraries such as NumPy. If you experience strange
results try falling back to the standard way of passing input into a pipe.But wait, there is moreCheckoutthe Maybe pipe,partial application on steroidsorautomatic data structure creationin thefull documentation.