Package: pipepy
A Python library for invoking and interacting with shell commands.Table of contentsTable of contentsWhy? Comparison with other similar frameworksInstallation and testingIntro, basic usageCustomizing commandsLazinessBackground commandsRedirecting output from/to filesPipesInteracting with background processesAltering the behavior of commandsMiscellaneouspymakeTODOsWhy? Comparison with other similar frameworksXonsh: Xonsh allows you to combine shell and Python
and enables very powerful scripting and interactive sessions. This library
does the same to a limited degree. However, Xonsh introduces a new language
that is a superset of Python. The main goal of this library that sets it
apart is that it is intended to be a pure Python implementation, mainly
aimed at scripting.shandpieshell: These are much closer to
the current library in that they are pure Python implementations. The
current library, however, tries to improve on the following aspects:It tries to apply more syntactic sugar to make the invocations feel more
like shell invocations.It tries to offer ways to have shell commands interact with python code in
powerful and intuitive ways.Installation and testingpython-mpipinstallpipepyOr, if you want to modify the code while trying it out:gitclonehttps://github.com/kbairak/pipepycdpipepy
python-mpipinstall-e.To run the tests, you need to first install the testing requirements:python-mpipinstall-rtest_requirements.txt

pymaketest# orpytestThere are a few morepymaketargets to assist with testing during
development:covtest: Produces and opens a coverage reportwatchtest: Listens for changes in the source code files and reruns the
tests automaticallydebugtest: Runs the tests without capturing their output so that you can
insert a debug statementpymakeis a console script that is part ofpipepythat aims to be a
replacement for GNUmake, with the difference that theMakefiles are
written in Python. More on thisbelow.Intro, basic usagefrompipepyimportls,grepprint(ls)# prints contents of current folderifls|grep('info.txt'):print('info.txt found')Most shell commands are importable straight from thepipepymodule. Dashes in
commands' names are converted to underscore (docker-compose→docker_compose). Commands that cannot be found automatically can be created
with the PipePy constructor:frompipepyimportPipePycustom_command=PipePy('./bin/custom')python_script=PipePy('python','script.py')Customizing commandsCalling a command with non empty arguments will return a modified unevaluated
copy. So the following are equivalent:frompipepyimportPipePyls_l=PipePy('ls','-l')# Is equivalent tols_l=PipePy('ls')('-l')There is a number of other ways you can customize a command:Globs: globbing will be applied to all positional arguments:frompipepyimportechoprint(echo('*'))# Will print all files in the current folderYou can useglob.escapeif you want to avoid this functionality:importglobfrompipepyimportls,echoprint(ls)# <<< **a *a *aaprint(echo('*a'))# <<< **a *a *aaprint(echo(glob.escape('*a')))# <<< *aKeyword arguments:frompipepyimportlsls(sort="size")# Equivalent to ls('--sort=size')ls(I="files.txt")# Equivalent to ls('-I', 'files.txt')ls(sort_by="size")# Equivalent to ls('--sort-by=size')ls(escape=True)# Equivalent to ls('--escape')ls(escape=False)# Equivalent to ls('--no-escape')Since keyword arguments come after positional arguments, if you want the
final command to have a different ordering you can invoke the command
multiple times:frompipepyimportlsls('-l',sort="size")# Equivalent to ls('-l', '--sort=size')ls(sort="size")('-l')# Equivalent to ls('--sort=size', '-l')Attribute access:frompipepyimportgitgit.push.origin.bugfixes# Equivalent to git('push', 'origin', 'bugfixes')Minus sign:frompipepyimportlsls-'l'# Equivalent to ls('-l')ls-'default'# Equivalent to ls('--default')This is to enable making the invocations look more like the shell:frompipepyimportlsl,t='l','t'ls-l-t# Equivalent to ls('-l', '-t')You can callpipepy.overload_chars(locals())in your script to assign all
ascii letters to variables of the same name.importpipepyfrompipepyimportlspipepy.overload_chars(locals())ls-l-t# Equivalent to ls('-l', '-t')LazinessCommands are evaluated lazily. For example, this will not actually do anything:frompipepyimportwgetwget('http://...')Invoking aPipePyinstance with non-empty arguments will return anunevaluatedcopy supplied with the extra arguments. A command will be
evaluated when its output is used. This can be done with the following ways:Accessing thereturncode,stdoutandstderrproperties:frompipepyimportechocommand=echo("hello world")command.returncode# <<< 0command.stdout# <<< 'hello world\n'command.stderr# <<< ''Evaluating the command as a string objectfrompipepyimportlsresult=str(ls)# orprint(ls)Converting a command to astrreturns itsstdout.Evaluating the command as a boolean object:frompipepyimportls,grepcommand=ls|grep('info.txt')bool(command)# <<< Trueifcommand:print("info.txt found")The command will be truthy if itsreturncodeis 0.Invoking the.as_table()method:frompipepyimportpsps.as_table()# <<< [{'PID': '11233', 'TTY': 'pts/4', 'TIME': '00:00:01', 'CMD': 'zsh'},# ...  {'PID': '17673', 'TTY': 'pts/4', 'TIME': '00:00:08', 'CMD': 'ptipython'},# ...  {'PID': '18281', 'TTY': 'pts/4', 'TIME': '00:00:00', 'CMD': 'ps'}]Iterating over a command object:frompipepyimportlsforfilenameinls:print(filename.upper)command.iter_words()iterates over the words of the command'sstdout:frompipepyimportpslist(ps.iter_words())# <<< ['PID', 'TTY', 'TIME', 'CMD',# ...  '11439', 'pts/5', '00:00:00', 'zsh',# ...  '15532', 'pts/5', '00:00:10', 'ptipython',# ...  '15539', 'pts/5', '00:00:00', 'ps']Redirecting the output to something else (this will be further explained
below):frompipepyimportls,grepls>'files.txt'ls>>'files.txt'ls|grep('info.txt')# `ls` will be evaluated, `grep` will notls|lambdaoutput:output.upper()If you are not interested in the output of a command but want to evaluate it
nevertheless, you can call it with empty arguments. So, this will actually
invoke the command (and wait for it to finish).frompipepyimportwgetwget('http://...')()Background commandsCalling.delay()on aPipePyinstance will return a copy that, although not
evaluated, will have started running in the background (taking inspiration from
Celery's.delay()method for the name). Again, if you try to access its output, it will perform
the rest of the evaluation process, which is simply to wait for it to finish:frompipepyimportwgeturls=[...]# All downloads will happen in the background simultaneouslydownloads=[wget(url).delay()forurlinurls]# You can do something else here in Python while the downloads are working# This will call __bool__ on all downloads and thus wait for themifnotall(downloads):print("Some downloads failed")If you are not interested in the output of a background command, you should
take care at some point to call.wait()on it. Otherwise its process will not
be waited for and if the parent Python process ends, it will kill all the
background processes:frompipepyimportwgetdownload=wget('...').delay()# Do something elsedownload.wait()You can supply the optionaltimeoutargument towait. If the timeout is
set, it expires and the process hasn't finished, aTimeoutExpiredexception
will be raised. (This is the sameTimeoutExpiredexception class from thesubprocessmodule but you can import it from thepipepymodule too)frompipepyimportsleepcommand=sleep(100).delay()command.wait(5)# <<< TimeoutExpired: Command '['sleep', '30']' timed out after 5 secondsAt any point, you can callpipepy.jobs()to get a list of non-waited-for
commands. In case you want to do some cleaning up, there is also apipepy.wait_jobs()function. This should be used with care however as, if any
of the background jobs aren't finished or are stuck,wait_jobs()may hang for
an unknown amount of time.wait_jobsalso accepts the optionaltimeoutargument.Redirecting output from/to filesThe>,>>and<operators work similar to how they work in a shell:ls>'files.txt'# Will overwrite files.txtls>>'files.txt'# Will append to files.txtgrep('info.txt')<'files.txt'# Will use files.txt as inputThese also work with file-like objects:importosfrompipepyimportls,grepbuf=io.StringIO()ls>bufls('subfolder')>>bufbuf.seek(0)grep('filename')<bufIf you want to combine input and output redirections, you have to put the first
redirection inside parentheses because of how python likes to deal with
comparison chains:frompipepyimportgzipgzip=gzip(_text=False)gzip<'uncompressed.txt'>'uncompressed.txt.gz'# Wrong!(gzip<'uncompressed.txt')>'uncompressed.txt.gz'# Correct!PipesThe|operator is used to customize where a command gets its input from and
what it does with its output. Depending on the types of the operands, different
behaviors will emerge:1. Both operands arePipePyinstancesIf both operands are commands, the result will be as similar as possible to
what would have happened in a shell:frompipepyimportgit,grepifgit.diff(name_only=True)|grep('readme.txt'):print("readme was changed")If the left operand was previously evaluated, then it's output (stdout) will
be passed directly as input to the right operand. Otherwise, both commands will
be executed in parallel andleft's output will be streamed intoright.2. Left operand is any kind of iterable (including string)If the left operand is any kind of iterable, its elements will be fed to the
command's stdin one by one:importrandomfrompipepyimportgrepresult=["John is 18 years old\n","Mary is 25 years old"]|grep("Mary")print(result)# <<< Mary is 25 years olddefmy_stdin():for_inrange(500):yieldf"{random.randint(1,100)}\n"result=my_stdin()|grep(17)print(result)# <<< 17# ... 17# ... 17# ... 17# ... 17If it's a string, it will be fed all at onceresult="John is 18 years old\nMary is 25 years old"|grep("Mary")# Equivalent toresult=["John is 18 years old\nMary is 25 years old"]|grep("Mary")In both cases, ie in all cases where the right operand is aPipePyobject,
the return value of the pipe operation will be anunevaluatedcopy, which
will be evaluated when we try to access its output. This means that we can take
advantage of our usual background functionality:frompipepyimportfind,xargscommand=find('.')|xargs.wccommand=command.delay()# Do something else in the meantimeforlineincommand:# Here we wait for the command to finishlinecount,wordcount,charcount,filename=line.split()# ...It also means that the left operand, if it's an iterable, will be consumed when
the command is evaluated.frompipepyimportgrepiterable=(lineforlinein["foo\n","bar\n"])command=iterable|grep("foo")command.stdout# <<< 'foo\n'list(iterable)# <<< []iterable=(lineforlinein["foo\n","bar\n"])command=iterable|grep("foo")list(iterable)# Lets consume the iterable prematurely# <<< ["foo\n", "bar\n"]command.stdout# <<< ''Also, if you prefer an invocation style that resembles a function call more
than a shell pipe operation, ie if you want to pass a command's input as an
argument, you can use the_inputkeyword argument:frompipepyimportgrep,lsgrep('setup',_input=ls)# Is equivalent tols|grep('setup')or use the square-bracket notation:frompipepyimportgrep,lsgrep('setup')[ls]# Is equivalent tols|grep('setup')(We use parentheses for arguments and square brackets for input because
parentheses allow us to take advantage of keyword arguments which are a good
fit for command-line options)This works both for inputs that are iterables and commands.3. Right operand is a functionThe function's arguments need to either be:a subset ofreturncode,output,errorsora subset ofstdout,stderrThe ordering of the arguments is irrelevant since the function's signature will
be inspected to assign the proper values.In the first case, the command will be waited for and its evaluated output will
be made available to the function's arguments.frompipepyimportwcdeflines(output):forlineinoutput.splitlines():try:lines,words,chars,filename=line.split()exceptValueError:continueprint(f"File{filename}has{lines}lines,{words}words and{chars}""characters")wc('*')|lines# <<< File demo.py has 6 lines, 15 words and 159 characters# ... File main.py has 174 lines, 532 words and 4761 characters# ... File interactive2.py has 10 lines, 28 words and 275 characters# ... File interactive.py has 12 lines, 34 words and 293 characters# ... File total has 202 lines, 609 words and 5488 charactersIn the second case, the command and the function will be executed in parallel
and the command'sstdoutandstderrstreams will be made available to the
function.importrefrompipepyimportpingdefmean_ping(stdout):pings=[]forlineinstdout:match=re.search(r'time=([\d\.]+) ms$',line.strip())ifnotmatch:continuetime=float(match.groups()[0])pings.append(time)iflen(pings)%10==0:print(f"Mean time is{sum(pings)/len(pings)}ms")ping('-c',30,"google.com")|mean_ping# >>> Mean time is 71.96000000000001 ms# ... Mean time is 72.285 ms# ... Mean time is 72.19666666666667 msIf the command ends before the function, thennext(stdout)will raise aStopIteration. If the function ends before the command, the command'sstdinwill be closed.The return value of the pipe operation will be the return value of the
function. The function can even include the wordyieldand thus return a
generator that can be piped into another command.Putting all of this together, we can do things like:frompipepyimportcat,grepdefmy_input():yield"line one\n"yield"line two\n"yield"line two\n"yield"something else\n"yield"line three\n"defmy_output(stdout):forlineinstdout:yieldline.upper()print(my_input()|cat|grep('line')|my_output|grep("TWO"))# <<< LINE TWO# ... LINE TWO4. Right operand is a generatorThis is one of the more exotic forms of piping. Here we take advantage of
Python'spassing values into a generatorfunctionality. The original generator must send and receive data with thea = (yield b)syntax. The result of the pipe operation will be another
generator that will yield whatever the original generator yields while, in the
original generator, the return value of eachyieldcommand will be the next
non-empty line of thePipePyinstance:frompipepyimportechodefupperize():line=yieldwhileTrue:line=(yieldline.upper())# Remember, `upperize` is a function, `upperize()` is a generatorlist(echo("aaa\nbbb")|upperize())# <<< ["AAA\n", "BBB\n"]And, since the return value of the pipe operation is a generator, it can be
piped into another command:print(echo("aaa\nbbb")|upperize()|grep("AAA"))# <<< AAAInteracting with background processesThere are 3 ways to interact with a background process:read-only,write-onlyandread/write. We have already coveredread-onlyandwrite-only:1. Incrementally sending data to a commandThis is done by piping from an iterable to a command. The command actually runs
in in parallel with the iterable and the iterable's data is fed to the command
as it becomes available. We will slightly modify the previous example to better
demonstrate this:importrandomimporttimefrompipepyimportgrepdefmy_stdin():start=time.time()for_inrange(500):time.sleep(.01)yieldf"{time.time()-start}{random.randint(1,100)}\n"command=my_stdin()|grep('-E',r'\b17$',_stream_stdout=True)command()# <<< 0.3154888153076172 17# ... 1.5810892581939697 17# ... 1.7773401737213135 17# ... 2.8303775787353516 17# ... 3.4419643878936768 17# ... 4.511774301528931  17Here,grepis actually run in in parallel with the generator and matches are
printed as they are found since the command's output is being streamed to the
console, courtesy of the_stream_stdoutargument (more on thisbelow).2. Incrementally reading data from a commandThis can be done either by piping the output of a command to a function with a
subset ofstdin,stdoutandstderras its arguments, or a generator, as
we demonstratedbefore, or by iterating over
a command's output:importtimefrompipepyimportpingstart=time.time()forlineinping('-c',3,'google.com'):print(time.time()-start,line.strip().upper())# <<< 0.15728354454040527 PING GOOGLE.COM (172.217.169.142) 56(84) BYTES OF DATA.# ... 0.1574106216430664  64 BYTES FROM SOF02S32-IN-F14.1E100.NET (172.217.169.142): ICMP_SEQ=1 TTL=103 TIME=71.8 MS# ... 1.1319730281829834  64 BYTES FROM 142.169.217.172.IN-ADDR.ARPA (172.217.169.142): ICMP_SEQ=2 TTL=103 TIME=75.3 MS# ... 2.1297826766967773  64 BYTES FROM 142.169.217.172.IN-ADDR.ARPA (172.217.169.142): ICMP_SEQ=3 TTL=103 TIME=73.4 MS# ... 2.129857063293457# ... 2.129875659942627   --- GOOGLE.COM PING STATISTICS ---# ... 2.1298911571502686  3 PACKETS TRANSMITTED, 3 RECEIVED, 0% PACKET LOSS, TIME 2004MS# ... 2.129910707473755   RTT MIN/AVG/MAX/MDEV = 71.827/73.507/75.253/1.399 MSAgain, thepingcommand is actually run in parallel with the body of the
for-loop and each line is given to the body of the for-loop as it becomes
available.3. Reading data from and writing data to a commandLets assume we have a command that makes the user take a math quiz. A normal
interaction with this command would look like this:→ math_quiz
3 + 4 ?
→ 7
Correct!
8 + 2 ?
→ 12
Wrong!
→ Ctrl-dUsing python to interact with this command in a read/write fashion can be done
with awithstatement:frompipepyimportmath_quizresult=[]withmath_quizas(stdin,stdout,stderr):stdout=(line.strip()forlineinstdoutifline.strip())try:for_inrange(3)question=next(stdout)a,_,b,_=question.split()answer=str(int(a)+int(b))stdin.write(answer+"\n")stdin.flush()verdict=next(stdout)result.append((question,answer,verdict))exceptStopIteration:passresult# <<< [('10 + 7 ?', '17', 'Correct!'),# ...  ('5 + 5 ?', '10', 'Correct!'),# ...  ('5 + 5 ?', '10', 'Correct!')]stdin,stdoutandstderrare the open file streams of the background
process. When the body of thewithblock finishes, an EOF is sent to the
process and it is waited for.You need to remember to end lines fed tostdinwith a newline character if
the command expects it. Also, don't forget to callstdin.flush()every now
and then.You can callwithon a pipe expression that involvesPipePyobjects. In
that case, eachPipePyobject'sstdoutwill be connected to the next one'sstdin, thestdinoffered to the body of thewithblock will be thestdinof the leftmost command and thestdoutandstderroffered to the
body of thewithblock will be thestdoutandstderrof the rightmost
command:frompipepyimportcat,grepcommand=cat|grep("foo")|cat|cat|cat# We might as well keep goingwithcommandas(stdin,stdout,stderr):stdin.write("foo1\n")stdin.write("bar2\n")stdin.write("foo3\n")stdin.close()assertnext(stdout).strip()=="foo1"assertnext(stdout).strip()=="foo3"Altering the behavior of commandsBinary modeAll commands are executed in text mode, which means that they deal withstrobjects. This can cause problems. For example:frompipepyimportgzipresult="hello world"|gzipprint(result.stdout)# <<< Traceback (most recent call last):# ... ...# ... UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start bytegzipcannot work in text mode because its output is binary data that cannot
be utf-8-decoded. When text mode is not desirable, a command can be converted
to binary mode setting its_textparameter toFalse:frompipepyimportgzipgzip=gzip(_text=False)result="hello world"|gzipprint(result.stdout)# <<< b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\xe1\x02\x00-;\x08\xaf\x0c\x00\x00\x00'Input and output will be converted from/to binary by using the 'UTF-8'
encoding. In the previous example, our input's type wasstrand was
utf-8-encoded before being fed intogzip. You can change the encoding with
the_encodingkeyword argument:frompipepyimportgzipgzip=gzip(_text=False)result="καλημέρα"|gzipprint(result.stdout)# <<< b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x01\x10\x00\xef\xff\xce\xba\xce\xb1\xce\xbb\xce\xb7\xce\xbc\xce\xad\xcf\x81\xce\xb1"\x15g\xab\x10\x00\x00\x00'result="καλημέρα"|gzip(_encoding="iso-8859-7")print(result.stdout)# <<< b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03{\xf5\xf0\xf5\xf37w?>\x04\x00\x1c\xe1\xc0\xf7\x08\x00\x00\x00'Streaming to consoleDuring invocation, you can set the_stream_stdoutand_stream_stderrkeyword arguments toTrue. This means that the respective stream will not be
captured by the result, but streamed to the console. This allows the user to
interact with interactive commands. Consider the following 2 examples:fzfworks like this:It gathers a list of choices from itsstdinIt displays the choices onstderr, constantly refreshing it depending
on what the user inputsIt starts directly capturing keystrokes on the keyboard, bypassingstdin, to allow the user to make their choice.When the user presses Enter, it prints the choice to itsstdoutTaking all this into account, we can do the following:frompipepyimportfzffzf=fzf(_stream_stderr=True)# This will open an fzf session to let us choose between "John" and "Mary"print("John\nMary"|fzf)# <<< Marydialogworks similar tofzf,
but swapsstdoutwithstderr:It gathers a list of choices from its argumentsIt displays the choices onstdout, constantly refreshing it depending
on what the user inputsIt starts directly capturing keystrokes on the keyboard, bypassingstdin, to allow the user to make their choice.When the user presses Enter, it prints the choice to itsstderrTaking all this into account, we can do the following:frompipepyimportdialogdialog=dialog(_stream_stdout=True)# This will open a dialog session to let us choose between "John" and "Mary"result=dialog(checklist=True)('Choose name',30,110,0,"John",'',"on","Mary",'',"off")print(result.stderr)# <<< JohnAlso, during a script, you may not be interested in capturing the output of a
command but may want to stream it to the console to show the command's output
to the user. You can force a command sto stream its whole output by setting the_streamparameter:frompipepyimportwgetwget('https://...',_stream=True)()Whilestdoutandstderrwill not be captured,returncodewill and thus
you can still use the command in boolean expressions:frompipepyimportwgetifwget('https://...',_stream=True):print("Download succeeded")else:print("Download failed")You can callpipepy.set_always_stream(True)to make streaming to the console
the default behavior. This may be desirable in some situations, like Makefiles
(seebelow).importpipepyfrompipepyimportlspipepy.set_always_stream(True)ls()# Alsost equivalent to `ls(_stream=True)()`pipepy.set_always_stream(False)Similarly to how setting_stream=Trueforces a command to stream its output
to the console, setting_stream=Falseforces it to capture its output even ifset_always_streamhas been called:importpipepyfrompipepyimportlspipepy.set_always_stream(True)ls()# Will stream its outputls(_stream=False)()# Will capture its outputpipepy.set_always_stream(False)ExceptionsYou can call.raise_for_returncode()on anevaluatedresult to raise an
exception if its returncode is not 0 (think ofrequests's.raise_for_status()):frompipepyimportping,PipePyErrorresult=ping("asdf")()# Remember, we have to evaluate it firstresult.raise_for_returncode()# <<< PipePyError: (2, '', 'ping: asdf: Name or service not known\n')try:result.raise_for_returncode()exceptPipePyErrorasexc:print(exc.returncode)# <<< 2print(exc.stdout)# <<< ""print(exc.stderr)# <<< ping: asdf: Name or service not knownYou can callpipepy.set_always_raise(True)to haveallcommands raise an
exception if their returncode is not zero.importpipepyfrompipepyimportpingpipepy.set_always_raise(True)ping("asdf")()# <<< PipePyError: (2, '', 'ping: asdf: Name or service not known\n')If "always raise" is set, you can still force a command to suppress its
exception by setting_raise=False:importpipepyfrompipepyimportpingpipepy.set_always_raise(True)try:ping("asdf")()# Will raise an exceptionexceptExceptionasexc:print(exc)# <<< PipePyError: (2, '', 'ping: asdf: Name or service not known\n')try:ping("asdf",_raise=False)()# Will not raise an exceptionexceptExceptionasexc:print(exc)"Interactive" modeWhen "interactive" mode is set, the__repr__method will simply returnself.stdout + self.stderr. This enables some very basic functionality for the
interactive python shell. To set interactive mode, runpipepy.set_interactive(True):importpipepyfrompipepyimportls,overload_charspipepy.set_interactive(True)ls# <<< demo.py# ... interactive2.py# ... interactive.py# ... main.pyoverload_chars(locals())ls-l# <<< total 20# ... -rw-r--r-- 1 kbairak kbairak  159 Feb  7 22:05 demo.py# ... -rw-r--r-- 1 kbairak kbairak  275 Feb  7 22:04 interactive2.py# ... -rw-r--r-- 1 kbairak kbairak  293 Feb  7 22:04 interactive.py# ... -rw-r--r-- 1 kbairak kbairak 4761 Feb  8 20:42 main.pyMaking alterations "permanent"SincePipePyobjects treat their list of arguments as list of strings simply
passed onto thesubprocess.Popenfunction, and since there is no special
significance to the first argument even though it is technically the command
being executed, you can cretePipePyinstances with the alterations we
discussed and use them as templates for commands that will inherit these
alterations:stream_sh=PipePy(_stream=True)stream_sh# <<< PipePy()stream_sh._stream# <<< Truestream_sh.ls# <<< PipePy('ls')stream_sh.ls._stream# <<< Truer=stream_sh.ls()# <<< check_tag.py  Makefile.py     setup.cfg  tags# ... htmlcov       pyproject.toml  setup.py   test_requirements.txt# ... LICENSE       README.md       srcr.stdout# <<< Noner.returncode# <<< 0raise_sh=PipePy(_raise=True)raise_sh# <<< PipePy()raise_sh.false# <<< PipePy('false')raise_sh.false()# <<< Traceback (most recent call last):# ... ...# ... pipepy.exceptions.PipePyError: (1, '', '')This can work as a more contained alternative toset_always_streamandset_always_raise.Miscellaneous.terminate(),.kill()and.send_signal()simply forward the method call
to the underlyingPopenobject.Here are some utilities implemented withinpipepythat don't make use of
shell subprocesses, but we believe are useful for scripting.cdIn its simplest form,pipepy.cdis an alias toos.chdir:frompipepyimportcd,pwdprint(pwd())# <<< /foocd('bar')print(pwd())# <<< /foo/barcd('..')print(pwd())# <<< /fooBut it can also be used as a context processor for temporary directory changes:print(pwd())# <<< /foowithcd("bar"):print(pwd())# <<< /foo/barprint(pwd())# <<< /fooexportIn its simplest form,pipepy.exportis an alias toos.environ.update:importosfrompipepyimportexportprint(os.environ['HOME'])# <<< /home/fooexport(PATH="/home/foo/bar")print(os.environ['HOME'])# <<< /home/foo/barBut it can also be used as a context processor for temporary environment
changes:print(os.environ['HOME'])# <<< /home/foowithexport(PATH="/home/foo/bar"):print(os.environ['HOME'])# <<< /home/foo/barprint(os.environ['HOME'])# <<< /home/fooIf an environment variable is further modified within the body of thewithblock, it is not reverted upon exit:withexport(PATH="/home/foo/bar"):export(PATH="/home/foo/BAR")print(os.environ['HOME'])# <<< /home/foo/BARsourceThesourcefunction runs a bash script, extracts the resulting environment
variables that have been set in the script and saves them on the current
environment. Similarly toexport, it can be used as a context processor (in
fact, it usesexportinternally):# envexportAAA=aaaimportosfrompipepyimportsourcewithsource('env'):print(os.environ['AAA'])# <<< aaa'AAA'inos.environ# <<< Falsesource('env')print(os.environ['AAA'])# <<< aaaThe following keyword-only arguments are available tosource:recursive(boolean, defaults toFalse): If set, all files with the same
name in the current directory and all its parents will be sourced, in reverse
order. This allows nesting of environment variables:- /
  |
  + - home/
      |
      - kbairak/
        |
        + - env:
        |     export COMPOSE_PROJECT_NAME="pipepy"
        |
        + - project/
            |
            + - env:
                  export COMPOSE_FILE="docker-compose.yml:docker-compose-dev.yml"frompipepyimportcd,source,docker_composecd('/home/kbairak/project')source('env',recursive=True)# Now I have both `COMPOSE_PROJECT_NAME` and `COMPOSE_FILE`The files/home/kbairak/envand/home/kbairak/project/envwere sourced,
in that order.quiet(boolean, defaults toTrue): If the sourced file fails,sourcewill usually skip its sourcing without complaint and move on to the next one
(ifrecursiveis set). Withquiet=False, an exception will be raised and
the environment will not be updated.shell(string, defaults to'bash'): The shell command used to perform
the sourcing.pymakeBundled with this library there is a command calledpymakewhich aims to
replicate the syntax and behavior of GNUmakeas much as possible, but in
Python. AMakefile.pyfile looks like this (this is actually part of the
Makefile of the current library):importpipepyfrompipepyimportpython,rmpipepy.set_always_stream(True)pipepy.set_always_raise(True)defclean():rm('-rf',"build","dist")()defbuild(clean):python('-m',"build")()defpublish(build):python('-m',"twine").upload("dist/*")()You can now runpymake publishto run thepublishmake target, along with
its dependencies. The names of the functions' arguments are used to define the
dependencies, socleanis a dependency ofbuildandbuildis a dependency
ofpublish.(You don't have to usepipepycommands insideMakefile.py, but admittedly
it's a very good fit)The arguments hold any return values of the dependency targets:defa():return1defb():return2defc(a,b):print(a+b)→pymakec# ← 3Each dependency will be executed at most once, even if it's used as a
dependency more than once:defa():print("pymake target a")defb(a):print("pymake target b")defc(a,b):print("pymake target c")→pymakec# ← pymake target a# ← pymake target b# ← pymake target cYou can set theDEFAULT_PYMAKE_TARGETglobal variable to define the default
target.frompipepyimportpytestDEFAULT_PYMAKE_TARGET="test"deftest():pytest(_stream=True)()pymakevariablesApart from dependencies, you can use function arguments to define variables
that can be overridden by the invocation ofpymake. This can be done in 2
ways:Using the function's keyword arguments:# Makefile.pydefgreeting(msg="world"):print(f"hello{msg}")→pymakegreeting# ← hello world→pymakegreetingmsg=Bill# ← hello BillUsing global variables defined inMakefile.py:# Makefile.pymsg="world"defgreeting():print(f"hello{msg}")→pymakegreeting# ← hello world→pymakegreetingmsg=Bill# ← hello BillShell completion forpymakepymakesupports shell completion for bash and zsh.In bash, run:eval$(pymake--setup-bash-completion)Then you will be able to see things like (example taken frompipepy's
Makefile):[kbairak@kbairakdelllaptop pipepy]$ pymake <TAB><TAB>
build      clean      debugtest  publish    watchtest
checks     covtest    html       testIn zsh, run:eval$(pymake--setup-zsh-completion)Then you will be able to see things like (example taken frompipepy's
Makefile):(pipepy) ➜  pipepy git:(master) ✗ pymake <TAB>
build      -- Build package
checks     -- Run static checks on the code (flake8, isort)
clean      -- Clean up build directories
covtest    -- Run tests and produce coverge report
debugtest  -- Run tests without capturing their output. This makes using an interactive debugger possible
html       -- Run tests and open coverage report in browser
publish    -- Publish package to PyPI
test       -- Run tests
watchtest  -- Automatically run tests when a source file changesThe descriptions are taken from thepymaketargets' docstrings.You can put theevalstatements in your.bashrc/.zshrc.TODOsTimeout for waitRedirect input/output from/to file-like objectsStream and capture at the same time (wrapper class for file-like object?)withblocks where PipePy invocations forward to the context's stdin, eg:frompipepyimportsshwithssh("some-host")ashost:r=host.ls()# Will actually send 'ls\n' to ssh's stdin