Package: shellous
Async Processes and Pipelinesshellousprovides a concise API for running subprocesses usingasyncio. It is
similar to and inspired bysh.importasynciofromshellousimportshasyncdefmain():result=awaitsh("echo","hello")print(result)asyncio.run(main())BenefitsRun programs asynchronously in a single line.Redirect stdin, stdout and stderr to files, memory buffers, async streams or loggers.Iterate asynchronously over subprocess output.Set timeouts and reliably cancel running processes.Run a program with a pseudo-terminal (pty).Use send() and expect() to manually control a subprocess.Constructpipelinesand useprocess substitutiondirectly from Python (no shell required).Runs on Linux, MacOS, FreeBSD and Windows.Monitor processes being started and stopped withaudit_callbackAPI.RequirementsRequires Python 3.9 or later.Requires an asyncio event loop.Pseudo-terminals require a Unix system.Process substitution requires a Unix system with /dev/fd support.Running a CommandThe tutorial in this README uses the asyncio REPL built into Python. In these examples,>>>is the REPL prompt.Start the asyncio REPL by typingpython3 -m asyncio, and importshfrom theshellousmodule:>>>fromshellousimportshHere's a command that runsecho "hello, world".>>>awaitsh("echo","hello, world")'hello, world\n'The first argument toshis the program name. It is followed by zero or more arguments. Each argument will be
converted to a string. If an argument is a list or tuple, it is flattened recursively.>>>awaitsh("echo",1,2,[3,4,(5,6)])'1 2 3 4 5 6\n'A command does not run until youawaitit. When you run a command usingawait, it returns the value of the standard output interpreted as a UTF-8 string.
It is safe toawaitthe same command object more than once.Here, we create our own echo command with "-n" to omit the newline. Note,echo("abc")will run the same command asecho -n "abc".>>>echo=sh("echo","-n")>>>awaitecho("abc")'abc'Commands areimmutableobjects that represent a program invocation: program name, arguments, environment
variables, redirection operators and other settings. When you use a method to modify aCommand, you are
returning a newCommandobject. The original object is unchanged.You can wrap your commands in a function to improve type safety:>>>fromshellousimportCommand>>>defexclaim(word:str)->Command[str]:...returnsh("echo","-n",f"{word}!!")...>>>awaitexclaim("Oh")'Oh!!'The type hintCommand[str]indicates that the command returns astr.ResultsWhen a command completes successfully, it returns the standard output (or "" if stdout is redirected). For a more detailed response, you can specify that the command should return aResultobject by using the.resultmodifier:>>>awaitecho.result("abc")Result(exit_code=0, output_bytes=b'abc', error_bytes=b'', cancelled=False, encoding='utf-8')AResultobject contains the command'sexit_codein addition to its output. AResultis True if
the command's exit_code is zero. You can access the string value of the output using the.outputproperty:ifresult:=awaitsh.result("cat","some-file"):output=result.outputelse:print(f"Command failed with exit_code={result.exit_code})You can retrieve the string value of the standard error using the.errorproperty. (By default, only the
first 1024 bytes of standard error is stored.)If a command was terminated by a signal, theexit_codewill be the negativesignalnumber.ResultErrorIf you are not using the.resultmodifier and a command fails, it raises aResultErrorexception:>>>awaitsh("cat","does_not_exist")Traceback (most recent call last):...shellous.result.ResultError:Result(exit_code=1, output_bytes=b'', error_bytes=b'cat: does_not_exist: No such file or directory\n', cancelled=False, encoding='utf-8')TheResultErrorexception contains aResultobject with the exit_code and the first 1024 bytes of standard error.In some cases, you want to ignore certain exit code values. That is, you want to treat them as if they are
normal. To do this, you can set theexit_codesoption:>>>awaitsh("cat","does_not_exist").set(exit_codes={0,1})''If there is a problem launching a process, shellous can also raise a separateFileNotFoundErrororPermissionErrorexception.Async ForUsingawaitto run a command collects the entire output of the command in memory before returning it. You
can also iterate over the output lines as they arrive usingasync for.>>>[lineasyncforlineinecho("hi\n","there")]['hi\n', ' there']Use anasync forloop when you want to examine the stream of output from a command, line by line. For example, suppose you want to run tail on a log file.asyncforlineinsh("tail","-f","/var/log/syslog"):if"ERROR"inline:print(line.rstrip())Async WithYou can use a command as an asynchronous context manager. There are two ways to run a program using
a context manager: a low-level API and a high-level API.Byte-by-Byte (Low Level)Useasync withdirectly when you need byte-by-byte
control over the individual streams: stdin, stdout and stderr. To control a standard stream, you
must tell shellous to "capture" it (For more on this, seeRedirection.)cmd=sh("cat").stdin(sh.CAPTURE).stdout(sh.CAPTURE)asyncwithcmdasrun:run.stdin.write(b"abc")run.stdin.close()print(awaitrun.stdout.readline())result=run.result()The streamsrun.stdoutandrun.stderrareasyncio.StreamReaderobjects. The streamrun.stdinis anasyncio.StreamWriterobject. If we didn't specify that stdin/stdout aresh.CAPTURE, the
streamsrun.stdinandrun.stdoutwould beNone.The return value ofrun.result()is aResultobject. Depending on the command settings, this
function may raise aResultErroron a non-zero exit code.:warning: When reading or writing individual streams, you are responsible for managing reads and writes so they don't
deadlock. You may userun.create_taskto schedule a concurrent task.You can also useasync withto run a server. When you do so, you must tell the server
to stop usingrun.cancel(). Otherwise, the context manager will wait forever for the process to exit.asyncwithsh("some-server")asrun:# Send commands to the server here...# Manually signal the server to stop.run.cancel()Prompt with Send/Expect (High Level API)Use theprompt()method to control a process usingsendandexpect. Theprompt()method returns an asynchronous context manager (thePromptclass) that facilitates reading and
writing strings and matching regular expressions.cmd=sh("cat").set(pty=True)asyncwithcmd.prompt()asclient:awaitclient.send("abc")output,_=awaitclient.expect("\r\n")print(output)ThePromptAPI automatically capturesstdinandstdout.Here is another example of controlling a bash co-process running in a docker container.asyncdeflist_packages():"Run bash in an ubuntu docker container and list packages."bash_prompt=re.compile("root@[0-9a-f]+:/[^#]*# ")cmd=sh("docker","run","-it","--rm","-e","TERM=dumb","ubuntu")asyncwithcmd.set(pty=True).prompt(bash_prompt,timeout=3)ascli:# Read up to first prompt.awaitcli.expect()# Disable echo. The `command()` method combines send *and* expect methods.awaitcli.command("stty -echo")# Return list of packages.result=awaitcli.command("apt-cache pkgnames")returnresult.strip().split("\r\n")# You can check the result object's exit code. You can only# access `cli.result` outside the `async with` block.assertcli.result.exit_code==0Theprompt()API does not raise aResultErrorwhen a command exits with an error status.
Typically, you'll see an EOFError when you were expecting to read a response. You can check the
exit status by retrieving the Prompt'sresultproperty outside of theasync withblock.Redirectionshellous supports the redirection operators|and>>. They work similar to how they work in
the unix shell. Shellous does not support use of<or>for redirection. Instead, replace these
with|.To redirect to or from a file, use apathlib.Pathobject. Alternatively, you can redirect input/output
to a StringIO object, an open file, a Logger, or use a special redirection constant likesh.DEVNULL.:warning: When combining the redirect operators withawait, you must use parentheses;awaithas higher
precedence than|and>>.Redirecting Standard InputTo redirect standard input, use the pipe operator|with the argument on theleft-side.
Here is an example that passes the string "abc" as standard input.>>>cmd="abc"|sh("wc","-c")>>>awaitcmd'       3\n'To read input from a file, use aPathobject frompathlib.>>>frompathlibimportPath>>>cmd=Path("LICENSE")|sh("wc","-l")>>>awaitcmd'     201\n'Shellous supports different STDIN behavior when using different Python types.Python TypeBehavior as STDINstrRead input from string object.bytes, bytearrayRead input from bytes object.PathRead input from file specified byPath.File, StringIO, ByteIORead input from open file object.intRead input from existing file descriptor.asyncio.StreamReaderRead input fromStreamReader.sh.DEVNULLRead input from/dev/null.sh.INHERITRead input from existingsys.stdin.sh.CAPTUREYou will write to stdin interactively.Redirecting Standard OutputTo redirect standard output, use the pipe operator|with the argument on theright-side. Here is an
example that writes to a temporary file.>>>output_file=Path("/tmp/output_file")>>>cmd=sh("echo","abc")|output_file>>>awaitcmd''>>>output_file.read_bytes()b'abc\n'To redirect standard output with append, use the>>operator.>>>cmd=sh("echo","def")>>output_file>>>awaitcmd''>>>output_file.read_bytes()b'abc\ndef\n'Shellous supports different STDOUT behavior when using different Python types.Python TypeBehavior as STDOUT/STDERRPathWrite output to the file path specified byPath.bytearrayWrite output to a mutable byte array.File, StringIO, ByteIOWrite output to an open file object.intWrite output to existing file descriptor at its current position. ◆logging.LoggerLog each line of output. ◆asyncio.StreamWriterWrite output toStreamWriter. ◆sh.CAPTURECapture output forasync with. ◆sh.DEVNULLWrite output to/dev/null. ◆sh.INHERITWrite output to existingsys.stdoutorsys.stderr. ◆◆ For these types, there is no difference between using|and>>.Shellous doesnotsupport redirecting standard output/error to a plainstrorbytesobject.
If you intend to redirect output to a file, you must use apathlib.Pathobject.Redirecting Standard ErrorBy default, the first 1024 bytes read from standard error are stored in the Result object.
Any further bytes are discarded. You can change the 1024 byte limit using theerror_limitoption.To redirect standard error, use thestderrmethod. Standard error supports the
same Python types as standard output. To append, setappend=Truein thestderrmethod.To redirect stderr to the same place as stdout, use thesh.STDOUTconstant. If you also
redirect stdout tosh.DEVNULL, you will only receive the standard error.>>>cmd=sh("cat","does_not_exist").stderr(sh.STDOUT)>>>awaitcmd.set(exit_codes={0,1})'cat: does_not_exist: No such file or directory\n'To redirect standard error to the hosting program'ssys.stderr, use thesh.INHERITredirect
option.>>>cmd=sh("cat","does_not_exist").stderr(sh.INHERIT)>>>awaitcmdcat: does_not_exist: No such file or directoryTraceback (most recent call last):...shellous.result.ResultError:Result(exit_code=1, output_bytes=b'', error_bytes=b'', cancelled=False, encoding='utf-8')If you redirect stderr, it will no longer be stored in the Result object, and theerror_limitoption
will not apply.Default RedirectionsFor regular commands, the default redirections are:Standard input is read from the empty string ("").Standard out is buffered and stored in the Result object (BUFFER).First 1024 bytes of standard error is buffered and stored in the Result object (BUFFER).However, the default redirections are adjusted when using a pseudo-terminal (pty):Standard input is captured and ignored (CAPTURE).Standard out is buffered and stored in the Result object (BUFFER).Standard error is redirected to standard output (STDOUT).When you use thePromptAPI, the standard input and standard output are automatically redirected to CAPTURE.PipelinesYou can create a pipeline by combining commands using the|operator. A pipeline feeds the standard out of one process into the next process as standard input. Here is the shellous
equivalent to the bash command:ls | grep README>>>pipe=sh("ls")|sh("grep","README")>>>awaitpipe'README.md\n'A pipeline returns aResultif the last command in the pipeline has the.resultmodifier. To set other
options likeencodingfor a Pipeline, set them on the last command.>>>pipe=sh("ls")|sh("grep","README").result>>>awaitpipeResult(exit_code=0, output_bytes=b'README.md\n', error_bytes=b'', cancelled=False, encoding='utf-8')Error reporting for a pipeline is implemented similar to using the-o pipefailshell option.Pipelines support the sameawait/async for/async withoperations that work on a single command, including
thePromptAPI.>>>[line.strip()asyncforlineinpipe]['README.md']Process Substitution (Unix Only)You can pass a shell command as an argument to another. Here is the shellous equivalent to the bash
command:grep README <(ls).>>>cmd=sh("grep","README",sh("ls"))>>>awaitcmd'README.md\n'Use.writableto write to a command instead.>>>buf=bytearray()>>>cmd=sh("ls")|sh("tee",sh("grep","README").writable|buf)|sh.DEVNULL>>>awaitcmd''>>>bufbytearray(b'README.md\n')The above example is equivalent tols | tee >(grep README > buf) > /dev/null.TimeoutsYou can specify a timeout using thetimeoutoption. If the timeout expires, shellous will raise
aTimeoutError.>>>awaitsh("sleep",60).set(timeout=0.1)Traceback (most recent call last):...TimeoutErrorTimeouts are just a special case ofcancellation. When a command is cancelled, shellous terminates
the running process and raises aCancelledError.>>>t=asyncio.create_task(sh("sleep",60).coro())>>>t.cancel()True>>>awaittTraceback (most recent call last):...CancelledErrorBy default, shellous will send a SIGTERM signal to the process to tell it to exit. If the process does not
exit within 3 seconds, shellous will send a SIGKILL signal. You can change these defaults with thecancel_signalandcancel_timeoutsettings. A command is not considered fully cancelled until the
process exits.Pseudo-Terminal Support (Unix Only)To run a command through a pseudo-terminal, set theptyoption to True.>>>awaitsh("echo","in a pty").set(pty=True)'in a pty\r\n'Alternatively, you can pass aptyfunction to configure the tty mode and size.>>>ls=sh("ls").set(pty=shellous.cooked(cols=40,rows=10,echo=False))>>>awaitls("README.md","CHANGELOG.md")'CHANGELOG.md\tREADME.md\r\n'Shellous provides three built-in helper functions:shellous.cooked(),shellous.raw()andshellous.cbreak().Context ObjectsYou can store shared command settings in an immutable context object (CmdContext). To create a new
context object, specify your changes to the default contextsh:>>>auditor=lambdaphase,info:print(phase,info["runner"].name)>>>sh_audit=sh.set(audit_callback=auditor)Now all commands created withsh_auditwill log their progress using the audit callback.>>>awaitsh_audit("echo","goodbye")start echostop echo'goodbye\n'You can also create a context object that specifies all return values areResultobjects.>>>rsh=sh.result>>>awaitrsh("echo","whatever")Result(exit_code=0, output_bytes=b'whatever\n', error_bytes=b'', cancelled=False, encoding='utf-8')OptionsBothCommandandCmdContextsupport options to control their runtime behavior. Some of these options (timeout,
pty, audit_callback, and exit_codes) have been described above. See theshellous.Optionsclass for
more information.You can retrieve an option fromcmdwithcmd.options.<option>. For example, usecmd.options.encodingto obtain the encoding:>>>cmd=sh("echo").set(encoding="latin1")>>>cmd.options.encoding'latin1'CommandandCmdContextuse the.set()method to specify most options:OptionDescriptionpathSearch path to use instead of thePATHenvironment variable. (Default=None)envAdditional environment variables to pass to the command. (Default={})inherit_envTrue if command should inherit the environment variables from the current process. (Default=True)encodingText encoding of input/output streams. You can specify an error handling scheme by including it after a space, e.g. "ascii backslashreplace". (Default="utf-8 strict")exit_codesSet of exit codes that do not raise aResultError. (Default={0})timeoutTimeout in seconds to wait before cancelling the process. (Default=None)cancel_timeoutTimeout in seconds to wait for a cancelled process to exit before forcefully terminating it. (Default=3s)cancel_signalThe signal sent to a process when it is cancelled. (Default=SIGTERM)alt_nameAlternate name for the process used for debug logging. (Default=None)pass_fdsAdditional file descriptors to pass to the process. (Default={})pass_fds_closeTrue if descriptors inpass_fdsshould be closed after the child process is launched. (Default=False)ptyUsed to allocate a pseudo-terminal (PTY). (Default=False)close_fdsTrue if process should close all file descriptors when it starts. This setting defaults to False to align withposix_spawnrequirements. (Default=False)audit_callbackProvide function to audit stages of process execution. (Default=None)coerce_argProvide function to coerceCommandarguments to strings whenstr()is not sufficient. (Default=None)error_limitMaximum number of initial bytes of STDERR to store inResultobject. (Default=1024)envUse theenv()method toaddto the list of environment variables. Theenv()method supports keyword parameters.
You can callenv()more than once and the effect is additive.>>>cmd=sh("echo").env(ENV1="a",ENV2="b").env(ENV2=3)>>>cmd.options.env{'ENV1': 'a', 'ENV2': '3'}Use theenvoption withset()when you want toreplace allthe environment variables.input, output, errorWhen you apply a redirection operator to aCommandorCmdContext, the redirection targets
are also stored in theOptionsobject. To change these, use the.stdin(),.stdout(), or.stderr()methods or the redirection operator|.OptionDescriptioninputThe redirection target for standard input.input_closeTrue if standard input should be closed after the process is launched.outputThe redirection target for standard output.output_appendTrue if standard output should be open for append.output_closeTrue if standard output should be closed after the process is launched.errorThe redirection target for standard error.error_appendTrue if standard error should be open for append.error_closeTrue if standard error should be closed after the process is launched.Type CheckingShellous fully supports PEP 484 type hints.CommandsCommands are generic on the return type, eitherstrorResult. You will specify the
type of a command object asCommand[str]orCommand[Result].Use theresultmodifier to obtain aCommand[Result]from aCommand[str].fromshellousimportsh,Command,Resultcmd1:Command[str]=sh("echo","abc")# When you `await cmd1`, the result is a `str` object.cmd2:Command[Result]=sh.result("echo","abc")# When you `await cmd2`, the result is a `Result` object.CmdContextTheCmdContextclass is also generic on eitherstrorResult.fromshellousimportsh,CmdContext,Resultsh1:CmdContext[str]=sh.set(path="/bin:/usr/bin")# When you use `sh1` to create commands, it produces `Command[str]` object with the given path.sh2:CmdContext[Result]=sh.result.set(path="/bin:/usr/bin")# When you use `sh2` to create commands, it produces `Command[Result]` objects with the given path.LoggingFor verbose logging, shellous supports aSHELLOUS_TRACEenvironment variable. Set the
value ofSHELLOUS_TRACEto a comma-delimited list of options:detail:  Enables detailed logging used to trace the steps of running a command.prompt: Enables logging in thePromptclass when controlling a program
using send/expect.all: Enables all logging options.Shellous uses the built-in Pythonloggingmodule. After enabling these options,
theshellouslogger will display log messages at theINFOlevel.Without these options enabled, Shellous generates almost no log messages.