Package: command-runner
command_runnerPlatform agnostic command execution, timed background jobs with live stdout/stderr output capture, and UAC/sudo elevationcommand_runner's purpose is to run external commands from python, just like subprocess on which it relies,
while solving various problems a developer may face among:Handling of all possible subprocess.popen / subprocess.check_output scenarios / python versions in one handy function without encoding / timeout hassleAllow stdout/stderr stream output to be redirected to callback functions / output queues / files so you get to handle output in your application while commands are runningCallback to optional stop check so we can stop execution from outside command_runnerCallback with optional process information so we get to control the process from outside command_runnerCallback once we're finished to easen thread usageOptional process priority and io_priority settingsSystem agnostic functionality, the developer shouldn't carry the burden of Windows & Linux differencesOptional Windows UAC elevation module compatible with CPython, PyInstaller & NuitkaOptional Linux sudo elevation compatible with CPython, PyInstaller & NuitkaIt is compatible with Python 2.7+, tested up to Python 3.11 (backports some newer Python 3.5 functionality) and is tested on both Linux and Windows.
It is also compatible with PyPy Python implementation.
...and yes, keeping Python 2.7 compatibility has proven to be quite challenging.command_runnercommand_runner is a replacement package for subprocess.popen and subprocess.check_output
The main promise command_runner can do is to make sure to never have a blocking command, and always get results.It works as wrapper for subprocess.popen and subprocess.communicate that solves:Platform differencesHandle timeouts even for windows GUI applications that don't return anything to stdoutPython language version differencesHandle timeouts even on earlier Python implementationsHandle encoding even on earlier Python implementationsKeep the promise to always return an exit code (so we don't have to deal with exit codes and exception logic at the same time)Keep the promise to always return the command output regardless of the execution state (even with timeouts, callback interrupts and keyboard interrupts)Can show command output on the fly without waiting the end of execution (withlive_output=Trueargument)Can give command output on the fly to application by using queues or callback functionsCatch all possible exceptions and log them properly with encoding fixesBe compatible, and always return the same result regarless of platformcommand_runner also promises to properly kill commands when timeouts are reached, including spawned subprocesses of such commands.
This specific behavior is achieved via psutil module, which is an optional dependency.command_runner in a nutshellInstall withpip install command_runnerThe following example will work regardless of the host OS and the Python version.fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('ping 127.0.0.1',timeout=10)Guide to command_runnerSetuppip install command_runneror download the latest git releaseAdvanced command_runner usageSpecial exit codesIn order to keep the promise to always provide an exit_code, spcial exit codes have been added for the case where none is given.
Those exit codes are:-250 : command_runner called with incompatible arguments-251 : stop_on function returned True-252 : KeyboardInterrupt-253 : FileNotFoundError, OSError, IOError-254 : Timeout-255 : Any other uncatched exceptionsThis allows you to use the standard exit code logic, without having to deal with various exceptions.Default encodingcommand_runner has anencodingargument which defaults toutf-8for Unixes andcp437for Windows platforms.
Usingcp437ensures that mostcmd.exeoutput is encoded properly, including accents and special characters, on most locale systems.
Still you can specify your own encoding for other usages, like Powershell whereunicode_escapeis preferred.fromcommand_runnerimportcommand_runnercommand=r'C:\Windows\sysnative\WindowsPowerShell\v1.0\powershell.exe --help'exit_code,output=command_runner(command,encoding='unicode_escape')Earlier subprocess.popen implementations didn't have an encoding setting so command_runner will deal with encoding for those.You can also disable command_runner's internal encoding in order to get raw process output (bytes) by passing False boolean.Example:fromcommand_runnerimportcommand_runnerexit_code,raw_output=command_runner('ping 127.0.0.1',encoding=False)On the fly (interactive screen) outputNote: for live output capture and threading, see stream redirection. If you want to run your application while command_runner gives back command output, the best way to go is queues / callbacks.command_runner can output a command output on the fly to stdout, eg show output on screen during execution.
This is helpful when the command is long, and we need to know the output while execution is ongoing.
It is also helpful in order to catch partial command output when timeout is reached or a CTRL+C signal is received.
Example:fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('ping 127.0.0.1',shell=True,live_output=True)Note: using live output relies on stdout pipe polling, which has lightly higher cpu usage.Timeoutscommand_runner has atimeoutargument which defaults to 3600 seconds.This default setting ensures commands will not block the main script execution.
Feel free to lower / higher that setting withtimeoutargument.
Note that a command_runner kills the whole process tree that the command may have generated, even under Windows.fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('ping 127.0.0.1',timeout=30)Remarks on processesUsingshell=Truewill spawn a shell which will spawn the desired child process.
Be aware that under MS Windows, no direct process tree is available.
We fixed this by walking processes during runtime. The drawback is that orphaned processes cannot be identified this way.Disabling logs / silencingcommand_runnerhas it's own logging system, which will log all sorts of error logs.
If you need to disable it's logging, just run with argument silent.
Be aware that logging.DEBUG log levels won't be silenced, by design.Example:fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('ping 127.0.0.1',silent=True)If you also need to disable logging.DEBUG level, you can run the following code which will required logging.CRITICAL only messages whichcommand_runnernever does:importloggingimportcommand_runnerlogging.getLogger('command_runner').setLevel(logging.CRITICAL)Capture methodcommand_runnerallows two different process output capture methods:method='monitor'which is default:A thread is spawned in order to check stop conditions and kill process if neededA main loop waits for the process to finish, then uses proc.communicate() to get it's outputPros:less CPU usageless threadsCons:cannot read partial output on KeyboardInterrupt or stop_on (still works for partial timeout output)cannot use queues or callback functions redirectorsis 0.1 seconds slower than poller methodmethod='poller':A thread is spawned and reads stdout/stderr pipes into output queuesA poller loop reads from the output queues, checks stop conditions and kills process if neededPros:Reads on the fly, allowing interactive commands (is also used withlive_output=True)Allows stdout/stderr output to be written live to callback functions, queues or files (useful when threaded)is 0.1 seconds faster than monitor method, is preferred method for fast batch runningsCons:lightly higher CPU usageExample:fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('ping 127.0.0.1',method='poller')exit_code,output=command_runner('ping 127.0.0.1',method='monitor')stdin stream redirectioncommand_runnerallows to redirect some stream directly into the subprocess it spawns.Example codeimportsysfromcommand_runnerimportcommand_runnerexit_code,output=command_runner("gzip -d",stdin=sys.stdin.buffer)print("Uncompressed data",output)The above program, when run withecho "Hello, World!" | gzip | python myscript.pywill show the uncompressed  stringHello, World!You can use whatever file descriptor you want, basic ones being sys.stdin for text input and sys.stdin.buffer for binary input.stdout / stderr stream redirectioncommand_runner can redirect stdout and/or stderr streams to different outputs:subprocess pipes/dev/null or NULfilesqueuescallback functionsUnless an output redirector is given forstderrargument, stderr will be redirected tostdoutstream.
Note that both queues and callback function redirectors requirepollermethod and will fail if method is not set.Possible output redirection options are:subprocess pipesBy default, stdout writes into a subprocess.PIPE which is read by command_runner and returned asoutputvariable.
You may also pass any other subprocess.PIPE int values tostdoutorstderrarguments./dev/null or NULIfstdout=Falseand/orstderr=Falseargument(s) are given, command output will not be saved.
stdout/stderr streams will be redirected to/dev/nullorNULdepending on platform.Output will always beNone. Seesplit_streamsfor more details using multiple outputs.filesGivingstdoutand/orstderrarguments a string,command_runnerwill consider the string to be a file path where stream output will be written live.Examples:fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('dir',stdout=r"C:/tmp/command_result",stderr=r"C:/tmp/command_error",shell=True)fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('dir',stdout='/tmp/stdout.log',stderr='/tmp/stderr.log',shell=True)Opening a file with the wrong encoding (especially opening a CP437 encoded file on Windows with UTF-8 coded might endup with UnicodedecodeError.)queuesQueue(s) will be filled up by command_runner.In order to keep your program "live", we'll use the threaded version of command_runner which is basically the same except it returns a future result instead of a tuple.Note: With all the best will, there's no good way to achieve this under Python 2.7 without using more queues, so the threaded version is only compatible with Python 3.3+.For Python 2.7, you must create your thread and queue reader yourself (see footnote for a Python 2.7 comaptible example).Threaded command_runner plus queue example:importqueuefromcommand_runnerimportcommand_runner_threadedoutput_queue=queue.Queue()stream_output=""thread_result=command_runner_threaded('ping 127.0.0.1',shell=True,method='poller',stdout=output_queue)read_queue=Truewhileread_queue:try:line=output_queue.get(timeout=0.1)exceptqueue.Empty:passelse:iflineisNone:read_queue=Falseelse:stream_output+=line# ADD YOUR LIVE CODE HERE# Now we may get exit_code and output since result has become available at this pointexit_code,output=thread_result.result()You might also want to read both stdout and stderr queues. In that case, you can create a read loop just like in the following example.
Here we're reading both queues in one loop, so we need to observe a couple of conditions before stopping the loop, in order to catch all queue output:importqueuefromtimeimportsleepfromcommand_runnerimportcommand_runner_threadedstdout_queue=queue.Queue()stderr_queue=queue.Queue()thread_result=command_runner_threaded('ping 127.0.0.1',method='poller',shell=True,stdout=stdout_queue,stderr=stderr_queue)read_stdout=read_stderr=Truewhileread_stdoutorread_stderr:try:stdout_line=stdout_queue.get(timeout=0.1)exceptqueue.Empty:passelse:ifstdout_lineisNone:read_stdout=Falseelse:print('STDOUT:',stdout_line)try:stderr_line=stderr_queue.get(timeout=0.1)exceptqueue.Empty:passelse:ifstderr_lineisNone:read_stderr=Falseelse:print('STDERR:',stderr_line)# ADD YOUR LIVE CODE HEREexit_code,output=thread_result.result()assertexit_code==0,'We did not succeed in running the thread'callback functionsThe callback function will get one argument, being a str of current stream readings.
It will be executed on every line that comes from streams.
Example:fromcommand_runnerimportcommand_runnerdefcallback_function(string):# ADD YOUR CODE HEREprint('CALLBACK GOT:',string)# Launch command_runnerexit_code,output=command_runner('ping 127.0.0.1',stdout=callback_function,method='poller')stop_onIn some situations, you want a command to be aborted on some external triggers.
That's wherestop_onargument comes in handy.
Just pass a function tostop_on, as soon as function result becomes True, execution will halt with exit code -251.Example:fromcommand_runnerimportcommand_runnerdefsome_function():returnTrueifwe_must_stop_executionexit_code,output=command_runner('ping 127.0.0.1',stop_on=some_function)Checking intervalsBy default, command_runner checks timeouts and outputs every 0.05 seconds.
You can increase/decrease this setting viacheck_intervalsetting which accepts floats.
Example:command_runner(cmd, check_interval=0.2)Note that loweringcheck_intervalwill increase CPU usage.Getting current process informationcommand_runnercan provide a subprocess.Popen instance of currently run process as external data.
In order to do so, just declare a function and give it asprocess_callbackargument.Example:fromcommand_runnerimportcommand_runnerdefshow_process_info(process):print('My process has pid:{}'.format(process.pid))exit_code,output=command_runner('ping 127.0.0.1',process_callback=show_process_info)Split stdout and stderrBy default,command_runnerreturns a tuple like(exit_code, output)in which output contains both stdout and stderr stream outputs.
You can alter that behavior by using argumentsplit_stream=True.
In that case,command_runnerwill return a tuple like(exit_code, stdout, stderr).Example:fromcommand_runnerimportcommand_runnerexit_code,stdout,stderr=command_runner('ping 127.0.0.1',split_streams=True)print('exit code:',exit_code)print('stdout',stdout)print('stderr',stderr)On-exit Callbackcommand_runnerallows to execute a callback function once it has finished it's execution.
This might help building threaded programs where a callback is needed to disable GUI elements for example.Example:fromcommand_runnerimportcommand_runnerdefdo_something():print("We're done running")exit_code,output=command_runner('ping 127.0.0.1',on_exit=do_something)Process and IO prioritycommand_runnercan set it's subprocess priority to 'low', 'normal' or 'high', which translate to 15, 0, -15 niceness on Linux and BELOW_NORMAL_PRIORITY_CLASS and HIGH_PRIORITY_CLASS in Windows.
On Linux, you may also directly use priority with niceness int values.You may also set subprocess io priority to 'low', 'normal' or 'high'.Example:fromcommand_runnerimportcommand_runnerexit_code,output=command_runner('some_intensive_process',priority='low',io_priority='high')Other argumentscommand_runnertakesanyargument thatsubprocess.Popen()would take.It also uses the following standard arguments:command (str/list): The command, doesn't need to be a list, a simple string worksvalid_exit_codes (list): List of exit codes which won't trigger error logstimeout (int): seconds before a process tree is killed forcefully, defaults to 3600shell (bool): Shall we use the cmd.exe or /usr/bin/env shell for command execution, defaults to Falseencoding (str/bool): Which text encoding the command produces, defaults to cp437 under Windows and utf-8 under Linuxstdin (sys.stdin/int): Optional stdin file descriptor, sent to the process command_runner spawnsstdout (str/queue.Queue/function/False/None): Optional path to filename where to dump stdout, or queue where to write stdout, or callback function which is called when stdout has outputstderr (str/queue.Queue/function/False/None): Optional path to filename where to dump stderr, or queue where to write stderr, or callback function which is called when stderr has outputno_close_queues (bool): Normally, command_runner sends None to stdout / stderr queues when process is finished. This behavior can be disabled allowing to reuse those queues for other functions wrapping command_runnerwindows_no_window (bool): Shall a command create a console window (MS Windows only), defaults to Falselive_output (bool): Print output to stdout while executing command, defaults to Falsemethod (str): Accepts 'poller' or 'monitor' stdout capture and timeout monitoring methodscheck interval (float): Defaults to 0.05 seconds, which is the time between stream readings and timeout checksstop_on (function): Optional function that when returns True stops command_runner executionon_exit (function): Optional function that gets executed when command_runner has finished (callback function)process_callback (function): Optional function that will take command_runner spawned process as argument, in order to deal with process info outside of command_runnersplit_streams (bool): Split stdout and stderr into two separate resultssilent (bool): Allows to disable command_runner's internal logs, except for logging.DEBUG levels which for obvious reasons should never be silencedpriority (str): Allows to set CPU bound process priority (takes 'low', 'normal' or 'high' parameter)io_priority (str): Allows to set IO priority for process (takes 'low', 'normal' or 'high' parameter)close_fds (bool): Like Popen, defaults to True on Linux and False on Windowsuniversal_newlines (bool): Like Popen, defaults to Falsecreation_flags (int): Like Popen, defaults to 0bufsize (int): Like Popen, defaults to 16384. Line buffering (bufsize=1) is deprecated since Python 3.7Note that ALL other subprocess.Popen arguments are supported, since they are directly passed to subprocess.UAC Elevation / sudo elevationcommand_runner package allowing privilege elevation.
Becoming an admin is fairly easy with command_runner.elevate
You only have to import the elevate module, and then launch your main function with the elevate function.elevation In a nutshellfromcommand_runner.elevateimportelevatedefmain():"""My main function that should be elevated"""print("Who's the administrator, now ?")if__name__=='__main__':elevate(main)elevate function handles arguments (positional and keyword arguments).elevate(main, arg, arg2, kw=somearg)will callmain(arg, arg2, kw=somearg)Advanced elevate usageis_admin() functionThe elevate module has a nifty is_admin() function that returns a boolean according to your current root/administrator privileges.
Usage:fromcommand_runner.elevateimportis_adminprint('Am I an admin ?%s'%is_admin())sudo elevationInitially designed for Windows UAC, command_runner.elevate can also elevate privileges on Linux, using the sudo command.
This is mainly designed for PyInstaller / Nuitka executables, as it's really not safe to allow automatic privilege elevation of a Python interpreter.Example for a binary in/usr/local/bin/my_compiled_python_binaryYou'll have to allow this file to be run with sudo without a password prompt.
This can be achieved in/etc/sudoersfile.Example for Redhat / Rocky Linux, where adding the following line will allow the elevation process to succeed without password:someuser ALL= NOPASSWD:/usr/local/bin/my_compiled_python_binaryFootnotescommand_runner Python 2.7 compatible queue readerThe following example is a Python 2.7 compatible threaded implementation that reads stdout / stderr queue in a thread.
This only exists for compatibility reasons.importqueueimportthreadingfromcommand_runnerimportcommand_runnerdefread_queue(output_queue):"""Read the queue as threadOur problem here is that the thread can live forever if we don't check a global value, which is...well ugly"""stream_output=""read_queue=Truewhileread_queue:try:line=output_queue.get(timeout=1)exceptqueue.Empty:passelse:# The queue reading can be stopped once 'None' is received.iflineisNone:read_queue=Falseelse:stream_output+=line# ADD YOUR LIVE CODE HERE# Create a new queue that command_runner will fill upoutput_queue=queue.Queue()# Create a thread of read_queue() in order to read the queue while command_runner executes the commandread_thread=threading.Thread(target=read_queue,args=(output_queue))read_thread.daemon=True# thread dies with the programread_thread.start()# Launch command_runner, which will be blocking. Your live code goes directly into the threaded functionexit_code,output=command_runner('ping 127.0.0.1',stdout=output_queue,method='poller')