Package: exec-helpers
exec-helpersExecution helpers for simplified usage of subprocess and ssh.
Why another subprocess wrapper and why no clearparamiko?Historicallyparamikooffers good ssh client, but with specific limitations:
you can call command with timeout, but without receiving return code,
or call command and wait for return code, but without timeout processing.In the most cases, we are need just simple SSH client with comfortable API for calls, calls via SSH proxy and checking return code/stderr.
This library offers this functionality with deadlock free polling and friendly result objects
(with inline decoding of XML Element tree, YAML, JSON, binary or just strings).
In addition this library offers the same API for subprocess calls, but with specific limitation: no parallel calls
(for protection from race conditions).Pros:STDOUT and STDERR polling during command execution - no deadlocks.The same API for subprocess and ssh.Free software: Apache licenseOpen Source:https://github.com/python-useful-helpers/exec-helpersPyPI packaged:https://pypi.python.org/pypi/exec-helpersSelf-documented code: docstrings with types in commentsTested: see badges on topSupport multiple Python versions:Python 3.8
Python 3.9
Python 3.10
Python 3.11
Python 3.12This package includes:SSHClient- historically the first one helper, which used for SSH connections.
Several API calls for sFTP also presents.SSHAuth- class for credentials storage.SSHClientdoes not store credentials as-is, but usesSSHAuthfor it.
Objects of this class can be copied between ssh connection objects, also it used forexecute_through_host.Subprocess-subprocess.Popenwrapper with timeouts, polling and almost the same API, asSSHClient(except specific flags, likecwdfor subprocess andget_ttyfor ssh).async_api.Subprocess- the same, asSubprocesshelper, but works with asyncio.
.. note:: for WindowsProactorEventLoopor another non-standard event loop should be used!ExecResult- class for execution results storage.
Contains exit code, stdout, stderr and getters for decoding as JSON, YAML, XML (and LXML) element tree, string, bytearray
and brief strings (up to 7 lines).ExitCodes- enumerator for standard Linux exit codes. BASH return codes (produced from signal codes) also available.InstallationStandard:pip install exec-helpersExtras:yaml- installPyYamlfor yaml decoding (PyYAMLis main decoder,ruamel.YAMLalso supported as fallback.)xml- installdefusedxmlfor safe XML parsing toxml.etree.ElementTree.Element.lxml- installlxmlfor advanced XML parsing. Can be unsafe.ALL_FORMATS(all-formats) - install all parsers. When new parsers will be added, it will ne also supported.UsageSSHClientBasic initialization ofSSHClientcan be done without construction of specific objects:client=exec_helpers.SSHClient(host,username="username",password="password")If ssh agent is running - keys will be collected by paramiko automatically,
but if keys are in specific location  - it should be loaded manually and provided as iterable object ofparamiko.PKey.For advanced cases or re-use of credentials,SSHAuthobject should be used.
It can be collected from connection object via propertyauth.Creation from scratch:auth=exec_helpers.SSHAuth(username='username',# str | Nonepassword='password',# str | Nonekey=None,# type: paramiko.PKey | Nonekeys=None,# type: Iterable[paramiko.PKey] | Nonekey_filename=None,# type: list[str] | Nonepassphrase=None,# str | None)Key is a main connection key (always tried first) and keys are alternate keys.
Key filename is a filename or list of filenames with keys, which should be loaded.
Passphrase is an alternate password for keys, if it differs from main password.
If main key now correct for username - alternate keys tried, if correct key found - it became main.
If no working key - password is used and None is set as main key.Context manager is available, connection is closed and lock is released on exit from context.Notecontext manager is strictly not recommended in scenarios with fast reconnect to the same host with te same credentials.SubprocessContext manager is available, subprocess is killed and lock is released on exit from context.Base methodsMain methods areexecute,check_callandcheck_stderrfor simple executing, executing and checking return code
and executing, checking return code and checking for empty stderr output.
This methods are almost the same forSSHClientandSubprocess, except specific flags.NoteBy default ALL methods have timeout 1 hour, infinite waiting can be enabled, but it’s special case.result:ExecResult=helper.execute(command,# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: int | float | None# Keyword only:log_mask_re=None,# str | Nonestdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boollog_stdout=True,# type: boolopen_stderr=True,# type: boollog_stderr=True,# type: bool**kwargs)result:ExecResult=helper.check_call(command,# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: type: int | float | Noneerror_info=None,# str | Noneexpected=(0,),# type: Iterable[int | ExitCodes]raise_on_err=True,# type: bool# Keyword only:log_mask_re=None,# str | Nonestdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boollog_stdout=True,# type: boolopen_stderr=True,# type: boollog_stderr=True,# type: boolexception_class=CalledProcessError,# type[CalledProcessError]**kwargs)result:ExecResult=helper.check_stderr(command,# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: type: int | float | Noneerror_info=None,# str | Noneraise_on_err=True,# type: bool# Keyword only:expected=(0,),# Iterable[int | ExitCodes]log_mask_re=None,# str | Nonestdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boollog_stdout=True,# type: boolopen_stderr=True,# type: boollog_stderr=True,# type: boolexception_class=CalledProcessError,# type[CalledProcessError])result:ExecResult=helper(# Lazy way: instances are callable and uses `execute`.command,# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: int | float | None# Keyword only:log_mask_re=None,# str | Nonestdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boollog_stdout=True,# type: boolopen_stderr=True,# type: boollog_stderr=True,# type: bool**kwargs)NoteIf command is provided asIterable[str],shell=Truewill be still used,
but all command components will be joined with escaping to protect from shell processing.If no STDOUT or STDERR required, it is possible to disable this FIFO pipes via**kwargswith flagsopen_stdout=Falseandopen_stderr=False.The next command level uses lower level and kwargs are forwarded, so expected exit codes are forwarded fromcheck_stderr.
Implementation specific flags are always set via kwargs.If required to mask part of command from logging,log_mask_reattribute can be set global over instance or provided with command.
All regex matched groups will be replaced by‘<*masked*>’.result:ExecResult=helper.execute(command="AUTH='top_secret_key'; run command",# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: Optional[int]log_mask_re=r"AUTH\s*=\s*'(\w+)'"# str | None)result.cmdwill be equal toAUTH=’<*masked*>’; run commandExecResultExecution result object has a set of useful properties:cmd- Commandexit_code- Command return code. If possible to decode using enumerators for Linux -> it used.ok->bool. Command return code is 0 (EX_OK).stdin->str. Text representation of stdin.stdout->tuple[bytes]. Raw stdout output.stderr->tuple[bytes]. Raw stderr output.stdout_bin->bytearray. Binary stdout output.stderr_bin->bytearray. Binary stderr output.stdout_str->str. Text representation of output.stderr_str->str. Text representation of output.stdout_brief->str. Up to 7 lines from stdout (3 first and 3 last if >7 lines).stderr_brief->str. Up to 7 lines from stderr (3 first and 3 last if >7 lines).stdout_json- STDOUT decoded as JSON.stdout_yaml- STDOUT decoded as YAML. Accessible only ifPyYAMLorruamel.YAMLlibrary installed.
(Extras:yaml)stdout_xml- STDOUT decoded as XML toElementTreeusingdefusedxmllibrary. Accessible only ifdefusedxmllibrary installed.
(Extras:xml)stdout_lxml- STDOUT decoded as XML toElementTreeusinglxmllibrary. Accessible only iflxmllibrary installed.
(Extras:lxml) Can be insecure.timestamp->Optional(datetime.datetime). Timestamp for received exit code.SSHClient specificSSHClient commands support get_pty flag, which enables PTY open on remote side.
PTY width and height can be set via keyword arguments, dimensions in pixels are always 0x0.Possible to call commands in parallel on multiple hosts if it’s not produce huge output:results:dict[tuple[str,int],ExecResult]=SSHClient.execute_together(remotes,# type: Iterable[SSHClient]command,# type: str | Iterable[str]timeout=1*60*60,# type: type: int | float | Noneexpected=(0,),# type: Iterable[int | ExitCodes]raise_on_err=True,# type: bool# Keyword only:stdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boolopen_stderr=True,# type: boollog_mask_re=None,# str | Noneexception_class=ParallelCallProcessError# type[ParallelCallProcessError])results# type: dict[tuple[str, int], exec_result.ExecResult]Results is a dict with keys = (hostname, port) and and results in values.
By default execute_together raises exception if unexpected return code on any remote.To open new connection using current as proxy is accessible methodproxy_to. Basic usage example:conn:SSHClient=client.proxy_to(host,username="username",password="password")Notefor full command API please rely API documentation.For execute through SSH host can be usedexecute_through_hostmethod:result:ExecResult=client.execute_through_host(hostname,# type: strcommand,# type: str | Iterable[str]# Keyword only:auth=None,# type: SSHAuth | Noneport=22,# type: inttimeout=1*60*60,# type: type: int | float | Noneverbose=False,# type: boolstdin=None,# type: bytes | str | bytearray | Noneopen_stdout=True,# type: boollog_stdout=True,# type: boolopen_stderr=True,# type: boollog_stderr=True,# type: boollog_mask_re=None,# str | Noneget_pty=False,# type: boolwidth=80,# type: intheight=24# type: int)Where hostname is a target hostname, auth is an alternate credentials for target host.SSH client implements fast sudo support via context manager:NoteIn case of combination sudo + chroot, chroot will be applied first. For alternative order write command with chroot manually.Commands will be run with sudo enforced independently from client settings for normal usage:withclient.sudo(enforce=True):...Commands will be runwithout sudoindependently from client settings for normal usage:withclient.sudo(enforce=False):...“Permanent client setting”:client.sudo_mode=mode# where mode is True or FalseSSH Client supports sFTP for working with remote files:withclient.open(path,mode='r')asf:...For fast remote paths checks available methods:exists(path)->bool>>>conn.exists('/etc/passwd')Truestat(path)->paramiko.sftp_attr.SFTPAttributes>>>conn.stat('/etc/passwd')<SFTPAttributes:[size=1882uid=0gid=0mode=0o100644atime=1521618061mtime=1449733241]>>>>str(conn.stat('/etc/passwd'))'-rw-r--r--   1 0        0            1882 10 Dec 2015  ?'isfile(path)->bool>>>conn.isfile('/etc/passwd')Trueisdir(path)->bool>>>conn.isdir('/etc/passwd')FalseAdditional (non-standard) helpers:mkdir(path: str)- execute mkdir -p pathrm_rf(path: str)- execute rm -rf pathupload(source: str, target: str)- upload file or from source to target using sFTP.download(destination: str, target: str)- download file from target to destination using sFTP.Subprocess specificKeyword arguments:cwd - working directory.env - environment variables dict.Noteshell=trueis always set.async_api.Subprocess specificAll standard methods are coroutines. Async context manager also available.Example:asyncwithhelper:result:ExecResult=awaithelper.execute(command,# type: str | Iterable[str]verbose=False,# type: booltimeout=1*60*60,# type: int | float | None**kwargs)TestingThe main test mechanism for the packageexec-helpersis usingtox.
Available environments can be collected viatox -lCI systemsFor code checking several CI systems is used in parallel:GitHub actions:is used for checking: PEP8, pylint, bandit, installation possibility and unit tests.