Package: pyshrimp
PyShrimpPyShrimp is combination of utilities designed to support easy creation of small python scripts instead of getting your
hands dirty with shell scripting languages like bash.When trying to write simple script to hack some ad-hoc task it's easy to end up with following choices:use bash - it's easy to write simple script which invokes some processes. But then it's hard to do something more
advanced (lacking good support for types, lacking support for arrays...)use python - is far better in handing different types, have good support for various collections, can easily be
expanded by installing dependencies from pip. But then there is painful overhead in setting up such script
(handling virtualenv to separate dependencies, or using one big env for all scripts) and doing more complex process
invocations beyond simplesubprocess.check_output(...)is also far from ideal.The PyShrimp aims at solving this - it's purpose is to remove barriers so developers can use python for simple
shell-scripting purpose. No more over-grown bash scripts, no more pain with env setup or subprocess handling!Note: the features provided are not expected to replace poetry and other tools like that - if you can afford the
complexity then it's probably better to use those ;). But for single-file scripts it should be way easier to go with
PyShrimp.TL;DR: ShowcaseThe rest of README explains features in detail. For those impatient there is quick example script - the code
is often worth 1000 words (although it covers only small subset of features):#!/usr/bin/env pyshrimp# $opts: magic,devloop# $requires: click==7.0,requests==2.22.0importrequestsfromclickimportstylefrompyshrimpimportlog,as_dot_dictdefslog(message,**kwargs):log(style(message,**kwargs))slog('Loading issues...')res=as_dot_dict(requests.get('https://jira.atlassian.com/rest/api/2/search',params={'jql':'created > -1d','maxResults':'5'}).json(),'res')slog(f'Issues ({res.total}in total):',fg='green',bold=True)forissueinres.issues:line_color='red'ifissue.fields.issuetype.name=='Bug'elseNonef=issue.fieldsslog(f' * [{issue.key}] [{f.issuetype.name}] [{f.status.name}]{f.summary}',fg=line_color)To run this script you need to have the PyShrimp installed:pipinstallpyshrimp
chmod+xthescript.py
./thescript.pyPyShrimp will automatically initialize new virtual environment, install dependencies declared in file header,
and execute the script with devloop:FeaturesVirtual env setup -$requiresPyShrimp parses script header and looks for the# $requires: ...lines.
Each such line can contain one or more pip-style requirements, for example:#!/usr/bin/env pyshrimp# $requires: click==7.0,requests==2.22.0# $requires: PyYAML==5.3.1Before script execution PyShrimp will create dedicated virtual env in~/.cache/pyshrimp/{hash}/where{hash}is hash value from the requirements. The environment is created only once - subsequent
runs of script will use already existing environment. Scripts with exactly the same dependencies will
re-use single virtual environment.The$requirements_filecan be used to load requirements list from external file:#!/usr/bin/env pyshrimp# $requirements_file: requirements.txtCreation of new scriptTo quickly create new script just run thenewcommand:pyshrimpnewmy-new-script.pyThe file created will contain skeleton of script. Script will have the executable mode set already.Script startup -runTherunfunction makes it easy to run the main function with some extra capabilities.
The default behavior is to set up logging and run the script main function.
In addition to this it's possible to enable additional tweaks:devloop- runs script in loop - PyShrimp will re-execute script automatically after it changeselevate- ensures that script is running as root - uses sudo to elevate permissionsExample script with both features enabled:#!/usr/bin/env pyshrimpfrompyshrimpimportrun,logdefmain():log('Hello world!')run(main,devloop=True,elevate=True)Power of magicThere is alternative 'magic' way to run script setup (logging configuration) and achieve the bonus features
like devloop and elevate. This is called magic as the PyShrimp no longer will only set up virtualenv but also
will wrap the code with extra "magical" setup.The following script will have exactly the same behavior as previously presented withrunmethod:#!/usr/bin/env pyshrimp# $opts: magic,devloop,elevatefrompyshrimpimportloglog('Hello world!')As you can see there is less "boilerplate" code when using this approach.There is however one downside - the code run with justpythonwill behave differently - the magic options will not
activate and also the logging will be not configured. This can be surprising when running script with debugger so
use the magic wisely ;).Useful utilitiesThere are few useful utilities provided:as_dot_dict- creates dictionary wrapper with support for property-like access
to the values:as_dot_dict(d).some_key.some_list[1].some_valueunwrap_dot_dict- un-wraps DotDicts back into the raw dict/listls,glob_ls- lists files and directorieswrite_to_file,read_file,read_file_bin- file content manipulationchmod_set,chmod_unset- sets/unsets file mode bitsacquire_file_lock,FileBasedLock- handles file based lockingre_match_all- runs regular expression matching across the list and returns selected
group from the matched elementsin_background- runs function in background thread poolStringWrapper- provides few methods especially useful for parsing process outputparse_table- parses table-like output intoParsedTablecreate_regex_splitter- createsregex_splitter- useful to handle unusual table/column-like outputwait_until,wait_until_gen- handles waiting for some result with timeout using periodic pollingYou can see example usage inexamplesand also intests.Easily run programs - simple subprocess helperThe subprocess helper simplifies the task of running processes and handling the results.frompyshrimpimportrun_processprint(run_process('echo -n 123 | wc -c',run_in_shell=True).raise_if_not_ok().standard_output)More advanced subprocess helperThecmdandshell_cmdwill produceCommandobject which can be executed with extra params:frompyshrimpimportshell_cmdwc_c=shell_cmd('echo -n "$1" | wc -c',check=True)print(wc_c('123456789').standard_output.strip())//9print(wc_c.exec('123').standard_output.strip())//3It is worth noting here that standard_output and error_output
arewrapped withStringWrapperto ease output parsing.Pipeline supportMotivationThe nice property about bash scripts is how easy it is to executed process and pass the output between processes. It's
all possible in python but the overhead is really too big to use in small scripts. It's more convenient to fallback into
subprocess with partial shell script than glue together the processes directly in python code.SolutionTheExecutionPipelineis designed to address those concerns. You can easily stitch together few processes and
functions together to process the input and output just like in shell scripts.There are two approaches - standard object-oriented python code and more shell like pipeline syntax.Object oriented usagefrompyshrimpimportExecutionPipeline,cmdp=ExecutionPipeline()# feed pipeline with text inputp.attach_text('Hello world!')# run the wc commandp.attach(cmd('wc'))# run awk, using the shell wrapperp.attach("awk '{print $3}'")# process the output with function - pad with zerosp.attach_function(lambdastdin:f'{int(stdin.strip()):05}')# close pipeline and collect outputres=p.close().stdoutprint(res)# 00012Shell-like pipe syntaxThe shell-like syntax is "abusing" the python feature which allows overriding behavior of binary or operator. It's not
very elegant (could be confusing for people) but on the other hand this fits nicely the purpose - being easy replacement
of shell pipes.frompyshrimpimportPIPE,PIPE_END_STDOUT,cmdres=(# feed pipeline with text inputPIPE.text('Hello world!')# run the wc command|cmd('wc')# run awk, using the shell wrapper|"awk '{print $3}'"# process the output with function - pad with zeros|(lambdastdin:f'{int(stdin.strip()):05}')# close pipeline and collect output|PIPE_END_STDOUT)print(res)# 00012Mixed approachYou can mix the shell-like and object-oriented syntax (if you have good reason to do so ;)). Under the cover
theExecutionPipelineis being used even in the shell-like syntax.frompyshrimpimportPIPE,cmdp=(# feed pipeline with text inputPIPE.text('Hello world!')# run the wc command|cmd('wc')# run awk, using the shell wrapper|"awk '{print $3}'")# process the output with function - pad with zerosp.attach_function(lambdastdin:f'{int(stdin.strip()):05}')# close pipeline and collect outputres=p.close().stdoutprint(res)# 00012LimitationsThings obviously missing in current version that you should be aware of:Limited error handling supportOnly text streams are officially supported, with UTF-8 hardcodedOnly connection of standard output is supported - error output behavior is undetermined (most likely flows to stderr)Those limitations can be addressed in future (if there is enough demand and willingness to introduce the change).TroubleshootingDiagnosing virtual env setup failureYou can setPYSHRIMP_LOGenvironment variable to1this will instruct boostrap code to
produce diagnostic messages:%PYSHRIMP_LOG=1./show_recently_created_issues.pyab[PyShrimp:bootstrap]INFO:target:./show_recently_created_issues.py[PyShrimp:bootstrap]INFO:args:['a','b'][PyShrimp:bootstrap]INFO:Usingrequirementsd82e6efbb5ea5ba895b6fe103b4c50bf3ac75eb3:['pyshrimp','click==7.0'][PyShrimp:bootstrap]INFO:Executingthescript:['~/.cache/pyshrimp/virtual_envs/d82e6e.../bin/python','-u','-m','pyshrimp._internal.wrapper.magicwrapper','--','./show_recently_created_issues.py','a','b']HelloworldSupported environmentThis project was developed onUbuntu Linux. It should work with any linux system, but I can imagine the tests
failing in case some system binaries are missing.It was not tested on macOS (probably will work fine) and some parts for sure will not work on Windows
(e.g. shell wrapping is depending on bash).Feel free to raise bugs found when using this project on macOS - it shouldn't be too hard to address them.The author does not have plans to introduce support for Windows but contributions are welcome ;).LicenseThe project is licensed underMIT Licensewith exceptions listed below.Project license exceptions:The files indoc/assets/img/logoanddocs/assets/img/logodirectory
are licensed underCC BY-SA 3.0 license.ContributionsFeel free to contribute to this project - I'll do my best to review and accept contributions.Please include at least some happy-path tests for your changes.Imaginary Q&AQ: Shouldn't this project be separated into few ones (e.g. pipelines, commands, script bootstrap)?A: Probably yes. And maybe it will be split in the future. But for now it's more convenient to manage single project.CreditsThe logo was created using"Shrimp" iconcreated by"elmars"and published
underCC BY-SA 3.0 license.