Package: python-pype
UsageBasicsAt the command prompt, usepypeto act on each item in the file with python commands:$ pype map x.upper() <<<'abc'
ABCChain python functions together with!:$ pype map 'x.upper() ! len(x)' <<<hello
5or by adding another command likemap <pipeline>$ pype map 'x.upper()' map 'len(x)' <<<hello
5Usexas a placeholder for the input at each stage:$ pype map ' x.split()[0] ! x.upper() + "!"' <<<'Hello world'
HELLO!

$ pype map 'x.split()[0] ! x.upper() + "!" ! x.replace("H", "J")' <<<'Hello world'
JELLO!Automatically import modules you need:$ pype stack 'itertools.repeat(x, 2) ! "".join' <<<hello,world!
hello,world!
hello,world!CommandsmapUsemapto act on each input item.$ pype map 'x * 2' <<<'a\nbb\n'
aa
bbbbfilterUsefilterto evaluate a condition on each line of input and exclude false values.$  pype filter 'len(x) > 1' <<<'a\nbb\nccc\n'
bb
cccapplyUseapplyto act on the sequence of items.$   pype apply 'len(x)' <<<'a\nbb\n'
2stackUsestackto treat the input as a single string, including newlines.$  pype stack 'len(x)' <<<'a\nbb\n'
5Useevalto evaluate a python expression without any input.$ pype eval 1+1
2reduceUsereduceto evaluate a function of two arguments successively over a sequence, likefunctools.reduce.For example, to multiply all the values together, first convert each value tointwithmap, then usereduceto successively multiply each item with the product.$ pype map int reduce operator.mul <<EOF
1
2
3
4
EOF

24AutocallYou don’t need to explicitly call the function withf(x); just usef. For example, instead of$ pype map 'len(x)' <<<'a\nbb'
5try$ pype map len <<<'a\nbb'
5AsyncMaking sequential requests is slow. These requests take 20 seconds to complete.$ time pype map 'requests.get ! x.text ! len' apply max <<EOF
http://httpbin.org/delay/5
http://httpbin.org/delay/1
http://httpbin.org/delay/4
http://httpbin.org/delay/3
http://httpbin.org/delay/4
EOF

302

0.61s user
0.06s system
19.612 totalConcurrent requests can go much faster. The same requests now take only 6 seconds. Useamap, orafilter, orreducewithawait some_async_functionto get concurrency out of the box.$ time pype amap 'await asks.get ! x.text ! len' apply max <<EOF
http://httpbin.org/delay/5
http://httpbin.org/delay/1
http://httpbin.org/delay/4
http://httpbin.org/delay/3
http://httpbin.org/delay/4
EOF

297

0.57s user
0.08s system
5.897 totalAsync streamingamapandafiltervalues are handled in streaming fashion, while retaining the order of the input items in the output. The order of function calls is not constrained – if you need the function to becalledwith items in a specific order, use the synchronous version.Making concurrent requests, each response is printed one at a time, as soon as (1) it is ready and (2) all of the preceding requests have already been handled.For example, the3 secondsitem is ready before the preceding4 secondsitem, but it is held until the4 secondsis ready because4 secondswas started first, so the ordering of the input items is maintained in the output.$ time pype --exec-before 'import datetime; now=datetime.datetime.utcnow; START_TIME=now(); print("Elapsed time | Response size")' map 'await asks.get !  f"{(now() - START_TIME).seconds} seconds    | {len(x.content)} bytes"'  <<EOF
http://httpbin.org/delay/1
http://httpbin.org/delay/2
http://httpbin.org/delay/4
http://httpbin.org/delay/3
EOF
Elapsed time | Response size
1 seconds    | 297 bytes
2 seconds    | 297 bytes
4 seconds    | 297 bytes
3 seconds    | 297 bytesConfigurationAdd code to automatically execute, into your config file.For example:# ~/.config/pype/config.toml

exec_before = """

from itertools import *
from collections import Counter

"""Then you can directly use the imported objects without referencing the module.$ pype map 'Counter ! json.dumps' <<<'hello\nworld\n'
{"h": 1, "e": 1, "l": 2, "o": 1}
{"w": 1, "o": 1, "r": 1, "l": 1, "d": 1}You can set any of thepypeoptions in your config. For example, to set a different default value for the concurrency maximumpype--max-concurrent, addmax_concurrentto your config file (note the underscore):# ~/.config/pype/config.toml

max_concurrent = 10then just usepypeas normal.AliasesDefine new commands in your config file which provide aliases to other commands. For example, this config adds ajsonlcommand for reading jsonlines streams into Python objects, by calling calling out to themaptraversal.[[alias]]

name = "jsonl"
short_help = "Load jsonlines into python objects."

[[alias.stage]]

command = "map"
options = []
arguments = [ "json.loads ! types.SimpleNameSpace(**x)" ]Now we can use it like a regular command:$ pype jsonl  <<< $'{"a":1, "b":2}\n{"a": 5, "b":9}'
X(a=1, b=2)
X(a=5, b=9)The new commandjsonlcan be used in pipelines as well. To get the maximum value in a sequence of jsonlines objects.$ pype jsonl map 'x.a' apply max <<< $'{"a":1, "b":2}\n{"a": 5, "b":9}'
5PluginsAdd new commands likemapandreduceby installing pype plugins. You can try them out without installing by adding them to any.pyfile in your~/.config/pype/modules/.InstallationGet it with pip:pip install python-pypeCaveatspypeassumestrusted command argumentsanduntrusted input stream data. It usesevalon your commands, not on the input stream data. If you useexec,eval,subprocess, or similar commands, you can execute arbitrary code from the input stream, like in regular python.StatusCheck theissues pagefor open tickets.This package is experimental and is subject to change without notice.Related workhttps://github.com/Russell91/pythonpyhttp://gfxmonk.net/dist/doc/piep/https://spy.readthedocs.io/en/latest/intro.htmlhttps://github.com/ksamuel/Pypedhttps://github.com/ircflagship2/pype