Package: xontrib-pipeliner
Easily process the lines using pipes inxonsh shell. Multicore processing supported.If you like the idea of pipeliner click ⭐ on the repo andtweet.Installxpipinstall-Uxontrib-pipelinerecho'xontrib load pipeliner'>>~/.xonshrc# Reload xonshUsageLet your pipe lines flow thru the Python code:<cmd>|...|pl"<lambda expression>"|<cmd>|...There are two variables available in lambda expression:linefrom pipe.numof the line starts with 0.Experimental featurespplis to run multicorepl. It tested mostly on Linux. See "Known issues" section.plxis the shorter way to execute the commands with pipe lines i.e.ls /home | plx 'du -sh /home/{line}'.ExamplesPython way to line modificationls-1/|pl"line + ' is here'"|head-n3bin is here
boot is here
dev is hereLine numberls-1/|head-n4|pl"f'{num} {line}'"0 bin
1 boot
2 cdrom
3 devIgnore line$ls-1/|head-n4|pl"f'{num} {line}' if num%2 == 0 else None"0 bin
2 cdromSplittingcat/etc/passwd|head-n3|pl"line.split(':')[6]"/bin/bash
/usr/sbin/nologin
/usr/sbin/nologinImportsimportre
cat/etc/passwd|head-n3|pl"re.sub('/bin/bash', '/usr/bin/xonsh', line)"root:x:0:0:root:/root:/usr/bin/xonsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologinArrayscat/etc/passwd|head-n3|pl"line.split(':')"|grepnologin|pl"':'.join(eval(line)[::-1])"/usr/sbin/nologin:/usr/sbin:daemon:1:1:x:daemon
/usr/sbin/nologin:/bin:bin:2:2:x:binPython headpl"'\\n'.join(list('ABCDEFG'))"|pl"line + ('!' if num%2 else '?')"|grep'!'B!
D!
F!Variables and operations chainingExpression is a lambda function so using variables and operations chaining since Python 3.8+ are available by trick with the walrus operator and the list:ls-1/|head-n3|pl"[s:='b', line.replace(s, s.upper()+')')][-1]"B)in
B)oot
devExecute command with the linels/|head-n3|pl"execx('du -sh /'+line) or 'Done command with /'+line"0       /bin
Done command with /bin
840M    /boot
Done command with /boot
4,0K    /cdrom
Done command with /cdromNote! If you do the operations with files (i.e.pl "execx(f'mv {line} prefix-{line}')") you could catchTypeError: an integer is requirederror that relates to wrong access rights to files. Fix it withchmodandchownbefore pipelining.PresetsThere are default presets:echo "  1" | pl strip
# 1

echo "1,2,3" | pl split ,
['1', '2', '3']

echo "a,b,c" | pl split , | pl list 0
# aYou can set your own presets:$XONTRIB_PIPELINER_PRESETS = {
    "upper": "line.upper()",
    "repeat": lambda args: f"line*int({repr(args[0])})"
}

echo 'hello' | pl upper
# HELLO

echo 'hey \nhi ' | pl repeat 3
# hey hey hey
# hi hi hiWrap pipeliner to get your own magicaliases['my_lovely_pl']=lambdaa,i,o:aliases['pl'](["'My lovely ' + "+a[0]],i,o)aliases['my_parallel_ppl']=lambdaa,i,o:aliases['ppl'](["'My parallel ' + "+a[0]],i,o)ls/|head-n3|my_lovely_pl"line + '!'"# My lovely bin!# My lovely boot!# My lovely cdrom!ls/|head-n3|my_parallel_ppl"line + '!'"# My parallel boot!# My parallel cdrom!# My parallel bin!Add your most useful solutions to xontrib-pipeliner. PRs are welcome!ExperimentalSyntax highlighting using xonsh promptIf you're using xonsh prompt and want to use pipeliner with syntax highlighting instead of string there is experimental
feature that catchpl @(<python>)calls and uses the expression from the xonsh python substitution as pipeliner argument.
Example:echoecho|pl@(line+'!')# In the xonsh prompt it's equals to:echoecho|pl"line + '!'"Syntax highlighting using xonsh macrosTo avoid writing Python inside the string and get the syntax highlighting there is a tricky way with usingxonsh macro:defpy(code):returncodeecho123|pl@(py!(line+'2'))Multicore pipeliningBy default pipeliner works using one CPU core. To use them all in parallel trypplcommand:head/etc/passwd|ppl"str(num) + ' ' + line.split(':')[0]"1 daemon
0 root
2 bin
4 sync
5 games
8 mail
9 news
6 man
7 lp
3 sysNote! The order of result lines is unpredictable because lines will be processed in parallel.
Thenumvariable contains the real line number.Pipeliner execThere areplxandpplxcommands to runexecx(f"{plx_command}")most shorter way.For example when you want to rename files you can do it Pythonic way:mkdir-p/tmp/plx-test&&cd/tmp/plx-test
touch111222333&&ls# 111 222 333ls|plx"mv {line} prefix-{line}"# mv 111 prefix-111# mv 222 prefix-222# mv 333 prefix-333ls# prefix-111 prefix-222 prefix-333Echo example:ls|plx'echo {line} # {num}'# echo prefix-111 # 0# prefix-111# echo prefix-222 # 1# prefix-222# echo prefix-333 # 2# prefix-333Pipeliner in xsh scriptsBy default xsh scripts haven't rc-file with xontribs loading. To add pipeliner to your script just doxontrib load pipelinerbefore usage.Known issues in experimental functionsplx: "Bad file descriptor" on huge amount of lineshttps://github.com/xonsh/xonsh/issues/4224ppl:On MacOS global variables are not accessible from child processesin multicore pipeliningOn Mac you can't access to the xonsh context (global variables and functions) in the expression. PR is welcome!ppl: On MacOS multicore pipelining freezes on endWorkaround is to addcatat the end:echo 1 | ppl 'line' | cat. PR is welcome!FuturePipeliner should be a part of xonsh and has shortcut and syntax highlighting. For example:echo'Pipeliner should be '|pl@{line+'a part of xonsh!'}# orecho'Pipeliner should be '|~(line+'a part of xonsh!')Pipeliner should be a part of xonsh!If you want to support this in xonsh add your Like and support message toPython code substitution in subproc mode.LinksThis package is the part ofrc-awesome- awesome snippets of code for xonshrc in xonsh shell.This package is the part ofergopack- the pack of ergonomic xontribs.This package was created withxontrib cookiecutter template.