Package: awking
awkingMake it easier to use Python as an AWK replacement.Basic usageExtracting groups of linesfromawkingimportRangeGrouperlines='''text 1text 2group start 1text 3group end 1text 4group start 2text 5group end 2text 6'''.splitlines()forgroupinRangeGrouper('start','end',lines):print(list(group))This will output:['group start 1', 'text 3', 'group end 1']
['group start 2', 'text 5', 'group end 2']Extracting fixed-width fieldsfromawkingimportrecordsps_aux='''USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMANDroot         1  0.0  0.0  51120  2796 ?        Ss   Dec22   0:09 /usr/lib/systemd/systemd --system --deserialize 22root         2  0.0  0.0      0     0 ?        S    Dec22   0:00 [kthreadd]root         3  0.0  0.0      0     0 ?        S    Dec22   0:04 [ksoftirqd/0]root         5  0.0  0.0      0     0 ?        S<   Dec22   0:00 [kworker/0:0H]root         7  0.0  0.0      0     0 ?        S    Dec22   0:15 [migration/0]root         8  0.0  0.0      0     0 ?        S    Dec22   0:00 [rcu_bh]root         9  0.0  0.0      0     0 ?        S    Dec22   2:47 [rcu_sched]saml      3015  0.0  0.0 117756   596 pts/2    Ss   Dec22   0:00 bashsaml      3093  0.9  4.1 1539436 330796 ?      Sl   Dec22  70:16 /usr/lib64/thunderbird/thunderbirdsaml      3873  0.0  0.1 1482432 8628 ?        Sl   Dec22   0:02 gvim -froot      5675  0.0  0.0 124096   412 ?        Ss   Dec22   0:02 /usr/sbin/crond -nroot      5777  0.0  0.0  51132  1068 ?        Ss   Dec22   0:08 /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplicasaml      5987  0.7  1.5 1237740 119876 ?      Sl   Dec26  14:05 /opt/google/chrome/chrome --type=renderer --lang=en-root      6115  0.0  0.0      0     0 ?        S    Dec27   0:06 [kworker/0:2]'''foruser,_,commandinrecords(ps_aux.splitlines(),widths=[7,58,...]):print(user,command)This will output:USER    COMMAND
root    /usr/lib/systemd/systemd --system --deserialize 22
root    [kthreadd]
root    [ksoftirqd/0]
root    [kworker/0:0H]
root    [migration/0]
root    [rcu_bh]
root    [rcu_sched]
saml    bash
saml    /usr/lib64/thunderbird/thunderbird
saml    gvim -f
root    /usr/sbin/crond -n
root    /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplica
saml    /opt/google/chrome/chrome --type=renderer --lang=en-
root    [kworker/0:2]The problemDid you ever have to scan a log file for XMLs? How hard was it for you to
extract a set of multi-line XMLs into separate files?You can usere.findallorre.finditerbut you need to read the entire log
file into a string first. You can also use an AWK script like this one:#!/usr/bin/awk -f/^Payload: <([-_a-zA-Z0-9]+:)?Request/{ofname="request_"(++index)".xml"sub(/^Payload: /,"")}/<([-_a-zA-Z0-9]+:)?Request/,/<\/([-_a-zA-Z0-9]+:)?Request/{print>ofname}/<\/([-_a-zA-Z0-9]+:)?Request/{if(ofname){close(ofname)ofname=""}}This works, and quite well. (Despite this being a Python module I encourage you
to learn AWK if you don't already know it.)But what if you want to build this kind of stuff into your Python application?
What if your input is not lines in a file but a different type of objects?Python equivalent usingawkingTheRangeGrouperclass groups elements from the input iterable based on
predicates for the start and end element. This is a bit like Perl's range
operator or AWK's range pattern, except that your ranges get grouped intoSTART..ENDiterables.An equivalent of the above AWK script might look like this:fromawkingimportRangeGrouperimportreimportsysg=RangeGrouper(r'^Payload: <([-_a-zA-Z0-9]+:)?Request',r'</([-_a-zA-Z0-9]+:)?Request',sys.stdin)forindex,requestinenumerate(g,1):withopen(f'request_{index}.xml','w')asf:forlineinrequest:line=re.sub(r'^Payload: ','',line)# Not optimalprint(line,file=f,end='')The predicates may be regular expressions, either asre.compile()objects or
strings; or they may be any callables that accept a single argument and return
a true/false value.CaveatsThe grouping algorithm reads the input iterable lazily. You can still run out
of memory if you keep references to previous groups without consuming them.