Package: rextract
rextract is a powerful commandline tool to extract and manipulate strings using regular exressions.Think of it somewhat similar to “sed” or “grep”, except it gives you complete control to rearrange/omit/duplicatethe output, so far as regular expressions support.It supports python extended regular expressions (compatible with perl-style).UsageUsage: rextract (Options) [regex pattern] ([output format])Reads from stdin and applies provided regex pattern line-by-line,optionally outputting in a format specified by “output format.”Options:========--debug     Enable debug mode--version   Print version and exitUsage:=====Some examples of usage could be:Extracting one or more groups from the inputOmitting one or more groups from the inputRearranging the inputUsing some textual input (csv?) into a series of commands to executeCreating reportsMany others!Regex Pattern Format/Tips:==========================rextract supports extended regular expression syntax (python/perl-style), specificallythat provided by the standard python “re” module.Some examples can be found athttps://docs.python.org/3/howto/regex.htmlEach pattern contained within parenthesis counts as a group.Each group left-to-right is assigned a numerical value, starting with 1.You can assign a name to a group like so:(?P<my_group_name>[a-zA-Z][a-zA-Z0-9]+)^ This creates a group which you can reference as:${my_group_name}A group must start with a letter a-z or underscore ( ‘_’ ),and must only contain letters a-z (any case), numbers 0-9, or underscore.Name a group like: (?P<name>.*)The regular expression will match at any point within the line, unless‘^’ (start of line) or ‘$’ (end of line) are present, in which case itwill have to match with those restrictions.Thus, there is no need to prefix with ‘.*’ to be able to match somewherein the center.If a line fails to match, it will not be provided to for use as output,it will just be silently skipped. You cannot logically format outputfrom an unknown input.You can specify a “NOT” within your pattern by starting a group with a caret ( ‘^’ )For example, to match that a line does not start with a semicolon, you can use apattern line:^([^;].*)Output Format=============The “Output Format” argument can be though of exactly like a bash string -characters all have their literal value except those preceded with anun-escaped dollar ( $ ) sign. These represent groups, either numeric orby-name. See the section above for more info on creating groups.If the “Output Format” argument is omitted, the entire matched region willbe printed, line-by-line. This is equivalent to providing an“Output Format” of ‘${0}’.Numeric Groups--------------Use ${1} or $1 for the first group, and ${3} for the third.Numerical groups are assigned left-to-right as they appear in the regex,even if the group is in a conditional, and despite nesting.For example:“((yellow|white) cheddar)|((whole|part-skim) mozerella)”“yellow|white cheddar” is the first group,“yellow|white” is the second group“whole|part-skim mozerella” is the third group“whole|part-skim” is the fourth group.If a group is not matched, it is assigned the value of an empty string.Use the special group $0 or ${0]} to print the entire matched section.Named Groups------------If you defined any groups to have a name (see section above on this topic),i.e. (?P<my_group>…) , then you can reference that group in the outputas $my_group or ${my_group}.NOTE: Make sure to single-quote the “Output Format” argument,or escape dollar [$] signs (by using $).rextract version 1.0.0 by Tim SavannahExamplespasswd fileExample, extract all the usernames and UIDs from /etc/passwd of folks who use “/bin/bash” as their shell, and reformat it.cat /etc/passwd | rextract ‘^(?P<username>[^:]+)[:][^:]*[:](?P<uid>[d]+).*/bin/bash$’ ‘${username} [${uid}]’Example output:joe55 [1000]tjoseph [1009]james [1011]Explained Expression:Match starts at first character of line. First group is named “username”, and contains at least 1 character and all characters that are not ‘:’.Then comes a colon ‘:’Then comes 0 or more characters which are not colon ‘:’Then comes a colonSecond group is named “uid”, and contains one or more digits.Then, match 0 or more charactersThen, match the string “/bin/bash” at the end of the line ( represented by ‘$’ )Our output is the username, followed by square brackets enclosing the uid.LogsExample, extract a sorted list of all pacman (archlinux) packages updated/installed on a specific date, and versionsrextract ‘^([2016-11-02).*(upgraded|installed) (?P<what>.*)$’ ‘$what’ < /var/log/pacman.log  | sortSample Output:asciidoc (8.6.9-3)accerciser (3.14.0-4 -> 3.22.0-1)accountsservice (0.6.42-1 -> 0.6.43-1)adwaita-icon-theme (3.20-2 -> 3.22.0-1)aisleriot (3.20.2-1 -> 3.22.0+5+gb3024a2-1)akonadi-qt4 (1.13.0-10 -> 1.13.0-11)Explanation:Here’s a few sample lines from pacman.log:[2016-11-02 16:45] [ALPM] installed asciidoc (8.6.9-3)[2016-11-02 22:42] [ALPM] upgraded tali (3.20.0-2 -> 3.22.0-1)[2016-11-02 22:42] [ALPM] upgraded totem (3.20.1-1 -> 3.22.0+5+ge0bf46e-1)[2016-11-02 22:42] [ALPM] warning: directory permissions differ on /etc/unrealircd/filesystem: 700  package: 755[2016-11-02 22:42] [ALPM] warning: directory permissions differ on /etc/unrealircd/aliases/filesystem: 700  package: 755[2016-11-02 22:42] [ALPM] upgraded unrealircd (4.0.6-1 -> 4.0.7-1)As you can see, there’s a lot of information here, some relevant, some not.Basically, this is an example that COULD be done with grep and sed, butis much more easily accomplished with rextract, and we may actually want to modifythe form of the output (see “More Advanced” below)So the filter expression says:Filter that line starts with the date of interestFilter that 0 or more characters occur between that date and either the word “upgraded” or “installed”Extract everything after the word “upgraded” or “installed” (excluding the space after), and place into a group called “what”And our output expression just contains the ‘what’ portion.Strip CommentsTo strip comments from a file, where a “comment” is defined by the character ‘;’ and everything following on a given line:cat myFile.ini | rextract ‘(?P<valid>[^;]*)’ > myFile.ini.strippedMore Advanced:Okay, so now let’s get more advanced. We want to produce a report that lists what software installations happened today,what the final version is, and whether it is new software (installation) or old software (upgrade). And we use the same log file.This is accomplished with the following:rextract ‘^([2016-11-02).*(?P<action>[ui])(pgraded|nstalled) (?P<name>[^ ]+)[ ][(](.+[-][>][ ]){0,1}(?P<final_version>.+)[)]$’ ‘$name = $final_version [$action]’ < /var/log/pacman.log  | sortSample output:asciidoc = 8.6.9-3 [i]vim = 8.0.0055-1 [u]vim-runtime = 8.0.0055-1 [u]yelp = 3.22.0+1+gfabd8eb-1 [u]yelp-tools = 3.18.0+1+g193c2bd-1 [u]yelp-xsl = 3.20.1-2 [u]Here we show the package name, the final version, and a marker if it was an install or an upgrade ( [i] == install, [u] == upgrade ).Filter Explanation:Start with today’s dateThis time, split the first letter of “upgraded” and “installed” into its own group, “action”.Ensure that following the “action” letter is the remainder of the word. Note, in theory this could match ‘ipgraded’ or ‘unstalled’, but with this given data, it won’t. However, in other cases, it might. For those cases, we can match with an “or” condition, and use two groups (you cannot repeat group names, even in an “or” condition):./rextract ‘^([2016-11-02).*[ ](((?P<a1>[u])pgraded)|((?P<a2>[i])nstalled))[ ](?P<name>[^ ]+)[ ][(](.+[-][>][ ]){0,1}(?P<final_version>.+)[)]$’ ‘$name - $final_version [${a1}${a2}]’ < /var/log/pacman.logSo here, note that we no longer can match “ipgraded” or “unstalled”. When a group is present in the pattern string, but does not appear in a matched group, its value is assigned as an empty string. Thus, where we used “$action” in the simpler form, we now use “${a1}${a2}”, as only one will hold a value (‘u’ or ‘i’), and the other will be blank.Anyway, I digress. Be sensible, unless lives are on the line, it’s OK to take shortcuts (like “(?P<action>[ui])(pgraded|nstalled)”) which are not “technically” 100% correct, but are 100% accurate with real-world data.After the word “upgraded/installed” and the following space, take all non-space characters (”[^ ]”) and assign to group “name”. This will be the package name.Next follows a space, and an open parenthesis.Then, is a conditional group. We match that there are at least one character followed by an arrow (“->”) followed by a space, 0 or 1 times. This may be confusing to some, basically, we are making a group of “{anything}-> “ and saying you may see that group 0 times, or 1 time. This covers the difference in representations of the “installed” and “upgraded” packages. ‘Upgrades’ will have matched that group 1 time (ok), ‘installs’ would have matched that group 0 times (ok).Now that we’ve discarded the first part of the parenthesis in the upgrade case, and remain just inside the paren in the install place, what is left between the cursor and the close-parenthesis is the final version. So we match everything from cursor to the final version.