Package: textomatic
textomaticScratchpad for tabular data transformationsGiven input of some tabular data (AKAINPUT),
and a command (AKACOMMAND), transform that input
into some other output (AKAOUTPUT).InstallationpipinstalltextomaticAt the moment, Python 3.8 or greater is required.RunningStart it by running$tmThe above will starttextomaticwith a blank slate.You can load a file by passing it as the first argument:$tm<PATH_TO_FILE>You can also pipe content from stdin:$ls|tmTo see what arguments/options are available, run:$ tm --helpWhentextomaticis running, typeF1to see available keyboard shortcuts.To exittextomaticuseCTRL-Cto exit without any output orCTRL-Oto print
currentOUTPUTinto standard out.UseCTRL-Pto put the currentOUTPUTin the system clipboard.ExamplesParsingps auxoutputPipeing content from shell usingps aux | tmand transforming it into json lines where
each line containes theUSERandPIDcolumns with lower cased keys.PIDis casted into an integer.COMMAND:hsaysINPUTincludes a headeri:shsays theINPUTshould be parsed like shell outputs:{user:USER,pid:PID}specifies the output structureo:jlspecifies the output format to be json linest:PID:ispecifies thePIDcolumn shouldd ge parsed as integerINPUT                                  │OUTPUT                                  
USER               PID  %CPU %MEM      │{"user": "dan", "pid": 63507}           
dan              63507   6.3  0.4  6178│{"user": "_windowserver", "pid": 250}   
_windowserver      250   4.4  0.3 12494│{"user": "dan", "pid": 54987}           
dan              54987   3.8 12.8 16080│{"user": "_hidd", "pid": 184}           
_hidd              184   2.0  0.0  5608│                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
vi-insert|COMMAND|live|in:sh|out:jl|delim:auto|header:true                      
> h;i:sh;s:{user:USER,pid:PID};o:jl;t:PID:iPretty printing csvCOMMAND:hsaysINPUTincludes a headero:tspecifies the output format to be a pretty printed tableINPUT                                  │OUTPUT                                  
Name,Age,City                          │╒═══════════╤═══════╤════════╕          
James Joe,34,NYC                       ││ Name      │   Age │ City   │          
John Doe,25,London                     │╞═══════════╪═══════╪════════╡          
                                       ││ James Joe │    34 │ NYC    │          
                                       │├───────────┼───────┼────────┤          
                                       ││ John Doe  │    25 │ London │          
                                       │╘═══════════╧═══════╧════════╛          
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
vi-insert|COMMAND|live|in:c|out:t|delim:auto|header:true                        
> h;o:tCorrecting bad inputCOMMAND:hsaysINPUTincludes a headert:Age:i/76/provides a default value when theAgecolumn cannot be parsed as integerINPUT                                  │OUTPUT                                  
Name,Age,City                          │[['James Joe', 34, 'NYC'],              
James Joe,34,NYC                       │ ['John Doe', 76, 'London'],            
John Doe,Not a number,London           │ ['Jane Row', 24, 'Tel Aviv']]          
Jane Row,24,Tel Aviv                   │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
vi-insert|COMMAND|live|in:c|out:l|delim:auto|header:true                        
> h;t:Age:i/76/InteractivejqCOMMAND:rputs evaluation into raw mode where input is not assumed to be raw basedo:jquse thejqoutput and pass arguments to it wrapped with backticks.INPUT                                  │OUTPUT                                  
{"one": 1, "two": 2, "three": 3}       │{"o":1,"t1":2,"t2":3}                   
{"one": 1.0, "two": 2.0, "three": 3.0} │{"o":1,"t1":2,"t2":3}                   
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
                                       │                                        
vi-insert|COMMAND|live|in:c|out:jq|delim:auto|header:false|raw:true             
> r;o:jq`{o: .one, t1: .two, t2: .three}`Usagetextomaticis split into 3 parts:INPUT: The input data that is to be transformedOUTPUT: The result of applying theCOMMANDonINPUTCOMMAND: The transformation logic using a succinct expression language described belowUse theTabkey to move between them.INPUTandCOMMANDare edited usingvimbindings.COMMANDExpression LanguageCOMMANDis composed of different parts separated by;, e.g.> h;i:c;o:jl;s:[1,2]The above will be explained in detail later on but for now we can see it has 4 parts:h: specifies that the inputcsvhas a headeri:cspecifies that the input is in fact acsvo:jlspecified that the output should be in jsonlines formats:[1,2]specifies that the output should only include the first and
second columns from the input, in that orderTo use a different expression separator, start the command with:<SEP>, e.g.> :|h|i:cExpressionsThehexpression (header)The simplest expression. It is basically a flag denoting whether the input
includes headers. This is relevent for inputs likecsvThedexpression (delimiter)Used by thecsvinput. In most cases, the delimiter can be automatically
deduced. In cases where it cannot, used, e.g. to set a,delimiter:> d:,To specify delimiters that are not easy to input, start the delimiter with\, the remaining part will then be parsed as a python literal
(wrapped in string), for example this will set the delimiter to the unicode
character─(U+2500):> d:\u2500Theiexpression (input)Used to specify the input format. Currently these are the available inputs:c(csv, this is the default. The delimiter musn't be a,as the name may imply)jl(jsonlines)sh(shell, e.g. the output ofps aux)jq(Usingjq)Theoexpression (output)Used to specify the output format. Currently these are the available outputs:l(python literal, this is the default)j(json)jl(jsonlines)c(csv)t(pretty printed table)h(table html)jq(Usingjq)Thetexpression (types)For inputs with no clear types (e.g. csv/shell), all columns are initially assumed to
be strings. To modify types of different columns uset.The types are:s(string, the default)f(float)i(integer)b(boolean, case insensitivetrue/yes/y/on/1will be parsed astrue)j(json, will JSON parse the column)l(literal, will parse a python literal usingast.literal_eval)Using positional syntax:> t:i,i,bIn the above:the first 2 columns will be parsed as integersthe third column will be parsed as booleanthe rest will be stringsUsing indexed syntax:> t:1:i,3:fIn the above:the first column will be parsed as integerthe third column will be parsed as floatthe rest will be stringsUsing named syntax:> t:col1:b,col4:iIn the above, assuming the input contains headers:the column namedcol1will be parsed as booleanthe column namedcol4will be parsed as integerthe rest will be stringsOptional types:If a certain value may be invalid, you can use?to
mark it as optional, in this case, its value will be
converted tonullwhen it is invalid:> t:col1:i?,col4:i?Defaults:If you want the specify a value different thannullfor invalid entries
use the following syntax:> t:col1:i/0/,col4:f/0.0/The value between the/'s will be evaluated as a python literal.Thesexpression (structure)Thesexpression is used to specify the structure of theOUTPUT.
Some of the options are catered to the python output but they will fallback
to a reasonable alternative for other outputs.Simple transformations:s:[]- Each row will be a list inOUTPUTs:()- Each row will be a tuple inOUTPUTs:{}- Each row will be an object inOUTPUT(assumes input has headers)s:d()- Same ass:{}s:s()- Each row will be as set inOUTPUTComplex transformations:s:[1,2,col6]- Each row will contain the first and second columns and a column
namedcol6. Note that it is wrapped with[]. This only
means the output row will be a list. You can just as well
wrap it with{}to get objects e.g.s:{1,2,col6}.
The different wrapping options are desribed in "Simple Transformations"
of the previous section.s:{first:1,second:2}- Each row will contain the first and second columns with
The specified new headers (firstandsecond)s:[-2,-1]- Each row will contain the two last columns fromINPUTs:{1,second:2}- Column definitions can be mixed.Nested transformations:s:[{1,2},{3,4}]- Each row will contain two objects, the first object will
contain the first and second columns, the second object
will contain the third and fourth columns.Nesting can be as complex as you wish and rules from previous sections
can be applied freely. As an overly complex example:> s:{k1:1,k2:{2,three},k3:[{four,5},d(-4,s7:seven)],eight,k4:(one, two, (four, five)),s:s(1,1,1)}Fetching values of nested data:s:[some_obj.key1.key2]- Each row will contain a single nested value from
thesome_objcolumn.s:[some_obj.key1?.key2]- Same as above, but don't fail ifkey1doesn't exist,
instead, replace it withnull.s:[some_obj.key1?.key2/100/]- Same as above, but use100instead ofnull.s:[some_obj.key1.key2/100/]- Default value without optional?is equivalent tos:[some_obj?.key1?.key2?/100/]A caveat of using//for default values is that the default value cannot
include/. This is mostly due to a very simple parser that is currently
implemented for the expression language.