Package: tabulog
An Example in PythonLet's again say you have the example logs in the fileaccesslog.txt.10.0.0.8 - - [2019-01-01:10:58:12 -0500] "https://mysite.com/index.html"
173.28.102.33 - - [2019-01-01:10:58:25 -0500] "https://mysite.com/login"We first define the template.template='{{ ip ip_address }} - - [{{ date date_time }}] "{{ url URL }}"'We then need to define our classes.ipandurlare builtins with the package, but dates come in a variety of formats so we must explicitly define ours here. Note you can see all builtins usingdefault_classes()importtabulog,datetimedate_parser=tabulog.Parser('[0-9]{4}\\-[0-9]{2}\\-[0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{2}[ ][\\-\\+][0-9]{4}',lambdax:datetime.datetime.strptime(x,'%Y-%m-%d:%H:%M:%S %z'),name='date')date_parserParser('[0-9]{4}\-[0-9]{2}\-[0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{2}[ ][\-\+][0-9]{4}', <function <lambda> at 0x7f7a07574e18>, 'date')forkeyin['ip','url']:print(tabulog.default_classes()[key])Parser('[0-9]{1,3}(\.[0-9]{1,3}){3}', <function <lambda> at 0x7f7a06c896a8>, 'ip')
Parser('(-|(?:http(s)?:\/\/)?[\w.-]+(?:\.[\w\.-]+)+[\w\-\._~:/?#[\]@!\$&\'\(\)\*\+,;=.]+)', <function <lambda> at 0x7f7a06c896a8>, 'url')Bothipandurlrequire no formatting, so they have the identity function, (lambda x:xin python), as their formatter.To get our final output in tabular format, we first combine everything into aTemplateobject.# We only need to pass our custom date parser class, the defaults will be included.T=tabulog.Template(template_string=template,classes=[date_parser])TTemplate("{{ ip ip_address }} - - [{{ date date_time }}] \"{{ url URL }}\"", classes = ...)Note that we only had to pass our custom classdate. The builtin classesipandurlwere included by default.Finally, we can read in our log file, and call thetabulatefunction in ourTemplateobject. The
final output is a Pandas DataFrame.withopen('accesslog.txt','r')asf:logs=f.read().split('\n')[:-1]T.tabulate(logs)ip_address                 date_time                            URL
0       10.0.0.8 2019-01-01 10:58:12-05:00  https://mysite.com/index.html
1  173.28.102.33 2019-01-01 10:58:25-05:00       https://mysite.com/loginA more elegant and portable way of completing this task would be to define the template and the custom class in the same file, which can be ported to other Tabulog libraries in other languages, leaving only the formatters to be defined in the R script.First, we define thetemplateand theclassesin a yaml file~$ cat accesslog_template.ymltemplate: '{{ ip ip_address }} - - [{{ date date_time }}] "{{ url URL }}"'
classes:
  date: '[0-9]{4}\-[0-9]{2}\-[0-9]{2}:[0-9]{2}:[0-9]{2}:[0-9]{2}[ ][\-\+][0-9]{4}'Next, we define the formatters for each of our classes. Here we only have one, but we still put it in a named list, with the name matching the name of the class in the template file.formatters={'date':lambdax:datetime.datetime.strptime(x,'%Y-%m-%d:%H:%M:%S %z')}Next, we make create our template again, this time using thefileargument.T=tabulog.Template(file='accesslog_template.yml',formatters=formatters)TTemplate("{{ ip ip_address }} - - [{{ date date_time }}] \"{{ url URL }}\"", classes = ...)Again, we get our final output with the same call totabulate.T.tabulate(logs)ip_address                 date_time                            URL
0       10.0.0.8 2019-01-01 10:58:12-05:00  https://mysite.com/index.html
1  173.28.102.33 2019-01-01 10:58:25-05:00       https://mysite.com/login