Package: collatelogs
collatelogsA simple log collator. That is, given one or more log file paths, collate them together such that they are in chronological order.MotivationIn complicated projects, it is common to have an assortment of logs being generated simultaneously. In a perfect world, these would all have the same log format, and thus could be easily collated/reconciled by appending them all and then sorting them.However, the world is not perfect, and you may be dealing with a variety of log formats, perhaps (and most importantly) even differences in timestamp format.collatelogshandles the consumption of logs based on the given regular expressions, then collates them and outputs them in a common format (also configurable).Installation$ pip install collatelogsUsageThis script probably won't work out of the box, unless your log files happen to have a prefix structure that matches one of the regular expressions in the example configuration. So, you'll probably see something like this:$ collatelogs PATH_WITH_LOGS/*.log
<snip>
ValueError: Line 'some line from the logs' did not match against any of the given regexes!So, the first thing you will need to do is define information on the expected log prefixes.The config fileFor ease of use, it is recommended that you create a config file to avoid having to pass a bunch of arguments every time you run the script. An example is included in the repo, atcollatelogs/example_config.yaml. This will also be installed alongside the package.The recommended place for a permanent config file is~/.cl_config.yaml. Other paths searched are listed in the help.It is probably easiest to$ cp example_config.yaml ~/.cl_config.yamlbefore beginning to make your changes.timestamp_output_format(optional)The format of the timestamp output. This will have no affect unless--parse-timestampsis passed as an argument at runtime, orparse_timestampsis set toTruein the config file.line_output_format(required)A PEP-3101 compliant format string that defines the output format for each line. The keywords here must be a subset of the regex capturing groupsplusany keywords contributed by the meta handlers (see below). Put another way: if you try to include keywords here that aren't being captured in each regular expression, you are going to get an error.log_parsing_info(required)then begin replacing the example entries inlog_parsing_infowith your own entries (leaving the examples will only slow down execution if they are never going to match anything).Eachdictin thelog_parsing_infolisthas four possible parts:regex(required): The regular expression used to parse log linestimestamp_input_format(optional): The format of the timestamp for lines captured byregex. If this is not given,dateutil.parsewill be used to generically consume the timestamp, but this will be ~5x slower!timestamp_input_timezone(optional): The timezone that the log timestamps were output in. If this is not given, it defaults to the local timezone of your computertimestamp_output_timezone(optional): The timezone that the output log timestamps will be in. If this is not given, it defaults to the local timezone of your computerNote that this must utilize capturing groups such that every keyword in theline_output_formatformat string is represented.Creating Regex Prefixes and Output Format StringsThis isn't too hard, provided you have experience with regular expressions.Examine the log and identify the prefixBreak it down into useful parts (timestamp, message, etc. -- see config.yaml for examples)Formulate a regular expression that captures these parts into aptly-named groupsFormulate a format string that outputs these groups into a sensible formatLet's see an example. Let's say our log lines are of this format:2018-04-20 11:30:01 circus[1624] [INFO] circusd-stats stoppedEven without examining the logger that created this, it can be broken down into four parts:{
    'timestamp': '2018-04-20 11:30:01'
    'module': 'circus[1624]', # Not technically a module, but close enough
    'level': 'INFO'
    'message': 'circusd-stats stopped'
}The above is what the regex should output via itsgroupdict. So, what regular expression will accomplish this? Well, this is actually pulled fromconfig.yaml:(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<module>\S+) (?P<level>\S+) (?P<message>.*)Because this prefix format is delimited by spaces (and I'm not exactly a regex expert), the regex is pretty verbose. I find that a tool such as regex 101 helps a lot:https://regex101.com/r/bTzhf4/1Now that the regex has been constructed, add it to the config file as per the above section.meta_handlersThese handle metadata associated with each log file, making it available toline_output_formatas keyword arguments.There are currently two available handlers:user: The owner of the log filefilename: The filename (base name) of the log fileYou will see that these are both present in the exampleline_output_formatThere's currently no clean way of adding your own, but you can easily hack them intometahandlers.pyby defining them, then mapping a name to them inall_meta_handlers.