Package: regex-log-parser
Regex Log ParserRegex Log Parser is a simple and easy to use Python library for log parsing/processing. It allows the user to define a dictionary of regex rules and handler functions which determine how logs should be processed. See the examples below for more information.This was originally developed atMWA Telescopein order to perform data mining of a large amount of log data in order to gain useful insights from it. Following the success of the project we have open sourced and published it in the hope that it may be useful for somebody else.We built this project to extract data from log files and load it into a PostgreSQL database so that it may be queried. However, it has been developed for extensibility and may be used to ingest into other data stores such as MySQL, SQLite, MongoDB, and more.Only Postgres is supported currently, if you would like to see more data stores supported, see the contributing section below.Basic IdeaImagine that you have a directory containing a number of log files. These files may be generated by different systems (i.e. a web server) and different versions of those systems./logsweb1_1.logweb1_2.logweb2_1.logweb2_2.logThe log files contain historical information about activity on the system like so:[2021-11-2504:29:55,015]INFO,192.168.0.1"POST /login HTTP/1.1"200"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36"[2021-11-2504:29:56,542]INFO,192.168.0.1"GET /logout HTTP/1.1"200"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36"[2021-11-2504:30:05,731]INFO,192.168.0.1"GET / HTTP/1.1"200"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36"You want to parse these files to answer questions about the number of logins that have taken place or some other kind of event.Rather than writing code to read each file line by line and doing string splitting, Regex Log Parser allows you to define a set of rules that will be used to parse files within a directory, as well as rules to process lines within matched files.rules={'web1_*':{'[(.*)] INFO, (.*) .*/login.*':'my_handler','.*':'skip'}}In the example rules above, we are defining a dictionary where the key is some regex that will be matched against path/filenames within your directory, and the value is a dictionary which defines how to process the file.By using regex capture groups, we can pull out the information we want from each line (datetime and IP address in the example above) and pass them to some handler function (here called my_handler), which can store the information in a database or do something else with it.Important NoteThis library FORCES you to handle all lines in a file. i.e. there must be at least one rule to match a line within a file. If not, an exception will be raised.This was done deliberately to ensure that users are handling all cases. Once you're confident that you are handling the lines that you care about, add a catch-all rule to skip everything else:rules={'web1_*':{'.*':'skip'}}InstallationPrerequisitesPython >= 3.10Install the packagepipinstallregex_log_parserIf you wish to use the included functionality for uploading data into a postgres database, install the extra dependencies like so:pipinstallregex_log_parser[postgres]UsageCreate a file and import the LogProcessor class. Create an instance of this object then call the run method, passing in a directory containing some logs that you would like to process.Two things are required to setup the processor. A rules dictionary and a handler object.fromregex_log_parserimportLogProcessor,HandlerBaselog_processor=LogProcessor(rules=rules,handler=handler,dry_run=False)log_processor.run('/path/to/my/logs')RulesRules is a standard python dictionary of the format:rules={"file_regex":{"line_regex":"handler_function",}}Where:file_regexis some regex to match the name of a file,line_regexis some regex to match a line within the file,handler_functionis the name of a function in your handler object which will be used to process the line.HandlersThe handler object should be subclassed from the HandlerBase class inhandlers.py. Or, if you wish to parse your logs and upload into a Postgresql database, you can subclass from thePostgresHandlerclass.The handler class can implementstartupandshutdownmethods. Which will be ran at the start and end of the processing run respectively. These can be used to perform some database setup or cleanup.handler_functions will have the signature:defhandler(self,file_path,line,match):Where:file_pathis the path to the file of the current linelineis the line in the log file to be handledmatchis the regex match groupWhen using thePostgresHandler, you can callself.queue_op(sql, params)in your handler functions to queue a database operation. By default this will run SQL operations in batches of 1000, you can customise this by passing theBATCH_SIZEparameter in the constructor toPostgresHandler. If you want to run a database operation immediately, callself.queue_op(sql, params, run_now=True).Full examplefromProcessorimportLogProcessorfromhandlerimportPostgresHandlerclassMyHandler(PostgresHandler):def__init__(self,*args,**kwargs):super().__init__(*args,**kwargs):defstartup(self):"""Optionally run some setup"""passdefshutdown(self):"""Optionally run some cleanup"""passdefmy_handler(self,file_path,line,match):field_1=match.group(1)field_2=match.group(2)sql="""INSERT INTO my_table (field_1, field_2)VALUES(%s,%s);"""params=(field_1,field_2)self.queue_op(sql,params,run_now=False)rules={'example\.log':{'(.*),(.*)':'my_handler','.*':'skip'}}handler=MyHandler(dsn='user:pass@localhost:5432/test',setup_script='path/to/db_setup')log_processor=LogProcessor(rules=rules,handler=handler,dry_run=False)log_processor.run('/path/to/my/logs')The Handler classThe library only stipulates that the handler object passed to theLogProcessorobject is an instance ofHandlerBase.You should subclass fromHandlerBaseand add your own methods to handle the lines found by your rules.Override thestartupandshutdownmethods in your handler class to run a function at the start and end of parsing, respectively.The PostgresHandler classOr, if you wish to make use of the includedPostgresHandlerfor uploading data into a PostgreSQL database, subclass from that instead.The PostgresHandler object has the following constructor:classPostgresHandler(HandlerBase):def__init__(self,dsn:Optional[str]=None,connection:Optional[Connection]=None,setup_script:Optional[str]=None,BATCH_SIZE:int=1000):dsnoptionally provide a dsn string which will be used to connect to an existing PostgreSQL database or;connectionoptionally provide an existing psycopg3 connection. Useful for unit tests.setup_scriptoptionally provide the path to a SQL file in order to perform some database setup/cleanup in between runs.BATCH_SIZEExecute database operations in batches of BATCH_SIZE, defaults to 1000.In your handler functions, define a SQL string and args tuple, and pass them to thequeue_opfunction. These should be setup according to the psycopg3 format, see the example above. If you wish to execute a database operation immediately, passrun_now=Truetoqueue_op, otherwise, it will be added to a queue, and executed in sequence when the size of the queue reachesBATCH_SIZE.ContributingAs mentioned above, the only data store that is currently supported is Postgres. If you would like to add support for another data store such as MySQL or MongoDB, then please open a pull request.