Package: logger-to-kibana
logger-to-kibanaThis project is inteded to generate view from the log messages encountered.The python executable can be found inhttps://pypi.org/project/logger-to-kibana/You will need to install the dependences by runningpipinstall-rrequirements.txtTo get the programs help just type:pythonmain.pyThis returns:Usage:main.py[OPTIONS]COMMAND[ARGS]...Processthefileprovidedaccordingtoconditions

Options:--helpShowthismessageandexit.

Commands:pommands:processProcessthefolderprocess_and_generateProcessthefolderandgeneratevisualizationprocess_generate_and_sendProcessthefolder,generatevisualizationandsendI have created a dev.to series explaining how this project works and any learning that I have taken from doing so. You can find it inhttps://dev.to/ismaelmartinez/setting-up-my-first-opensource-python-project-4k1oDefault settingsThe default settings can be found in thesettings.inifile. You can provide a different settings
file by specifying it as an environment variable LOGGER_TO_KIBANA_CONFIGcommandsThe current available commands are:processProcess a folder and prints out the processed functions/logs in the following format:[{'subfolder':'<folder_name>','filename':'<filename>','function':'<function_name>','type':'<log_type>','query':'message: "<log_filter>"','label':'<log_type>: <log_filter>'}]To execute the command run:pythonmain.pyprocess-f<folder_location>Check the table underHow does it worksection to get more info about log_type and log_filter.process_and_generateProcess a folder (as shown in the process section) and generates a table visualization for kibana.To execute the command run:pythonmain.pyprocess_and_generate-f<folder_location>process_generate_and_sendProcess a folder, generates a table visualization for kibana and send it to kibana (currently in localhost:5601)To execute the command run:pythonmain.pyprocess_and_generate-f<folder_location>How does it workBy default, it scans for files under the folder specified and with the  patternapp/src/**/*.py. You can specify another patter in the FilesMatchFilter in thesettings.iniThis program uses different regexdetectorsto filter logs and files to process.Those can be changed in thesettings.inifile.The current available detectors are:DetectorDefault ValueProposeFilesMatchFilterapp/src/**/*.pyFilter the files to process in the provided folderFunctionMappingDetectordefDetect a functionFunctionMappingFilter(?<=def ).*?(?=()Filter the function nameLogDebugDetectorLOG.debugDetect the log debug messageLogDebugFilter(?<=LOG.debug(["']).*?(?=["'])Filter the log debug messageLogInfoDetectorLOG.infoDetect the log info messageLogInfoFilter(?<=LOG.info(["']).*?(?=["'])Filter the log info messageLogWarnDetectorLOG.warnDetect the log warn messageLogWarnFilter(?<=LOG.warn(["']).*?(?=["'])Filter the log warn messageLogErrorDetectorLOG.errorDetect the log error messageLogErrorFilter(?<=LOG.error(["']).*?(?=["'])Filter the log error messageLogCriticalDetectorLOG.criticalDetect the log critical messageLogCriticalFilter(?<=LOG.critical(["']).*?(?=["'])Filter the log critical messageLogExceptionDetectorLOG.exceptionDetect the log exception messageLogExceptionFilter(?<=LOG.exception(["']).*?(?=["'])Filter the log exception messageOther configuration available in the settings.ini file are:TypeValueProposeBaseUrlhttp://localhost:5601Kibana base urlIndex59676040-e7fd-11e9-9209-1f165c3af176Kibana indexVisualizationTypemetricType of visualization to generate. Valid options are metric or tableAuthTypeNoneAuthentication Type. You can use 'aws' to use requests_aws4auth to authenticate with awsThe processThe commands for the application are done in the following logical order.process->generate->sendAs an example, when processing a file intests/unit/resources/example.pywith the content:deflambda_handler(_event:dict,_context):LOG.debug('Initialising')LOG.info('Processing')LOG.warn('Success')LOG.error('Failure')LOG.critical('Bananas')LOG.exception('Exception'))Will return the next object:[{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'debug','query':'message: "Initialising"','label':'debug: Initialising'},{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'info','query':'message: "Processing"','label':'info: Processing'},{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'warn','query':'message: "Success"','label':'warn: Success'},{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'error','query':'message: "Failure"','label':'error: Failure'},{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'critical','query':'message: "Bananas"','label':'critical: Bananas'},{'subfolder':'resources','filename':'example.py','function':'lambda_handler','type':'exception','query':'message: "Exception"','label':'exception: Exception'}]It will generate a metric visualization with filters for all the logs that have found.Those visualisations will be split by function, then filename and then subfolder.You can change the type of visualization generated by modifying the VisualizationType in thesettings.ini. The current available values are metric or table. The default value is metric.To finish, it sends the generated visualization to Kibana with the following name format:[Generated-<folder><subfolder><filename><function>]LimitationsIt does not detect logs assign to variables or constants. They will not appear in your visualizations.#40should address this.