Package: cloudwatch-logging
OverviewThis package provides structured JSON logging faculties designed to seamlessly interoperate with log aggregators like
AWS Cloudwatch, ELK, and others. These faculties are provided via extension of the python's existing logging library
and can be used as a drop in replacement.Usage:If you want to generate simple structured logs, use this package just like the Python logging library.If you want inject contextual information into your structured logs, add a LogAppender. This is great option if you want
to run your application in somewhere like AWS Lambda, where you would want your logs to contain aws_request_id and other
function specific metadata. See Use Case: AWS Lambda for more.If you want to ensure that unwanted information is removed from your logs, use a LogFilter. Filters are
a great way to trim the size of your log lines to cut down on logging costs. Filters can also be useful in specific log
stacks - namely the ELK stack - that need to inject specific keys like thetypekey to index logs correctly.Filters and Appenders can be mixed and matched through your logger's addFilter and removeFilter methods.
Appenders and Filters are both dynamically configurable via the logger's update_appender and update_filter methods.Basic use: JSON loggingimportloggingfromcloudwatch_loggingimportCloudwatchLogging# create a structured JSON logger that works well with Cloudwatch and others; optionally with runtime specific appenders and filters to remove unwanted fieldslogger=CloudwatchLogging.create_logger(__name__,appender=None,filter=None)logger.setLevel(logging.INFO)logger.warning("Structured logging with custom fields",extra={"custom_field":"custom_value"})Use case; AWS Lambdaimportloggingfromcloudwatch_loggingimportCloudwatchLogging# Setup logginglogger=CloudwatchLogging.create_logger("your_lambda_function")logger.propagate=False# disable Lambda runtime default logger from double logging lines sent to this loggerlogger.setLevel(logging.INFO)defyour_lambda_handler(event,context):lamdbda_appender=CloudwatchLogging.LogAppender(context)logger.addFilter(lamdbda_appender)logger.info("This line will be logged with info from the Lambda context object!",extra={"tapped_in":True})logger.removeFilter(lamdbda_appender)# ensure you do this so you don't add duplicate appenders on the same loggerUse case; Cloudwatch Cost Optimizationimportloggingfromcloudwatch_loggingimportCloudwatchLogging,Filterslogger=CloudwatchLogging.create_logger("your_lambda_function")logger.setLevel(logging.INFO)# add a LogTrimmer on the logger and save some money on CW storage costslogger.addFilter(CloudwatchLogging.LogFilter(Filters.COST_EFFECTIVE))logger.warning("Structured logging on the cheap!",extra={"field":"value"})DevelopmentSetupThis is a pure Python library; it can easily be installed usingpip install cloudwatch_logging. Use the-earg if you
plan to develop locally.TestRun python3 test_cloudwatch_logging.py. Inspect the output to see what is logged. Someone should write proper tests
but I wrote this on vacation and couldn't be bothered.To-DoCheck error and exception trace handling - how can we make these better?Add examples for other AWS runtimes (Fargate?)