Package: okdata-aws
okdata-awsCollection of helpers for working with AWS.Logging for LambdaBased onStructlog.Structured and enriched logging for AWS Lambda functions.TL;DR:Decorate handler withlogging_wrapperEncrich logs with key/value pairs usinglog_addTime functions withlog_durationLog exceptions withlog_exceptionUsageRegular Lambda functionsWrap your Lambda handler withlogging_wrapper. Badabing badabom, you're good
to go!You can set the service name using thelogging.initmethod, or configure it
using theSERVICE_NAMEenvironment variable.fromokdata.awsimportloggingfromokdata.aws.loggingimportlogging_wrapperlogging.init("my_fantastic_lambda")@logging_wrapperdefhandler(event,context):iferror:return{"statusCode":500,"body":"Automatically logs bodies from error responses even!",}FastAPI applicationsCalladd_fastapi_loggingwith the FastAPI application as a parameter
to add the logging middleware.fromokdata.aws.loggingimportadd_fastapi_loggingapp=FastAPI()add_fastapi_logging(app)Encriching logsBy automagic logs will be enriched with git revisions, cold start y/n call
duration and much more, but to add even more magic you can uselog_addandlog_duration.fromokdata.aws.loggingimportlogging_wrapper,log_add,log_duration@logging_wrapperdefhandler(event,context):log_add(dataset=event["dataset"],foo=context["foo"])log_duration(lambda:slow_thinger(event["dataset"]),"my_slow_thinger")...andsoondefslow_thinger():sleep(9999999999999999)ExceptionsStruct log can extract exception info if we log the exception to the specialexc_infokey.For convenience we catch and log uncaught exceptions using this already.If you need to process an exception you can uselog_exceptionto log it to theexc_infokey.fromokdata.aws.loggingimportlogging_wrapper,log_exception@logging_wrapperdefhandler(event,context):try:thing()exceptMyExceptionase:log_exception(e)return{...}Status wrapperThe status wrapper logs details about a Lambda function execution and sends it
to the status API.The first component that touches the data (typically on upload) sets a "trace
ID", which is then inherited by the following processing steps. This allows the
status API to track what has happened to the data, from upload through the
various processing steps until the data is ready for consumption.For pipeline components, the status wrapper picks up the trace ID from the
Lambda event automatically.The status wrapper expects theSERVICE_NAMEof the Lambda component to be set
in an environment variable, along withGIT_REVandGIT_BRANCH.UsageTag the Lambda handler function with@status_wrapper.The handler function should set thedomainanddomain_idvalues using thestatus_addmethod:fromokdata.aws.statusimportstatus_wrapper,status_add@status_wrapperdefmy_lambda_handler(event,context):status_add(domain="dataset",domain_id=f"{dataset_id}/{version}")# Regular handler logic here ...# The handler can also add a body object containing component-specific informationstatus_body={"input":"/tmp/file.txt","output":"/tmp/file.csv","transformation":"text-to-csv",}status_add(status_body=status_body)By default, this will send a status event with event statusOKand trace
statusCONTINUE, meaning that the data pipeline is still running. If the
handler function fails, e.g. throws an exception, it will send event statusFAILEDand trace statusFINISHED, in addition to the failure details
(exception).