Package: cumulus-message-adapter-python
cumulus-message-adapter-pythonWhat is Cumulus?Cumulus is a cloud-based data ingest, archive, distribution and management
prototype for NASA's future Earth science data streams.Read theCumulus DocumentationWhat is the Cumulus Message Adapter?The Cumulus Message Adapter is a library that adapts incoming messages in the
Cumulus protocol to a format more easily consumable by Cumulus tasks, invokes
the tasks, and then adapts their response back to the Cumulus message protocol
to be sent to the next task.Installationpip install cumulus-message-adapter-pythonTask definitionIn order to use the Cumulus Message Adapter, you will need to create two
methods in your task module: a handler function and a business logic function.The handler function is a standard Lambda handler function which takes two
parameters (as specified by AWS):eventandcontext.The business logic function is where the actual work of your task occurs. It
should take two parameters:eventandcontext.Theeventobject contains two keys:input- the task's input, typically thepayloadof the message, produced
at runtimeconfig- the task's configuration, with any templated variables resolvedThecontextparameter is the standard Lambda context as passed by AWS.The return value of the business logic function will be placed in thepayloadof the resulting Cumulus message.Expectations for input, config, and return values are all defined by the task,
and should be well documented. Tasks should thoughtfully consider their inputs
and return values, as breaking changes may have cascading effects on tasks
throughout a workflow. Configuration changes are slightly less impactful, but
must be communicated to those using the task.Cumulus Message Adapter interfaceThe Cumulus Message adapter for python provides one method:run_cumulus_task. It takes four parameters:task_function- the function containing your business logic (as described
above)cumulus_message- the event passed by Lambda, and should be a Cumulus
Message,ora CMA parameter encapsulated message (seeCumulus Workflow
Documentation):{"cma":{"event":"<cumulus message object>","SomeCMAConfigKey":"CMA configuration object>"}}context- the Lambda contextschemas- optional: a dict withinput,config, andoutputproperties.
Each should be a string set to the filepath of the corresponding JSON schema
file. All three properties of this dict are optional. If ommitted, the message
adapter will look in/<task_root>/schemas/<schema_type>.json, and if not
found there, will be ignored.taskargs- Optional. Additional keyword arguments for thetask_functionExampleSimple example of using this package'srun_cumulus_taskfunction as a wrapper
around another function:>>>fromrun_cumulus_taskimportrun_cumulus_task# simple task that returns the event>>>deftask(event,context):...returnevent# handler that is provided to aws lambda>>>defhandler(event,context):...returnrun_cumulus_task(task,event,context)For a full example see theexample folder.Creating a deployment packageTasks that use this library are just standard AWS Lambda tasks. Seecreating release packages.Usage in a Cumulus DeploymentFor documenation on how to utilize this package in a Cumulus Deployment, view
theCumulus Workflow Documenation.DevelopmentDependency Installation$ pip install -r requirements-dev.txt
$ pip install -r requirements.txtLogging withCumulusLoggerIncluded in this package is thecumulus_loggerwhich contains a logging classCumulusLoggerthat standardizes the log format for Cumulus. Methods are
provided to log error, fatal, warning, debug, info, and trace.Import theCumulusLoggerclass:>>>fromcumulus_loggerimportCumulusLoggerInstantiate the logger inside the task definition (name and level are
optional):>>>importlogging>>>logger=CumulusLogger("event_name",logging.ERROR)Use the logging methods for different levels:>>>logger.trace('<your message>')>>>logger.debug('<your message>')>>>logger.info('<your message>')>>>logger.warn('<your message>')>>>logger.error('<your message>')>>>logger.fatal('<your message>')It can also take additional non-keyword and keyword arguments as in Python
Logger.Themsgis the message format string, theargsandkwargsare the
arguments for string formatting.Ifexc_infoinkwargsis notFalse, the exception information in theexc_infoorsys.exc_info()is added to the message.>>>logger.debug(msg,*args,**kwargs)Example usage:>>>importos>>>importsys>>>fromrun_cumulus_taskimportrun_cumulus_task>>>fromcumulus_loggerimportCumulusLogger# instantiate CumulusLogger>>>logger=CumulusLogger()>>>deftask(event,context):...logger.info('task executed')......# log error when an exception is caught...logger.error("task formatted message{}exc_info ","bar",exc_info=True)......# return the output of the task...return{"example":"output"}>>>defhandler(event,context):...# make sure event & context metadata is set in the logger...logger.setMetadata(event,context)...returnrun_cumulus_task(task,event,context)Running TestsRunning tests requireslocalstack.Tests only require localstack running S3, which can be initiated with the
following command:$ SERVICES=s3 localstack startAnd then you can check tests pass with the following nosetests command:$ CUMULUS_ENV=testing nose2Linting$ pylint run_cumulus_task.pyWhy?This approach has a few major advantages:It explicitly prevents tasks from making assumptions about data structures
likemetaandcumulus_metathat are owned internally and may therefore
be broken in future updates. To gain access to fields in these structures,
tasks must be passed the data explicitly in the workflow configuration.It provides clearer ownership of the various data structures. Operators ownmeta. Cumulus ownscumulus_meta. Tasks define their ownconfig,input, andoutputformats.The Cumulus Message Adapter greatly simplifies running Lambda functions not
explicitly created for Cumulus.The approach greatly simplifies testing for tasks, as tasks don't need to
set up cumbersome structures to emulate the message protocol and can just
test their business function.LicenseApache 2.0