Package: raise-me
raise-meServerless event broker which allows connecting native cloud events to targets declaratively.The native cloudeventscan be raised from eitherAWSorGCP.Thetargetscan be eitherHTTP endpointsorOpenWhisk Actions.ExampleDeclare your event defintions inraise-events.yaml:events:s3-to-cloudfunction:# Event's logical name.source:provider:awsfilters:# AWS EventBridge event patterns.-'source:["aws.s3"]'-'detail-type:["ObjectCreated"]'-'detail:{"bucket":{"name":["my-bucket-name"]}}'targets:-http:method:post# Sends event data in CloudEvents format.url:my-cloudfunction-url-action:name:my-openwhisk-action-namecloudstorage-to-lambda:source:provider:gcpfilters:# GCP Eventrac filters.-'type=google.cloud.audit.log.v1.written'-'serviceName=storage.googleapis.com'-'methodName=storage.objects.create'targets:-http:method:geturl:my-lambda-urlIntroductionThis solution enables declarative native cloud event routing (e.g. AWS S3's "Object Created") to either HTTP endpoints or user-defined serverless functions deployed asApache OpenWhiskActions.An example use case could be triggering aGCP Cloud Functionwhen a file is uploaded to anAWS S3bucket.
Such process is accomplished by creating necessary cloud resources that will route the event to the OpenWhisk deployment, which will then receive the event's data in theCloudEventsformat and forward it to the Cloud Function through an API call.Table of ContentsMotivationRequirementsConfigurationPulumiEnable Google ServicesInstallationTestingUsage1. Prepare Inputs2. Create Resources3. CleanupArchitectureContriburingLicenseFuture FeaturesMotivationThe implementation alternatives for the previous example (trigger Cloud Function from S3 event) are countless. A straightforward approach could be linking a Lambda to the event, which is fairly simple within the AWS ecosystem, and communicate with the Cloud Function from there.Although this alternative may seem initially as a good approach, there are several design limitations. If we were to trigger 2 Cloud functions instead of 1, we would have to deploy a new Lambda with code containing logic to reach both Cloud functions.
What if we want to reach 100 endpoints from that single event? Lambdas have a 15-minute time limit on their execution, so extending this event-based communication solution would have limitations in terms of:Extensibility– the number of endpoints would be limited by Lambda. Perhaps, we would have to create more Lambdas to handle different endpoints.Maintainability– the logic at the Lambda layer might require frequent changes.Complexity– the logic at the Lambda layer will increase.Portability– this schema would only work for native AWS events, but not for other clouds, such as GCP.The solution thatraise-meprovides addresses all of these limitations by using serverless services and an abstraction layer that simplifies the management of the required infrastrcture that makes the event routing possible.RequirementsApache OpenWhiskdeployment -where the event processors run; OpenWhisk has severaldeployment options, including Kubernetes.Pulumi-used to create cloud resources.AWS Account &AWS CLI-optional, if AWS events are to be listened.GCP Account -optional, if GCP events are to be listened.ConfigurationPulumiConfigure Pulumi to access your AWS accountand/orGoogle Cloud account. If you are just interested in one of those event sources, you can ignore the other.Create a Pulumi project. You can use either a Python template, AWS or GCP. For example:$mikdiryour-project-name
$puluminewaws-pythonActivate your pulumi project's virtual environment and installraise-me(available on PyPi):$source./venv/bin/activate
$./venv/bin/python.exe-mpipinstallraise-meInPulumi.<stack-name>.yaml, make sure you have the appropriate configuration for the corresponding clouds:config:aws:region:your-target-regiongcp:project:your-project-idIn your__main__.pyPulumi file, include the code that will create the cloud resources for you:fromraise_me.buildimportCloudBuilderbuilder=CloudBuilder(config_path='path/to/raise-config.yaml')builder.update_stack(events_path='path/to/raise-events.yaml')Enable Google ServicesEnable the following Google services:Eventrac-eventrac triggers will be created.PubSub-used internally by Eventrac.Workflows-workflows will be created.Cloud Logging-provides more event options.If you are interested in handling events from services that log into Cloud Logging,enable the logging of those services.InstallationRun (available on PyPi):$pipinstallraise-meVerify installation:$raise--helpTestingInstallPoetryif you don't have it already:$pipinstallpoetryClone the repository, create a virtual environment at the project level and activate it:$python3-mvenvmy-venv
$sourcemy-venv/bin/activateInstall project dependencies using Poetry:$poetryinstallModify thetests/raise-config.yamland provide the connection details of the target OpenWhisk deployment (seeUsagefor more details). You can also modify the contents oftests/raise-events.yaml, but it is not necessary for these tests.After this, you can run the following integration tests that verify that the connection with the OpenWhisk deployment is successful:$pytest-v-mclient# Tests OpenWhisk API client.$pytest-v-mpaginator# Tests paginator interface using the API client.$pytest-v-mbuilder# Tests creation/destruction of raise-me OpenWhisk resources.Usage1. Prepare InputsThere are 2 required inputs, namelyraise-config.yamlandraise-events.yaml.Theraise-config.yamlfile contains configuration settings that allow connecting to the OpenWhisk deployment and creating cloud resources.(The auth field in the example contains default values for a basic unchanged OpenWhisk deployment.)openwhisk:namespace:guest# Storage location of Openwhisk resources.endpoint:https://my-endpoint:443# Openwhisk deployment endpoint.auth:# Authentication for REST API.username:23bc46b1-71f6-4ed5-8c54-816aa4f8c502password:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwPgcp:# Used for cloud resource creation.project-id:my-project-idregion:my-region# Should match pulumi config.Theraise-events.yamlwill contain the event definitions. Each event definition is composed by a Source and a list of Targets.
To specify the desired events, we use each cloud's selection strategy; event patterns for AWS and Eventrac filters for GCP.events:s3-to-cloudfunction:# Your event's logical name.source:provider:awsfilters:# AWS EventBridge event patterns.-'source:["aws.s3"]'-'detail-type:["ObjectCreated"]'-'detail:{"bucket":{"name":["my-bucket-name"]}}'targets:-http:method:post# Sends event data in CloudEvents format.url:my-cloudfunction-url-action:name:my-openwhisk-action-namecloudstorage-to-lambda:source:provider:gcpfilters:# GCP Eventrac filters.-'type=google.cloud.audit.log.v1.written'-'serviceName=storage.googleapis.com'-'methodName=storage.objects.create'targets:-http:method:geturl:my-lambda-urlNotes on theraise-events.yaml:Each event definition must have only 1providerand at least 1 target.The providers available areawsandgcp.Targets available includehttpandaction.httptargets must containmethodand aurl. Current available methods aregetandpost, where the later sends the event's data to the target.actiontargets must contain thenameof the OpenWhisk action to trigger, which should be available within the namespace specified in theraise-config.yaml.OpenWhisk Actions are language-agnostic, so if you need more advanced event-handling logic, you can deploy your own actions and link them to the one or more events seamlessly!2. Create ResourcesOnce having the inputs ready, we just need to create:Cloud resources that will route the events to our OpenWhisk deployment.OpenWhisk resources that will handle the incoming events.Create the OpenWhisk resources using theraisecommand:$raiseup--config-pathpath/to/raise-config.yaml--events-pathpath/to/raise-events.yamlCreate the cloud resources with Pulumi as you usually would:~/pulumi/project/$pulumiupAnd done! You should have now the required infrastructure in place to route the specified events to their corresponding targets.3. CleanupTo delete the OpenWhisk resources created byraise-me, run:$raisedestroy--config-pathpath/to/raise-config.yamlDelete the cloud resources running:~/pulumi/project/$pulumidestroyArchitectureLet's consider the example given in the introduction and an OpenWhisk deployment inAWS EKS.To forward S3 events to OpenWhisk,raise-meusesAWS EventBridge. It creates anEventRulethat routes the event to aLambdafunction that parses the event and forwards it to the OpenWhisk deployment by firing anOpenWhisk Trigger. Such Trigger will then invoke the necessary Actions to reach the Cloud Function's endpoint.Note that:The services used to route the events to our target, namelyEventBridge,LambdaandOpenWhisk, are serverless and can scale seamlessly.The dotted arrows represent AWS managed communication, whereas the solid lines represent REST API calls implemented in the solution.Although OpenWhisk implements internal load balancing, a deployment in EKS will also include an Elastic Load Balancer (docs). This component is omitted in the diagram as it is AWS-specific.Considering the same OpenWhisk deployment, let's say that we would like to trigger a Lambda function when a file is uploaded to a Cloud Storage bucket. The architecture would look as follows:ContributingPull requests are welcome. For major changes, please open an issue first to discuss the change.Please ensure tests are up to date.LicenseMITFuture FeaturesThorough exception handlingSecure OpenWhisk API credentialsAWS: SecretsManagerGCP: SecretManagerHttp request body/parametersInternal API crendential storage (for HTTP targets)Logging