Package: xFlow
xFlow=====A serverless workflow architecture using AWS Lambda functions and Kinesis.How it works============The concept behind xFlow is to define a workflow which comprises ofa number steps that gets executed to complete a task such that the outputof one step is fed in as the input to the next step - except that instead of theoutput being fed in as input, the preceding step publishes an event which is subscribedby the next step. This way each step is executed independently and can run in parallel.These steps are AWS Lambda functions and primarily, they should do the following:- It should be subscribed to listen on certain events- Do work based on these events- Optionally emit an output event for other lambda functions to work onxFlow uses AWS Kinesis as the streaming engine to publish and subscribe events in a workflow.As described above, these events would then be processed via AWS Lambda functions after theyhave been subscribed to it. The configuration file that you provide to xFlow would be the gluethat combines events to the processors in a workflow.xFlow will internally create the AWS Lambda functions and subscribe them to AWS Kinesis.All you need to do is write the body of the lambda functions.xFlow supports all languages AWS Lambda supports. At this moment, lambdas can only beuploaded from the local filesystem or s3. In the future we will support integration withgithub.Creating a sample workflow:==========================- Create a config that defines your workflow. For example:```yamlgeneral:lambda_timeout_time: 3aws:region:subnet_ids:security_group_ids:lambda_execution_role_name: lambda-executelambdas:- name: lambda_readerdescription: Reads contents of a file that was uploaded and publishes those contents.source: /xFlow/examples/wordcount/lambda_reader.pyhandler: readruntime: python2.7- name: lambda_parserdescription: Reads contents, parses it into an array of words.source: s3://xFlow/flows/wordcount/lambda_parser.pyhandler: parseruntime: python2.7- name: lambda_filterdescription: Filters non-letters from words and non-words.source: /xFlow/examples/wordcount/lambda_filter.pyhandler: filter_out_non_wordsruntime: python2.7- name: lambda_aggregatordescription: Groups similar words and counts them.source: /xFlow/examples/wordcount/lambda_aggregator.pyhandler: aggregateruntime: python2.7- name: lambda_summarizerdescription: Prints unique word counts.source: /xFlow/examples/wordcount/lambda_summarizer.pyhandler: summarizeruntime: python2.7subscriptions:- event: FileUploadedsubscribers:- lambda_reader- event: FileDownloadedsubscribers:- lambda_parser- event: FileParsedsubscribers:- lambda_filter- event: FileFilteredsubscribers:- lambda_aggregator- event: FileAggregatedsubscribers:- lambda_summarizer- event: FileSummarizedsubscribers:# Optional - Can be used to track a complete workflowworkflows:- id: compute_word_countflow:- FileUploaded- FileDownloaded- FileParsed- FileFiltered- FileAggregated- FileSummarized```- Setup the workflow via the following command:`xflow word_count.cfg --configure`Behind the scenes, this will do the following:- Create (or update) AWS Lambda functions.- For each lambda function, subscribe them to AWS Kinesis based on the events they are listening for.- Lambda functions will be executed once an event is published to AWS Kinesis.- Tracking the workflow is done via the following command:`xflow word_count.cfg --track <WORKFLOW_ID> <EXECUTION_ID>`An instance of a particular workflow is tracked via its `execution_id`. Therefore all events definedin a workflow must contain this field. Since it is required to have the events defined in order itwill then be easy to determine which events were successfully processed and which were not at anyparticular time in the last 7 days.The goal of the tracker is to return the state of all events in the workflow given as inputthe `workflow_id` and its `execution_id`. The state would indicate whether the event was processed or not.It will then grab the last unprocessed event and fetch all logs for its subscribers (the lambdas) that failedto process that event.For this to work, there would be a separate tracker for every workflow defined. The tracker would subscribeitself to every stream in that workflow and hence it would receive events published to that stream. The tracker itself would be a lambda function and its name would be `tracker_<workflow_id>`. Hence it will be invoked on everysuccessful publishing of an event to the stream.The actual tracking is done via AWS CloudWatchLogs. For every workflow a CloudWatch `LogGroup` is created. And for every instance of a workflow execution, a `LogStream` would be created.Hence, when an event is published to a stream, the tracker would receive the event and extract the`execution_id`. It will create a new log stream from this execution_id if not already created. It willthen store the event in that stream. Since the lambda's name is generated from the workflow_id, it is alsoeasy to extract the log group for the log stream.For every subsequent event published to the kinesis stream, the corresponding lambda for the workflow will be invoked and it will save the event to the log stream.- Running in server mode:`xflow word_count.cfg --server`This will run xflow via server mode. On startup, the server will setup the necessary streams, lambda functions and workflows. You can then publish events to a stream or track workflow executions in a RESTful way. Following are examples how you would do this.Publishing:`curl -XPOST localhost/publish -d '{"stream":"FileUploaded", "event":{"execution_id":"ex1", "message":"Test with ccc"}}'`Tracking:`curl -v localhost/track/workflows/compute_word_count/executions/ex1`Installation:=============- Via pip```bash>> pip install xflow```- Via github```bash>> git clone git@github.com:dsouzajude/xFlow.git>> cd xFlow>> python setup.py install```xFlow Requirements (and roadmap):=================================- Integration with github (so we can download the lambda function from there given the link)- Monitoring around lambda functions- Centralized Logging (or ability to route logs) to log server- CI/ CD for lambda functions