Package: echostream-node
echostream-nodeEchoStream library for implementing remote nodes that can be used in the echostream system.This package supports creating External Nodes and Managed Node Types,
and supports the following EchoStream use cases:An External Node in an External App or Cross Account App that is a stand-alone application or part of another application, using eitherthreadingorasyncio.An External Node in a Cross Account App that is an AWS Lambda function. This use case only supportsthreading.A Managed Node Type, using eitherthreadingorasyncioInstallationPythonpipinstallechostream-nodeAWS LambdaYou may use the publiclally provided layer instead of directly installingechostream-nodein your lambda package. This layer includesechostream-nodeand all of the Python dependenciesexceptthose built-in to the AWS Lambda environment for Python.The Layer arn is:arn:aws:lambda:{region}:226390263822:layer:echostream-node-{version}:1where{version}is the version ofechostream-nodethat you want, with.replaced with_and{region}is the AWS region that your Lambda will run in. Currently,us-east-1,us-east-2,us-west-1andus-west-2are supported.For example, forechostream-node==0.3.7in theus-east-1region the layer arn would be:arn:aws:lambda:us-east-1:226390263822:layer:echostream-node-0_3_7:1UsageConfigurationTo instantiate a Node a number of variables are required. These can be provided either as environment variables or directly on Node creation:ParameterEnvironment VariableDescriptionappsync_endpointAPPSYNC_ENDPOINTThe URL to the EchoStream API endpoint.client_idCLIENT_IDThe Application Client ID for the App's Cognito Client Application.nameNODEThe Node's name.passwordPASSWORDThe password for the App User for the Node's App.tenantTENANTThe name of the Tenant that the Node is a part of.usernameUSER_NAMEThe name of the App User for the Node's App.user_pool_idUSER_POOL_IDThe User Pool Id for the App's Cognito User Pool.Threading Application NodefromsignalimportSIGHUP,SIGINT,SIGTERM,signal,strsignalfromechostream_nodeimportMessagefromechostream_node.threadingimportAppNodeclassMyExternalNode(AppNode):defhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)defsignal_handler(self,signum:int,_:object)->None:print(f"{strsignal(signum)}received, shutting down")self.stop()defstart(self)->None:super().start()signal(SIGHUP,self.signal_handler)signal(SIGINT,self.signal_handler)signal(SIGTERM,self.signal_handler)try:my_external_node=MyExternalNode()my_external_node.start()foriinrange(100):message=my_external_node.create_message(str(i))my_external_node.send_message(message)my_external_node.audit_message(message)my_external_node.join()exceptException:print("Error running node")Asyncio Application Nodeimportasyncioimportaiorunfromechostream_nodeimportMessagefromechostream_node.asyncioimportNodeclassMyExternalNode(Node):asyncdefhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)asyncdefmain(node:Node)->None:try:awaitnode.start()foriinrange(100):message=my_external_node.create_message(str(i))my_external_node.send_message(message)my_external_node.audit_message(message)awaitnode.join()exceptasyncio.CancelledError:passexceptException:print("Error running node")if__name__=="__main__":aiorun.run(main(MyExternalNode()),stop_on_unhandled_errors=True,use_uvloop=True)Cross Account Lambda Nodefromechostream_nodeimportMessagefromechostream_node.threadingimportLambdaNodeclassMyExternalNode(LambdaNode):defhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)MY_EXTERNAL_NODE=MyExternalNode()deflambda_handler(event,context):MY_EXTERNAL_NODE.handle_event(event)Concurrent vs Sequential Message ProcessingBy default, all Nodes created using the package will process messages sequentially.
This is normally the behavior that you want, as many messaging protocols require
guaranteed ordering and therefore sequential processing within your Nodes. If this is
the behavior that you require, nothign special is needed to gain it fromechostream-node.However, there are use cases where message ordering is not important but processing speed is.
In these cases, you may configure your Node upon creation to concurrently process the messages
that it receives.Making a Threading Application Node ConcurrentIf your Node inherits from theechostream_node.threading.AppNodeclass you can achieve concurrency
using threading.This will create an AppNode that uses the providedThreadPoolExecutorto concurrently
process receivedMessages. Note that while you can set the maximum number of workers to
less than 10, there is no gain to setting it to more than 10 since Nodes will only process
up to 10 messages at a time.fromconcurrent.futuresimportThreadPoolExecutorfromechostream_nodeimportMessagefromechostream_node.threadingimportAppNodeclassMyExternalNode(AppNode):def__init__(self)->None:super().__init__(executor=ThreadPoolExecutor(max_workers=10))defhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)Making a Asyncio Application Node ConcurrentIf your Node inherits from theechostream_node.asyncio.Nodeyou can set the Node to
process incomingMessages concurrently. There is no setting for the maximum number of tasks;
a task is created per receivedMessage.importasynciofromechostream_nodeimportMessagefromechostream_node.asyncioimportNodeclassMyExternalNode(Node):def__init__(self)->None:super().__init__(concurrent_processing=True)asyncdefhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)Making a Lambda Node ConcurrentThe AWS Lambda platform does not support shared memory, and therefore will only support concurrency
via threading. This will create a LambdaNode that uses an optimized (to your Lambda function's resources)ThreadPoolExecutorto concurrently process receivedMessages.fromechostream_nodeimportMessagefromechostream_node.threadingimportLambdaNodeclassMyExternalNode(LambdaNode):def__init__(self)->None:super().__init__(concurrent_processing=True)defhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)Lambda Nodes and Partial Success ReportingWhen you connect an Edge's SQS Queue to the AWS Lambda function implementing your
Lambda Node, you can choose to Report Batch Item Failures. This allows your Lambda Node
to report partial success back to the SQS Queue, but it does require that your Lambda Node
operate differently.If you wish to take advantage of this, setreport_batch_item_failureswhen you create your
Lambda Node. This can be set even if your Node isnotconcurrent processing.fromechostream_nodeimportMessagefromechostream_node.threadingimportLambdaNodeclassMyExternalNode(LambdaNode):def__init__(self)->None:super().__init__(report_batch_item_failures=True)defhandle_received_message(self,*,message:Message,source:str)->None:print(f"Got a message:\n{message.body}")self.audit_message(message,source=source)Full documentation may be found athttps://docs.echostream-node.echo.stream.