Package: memphis-functions
Cloud-Docs-X-YouTubeMemphis.devis more than a broker. It's a new streaming stack.Memphis.dev is a highly scalable event streaming and processing engine.AboutBefore Memphis came along, handling ingestion and processing of events on a large scale took months to adopt and was a capability reserved for the top 20% of mega-companies. Now, Memphis opens the door for the other 80% to unleash their event and data streaming superpowers quickly, easily, and with great cost-effectiveness.This repository is responsible for the Memphis Functions Python SDKInstallation$pip3installmemphis-functionsImportingfrommemphisimportcreate_functionCreating a Memphis functionMemphis provides a create_function utility for more easily creatin Memphis Functions.The user createdevent_handlerwill be called for every message in the given batch of events. The user'sevent_handlerwill take in amsg_payloadas bytes,msg_headersas a dict andinputsas a dict, and should return a modified version of the payload and headers in the same data types.The user function should raise an exception if the message processing has failed. If any exception is raised (deliberately or by a failed operation) the message will be sent to the dead letter station.If the returned modified version of themsg_payloadormsg_headersare returned asNone, then the message will be skipped and will not be sent to the station or dead letter station.Make sure to encode the modifiedmsg_payloadbytes object with utf-8 encoding!This example function takes the bytes objectmsg_payloadand encodes it into a string so that it may be parsed as JSON.importjsonimportbase64frommemphisimportcreate_functiondefhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler)defevent_handler(msg_payload,msg_headers,inputs):payload=str(msg_payload,'utf-8')as_json=json.loads(payload)as_json['modified']=Truereturnbytes(json.dumps(as_json),encoding='utf-8'),msg_headersInstead of takingmsg_payloadas a bytes object, the as_dict flag can be used to have the JSON parsed to a dictionary.importjsonimportbase64frommemphisimportcreate_functiondefhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler,as_dict=True)defevent_handler(msg_payload,msg_headers,inputs):msg_payload['modified']=Truereturnmsg_payload,msg_headersMemphis Functions support using Async functions through asyncio. When functions are async, set the use_async parameter to true.importjsonimportbase64importasynciofrommemphisimportcreate_functiondefhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler,use_async=True)asyncdefevent_handler(msg_payload,msg_headers,inputs):payload=str(msg_payload,'utf-8')as_json=json.loads(payload)as_json['modified']=Trueasyncio.sleep(1)returnbytes(json.dumps(as_json),encoding='utf-8'),msg_headersIf the user would want to have a message that they would want to validate and send to the dead letter station if the validation fails then the user can raise an exception. In the following example, the fieldcheckis simply a boolean. The following function will send any messages which fail thecheckto the dead letter station.importjsonimportbase64frommemphisimportcreate_functiondefhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler)defevent_handler(msg_payload,msg_headers,inputs):payload=str(msg_payload,'utf-8')as_json=json.loads(payload)ifas_json['check']==False:raiseException("Validation Failed!")returnbytes(json.dumps(as_json),encoding='utf-8'),msg_headersIf a user would rather just skip the message and not have it be sent to the station or dead letter station, the cuser could instead returnNone,None:importjsonimportbase64frommemphisimportcreate_functiondefhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler)defevent_handler(msg_payload,msg_headers,inputs):payload=str(msg_payload,'utf-8')as_json=json.loads(payload)ifas_json['check']==False:returnNone,Nonereturnbytes(json.dumps(as_json),encoding='utf-8'),msg_headersLastly, if the user is using another data format like Protocol Buffers, the user may simply decode themsg_payloadinto that format instead of JSON. Assuming we have a .proto definition like this:syntax="proto3";packageprotobuf_example;messageMessage{stringdata_field=1;}We can decode this and get the data_field out like this:importjsonimportbase64frommemphisimportcreate_functionimportmessage_pb2defhandler(event,context):# The name of this file and this function should match the handler field in the memphis.yaml file in the following format <file name>.<function name>returncreate_function(event,event_handler=event_handler)defevent_handler(msg_payload,msg_headers,inputs):message=message_pb2.Message()message.ParseFromString(base64.b64decode(encoded_str))# Arbitrarily changing the data_fieldmessage.data_field="my new data"# SerializeToString returns bytes, which is the type we wantreturnmessage.SerializeToString(),msg_headers