Package: commlib-py
commlib-pyCommlib is aninternal DSLfor communication and messaging inCyber-Physical Systems. Can be used for
rapid development of the communication layer on-device, at the Edge and on the Cloud.The goal of this project is to implement a simple Protocol-agnostic API (AMQP, Kafka, Redis, MQTT, etc)
for common communication patterns in the context of Cyber-Physical Systems, using message broker
technologies. Such patterns include PubSub, RPC and Preemptive Services (aka Actions), among others.InstallationInstall from PyPi:pipinstallcommlib-pyAlternatively, download this repo and install using setuptools:git clone git@github.com:robotics-4-all/commlib-py.git
cd commlib-py && python setup.py installCommlib can also be used in a virtual envirtonmentpython -m venv myvenv
pip install commlib-pyJSON SerializationIt is recommended to use a fast json library, such asorjsonorujson.The framework will load and use the most performance optimal library based
on installations. The default isujson.GuideNodeANodeis a software component that follows the Component-Port-Connector model.
It has input and output ports for communicating with the world. Each
port defines an endpoint and can be of the following types.Input Port:SubscriberRPC ServiceAction ServiceOutput Port:PublisherRPC ClientAction ClientInOut Port:RPCBridge: Bridge RPC Communication between two brokers. Directional.TopicBridge: Bridge PubSub Communication between two brokers. Directional.PTopicBridge: Bridge PubSub Communication between two brokers, based on a topic pattern. Directional.Furthermore, it implements several features:Publish Heartbeat messages in the background for as long as the node is activeProvide control interfaces, tostartandstopthe execution of the NodeProvides methods to create endpoints and bind to Node ports.fromcommlib.nodeimportNode,TransportTypefromcommlib.msgimportRPCMessage## Import the Redis transports## Imports are lazy handled internallyfromcommlib.transports.redisimportConnectionParametersclassAddTwoIntMessage(RPCMessage):classRequest(RPCMessage.Request):a:int=0b:int=0classResponse(RPCMessage.Response):c:int=0defon_request(msg):print(f'On-Request:{msg}')resp=AddTwoIntMessage.Response(c=msg.a+msg.b)returnrespif__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='add_two_ints_node',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)rpc=node.create_rpc(msg_type=AddTwoIntMessage,# rpc_name='add_two_ints_node.add_two_ints',on_request=add_two_int_handler)node.run_forever(sleep_rate=1)A Node always binds to a specific broker for implementing the input and
output ports. Of course you can instantiate and run several Nodes in a single-process
application.Node class:classNode:def__init__(self,node_name:Optional[str]='',connection_params:Optional[Any]=None,transport_connection_params:Optional[Any]=None,debug:Optional[bool]=False,heartbeats:Optional[bool]=True,heartbeat_uri:Optional[str]=None,compression:CompressionType=CompressionType.NO_COMPRESSION,ctrl_services:Optional[bool]=False):Node methods to create and run Endpoints::Node:create_action(self,*args,**kwargs)create_action_client(self,*args,**kwargs)create_event_emitter(self,*args,**kwargs)create_heartbeat_thread(self)create_mpublisher(self,*args,**kwargs)create_psubscriber(self,*args,**kwargs)create_publisher(self,*args,**kwargs)create_rpc(self,*args,**kwargs)create_rpc_client(self,*args,**kwargs)create_start_service(self,uri:str=None)create_stop_service(self,uri:str=None)create_subscriber(self,*args,**kwargs)run_forever(self,sleep_rate:float=0.001)run(self)stop(self)Endpoint (Low-level API)It is possible to construct endpoints without binding them to a specific
Node. This is a feature to support a wider range of applications, where the
concept Node might not be usable.One can create endpoint instances by using the following classes of each
supported transportRPCClientRPCServerPublisherSubscriberMPublisher (Multi-topic Publisher)PSubscriber (Pattern-based Subscriber)ActionService (Preemptable Services with feedback)ActionClientfromcommlib.transports.redisimportRPCServicefromcommlib.transports.amqpimportSubscriberfromcommlib.transports.mqttimportPublisher,RPCClient...Or use theendpoint_factoryto construct endpoints.importtimefromcommlib.endpointsimportendpoint_factory,EndpointType,TransportTypedefcallback(data):print(data)if__name__=='__main__':topic='factory_test_topic'mqtt_sub=endpoint_factory(EndpointType.Subscriber,TransportType.MQTT)(topic=topic,on_message=callback)mqtt_sub.run()mqtt_pub=endpoint_factory(EndpointType.Publisher,TransportType.MQTT)(topic=topic,debug=True)data={'a':1,'b':2}whileTrue:mqtt_pub.publish(data)time.sleep(1)Communication PattersReq/Resp (RPC) CommunicationServer Side Examplefromcommlib.msgimportRPCMessagefromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersclassAddTwoIntMessage(RPCMessage):classRequest(RPCMessage.Request):a:int=0b:int=0classResponse(RPCMessage.Response):c:int=0defadd_two_int_handler(msg):print(f'Request Message:{msg.__dict__}')resp=AddTwoIntMessage.Response(c=msg.a+msg.b)returnrespif__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='add_two_ints_node',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)rpc=node.create_rpc(msg_type=AddTwoIntMessage,rpc_name='add_two_ints_node.add_two_ints',on_request=add_two_int_handler)node.run_forever(sleep_rate=1)Client Side Exampleimporttimefromcommlib.msgimportRPCMessagefromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersclassAddTwoIntMessage(RPCMessage):classRequest(RPCMessage.Request):a:int=0b:int=0classResponse(RPCMessage.Response):c:int=0if__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='myclient',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)rpc=node.create_rpc_client(msg_type=AddTwoIntMessage,rpc_name='add_two_ints_node.add_two_ints')node.run()# Create an instance of the request objectmsg=AddTwoIntMessage.Request()whileTrue:# returns AddTwoIntMessage.Response instanceresp=rpc.call(msg)print(resp)msg.a+=1msg.b+=1time.sleep(1)PubSub CommunicatonTraditional Topic-based Publish-Subscribe pattern for asynchronous communication as depicted below.An example of using PubSub communication is located atexamples/simple_pubsub.Write a Simple Topic Publisherfromcommlib.msgimportMessageHeader,PubSubMessagefromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersclassSonarMessage(PubSubMessage):header:MessageHeader=MessageHeader()range:float=-1hfov:float=30.6vfov:float=14.2classSonarMessage(PubSubMessage):distance:float=0.001horizontal_fov:float=30.0vertical_fov:float=14.0if__name__=="__main__":conn_params=ConnectionParameters(host='localhost',port=1883)node=Node(node_name='sensors.sonar.front',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)pub=node.create_publisher(msg_type=SonarMessage,topic='sensors.sonar.front')node.run()msg=SonarMessage()whileTrue:pub.publish(msg)msg.range+=1time.sleep(1)Write a Simple Topic Subscriber#!/usr/bin/env pythonimporttimefromcommlib.msgimportMessageHeader,PubSubMessagefromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersclassSonarMessage(PubSubMessage):header:MessageHeader=MessageHeader()range:float=-1hfov:float=30.6vfov:float=14.2defon_message(msg):print(f'Received front sonar data:{msg}')if__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='obstacle_avoidance_node',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)node.create_subscriber(msg_type=SonarMessage,topic='sensors.sonar.front',on_message=on_message)node.run_forever(sleep_rate=1)Pattern-based Topic SubscriptionFor pattern-based topic subscription one can also use thePSubscriberclass directly.For multi-topic publisher one can also use theMPublisherclass directly.#!/usr/bin/env python### Pattern-based Subscriber##fromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersdefon_message(msg,topic):print(f'Message at topic <{topic}>:{msg}')if__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='example5_listener',connection_params=conn_params,debug=True)node.create_psubscriber(topic='topic.*',on_message=on_message)node.run_forever()#!/usr/bin/env python### Multi-Topic Puiblisher##fromcommlib.nodeimportNodefromcommlib.transports.mqttimportConnectionParametersdefon_message(msg,topic):print(f'Message at topic <{topic}>:{msg}')if__name__=='__main__':conn_params=ConnectionParameters()node=Node(node_name='example5_publisher',connection_params=conn_params,debug=True)pub=node.create_mpublisher()node.run()topicA='topic.a'topicB='topic.b'whileTrue:pub.publish({'a':1},topicA)pub.publish({'b':1},topicB)time.sleep(1)Pythonic implementation of Subscribers and RPCs using decoratorsfromcommlib.msgimportMessageHeader,PubSubMessage,RPCMessagefromcommlib.nodeimportNode,TransportTypefromcommlib.transports.redisimportConnectionParametersclassSonarMessage(PubSubMessage):header:MessageHeader=MessageHeader()range:float=-1hfov:float=30.6vfov:float=14.2classAddTwoIntMessage(RPCMessage):classRequest(RPCMessage.Request):a:int=0b:int=0classResponse(RPCMessage.Response):c:int=0conn_params=ConnectionParameters()node=Node(node_name='obstacle_avoidance_node',connection_params=conn_params,debug=True)@node.subscribe('sensors.sonar.front',SonarMessage)defon_message(msg):print(f'Received front sonar data:{msg}')@node.rpc('add_two_ints_node.add_two_ints',AddTwoIntMessage)defadd_two_int_handler(msg):print(f'Request Message:{msg.__dict__}')resp=AddTwoIntMessage.Response(c=msg.a+msg.b)returnrespnode.run_forever(sleep_rate=0.01)Preemptable Services with Feedback (Actions)Actions arepre-emptable serviceswith support for asynchronous feedback publishing. This communication pattern
is used to implement services which can be stopped and can provide feedback data, such
as the move command service of a robot.Write an Action Serviceimporttimefromcommlib.actionimportGoalStatusfromcommlib.msgimportActionMessagefromcommlib.transports.redisimportConnectionParameters)classExampleAction(ActionMessage):classGoal(ActionMessage.Goal):target_cm:int=0classResult(ActionMessage.Result):dest_cm:int=0classFeedback(ActionMessage.Feedback):current_cm:int=0defon_goal(goal_h):c=0res=ExampleAction.Result()whilec<goal_h.data.target_cm:ifgoal_h.cancel_event.is_set():breakgoal_h.send_feedback(ExampleAction.Feedback(current_cm=c))c+=1time.sleep(1)res.dest_cm=creturnresif__name__=='__main__':action_name='testaction'conn_params=ConnectionParameters()node=Node(node_name='action_service_example_node',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)node.create_action(msg_type=ExampleAction,action_name=action_name,on_goal=on_goal)node.run_forever()Write an Action Clientimporttimefromcommlib.actionimportGoalStatusfromcommlib.msgimportActionMessagefromcommlib.transports.redisimportActionClient,ConnectionParametersclassExampleAction(ActionMessage):classGoal(ActionMessage.Goal):target_cm:int=0classResult(ActionMessage.Result):dest_cm:int=0classFeedback(ActionMessage.Feedback):current_cm:int=0defon_feedback(feedback):print(f'ActionClient <on-feedback> callback:{feedback}')defon_result(result):print(f'ActionClient <on-result> callback:{result}')defon_goal_reached(result):print(f'ActionClient <on-goal-reached> callback:{result}')if__name__=='__main__':action_name='testaction'conn_params=ConnectionParameters()node=Node(node_name='action_client_example_node',connection_params=conn_params,# heartbeat_uri='nodes.add_two_ints.heartbeat',debug=True)action_client=node.create_action_client(msg_type=ExampleAction,action_name=action_name,on_goal_reached=on_goal_reached,on_feedback=on_feedback,on_result=on_result)node.run()goal_msg=ExampleAction.Goal(target_cm=5)action_client.send_goal(goal_msg)resp=action_client.get_result(wait=True)print(f'Action Result:{resp}')node.stop()Broker-to-broker (B2B) bridgesIn the context of IoT and CPS, it is a common requirement to bridge messages
between message brokers, based on application-specific rules. An example is to
bridge analytics (preprocessed) data from the Edge to the Cloud. And what happens
if the brokers use different communication protocols?Below are examples of an MQTT Redis-to-MQTT Bridge and a Redis-to-MQTT
Topic Bridge.#!/usr/bin/env pythonimporttimeimportcommlib.transports.amqpasacommimportcommlib.transports.redisasrcommimportcommlib.transports.mqttasmcommfromcommlib.bridgesimport(RPCBridge,RPCBridgeType,TopicBridge,TopicBridgeType)defredis_to_mqtt_rpc_bridge():"""[RPC Client] ----> [Broker A] ------> [Broker B] ---> [RPC Service]"""bA_params=rcomm.ConnectionParameters()bB_params=mcomm.ConnectionParameters()bA_uri='ops.start_navigation'bB_uri='thing.robotA.ops.start_navigation'br=RPCBridge(RPCBridgeType.REDIS_TO_MQTT,from_uri=bA_uri,to_uri=bB_uri,from_broker_params=bA_params,to_broker_params=bB_params,debug=False)br.run()defredis_to_mqtt_topic_bridge():"""[Producer Endpoint] ---> [Broker A] ---> [Broker B] ---> [Consumer Endpoint]"""bA_params=rcomm.ConnectionParameters()bB_params=mcomm.ConnectionParameters()bA_uri='sonar.front'bB_uri='thing.robotA.sensors.sonar.font'br=TopicBridge(TopicBridgeType.REDIS_TO_MQTT,from_uri=bA_uri,to_uri=bB_uri,from_broker_params=bA_params,to_broker_params=bB_params,debug=False)br.run()if__name__=='__main__':redis_to_mqtt_rpc_bridge()redis_to_mqtt_topic_bridge()A Pattern-based Topic Bridge (PTopicBridge) example is also shown below.
In this example, we use static definition of messages (SonarMessage), also
referred astyped communication.#!/usr/bin/env pythonimporttimefromcommlib.msgimportPubSubMessagefromcommlib.bridgesimportPTopicBridgeimportcommlib.transports.amqpasacommimportcommlib.transports.redisasrcommclassSonarMessage(PubSubMessage):distance:float=0.001horizontal_fov:float=30.0vertical_fov:float=14.0if__name__=='__main__':"""[Broker A] ------------> [Broker B] ---> [Consumer Endpoint]"""bA_uri='sensors.*'bB_namespace='myrobot'bA_params=rcomm.ConnectionParameters()bB_params=mcomm.ConnectionParameters()br=PTopicBridge(TopicBridgeType.REDIS_TO_MQTT,bA_uri,bB_namespace,bA_params,bB_params,msg_type=SonarMessage,debug=False)br.run()Action bridgesTCP BridgeTCP bridge forwards tcp packages between two endpoints:[Client] -------> [TCPBridge, port=xxxx] ---------> [TCP endpoint, port=xxxx]A one-to-one connection is performed between the bridge and the endpoint.REST ProxyImplements a REST proxy, that enables invocation of REST services via
broker communication. The proxy uses an RPCService to run the broker endpoint and
an http client for calling REST services. An RPC call is transformed into proper,
REST-compliant, http request, based on the input parameters.classRESTProxyMessage(RPCMessage):classRequest(RPCMessage.Request):base_url:strpath:str='/'verb:str='GET'query_params:Dict={}path_params:Dict={}body_params:Dict={}headers:Dict={}classResponse(RPCMessage.Response):data:Union[str,Dict,int]headers:Dict[str,Any]status_code:int=200Responses from the REST services are returned to clients in the form of aRPCMessage.Responsemessage.TransportsAMQP / RabbitMQRPC (request/reply) and PubSub Endpoints are supported by the protocol itself (AMQP), using dedicated exchanges.For RPC enpoints aDirect Exchangeis used to route requests and responses,
optionally using theDirect Reply-to.
If theDirect Reply-tofeature is used, then RPC endpoints must publish
to the default exchange"".To useDirect Reply-to, an RPC client should:Consume from the pseudo-queueamq.rabbitmq.reply-toin no-ack mode.Set thereply-toproperty in their request message toamq.rabbitmq.reply-to.Meta-information such as the serialization method used, is passed through themessage propertiesmetadata, as specified my AMQP.RedisReq/Resp communication (RPC) is not supported out-of-the-box. To support
RPC communication over Redis, a custom layer implements the pattern for both endpoints
using Redis Lists to represent queues. RPC server listens for requests from
a list (LPOP / BLPOP), while an RPC client sends request messages to that list (RPUSH).
In order for the client to be able to receive responses, he must listen to a temporary queue.
To achieve this, the request message must include areply_toproperty that is
used by the RPCServer implementation to send the response message. Furthermore,serializationandencodingproperties are defined. Finally, the header
includes atimestamp, that indicates the time that the message was sent to to wire.Below is the  data model of the request message.{
  'data': {},
  'header': {
    'timestamp': <int>,
    'reply_to': <str>,
    'content_type': 'application/json',
    'content_encoding': 'utf8',
    'agent': 'commlib'
  }
}Note: TheRPC Clientimplementation is responsible to remove any created
temporary queues!MQTTReq/Resp communication (RPC) is not supported out-of-the-box. To support
RPC communication over MQTT, a custom layer implements the pattern for both endpoints
using MQTT topics. RPC server listens for requests at a specific topic,
while an RPC client listens to a temporary topic for response messages.
For the server to know where to send the response, the request message must include areply_toproperty that is
used by the RPCServer implementation to send the response message. Furthermore,serializationandencodingproperties are defined. Finally, the header
includes atimestamp, that indicates the time that the message was sent to to wire.Below is the data model of the Request message.{
  'data': {},
  'header': {
    'timestamp': -1,
    'reply_to': "UNIQUE_NAME",
    'content_type': 'application/json',
    'content_encoding': 'utf8',
    'agent': 'commlib',
  }
}ExamplesExamples can be found at theexamples/directory of this repository.TestsRun tests by executingtoxcommand under this repo directory:maketestsTODO - Currently working on!!DocsMake docs by running:makedocsTODO - Currently working on!!Contributionsklpanagi