Package: zmq-tubes
ZMQ TubesZMQ Tubes is a managing system for ZMQ communication.
It can manage many ZMQ sockets by one interface.
The whole system is hierarchical, based on topics
(look atMQTT topics).ClassesTubeMessage- This class represents a request/response message.
Some types of tubes require a response in this format.Tube- This class wraps a ZMQ socket.
It represents a connection between client and server.TubeMonitor- The class can sniff of the ZMQTube communication.TubeNode- This represents an application interface for communication via tubes.Asyncio / ThreadingThe library support bot method. Asyncio from Python 3.7.fromzmq_tubesimportTubeNode,Tube# Asyncio classesfromzmq_tubes.threadsimportTubeNode,Tube# Threads classesUsage:Node definitions in yml fileWe can define all tubes for one TubeNode by yml file.
Next examples require install these packagesPyYAML,pyzmqandzmq_tubes.Client service (asyncio example)# client.ymltubes:-name:Client REQaddr:ipc:///tmp/req.pipetube_type:REQtopics:-foo/bar-name:Client PUBaddr:ipc:///tmp/pub.pipetube_type:PUBtopics:-foo/pub/## client.pyimportasyncioimportyamlfromzmq_tubesimportTubeNode,TubeMessageasyncdefrun():withopen('client.yml','r+')asfd:schema=yaml.safe_load(fd)node=TubeNode(schema=schema)asyncwithnode:print(awaitnode.request('foo/bar','message 1'))awaitnode.publish('foo/pub/test','message 2')if__name__=="__main__":asyncio.run(run())>pythonclient.py
topic:foo/bar,payload:responseServer service (threads example)# server.ymltubes:-name:server ROUTERaddr:ipc:///tmp/req.pipetube_type:ROUTERserver:Truetopics:-foo/bar-name:server SUBaddr:ipc:///tmp/pub.pipetube_type:SUBserver:Truetopics:-foo/pub/## server.pyimportyamlfromzmq_tubes.threadsimportTubeNode,TubeMessagedefhandler(request:TubeMessage):print(request.payload)ifrequest.tube.tube_type_name=='ROUTER':returnrequest.create_response('response')defrun():withopen('server.yml','r+')asfd:schema=yaml.safe_load(fd)node=TubeNode(schema=schema)node.register_handler('foo/#',handler)withnode:node.start().join()if__name__=="__main__":run()>pythonserver.py
message1message2YAML definitionThe yaml file starts with a root elementtubes, which contains list of all our tube definitions.name- string - name of the tube.addr- string - connection or bind address in formattransport://address(see morehttp://api.zeromq.org/2-1:zmq-connect)server- bool - is this tube server side (bind toaddr) or client side (connect toaddr)tube_type- string - type of this tube (see morehttps://zguide.zeromq.org/docs/chapter2/#Messaging-Patterns)identity- string - (optional) we can setup custom tube identityutf8_decoding- bool - (default = True), if this is True, the payload is automatically UTF8 decode.sockopts- dict - (optional) we can setup sockopts for this tube (see morehttp://api.zeromq.org/4-2:zmq-setsockopt)monitor- string - (optional) bind address of tube monitor (see moreDebugging / Monitoring)Request / ResponseThis is a simple scenario, the server processes the requests serially.Server:fromzmq_tubesimportTube,TubeNode,TubeMessageasyncdefhandler(request:TubeMessage):print(request.payload)return'answer'# or return request.create_response('response')tube=Tube(name='Server',addr='ipc:///tmp/req_resp.pipe',server=True,tube_type='REP')node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.start()# output: 'question'Client:fromzmq_tubesimportTube,TubeNodetube=Tube(name='Client',addr='ipc:///tmp/req_resp.pipe',tube_type='REQ')node=TubeNode()node.register_tube(tube,'test/#')response=awaitnode.request('test/xxx','question')print(response.payload)# output: 'answer'The methodrequestaccepts the optional parameterutf8_decoding. When we set this parameter toFalsein previous
example, the returned payload is not automatically decoded, we get bytes.Subscribe / PublisherServer:fromzmq_tubesimportTube,TubeNode,TubeMessageasyncdefhandler(request:TubeMessage):print(request.payload)tube=Tube(name='Server',addr='ipc:///tmp/sub_pub.pipe',server=True,tube_type='SUB')node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.start()# output: 'message'Client:fromzmq_tubesimportTube,TubeNodetube=Tube(name='Client',addr='ipc:///tmp/sub_pub.pipe',tube_type='PUB')# In the case of publishing, the first message is very often# lost. The workaround is to connect the tube manually as soon as possible.tube.connect()node=TubeNode()node.register_tube(tube,'test/#')node.publish('test/xxx','message')Request / RouterThe server is asynchronous. It means it is able to process
more requests at the same time.Server:importasynciofromzmq_tubesimportTube,TubeNode,TubeMessageasyncdefhandler(request:TubeMessage):print(request.payload)ifrequest.payload=='wait':awaitasyncio.sleep(10)returnrequest.create_response(request.payload)tube=Tube(name='Server',addr='ipc:///tmp/req_router.pipe',server=True,tube_type='ROUTER')node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.start()# output: 'wait'# output: 'message'Client:importasynciofromzmq_tubesimportTube,TubeNodetube=Tube(name='Client',addr='ipc:///tmp/req_router.pipe',tube_type='REQ')asyncdeftask(node,text):print(awaitnode.request('test/xxx',text))node=TubeNode()node.register_tube(tube,'test/#')asyncio.create_task(task(node,'wait'))asyncio.create_task(task(node,'message'))# output: 'message'# output: 'wait'Dealer / ResponseThe client is asynchronous. It means it is able to send
more requests at the same time.Server:fromzmq_tubesimportTube,TubeNode,TubeMessageasyncdefhandler(request:TubeMessage):print(request.payload)return'response'# or return requset.create_response('response')tube=Tube(name='Server',addr='ipc:///tmp/dealer_resp.pipe',server=True,tube_type='REP')node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.start()# output: 'message'Client:fromzmq_tubesimportTube,TubeNode,TubeMessagetube=Tube(name='Client',addr='ipc:///tmp/dealer_resp.pipe',tube_type='DEALER')asyncdefhandler(response:TubeMessage):print(response.payload)node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.send('test/xxx','message')# output: 'response'Dealer / RouterThe client and server are asynchronous. It means it is able to send and process
more requests/responses at the same time.Server:importasynciofromzmq_tubesimportTube,TubeNode,TubeMessageasyncdefhandler(request:TubeMessage):print(request.payload)ifrequest.payload=='wait':awaitasyncio.sleep(10)returnrequest.create_response(request.payload)tube=Tube(name='Server',addr='ipc:///tmp/dealer_router.pipe',server=True,tube_type='ROUTER')node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.start()# output: 'wait'# output: 'message'Client:fromzmq_tubesimportTube,TubeNode,TubeMessagetube=Tube(name='Client',addr='ipc:///tmp/dealer_router.pipe',tube_type='DEALER')asyncdefhandler(response:TubeMessage):print(response.payload)node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.send('test/xxx','wait')awaitnode.send('test/xxx','message')# output: 'message'# output: 'wait'Dealer / DealerThe client and server are asynchronous. It means it is able to send and process
more requests/responses at the same time.Server:fromzmq_tubesimportTube,TubeNode,TubeMessagetube=Tube(name='Server',addr='ipc:///tmp/dealer_dealer.pipe',server=True,tube_type='DEALER')asyncdefhandler(response:TubeMessage):print(response.payload)node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.send('test/xxx','message from server')# output: 'message from client'Client:fromzmq_tubesimportTube,TubeNode,TubeMessagetube=Tube(name='Client',addr='ipc:///tmp/dealer_dealer.pipe',tube_type='DEALER')asyncdefhandler(response:TubeMessage):print(response.payload)node=TubeNode()node.register_tube(tube,'test/#')node.register_handler('test/#',handler)awaitnode.send('test/xxx','message from client')# output: 'message from server'Debugging / MonitoringWe can assign a monitor socket to our zmq tubes. By this monitor socket, we can sniff zmq communication or get a zmq tube
configuration.tubes:-name:ServerRouteraddr:ipc:///tmp/router.pipemonitor:ipc:///tmp/test.monitortube_type:ROUTERserver:yestopics:-foo/#This is example of a yaml definition. We can use the same monitor socket for more tubes in the same tubeNode.
When we add the monitor attribute to our tube definition, the application automatically create a new socket monitor:/tmp/test.monitor. Your application works as a server side. The logs are sent to the socket only for the time, when the monitoring
tool is running.Monitoring toolAfter enabling of the monitoring in the application, we can use the monitoring tool for sniff.# get the server tube configuration>zmqtube-monitorget_schemaipc:///tmp/display.monitortubes:-addr:ipc:///tmp/router.pipemonitor:ipc:///tmp/test.monitorname:ServerRouterserver:'yes'tube_type:ROUTER# the log tube communication. Logs will be saved to dump.rec as well.>zmqtube-monitorlogs-d./dump.recipc:///tmp/display.monitor0.28026580810546875ServerRouter<foo/testRequest0.0901789665222168ServerRouter>foo/testResponse# The format of output# <relative time> <tube name> <direction> <topic> <message>`Simulation of the client sideWhen we have a dump file (e.g.dump.rec), we can simulate the communication with our app.
The first step is prepare the mock client schema file.
For this, We can get the tube node configuration from our application and after that edit it.>zmqtube-monitorget_schemaipc:///tmp/display.monitor>mock_schema.yaml
>vimmock_schema.yaml
...# Now, we have to update the file mock_schema.yaml.# We change configuration to the mock client configuration.# The names of the tubes must be the same as are in your app.# We can remove monitoring attribute and change server and# tube_type attributes. In this mock file, the topics are not# required, they are ignored.>catmock_schema.yaml
tubes:
-addr:ipc:///tmp/router.pipename:ServerRoutertube_type:REQNow, we can start the simulation of the client communication.>zmqtube-monitorsimulatemock_schema.yamldump.recIf the response of our app is not the same as tool expects (the response saved in dump file), then
the monitoring tool warns us.We can modify speed of the simulation by the parameter--speed.In the default configuration, is  the simulation run the same
speed as original communication (parameter--speed=1).Speeddescription0no blocking simulation0.5twice faster than original1original speed2twice slower than originalExample of programming declaration of the monitoring.importzmqfromzmq_tubes.threadsimportTube,TubeNode,TubeMessage,TubeMonitordefhandler(request:TubeMessage):print(request.payload)returnrequest.create_response('response')resp_tube=Tube(name='REP',addr='ipc:///tmp/rep.pipe',server='yes',tube_type=zmq.REP)req_tube=Tube(name='REQ',addr='ipc:///tmp/rep.pipe',tube_type=zmq.REQ)node=TubeNode()node.register_tube(resp_tube,f"foo/#")node.register_tube(req_tube,f"foo/#")node.register_handler(f"foo/#",handler)node.register_monitor(resp_tube,TubeMonitor(addr='ipc:///tmp/test.monitor'))withnode:print(node.request('foo/xxx','message 2'))