Package: twisted-mqtt
MQTT Client protocol for Twisted.Descriptiontwisted-mqttis a library using the Twisted framework and
implementing the MQTT protocol (v3.1 & v3.1.1) in these flavours:pure subscriberpure publisheror a mixing of both. This is useful to subscribe and publish through
the same broker using only one TCP connection.InstalationJust type:sudo pip installtwisted-mqttor from GitHub:git clone https://github.com/astrorafael/twisted-mqtt.git
cd twisted-mqtt
sudo python setup.py installCreditsI started writting this software after findingAdam Rudd’s MQTT.py
code. A small
part his code is still there. However, I soon began taking my own
direction both in design and scope.Function/methods docstrings contain quotes of the OASISmqtt-v3.1.1standard.MQTT Version 3.1.1. Edited by Andrew Banks and Rahul Gupta. 29 October
2014. OASIS Standard.http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html.
Latest version:http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html.UsageThe APIs are described in thelibrary defined
interfacesThis library buildsMQTTProtocolobjects and is designed to beused
rather than inherited.ExamplesThese examples show my library intended usage: managed by a service.
Your Twisted application should probably be designed as a collection of
services and one of these would be an MQTT Service. Note that a service
is simply an object that can be started bystartService()and
stopped bystopService().Probably you also want your service to handle automatic reconnections to
the MQTT broker and that’s where Twisted’sClientServiceclass comes
in. AClientServiceinstance detects its transport has been closed
and re-opens the connection to the MQTT Broker.However, this is not enough for the MQTT protocol since the broker
expects a CONNECT packet request shortly after the socket has been
opened. For this reason, we must subclassClientServiceto overridestartService(). Also we will add some MQTT connection/disconnection
handling code. This requires us to obtain somehow the protocol instance
built by the factory.In the startup code, we create aClientServiceinstance, passing the
proper MQTT protocol factory and we simply start the service. InsidestartService()we invoke ClientService’s methodwhenConnected()that returns aDeferred. ThisDeferred- when fired - will
invoke a user function with the protocol object been created as the
parameter.Our customClientServicesubclass defines aconnectToBroker()method, receiving the protocol object just built. At minimun, we will
store a reference to this protocol for further reference. If we wish to
handle automatic reconnections, we should set the MQTT protocolonDisconnectionattribute to a callback that will handle what to do
in such cases. Our serviceonDisconnection()callback will simple
tell us to rebuild a new protocol instance and callconnectToBroker()again when done. In this way, we start the whole
MQTT CONNECT thing all over again.Finally, our customClientServiceexample subclass may define a
custom retry policy by customizingbackoffPolicy()default argumentsinitialDelay,maxDelayandfactor. See thetwisted.application.internet.backoffPolicy()API reference for
further details.Publisher ExampleA publisher is built by obtaining a factory for theMQTTFactory.PUBLISHERprofile.Your MQTT Publisher service should configure a couple of things in theconnectToBroker()method:The MQTT protocolonDisconnectionattribute storing a callback
that will be invoked when a disconnection occurs.The maximun Window Size - that is - how many asynchronous PUBLISH
request you will issue in a row to the library, before getting and
acknowledge from the Broker (Qos=1 and 2 only). By thefault, the
window size is 1 and this guarantees in-order delivery of published
messages.This example additionally starts a periodic task to publish sample data.importsysfromtwisted.internetimportreactor,taskfromtwisted.internet.deferimportinlineCallbacks,DeferredListfromtwisted.application.internetimportClientService,backoffPolicyfromtwisted.internet.endpointsimportclientFromStringfromtwisted.loggerimport(Logger,LogLevel,globalLogBeginner,textFileLogObserver,FilteringLogObserver,LogLevelFilterPredicate)frommqtt.client.factoryimportMQTTFactory# ----------------# Global variables# ----------------# Global object to control globally namespace logginglogLevelFilterPredicate=LogLevelFilterPredicate(defaultLogLevel=LogLevel.info)BROKER="tcp:test.mosquitto.org:1883"# -----------------# Utility Functions# -----------------defstartLogging(console=True,filepath=None):'''
    Starts the global Twisted logger subsystem with maybe
    stdout and/or a file specified in the config file
    '''globallogLevelFilterPredicateobservers=[]ifconsole:observers.append(FilteringLogObserver(observer=textFileLogObserver(sys.stdout),predicates=[logLevelFilterPredicate]))iffilepathisnotNoneandfilepath!="":observers.append(FilteringLogObserver(observer=textFileLogObserver(open(filepath,'a')),predicates=[logLevelFilterPredicate]))globalLogBeginner.beginLoggingTo(observers)defsetLogLevel(namespace=None,levelStr='info'):'''
    Set a new log level for a given namespace
    LevelStr is: 'critical', 'error', 'warn', 'info', 'debug'
    '''level=LogLevel.levelWithName(levelStr)logLevelFilterPredicate.setLogLevelForNamespace(namespace=namespace,level=level)# -----------------------# MQTT Publishing Service# -----------------------classMQTTService(ClientService):def__init(self,endpoint,factory):ClientService.__init__(self,endpoint,factory,retryPolicy=backoffPolicy())defstartService(self):log.info("starting MQTT Client Publisher Service")# invoke whenConnected() inherited methodself.whenConnected().addCallback(self.connectToBroker)ClientService.startService(self)@inlineCallbacksdefconnectToBroker(self,protocol):'''
        Connect to MQTT broker
        '''self.protocol=protocolself.protocol.onDisconnection=self.onDisconnection# We are issuing 3 publish in a row# if order matters, then set window size to 1# Publish requests beyond window size are enqueuedself.protocol.setWindowSize(3)self.task=task.LoopingCall(self.publish)self.task.start(5.0)try:yieldself.protocol.connect("TwistedMQTT-pub",keepalive=60)exceptExceptionase:log.error("Connecting to{broker}raised{excp!s}",broker=BROKER,excp=e)else:log.info("Connected to{broker}",broker=BROKER)defonDisconnection(self,reason):'''
        get notfied of disconnections
        and get a deferred for a new protocol object (next retry)
        '''log.debug(" >< Connection was lost ! ><, reason={r}",r=reason)self.whenConnected().addCallback(self.connectToBroker)defpublish(self):def_logFailure(failure):log.debug("reported{message}",message=failure.getErrorMessage())returnfailuredef_logAll(*args):log.debug("all publihing complete args={args!r}",args=args)log.debug(" >< Starting one round of publishing >< ")d1=self.protocol.publish(topic="foo/bar/baz1",qos=0,message="hello world 0")d1.addErrback(_logFailure)d2=self.protocol.publish(topic="foo/bar/baz2",qos=1,message="hello world 1")d2.addErrback(_logFailure)d3=self.protocol.publish(topic="foo/bar/baz3",qos=2,message="hello world 2")d3.addErrback(_logFailure)dlist=DeferredList([d1,d2,d3],consumeErrors=True)dlist.addCallback(_logAll)returndlistif__name__=='__main__':importsyslog=Logger()startLogging()setLogLevel(namespace='mqtt',levelStr='debug')setLogLevel(namespace='__main__',levelStr='debug')factory=MQTTFactory(profile=MQTTFactory.PUBLISHER)myEndpoint=clientFromString(reactor,BROKER)serv=MQTTService(myEndpoint,factory)serv.startService()reactor.run()Subscriber ExampleA subscriber is built by obtaining a factory for theMQTTFactory.SUBSCRIBERprofile.Your MQTT Subscriber service should configure the following things in
theconnectToBroker()method:The MQTT protocolonDisconnectionattribute storing a callback
that will be invoked when a disconnection occurs.The maximun Window Size - that is - how many asynchronous SUBSCRIBE
or UNSUBSCRIBE request you will issue in a row to the library, before
getting and acknowledge from the Broker.The MQTT protocolonPublishattribute storing a callback that
will be fired whenever a new PUBLISH packed is delivered to the
subscriber.importsysfromtwisted.internet.deferimportinlineCallbacks,DeferredListfromtwisted.internetimportreactorfromtwisted.internet.endpointsimportclientFromStringfromtwisted.application.internetimportClientService,backoffPolicyfromtwisted.loggerimport(Logger,LogLevel,globalLogBeginner,textFileLogObserver,FilteringLogObserver,LogLevelFilterPredicate)frommqtt.client.factoryimportMQTTFactory# ----------------# Global variables# ----------------# Global object to control globally namespace logginglogLevelFilterPredicate=LogLevelFilterPredicate(defaultLogLevel=LogLevel.info)BROKER="tcp:test.mosquitto.org:1883"# -----------------# Utility Functions# -----------------defstartLogging(console=True,filepath=None):'''
    Starts the global Twisted logger subsystem with maybe
    stdout and/or a file specified in the config file
    '''globallogLevelFilterPredicateobservers=[]ifconsole:observers.append(FilteringLogObserver(observer=textFileLogObserver(sys.stdout),predicates=[logLevelFilterPredicate]))iffilepathisnotNoneandfilepath!="":observers.append(FilteringLogObserver(observer=textFileLogObserver(open(filepath,'a')),predicates=[logLevelFilterPredicate]))globalLogBeginner.beginLoggingTo(observers)defsetLogLevel(namespace=None,levelStr='info'):'''
    Set a new log level for a given namespace
    LevelStr is: 'critical', 'error', 'warn', 'info', 'debug'
    '''level=LogLevel.levelWithName(levelStr)logLevelFilterPredicate.setLogLevelForNamespace(namespace=namespace,level=level)# -----------------------# MQTT Subscriber Service# ------------------------classMQTTService(ClientService):def__init(self,endpoint,factory):ClientService.__init__(self,endpoint,factory,retryPolicy=backoffPolicy())defstartService(self):log.info("starting MQTT Client Subscriber Service")# invoke whenConnected() inherited methodself.whenConnected().addCallback(self.connectToBroker)ClientService.startService(self)@inlineCallbacksdefconnectToBroker(self,protocol):'''
        Connect to MQTT broker
        '''self.protocol=protocolself.protocol.onPublish=self.onPublishself.protocol.onDisconnection=self.onDisconnectionself.protocol.setWindowSize(3)try:yieldself.protocol.connect("TwistedMQTT-subs",keepalive=60)yieldself.subscribe()exceptExceptionase:log.error("Connecting to{broker}raised{excp!s}",broker=BROKER,excp=e)else:log.info("Connected and subscribed to{broker}",broker=BROKER)defsubscribe(self):def_logFailure(failure):log.debug("reported{message}",message=failure.getErrorMessage())returnfailuredef_logGrantedQoS(value):log.debug("response{value!r}",value=value)returnTruedef_logAll(*args):log.debug("all subscriptions complete args={args!r}",args=args)d1=self.protocol.subscribe("foo/bar/baz1",2)d1.addCallbacks(_logGrantedQoS,_logFailure)d2=self.protocol.subscribe("foo/bar/baz2",2)d2.addCallbacks(_logGrantedQoS,_logFailure)d3=self.protocol.subscribe("foo/bar/baz3",2)d3.addCallbacks(_logGrantedQoS,_logFailure)dlist=DeferredList([d1,d2,d3],consumeErrors=True)dlist.addCallback(_logAll)returndlistdefonPublish(self,topic,payload,qos,dup,retain,msgId):'''
        Callback Receiving messages from publisher
        '''log.debug("msg={payload}",payload=payload)defonDisconnection(self,reason):'''
        get notfied of disconnections
        and get a deferred for a new protocol object (next retry)
        '''log.debug(" >< Connection was lost ! ><, reason={r}",r=reason)self.whenConnected().addCallback(self.connectToBroker)if__name__=='__main__':importsyslog=Logger()startLogging()setLogLevel(namespace='mqtt',levelStr='debug')setLogLevel(namespace='__main__',levelStr='debug')factory=MQTTFactory(profile=MQTTFactory.SUBSCRIBER)myEndpoint=clientFromString(reactor,BROKER)serv=MQTTService(myEndpoint,factory)serv.startService()reactor.run()Publisher/Subscriber ExampleA Publisher/Subscriber example is no more than a mix of the previous
examples, not forgetting to set the MQTT factory profile toMQTTFactory.PUBLISHER | MQTTFactory.SUBSCRIBER.Design NotesThere is a separateMQTTProtocolin each module implementing a
different profile (subscriber, publiser, publisher/subscriber). TheMQTTBaseProtocoland the variousMQTTProtocolclasses implement
a State Pattern to avoid the “if spaghetti code” in the connection
states. A basic state machine is built into theMQTTBaseProtocoland
theConnectedStateis patched according to the profile.Previous 0.1.x implementations used two separate (subclases, publisher)
and with separate logic for both roles. The publisher/subscriber was a
mixin class implemented by delegation that managed the connection state
and forwarded all client requests and network events to the proper
delegate.However, this approach had some quirks and issues with sharing state. It
has been re-written to a single publisher/subscriber class that manages
everything.To maintain the former API, separate subclasses has been derived to
implement a pure subscriber or publisher roles. The subclassing simply
patches the state machine in order to honor only the methods for a given
role.LimitationsThe current implementation has the following limitations:This library does not claim to be full comformant to the standard.There is a limited form of session persistance for the publisher.
Pending acknowledges for PUBLISH and PUBREL are kept in RAM and
outlive the connection and the protocol object while Twisted is
running. However, they are not stored in a persistent medium.TODOI wrote this library for my pet projects and learn Twisted. However, it
goes a long way from an apparently looking good library to an
industrial-strength, polished product. I don’t simply have the time,
energy and knowledge to do so.Some areas in which this can be improved:Bug fixingInclude a thorough test battery.Improve documentation.etc.