Package: amqp-dispatcher
A daemon to run AMQP consumersRequirementsPython 2.6+InstallationUsing PIP:From Github:pip install git+git://github.com/opschops/amqp-dispatcher.git@0.11.0#egg=amqp-dispatcherFrom PyPI:pip install amqp-dispatcher==0.11.0Runningamqp-dispatcher--configamqp-dispatcher-config.ymlThe environment variableRABBITMQ_URLcan also be used which will cause
attempt to connect to the defined data source name. Hosts are separated
via commas, and they are connected to in random order.Note that you can specify a heartbeat via theheartbeatquerystring value
on theRABBITMQ_URL. By default, it is set toNone, which ensures that
the client respects the broker specified heartbeat settings. You may wish to
override this for a particular environment.Additionally, you can specifically set a heartbeat override via theRABBITMQ_HEARTBEATenvironment variable.
This will take precedence over the heartbeat set inRABBITMQ_URL.To see an example run:makeexampleConsumersConsumers are a class with 2 required methods:consumeandshutdown.
amqp-dispatcher will not monkey patch the environment, you will have to do
that yourself.consume:consumeis called once for each message being handled. It should take 2 parameters, a proxy for AMQP operations (amqp) and the message (msg).shutdown-shutdownis called before the instance of the consumer is removed. It takes a single argumentexceptionwhich may beNone. If your consumer raises an exception while consuming theshutdownmethod will be called. Onceshutdownis finished a new instance of your consumer will be created to replace the one that raised the exception. If you would like to rate limit instance replacement you can callgevent.sleep(X)to sleep forXseconds after a failure.Example consumer:classConsumer(object):def__init__(self):self.init_msg="I've been initiliazed"defconsume(self,amqp,msg):print'Consuming message',msg.bodygevent.sleep(1)val=random.random()ifval>.8:print'publishing'amqp.publish('test_exchange','test_routing_key',{},'New body!')ifval<.5:raiseValueError()print'Done sleeping'amqp.ack()defshutdown(self,exception=None):print'Shut down'Configurationamqp-dispatcher will read environment variable for connection information and a
YAML file for worker configuration.You can validate the yaml file configuration with the following command:amqp-dispatcher--validate--configamqp-dispatcher-config.ymlThis will validate that thestartup_handlerandconsumersexist and can be
imported. Note that if there is any logic contained outside of those functions, said
logic will be executed.Environment VariablesRABBITMQ_URL: Connection string of the formamqp://USER:PASS@HOST:PORT/VHOSTStartup ConfigurationIf you need to perform custom actions (configure your logging, create initial objects) you can add a startup handler.This is configured in the config yml with thestartup_handleroption.startup_handler:amqpdispatcher.example_startup:startupQueue configurationQueues can be created on the fly by amqp dispatcher, and may bind existing exchanges on the fly as well.There are a few obvious constraints:To create a non-passive queue (typical behavior) the current user must haveconfigure=queuepermissionTo bind to an exchange, the current user must havereadpermission on the binding exchangeQueue configuration is as follows:queue: (required) name of the queuedurable: (optional) queue created in “durable” mode (default = True)auto_delete: (optional) queue created in “auto_delete” mode (default = False), meaning it will be deleted automatically once all consumers disconnect from it (e.g. on restart)exclusive: (optional) queue created in “exclusive” mode (default = False) meaning it will only be accessible by this processx_dead_letter_exchange: (optional) name of dead letter exchangex_dead_letter_routing_key: (optional) dead letter routing keyx_max_length: (optional) maximum length of ready messages. (default = INFINITE)x_expires: (optional) How long a queue can be unused for before it is automatically deleted (milliseconds) (default=INFINITE)x_message_ttl: (optional) How long a message published to a queue can live before it is discarded (milliseconds) (default=INFINITE)Bindingsbindingsshould contain a list ofexchange/routing_keypairs and defines the binding for the queue (there can be multiple)A complete configuration example would look like:queues:-queue:notify_mat_jobdurable:trueauto_delete:falsepassive:trueexclusive:falsex_dead_letter_exchange:nullx_dead_letter_routing_key:nullx_max_length:nullx_expires:nullx_message_ttl:nullbindings:-exchange:notifyrouting_key:transaction.*-exchange:notifyrouting_key:click.*-queue:notify_apsalar_jobbindings:-exchange:notifyrouting_key:transaction.*-exchange:notifyrouting_key:click.*Worker configurationWorkers are autoloaded when AMQP Dispatcher starts. This means your worker must
be importable from the environment.A complete configuration example would look like:consumers:-consumer:workers.module:Consumerconsumer_count:1queue:test_queueprefetch_count:2-consumer:workers.module_2:Consumerconsumer_count:2queue:test_queue_2prefetch_count:10prefetch_countis the AMQPprefetch_countwhen consuming. Theconsumer_countis the number of instances of your consumer to handle messages
from that queue.  Connection pools are highly recommended. MySQL will require theMySQL Connectorinstead ofmysqldbin order for gevent to switch properly.Pools can be created and attached to the consumer class during the__init__. Example with SQLAlchemyclassConsumer(object):session_maker=Nonedef__init__(self):self.session=NoneifConsumer._engineisNone:print'Creating session maker'Consumer._engine=create_engine(...)Consumer.sessionmaker=sessionmaker(bind=Consumer._engine)And then a session created during the consume method.defconsume(self,proxy,msg):session=self.sessionmaker()# Do something with the sessionsession.close()LoggingLogging is performed on the loggeramqp-dispatcher. The RabbitMQ connection
provided by Haigha will log onamqp-dispatcher.haigha.You can also configure the logger by using theLOGGING_FILE_CONFIGenvironment variable to specify a file config path. This will be used bylogging.config.fileConfigbefore creating the initial logger.