Package: yagi
# YagiA modular OpenStack notification event processor/broadcaster written in Python.Yagi is designed to efficiently gather amqp messages in the json formatused by OpenStack projects notification busses, from a large andconfigurable number of queues, and proccess them through an extensibleset of simple handlers.Handlers are simple to write, and can be chained in a WSGI-like architecture.Yagi handles fetching messages, a batch at a time, and passes fetchedmessages to each handler, handling AMQP message semantics so the handlerscan concentrate on the task at hand.In addition, a feed daemon is included that can generate a paged Atom feedof notification events that have been persisted in a datastore.## Available Handlers* AtomPub: Formats notifications in Atom format, and pushes them to afeed server using the AtomPub protocol. Useful with feedservers such as AtomHopper (http://atomhopper.org/)* Redis:   Persists notifications to a Redis database. Can be used withYagi\'s feed daemon.* PubSubHubub: Pings a pubsubhubub hub when notifications arrive.Together with a hub, and yagi\'s feed daemon, this canenable publish/subscribe subscriptions to notification events.* StackTackPing: Works with the StackTach openstack monitoring tool tomonitor event feeds. If you are using Yagi to providefeeds of openstack notifications, this will pingstacktach when those feeds are updated, informing itof the success or failure of the updates, letting youcatch if the feed server is down, or some system isdropping events.## Installation and runningThe current version of Yagi can be fetched from the code repositoryat: https://github.com/rackerlabs/yagicd to the yagi directory and run:sudo python setup.py installThe launch the yagi process:yagi-eventAn altername config file may be passed to yagi like this:yagi-event -c /path/to/config/fileYagi does not daemonize. use your favorite daemon manager to do that.## ConfigurationA sample yagi.conf can be found in the etc directory.Sections to note:* rabbit_broker: Your rabbit connection info goes here.* event_feed: If using the feed daemon, remember to set the feed_host tothe name of the host it is running on. This allows yagito correctly construct links in the feed.* persistence: If using the redis handler, put your reddisc connectioninfo here.* consumers: the 'queues' config variable lists the queues yagi shouldlisten on.* consumer:$queue_name: For each queue Yagi is listening on there shouldbe a consumer section in the config file(for example if you have a queue namedsome.queue listed in the [consumers] section,there should be a [consumer:some.queue]section with configuration for that specific queue.)This should list propertiesfor the queue, such as if itshould be durable.Important variables here are 'apps',which is a comma separated list of handlers thatmessages from that queues should bepassed to, and 'max_messages' which is the maximumnumber of messages that Yagi will pull from thatqueue at one time. (it will then go to the next queue,eventually coming back around, if there are stillmessages waiting)Handlers may also have their own, additional configuration.This is usually found in a section named after the handler (alllowercase, one word)## ScalingYagi is designed to scale by running multiple processes. Simplylaunch as many yagi-event processes asyou need to handle your load. (yagi-event is fairly lightweight)## Dependencies:* anyjson* argparse* feedgenerator* httplib2* requests (eventually httplib2 will be replaced with requests)* redis (if using redis handler)* webob* eventlet* python-dateutil* daemon* pubsubhubbub_publish (if using pubsubhubub handler) (available underthe publisher_clients folder after checking out the project from[Google Code](http://code.google.com/p/pubsubhubbub/source/checkout)NOTE: the plan is to replace this dependency later with our implementation* carrot (if using Rabbit)* routes## Running the feed daemonSimply run:yagi-feed## Setting up a hub (if using PubSubHubub)Download the Google App Engine SDK for Linux and add it to your pathhttp://code.google.com/appengine/downloads.htmlThen checkout the reference hub.svn checkout http://pubsubhubbub.googlecode.com/svn/trunk/ pubsubhubbub-read-onlyInstall pubsubhubbub_publisher for pythoncd pubsubhubbub-read-only/publisher_clients/pythonsudo python setup.py installStart the hubcd pubsubhubbub-read-onlydev_appserver.py hub/ -p<port number specified in yagi.conf>## Testing subscriptions for PubSubHubub.cd yagi# You'll want to run this in multiple screen windows or terminal# sessions, as the callback process won't daemonizepython subscriber/callback.py <sub_port>python subscriber/sub.py <topic> <callback> <hub># I usually load other/push_rabbit.py in an iPython sessioncd otheripythonimport push_rabbit# the cast below is assuming you setting up yagi to listen# on a queue named 'notifications.warn'push_rabbit.cast(dict(a=3), 'instance', 'notifications', 'warn')You should see XML content being pushed to your window runningcallback.py, above.## Brokers, Consumers and Handlers(Apps)The top-level object for Yagi is the Event Broker, which is defined in theconfiguration file as:[event_worker]event_driver = yagi.broker.rabbit.BrokerThe Broker will create a Consumer object for each input queue defined.[consumers]queues = queue1, queue2[consumer:queue1]apps = yagi.handler.handler1, myhandlers.handlerrouting_key = notifications.warn[consumer:queue2]apps = ...The Consumer grabs the notifications as they come in and give them to thehandlers for processing. Handlers are also known as apps for legacy reasons.Handlers are chained together, and it's the responability of the handlerto call child handlers if they think it's appropriate (the default behavior).However, some handler may wish to kill the chain on an error some othercondition. Each handler also gets to process the messages themself. Firstnotifications are checked against the filter list for that handler and, ifaccepted, the `handle_messages()` method is called. Child handlers always getcalled unfiltered.[filters]cufpub = compute.instance.exists.verified,compute.instance.existsCurrently, filters are applied to all handlers, but this should change toa per-handler filter list.Look at `yagi.handlers.__init__.py` for details.