Package: twitcher
=================== Twitcher ===================1. License2. Overview3. Installation4. Configuration Language4.1 Examples4.2 Things to Avoid5. Known Issues6. Future Features7. Change Log1. License==========This software is licensed under the Apache 2.0 license. Please see LICENSEin the top level of this git repository for more information.2. Overview===========Twitcher is a minimal tool used to run scripts when a znode in ZooKeeperchanges. The intent is to allow various system tasks to take place on awatched model rather than via polling.The best description of Twitcher is to highlight an example that it solvesvery well. Lets assume we are facing a problem where we need a single gitrepository to be checked out and up to date on 100 machines in a cluster. Thetypical solution to this is to add a cron job that performs a git pull everyminute. This can be problematic as your git server has to endure the load of100 servers performing a pull every minute. You can configure Twitcher on allmachines to watch /git/head on your ZooKeeper instance. When that znodechanges Twitcher can run git pull. Add a post-commit script in your maingit repository that pokes that file and you will be able to fetch updateswithin few seconds of a commit on all machines while not requiring constantfetching when no changes are being made.3. Installation===============The default installation of Twitcher places the twitcher binary (bin/twitcher)in /usr/sbin and the twitcher module (twicher/*) in the site-packagesdirectory which is often /usr/lib/python2.6/sites-packages/twitcher.The configs are read recursively from /etc/twitcher while twitcher is running.Note that Twitcher uses inofity to update when files change so it canautomatically reload them. This is used to allow quick adding and removingof configs without restarting the server binary. If a config fails to parsethen it will not be updated in the running binary and a log message will bewritten.By default Twitcher uses syslog under daemon as the default logging method.4. Configuration Language=========================The configuration language for Twitcher is actually just python. Eachconfiguration must register a znode, and an action for each watch it wants.This is done via a RegisterWatch function which takes several arguments.RegisterWatch(): Creates a watch that will run a script when a Zookeeper nodeis updated.znode: This is the znode in ZooKeeper that is to be watched.action: This is a function or lambda that will perform an action whenthe znode is modified. The most common use here is to call Exec()which is a function documented later.pipe_stdin: If True then the contents of 'znode' will be piped to the stdinof the processes running action. Default is True.run_on_load: If True then the action will be executed when Twitcher starts.Without this your action may miss updates.The default is True.run_mode: This defines how Twitcher will react when 'znode' is updatedwhile it is running 'action' for a previous update. The optionalmodes are:QUEUE: This will queue the watch until the currently running'action' finishes, then run 'action' with the mostrecent contents. If several watches are received while'action' is running then only the last update willbe executed.DISCARD: Ignore the update, don't run the script, butre-register the watch.PARALLEL: Run the script in parallel for every update received.The default run mode is QUEUE.uid: The user id to run the process as (string or int). The default is torun as root.gid: The group id to run the process as (string or int). The default is torun as root.notify_signal: This is the signal that will be send to the spawnedprocess that is currently running when a new update hasbeen received. By default this is not used, and it will onlywork if the run_mode is QUEUE.timeout: The number of seconds before any spawned process should be senta SIGTERM. Five seconds later the process will be sent a SIGKILL.description: This is a text description of the watch which will be usedfor logging. The default is to name watches after the filethey are configured in.Exec(): Returns a lambda that will execute a given command when run.command: If this is a string then the command will be invoked in ashell interpreter. If its a list then it will be executedexactly as specified.By default the configuration files should be placed in /etc/twitcher andshould use an extension of ".twc".4.1. Examples=============In this example config a node called '/twitcher/example1' will be watched forupdates, and when it changes it will execute"date >> /tmp/twitcher_example1.out".RegisterWatch(znode='/twitcher/example1',action=Exec('date >> /tmp/twitcher_example1.out'))As another example we can run a python function rather than executing acommand. For this example we will execute the function example2() every time'/twitcher/example2' is updated.def example2():open('/tmp/twitcher_example2.out', 'w').write('testing')RegisterWatch(znode='/twitcher/example2',action=example2)Keep in mind that the action function is run in a process that has been forkedoff from the main Twitcher instance. As such the following exampleDOES NOT WORK AS EXPECTED. The output will always be 1 since the incrementhappens in the child process which exits after doesnotwork() finishes.x = 0def doesnotwork():global xx += 1open('/tmp/twitcher_badexample.out', 'w').write(x)RegisterWatch(znode='/twitcher/doesnotwork',action=doesnotwork)4.2 Things to Avoid===================Any action run by Twitcher is expected to by idempotent. Since the script maybe executed at any time and can be triggered rapidly it is expected thateach script does basic health and sanity checking before performing expensiveor destructive actions.Even if the script is in QUEUE mode there is a possibility that it may be runtwice. If Twitcher is restarted while the script is running it will thenrestart the script when Twitcher restarts which leaves two running at thesame time. In order to prevent this your script should use file based sigilor other method to verify exclusivity.5. Known Issues===============None at the moment.6. Future Features==================Failure handling on the child process: Allow the config to specify what shouldbe done in the case where the script fails (returns non zero). Some examplesinclude "retry x times" "fail but keep watch" "fail and stop watching"7. Change Log=============Release 1.7.5:2014/05/13: Trusty support for Ubuntu.Release 1.7.4:2014/04/02: Allow uploading of builds to be skipped so they are easier totest locally.2014/04/02: Fix a bug in the upstart script.2014/04/02: Better error handling and reporting, as well as a format stringfix that caused connections to be impossible.Release 1.7.32014/04/02: Imported print() from future to support python3.0 properly.Release 1.7.22014/03/21: Better Python 3.0 support.Release 1.7.12014/03/19: Integrated wickman and jtrosby's improvements into the code base.Release 1.7.02014/03/19: No signifigant code changes but major changes to the deliveryand build structure.Release: 1.52010/11/08: Make connection re-establish all existing watches.Release: 1.42010/10/25: Fix client session expiration recovery.2010/8/3: Documentation fixes.Release: 1.32010/8/2: Added notify_signal and timeout support and fix a few bugs.Release: 1.2, Open source under the Apache 2.0 license.2010/7/15: Added uid/gid supportRelease: 1.12010/6/24: Fixed a bug that allowed child processes to get ignored forever.Release 1.02010/6/19: Initial review.