Package: nd_service_registry
# Nextdoor Service Registry*nd_service_registry* is a Python module that provides a simple way to leverage*Apache Zookeeper* as a dynamic configuration and service registry.The goal of the package is to provide a single foundational class that can beleveraged by any Python program for registering and monitoring services throughZookeeper.The current use cases are:* Register a server providing a service (and subsequently retrieve them)* Store a simple dict/key value in Zookeeper (and retreive it)* Execute callback methods whenever any node list or data setting changes* Get a temporary (but global!) lock while you do somethingThe main benefit of using this module is if you have several different toolsin your network that all need to interact with Zookeeper in a common way. Themost common functions are handled in this singular module allowing you to focuson your own app development more.## InstallationTo install, runpython setup.py installorpip install nd_service_registry## Usage### Instantiating KazooServiceRegistryCreate a logger object>>> import logging>>> logging.basicConfig(level=logging.DEBUG)To create your initial connection object>>> import nd_service_registry>>> nd = nd_service_registry.KazooServiceRegistry()The KazooServiceRegistry object is a child of nd\_service\_registry thatconforms to our ServiceRegistry specs, while leveraging Kazoo as the backend.The object handles all of your connection states - there is no need tostart/stop or monitor the connection state at all.### Registering Data#### Ephemeral NodesRegistering a node in Zookeeper thats Ephemeral (disappears when the nodegoes offline, or if the connection is lost) is highly useful for keepingtrack of which servers are offering specific services.>>> nd.set_node('/production/ssh/server1:22')>>> nd.set_node('/production/ssh/server2:22')>>> nd.set_node('/production/ssh/server3:22')>>> nd.set_node('/production/web/server2:22',data={'type': 'apache'})>>> nd.get('/production/ssh'){'children': [u'server1:22', u'server2:22', u'server3:22' ],'data': None,'path': '/production/ssh','stat': ZnodeStat(czxid=27, mzxid=27, ctime=1355533229452,mtime=1355533229452, version=0, cversion=5,aversion=0, ephemeralOwner=0, dataLength=0,numChildren=3, pzxid=45)}To register the host as providing a particular service>>> nd.set_node('/services/ssh/server1:22', data={ 'foo': 'bar'})Getting a list of servers at a path>>> nd.get('/services/ssh'){'children': {u'server1:22': {u'foo': u'bar',u'created': u'2012-12-15 00:45:03',u'pid': 10733}},'data': None,'path': '/services/ssh','stat': ZnodeStat(czxid=6, mzxid=6, ctime=1355532303688,mtime=1355532303688, version=0, cversion=1,aversion=0, ephemeralOwner=0, dataLength=0,numChildren=1, pzxid=7)}#### Arbitrary DataWhen you want to store arbitrary (but simple!) data objects in, its simple!>>> sr.set_data('/config/my_app', data={'state': 'enabled'})True>>> sr.get('/config/my_app'){'children': []'data': {u'created': u'2014-06-03 13:37:53',u'pid': 2546,u'state': u'enabled'},'path': '/config/my_app','stat': ZnodeStat(czxid=76, mzxid=77, ctime=1401827873764,mtime=1401827873766, version=1, cversion=0,aversion=0, ephemeralOwner=0, dataLength=62,numChildren=0, pzxid=76)}### Distributed LocksOne of Zookeepers great features is using it as a global lock manager. We providetwo models for getting a lock. In one model, your lock is only active as long asyour code is running>>> with nd.get_lock('/foo', simultaneous=1):...      <do some work>...>>>Another example is explicitly locking a path for some period of time, thenreleasing it explicitly (eg, locking during one method, and waiting for anentirely different method to handle the unlock)>>> nd.acquire_lock('/foo', simultaneous=1)>>> <do your work... >>>> nd.release_lock('/foo')### DjangoWe initially wrote this code to simplify our use of Zookeeper in Django.Included is a very simple Django utility package that makes it dead simpleto use Zookeeper inside Django for retreiving information. To use, addthe following to your *settings* module in Django:from nd_service_registry.contrib.django import utils as ndsrndsr.SERVER = '127.0.0.1'ndsr.PORT = '2181'ndsr.TIMEOUT = 5ndsr.CACHEFILE = '/tmp/serviceregistry.cache'Example usage to grab a single string and use it as a setting:MY_CRED = ndsr.get('/creds/some_cred')['data']['mycred']Example use in your code, calling a method every time a value is updatedin Zookeeper:>>> def do_something(data):...     print "New server data: %s" % data...>>> ndsr.get('/services/staging/uswest2/memcache', callback=do_something)New server data: { 'path': '/services/staging/uswest2/memcache','stat': ZnodeStat(czxid=8589934751, mzxid=8589934751,ctime=1354785240728,mtime=1354785240728, version=0,cversion=45, aversion=0,ephemeralOwner=0, dataLength=0,numChildren=1, pzxid=30064903926),'data': None,'children': { u'ec2-123-123-123-123.us-west-2:11211': {u'created': u'2013-01-08 16:51:12',u'pid': 3246}}}Warning: LC\_ALL and LANG settingsDue to an unknown bug, if Django cannot find your LC\_ALL LOCALE settings(which often default to 'C'), *nd_service_registry* or *kazoo* crash andburn during the init phase. Its unknown why at this point, but we've foundthat its best to *unset LC\_ALL* and set *LANG=en_US:UTF-8* (or some othervalid setting) before you start up your Django app.If you use Celery, set these options in */etc/default/celeryd*.If you use uWSGI, set them in your uWSGI config file.Running the Django shell# unset LC_ALL; LANG=en_US:UTF-8 python manage.py shell## Connection HandlingThe ServiceRegistry object tries everything that it can to make sure thatthe backend Zookeeper connection is always up and running.### Fork BehaviorIn the event that your code has created an ServiceRegistry object but thengone and forked the process (celery, as an example), we do our best todetect this and re-create the connection, watchers and registrations.When we detect a fork (more on that below), we re-create our Zookeeperconnection, and then re-create all Watcher and Registration objects as well.### Fork DetectionDetecting the fork is extremely tricky... we can only really detect it whenyou call the module for new data. This means that if you have created aWatcher or Registration object, those objects will not be aware of the fork(and thus the loss of their connection to Zookeeper) until you make anothercall to them.Because of this, I strongly recommend that if you can detect the fork fromwithin your application (Django signals perhaps?), you should immediately callthe *rebuild()* method on your ServiceRegistry object.>>> from nd_service_registry import KazooServiceRegistry>>> k = KazooServiceRegistry()>>> do_fork()>>> k.rebuild()## ExceptionsThe goal of this module is to be as self-contained as possible and requireas little code in your app as possible. To that end, we *almost never* raisean Exception once the module is loaded up and connected.We do raise a few exceptions, and each one is documented here. Whenever wecan though, we instead just *return False* as a way of indicating that we wereunable to perform your command now ... but that we will take care of it later.Whenever we do this, we throw a WARNING log message as well.### nd\_service\_registry.exceptions.NoConnectionThrown if you attempt any operation that requires immediate access to thebackend Zookeeper service. Either a *set()* operation, or a *get()*operation on a path for the first time.Also thrown during initial connection to Zookeeper, if *lazy=False*.(It should be noted, a *get()* will actually return the cached results evenif Zookeeper is down. This allows the service to fail temporarily in thebackground but your app is still able to get the *last known* results.)### nd\_service\_registry.exceptions.ReadOnlyIf *readonly=True*, any operation that would result in a *write* will throwthis exception. Most notably, a *set()* operation will fail with thisexception if *readonly=True*.## API DocumentationDetailed implementation details and instructions are in the individuallibrary files.## Development### Unit TestsRunning them$ python setup.py testrunning testtest_creates_simple_json_outputformat (nd_service_registry.bin.ndsr.get_tests.GetTests) ... ok......test_default_data_produces_expected_dict (nd_service_registry.funcs_tests.FuncsTests) ... oktest_encode_creates_dict_from_single_string (nd_service_registry.funcs_tests.FuncsTests) ... okRan 15 tests in 0.108sOK### Integration TestsIngegration tests download and install Zookeeper in a temporary path in yourworkspace.  They can be executed like this$ make integrationrunning integration...Make sure that the enter/exit functionality works in non-blocking ... okTest that a blocking Lock works ... oktest_decode_converts_json_to_dict (nd_service_registry.funcs_tests.FuncsTests) ... ok----------------------------------------------------------------------Ran 19 tests in 3.360sOK## Apps using nd\_service\_registry* [zk_monitor](http://github.com/nextdoor/zkmonitor): Zookeeper Path Monitoring Agent* [zk_watcher](http://github.com/nextdoor/zkwatcher): Registers third-party (memcache/etc)services in Zookeeper.