Package: pyrainer
Rainer is a configuration management library that is based around versioned key/value pairs called "commits". We
created it after we noticed that we had a lot of services that each managed configuration in slightly different ways
(with slightly different flaws!). It is composed of a set of APIs that can be used independently, or together as
part of a powerful configuration management system. In addition to key/value access, it can also provide:Full audit trail of historical commits for each key, including who, why, and when they were committed.Ability to detect concurrent modifications and prevent users from clobbering each other.Extensibility to handle a variety of storage backends and commit payload types.Optional ZooKeeper-backed views and notifications, for immediate cluster-wide updates.Optional HTTP API along with clients for the JVM, Python, and the command line.Rainer is written in Scala, as are the examples on this page, but it should also be usable from other JVM
languages (like Java).CommitsRainer is based around versioned key/value pairs called "commits". Commits have the attributes:Key, which is a String.Payload, which can be an arbitrary type. Think of this like a short document.Version, which must increase by one with each successive commit.Author, the entity that created the commit.Comment, some free-form string describing the commit.Mtime, the timestamp when the commit was created.Each key's commits are separate from every other key's commits, and are versioned independently. In particular,
there is no concept of a global database version.Your payload type must be deserializable, so you need to provide a KeyValueDeserialization for it. You can optionally
provide a KeyValueSerialization as well.If you are not using a serializer, then when you create a new Commit, instead of providing the payload as an object,
you just provide some bytes that are deserializable using your KeyValueDeserialization. This mode is useful if you want
to allow humans to edit the serialized documents directly, since it will preserve comments, whitespace, and so on.UsageRainer offers three server-side components that can work together or separately:CommitStorage (persistent journaled storage)CommitKeeper (ZooKeeper-backed views and notifications)RainerServlet (HTTP API for remote inspection and modification)If you want to use all three together, the simplest way is using the "Rainers" builder, which creates all three and
links them together:implicitvalserialization=KeyValueSerialization.usingJackson[ValueType](objectMapper)implicitvaldeserialization=KeyValueDeserialization.usingJackson[ValueType](objectMapper)valdb=newMySQLDB(dbConfig)withDbCommitStorageMySQLMixinvalrainers=Rainers.create[ValueType](curator,"/path/in/zk",newDbCommitStorage[ValueType](db,"table_name"))db.start()rainers.start()// Do what you will with these:valmyStorage:CommitStorage[ValueType]=rainers.storagevalmyKeeper:CommitKeeper[ValueType]=rainers.keepervalmyServlet:RainerServlet[ValueType]=rainers.servlet// For example:myStorage.save(Commit.fromValue(key="foo",version=1,payload=newValueType,author="me",comment="Creating foo",mtime=DateTime.now))ComponentsCommitStorage: Persistent journaled storageThe CommitStorage trait represents a key/value store that retains every version of every commit. The main builtin
implementation is DbCommitStorage, which is backed by an RDBMS.implicitvaldeserialization=KeyValueDeserialization.usingJackson[ValueType](objectMapper)valdb=newMySQLDB(dbConfig)withDbCommitStorageMySQLMixinvalstorage=newDbCommitStorage[ValueType](db,"table_name")db.start()storage.start()// Revert "foo" to previous versionvalfoo=storage.get("foo")valpreviousFoo=fooflatMap(x=>storage.get(x.key,x.version-1))previousFooforeach(x=>storage.save(Commit.fromBytes(x.key,x.version+2,x.payload,"me","Reverting foo",DateTime.now))CommitKeeper: ZooKeeper viewsThe CommitKeeper class represents the most recent versions of each commit stored in ZooKeeper. It supports
on-demand access, live-updating views, and instant notifications. It does not keep a full history for each key; only
CommitStorages do that.implicitvaldeserialization=KeyValueDeserialization.usingJackson[ValueType](objectMapper)valkeeper=newCommitKeeper[ValueType](curator,"/path/in/zk")// On-demand access (synchronous ZK operations):valfoo=keeper.get("foo")valkeys=keeper.keys// Create a mirror (async updates from ZK):valmirror:Var[Map[String,Commit[ValueType]]]=keeper.mirror()// Which, for example, you can use to update an AtomicReference:valref=newAtomicReference[Map[String,Commit[ValueType]]]valc=mirror.changes.register(Witness(ref))// When you're no longer interested in updates, close the mirror:Await.result(c.close())Using CommitKeeper with CommitStorageCombining a CommitStorage with a CommitKeeper is a common pattern used for managing configuration of a cluster of
machines.  Combined setups can provide full-history journaled updates in an RDBMS, coupled with low latency
notifications through ZooKeeper. The idea is to do all of your saves through a keeperPublishing CommitStorage (which
will update ZooKeeper any time you save something) and to do your reads through a mirror provided by a CommitKeeper. To
prevent inopportune crashes or out of band ZooKeeper modifications from causing ZooKeeper to become out of sync, an
autoPublisher can periodically detect and push any unpushed updates from your storage.The "Rainers" builder sets all of this up for you, but you can also do it manually with something like:// During setup, link your underlying storage and keeper together.// Afterwards, use "storage", not "underlyingStorage" for your own operations.valkeeper=newCommitKeeper[ValueType](curator,"/path/in/zk")valstorage=CommitStorage.keeperPublishing(underlyingStorage,keeper)valautoPublisher=keeper.autoPublisher(underlyingStorage,period,periodFuzz)autoPublisher.start()// When your application exits, shut down your autoPublisher.Await.result(autoPublisher.close())When using a linked CommitStorage and CommitKeeper, make sure to send writes through the CommitStorage only. This is
because the CommitStorage is treated as the system of record.// You can get a mirror from your keeper, which will now reflect the most recent commits from your storage.valmirror:Var[Map[String,Commit[ValueType]]]=keeper.mirror()// Which, for example, you can use to update an AtomicReference:valref=newAtomicReference[Map[String,Commit[ValueType]]]valc=mirror.changes.register(Witness(ref))// When you save to the storage, ZooKeeper and the mirror will be updated automatically. (Don't save using the keeper!)storage.save(someCommit)Using CommitStorage without CommitKeeperYou can use a CommitStorage by itself. The only thing you lose are the ZooKeeper views, which means you can't set up
mirrors. In this case you'll either need to access the underlying storage on-demand, or set up a cache yourself that
refreshes periodically.Using CommitKeeper without CommitStorageYou can use CommitKeeper by itself, too, without any backing journaled storage. You won't be able to access old
versions of any commits. Just callkeeper.save(commit)to publish new commits, and don't use the autoPublisher.
The other CommitKeeper features will work normally, including gets and mirrors.HTTP serverYou can use the RainerServlet to gain a nice HTTP API to your CommitStorage. It does not use, or require, a
CommitKeeper. The API can be used like this with any HTTP client:$ curl \
  -s \
  -XPOST \
  -H'Content-Type: application/octet-stream' \
  -H'X-Rainer-Author: gian' \
  -H'X-Rainer-Comment: rofl' \
  --data-binary '{"hey":"what"}' \
  http://localhost:8080/diary/foo/1
{"author":"gian","comment":"rofl","key":"foo","mtime":"2014-02-11T14:01:28.839-08:00","version":1}

$ curl -s http://localhost:8080/diary/foo
{"hey":"what"}

$ curl -s http://localhost:8080/diary/foo/1
{"hey":"what"}

$ curl -s http://localhost:8080/diary/foo/1/meta
{"author":"gian","key":"foo","version":1,"mtime":"2014-02-11T14:01:28.839-08:00","comment":"rofl"}ClientsIf you have a RainerServlet running, there are a few pre-built clients that can be used to inspect and modify
configurations remotely.JVM HTTP clientYou can use HttpCommitStorage to access a RainerServlet running on another machine:valuri=newURI("http://localhost:8080/diary")valclient=ClientBuilder().name(uri.toString).codec(Http()).group(Group.fromVarAddr(InetResolver.bind(uri.getAuthority))).hostConnectionLimit(2).build()valstorage=newHttpCommitStorage[ValueType](client,uri)// Use it like any other CommitStorage:valcommit=storage.get("hey")Python HTTP clientFrom Python, you can use "pyrainer". You can install it withpip install pyrainerand use it like this:importpyrainer.httpclient=pyrainer.http.RainerClient("http://localhost:8080/diary")commits=client.list_full()forkeyinsorted(commits.keys()):version=commits[key].versionprint"%s[current version =%s]"%(key,str(version))hey=client.get_commit("hey")print"Commit version%s=%s"%(hey.meta["version"],hey.value)client.post_commit({"key":"hey","version":2,"author":"sue","comment":"rofl"},"new value")Command-line clientIf you have pyrainer installed (pip install pyrainer), you can use the command line tool interactively or in
shell scripts.$ python -m pyrainer.rainer --url http://localhost:8080/diary list
foo     1       http://localhost:8080/diary/foo/1
hey     6       http://localhost:8080/diary/hey/6

$ python -m pyrainer.rainer --url http://localhost:8080/diary show hey
{"hey":"whatsit"}

$ python -m pyrainer.rainer --url http://localhost:8080/diary edit hey
[...a wild $EDITOR appears!]You can also create a wrapper for your service that makes invocation simpler. An executable script like this should
do it:#!/bin/sh -eexecpython-mpyrainer.rainer--url"http://example.com/foo""$@"