Durable data

I'm not looking to actually merge this PR, just get some feedback on two possible approaches.

If there is agreement on one approach, then I will code it up, and convert this design document into user documentation.  

I'm inclined to add a new type of REST resource to model Durable local data.  I know new resources are always  controversial, so I've written this doc to compare the alternatives.  Comments welcome from anyone.  I'd particularly like comments from @dchen1107 @bgrant0607 @thockin @brendanburns @jbeda @lavalamp @smarterclayton.

Agreed. High-availability services with a lot of local state (even in memory rather than on disk/ssd) will likely want to wait longer before giving up than, say, 2-second compilation tasks. It's all about expected time to repair.