DNM: new watch/notify framework

This is mostly complete.  The main remaining items are to convert rgw and
librbd over to use the new version of the API.

One thing I ran out of steam before doing is writing a reference user for
the API that uses all elements to full effect.  I think the use case I have in
mind is a shared cache with invalidates, like what RGW is doing.  But instead of
writing a toy or test implementation, it would be best to write a generic
shared-cache module that has all the features RGW needs and then migrate to using
it.  Maybe that will be simpler than fixing up RGW to make use of watch_check()
properly...?
On Thu, 20 Nov 2014, Jason Dillaman wrote:
> I think my questions were poorly worded.
> 
> I guess my concern is that a client calling "watch_check" can only compare the result to its local config setting for osd_client_watch_timeout to know whether
> or not the watch should still be considered valid from the server (barring other issues). What the client really wants is to compare the return value with the
> known timeout value the OSD is using (not the potentially different client-side config setting). i.e. an OSD is configured to timeout in 15 seconds but the
> client's default config says the timeout is 30 seconds. I wasn't suggesting allowing clients to tell the OSD what timeout value to use.
> 
> My second question was really whether watch_check could just be simplified to return 0 if the watch should still be valid or a negative error code if it
> shouldn't be considered valid. If the client is really only using the result to know if the last ACKed ping was within the OSD's timeout interval, it seems
> like it could be simplified to that. If it is possible to simplify it, Objecter would no longer need to keep track of the timestamp of when an error was
> detected on the watch.

Oh, I see.

The idea is that the 'timeout' we use here is application dependent, and 
independent of what the watch timeout is on the OSD (as long as it is 
<=).  For example, it may be that I have a hot cache and I'm 
willing to wait 3 seconds to invalidate before giving up and ignoring the 
cache.  For RBD, we may decide that the leader has to respond in 10s or 
else we take over the lock.  AFAICS it's okay that the watch registration 
actually lasts longer than that (and often it will take much longer 
to expire because of OSD peering or whatever) because we are not relying 
on the watch timeout to reset, but instead doing watch_check() to confirm 
we are still alive and allowed to do whatever we're doing.

For RBD, for example, we would guard any leader activities with 

 int r = watch_check();
 if (r > 0 && r < liveness) { ... }

...and, ideally, also make the IO operations also conditional on the lock 
attr on the object.  If we fail that check, either due to an error or 
stale result, then we refresh and make sure we are still in 
control.

That's the idea... does it cover all the bases for RBD?