librados: watch_check() to check on status of watch

Return either an accumulated error code, or the time since the watch was
last confirmed.

Signed-off-by: Sage Weil <sage@redhat.com>
messages/MWatchNotify: print more nicer

Signed-off-by: Sage Weil <sage@redhat.com>
INCOMPLETE rgw: switch to new watch/notify API

Define a 'liveness' option that controls how much latency we will tolerate
on the notification mechanism before we give up and skip the cache.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: expose LingerOp instead of linger_id

We want a reference-counted handle, not an id.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: spit out linger register step

Signed-off-by: Sage Weil <sage@redhat.com>
librados: use Objecter's linger_id as the cookie

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: take Finisher ref from creator

Only needed for watch/notify users, so most can pass NULL.

Signed-off-by: Sage Weil <sage@redhat.com>
rados: break out notify replies by client

Signed-off-by: Sage Weil <sage@redhat.com>
librados: move watch/notify into the Objecter

Several things here:
 - we move all of the junk from librados' RadosClient into Objecter
 - we use fast-dispatch to schedule the watch-notify events (this will
   keep them ordered wrt ping)
 - we use the LingerOp * as the librados-exposed handle so that we can
   avoid any lookups for watch-check.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: drop ACK callback for watch

It is semantically useless.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: only reconnect watches; skip ack unless needed

Only watches get a RECONNECT op; lingers just get resent.

Only request an ACK in the non-watch reconnect case and when an ack
callback is specified.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: ignore pings from previous registrations

If we have a ping in flight and then reregister (due to interval change
or whatever), ignore the result of the previous ping.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: rejigger watch connect/disconnect callers

Signed-off-by: Sage Weil <sage@redhat.com>
encoding: multiset encode/decode

Signed-off-by: Sage Weil <sage@redhat.com>
librados: include missed watchers in notify reply data

Include a list of who missed their notify in the notify reply.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: use (gid,cookie) to identify notify participants

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: use recast LingerOp* as OTW cookie, too

This makes the OSD reply's cookie values match up with what the librados
users see.  And it makes life less confusing when debugging.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: make watch_check time reflect async delivery

When a librados user calls watch_check(), the age they get back should
establish a timestamp for which:

 1) we know that the watch was registered at least through this time,
 2) we have received and processed notifies sent as of this time,
 3) we have processed any async error events as of this time.

We already accomplish 1 by updating watch_valid_thru based on the time
the ping is sent (not received).

This patch gets us 2 and 3 by using the MIN of watch_valid_thru and the
oldest queued async event for the watch_check result.

Signed-off-by: Sage Weil <sage@redhat.com>