osdc/Objecter: take Finisher ref from creator

Only needed for watch/notify users, so most can pass NULL.

Signed-off-by: Sage Weil <sage@redhat.com>
rados: break out notify replies by client

Signed-off-by: Sage Weil <sage@redhat.com>
librados: move watch/notify into the Objecter

Several things here:
 - we move all of the junk from librados' RadosClient into Objecter
 - we use fast-dispatch to schedule the watch-notify events (this will
   keep them ordered wrt ping)
 - we use the LingerOp * as the librados-exposed handle so that we can
   avoid any lookups for watch-check.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: drop ACK callback for watch

It is semantically useless.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: only reconnect watches; skip ack unless needed

Only watches get a RECONNECT op; lingers just get resent.

Only request an ACK in the non-watch reconnect case and when an ack
callback is specified.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: ignore pings from previous registrations

If we have a ping in flight and then reregister (due to interval change
or whatever), ignore the result of the previous ping.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: rejigger watch connect/disconnect callers

Signed-off-by: Sage Weil <sage@redhat.com>
encoding: multiset encode/decode

Signed-off-by: Sage Weil <sage@redhat.com>
librados: include missed watchers in notify reply data

Include a list of who missed their notify in the notify reply.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: use (gid,cookie) to identify notify participants

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: use recast LingerOp* as OTW cookie, too

This makes the OSD reply's cookie values match up with what the librados
users see.  And it makes life less confusing when debugging.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: make watch_check time reflect async delivery

When a librados user calls watch_check(), the age they get back should
establish a timestamp for which:

 1) we know that the watch was registered at least through this time,
 2) we have received and processed notifies sent as of this time,
 3) we have processed any async error events as of this time.

We already accomplish 1 by updating watch_valid_thru based on the time
the ping is sent (not received).

This patch gets us 2 and 3 by using the MIN of watch_valid_thru and the
oldest queued async event for the watch_check result.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: use RWLock for watch_lock

This makes the linger_check fast path an RWLock read lock.  Drop the
unnecessary write lock on rwlock and drop the unused Cond.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: set last_error before async error event delivery

No reason to delay this.

Signed-off-by: Sage Weil <sage@redhat.com>
test_async_driver: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>
PendingReleaseNotes: a few notes about watch/notify changes

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_rados_api_watch_notify: test ENOENT case

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_rados_watch_notify: do not unwatch on invalid handle

It is not safe to call rados_unwatch on an undefined handle.

Signed-off-by: Sage Weil <sage@redhat.com>