osdc/Objecter: do notify completion callback in fast-dispatch context

The notify completion has exactly one user, the librados caller which
does nothing but take a local (inner) lock and signal a Cond.  Do this
in the fast-dispatch context for simplicity.

Notably, this makes the notify completion (and timeout) trigger a
notify2() return (with ETIMEDOUT) even when the finisher queue that
normally delivers notify is busy.. for example with a notify that is
being very slow.  In our case, the unit test is doing a sleep(3) to
test timeouts but also prevented the ETIMEDOUT notification from
being delivered to the caller.  This patch resolves that.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: add rados_watch_flush() call

Add a call so that callers can make sure all queued callbacks have
completed before shutting down the ioctx.  This avoids a segv triggered
by the LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1
test due to the ioctx being destroyed when the in-progress callback
does a notify_ack.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: watch_flush() on shutdown

Users can easily forget this. It makes shutdown potentially block, but if
they have racing callbacks they get what they ask for.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: warn about rados_watch_flush() prior to ioctx shutdown

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #3222 from ceph/wip-watch-notify-2

librados: fix watch/notify event flushing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>