librados: define error callback

This is the async notification of a watch error (failure to reconnect to
the object's OSD after a peering event or network disconnect).

Signed-off-by: Sage Weil <sage@redhat.com>
librados: add infrastructure to deliver an error notification

Use a reusable context in the WatchNotifyInfo to trigger an error
event, delivered via the existing Finisher thread.  Re-lookup the cookie
in the thread to cope with races with unregister (just as we do with
notify events).

Signed-off-by: Sage Weil <sage@redhat.com>
rados: improve watch command output

- hexdump the notify payload, and include all of the new metadata
- print error events too

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: separate WATCH from RECONNECT

Use WATCH op for the initial registration.  This is idempotent in that
it will succeed whether the watch information has been persisted or not.
It is used by the client if it does not know that it is registered.

The RECONNECT op is used for any subsequent session reconnect.  It will
fail if the watch state isn't already persisted on the OSD.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: queue an error if reconnect fails

If we fail to reconnect, trigger the on_error (reusable) context to
let the user know about it.

Signed-off-by: Sage Weil <sage@redhat.com>
osdc/Objecter: send regular PING ops

Send a full PING op to the object to ensure we are still connected. For
now just use the existing ping interval; we may want to change this in
the future.

Signed-off-by: Sage Weil <sage@redhat.com>
Revert "osd: assume WATCH_NOTIFY for MWatchNotify"

This reverts commit 1197cfcb794b64b7c1e2ab8ce759c5e47330ec31.
clean up CEPH_WATCH_EVENT_* namespace, and make COMPLETE distinct

Use a proper prefix and make a separate event for the notify completion
message.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: only send pending notifies on reconnect, not fresh watches

If an existing watch reconnects, we must give them a chance on the
in-progress notify because it will fail anyway if they don't ack it.

However, a new watch should not receive a notify that initiated before it
tried to add the watch.  It doesn't make sense from an API perspective, and
on a practical level it means that even a well-behaved client may cause
a notify to fail by adding the watch near the end of the notify timeout
period when there isn't enough time for the notify to be processed.  (That
would happen if another watcher is acking just in the nick of time.)

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_rados_api_watch_notify: s/handle/cookie/

Use consist nameing for C and C++ tests.

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_rados_api_watch_notify: verify we get the right # of notifies

Signed-off-by: Sage Weil <sage@redhat.com>
osd/Watch: fix comment typo

Signed-off-by: Sage Weil <sage@redhat.com>
Revert "osd: only send pending notifies on reconnect, not fresh watches"

This reverts commit a883cd392258bade3e21a78441c6859d265e762e.
osd/Watch: drop redundant in_progress_watchers

This is essentially watchers.size() at all times and obfuscates the code.
In particular this makes the maybe_complete_notify() logic much clearer.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: behave if errcb is NULL

Signed-off-by: Sage Weil <sage@redhat.com>
librados: EINVAL on bad args to watch2

Signed-off-by: Sage Weil <sage@redhat.com>
osd/Watch: inform watchers when they cause a notify timeout

If a watcher doesn't ack in time, let them know that they triggered a
notify failure.

Signed-off-by: Sage Weil <sage@redhat.com>
librados: tell watcher if we cause a notify timeout

If we are a watcher and we fail to notify in a timely manner, or
circumstances otherwise conspire to prevent out ack from arriving in time,
initiate a callback.

Signed-off-by: Sage Weil <sage@redhat.com>
osd, librados: distinguish between old and new watchers

Use a new watch op code that is different from the legacy one so we can
tell when the client is doing all the new stuff (like sending pings).

Signed-off-by: Sage Weil <sage@redhat.com>
rados fix notify ack