client: Wait for caps to flush when flushing metadata.

Embarrassingly, this conditional has been backwards since
I committed it in 818e7939. But we want to do the wait when
we did do the flush and are syncing metadata. As the code
previously stood, we only did the wait when we didn't care,
and it didn't wait anyway since it was waiting for 0. So,
bad all around.
Signed-off-by: Greg Farnum <greg@inktank.com>
client: return errors to the user if fsync fails

To do so, we allow callers of _flush(Inode) to pass in a Context
as well. This Context is then given to the ObjectCacher as
the completion callback instead of the C_Client_PutInode that
it uses by default. _fsync() passes in a SafeCond which is
pointing to its return code, and presto! The return code gets
a failure if the Objecter returns any.
To preserve existing semantics, _fsync() (and any similar
caller) is now required to take a reference to the inode.

However, this is a behavioral change: previously, a call to fsync
in the userspace code would simply spin as it waited for all the
data to flush out. That obviously wasn't desirable, but now a user
can call fsync, ignore the return value, and think that everything is
hunky-dory when in fact the client is repeatedly trying to flush out
the dirty data. Hrm...

This fixes the fsync case of #2753 for ceph-fuse.

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge branch 'wip-2753-fsync-errors'

Reviewed-by: Sage Weil <sage@inktank.com>