client: Respect O_SYNC, O_DSYNC, and O_RSYNC

If the file is opened with O_SYNC, O_DSYNC, or O_RSYNC, we need to
flush cached data (and metadata for O_SYNC) on a write.
For O_RSYNC, we need to flush dirty data on a read.
This patch adds a file_flush() call to the objectCacher
to allow a specific range to be flushed from the cache, and
in the O_SYNC,O_DSYNC case for write and O_RSYNC case for read,
calls that function waiting for the flush to complete.  The patch
also adds a flags field directly to the file handle struct, and
replaces the append boolean with the use of the flags field directly.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
objectcacher: Remove commit_set, use flush_set

commit_set() and flush_set() are identical in functionality,
so use flush_set everywhere and remove commit_set from
the code.

Also fixes a bug in flush_set where the finisher context was
getting freed twice if no objects needed to be flushed.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
qa/workunit: Add iozone test script for sync

The iozone-sync.sh script runs iozone testing
various sync flags, O_SYNC, O_DSYNC, O_RSYNC.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
osd: reduce op thread heartbeat default 30 -> 15 seconds

If the thread stalls for 15 seconds, let our internal heartbeat fail.
This will let us internally respond more quickly to a stalled or failing
disk.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: do not reply to ping if internal heartbeat is not healthy

If we find that our internal threads are stalled, do not reply to ping
requests.  If we do this long enough, peers will mark us down.  If we are
only transiently unhealthy, we will reply to the next ping and they will
be satisfied.  If we are unhealthy and marked down, and eventually recover,
we will mark ourselves back up.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: hold lock while calling start_boot on startup

This probably doesn't strictly matter because start_boot doesn't need the
lock (currently) and few other threads should be running, but it is
better to be consistent.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: do not join cluster if not healthy

If our internal heartbeats are failing, do not send a boot message and try
to join the cluster.

Signed-off-by: Sage Weil <sage@inktank.com>
os/FileStore: add stall injection into filestore op queue

Allow admin to artificially induce a stall in the op queue.  Forces the
thread(s) to sleep for N seconds.  We pause for 1 second increments and
recheck the value so that a previously stalled thread can be unwedged by
reinjecting a lower value (or 0).  To stall indefinitely, just injust
very large number.

Signed-off-by: Sage Weil <sage@inktank.com>
common/HeartbeatMap: inject unhealthy heartbeat for N seconds

This lets us test code that is triggered by an unhealthy heartbeat in a
generic way.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: implement safety interlock for deleting pools

This is a very easy way for users to accidentally to a *lot* of damage.
Make it an annoying manual process to actually do this.

Signed-off-by: Sage Weil <sage@inktank.com>
rados: safety interlock on 'rmpool' command

This is a very easy way for a user to do a lot of damage with no way back.
Make sure they mean it.

Signed-off-by: Sage Weil <sage@inktank.com>
Merge branch 'wip-osd-hb'

Reviewed-by: Samuel Just <sam.just@inktank.com>
Merge remote-tracking branch 'gh/wip-3832-oc-flushrange'

Reviewed-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branch 'gh/wip-rm-pool'

Reviewed-by: Samuel Just <sam.just@inktank.com>
PendingReleaseNotes: pool removal cli changes

Signed-off-by: Sage Weil <sage@inktank.com>
Merge git://github.com/ceph/ceph