common/Formatter: add newline to flushed output if m_pretty

This applies to json-pretty and xml-pretty modes.

Signed-off-by: Sage Weil <sage@inktank.com>
objecter: don't resend paused ops

Paused ops are meant to block on the client side until a new map that
unpauses them is recieved. If we send paused writes when the FULL flag
is set, we'll get -ENOSPC from the osds, which is not what Objecter
users expect. This may cause rbd without caching to produce an I/O
error instead of waiting for the cluster to have capacity.

Fixes: #6725
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
objecter: set op->paused in recalc_op_target(), resend in not paused

When going through scan_requests() in handle_osd_map() we need to make
sure that if an op should not be paused anymore then set it on the op
itself, and return NEED_RESEND. Otherwise we're going to miss reset of
the full flag.
Also in handle_osd_map(), make sure that op shouldn't be paused before
sending it. There's a lot of cleanup around that area that we should
probably be doing, make the code much more tight.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
objecter: clean pause / unpause logic

op->paused holds now whether operation should be paused or not, and it's
being updated when scanning requests. No need to do a second scan.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
osd: drop writes when full instead of returning an error

There's a race between the client and osd with a newly marked full
osdmap.  If the client gets the new map first, it blocks writes and
everything works as expected, with no errors from the osd.

If the osd gets the map first, however, it will respond to any writes
with -ENOSPC. Clients will pass this up the stack, and not retry these
writes later.  -ENOSPC isn't handled well by all clients. RBD, for
example, may pass it on to qemu or kernel rbd which will both
interpret it as EIO.  Filesystems on top of rbd will not behave well
when they receive EIOs like this, especially if the cluster oscillates
between full and not full, so some writes succeed.

To fix this, never return ENOSPC from the osd because of a map marked
full, and rely on the client to retry all writes when the map is no
longer marked full.

Old clients talking to osds with this fix will hang instead of
propagating an error, but only if they run into this race
condition. ceph-fuse and rbd with caching enabled are not affected,
since the ObjectCacher will retry writes that return errors.

Refs: #6938
Backport: dumpling, emperor
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
objecter: resend all writes after osdmap loses the full flag

Now that the osd does not respond if it gets a map with the full flag
set first, clients need to resend all writes.

Clients talking to old osds are still subject to the race condition,
so both sides must be upgraded to avoid it.

Refs: #6938
Backport: dumpling, emperor
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
rbd: check write return code during bench-write

This is allows rbd-bench to detect http://tracker.ceph.com/issues/6938
when combined with rapidly changing the mon osd full ratio.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
osd/osd_types: make pref optional in pg_t constructor

We don't use preferred placements any more, so this will
make it easier to start dropping references to it in new code.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: drop redundant condition

We are inside an if (response_size) block.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: separate explicit pg target from current target

The pgid field is used to store the pg the op mapped to.  We were just
setting it directly for PGLS.  Instead, fill in a new base_pgid, and copy that
to pgid in recalc_op_target(), the same way we do when we map an object
name to a PG.

In particular, we take this opportunity to map a raw pgid to an actual
pgid.  This means the base_pg could come from a raw hash value (although
it doesn't, yet).

Signed-off-by: Sage Weil <sage@inktank.com>
Signed-off-by: Greg Farnum <greg@inktank.com>
osdc/Objecter: factor pg_read out of list_objects code

This will get used later for other ops against PGs (instead of objects).

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: stick bl inside ListContext

This is simpler and less error-prone.

Signed-off-by: Sage Weil <sage@inktank.com>
librados: add get_pg_hash_position to determine pg while listing objects

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/Objecter: reimplement list_objects

Return to caller at the end of each PG.  This allows the caller to look at
the [pg_]hash_position and get something meaningful.

If there are no objects in the PG, we skip it so that every callback has
*some* data (unless the pool is totally empty!).  So the real difference
here is that we don't move on to the next PG just to reach max_entries.

This gives the client some data sooner, but may mean more callbacks into
client code.

Signed-off-by: Sage Weil <sage@inktank.com>
mon/MonClient: constify inbl command args

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
osdc/Objecter: constify inbl command args

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
librados: constify inbl command args

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
librados: document aio_flush()

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
librados: add mon_command to C++ API

This way librados users can execute monitor commands.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
encoding: allow users to specify a different compatv after encoding

This way we can set the compatv preferentially depending on whether
we've actually encoded new information or not.

Signed-off-by: Greg Farnum <greg@inktank.com>