osd/ReplicatedPG: do not queue NULL dup_op

We call start_flush() with a NULL op in a couple different places.  Do not
put a NULL pointer on the dup_ops list or we will crash later.

Fixes: #8328
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0d67f9b0695765824bdc4a65fbed88edf8ea232e)
osd: handle race between osdmap and prepare_to_stop

If we get a MOSDMarkMeDown message and set service.state == STOPPING, we
kick the prepare_to_stop() thread.  Normally, it will wake up and then
set osd.state == STOPPING, and when we process the map message next we
will not warn.  However, if dispatch() takes the lock instead and processes
the map, it will fail the preparing_to_stop check and issue a spurious
warning.

Fix by checking for either preparing_to_stop or stopping.

Fixes: #8319
Backport: firefly, emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 6b858be0676f937a99dbd51321497f30c3a0097f)
osd/osd_types: add last_force_op_resend to pg_pool_t

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 3152faf79f498a723ae0fe44301ccb21b15a96ab)
osdc/Objecter: resend ops in the last_force_op_resend epoch

If we are a client, and process a map that sets last_force_op_resend to
the current epoch, force a resend of this op.

If the OSD expects us to do this, it will discard our previous op.  If the
OSD is old, it will process the old one, this will appear as a dup, and we
are no worse off than before.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dd700bdf7115223cb3e517b851f462d75dd76a2b)
osd: discard client ops sent before last_force_op_resend

If an op is sent before last_force_op_resend, and the client's feature is
present, drop the op because we know they will resend.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 45e79a17a932192995f8328ae9f6e8a2a6348d10)
mon/OSDMonitor: force op resend when pool overlay changes

If a client is sending a sequence of ops (say, a, b, c, d) and partway
through that sequence it receives an OSDMap update that changes the
overlay, the ops will get send to different pools, and the replies will
come back completely out of order.

To fix this, force a resend of all outstanding ops any time the overlay
changes.

Fixes: #8305
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 63d92ab0969357f78fdade749785136a509bc81b)
Objecter::_op_submit: only replace the tid if it's 0

Otherwise, redirected ops will suddenly have a different tid
and will become uncancelable.

Fixes: #7588
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 76568aa0db4e16ac1af8fe6405edade1e61cbc81)
msg: Fix inconsistent message sequence negotiation during connection reset

Backport: firefly, emperor, dumpling

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit bdee119076dd0eb65334840d141ccdf06091e3c9)
ReplicatedPG: block scrub on blocked object contexts

Fixes: #8011
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 7411477153219d66625a74c5886530029c516036)
osd: skip out of order op checks on tiered pools

When we send redirected ops, we do not assign a new tid, which means that
a given client's ops for a pool may not have strictly ordered tids.  Skip
this check if the pool is tiered to avoid false positives.

Fixes: #8380
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit cf2b172c843da0599164901956b66c306a59e570)
OSD: fix an osdmap_subscribe interface misuse

When calling osdmap_subscribe, you have to pass an epoch newer than the
current map's. _maybe_boot() was not doing this correctly -- we would
fail a check for being *in* the monitor's existing map range, and then
pass along the map prior to the monitor's range. But if we were exactly
one behind, that value would be our current epoch, and the request would
get dropped. So instead, make sure we are not *in contact* with the monitor's
existing map range.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 290ac818696414758978b78517b137c226110bb4)
test: fix some templates to match new output code

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 00225d739cefa1415524a3de45fb9a5a2db53018)
ceph-disk: partprobe before settle when preparing dev

Two users have reported this fixes a problem with using --dmcrypt.

Fixes: #6966
Tested-by: Eric Eastman <eric0e@aol.com>
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0f196265f049d432e399197a3af3f90d2e916275)
look for "jni.h" in gcj-jdk path, needed to find "jni.h" with gcj-jdk_4.9.0

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
(cherry picked from commit 0f4120c0115e7977ae7c03458addcc2b2916db07)
pass '-classpath' option (gcj/javah ignores CLASSPATH environment variable).

 This should not affect OpenJDK which understands '-classpath' as well.

 With gcj-jdk we still get FTBFS later:

 ~~~~
    java/native/libcephfs_jni.cc:2878:55: error: invalid conversion from 'const jbyte* {aka const signed char*}' to 'jbyte* {aka signed char*}' [-fpermissive]
                 reinterpret_cast<const jbyte*>(rawAddress));
                                                           ^
    In file included from java/native/libcephfs_jni.cc:27:0:
    /usr/lib/gcc/x86_64-linux-gnu/4.8/include/jni.h:1471:8: error:   initializing argument 4 of 'void _Jv_JNIEnv::SetByteArrayRegion(jbyteArray, jsize, jsize, jbyte*)' [-fpermissive]
       void SetByteArrayRegion (jbyteArray val0, jsize val1, jsize val2, jbyte * val3)
            ^
    make[5] *** [java/native/libcephfs_jni_la-libcephfs_jni.lo] Error 1
 ~~~~

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
(cherry picked from commit 89fe0353582bde7e2fbf32f1626d430a20002dd0)
prioritise use of `javac` executable (gcj provides it through alternatives).

 On Debian this fixes FTBFS when gcj-jdk and openjdk-7-jdk are installed at
 the same time because build system will use default `javac` executable
 provided by current JDK through `update-alternatives` instead of blindly
 calling GCJ when it is present.

Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
(cherry picked from commit 8b682d167e4535df582f1c77542e2b1ea0981228)
OSDMonitor: set next commit in mon primary-affinity reply

Commit 8c5c55c8b47e ("mon: set next commit in mon command replies")
fixed MMonCommand replies to include the right version, but the
primary-affinity handler was authored before that.  Fix it.

Backport: firefly
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
(cherry picked from commit a78b14ec1769ef37bef82bfda6faabb581b4cd7d)
cephfs-java: build against older jni headers

Older versions of the JNI interface expected non-const parameters
to their memory move functions. It's unpleasant, but won't actually
change the memory in question, to do a cast_const in order to satisfy
those older headers. (And even if it *did* modify the memory, that
would be okay given our single user.)

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 4d4b77e5b6b923507ec4a0ad9d5c7018e4542a3c)
rest-api: key missing for per "rx" and "rwx"

commit 85a1cf31e6 and db266a3fb2 introduce new per "rx" and "rwx", but key missing for per "rx" and "rwx" in permmap

Signed-off-by: Ailing Zhang <ailzhang@cisco.com>
(cherry picked from commit 0b5a67410793ec28cac47e6e44cbbcf5684d77e7)
add backport of collections.Counter for python2.6

Using Raymond Hettinger's MIT backport

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>
(cherry picked from commit 23b75b550507438c79b3aa75e06721e5f7b134a4)