mon: fix occasional message leak after session reset

Consider:

 - we get a message, put it on a wait list
 - the client session resets
 - we go back to process the message later and discard
   - _ms_dispatch returns false, but nobody drops the msg ref

Since we call _ms_dispatch() a lot internally, we need to always return
true when we are an internal caller.

Fixes: #9176
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>
mon: always process the message in dispatch

Nobody should be chained after teh mon, so we can safely drop any message
we don't understand.

Signed-off-by: Sage Weil <sage@redhat.com>
mon: make dispatch(), _ms_dispatch() void

They always return true.

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2301 from ceph/wip-9176

mon: fix occasional memory leak; clean up dispatch return codes

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
rgw: separate civetweb log from rgw log

The civetweb log now is independent from the rgw log.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
rgw: use a separate callback for civetweb access log

Access log is separate from the regular civetweb logging. Also, changed
the log level for the regular logging as it's used mostly for error
messages.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
CollectionIndex: Collection name is added to the access_lock name

The CollectionIndex constructor is changed to accept the coll_t
so that the collection name can be used to form access_lock(RWLock)
name.This is needed otherwise lockdep will report a recursive lock error
and assert. lockdep needs unique lock names for each Index object.

Fixes: #9145

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>
(cherry picked from commit 615d2d904024526cc58557ee5250c2536a3cd5c8)
ceph_test_rados_api_tier: make PromoteOn2ndRead test tolerate retries

If there is an ill-timed connection reset our read could get sent twice.
Weaken our assertion if the read was slow to tolerate this case.

Fixes: #9221
Signed-off-by: Sage Weil <sage@redhat.com>
rgw: fix test to identify whether object has tail

Fixes: #9226
Reported-by: Sylvain Munaut <s.munaut@whatever-company.com>
Backport: firefly

We need to identify whether an object is just composed of a head, or
also has a tail. Test for pre-firefly objects ("explicit objs") was
broken as it was just looking at the number of explicit objs in the
manifest. However, this is insufficient, as we might have empty head,
and in this case it wouldn't appear, so we need to check whether the
sole object is actually pointing at the head.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Merge pull request #2320 from ceph/wip-9221

ceph_test_rados_api_tier: make PromoteOn2ndRead test tolerate retries

Reviewed-by: Samuel Just <sam.just@inktank.com>
osd/OSDMap: encode blacklist in deterministic order

When we use an unordered_map the encoding order is non-deterministic,
which is problematic for OSDMap.  Construct an ordered map<> on encode
and use that.  This lets us keep the hash table for lookups in the general
case.

Fixes: #9211
Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2321 from ceph/wip-9226

rgw: fix test to identify whether object has tail

Reviewed-by: Sage Weil <sage@redhat.com>
PG: make the reservation sets more descriptively named

These sets won't precisely be the backfill_targets or actingbackfill
shortly.

%s/sorted_backfill_set/remote_shards_to_reserve_backfill/g
%s/acting_osd_it/remote_recovery_reservation_it/g
%s/sorted_actingbackfill_set/remote_shards_to_reserve_recovery/g

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: recover from each osd at most once

Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge pull request #2330 from ceph/wip-9211

osd/OSDMap: encode blacklist in deterministic order

Reviewed-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG::start_flush: remove superfluous loop

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG:start_flush send a second delete

Suppose we start with the following in the cache pool:

30:[29,21,20,15,10,4]:[22(21), 15(15,10), 4(4)]+head

The object doesn't exist at 29 or 20.

First, we flush 4 leaving the backing pool with:

3:[]+head

Then, we begin to flush 15 with a delete with snapc 4:[4] leaving the
backing pool with:

4:[4]:[4(4)]

Then, we finish flushing 15 with snapc 9:[4] with leaving the backing
pool with:

9:[4]:[4(4)]+head

Next, snaps 10 and 15 are removed causing clone 10 to be removed leaving
the cache with:

30:[29,21,20,4]:[22(21),4(4)]+head

We next begin to flush 22 by sending a delete with snapc 4(4) since
prev_snapc is 4 <---------- here is the bug

The backing pool ignores this request since 4 < 9 (ORDERSNAP) leaving it
with:

9:[4]:[4(4)]

Then, we complete flushing 22 with snapc 19:[4] leaving the backing pool
with:

19:[4]:[4(4)]+head

Then, we begin to flush head by deleting with snapc 22:[21,20,4] leaving
the backing pool with:

22[21,20,4]:[22(21,20), 4(4)]

Finally, we flush head leaving the backing pool with:

30:[29,21,20,4]:[22(21*,20*),4(4)]+head

When we go to flush clone 22, all we know is that 22 is dirty, has snaps
[21], and 4 is clean. As part of flushing 22, we need to do two things:
1) Ensure that the current head is cloned as cloneid 4 with snaps [4] by
sending a delete at snapc 4:[4].
2) Flush the data at snap sequence < 21 by sending a copyfrom with snapc
20:[20,4].

Unfortunately, it is possible that 1, 1&2, or 1 and part of the flush
process for some other now non-existent clone have already been
performed.  Because of that, between 1) and 2), we need to send
a second delete ensuring that the object does not exist at 20.

Fixes: #9054
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD::session_notify_pg_create: requeue at the start of the queue

Introduced: 2120f4bb6c5ba0f066d4541a51ce1d43c8ab6881
Fixes: #9205
Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: mark_log_for_rewrite on resurrection

Fixes: #8777
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>
rgw: clear bufferlist if write_data() successful

Fixes: #9201
Backport: firefly

We sometimes need to call RGWPutObjProcessor::handle_data() again,
so that we send the pending data. However, we failed to clear the buffer
that was already sent, thus it was resent. This triggers when using non
default pool alignments.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>