filestore: fix collection_move guard

We had a sequence like:

 1- write A block 1
 2- write A block 2
 3- write A block 3
 4- write A block 4
 5- move A -> B
     - link B
     - unlink A
     - set guard on B   <crash>
  - replay 3, 4, 5

with the result being B with only half of its content.  The problem is that
we destroyed the old link _and_ didn't guard the new content.  Instead,
set the guard before the link, and replay the unlink step here
unconditionally.

Fixes: #2178
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: fix collection_add guard

If we crash between the link() and setting the guard, we will get
EEXIST.  Tolerate that.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: fix collection_rename guard

If we crash between the rename and setting the guard, we can get EEXIST
or ENOTEMPTY on rename.  Tolerate that.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: fix fd leak on collection_rename

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: cleanup: flip sense of replay guard check

The other are all if (_check_replay_guard(..)) do_it;.  Make this one
match.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: implement collection_move() as add + remove

This ensures we get add and remove steps with different spos values, which
makes the guard work.  The collection_move implementation breaks on replay
because those values match, so the just-set guard prevents unlink from
happening.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: replay collection_move using add+remove

This approximates the buggy collection_move.  It is still buggy.  It is
only there to replay old journals.

Rip out buggy (and now unused) collection_move code.

For the record, the problem there is that a crash between setting the guard
and unlinking the old name will not remove the old name on replay because
the guard for the link stage is indistinguishable from that for the unlink
stage.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: fix collection_add argument names

No functional changes, just fixing and clarifying argument names so that it
is less confusing/wrong.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: simple failure injections via --filestore-kill-at <n>

This will make filestore suicide (_exit(1)) on the n'th potential failure
call site.  We can potentially fail:

     - before a transaction
     - between each op
     - at the end

Additionally, we instrument the guards:

     - before/after/inside _set_replay_guard
     - between significant steps of callers of _set_replay_guard

All instrumentation points are inside _do_transactions(), so if everything
is done in a single sequencer (or from a single thread) the failure
point is deterministic.

That said, use an atomic so we will still reliably fail (at some point)
when there are multiple filestore threads in action.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: two-phase guard

For certain operations (collection_add) we need a two-phase guard, and an
"in-progress" state.

 - before exposing an object in a new location, we need to mark it so that
   old operations affecting the target name don't touch the new object.
 - can't just set the guard before starting or else we can't distinguish
   between a collection_add that was in-progress and one that happend a
   long time ago.

We may need the same for collection_rename().

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
TestFileStoreState: Represent a FileStore's state to be used by tests.

Instead of having each test creating the same representation of a
FileStore's state, with a map/set of collections and objects, as well as
multiple init() functions for each test that are in all similar in
nature, provide this in a single class that can be inheritted by test
classes.
DeterministicOpSequence: Generate a reproducible sequence of operations.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
VerifyFileStore: Check if two FileStore's match after applying a set of operations.

With DeterministicOpSequence we are able to reproduce exactly the same
sequence of operations, over and over. However, if the filestore fails
(e.g., because we injected a failure), we want to check if it is kept
consistent after replaying its journal.

With VerifyFileStore, which extends DeterministicOpSequence, we are able
to bring a brand new filestore to the state the failed filestore would
reach had it not failed. We can then compare to check if the failure
introduced inconsistencies after replaying the journal.

(This is still work in progress and not fully functional)

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
test_idempotent_sequence: Generate a reproducible sequence of txs.

With this test we aim at reproducing the same sequence of transactions
as long as we are provided with the same seed between runs.

We also allow failures to be injected onto the filestore if the
--filestore-kill-at <VAL> argument is passed, and we provide verification
when --test-verify-at <VAL> is provided.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
deterministicopseq: add collection_rename() support
test_idempotent_sequence: no need to reinject value that is already there

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
filestore: name internally

We need to allow the perfcounter name to be controlled so that we can have
two instances of FileStore in the same process that don't step on each
other.  Default to 'filestore'.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
test_idempotent_sequence: Add commands and lose a couple of optional args.

Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
filestore: init filestore_kill_at in ctor

Otherwise we don't get the option for FileStore instances created after
common_init_finish() (which does md_config_t::call_all_observers()).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
test_idempotent_sequence: require nonexistent filestore dir

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>