MB-10171: added capturing of master vbucket _local docs to /diag

Particularly so that we can see xdcr checkpoints.

Change-Id: I658852aa3587541527684049529f4e0c9ee35447
Reviewed-on: http://review.couchbase.org/34028
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
bp:MB-100 use sockets pool for long blocking memcached calls

so far it is just wait_for_checkpoint_persistence but there
will be more of them

Change-Id: Iaeca8739de6358e2309d07c04d2a9a18dca63be6
Orig-Reviewed-on: http://review.couchbase.org/32748
Reviewed-on: http://review.couchbase.org/34029
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
bp:MB-100 allow free form Destination in ns_connection_pool

there's no need to require it to be {Host, Port, Ssl}

Change-Id: Icee603aeab20a5199a640abb2c2ce89dd00b6a8b
Orig-Reviewed-on: http://review.couchbase.org/32747
Reviewed-on: http://review.couchbase.org/34030
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
bp:MB-100 dialyzer fix

Change-Id: Id738748957f12da4accd5a03174dfb18d3531f53
Orig-Reviewed-on: http://review.couchbase.org/32828
Reviewed-on: http://review.couchbase.org/34031
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: don't keep checkpoint status error message

Because it's not used anywhere

Change-Id: I840885e3e2152d05fdf23adef5c7ecc85a936024
Reviewed-on: http://review.couchbase.org/34032
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: killed unused source_seq field

Change-Id: I6ac2110f3f91634cc51f7bffe0e4dd24332c8d34
Reviewed-on: http://review.couchbase.org/34033
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: removed useless misc:flush(checkpoint) calls

Because checkpoint is not plain message but a gen_server:cast

Change-Id: Iba726f68f6bcb0eede22df820be8dd22ca0aa88f
Reviewed-on: http://review.couchbase.org/34034
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: made pre-2.5.1 incoming xdcr checkpointing ineffective

2.5.0 and previous releases have buggy checkpointing. We're
implementing new checkpointing protocol. Which is incompatible with
previous releases.

For previous release we have to pretend that their's checkpointing
protocol "works". Without actually saving checkpoints. Otherwise
previous releases would fail to replicate to 2.5.1+ at all.

Our approach is to simply do nothing on ensure_full_commit and to drop
_local/ docs update on the floor, pretending that it actually
succeeded.

Change-Id: Ib495d3862c810b95c90c585dc7c1c5e79985200d
Reviewed-on: http://review.couchbase.org/34035
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: extracted capi_frontend:verify_bucket_auth

In order to make auth checks from other parts of capi layer.

Change-Id: Icd91a0e394d5e114ee52bbd6dbae1fa8beee13c6
Reviewed-on: http://review.couchbase.org/34036
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: added ns_memcached api for xdcr checkpointing ops

Notably we're adding new implementation of
wait_for_checkpoint_persistence which is using pool of "very long
call" memcached sockets and not usual path via ns_memcached workers.

It's different than plain wait_for_checkpoint_persistence because we
expect greater concurrency of xdcr wait_for_checkpoint_persistence
requests. Also xdcr variant handles retries and externally specified
timeout which normal variant does not. And for 2.5.1 I want to avoid
changing "normal" code.

Change-Id: I5c05b3214fd1ec661bbecaee1bbe45fcdc712fa7
Reviewed-on: http://review.couchbase.org/34037
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: implemented vbuuid creation

The trick we use is to update vbucket database during mccouch
notification. At this time we know that ep-engine does not touch
.couch file.

Notable implementation detail is that we have to manually spawn
duplicate couchdb process for actual update. That's because normal
couchdb instance we have for normal vbuckets refuse any updates (which
is a feature).

Change-Id: I1e245ef0462cfc45d282fe66c78f9a80f97e582c
Reviewed-on: http://review.couchbase.org/34038
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: implemented REST API for incoming xdcr checkpointing

Change-Id: Iacdb11fc021a73bb6e2f5ab92a2f88937fdfcfc0
Reviewed-on: http://review.couchbase.org/34039
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: replaced outgoing xdcr checkpointing implementation

Change-Id: Ia540f770c2daebff386a3bcc6acc782f4a83eb07
Reviewed-on: http://review.couchbase.org/34040
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-10057: stop trying to report checkpoint error messages

Change-Id: If234128166d574573ef3df110c2f5fc098516163
Reviewed-on: http://review.couchbase.org/34041
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: propagate checkpointing errors slightly better

Particularly, propagate real reason of failure rather than useless
generic error message.

Change-Id: I423496398c4315f30e4eca0deb14e9f4ac0b6d7d
Reviewed-on: http://review.couchbase.org/34042
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: expose /diag/eval-able API for resetting vbucket uuids

I.e. for testing and diagnostics/troubleshooting.

Change-Id: I7eac55d4f84d8083fec6f68397e18ae3798d7cb8
Reviewed-on: http://review.couchbase.org/34070
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: expose testrunner api that returns json of checkpoints

I.e. so that checkpoints can be verified from tests

Change-Id: I5011229de5c5e94d425fcab662a3bf4a9294fd51
Reviewed-on: http://review.couchbase.org/34071
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10057: added bucket uuid to remote checkpointing payload

It's better if we can detect speaking to wrong bucket asap.

Change-Id: I39492992c97ac9b3b6c50a58fc8eeeb638fa3d49
Reviewed-on: http://review.couchbase.org/34072
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-10352: handle checkpointing against older nodes specially

Older nodes (>= 2.5.0) don't support new checkpointing protocol. And
because checkpointing code currently crashes replicator if for any
reason checkpointing fails and also because we actually need to detect
crashes of destination, we have to treat older node specially.

Current behavior is to react on 404 from _pre_replicate and grab
remote instance start time as special form of vbopaque.

When it's time to checkpoint we detect that special form of
vbopaque. And we compare previous instance start time and current
instance start time. If they don't match, then we crash replicator (as
per usual xdcr checkpointing behavior). If they do match we "fake"
checkpoint by doing nothing. Faking of checkpoint is necessary because
older nodes cannot reliably support checkpointing.

Change-Id: I531a0baddaa924cee8875a03002ef4cf6f3fe6cb
Reviewed-on: http://review.couchbase.org/34168
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>