don't try to delete vbuckets on undefined nodes

Change-Id: I621ad219b65aede966cd10033b23fc9a69af5f28
just activate new master when old master is missing

See also http://review.couchbase.org/14315

Change-Id: I76b39705f5d79ae99dc1e8003350ad71be98a9af
implemented master events keeping & streaming facility.MB-5018

In order to observe & verify cluster-wide operations we need some way
to get notified when something notable happens.

We now have this.

By doing GET for /diag/masterEvents on master node you'll get back infinite stream of
json events. There will be one line per json event with lines
separated by '\n'.

Events streamed are:

* node became master
  {"type":"becameMaster","ts":"1333576032.53991","node":"'n_0@10.17.2.163'","host":"10.17.2.163:12000"}

* ebucketmigrator started
  {"vbuckets":[85],"checkpoints":{"85":0},"username":"default","type":"ebucketmigratorStart","ts":"1333576052.81971","takeover":"false","src":"10.17.2.163:12000","pid":"<13013.532.0>","node":"'n_2@10.17.2.163'","name":"replication_building_85_'n_2@10.17.2.163'","dst":"10.17.2.163:12004","bucket":"undefined"}

* ebucketmigrator died
  {"type":"ebucketmigratorTerminate","ts":"1333576047.859331","reason":"shutdown","pid":"<13014.432.0>"}

* vbucket state changed by ns_server
  {"vbucket":"0","type":"vbucketStateChange","ts":"1333576042.549572","state":"active","host":"10.17.2.163:12000","bucket":"default"}

* bucket created
  {"type":"createBucket","ts":"1333582750.502623","bucketType":"memcached","bucket":"mcd","params":{"sasl_password":"","ram_quota":"104857600","auth_type":"sasl"}}

* bucket deleted
  {"type":"deleteBucket","ts":"1333582734.895535","bucket":"mcd"}

* rebalance started
  {"type":"rebalanceStart","ts":"1333576051.72607","pid":"<0.533.0>","keepNodes":["10.17.2.163:12000","10.17.2.163:12002","10.17.2.163:12004"],"ejectNodes":[],"failedNodes":[]}

* rebalance completed/failed/stopped
  {"type":"rebalanceEnd","ts":"1333582913.797543","reason":"normal","pid":"<0.782.0>"}

* vbucket map or fast-forward map changed (with event per vbucket
  changed)
  {"type":"resetFastForwardMap","ts":"1333576047.79036","bucket":"default"}
  or
  {"vbucket":"43","type":"updateMap","ts":"1333576047.814047","bucket":"default","chainBefore":["10.17.2.163:12002","10.17.2.163:12004"],"chainAfter":["10.17.2.163:12002",""]}
  or
  {"vbucket":"36","type":"updateFastForwardMap","ts":"1333576052.76132","bucket":"default","chainBefore":[],"chainAfter":["10.17.2.163:12000","10.17.2.163:12002"]}

* vbucket mover spawned
  {"vbucket":"43","type":"vbucketMoverStart","ts":"1333576052.782249","pid":"<0.592.0>","node":"'n_1@10.17.2.163'","bucket":"default","oldChain":["10.17.2.163:12002",""],"newChain":["10.17.2.163:12002","10.17.2.163:12004"]}

* vbucket move completed
  {"vbucket":"43","type":"vbucketMoveDone","ts":"1333576052.926554","bucket":"default"}

* node failed over
  {"type":"failover","ts":"1333576047.785782","host":"10.17.2.163:12004"}

* tap name deregister
  {"type":"deregisterTapName","ts":"1333584275.53477","pidNode":"'n_0@10.17.2.163'","pid":"<0.9312.0>","name":"replication_building_1023_'n_2@10.17.2.163'","host":"10.17.2.163:12002","bucket":"default"}

Here's common types of fields:

* type - type of event

* ts - event timestamp as seconds since Unix epoch

* host - memcached host/port for node

* node - erlang atom of node

* vbucket - vbucket number

* bucket - bucket name (note for ebucketmigrator events bucket is
  usually undefined and real bucket name is username)

* pid - pid of process

Implementation approach is to have gen_event (registered locally under
master_activity_events_ingress) on all nodes and have note_XXX calls
do notify to this process on master nodes. We expect in most cases
it'll be local notify calls. In order to mitigate any basd
side-effects of discovering who is current master and doing this
potentially remote sends, each note_XXX is actually queuing this work
to per-node work queue service (named master_activity_events_srv).

On master_activity_events_ingress there are only one listener that's
adding timestamps to all events and forwards them to another local
gen_event server. This one is our main one and is locally registered
as master_activity_events.

Additionally there's simple events archiver process that keeps last 4k
events and events streaming function
(master_activity_events:stream_events/2) which will invoke passed
callback for all recently archived events and all future events. This
function normally doesn't return.

Change-Id: Ic85da9e306bbe1620e0b58b9901f3bf2a4da03d3
built simple quote transform to be able to eval lambdas remotely

Change-Id: Iac9fe81d820e1aed9fa399b6d373311593cff55d
fixed ebucketmigrator_srv typo

Change-Id: I1ea57c22793f43a045a42008222c7995537b6a79
implemented ns_process_registry

Which is a simple registry that maps arbitrary terms to processes and
automatically cleans up entries for dead processes.

Change-Id: I9f50de502cebab223451fcfa9f83535c8592a80d
return sane value from ebucketmigrator_srv:confirm_sent_messages

So that callers can know if that indeed was successful

Change-Id: I25e1abf000297bea012a11be0c8dfbe982f7762b
only reuse tap name when changing vbucket filter.MB-4366

Now for all ebucketmigrators we're killing named tap before
establishing connection, except as noted below.

Basically we're trying to avoid named tap reuse as much as
possible. And there's only one case where it's worthwhile reusing
it. That is our quite frequent change of vbucket filter for most
replication connections during rebalance. If we're not using named tap
and only relying on checkpoints, it'll cause excessing resending of
already received data, because our checkpoints are normally quite
large.

The approach is the following:

* when we detected vbucket filter may be changed

* we send erl_eval lambda (via rpc:call) to node that hosts some
  ebucketmigrator

* that lambda checks if it support this node supports vbucket filter
  change and if it does calls local vbucket filter change code

* that code spawns separate process where it's work is performed

* that process is registered in vbucket_filter_changes_registry. This
  process represents 'vbucket filter change transaction'. And
  everything is linked carefully so that any crash will not leave any
  resources orphan.

* vbucket filter change then proceeds with finding current
  ebucketmigrator for stream we want change filter for

* it then calls start_vbucket_filter_change/1 on it

* which is handled inside ebucketmigrator by creating new downstream socket

* checking if current downstream socket is alive (by use of confirm_sent_messages)

* and if all is well replying with that new downstream, original
  ebucketmigrator enters 'zombie' state then where it'll only handle
  EXIT messages

* vbucket filter changing code then shuts down original
  ebucketmigrator process through it's supervisor (terminate & remove
  child)

* it then spawns new child for which it adds extra option:
  downstream_retriever

* function that's value of this option is called by new
  ebucketmigrator to check if downstream is being passed to it

* implementation of this function then consults
  vbucket_filter_changes_registry and if there's still 'transaction
  process', it is asked for downstream via gen_server:call

* if that succeeds new ebucketmigrator is then using it instead of
  opening it's own downstream _and_ only in this case tap name is not
  killed before sending tap_connect command

As can be seen this is quite complicated procedure. Why?

* erl_eval lambda sending trick is part of our handling of old
  nodes. Old nodes _will_ reuse named tap and quite obviously do not
  support special vbucket filter change operation. For them we fall
  back to old sequence of stopping child and starting it back (with
  updated vbucket filter).

* we need to handle ns_vbm_sup restarting our ebucketmigrators with
  same arguments we used. Thus instread of just passing downstream as
  one of options/parameters we're using that trick with process
  registry.

* also because vbucket filter is used as part of process's child_id
  inside vbm_sup, we seemingly really need this dance with shutting
  down old process and creating new

* and we're creating downstream in old process because:

  a) we cannot afford, even temporarily, two ebucketmigrators for same
  name. Otherwise some crash in the middle may cause supervisor to
  actually (re)spawn them both.

  b) we want to make sure nothing can happen with downstream memcached
  in between

  So we have this complex procedure of creating new downstream while
  old downstream is alive and only after old downstream passed check
  we're passing new dowsntream up where it's then passed into new
  vbucketmigrator

Change-Id: I36fd85c25d5c36c96f822dc2dbb8169f2d683dae
CRAP

Change-Id: Iac47839b8521e6950be49fdf8fd86f599a5dc1e5