basho/riak_core — pull request #415 by slfritchie (2013-10-09 07:06:33+00:00)
Title: Add infrastructure to change the OTP net_kernel's net_ticktime

Changing the net_ticktime on a single node is easy: it's a single
function call. However, in a cluster environment, that function
must be run on all nodes simultaneously, including any new nodes
that connect to the cluster during the time change ... and that
includes any hidden nodes.

In Riak's case, those hidden nodes are usually `nodetool`, which
is an escript. Those scripts don't have access to `riak_core_util.erl`
or to any other module module that a fun sent over-the-network
might refer to. So there's this icky dance of checking the remote
node's code server path and appending to the code path if the
riak_core dir is missing.

There is additional icky spawns and timeout checking and exit()
use to avoid: 1. being blocked by the local net_kernel when trying
to use `rpc:call/4` or `spawn/2` to a remote node that's down/slow,
and 2. leaking zillions of processes during such net_kernel instability.

Yes, there's an accompanying riak_test test.