Vnode nonblocking reply, First draft (3rd edition), ready for some review

Vnode replies always go via reply(), and reply() always uses unreliable
messaging.  (As opposed to the usual (and more reliable) send-and-pray
messaging.)

During handoff, all forwarding requests use unreliable vnode master
commands to avoid net_kernel blocking interference.

My testing procedure:

1. Use "make stagedevrel" on box A for nodes 1-4.  Configure for
multi-box use (i.e. change all 127.0.0.1 appearances in vm.args and
app.config as appropriate).

2. Use "make stage" on box B for node 5.  Configure for multi-box
use.

3. Start all nodes, join them all together, and commit.

4. Run basho_bench on box A, using this config:

    {mode, {rate, 50}}.
    {report_interval, 1}.
    {duration, 30}.
    {concurrent, 50}.
    
    {driver, basho_bench_driver_riakc_pb}.
    {riakc_pb_ips, [
                    {{127,0,0,1}, 10017}, %% {Ip, Port}
                    {{127,0,0,1}, [10027, 10037, 10047]} %% {Ip, Ports}
                   ]}.
    {riakc_pb_replies, 1}.
    {operations, [{get, 1}, {update, 1}]}.
    {pb_timeout_general, 5000}.

    {key_generator, {int_to_bin, {uniform_int, 990000}}}.
    {value_generator, {fixed_bin, 10}}.
    
5. On box B:

    sh -c 'for i in 1 2 3 4 5; do date; sleep 1 ; ( date ; ifdown eth0 ; sleep 100 ; ifup eth0 ; date ) > /tmp/slf 2>& 1 ; cat /tmp/slf; sleep 70; done'

6. Check the results via:

    awk -F, '{if ($5 == 0) { print "ok"; } else { print "error" } }' tests/current/summary.csv | uniq -c | less

7. There's a pretty darn obvious qualitative way to view the results
of the pre-patch and post-patch behavior.  The sampling rate used by
the basho_bench config above, i.e., `{report_interval, 1}`, creates
some noise in the step 6 results.  So, the best advice that I can give
for looking at the quantative data is to look for ~100 seconds of `ok`
stability after node 1's network interface has been shut off.

7a. Good results from step 6:

      1 error
     19 ok
      3 error
      2 ok
      4 error
      1 ok
     57 error (definitely down ifdown iteration  1)
      2 ok
      1 error
    105 ok (about ~100 seconds of goodness after down ifdown iteration  1)
      4 error
      2 ok
      4 error
      1 ok
     61 error (definitely down ifdown iteration  2)
      1 ok
      2 error
    101 ok (about ~100 seconds of goodness after down ifdown iteration  1)
      3 error
      2 ok
     55 error (definitely down ifdown iteration  3)
      2 ok
      2 error
    112 ok (about ~100 seconds of goodness after down ifdown iteration  1)
      4 error
      1 ok
     58 error (definitely down ifdown iteration  4)
      1 ok
      3 error
    109 ok (about ~100 seconds of goodness after down ifdown iteration  1)
      3 error
      2 ok
      4 error
      1 ok
      4 error
      1 ok
     52 error (definitely down ifdown iteration  5)
      1 ok
      4 error
    188 ok  (about ~100 seconds of goodness after down ifdown iteration  1)
            (end of interface flapping, ignore the quantity here)

7b. Bad results from step 6:

      1 error
     24 ok
      4 error (probably down ifdown iteration  1)
      1 ok
     10 error (definitely down ifdown iteration  1)
      1 ok
     46 error
      1 ok
      7 error
    106 ok (about ~100 seconds of goodness after down ifdown iteration  1)
      3 error
      2 ok
      4 error
      1 ok
     61 error (definitely down ifdown iteration 2 ... but this is
               where it becomes much less obvious where ifdown and
               ifup events actually happened)
      2 ok
      5 error
      1 ok
     21 error
     75 ok
      4 error
      1 ok
      5 error
      1 ok
      4 error
      1 ok
     46 error (down ifdown iteration  3?)
      1 ok
     38 error
     75 ok
      3 error
      2 ok
      4 error
      1 ok
      5 error
      1 ok
     84 error (Huge problem ... Step 7a's results never had an error
               rate larger than 61 samples in a row.)
     75 ok
      3 error
      2 ok
      5 error
      1 ok
     57 error (probably down here ifdown iteration  5)
      1 ok
     32 error (if "57 error" was ifdown, then node 1's interface is
               now up, and this error rate is too high.)
    974 ok (end of interface flapping, ignore the quantity here)