basho/riak_core — pull request #299 by slfritchie (2013-04-12 15:51:33+00:00)
Title: First draft (3rd edition), ready for some review

Vnode replies always go via reply(), and reply() always uses unreliable
messaging. (As opposed to the usual (and more reliable) send-and-pray
messaging.)

During handoff, all forwarding requests use unreliable vnode master
commands to avoid net_kernel blocking interference.

My testing procedure:

1. Use "make stagedevrel" on box A for nodes 1-4. Configure for
multi-box use (i.e. change all 127.0.0.1 appearances in vm.args and
app.config as appropriate).

2. Use "make stage" on box B for node 5. Configure for multi-box
use.

3. Start all nodes, join them all together, and commit.

4. Run basho_bench on box A, using this config:

{mode, {rate, 50}}.
{report_interval, 1}.
{duration, 30}.
{concurrent, 50}.

{driver, basho_bench_driver_riakc_pb}.
{riakc_pb_ips, [
{{127,0,0,1}, 10017}, %% {Ip, Port}
{{127,0,0,1}, [10027, 10037, 10047]} %% {Ip, Ports}
]}.
{riakc_pb_replies, 1}.
{operations, [{get, 1}, {update, 1}]}.
{pb_timeout_general, 5000}.

{key_generator, {int_to_bin, {uniform_int, 990000}}}.
{value_generator, {fixed_bin, 10}}.

5. On box B:

sh -c 'for i in 1 2 3 4 5; do date; sleep 1 ; ( date ; ifdown eth0 ; sleep 100 ; ifup eth0 ; date ) > /tmp/slf 2>& 1 ; cat /tmp/slf; sleep 70; done'

6. Check the results via:

awk -F, '{if ($5 == 0) { print "ok"; } else { print "error" } }' tests/current/summary.csv | uniq -c | less

7. There's a pretty darn obvious qualitative way to view the results
of the pre-patch and post-patch behavior. The sampling rate used by
the basho_bench config above, i.e., `{report_interval, 1}`, creates
some noise in the step 6 results. So, the best advice that I can give
for looking at the quantative data is to look for ~100 seconds of `ok`
stability after node 1's network interface has been shut off.

7a. Good results from step 6:

1 error
19 ok
3 error
2 ok
4 error
1 ok
57 error (definitely down ifdown iteration 1)
2 ok
1 error
105 ok (about ~100 seconds of goodness after down ifdown iteration 1)
4 error
2 ok
4 error
1 ok
61 error (definitely down ifdown iteration 2)
1 ok
2 error
101 ok (about ~100 seconds of goodness after down ifdown iteration 1)
3 error
2 ok
55 error (definitely down ifdown iteration 3)
2 ok
2 error
112 ok (about ~100 seconds of goodness after down ifdown iteration 1)
4 error
1 ok
58 error (definitely down ifdown iteration 4)
1 ok
3 error
109 ok (about ~100 seconds of goodness after down ifdown iteration 1)
3 error
2 ok
4 error
1 ok
4 error
1 ok
52 error (definitely down ifdown iteration 5)
1 ok
4 error
188 ok (about ~100 seconds of goodness after down ifdown iteration 1)
(end of interface flapping, ignore the quantity here)

7b. Bad results from step 6:

1 error
24 ok
4 error (probably down ifdown iteration 1)
1 ok
10 error (definitely down ifdown iteration 1)
1 ok
46 error
1 ok
7 error
106 ok (about ~100 seconds of goodness after down ifdown iteration 1)
3 error
2 ok
4 error
1 ok
61 error (definitely down ifdown iteration 2 ... but this is
where it becomes much less obvious where ifdown and
ifup events actually happened)
2 ok
5 error
1 ok
21 error
75 ok
4 error
1 ok
5 error
1 ok
4 error
1 ok
46 error (down ifdown iteration 3?)
1 ok
38 error
75 ok
3 error
2 ok
4 error
1 ok
5 error
1 ok
84 error (Huge problem ... Step 7a's results never had an error
rate larger than 61 samples in a row.)
75 ok
3 error
2 ok
5 error
1 ok
57 error (probably down here ifdown iteration 5)
1 ok
32 error (if "57 error" was ifdown, then node 1's interface is
now up, and this error rate is too high.)
974 ok (end of interface flapping, ignore the quantity here)