basho/riak_kv — pull request #243 by rustyio (2011-10-27 21:55:30+00:00)
Title: Add siblings and objsize stats (AZ895)

## Overview

This code adds Riak stats for `node_get_fsm_siblings` and
`node_get_fsm_objsize`, and supports hot upgrades (no need to restart
the node.)

## Hot Upgrade

On a running Riak node:

1. Backup the existing files `riak_kv_stat.beam` and
`riak_kv_get_fsm.beam`, found in
`/usr/lib/riak/lib/riak_kv-1.0.1/ebin`.

2. Copy `riak_kv_stat.beam` and `riak_kv_get_fsm.erl` to
`/usr/lib/riak/lib/riak_kv-1.0.1/ebin`.

3. Run `riak attach` to connect to the Erlang shell and paste the
following commands:

```erlang
sys:suspend(riak_kv_stat).
code:purge(riak_kv_stat).
code:load_file(riak_kv_stat).
sys:change_code(riak_kv_stat, riak_kv_stat, {up, siblings_and_objsize_stats}, []).
sys:resume(riak_kv_stat).
code:purge(riak_kv_get_fsm).
code:load_file(riak_kv_get_fsm).
```

4. Press Control-D to disconnect from the Erlang shell.

5. To verify installation, run `curl localhost:8098/stats` and look
for a stat named `node_get_fsm_siblings_total`.

## Hot Downgrade

On a running Riak node:

1. Copy the backed-up versions of `riak_kv_stat.beam` and `riak_kv_get_fsm.beam`
to `/usr/lib/riak/lib/riak_kv-1.0.1/ebin`.

2. Run `riak attach` to connect to the Erlang shell and paste the
following commands:

```erlang
code:purge(riak_kv_get_fsm).
code:load_file(riak_kv_get_fsm).
sys:suspend(riak_kv_stat).
sys:change_code(riak_kv_stat, riak_kv_stat, {down, siblings_and_objsize_stats}, []).
code:purge(riak_kv_stat).
code:load_file(riak_kv_stat).
sys:resume(riak_kv_stat).
```

3. Press Control-D to disconnect from the Erlang shell.

4. To verify, run `curl localhost:8098/stats` and ensure that the stat
named `node_get_fsm_siblings_total` does not exist.

## Rolling Upgrade / Downgrade

On a running Riak node:

1. Backup the existing files `riak_kv_stat.beam` and
`riak_kv_get_fsm.beam`, found in
`/usr/lib/riak/lib/riak_kv-1.0.1/ebin`.

2. Copy `riak_kv_stat.beam` and `riak_kv_get_fsm.erl` to
`/usr/lib/riak/lib/riak_kv-1.0.1/ebin`.

3. Restart the Riak node.

To downgrade, restore the previously backed up .beam files.

## Notes

+ Updated the `riak_kv_stat` module to support `node_get_fsm_siblings` and
`node_get_fsm_objsize` stats, tracking number of siblings and object
size, respectively. Also updated stats module `code_changed/N`
function to enable hot upgrades. (No need to take down the VM if
upgrading from Riak 1.0.1.)

+ Updated the `riak_kv_get_fsm` module to send stats about # of siblings
and object size to the stats module.

+ Noticed that sometimes 99% stats are larger than 100% stats. Assuming
that this is due to a small sample size plus the fact that 99% stats
being estimated while 100% stats are the actual max.

+ Calculating object size by summing the bucket size, key size, using
`term_to_binary/1` on the vector clock, and the metadata and value
sibling. This isn't as exact of a calculation because it doesn't
capture some of the overhead (ie: it doesn't count the size of the
tuple that wraps the metadata and value.) But from what I've seen it's
within 10% of the object size.

+ Tested performance of old and new code via basho_bench. Adding the new
stats had no discernible affect on performance.

## Test Code

```erlang
%% Some rough smoke-testing code. Just copy this code and paste into
%% an Erlang shell. Creates siblings of different sizes, prints out
%% the stats, then pauses for 5 seconds.

%% The CreateSiblings/1 function creates creates an object with the
%% specified number of siblings.
CreateSiblings = fun(N) ->
{ok, Client} = riak:local_client(),
Client:set_bucket(<<"mybucket">>, [{allow_mult, true}]),
Key = erlang:md5(term_to_binary(now())),
Obj = riak_object:new(<<"mybucket">>, Key, <<"1">>),
[Client:put(Obj) || _X <- lists:seq(1, N)],
Client:get(<<"mybucket">>, Key),
ok
end.

%% Print `fsm_siblings` and `fsm_objsize` stats.
PrintStats = fun() ->
%% Use apply so that we don't hold a reference to old code.
Stats = erlang:apply(riak_kv_stat, get_stats, []),
io:format("---~n"),
io:format("~p~n", [lists:keyfind(node_get_fsm_siblings_mean, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_siblings_median, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_siblings_95, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_siblings_99, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_siblings_100, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_objsize_mean, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_objsize_median, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_objsize_95, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_objsize_99, 1, Stats)]),
io:format("~p~n", [lists:keyfind(node_get_fsm_objsize_100, 1, Stats)]),
ok
end.

CreateSiblings(1).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(10).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(100).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(2).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(20).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(200).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(3).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(30).
PrintStats().
timer:sleep(5 * 1000).

CreateSiblings(300).
PrintStats().
```