basho/riak_core — pull request #106 by jtuple (2011-10-31 19:58:10+00:00)
Title: Refactor riak_core vnode management (part 1)

Move vnode spawning and pid tracking logic from riak_core_master to
riak_core_manager. This change sets the stage for future work that will
make the vnodes more pure, with state transitions coordinated by the
vnode manager. A side-effect is that there is now a central process
managing pids for all registered vnode modules, rather than one master
per vnode module.

Add registered and supervised vnode proxy processes. Requests can be
sent directly to a vnode proxy process for a desired node/index/module
and the request will be forwarded to the proper vnode. The proxy process
coordinates with the vnode manager to know the proper pid for routing,
and monitors the vnode pid in order to clean-up on shutdown/crashes. While
vnodes may be spun up and down on demand by the vnode manager, the relevant
proxy processes are always alive and can be counted on for direct routing.

Change riak_core_vnode_master to use the new vnode proxy support in most
cases. All requests that are routed through a vnode master process will
now use the proxy logic to dispatch requests. Likewise, the non-sync API
exposed by the vnode master module no longer routes through the master
itself, but directly dispatches requests to the proxy processes. Sync
commands still route through the master, which then routes through the
proxy in handle_call. A side-effect is that sync commands now require
three local hops (master, proxy, vnode-pid) rather than two (master,
vnode-pid).

A few notes. You need to add {legacy_vnode_routing, false} to the riak_core portion of app.config if you want the direct routing (ie. bypassing vnode_master) behavior. This is necessary for rolling upgrade support. With or with the setting, vnode master will always route through the proxies, but the setting determines if non-sync requests are sent directly to a proxy or first through vnode master (the second case being safe for old nodes that have a vnode master but no proxies).

The changes effect the underlying vnode behavior of riak_core, so for testing I just went with basho_expect using this branch for stage injection. basho_expect individual node tests seem to pass the same as they do on master (ie. the same tests fail that fail on master for me). The cluster tests are hit or miss. A few failed, but I think it may have been an issue with the stage beam clobbering approach. Re-running with a package containing my changes to test things out.

I'll likely add an automated integration test that monitors trace messages to determine that requests are proxied as desired. For now, I've manually done similar with redbug (monitoring master, manager, and proxy process handle_calls/casts) and everything looks good.