basho/riak_core — pull request #314 by jtuple (2013-05-04 10:55:19+00:00)
Title: Implement vnode overload protection

This pull-request provides overload protection for vnodes, limiting the size of a given vnode's mailbox and dropping message if the limit is reached. The overload logic is implemented by the vnode proxy, which decides to either forward the message to the relevant vnode or drop the message -- potentially informing a waiting `riak_client` about the overload condition.

Every time the proxy process sends a message to the vnode, it increments a counter that estimates the size of the vnode's mailbox. If the counter hits the limit, messages are dropped. This limit is configured via `riak_core/vnode_overload_threshold` setting. This estimated mailbox size is periodically adjusted to make it match the actual mailbox size.

After sending a configurable number of messages (determined by `riak_core/vnode_check_request_interval'), the proxy will send a ping message to the vnode. Whenever a pong arrives, the proxy knows all messages sent before the ping have been handled and can adjust the estimate accordingly. In the common case, this ping/pong approach keeps the mailbox estimate up-to-date without ever falling back to the more expensive approach described below.

After sending even more messages (determined by 'riak_core/vnode_check_interval'), the proxy will directly query the size of the vnode's mailbox using `process_info`. This call is potentially expensive. If the vnode is currently running, the proxy will be suspended until the vnode is descheduled.

This pull-request also provides the `dropped_vnode_requests_total` stat, which is incremented anytime a request is dropped.

I will edit this pull-request with a link to the relevant riak_test for this work, which also tests the the related FSM overload protection feature.