antirez/redis — Issue #327: maxmemory + evicting policy + slaves = death
Author: antirez
Created: 2012-02-04 07:57:44+00:00
State: open
Labels: 2.4 merge candidate, critical bug, more design needed

Pieter Noordhuis discovered this interesting bug, with the help of an user experiencing strange things in a Redis instance with multiple slaves and maxmemory configured at the same time: when maxmemory was reached the DB on the slave was completely erased (thanks to Pieter, Dane and Jemie for the investigation).

The issue happens for the following reason:
+ Redis reached the configured limit, so it tries to expire keys.
+ Evicting keys turns into explicit DELs sent to slaves, since masters control the eviction of slaves for well known reasons.
+ But this way if there are enough slaves, emitting the protocol in the output buffers will actually take more memory than the amount freed removing keys...
+ So the key eviction process starts to enter into an infinite loop.

Up to a given point the fact that there is a static buffer part in the output queue of every client (including slaves) mitigate this in certain conditions, but once Redis can't use the output buffer but must use the queue of objects the infinite loop is triggered.

I'm working to fix this problem, there are different ways to do this and I'm doing experiments to find the best solution. Among the possible solutions there is to ignore output buffer of slaves when computing max memory, or to call the function that flushes data to the slave while we are inside the eviction loop if we detect that the memory is raising after every deletion instead of decreasing, or to even tolerate the fact that the memory increases inside the loop but up to a certain limit, and break the loop when a few keys are evicted (enough to go under the maxmemory limit, if we subtract the size taken by the slave output buffers).

I'm experimenting with the above solutions, but also improving the memory used generating the protocol by using more shared objects for common parts of the protocol, that helps to make this issue (and the code path resolving it, that should be considered an exception) less likely to be triggered.