antirez/redis — Issue #162: redis-server should deal with rlimits better
Author: davebarr
Created: 2011-10-26 23:43:15+00:00
State: open

This is partly a packaging issue, but also a coding issue. Linux has a default ulimit -n of 1024. For many Redis users, this is insufficient. There are various ugly workarounds that are used to raise the limit. The basic problem is that when the system boots up, root processes get the default ulimit -n of 1024. The only way to fix is is to run ulimit -n in a parent process to raise the limit to something higher. PAM has a way to do this with pam_limits, but this applies only to processes that start from something that can and does use pam_limits.so, which doesn't cover most of the usual start paths (like the system booting).

Method 1:
If you use init scripts that source /etc/default/[service], if you put "ulimit -n 32767" in there, it will raise the limit before starting the daemon (e.g. with start-stop-server). Note that upstart services do not support this, so you need some other method (I say this because Rowan's 2.4 launchpad PPA packages now use upstart).

Method 2:
Hack the startup script itself to run ulimit before starting the daemon. This is less ideal from a packaging perspective, as you are now editing a script that can change from release to release, and is much more OS-specific. Doing this with upstart packages is extremely ugly and prone to breakage, as you need to do something like "exec bash -c "ulimit -n 32767; exec start-stop-daemon ...", as "script start" stanzas run in a different process.

Method 3:
Fix this in the code and set the limit directly with setrlimit(2) based on your config option. This is what most programmers end up doing.

Take a look at memcached or gearmand, for example. The upside of this is that you get full control of your environment and limits within your redis config file. There's no editing things in multiple places. The only other change that needs to be made is you must start the daemon as root so it can do the setrlimit without restriction, then setuid/setgid to your user/group. This means your user/group to run as is now a config setting in your redis.conf and/or a command line flag. This still can be optional of course. If you don't start as root it can still attempt to raise it with setrlimit and spit out a warning if it is unsuccessful. But in either case it's useful to know in the logs on startup what the current file descriptor limit is, without having to look in /proc/PID/limits.