mozilla-services/circus — pull request #610 by suligap (2013-10-26 12:01:01+00:00)
Title: Added so_reuseport option to circus sockets

It would be nice if circus could utilize the SO_REUSPORT socket option for tcp sockets. Present long before in other operating systems this option was [added](http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c617f398edd4db2b8567a28e899a88f8f574798d) to Linux not so long ago.

This report is inspired by [this one](https://github.com/benoitc/gunicorn/issues/598) from gunicorn.

So the example config is (`so_reuseport` added to socket section):

[watcher:dummy]
cmd = python worker.py --fd $(circus.sockets.foo)
use_sockets = True
warmup_delay = 0
numprocesses = 3

[socket:foo]
host = 127.0.0.1
port = 8888
so_reuseport = yes

If `so_reuseport` is set to True and SO_REUSEPORT is available on target platform, circus will create and bind new SO_REUSEPORT socket(s) for every worker it starts which is a user of this socket(s).

for this example setup lsof, netstat output would look like:

$ netstat -tnpl | grep 8888
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 7342/python
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 7341/python
tcp 0 0 127.0.0.1:8888 0.0.0.0:* LISTEN 7340/python

$ lsof -i | grep 8888
python 7340 user 3u IPv4 74754 0t0 TCP localhost:8888 (LISTEN)
python 7340 user 21u IPv4 74754 0t0 TCP localhost:8888 (LISTEN)
python 7341 user 3u IPv4 74758 0t0 TCP localhost:8888 (LISTEN)
python 7341 user 21u IPv4 74758 0t0 TCP localhost:8888 (LISTEN)
python 7342 user 3u IPv4 74762 0t0 TCP localhost:8888 (LISTEN)
python 7342 user 21u IPv4 74762 0t0 TCP localhost:8888 (LISTEN)

On my machine I can tell the difference in how new connections are distributed. When sending sequential requests:
- with SO_REUSPORT on, there is no regularity in which worker will accept the connection.
- with SO_REUSPORT off, workers would accept in round robin fashion

`uid` and `gid` watcher opts work as expected.

Maybe a more elegant solution would be to create and bind the socket after `fork` (before `setuid` and `setgid`) and before `exec`.
But `Popen`'s, `preexec_fn` is of no help here as it does not allow to alter process' args (and we need to obtain new socket's `FD` and pass it to `exec`)

Anyway, I'd love to get some feedback on this PR before continuing (no tests yet).