bruceg/daemontools-encore — pull request #9 by snakpak (2013-08-03 17:37:41+00:00)
Title: Add fifo program to hold named pipes open for feeding multilog

The primary focus of this update is to resurrect the fifo program, which has been missing from daemontools since version 0.60.

The reason for doing this is to support daemontools users of Nginx who would rather log with DJBWare versus using the developer-recommended cron/logrotate combination.

For background, I'll just ramble on a bit about how I arrived at rewriting fifo.

Nginx is a high performance web server capable of serving tens of thousands of requests per second. In testing I've seen it do 39,000 static pages per second across localhost using less than half the machine. (ApacheBench was using the rest.)

It runs as a 'master' root process controlling a number of reduced-privilege 'worker' processes. The workers write the logs. Somehow though, standard in/out/error of the workers is owned by root, so the accepted practice of piping those to multilog doesn't work. Ditto for /dev/std* and /proc/self/fd/*.

So, since the developers won't support anything other than flat-file logs, that leaves a named pipe. Trying to redirect it into multilog's stdin though also won't work since Nginx opens and closes the log for every entry. As soon as multilog catches up, and it will, it sees that end of input as a signal to shut down, which is probably not a very good idea on a heavily loaded machine.

What's needed is a way to hold the named pipe open for writing by Nginx while at the same time insulating all the open/close action from multilog. Enter fifo...

#!/bin/sh
exec env - PATH=/usr/local/daemontools/bin \
setuidgid loguser \
fifo logpipe \
multilog t s10000000 n400 ./main

Notice that this version of fifo uses a fork/exec model to spawn multilog (or any other program). The 0.53 version had to be explicitly piped to multilog.

The biggest part of all this is naturally the fifo program itself, but along the way I can think of a couple other things that got changed, notably:

* Restore DJB conf-cc plus -Wall and fix warnings.

* Add error_child/ECHILD in error.[c,h]

* Add fifomain.c and build-system mods.

* Add fifo.8.

* Add fifo tests.

More detail is of course available in fifo.8 and if interested some simple benchmarks are in the comments at the bottom of fifomain.c comparing Nginx using fifo/multilog to Nginx logging both to flat file and /dev/null.

Enjoy!