square/prodeng — pull request #31 by chrislmar (2013-12-05 02:45:55+00:00)
Title: mcmd: close FDs when finished so we don't run out

Close file descriptors (FDs) for the pipes between the MultipleCmd
parent and its children to prevent the parent from exceeding the default
per-process Linux limit of 1024 FDs.

The pipes created for communication between the parent process and each
subprocess are unidirectional. Close the unused ends of the pipes in the
parent and child as soon as a child is spawned.

In the parent, keep track of the other FDs for each subprocesses, and
close the FDs for a subproccess when it's reaped.

MultipleCmd now requires only (3 * maxflight) open FDs instead of
(6 * total number of commands). Given a per-process FD limit of 1024,
the theoretical limit for maxflight is 341, versus the previous limit of
170.