wal-e/wal-e — pull request #96 by fdr (2014-02-14 00:30:09+00:00)
Title: Take total control of buffering and syscall dispatch for pipes

With this patch, CPU usage is greatly reduced to run WAL-E. Per some
user reports, the CPU usage was high enough to be a bottleneck in
practical use.

The approach for this is documented in pipebuf.py:

Detailed handling of pipe buffering

This module attempts to reduce the number of system calls to
non-blocking pipes via careful control over pipe buffering.

Buffering occurs both within the application and in the kernel.
This module attempts to account for both.

The general approach is to defer defragmentation of memory submitted
to or from the kernel from Python by storing them as fragments in a
deque until the latest possible time. The deque is exploited to
"unget" partial fragments.

In order to make this optimization work well, taking advantage of
the kernel's ability to buffer large amounts of memory is required.
This is, uniformly, the size of system call made to read or write
except in cases like flushing.

This is only supported on Linux but can be ported to other systems
that support a similar facility.

The flushing model is also delayed maximally: it waits until the
number of bytes in the deque are larger than the kernel's pipe
buffer, and then perform kernel flushes of maximum size until what
remains in the buffer is less than the largest possible size that
can be accepted. Inversely, reads always read the maximum amount
from the pipe and store the result in the deque.