wal-e/wal-e — pull request #92 by fdr (2014-02-06 07:11:26+00:00)
Title: Redo questionable the non-blocking-pipe I/O routines

There are a few reasons to revisit this code.

Most distressingly is this:

buf = StringIO(data)

while bytes_written < bytes_total:
try:
# self._fp.write() doesn't return anything, so use
# os.write.
bytes_written += os.write(self._fp.fileno(),
buf.read(PIPE_BUF_BYTES))
except IOError, ex:
if ex[0] != errno.EAGAIN:
raise
sys.exc_clear()

In particular, the cursor-advancing side effect of "buf.read" (which
has side effects) wrapped inside a loop that might retry in event of
receiving EAGAIN from an "os.write". It has now been re-spelled to
avoid this.

Also changed is catching EnvironmentError rather than IOError, because
of a report from Art Fedorov on Macintosh that came
from an OSError which is not caught and retried when it possesses the
errno EAGAIN as one might hope.

Also, the prior code tended to call gevent.socket.wait_read and
gevent.socket.wait_write too often, which may be vastly increasing the
number of system calls. Now those are only called when EAGAIN is
encountered.

Finally, I wonder of the possibility of mistaken increments on the
cursor on the CStringIO object above caused some of the reported
Macintosh deadlock problems, solved by reading the atomic pipe buffer
size from the operating system (see
0dd95420f354b4656d4b7ce5b1ba1720405b908c). If so, there may be a new
opportunity to use larger buffer sizes and reduce overhead.