geemus/excon — pull request #87 by evanphx (2011-12-10 21:43:58+00:00)
Title: Don't ignore OpenSSL errors

The current code, given an OpenSSL exception that is not "write would block", swallows the exception, returns the the top of the loop and does the whole thing over again. If the socket error is persistent (common) then this code loops forever chewing up as much CPU as it can.

This problem was experienced in the wild, specifically on rubygems.org with fog (using excon) attempting to upload gems to S3.

The exception message for the error is "SSL_write:: bad write retry". Some investigation shows that this error comes from using OpenSSL in write_nonblock mode and changing the address buffer passed in after a request has indicated it would block on write (which this code definitely does since the #slice will copy and return a different address each time, even if a previous request had been a "write would block" exception.

Here is a python bug that describes the same issue: http://bugs.python.org/issue8240

As a side note, the usage of the @write_buffer in Socket#write causes a huge amount of memory usage when trying to write large payloads (rubygems.org passes a 1M String to #write) because of: 1) the append into the buffer, 2) the slice to pass a chunk, and 3) the slice! to prune the buffer.

I actually found this issue by seeing that it appeared that the process was just continually garbage collecting but in fact what was happening is the loop was doing #slice so often that ever 2 or 3 times through the loop, the GC would run.