Repository: urllib3/urllib3

Issue:
SecureTransport backend can lead to long pauses on Python 3.
Spotted by someone testing out pip with SecureTransport backing via the new urllib3.

The symptom is this: if you take pip from the branch on pypa/pip#4454, install it on Python 3, and run `pip install -I --no-cache-dir django` you'll find it takes a surprisingly long time. If you do the same on Python 2, it takes something around 7 seconds, while on Python 3 it can take more than 40! This is clearly not a good sign.

I've spent some time debugging this, and I have an explanation of what's going on. I'm going to write it out at length to help people understand the moving parts here, and to help understand why I've chosen the fix approach that I have.

The major relevant difference between Python 2 and Python 3 is their approach to buffering inside httplib. Specifically, Python 3 uses the fancy `io.BufferedReader`, while Python 2 does not (because it didn't exist when httplib was originally written). On Python 3, when making a `HTTPResponse.read` call, the objects involved are, in order: `http.client.HTTPResponse`, `io.BufferedReader`, `socket.SocketIO`, `securetransport.WrappedSocket`, and `socket.socket`.

`io.BufferedReader` has an interesting mechanism. It is initialized with a buffer size (by default 8192 bytes). When asked to read some bytes, it checks how many it was asked for. If the size is greater than the buffer, it'll just pass the read straight through and won't bother attempting to do any buffering (this was something of a surprise for me). However, when asked to read *less* than the buffer size it'll attempt to fill its buffer. It does this by passing its own internal buffer to the lower-level `readinto` function and asking for that buffer to be filled. This means that if httplib issues a `read` for fewer than 8192 bytes, in most cases the `io.BufferedReader` will intercept that call and ask for a read of 8192 bytes instead.

Interestingly, in most cases pip will circumvent the buffered reader. The buffering is used a bit early on when reading headers, but once pip starts reading the body it does so by using `iter_content(10240)`, which issues reads sized at 10240 bytes. This, eagle-eyed readers will notice, is larger than the buffer size of the `io.BufferedReader`, and so it will mostly stay out of the way. This changes only once: when reading the last segment of the body. At this time, httplib will notice that a `read` of 10240 bytes was asked for but only, say, 5192 bytes are left in the body, and so it will quietly clamp the read down to 5192 bytes before issuing the read to `io.BufferedReader`. At this point, `io.BufferedReader`, the hero of the day, will think "NOW IS MY TIME" and leap in and quietly *re-expand* the read to 8192 bytes before it passes it further on down the line. This is despite the fact that any *human being* could have told it that there will be no extra data to buffer ("the response is over, goddamn it!").

Now, for other TLS backends this does not matter, but for SecureTransport it does. This is because our ST backend just calls `SSLRead` with the buffer provided by the higher level (which in this case is 8192 bytes long). And, it turns out, that `SSLRead` *also* buffers. That is, when asked to read 8192 bytes, it will damn well do so until it gets a read that times out before *any* data is returned. If even one byte is returned by the timeout then it will keep issuing calls to the read callback until it is done or it gets a true timeout.

Here is the issue. httplib asked for exactly the number of bytes it needed. If `io.BufferedReader` hadn't tried to be helpful by over-reading, `SSLRead()` would have been asked for that many application bytes, and so would have returned directly. But it *wasn't*, and so it elected to keep issuing reads. This meant that after it had read the 5192 bytes httplib needed SecureTransport issued *another* socket read, which waited for the timeout before failing to read and causing `SSLRead` to return with `errSSLWouldBlock`. This is where our long delay happens: this will happen once for each package pip downloads. To put it in a TL;DR kinda way, we have two layers doing buffering, and they're getting in each other's way.

So, how do we fix it? As far as I can tell, there's really only one way: stop the `io` module from doing any buffering when we use SecureTransport. Anything else involves trying to second-guess whether the `io` module is just trying to fill its own buffer or whether it actually needs the buffering. This other approach allows SecureTransport to continue to do the buffering it wants to do, without getting in the way.

The risk here is that there is some code somewhere in httplib that assumes it can get a `BufferedReader` unconditionally. I certainly hope that doesn't exist.


Discussion:

Resolving patch:
diff --git a/urllib3/contrib/securetransport.py b/urllib3/contrib/securetransport.py
--- a/urllib3/contrib/securetransport.py
+++ b/urllib3/contrib/securetransport.py
@@ -685,7 +685,11 @@ def makefile(self, mode, bufsize=-1):
         self._makefile_refs += 1
         return _fileobject(self, mode, bufsize, close=True)
 else:  # Platform-specific: Python 3
-    makefile = backport_makefile
+    def makefile(self, mode="r", buffering=None, *args, **kwargs):
+        # We disable buffering with SecureTransport because it conflicts with
+        # the buffering that ST does internally (see issue #1153 for more).
+        buffering = 0
+        return backport_makefile(self, mode, buffering, *args, **kwargs)
 
 WrappedSocket.makefile = makefile
 


Test patch: