Repository: urllib3/urllib3

Issue:
pyopenssl performance hit from issue#412
The fix for issue #412 (25755abf27790118cbe1df9bd3b967085c039e9d) introduced a significant performance hit to large file uploads.

In my benchmarks testing with a 50 MB file:
- curl can upload the file in 0.5s (just for context)
- requests + urllib3 using the current implementation takes over 20 seconds (20-22s)
- requests + urllib3 with that fix reverted (`sendall` instead invokes `self.connection.sendall(data)`) uploads the file in 1.15s

The cost appears to be a result of a combination of two factors:
- OpenSSL will only write up to 16384 bytes at a time (the limit derives from the maximum amount of plaintext in a single SSL record)
- sendall() makes a copy of `data` (removing what was already sent), which is copying the entire payload

So with a 50 MB file, urllib3 is performing over 3000 allocations with an average size around 25 MB.  It gets worse when I try uploading even larger files...

Something like the following restores the performance.  There are still a large number of string copies, but they're of a limited (fixed) size.  I should add that I thought about using a memoryview here instead of the slices, but pyOpenSSL immediately turns around and casts back to bytes, so I didn't see a need for the complexity.

``` python
    def sendall(self, data):
        total_sent = 0
        while total_sent < len(data):
            sent = self._send_until_done(data[total_sent:total_sent+16384])
            total_sent += sent

```

With this implementation, I'm again seeing uploads complete in about 1.15s.

pyopenssl performance hit from issue#412
The fix for issue #412 (25755abf27790118cbe1df9bd3b967085c039e9d) introduced a significant performance hit to large file uploads.

In my benchmarks testing with a 50 MB file:
- curl can upload the file in 0.5s (just for context)
- requests + urllib3 using the current implementation takes over 20 seconds (20-22s)
- requests + urllib3 with that fix reverted (`sendall` instead invokes `self.connection.sendall(data)`) uploads the file in 1.15s

The cost appears to be a result of a combination of two factors:
- OpenSSL will only write up to 16384 bytes at a time (the limit derives from the maximum amount of plaintext in a single SSL record)
- sendall() makes a copy of `data` (removing what was already sent), which is copying the entire payload

So with a 50 MB file, urllib3 is performing over 3000 allocations with an average size around 25 MB.  It gets worse when I try uploading even larger files...

Something like the following restores the performance.  There are still a large number of string copies, but they're of a limited (fixed) size.  I should add that I thought about using a memoryview here instead of the slices, but pyOpenSSL immediately turns around and casts back to bytes, so I didn't see a need for the complexity.

``` python
    def sendall(self, data):
        total_sent = 0
        while total_sent < len(data):
            sent = self._send_until_done(data[total_sent:total_sent+16384])
            total_sent += sent

```

With this implementation, I'm again seeing uploads complete in about 1.15s.



Discussion:

Resolving patch:
diff --git a/urllib3/contrib/pyopenssl.py b/urllib3/contrib/pyopenssl.py
--- a/urllib3/contrib/pyopenssl.py
+++ b/urllib3/contrib/pyopenssl.py
@@ -204,9 +204,11 @@ def _send_until_done(self, data):
                 continue
 
     def sendall(self, data):
-        while len(data):
-            sent = self._send_until_done(data)
-            data = data[sent:]
+        total_sent = 0
+        while total_sent < len(data):
+            # OpenSSL will only write 16K at once
+            sent = self._send_until_done(data[total_sent:total_sent+16384])
+            total_sent += sent
 
     def shutdown(self):
         # FIXME rethrow compatible exceptions should we ever use this


Test patch: