requests has poor performance streaming large binary responses

https://github.com/alex/http-client-bench contains the benchmarks I used.

The results are something like:

|  | requests/http | socket |
| --- | --- | --- |
| CPython | 12MB/s | 200MB/s |
| PyPy | 80MB/s | 300MB/s |
| Go | 150MB/s | n/a |

requests imposes a considerable overhead compared to a socket, particularly on CPython.


That overhead is unexpectedly large. However, avoiding it might be tricky.

The big problem is that we do quite a lot of processing per chunk. That's all the way down the stack: requests, urllib3 and httplib. It would be extremely interesting to see where the time is being spent to work out who is causing the inefficiency.


guess a next step would be to try profiling httplib / urllib3 to see the
performance there?

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Thu, Dec 4, 2014 at 5:01 PM, Cory Benfield notifications@github.com
wrote:

> That overhead is unexpectedly large. However, avoiding it might be tricky.
> 
> The big problem is that we do quite a lot of processing per chunk. That's
> all the way down the stack: requests, urllib3 and httplib. It would be
> extremely interesting to see where the time is being spent to work out who
> is causing the inefficiency.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65732050
> .


Just ran benchmarks with urllib3:

PyPy: 120MB/s
CPython: 70MB/s

And I re-ran CPython + requests: 35MB/s

(My machine seems to be experiencing a far bit of noise in benchmarks, if anyone has a quieter system they can these on, that'd be awesome)


I tried running these on my machine after shutting down every other
application & terminal window and got a fair amount of noise as well - the
socket benchmark was anywhere from 30mb/s to 460mb/s.

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Thu, Dec 4, 2014 at 9:24 PM, Alex Gaynor notifications@github.com
wrote:

> Just ran benchmarks with urllib3:
> 
> PyPy: 120MB/s
> CPython: 70MB/s
> 
> And I re-ran CPython + requests: 35MB/s
> 
> (My machine seems to be experiencing a far bit of noise in benchmarks, if
> anyone has a quieter system they can these on, that'd be awesome)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65748982
> .


I made the benchmarks easier to run now, so other folks can hopefully verify my numbers:

CPython:

```
BENCH SOCKET:
   8GiB 0:00:22 [ 360MiB/s] [======================================================>] 100%
BENCH HTTPLIB:
   8GiB 0:02:34 [53.1MiB/s] [======================================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:30 [90.2MiB/s] [======================================================>] 100%
BENCH REQUESTS
   8GiB 0:01:30 [90.7MiB/s] [======================================================>] 100%
BENCH GO HTTP
   8GiB 0:00:26 [ 305MiB/s] [======================================================>] 100%
```

PyPy:

```
BENCH SOCKET:
   8GiB 0:00:22 [ 357MiB/s] [======================================================>] 100%
BENCH HTTPLIB:
   8GiB 0:00:43 [ 189MiB/s] [======================================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:07 [ 121MiB/s] [======================================================>] 100%
BENCH REQUESTS
   8GiB 0:01:09 [ 117MiB/s] [======================================================>] 100%
BENCH GO HTTP
   8GiB 0:00:26 [ 307MiB/s] [======================================================>] 100%
```


Uh...those numbers are weird. CPython's httplib is slower than requests or urllib3, even though both libraries use httplib? That just cannot be right.


They reproduce consistently for me -- can you try the benchmarks and see if
you can reproduce? Assuming you can, do you see anything wrong with the
benchmarks?

On Fri Dec 05 2014 at 11:16:45 AM Cory Benfield notifications@github.com
wrote:

> Uh...those numbers are weird. CPython's httplib is slower than requests or
> urllib3, even though both libraries use httplib? That just cannot be right.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65821989
> .


I'm just grabbing a known-quiet machine now. Should take a few minutes to become available because it's a physical box that has to get installed (god I love MAAS).


`CPython 2.7.8`

```
BENCH SOCKET:
   8GiB 0:00:26 [ 309MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:02:24 [56.5MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:42 [79.7MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:01:45 [77.9MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:27 [ 297MiB/s] [================================>] 100%
```


For what it's worth:

[This patch](https://gist.github.com/frewsxcv/1c0f3c81cda508e1bca9), `CPython 3.4.2`:

```
BENCH SOCKET:
   8GiB 0:00:27 [ 302MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:00:53 [ 151MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:00:54 [ 149MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:00:56 [ 144MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:31 [ 256MiB/s] [================================>] 100%
```


You should be able to get that same effect on Python2 with
`env PYTHONUNBUFFERED=` or the `-u` flag.

On Fri Dec 05 2014 at 11:42:36 AM Corey Farwell notifications@github.com
wrote:

> For what it's worth:
> 
> This patch https://gist.github.com/frewsxcv/1c0f3c81cda508e1bca9, CPython
> 3.4.2:
> 
> BENCH SOCKET:
>    8GiB 0:00:27 [ 302MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:00:53 [ 151MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:00:54 [ 149MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:00:56 [ 144MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:31 [ 256MiB/s] [================================>] 100%
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65826239
> .


@alex Interestingly, neither `env PYTHONUNBUFFERED=` or `-u` has the same effect on Python 2. Results from my machine incoming.


Alright, the below data comes from a machine that is doing nothing else but running these tests. The last test was run with the Python `-u` flag set, and as you can see that flag has no effect.

```
Python 2.7.6
go version go1.2.1 linux/amd64
BENCH SOCKET:
   8GiB 0:00:16 [ 500MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:01:32 [88.6MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:01:21 [ 100MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:21 [ 385MiB/s] [================================>] 100%
```

```
Python 2.7.6
go version go1.2.1 linux/amd64
BENCH SOCKET:
   8GiB 0:00:16 [ 503MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:01:33 [87.8MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:01:22 [99.3MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:20 [ 391MiB/s] [================================>] 100%
```

```
Python 2.7.6
go version go1.2.1 linux/amd64
BENCH SOCKET:
   8GiB 0:00:16 [ 506MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:01:31 [89.1MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:21 [ 389MiB/s] [================================>] 100%
```

These numbers are extremely stable, and show the following features:
1. Raw socket reads are fast (duh).
2. Go is about 80% the speed of a raw socket read.
3. urllib3 is about 20% the speed of a raw socket read.
4. requests is slightly slower than urllib3, which makes sense as we add a couple of stack frames for the data to pass through.
5. httplib is slower than requests/urllib3. That's just impossible, and I suspect that we must be configuring httplib or the sockets library in a way that httplib is not.


FWIW, I just merged adding `buffering=True` from @kevinburke, do your runs
include that?

On Fri Dec 05 2014 at 12:04:40 PM Cory Benfield notifications@github.com
wrote:

> Alright, the below data comes from a machine that is doing nothing else
> but running these tests. The last test was run with the Python -u flag
> set, and as you can see that flag has no effect.
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 500MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:32 [88.6MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:21 [ 100MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:21 [ 385MiB/s] [================================>] 100%
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 503MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:33 [87.8MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:22 [99.3MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:20 [ 391MiB/s] [================================>] 100%
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 506MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:31 [89.1MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:21 [ 389MiB/s] [================================>] 100%
> 
> These numbers are extremely stable, and show the following features:
> 1. Raw socket reads are fast (duh).
> 2. Go is about 80% the speed of a raw socket read.
> 3. urllib3 is about 20% the speed of a raw socket read.
> 4. requests is slightly slower than urllib3, which makes sense as we
>    add a couple of stack frames for the data to pass through.
> 5. httplib is slower than requests/urllib3. That's just impossible,
>    and I suspect that we must be configuring httplib or the sockets library in
>    a way that httplib is not.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65829335
> .


Cory - see the latest version of the bench client which turns on the
buffering=True in httplib (as requests/urllib3 do)

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Fri, Dec 5, 2014 at 10:04 AM, Cory Benfield notifications@github.com
wrote:

> Alright, the below data comes from a machine that is doing nothing else
> but running these tests. The last test was run with the Python -u flag
> set, and as you can see that flag has no effect.
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 500MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:32 [88.6MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:21 [ 100MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:21 [ 385MiB/s] [================================>] 100%
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 503MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:33 [87.8MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:22 [99.3MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:20 [ 391MiB/s] [================================>] 100%
> 
> Python 2.7.6
> go version go1.2.1 linux/amd64
> BENCH SOCKET:
>    8GiB 0:00:16 [ 506MiB/s] [================================>] 100%
> BENCH HTTPLIB:
>    8GiB 0:01:31 [89.1MiB/s] [================================>] 100%
> BENCH URLLIB3:
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH REQUESTS
>    8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
> BENCH GO HTTP
>    8GiB 0:00:21 [ 389MiB/s] [================================>] 100%
> 
> These numbers are extremely stable, and show the following features:
> 1. Raw socket reads are fast (duh).
> 2. Go is about 80% the speed of a raw socket read.
> 3. urllib3 is about 20% the speed of a raw socket read.
> 4. requests is slightly slower than urllib3, which makes sense as we
>    add a couple of stack frames for the data to pass through.
> 5. httplib is slower than requests/urllib3. That's just impossible,
>    and I suspect that we must be configuring httplib or the sockets library in
>    a way that httplib is not.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65829335
> .


Yeah, that fixes the performance behaviour of httplib to make far more sense.

New results and conclusions:

```
Python 2.7.6
go version go1.2.1 linux/amd64
BENCH SOCKET:
   8GiB 0:00:16 [ 499MiB/s] [================================>] 100%
BENCH HTTPLIB:
   8GiB 0:01:12 [ 113MiB/s] [================================>] 100%
BENCH URLLIB3:
   8GiB 0:01:21 [ 100MiB/s] [================================>] 100%
BENCH REQUESTS
   8GiB 0:01:20 [ 101MiB/s] [================================>] 100%
BENCH GO HTTP
   8GiB 0:00:20 [ 391MiB/s] [================================>] 100%
```
1. Raw socket reads are fast (duh).
2. Go is about 80% the speed of a raw socket read.
3. httplib is just under 25% the speed of a raw socket read.
4. urllib3 is about 20% the speed of a raw socket read, adding some small overhead to httplib.
5. requests is slightly slower than urllib3, which makes sense as we add a couple of stack frames for the data to pass through.


So, arguably the real cost here is httplib. Speeding this up requires getting httplib out of the way.

I'm interested to work out what part of httplib is costing us though. I think profiling `bench_httplib.py` is a good next step.


I've ruled out the conversion of the socket to a file object through `socket.makefile` by adding that line to the `bench_socket.py` test, that doesn't slow it down at all. Weirdly, it appears to make it faster.


The answer is almost certainly the transfer-encoding: chunked handling.
See: https://github.com/alex/http-client-bench/pull/6 , switching to
Content-Length on the server produces some unexpected results.

On Fri Dec 05 2014 at 12:24:53 PM Cory Benfield notifications@github.com
wrote:

> So, arguably the real cost here is httplib. Speeding this up requires
> getting httplib out of the way.
> 
> I'm interested to work out what part of httplib is costing us though. I
> think profiling bench_httplib.py is a good next step.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65831653
> .


Interesting.

The chunked handling is almost certainly the problem, and I'm not really surprised that go handles it better, especially as chunked is the default HTTP mode for go.

However, requests being fast than a raw socket is...unexpected!

One thing worth noting: if the socket wasn't decoding the chunked encoding in the previous tests then it got an unfair advantage, as it was actually reading less data than the other methods were! They were all reading the chunked headers as well as the 8GB of data.

This leads to a follow-on question: do we still think all of these methods are actually reading the same amount of data?


Yes, the socket layer was cheating, it didn't decode the chunked metadata,
and technically read a bit less. It was there as a baesline for "how fast
can we read", not to prove anything.

On Fri Dec 05 2014 at 12:33:10 PM Cory Benfield notifications@github.com
wrote:

> Interesting.
> 
> The chunked handling is almost certainly the problem, and I'm not really
> surprised that go handles it better, especially as chunked is the default
> HTTP mode for go.
> 
> However, requests being fast than a raw socket is...unexpected!
> 
> One thing worth noting: if the socket wasn't decoding the chunked encoding
> in the previous tests then it got an unfair advantage, as it was actually
> reading less data than the other methods were! They were all reading the
> chunked headers as well as the 8GB of data.
> 
> This leads to a follow-on question: do we still think all of these methods
> are actually reading the same amount of data?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-65833299
> .


I wouldn't be surprised if this is related to the chunk size that we're reading off the socket at a time. 


Cake for @alex for being super helpful :cake:


@nelhage did some stracing of the various examples (in the transfer
encoding: chunked case) https://gist.github.com/nelhage/dd6490fbc5cfb815f762
are the results. It looks like there's a bug in httplib which results in it
not always reading a full chunk off the socket.

On Mon Dec 08 2014 at 9:05:14 AM Kenneth Reitz notifications@github.com
wrote:

> Cake for @alex https://github.com/alex for being super helpful [image:
> :cake:]
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-66147998
> .


So what we have here is a bug in a standard library that no one is really maintaining? (@Lukasa has at least 2 patch sets that  have been open for >1 year.) Maybe I'll raise a stink on a list somewhere tonight


Someone (I might get to it, unclear) probably needs to drill down with pdb
or something and figure out what exact code is generating those 20-byte
reads so we can put together a good bug report.

On Mon Dec 08 2014 at 9:14:09 AM Ian Cordasco notifications@github.com
wrote:

> So what we have here is a bug in a standard library that no one is really
> maintaining? (@Lukasa https://github.com/Lukasa has at least 2 patch
> sets that have been open for >1 year.) Maybe I'll raise a stink on a list
> somewhere tonight
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2371#issuecomment-66149522
> .


I'll try to fit that in tonight or tomorrow if no one else gets to it.


So, any news on the root cause? What's generating these short reads, and how much does the situation improve without them?


@kislyuk Not as far as I'm aware. Hopefully I'll have some time to chase it down this christmas holiday.


Thanks @Lukasa. I'm dealing with a performance issue where download speed on a chunked response using urllib3/requests is much slower than with curl and other libraries, and trying to understand if this is the culprit.