Documentation unclear on Response.elapsed

The documentation says that `Response.elapsed` is "the amount of time elapsed between sending the request and the arrival of the response". However it is very unclear what this actually means - the time to the first byte of the headers, the time to the first byte of the body, the time to the end of the body, etc? What value does it, and the other corresponding values in the `Response.history` list's objects, have if the response involved one or more redirects? Does it change if you access `Response.content`? Does it make a difference if `stream` is set on the request? A great deal more clarification on what this value actually tells you would be very helpful.


Good point!

The most accurate answer is that it's the time taken between passing the request to `httplib` and getting the response from it. This means it's time to the end of the _headers_ and the headers are finished parsing. Therefore, each element in `Response.history` measures its own `elapsed` value. It does not change if you access `Response.content` or set `stream=True`.

We should certainly write this in the documentation. =)


See #2512.


Actually @Lukasa that's not exactly correct. See https://github.com/kennethreitz/requests/pull/2209 for more details.


To be clear, it seems to me that `elapsed` serves almost no purpose at all if it does not work as @Lukasa describes - measuring the total time taken is trivial to do in the application, but measuring the "time to first byte" is impossible if Requests does not provide that information itself.


Maybe I'm going mad, but my reading of #2209 suggests that my description of what happens is accurate, but that it may not be what we _intend_ to happen. Is that correct?


If I remember #2209 correctly, there's a difference in behaviour between behaviour with `stream=True` and `stream=False`. In the former it is as you described, in the latter it's the time until the download finished.


Not according to the code. According to the code the `stream` kwarg has no effect on the `HTTPAdapter`.


This sort of confusion is why I thought it important that the expected behaviour be documented ;-)
TL;DR for the below: it looks like the behaviour changed between versions of Requests and @sigmavirus24 used to be correct but now @Lukasa is correct...

I've set up two URLs. `redirect.php`:

``` php
<? sleep(1); header("Location: /delay.php"); flush(); sleep(2); ?>
<p>Foo</p>
```

`delay.php`:

``` php
<p>Hello</p>
<? flush(); sleep(5); ?>
<p>world!</p>
```

With Requests 2.2.1, this is what happens:

``` python
>>> n=datetime.datetime.now(); r=requests.get("http://localhost/redirect.php"); print(r.history[0].elapsed, r.elapsed, datetime.datetime.now()-n);
0:00:03.023007 0:00:05.010399 0:00:08.049038
>>> n=datetime.datetime.now(); r=requests.get("http://localhost/redirect.php", stream=True); print(r.history[0].elapsed, r.elapsed, datetime.datetime.now()-n);
0:00:01.026764 0:00:00.012600 0:00:03.054618
```

With Requests 2.6.0, this happens:

``` python
>>> n=datetime.datetime.now(); r=requests.get("http://localhost/redirect.php"); print(r.history[0].elapsed, r.elapsed, datetime.datetime.now()-n);
0:00:01.020450 0:00:00.009139 0:00:08.033785
>>> n=datetime.datetime.now(); r=requests.get("http://localhost/redirect.php", stream=True); print(r.history[0].elapsed, r.elapsed, datetime.datetime.now()-n);
0:00:01.029155 0:00:00.013359 0:00:03.044638
```

So with `stream=True` the behaviour is consistent between versions, and matches what @Lukasa describes. With `stream=False`, in 2.2.1 `elapsed` changes from "time to first byte" to "time to last byte", but in Requests 2.6.0 there is no difference.

Can I humbly suggest that this change be regarded as a "bug fix" rather than a "bug to fix"? The new behaviour is more logical, consistent and useful, and to change it would be to remove a feature (i.e. the ability to get TTFB with `stream=False`).


I'm actually curious when and how this changed because it's a change that should have been documented. At the least, we should now close #2209, or provide another attribute to replace the old behaviour to get the time to last byte when `stream=False`.


I guess part of the problem would be, with `stream=True`, Requests simply cannot know the "time to last byte", until the entire body has been read (and in the case of automatically-handled redirects, it may never be available). Add to this the fact that finding "time to last byte" in the application is utterly trivial (as shown in my one-liners above), it might do more harm than good to add a "total_time" value that will only sometimes be available.


> in the case of automatically-handled redirects, it may never be available

To anyone else reading, this is inherently incorrect. `elapsed` is available on every response object and will be TTFB. There attribute will only be correct for that request/response pair though. So a total time with redirects will simply be equivalent to:

``` python
sum((r.elapsed for r in response.history), response.elapsed)
```

We also consume the entirety of each response's body on redirects to prevent a socket leak.


@sigmavirus24 you misread my comment. I said that time to _last_ byte may not be available, not that `elapsed` (time to _first_ byte) may not be available. And yes, that `sum` expression is exactly what I am using in my application to determine TTFB :-)


Final question on this - @Lukasa's documentation change specifies that the DNS-lookup time and socket connect() time are excluded from `elapsed`, which seems peculiar if true. Is it true?


It _may_ be true. It is TTLB for that response. If to get that byte we need to create a socket, then yes, it'll include DNS lookup. If we are able to re-use one, it will not.


That's great, that's what I expected (assuming 'TTLB' was a typo for 'TTFB'). The documentation change is still wrong then I'm afraid :-( But the code behaviour is what I need :-)