Consecutive requests with Session() raises 404

In [1]: import requests
In [2]: s = requests.Session()
In [3]: s.get('https://www.flavourly.com/start-subscription/12/', verify=False)
Out[3]: &lt;Response [200]&gt;
In [4]: s.get('https://www.flavourly.com/start-subscription/12/', verify=False)
Out[4]: &lt;Response [404]&gt;

Is it me or is this a bug? I've checked and it only happens in versions higher than 2.3.0.


I don't think it's a bug, I think it's because of cookies. Flavourly doesn't seem to like the cookies we're sending back. Try checking the difference between the value of `s.cookies` after the first request, once in something where you don't see the 'bug', and once in a version where you do.


In fact, I think the bug is on Flavourly's end, because the history is this:

``` python
>>> import requests
>>> s = requests.Session()
>>> r1 = s.get('https://www.flavourly.com/start-subscription/12/')
>>> r1.history
[<Response [302]>, <Response [301]>]
>>> for r in r1.history:
...     print r.url
... 
https://www.flavourly.com/start-subscription/12/
http://www.flavourly.com/signup/
>>> print r.url
http://www.flavourly.com/signup/
```


But the second time around, they 404 on the signup URL:

``` python
>>> r2 = s.get('https://www.flavourly.com/start-subscription/12/')
>>> for r in r2.history:
...     print r.url
... 
https://www.flavourly.com/start-subscription/12/
>>> print r2.url
https://www.flavourly.com/signup/
```


Oh, hey, look! For the second time my predictions about #2095 have come true and it's caused us problems:

```
import requests
>>> s = requests.Session()
>>> class NullDict(dict):
...     def __setitem__(self, key, value):
...         return
... 
>>> s.redirect_cache = NullDict()
>>> s.get('https://www.flavourly.com/start-subscription/12/')
<Response [200]>
>>> s.get('https://www.flavourly.com/start-subscription/12/')
<Response [200]>
```


So I retract my position, this is an interaction of a requests behaviour and a Flavourly behaviour. They 301 from http to https, which we cache. We're allowed to do that, that's what 301 means, as you can see [from the spec](https://tools.ietf.org/html/rfc7231#section-6.4.2):

> The 301 (Moved Permanently) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs. Clients with link-editing capabilities ought to automatically re-link references to the effective request URI to one or more of the new references sent by the server, where possible.

However, that seems to fall over with this particular redirect. That's weird because the 301 doesn't actually set any cookies or anything, and we don't send it any cookies either (because they stored their cookie as 'secure').


Ok, the problem is cookie handling. This request came from one with the redirect cache turned off:

``` python
>>> r.request.headers['Cookie']
'csrftoken=83Tg47VB3OtXBLsIQa3PZg5NEJFH5dhT; sessionid=j59ad65zb2lzfviux37kwxqjipytqthr'
```

And this one from one with the redirect cache turned on:

``` python
>>> r2.request.headers['Cookie']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'cookie'
```

Here are the two session cookiejars:

``` python
>>> s.cookies
<RequestsCookieJar[Cookie(version=0, name='csrftoken', value='83Tg47VB3OtXBLsIQa3PZg5NEJFH5dhT', port=None, port_specified=False, domain='www.flavourly.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=1452805132, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='sessionid', value='j59ad65zb2lzfviux37kwxqjipytqthr', port=None, port_specified=False, domain='www.flavourly.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=1422565132, discard=False, comment=None, comment_url=None, rest={'httponly': None}, rfc2109=False)]>
>>> s2.cookies
<RequestsCookieJar[Cookie(version=0, name='csrftoken', value='Pja7YWnMlcw4qtrvvf2I6BbeuV3S6iGX', port=None, port_specified=False, domain='www.flavourly.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=1452805188, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='sessionid', value='222eih12j1h8wr3c3n93pvza8hkg736e', port=None, port_specified=False, domain='www.flavourly.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=1422565189, discard=False, comment=None, comment_url=None, rest={'httponly': None}, rfc2109=False)]>
```


Ugh, FFS, I've found it.

The problem is that we handle the redirect cache in `Session.send` with this logic:

``` python
        checked_urls = set()
        while request.url in self.redirect_cache:
            checked_urls.add(request.url)
            new_url = self.redirect_cache.get(request.url)
            if new_url in checked_urls:
                break
            request.url = new_url
```

However, this is well _after_ we've decided what the cookies should be (in `Session.prepare_request`). This means that the cookies are chosen based on the original URL, but the actual URL hit is potentially redirected dramatically.

This is a definite bug introduced by #2095. Apologies for my earlier comment @RossLote, you're quite right that this is a bug.


@RossLote Until this gets fixed, feel free to use the workaround I posted above. It has no negative side-effects aside from occasionally slightly slowing your code down.


Alright, @sigmavirus24, I've thought about this overnight and I can't think of a good fix that uses the current code and doesn't commit wild layering violations. So here's my proposal.

We should rewrite the redirect cache.

The core problem is, fundamentally, that the redirect cache's implementation is misguided. I've come to this position because of closer reading of RFC 7231. It has the following two stanzas [under the 301 code](https://tools.ietf.org/html/rfc7231#section-6.4.2) that I find to be interesting:

> [A]ny future references to this resource ought to use one of the enclosed URIs. Clients with link-editing capabilities ought to automatically re-link references to the effective request URI to one or more of the new references sent by the server, where possible

and

> A 301 response is cacheable by default; i.e., unless otherwise indicated by the method definition or explicit cache controls.

This is a weird combination of text. If the 301 is supposed to cause a rewrite of the URL, why would I cache it?

My suspicion is that the correct flow in requests is actually not a 'redirect' cache, but a 301 cache, that literally returns 301 responses without contacting the network. This will lead to the correct flow in requests: everything looks exactly the same from the perspective of the upper layers of code, the response just came in really fast.

This is backed up by examining what Chrome does, which is serve 301s from its cache (click to see better):

![301](https://cloud.githubusercontent.com/assets/1382556/5773563/dc5f2bdc-9d5b-11e4-8581-5f83e76d8526.png)

I think now, as I thought then, that the 301 cache should not have been merged in its proposed form. I now think more strongly than that: we have to change it. I have three options on the table:
1. Remove the redirect cache entirely. People who want that function should use [CacheControl](https://github.com/ionrock/cachecontrol), as we have said many, many times. This will lead to the flow through requests I mentioned above, resolve this bug (and some others that are inevitably lying around), and provide additional benefits _on top_ of the standard redirect cache.
2. Replace the redirect cache with a special-case cache for 301s that operates at the `HTTPAdapter` layer. This is tricky, because caching 301s needs to obey all the rules for caching (another thing our 'redirect cache' doesn't do), including `Cache-Control` and `Vary` headers. We'd probably need to write about 30% of the CacheControl library, which raises my third option;
3. Bring CacheControl (or someone else) into the fold. Bless a requests caching solution as 'the one true way' to cache and bring it in tree, or use it as an external `pip` dependency, but include it by default. This will fix our 301 bugs and give us caching out-of-the-box by default.

Thoughts would be extremely appreciated at this time. Especially from @sigmavirus24, @kennethreitz, @shazow, @ionrock. Note also the discussion that occurred in #2095: my reading of it suggests that @johnsheehan, @dstufft, and @dknecht favoured my option 1, and @dolph favoured something closer to my option 3 (or possibly 2).


Option 1 and 3 seem the most reasonable to me. I agree with your reading of the spec. The redirect cache is really an operation that is implicitly dependent on the client having some caching implementation including both storage and the HTTP algorithms. Unfortunately, as I have a feeling that I assumed requests was responsible for it, I don't have support for caching 301s in CacheControl. That should obviously change before recommending CacheControl handle this aspect. 

Also, to specifically address option 3, I'm fine to merge or include CacheControl as a dependency. I'm happy to support the code no matter the decision there. Between to the two, I suspect that adding a dependency would be the better option, but in terms my stamp of approval, I'd support either option. Obviously, no big deal if option 3 doesn't happen either. In other words, I'm easy to please! :)


I'm in favor of option 1. If there's a bunch of complexity to setting up CacheControl to handle this case (which is @Ionrock's prerogative) we can put the common code in requests-toolbelt as necessary. I'm fine having a dependency in there on CacheControl. If we do choose option 3, I'd much rather see CacheControl as an outside dependency. Development on that moves quickly enough that it makes sense to allow people to upgrade as necessary rather than bundling it.

Frankly, I'm worried about removing the redirect cache haphazardly though. Perhaps the best first option would be to use a `NullCache` for v2.6.0 of requests. This means people will have to specifically turn it on for it to work (and thereby cause issues). We would document how to use `urllib3`'s `RecentlyUsedContainer`. I suspect the problem we'll have is with pickling again. We can serialize a `NullCache` as a special sentinel value. Then we'll have to change https://github.com/kennethreitz/requests/blob/d2d576b6b1101e2871c82f63adf2c2b534c2dabc/requests/sessions.py#L674 to check the popped value and determine if it's the sentinel. If it isn't, we'll use a `RecentlyUsedContainer`.

It's extra work but it will provide a level of backwards compatibility until we can break this out in 3.0. In documenting how to enable this, we should add the warning that it is a slipshod implementation of the cache that has been known to cause problems. It should also be noted that it is considered deprecated functionality and will be removed in 3.0. When it's available in CacheControl, we should then start pointing towards that as a real solution that is correct and more reliable than using the redirect cache.

How does this sound to everyone?


I've updated CacheControl to support 301 and 300 status codes. 300s are dealt with, more or less, like a normal response, where 301s are cached by default, while adhering to caching headers. 

I looked into the other 3xx headers and it seems the spec specifically provides guidance regarding caching by mentioning it specifically in its description. If others have different perspectives let me know. 

If folks want to test it out to verify it fufills the same results as the existing redirect cache, just checkout master and give it a try. Let me know if you find any issues. 


I'm having a similar problem as the original report here.  In my case the subsequent requests are returning a 400 rather than 404.

```
$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> auth
('xxxxx', 'xxxxxx')
>>> s = requests.session()
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [200]>
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [400]>
```

Subsequent requests are OK when not using a session:

```
>>> r = requests.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [200]>
>>> r = requests.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [200]>
>>>
```

And it works OK when using a session without the auth:

```
>>> s = requests.Session()
>>> r = s.get(url=url+"/changes/")
>>> r
<Response [200]>
>>> r = s.get(url=url+"/changes/")
>>> r
<Response [200]>
>>>
```

I've tried the `NullDict` workaround mentioned above, but that didn't make any difference.


@dpursehouse What's in the response history for each response?


@Lukasa the history seems to be empty for both responses

```
>>> s = requests.Session()
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [200]>
>>> r.history
[]
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [400]>
>>> r.history
[]
>>> 
```


I've also tried using the latest head of the master branch (previously using the released 2.7.0) and get the same results.


@dpursehouse That strongly suggests one of two things:
1. The connection is setting a cookie that it then does not expect to receive back. Unlikely.
2. The remote server doesn't like connection reuse. _Way_ more likely. Particularly if it's behind HAProxy, which can get all kinds of sad here.

Can you try mounting a HTTPAdapter with a poolsize of 1?

``` python
from requests.adapters import HTTPAdapter
import requests

h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
s = requests.Session()
s.mount('https://gerrit-review.googlesource.com', h)

# Make your requests here
```


Same result :(

```
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> from requests.adapters import HTTPAdapter
>>> h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
>>> s = requests.Session()
>>> s.mount('https://gerrit-review.googlesource.com', h)
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [200]>
>>> r = s.get(url=url+"/a/changes/", auth=auth)
>>> r
<Response [400]>
>>> 
```


Ah, crap, sorry, change the 1s to 0s. Sometimes I forget how to urllib3. =D


After changing them to 0 I get this exception on the `get` call:

```
requests.packages.urllib3.exceptions.ClosedPoolError: HTTPSConnectionPool(host='gerrit-review.googlesource.com', port=443): Pool is closed.
```


BTW in case it's relevant, on the call to `get` that succeeds, it emits this:

```
ENV/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
```

The subsequent failing one does not.

I omitted that from my previous logs because I was too lazy to edit out only the sensitive parts of my path and just removed the whole line.


That's to be expected: we only emit the InsecurePlatformWarning once.

Ok, new plan: can we do this instead?

``` python
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> from requests.adapters import HTTPAdapter
>>> h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
>>> s = requests.Session()
>>> s.mount('https://gerrit-review.googlesource.com', h)
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
```

Here we're trying not to consume the response, which should ensure the pool is empty and you get a new connection.


This time it emits the `InsecurePlatformWarning` on both requests, but the second request still returns 400.

```
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> from requests.adapters import HTTPAdapter
>>> h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
>>> s = requests.Session()
>>> s.mount('https://gerrit-review.googlesource.com', h)
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [200]>
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [400]>
>>>
```


Interesting. That suggests a cookie problem. Can you print `r.request.headers` for me in both cases?


Looks like there's a cookie in the second one that wasn't in the first.  Note that I've redacted the value of the cookie and the auth headers :)

```
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> from requests.adapters import HTTPAdapter
>>> h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
>>> s = requests.Session()
>>> s.mount('https://gerrit-review.googlesource.com', h)
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [200]>
>>> r.request.headers
{'Accept': '*/*', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Authorization': 'Basic xxxx', 'User-Agent': 'python-requests/2.7.0 CPython/2.7.6 Darwin/14.1.0'}
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [400]>
>>> r.request.headers
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.7.0 CPython/2.7.6 Darwin/14.1.0', 'Connection': 'keep-alive', 'Cookie': 'gi=xxxx', 'Authorization': 'Basic xxxx'}
>>>
```


So that cookie appears to be the core problem. Can you confirm that the header was correctly set? `r.headers['set-cookie']` will work. Can you obfuscate the value and then show me the rest?


The `set-cookie` header is present on the response from the first request, but not on the second.

```
>>> import requests
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> from requests.adapters import HTTPAdapter
>>> h = HTTPAdapter(pool_connections=1, pool_maxsize=1)
>>> s = requests.Session()
>>> s.mount('https://gerrit-review.googlesource.com', h)
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [200]>
>>> r.headers['set-cookie']
'gi=xxx;Path=/;Secure;HttpOnly'
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r
<Response [400]>
>>> r.headers['set-cookie']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'set-cookie'
>>>
```


That's bizarre. Why does the service not like the cookie that it itself sets?

Let's just confirm that this is actually what's happening. Can you run this?

``` python
import requests
from requests.cookies import RequestsCookieJar
url = "https://gerrit-review.googlesource.com"
auth = requests.utils.get_netrc_auth(url)
s = requests.Session()
r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
print r.status
s.cookies = RequestsCookieJar()
r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
print r.status
```


Had to change `r.status` to `r.status_code`, but it seems to work now.  Yay!

```
$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from requests.cookies import RequestsCookieJar
>>> url = "https://gerrit-review.googlesource.com"
>>> auth = requests.utils.get_netrc_auth(url)
>>> s = requests.Session()
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> print r.status_code
200
>>> s.cookies = RequestsCookieJar()
>>> r = s.get(url=url+"/a/changes/", auth=auth, stream=True)
/Users/david/git/pygerrit/ENV/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> print r.status_code
200
```


So the weird thing is: why don't they want that cookie back?