Returning cookies from Adapters

I've been trying to create a subclass of `BaseAdapter` for Requests that supports WSGI applications, (preferably based on the requests-wsgi-adapter package, or from scratch if need be).

The sticking point has been cookies and session handling.  From reading the Requests code, my understanding is that cookies are automatically extracted from the reponse object returned from the adapter's `send` method.  In order for the cookies to be automatically extracted the response needs to have a `r.raw._original_response.msg` chain of attributes and this should return an object with an `info` method that should return an `email.message.Message` instance (largely because of  `cookies.py:extract_cookies_to_jar`, but also because of how `http.cookiejar` behaves).

This seems like a needlessly complicated series of steps to go through, particularly as I'm not currently clear on how to correctly convert my response body to an `email.message.Message` (and a quick scan of the Python documentation makes this look non-trivial).

Have I misunderstood?  Are there any helper functions for performing these steps?

Would you consider a patch to `extract_cookies_to_jar` that also looks, for example, for a `_cookies_dict` property on the response, at the point that it currently silently returns if there is no `_original_response` attribute?

Alternatively, would you consider supporting something like https://pypi.python.org/pypi/cookies/ with a friendlier interface (kind of "Requests for cookies")?  I haven't dug into it a lot, but it looks relatively easy to add optional support for it (which I'd be happy to contribute).


> Have I misunderstood? Are there any helper functions for performing these steps?

You have not: there are no helper functions for performing these steps.

> Alternatively, would you consider supporting something like https://pypi.python.org/pypi/cookies/ with a friendlier interface (kind of "Requests for cookies")?

Nope. =) Note the README on that project:

> This doesn’t compete with the cookielib (http.cookiejar) module in the Python standard library, which is specifically for implementing cookie storage and similar behavior in an HTTP client such as a browser. Things cookielib does that this doesn’t:
>
> - Write to or read from browsers’ cookie stores or other proprietary formats for storing cookie data in files
> - Handle the browser/client logic like deciding which cookies to send or discard, etc.

`cookies` is not an appropriate replacement for `cookielib`. We need to continue to use `cookielib` for the moment.
> Would you consider a patch to `extract_cookies_to_jar` that also looks, for example, for a `_cookies_dict` property on the response, at the point that it currently silently returns if there is no `_original_response` attribute?

I am disinclined to do that. Generally speaking I don't think this is a common concern. Additionally, the surface area you actually need here is low. Your `Message` instance actually just needs to be an object with a few methods: `get_all`, and `getheaders`. That's it: everything else should be covered by the shim code. I think the two methods can even have the same implementation, and that in your case `getheaders` may be optional.

So I don't think this is completely unreasonable to work with, and given that it's very uncommonly needed the overhead of having an alternative escape method is probably not worthwhile.

@Singletoned thank you for doing this! WSGI app mounting is exactly one of the use cases I had in mind when designing the ConnectionAdapter interface — I've been waiting for someone to do this :)

✨🍰✨

@Lukasa

> cookies is not an appropriate replacement for cookielib. We need to continue to use cookielib for the moment.

I'm not suggesting that you replace `cookielib` with `cookies`, I'm suggesting that `cookies` would be a nicer thing to return attached to a response from an adapter than `email.message.Message`.  I've already wasted a day looking into this, and my experience of looking at the `cookielib` code makes me reluctant to get involved with it any further.  It appears to be legacy code that will involve needless pain to interface with, that will be very hard to debug when strange things happen.  All this just to pass a single header.

> given that it's very uncommonly needed the overhead of having an alternative escape method is probably not worthwhile.

The reason it is uncommonly needed is because no one is creating adapters.  If the interface between adapters and sessions is deeply entwined with urllib/cookielib, then that could discourage people from creating adapters.

At some point an interface between adapters and sessions will have to be designed (it can't always be based on urllib/cookielib).  I'd prefer it if it was sooner rather than later.  If you can agree on a design, I'd be willing to implement it.

@kennethreitz Thanks but @seanbrant did most of the work already on https://github.com/seanbrant/requests-wsgi-adapter.  I'm just hoping to stand on his shoulders, and get it working with sessions (or rewrite it if need be).  I want to be able to replace some code using Requests to test a legacy website with some that tests a new WSGI app.  I've also been playing with creating a web testing library that's based on Requests, but returns custom responses with helpers for filling in forms, using css selectors, etc.

I really like the idea of the adapters and the ability to mount them on particular addresses.  I can see it being useful in a surprising variety of ways.  It also ties in nicely (for me) with the ability to hook into response creation.

> I'm not suggesting that you replace cookielib with cookies, I'm suggesting that cookies would be a nicer thing to return attached to a response from an adapter than email.message.Message. 

Ah, sorry, I misunderstood. If we're redesigning this interface and adding code to Requests to shim from format x to `cookielib` there is no reason to invoke a third party library at all: we should just look straight at the header. We already have the headers, no reason to beat around the bush with an extra dependency. 

> The reason it is uncommonly needed is because no one is creating adapters.

Sure they are. I have done it myself several times. And I have [tackled this problem](https://github.com/Lukasa/hyper/blob/development/hyper/contrib.py#L173). It's not intractable at all: the interface is really as small as I suggested it was. 

However, I agree that we could bear to formalise this interface more strongly. The primary reason it takes the form that it does is that Requests throws away some key data about the header: namely, it joins them with commas, which does not work well with cookie handling. So we will need *a* formal interface that we can use, and we will need to map urllib3's responses through that interface. 

The design wouldn't need to be very complex: just an object that takes a header iterable and implements the two needed methods. Hell, you could lift the one I linked above from hyper, change its name, and use that. That would probably be my initial suggestion. 

@frostming's solution works fine for me (following your suggestion).  If you guys are open to it, I (or @frostming) could make a small utility to put in the adapters code that generates the relevant stub.  At the very least it should be documented somewhere so that other people don't have to go through the same chain of exploration.

@Singletoned What's the proposed patch for `adapters.py`? I'd be open to seeing what it looks like. Failing that I'd be happy to review a documentation patch for the `Transport Adapters` section of the documentation.

Having looked through the code while working on my own [WSGI adapter project](https://github.com/the-allanc/pyriform), it makes me wonder why we don't just use urllib3's HTTPHeaderDict directly from the urllib3 response object. I'm guessing that's there's an [API difference](https://github.com/shazow/urllib3/issues/670) which means that we can't plug it directly in.

If requests had its own `HTTPHeaderDict` which is inspired by the `urllib3` version, but is compatible with `CaseInsensitiveDict` -  would that make things much easier? If so, then I might play with that a little later - adding some methods (maybe just `add` and `getlist`) would do the job.