Requests Session doesn't use HTTP_PROXY settings when sending PreparedRequests?

I am using mitmproxy for development of a RESTful API client as I want to see what's sent. I noticed that sometimes I don't see all requests in my proxy logs. I narrowed down the code to a reproducible test case.

How to reproduce: start some proxy locally which allows you to see the requests, set the HTTP(S)_PROXY variables and execute the following code:

```
#!/usr/bin/env python3 
import requests
session = requests.Session()
def retry_hook(res, *args, **kwargs):
    req = res.request
    print(req)
    if 'X-Retried-Request' not in req.headers:
        print('(Retrying)')
        req.headers['X-Retried-Request'] = True
        return session.send(req)
    else:
        print('(Not retrying retried request)')

session.get('https://www.google.de', hooks={'response': retry_hook}, verify=False)
```
- I'm using google.de as I am in Germany and google.com results in redirects, which I don't want to see in my log. Depending on where you test this, use a different domain.

What the code does: it sends a request, and using a hook, when the response is ready, it re-sends the original request. The first time the request is re-sent is sets a custom header, so that the next time the request is not re-sent as the header is discovered. The first time the request is immediately re-sent, the second time it is not. Every time the hook is invoked the request is printed out.

I see the following output (ignoring a warning about unverified HTTPS requests):

```
<PreparedRequest [GET]>
(Retrying)
<PreparedRequest [GET]>
(Not retrying retried request)
```

However, in my proxy window, I only see the first request, the second one is not seen (the custom header is missing from the request, that's how I know it is the second one that's missing.

Now, this is a contrived example, let me explain a real world scenario. We have a RESTful service which requires OAuth2 tokens for authorization. The tokens we have are valid for 3 minutes. I am writing a tool in Python which allows us to call the service, and it handles authentication transparently: it gets a token initially, and as long as it is valid, it uses it. At some point, a 401 response will be returned, that's when the hook comes in - it recognizes 401, fetches another token, sets it on the session and request again, and retries the request once. The result of this retried request is the result of the hook, meaning that a unauthorized first request is 'forgotten', and the retried one is returned. Any other result than 401, and retried requests resulting in 401 get standard treatment.

There is an issue that such retried requests always fail when a proxy is in place, but as they correctly fetched the token and set it on the Session, subsequent requests don't have this problem.


Yup, sorry, this is an expected behaviour. See #2807 which also discusses the expected solution.


It seems I need code like this right before I re-send the request:

```
settings = session.merge_environment_settings(req.url, {}, False, False, None)
return session.send(req, **settings)
```

I have a few questions which I couldn't find answers for in the linked ticket, I hope you don't mind
- It seems a such a PreparedRequest doesn't keep all the information, like the proxies, whether it is streamed, etc. the original request was sent with. What would then be the best, most robust way to just re-send a request with a single change to a header value?
- Right now in the hook I would somehow need to determine what stream, verify etc. settings were set to in the original, which sometimes will be difficult as these settings are not available. I think the above code will break for requests for which I do want streaming - the re-sent request will not be streaming, as I explicitly configured it to be. How can I get the necessary information in the hook to pass it to the merge_xxx method? Note that the same hook is used for different types of requests, some of which are streamed and some are not.
- The called merge_environment_settings method takes explicit certs and stream parameters, which seem to never be read from env settings - is this correct? Should then they be in a method named like this?


> It seems a such a PreparedRequest doesn't keep all the information, like the proxies, whether it is streamed, etc. the original request was sent with. What would then be the best, most robust way to just re-send a request with a single change to a header value?

We're treading a bit into semantics.

The `PreparedRequest` keeps all of the information about the _request_ itself: that is, the HTTP parts of the protocol like headers and body format. It doesn't keep the information about _how_ the request was sent and what was done at the socket layer: proxies etc fall into that category. Think of the `PreparedRequest` as an idealised form of a request that could be sent in lots of different ways.

If you want to resend a specific request _exactly as before_ but with changes only to that request, I recommend swapping to the [PreparedRequest flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests) for the _entire_ request/response cycle: that is, build a `Request` object, call `Session.prepare_request` to get your `PreparedRequest`. Then, call `Session.merge_environment_settings` to get the settings, and `Session.send` with those settings. Those last two steps can be repeated as needed, so you can catch any errors and retry those in a loop.

> How can I get the necessary information in the hook to pass it to the merge_xxx method? Note that the same hook is used for different types of requests, some of which are streamed and some are not.

I recommend not using a hook at all. It's not entirely clear to me why you're using the hook instead of handling it manually, to be honest.

> The called merge_environment_settings method takes explicit certs and stream parameters, which seem to never be read from env settings - is this correct? Should then they be in a method named like this?

Yeah, this is somewhat inconsistent, I agree, but it's the result of some historical layout of code. The real reason they're there is because those two kwargs can be set _on the Session itself_, and that method is responsible for grabbing them.


Ok, thanks, I now understand the difference between the request data and how the request is sent in, thank you for the clarification. It does makes sense to separate this, of course.

I will look into the proposed flow. One more question, though - it seems I would need to write a loop myself for re-trying the request, is this correct? That was one of the reasons for the hook, as it is automatic. I think I could actually still implement it using hooks - create a method which uses the prepared request workflow, and in the hook, just call it again. What would be your suggested solution?

As for why I'm using a hook - I'm a novice in both Python and requests, I didn't know any better and by reading the docs I thought it would be the easiest thing possible. What made me wade deeper and deeper into the hook solution was the fact that it did in fact work most of the time; the times when it didn't I didn't see it or I basically put the fault on my code.


> I think I could actually still implement it using hooks - create a method which uses the prepared request workflow, and in the hook, just call it again. What would be your suggested solution?

Yup, this will certainly work as well. You could actually do it by providing that hook as a closure which closes over the relevant variables.

The hook is fine, but depending on what you want to do it may simply be easier to let the request run to completion and then handle the response separately. Entirely up to you though: the closure method will work just fine.


I wonder if we need to document the PreparedRequest workflow better.