RFE: Be able to pass 'no_proxy' in proxies dictionary argument

It would be good if were able to pass the 'no_proxy' value in the proxies dictionary argument. Currently it appears have to set the environment variable 'no_proxy'.

 For example:

```
proxies = {'https_proxy': 'http://proxy.example.com:3128',
           'no_proxy': '10.0.0.0/16,example.com,.example.com'}
requests.get('http://www.example.com', proxies=proxies)
```


This is reasonable request, but the use-case seems small, especially as we now support per-host proxies. The only risk is that you might get redirected off-host, to a host that you don't want to reach via a proxy, but in most cases I'd expect that to be relatively unlikely and covered by the per-host proxies functionality.

Is that incorrect?


We see this use case when using Ironic bare-metal provisioning in the OpenStack project to provision a system from scratch. So the underlying temporary installation OS isn't really configured with proxies in mind.

I have not heard of 'per-host proxies' before. So unsure if that would work or not.


The per-host proxies feature is mentioned in our changelog for 2.8.0, but it allows you to set up a proxies dictionary like so:

``` python
proxies = {'http://my-host.com/': 'http://proxy.example.com:3128'}
```

This will only use the proxy for requests to `my-host.com`: other requests will not use the proxy.


Unfortunately we don't have a good idea which hosts we will be accessing. Thus want the more general method.


So you don't know what hosts you _will_ access, just which ones you mustn't? It does feel like you should be setting the `NO_PROXY` environment variable, so that other tools obey these rules too.


Right. So this installation OS is thrown away when done. So doesn't have the variables set.  All the work is done by the Ironic Python Agent, which is a Python program.

Our current plan is to just use os.environ['no_proxy'] = '....'  but that seems a little hackish but will work. Thus why I filed this RFE.

I'd be happy to try to come up with a pull-request, given a few pointers on what to modify.


I think we could do it easily enough, but I'm increasingly reluctant to add further features to the already quite complicated proxy logic. In the short term, your solution will work. Longer term, I need to think about where the line is with further proxy features.


I _think_ this one is probably ok to have. But I'm going to start putting the brakes on proxy features very soon.


> I think this one is probably ok to have. But I'm going to start putting the brakes on proxy features very soon.

That said, we are in a feature freeze and have been making exemptions for proxies for a reason I'm still not entirely certain of.


The main reason I've been exempting proxies is because our proxies support has always been fairly limited, and it would be nice if it were better. I think 'better' is not very far from here, though.


Thank you very much. Please let me know if I can help out!


So, we haven't had a chance to make any forward progress on this and I believe we need to push 2.9.0 out soon, so I'm deferring this to the hypothetical 2.10.0 release.


I thought we had fixed this in 2.11.0 or 2.10.0 but it looks (from a cursory glance) that I'm wrong.


A proof of concept patch to fix the issue:

 https://github.com/JohnVillalovos/requests/commit/ee88bccaeb529f3023ddc0c9b92d4bc455c0a183

No unit tests though :(


@JohnVillalovos Generally that patch is fine. It'd better to pass the `no_proxy` entry around directly rather than the entire proxies dictionary, so I'd suggest refactoring to do that.

Feel free to open a PR with that patch and we can make suggestions about testing.


So I was working on things, doing unit tests and such, and it was looking pretty good until I noticed this:

https://github.com/kennethreitz/requests/blob/7c80222afa17e3c02a41eb465f886699dfd56687/requests/utils.py#L589

So I'm not sure of a good way to handle that call to proxy_bypass().

Ugly hack would be to write a context manager to set the 'no_proxy' variable and restore the old one. And use that before calling proxy_bypass. But it just seems ugly to me.

Followup: Decided to go with hack. Maybe not so ugly.


@JohnVillalovos We shouldn't even get to there if `no_proxy` has been set by the user. That function has short-circuit returns, so if `no_proxy` was set by the user we should be able to get out of the function well before we ever reach it.


@Lukasa Do you think that on https://github.com/JohnVillalovos/requests/blob/a3c5b0325f74517707ddda91de38d55071c9c76a/requests/utils.py#L613

It should maybe just be:

```
if no_proxy_arg:
    return False
try:
    ...
```


@JohnVillalovos Arg, further reading suggests that maybe it _is_ necessary. Sorry!


@Lukasa Arg, I just removed it!  heh.

Let me know if I should put it back, or if you think it will work.


Let's put it back. =) We can discuss alternatives with it there.


/me is glad he knows about 'git reflog' :)


Latest version:
https://github.com/JohnVillalovos/requests/commit/a3c5b0325f74517707ddda91de38d55071c9c76a