Proxy bypass detection could add delay of ~4.5 seconds to each request 

Requests uses `urlib.proxy_bypass` to check if the proxy should be bypassed. This uses `socket.gethostbyaddr` to check if the provided IP matches a hostname entered in the proxy bypass list. In some cases, `gethostbyaddr` takes up to 4.5 seconds on Windows (e.g. when the IP is found but the primary DNS doesn't recognize it).

Currently, the proxy bypass detection is done for _every_ request, making it pretty unusable when the conditions are met where `gethostbyaddr` is this slow.

A reasonable solution might be caching the results from `urlib.proxy_bypass`.


Yeah, 4.5 seconds is probably too long.

The concern with caching the result is that the resolver doesn't give enough information for us to invalidate that cache appropriately, but we will from time to time want to invalidate it (so that long running processes don't end up so broken that they need a restart).

A simple approach here might be to use an `@memoize` decorator with a cache that invalidates entries after a certain amount of time. Given that [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache) provides a `cache_clear` function, we may be able to provide a very simple wrapper function that handles this logic.


May I try this one :-)?

My plan for implementation:
1. wrap the `proxy_bypass` by our function like `cached_proxy_bypass`
2. cache our wrapper function

``` python
 @functools.lru_cache(maxsize=32)
  def cached_proxy_bypass(netloc):
      return proxy_bypass(netloc)
```

This can cache all the function call to `proxy_bypass` with the same `netloc` as long as they are in the latest 32 calls. However, if we want to invalidate the same `netloc` from time to time, we might maintain a counter, such that we define the wrapper function like this

``` python
@functools.lru_cache(maxsize=32)
def cached_proxy_bypass(netloc, _):
    return proxy_bypass(netloc)
```

and call it like

``` python
cached_proxy_bypass(netloc, int(counter / 5)); counter += 1;
```

5 is a magic number here, which can be any other number, such that lru will invalidate the result every 5 calls with the same `netloc` (because for every 5 calls, the passing arguments are the same, and thus lru will cache them).

How do you think?

Thanks a lot.


@lazywei That could definitely work. The issue there is that the cache gets busted after every _n_ calls regardless of how fast they happen or how broadly distributed they are. That seems problematic.

Another option would be to try getting a unix timestamp, dividing that by some interval (e.g. 60 seconds), and then using that as the 'counter'.


@Lukasa Oh, thanks for your advice! Using timestamp is brilliant!
I have made the first attempt and opened the PR!

Thanks!


Has this been solved?


@TetraEtc The pull-request is still open, I'm afraid.


Sorry I was busy and distracted last few weeks. I'll work on it this weekend. Hope I can deliver some progress by that time.


How is the progress on this issue?

Which workarounds (besides turning off proxy settings in Windows) are known? 


Right now there is a proposed patch in #2992 that is currently outstanding. Work would be needed to evaluate whether the patch there is suitable. Otherwise, the only work arounds involve monkeypatching the standard library to affect the results of those function calls.


Isn´t there a way to tell 'requests' to completly ignore proxy settings instead of checking which URLs/IPs can bypass it? 


Yes, you can do this:

``` python
import requests
s = requests.Session()
s.trust_env = False
```

This will prevent requests getting any information from its environment: specifically, it'll disable environment searches for proxies and for certificate bundles.


@Lukasa : Thanks! That´s exactly what I thought of. I tried to implement your suggestion into [robotframework-requests](https://github.com/bulkan/robotframework-requests) (see [PR 117](https://github.com/bulkan/robotframework-requests/pull/117))  but was not successful - still have a delay of approx. 5 seconds. But I am just starting out with Python and programming at all, so I am very sure that I missed something :(


Your patch there won't work.

You have this line: `s.trust_env = trust_env if trust_env else s.trust_env`.

This means that if you try to set `trust_env` to `False`, `trust_env` is not `True` and so you set `s.trust_env` to `s.trust_env`. In essence:

``` python
# If trust_env == True
s.trust_env = trust_env

# If trust_env == False
s.trust_env = s.trust_env
```

This means that there's no way to actually _set_ `trust_env` to `False`. Change line 82 to `s.trust_env = trust_env if trust_env is not None else s.trust_env`, and then change line 134 to default `trust_env` to `None`, not `True`.


@Lukasa : I knew I missed somethink :). Thanks again!  I fixed it now but still see a delay :( (more details [here](https://github.com/bulkan/robotframework-requests/issues/116#issuecomment-224866699))


Just to be sure: you did _actually_ end up calling with trust_env=False in your own code, yes?


Yes, at least Robotframework log tells so

![image](https://cloud.githubusercontent.com/assets/5635462/15931681/0d907b86-2e59-11e6-9846-94a3503b6c22.png)


@Tset-Noitamotua And you're entirely confident that the session is being used?


Hi @Lukasa, 

We are using the [requests.Session](https://github.com/bulkan/robotframework-requests/blob/master/src/RequestsLibrary/RequestsKeywords.py#L78)

[Get request](https://github.com/bulkan/robotframework-requests/blob/master/src/RequestsLibrary/RequestsKeywords.py#L840)


@bulkan Sorry, my question wasn't clear. I meant: are we really sure that the request that is being delayed is going through the `Session` constructed with `trust_env=False`?


Hi @Lukasa, @bulkan 
how can I help so that we make progress on this issue. Please give me some noob friendly instructions how to verify whether the used session is the expected one :-)


@Tset-Noitamotua You'll need some way to evaluate the value of `trust_env` while the request is hanging, or about to hang. 


@Lukasa  Thank you! Did as you said and it came out value of `trust_env` was string represantation  of `False` ... thus it was `'False'` (If you interessted [here](https://github.com/bulkan/robotframework-requests/issues/116#issuecomment-230270721) are more details).


@Lukasa Is this issue still open? Would be willing to take a shot at it using the `@functools.lru_cache` and time-stamping approach.

As far as I know, yes.

@schlamar would you be able to review?

Looks like this was fixed in #3885.

@sigmavirus24 Actually, an expiration time of 60 seconds doesn't help at all for my original use case. I have a long running GUI process which (might) does HTTP requests every few minutes. So while the expiration time is this low and not adjustable, this issue is not fixed for me.

However, thinking again about this issue, I would say that `urllib` is doing it wrong. Forward and reverse DNS lookups to check if a proxy should be bypassed doesn't feel right.