Repository: aio-libs/aiohttp

Issue:
aiohttp not honoring `no_proxy`
## Long story short
aiohttp not honoring `no_proxy` environment variable when `trust_env` is set.
<!-- Please describe your problem and why the fix is important. -->

## Expected behaviour
When there is a `no_proxy` `NO_PROXY` environment variable set, URLs listed in that variable should not use `http{s}_proxy`
<!-- What is the behaviour you expect? -->

## Actual behaviour
When `trust_env` is set and there are `http_proxy` and `no_proxy` set, the proxy is always used.
<!-- What's actually happening? -->

## Steps to reproduce
Set `http{s}_proxy` and `no_proxy`, make a request to a domain listed in `no_proxy`. The request will still go through the proxy.

<!-- Please describe steps to reproduce the issue.
     If you have a script that does that please include it here within
     markdown code markup -->

## Your environment
Linux 
aiohttp 3.6.2
Company proxy server
<!-- Describe the environment you have that lead to your issue.
     This includes aiohttp version, OS, proxy server and other bits that
     are related to your case. 
     
     IMPORTANT: aiohttp is both server framework and client library.
     For getting rid of confusing please put 'server', 'client' or 'both'
     word here.
     -->



Discussion:
The pull request is welcome!
Could you point me in the right direction? 

I'm guessing this is where I'll need to check the no proxy list https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client.py#L424 and blank the proxy if found?
Hmm, interesting.
The real code for extracting the proxy list from env vars is [`proxies_from_env()`](https://github.com/aio-libs/aiohttp/blob/master/aiohttp/helpers.py#L230-L254) function.

It doesn't parse itself but reuses `getproxies()` from `urllib.request`. 
I did not investigate more.
Does `getproxies()` ignore `NO_PROXY`?
No. That is why I suggesting doing it manually. I could only find a reference to `no_proxy` when [urllib.request.ProxyHandler](https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler) is used. I'm searching for something to compare the hostnames to the `no_proxy` list.

I could add a `is_in_no_proxy_list(url)` to the [helpers.py](https://github.com/aio-libs/aiohttp/blob/master/aiohttp/helpers.py) module.
Maybe a solution like 5d92cb5 might work. Would still have to write the tests. I should probably read up on the no_proxy list standards to make sure hostnames put in the no_proxy list are matching correctly. 
Actually I looks like urllib has code to handle [no_proxy](https://github.com/python/cpython/blob/3.8/Lib/urllib/request.py#L2490). I'll use the appropriate function from there.
Any ETA for this? We're suffering from the same problem with aiohttp not honoring no_proxy vars.
Even I am interested in this fix as a third party application we are using is dependent on this library.
We tested these changes and they worked for us!
Glad I found this bug report! Just wondering if there are any workarounds until this patch gets accepted?

I need to use a proxy to get to external services but that proxy has no access to other internal services, hence my need for no_proxy.

At the moment the only option I seem to have is to create separate sessions with and without proxies set, but that means my app would need to be aware of the network setup of where it gets installed, which is certainly not ideal. Unless there are other options I am missing?

Resolving patch:
diff --git a/aiohttp/client.py b/aiohttp/client.py
--- a/aiohttp/client.py
+++ b/aiohttp/client.py
@@ -8,6 +8,7 @@
 import sys
 import traceback
 import warnings
+from contextlib import suppress
 from types import SimpleNamespace, TracebackType
 from typing import (
     Any,
@@ -76,8 +77,8 @@
     BasicAuth,
     TimeoutHandle,
     ceil_timeout,
+    get_env_proxy_for_url,
     get_running_loop,
-    proxies_from_env,
     sentinel,
     strip_auth_from_url,
 )
@@ -483,11 +484,8 @@ async def _request(
                     if proxy is not None:
                         proxy = URL(proxy)
                     elif self._trust_env:
-                        for scheme, proxy_info in proxies_from_env().items():
-                            if scheme == url.scheme:
-                                proxy = proxy_info.proxy
-                                proxy_auth = proxy_info.proxy_auth
-                                break
+                        with suppress(LookupError):
+                            proxy, proxy_auth = get_env_proxy_for_url(url)
 
                     req = self._request_class(
                         method,
diff --git a/aiohttp/helpers.py b/aiohttp/helpers.py
--- a/aiohttp/helpers.py
+++ b/aiohttp/helpers.py
@@ -41,7 +41,7 @@
     cast,
 )
 from urllib.parse import quote
-from urllib.request import getproxies
+from urllib.request import getproxies, proxy_bypass
 
 import async_timeout
 import attr
@@ -297,6 +297,20 @@ def isasyncgenfunction(obj: Any) -> bool:
         return False
 
 
+def get_env_proxy_for_url(url: URL) -> Tuple[URL, Optional[BasicAuth]]:
+    """Get a permitted proxy for the given URL from the env."""
+    if url.host is not None and proxy_bypass(url.host):
+        raise LookupError(f"Proxying is disallowed for `{url.host!r}`")
+
+    proxies_in_env = proxies_from_env()
+    try:
+        proxy_info = proxies_in_env[url.scheme]
+    except KeyError:
+        raise LookupError(f"No proxies found for `{url!s}` in the env")
+    else:
+        return proxy_info.proxy, proxy_info.proxy_auth
+
+
 @attr.s(auto_attribs=True, frozen=True, slots=True)
 class MimeType:
     type: str


Test patch:
diff --git a/tests/test_helpers.py b/tests/test_helpers.py
--- a/tests/test_helpers.py
+++ b/tests/test_helpers.py
@@ -6,6 +6,7 @@
 import tempfile
 from math import isclose, modf
 from unittest import mock
+from urllib.request import getproxies_environment
 
 import pytest
 from multidict import MultiDict
@@ -497,6 +498,96 @@ async def test_get_running_loop_ok(loop) -> None:
     assert helpers.get_running_loop() is loop
 
 
+# --------------------- get_env_proxy_for_url ------------------------------
+
+
+@pytest.fixture
+def proxy_env_vars(monkeypatch, request):
+    for schema in getproxies_environment().keys():
+        monkeypatch.delenv(f"{schema}_proxy", False)
+
+    for proxy_type, proxy_list in request.param.items():
+        monkeypatch.setenv(proxy_type, proxy_list)
+
+    return request.param
+
+
+@pytest.mark.parametrize(
+    ("proxy_env_vars", "url_input", "expected_err_msg"),
+    (
+        (
+            {"no_proxy": "aiohttp.io"},
+            "http://aiohttp.io/path",
+            r"Proxying is disallowed for `'aiohttp.io'`",
+        ),
+        (
+            {"no_proxy": "aiohttp.io,proxy.com"},
+            "http://aiohttp.io/path",
+            r"Proxying is disallowed for `'aiohttp.io'`",
+        ),
+        (
+            {"http_proxy": "http://example.com"},
+            "https://aiohttp.io/path",
+            r"No proxies found for `https://aiohttp.io/path` in the env",
+        ),
+        (
+            {"https_proxy": "https://example.com"},
+            "http://aiohttp.io/path",
+            r"No proxies found for `http://aiohttp.io/path` in the env",
+        ),
+        (
+            {},
+            "https://aiohttp.io/path",
+            r"No proxies found for `https://aiohttp.io/path` in the env",
+        ),
+        (
+            {"https_proxy": "https://example.com"},
+            "",
+            r"No proxies found for `` in the env",
+        ),
+    ),
+    indirect=["proxy_env_vars"],
+    ids=(
+        "url_matches_the_no_proxy_list",
+        "url_matches_the_no_proxy_list_multiple",
+        "url_scheme_does_not_match_http_proxy_list",
+        "url_scheme_does_not_match_https_proxy_list",
+        "no_proxies_are_set",
+        "url_is_empty",
+    ),
+)
+@pytest.mark.usefixtures("proxy_env_vars")
+def test_get_env_proxy_for_url_negative(url_input, expected_err_msg) -> None:
+    url = URL(url_input)
+    with pytest.raises(LookupError, match=expected_err_msg):
+        helpers.get_env_proxy_for_url(url)
+
+
+@pytest.mark.parametrize(
+    ("proxy_env_vars", "url_input"),
+    (
+        ({"http_proxy": "http://example.com"}, "http://aiohttp.io/path"),
+        ({"https_proxy": "http://example.com"}, "https://aiohttp.io/path"),
+        (
+            {"http_proxy": "http://example.com,http://proxy.org"},
+            "http://aiohttp.io/path",
+        ),
+    ),
+    indirect=["proxy_env_vars"],
+    ids=(
+        "url_scheme_match_http_proxy_list",
+        "url_scheme_match_https_proxy_list",
+        "url_scheme_match_http_proxy_list_multiple",
+    ),
+)
+def test_get_env_proxy_for_url(proxy_env_vars, url_input) -> None:
+    url = URL(url_input)
+    proxy, proxy_auth = helpers.get_env_proxy_for_url(url)
+    proxy_list = proxy_env_vars[url.scheme + "_proxy"]
+    assert proxy == URL(proxy_list)
+    assert proxy_auth is None
+
+
 # ------------- set_result / set_exception ----------------------