Repository: encode/httpx
Pull request #835 (2020-02-29 03:56:20)

Enable NO_PROXY environment variable support 
This PR addresses issue https://github.com/encode/httpx/issues/360. I found this issue by attempting to use httpx in a environment that had both endpoints that needed to be proxied and ones that needed to stay in the local network. That is when I found out that there was no "NO_PROXY" support. 


Patch:
diff --git a/docs/advanced.md b/docs/advanced.md
index 5d3dba0..8dd7939 100644
--- a/docs/advanced.md
+++ b/docs/advanced.md
@@ -2,50 +2,86 @@
 
 ## Client Instances
 
-Using a Client instance to make requests will give you HTTP connection pooling,
-will provide cookie persistence, and allows you to apply configuration across
-all outgoing requests.
-
 !!! hint
-    A Client instance is equivalent to a Session instance in `requests`.
+    If you are coming from Requests, `httpx.Client()` is what you can use instead of `requests.Session()`.
 
-!!! note
-    Starting from `httpx==0.10.0`, the default and recommended Client class is `AsyncClient`. This should help prevent breaking changes once sync support is reintroduced.
+### Why use a Client?
+
+!!! note "TL;DR"
+    If you do anything more than experimentation, one-off scripts, or prototypes, then you should use a `Client` instance.
+
+#### More efficient usage of network resources
+
+When you make requests using the top-level API as documented in the [Quickstart](/quickstart) guide, HTTPX has to establish a new connection _for every single request_ (connections are not reused). As the number of requests to a host increases, this quickly becomes inefficient.
+
+On the other hand, a `Client` instance uses [HTTP connection pooling](https://en.wikipedia.org/wiki/HTTP_persistent_connection). This means that when you make several requests to the same host, the `Client` will reuse the underlying TCP connection, instead of recreating one for every single request.
+
+This can bring **significant performance improvements** compared to using the top-level API, including:
+
+- Reduced latency across requests (no handshaking).
+- Reduced CPU usage and round-trips.
+- Reduced network congestion.
 
-    A `Client` synonym remains for compatibility with `httpx==0.9.*` releases, but you are encouraged to migrate to `AsyncClient` as soon as possible.
+#### Extra features
+
+`Client` instances also support features that aren't available at the top-level API, such as:
+
+- Cookie persistance across requests.
+- Applying configuration across all outgoing requests.
+- Sending requests through HTTP proxies.
+- Using [HTTP/2](/http2).
+
+The other sections on this page go into further detail about what you can do with a `Client` instance.
 
 ### Usage
 
 The recommended way to use a `Client` is as a context manager. This will ensure that connections are properly cleaned up when leaving the `with` block:
 
 ```python
->>> with httpx.Client() as client:
-...     r = client.get('https://example.com')
-...
->>> r
-<Response [200 OK]>
+with httpx.Client() as client:
+    ...
 ```
 
 Alternatively, you can explicitly close the connection pool without block-usage using `.close()`:
 
 ```python
->>> client = httpx.Client()
->>> try:
+client = httpx.Client()
+try:
+    ...
+finally:
+    client.close()
+```
+
+### Making requests
+
+Once you have a `Client`, you can send requests using `.get()`, `.post()`, etc. For example:
+
+```python
+>>> with httpx.Client() as client:
 ...     r = client.get('https://example.com')
-... finally:
-...     client.close()
 ...
 >>> r
 <Response [200 OK]>
 ```
 
-Once you have a `Client`, you can use all the features documented in the [Quickstart](/quickstart) guide.
+These methods accept the same arguments as `httpx.get()`, `httpx.post()`, etc. This means that all features documented in the [Quickstart](/quickstart) guide are also available at the client level.
+
+For example, to send a request with custom headers:
+
+```python
+>>> with httpx.Client() as client:
+...     headers = {'X-Custom': 'value'}
+...     r = client.get('https://example.com', headers=headers)
+...
+>>> r.request.headers['X-Custom']
+'value'
+```
 
-### Configuration
+### Sharing configuration across requests
 
 Clients allow you to apply configuration to all outgoing requests by passing parameters to the `Client` constructor.
 
-For example, to apply a set of custom headers on every request:
+For example, to apply a set of custom headers _on every request_:
 
 ```python
 >>> url = 'http://httpbin.org/headers'
@@ -57,15 +93,47 @@ For example, to apply a set of custom headers on every request:
 'my-app/0.0.1'
 ```
 
-!!! note
-    When you provide a parameter at both the client and request levels, one of two things can happen:
+### Merging of configuration
 
-    - For headers, query parameters and cookies, the values are merged into one.
-    - For all other parameters, the request-level value is used.
+When a configuration option is provided at both the client-level and request-level, one of two things can happen:
 
-Additionally, `Client` accepts some parameters that aren't available at the request level.
+- For headers, query parameters and cookies, the values are combined together. For example:
 
-One particularly useful parameter is `base_url`, which allows you to define a base URL to prepend to all outgoing requests:
+```python
+>>> headers = {'X-Auth': 'from-client'}
+>>> params = {'client_id': 'client1'}
+>>> with httpx.Client(headers=headers, params=params) as client:
+...     headers = {'X-Custom': 'from-request'}
+...     params = {'request_id': 'request1'}
+...     r = client.get('https://example.com', headers=headers, params=params)
+...
+>>> r.request.url
+URL('https://example.com?client_id=client1&request_id=request1')
+>>> r.request.headers['X-Auth']
+'from-client'
+>>> r.request.headers['X-Custom']
+'from-request'
+```
+
+- For all other parameters, the request-level value takes priority. For example:
+
+```python
+>>> with httpx.Client(auth=('tom', 'mot123')) as client:
+...     r = client.get('https://example.com', auth=('alice', 'ecila123'))
+...
+>>> _, _, auth = r.request.headers['Authorization'].partition(' ')
+>>> import base64
+>>> base64.b64decode(auth)
+b'alice:ecila123'
+```
+
+If you need finer-grained control on the merging of client-level and request-level parameters, see [Request instances](#request-instances).
+
+### Other Client-only configuration options
+
+Additionally, `Client` accepts some configuration options that aren't available at the request level.
+
+For example, `base_url` allows you to prepend an URL to all outgoing requests:
 
 ```python
 >>> with httpx.Client(base_url='http://httpbin.org') as client:
@@ -75,7 +143,7 @@ One particularly useful parameter is `base_url`, which allows you to define a ba
 URL('http://httpbin.org/headers')
 ```
 
-For a list of all available client-level parameters, see the [`Client` API reference](/api/#client).
+For a list of all available client parameters, see the [`Client`](/api/#client) API reference.
 
 ## Calling into Python Web Apps
 
@@ -120,19 +188,37 @@ with httpx.Client(dispatch=dispatch, base_url="http://testserver") as client:
     ...
 ```
 
-## Build Request
+## Request instances
 
-You can use `Client.build_request()` to build a request and
-make modifications before sending the request.
+For maximum control on what gets sent over the wire, HTTPX supports building explicit [`Request`](/api#request) instances:
 
 ```python
->>> with httpx.Client() as client:
-...     req = client.build_request("OPTIONS", "https://example.com")
-...     req.url.full_path = "*"  # Build an 'OPTIONS *' request for CORS
-...     r = client.send(req)
-...
->>> r
-<Response [200 OK]>
+request = httpx.Request("GET", "https://example.com")
+```
+
+To dispatch a `Request` instance across to the network, create a [`Client` instance](#client-instances) and use `.send()`:
+
+```python
+with httpx.Client() as client:
+    response = client.send(request)
+    ...
+```
+
+If you need to mix client-level and request-level options in a way that is not supported by the default [Merging of parameters](#merging-of-parameters), you can use `.build_request()` and then make arbitrary modifications to the `Request` instance. For example:
+
+```python
+headers = {"X-Api-Key": "...", "X-Client-ID": "ABC123"}
+
+with httpx.Client(headers=headers) as client:
+    request = client.build_request("GET", "https://api.example.com")
+
+    print(request.headers["X-Client-ID"])  # "ABC123"
+
+    # Don't send the API key for this particular request.
+    del request.headers["X-Api-Key"]
+
+    response = client.send(request)
+    ...
 ```
 
 ## .netrc Support
diff --git a/httpx/_api.py b/httpx/_api.py
index 19ef229..d42277c 100644
--- a/httpx/_api.py
+++ b/httpx/_api.py
@@ -66,8 +66,6 @@ def request(
     file, key file, password).
     * **trust_env** - *(optional)* Enables or disables usage of environment
     variables for configuration.
-    * **proxies** - *(optional)* A dictionary mapping HTTP protocols to proxy
-    URLs.
 
     **Returns:** `Response`
 
diff --git a/httpx/_client.py b/httpx/_client.py
index e9d33a9..2a310e2 100644
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -43,7 +43,12 @@ from ._models import (
     URLTypes,
 )
 from ._status_codes import codes
-from ._utils import NetRCInfo, get_environment_proxies, get_logger
+from ._utils import (
+    NetRCInfo,
+    get_environment_proxies,
+    get_logger,
+    should_not_be_proxied,
+)
 
 logger = get_logger(__name__)
 
@@ -517,7 +522,7 @@ class Client(BaseClient):
         Returns the SyncDispatcher instance that should be used for a given URL.
         This will either be the standard connection pool, or a proxy.
         """
-        if self.proxies:
+        if self.proxies and not should_not_be_proxied(url):
             is_default_port = (url.scheme == "http" and url.port == 80) or (
                 url.scheme == "https" and url.port == 443
             )
@@ -1032,7 +1037,7 @@ class AsyncClient(BaseClient):
         Returns the AsyncDispatcher instance that should be used for a given URL.
         This will either be the standard connection pool, or a proxy.
         """
-        if self.proxies:
+        if self.proxies and not should_not_be_proxied(url):
             is_default_port = (url.scheme == "http" and url.port == 80) or (
                 url.scheme == "https" and url.port == 443
             )
diff --git a/httpx/_content_streams.py b/httpx/_content_streams.py
index ca48326..1d35167 100644
--- a/httpx/_content_streams.py
+++ b/httpx/_content_streams.py
@@ -291,7 +291,7 @@ class MultipartStream(ContentStream):
         self, data: dict, files: dict
     ) -> typing.Iterator[typing.Union["FileField", "DataField"]]:
         for name, value in data.items():
-            if isinstance(value, (list, dict)):
+            if isinstance(value, list):
                 for item in value:
                     yield self.DataField(name=name, value=item)
             else:


Test patch:
diff --git a/tests/client/test_proxies.py b/tests/client/test_proxies.py
index a100873..e76e520 100644
--- a/tests/client/test_proxies.py
+++ b/tests/client/test_proxies.py
@@ -87,3 +87,32 @@ def test_dispatcher_for_request(url, proxies, expected):
 def test_unsupported_proxy_scheme():
     with pytest.raises(ValueError):
         httpx.AsyncClient(proxies="ftp://127.0.0.1")
+
+
+@pytest.mark.parametrize(
+    ["url", "env", "expected"],
+    [
+        ("http://google.com", {}, None),
+        (
+            "http://google.com",
+            {"HTTP_PROXY": "http://example.com"},
+            "http://example.com",
+        ),
+        (
+            "http://google.com",
+            {"HTTP_PROXY": "http://example.com", "NO_PROXY": "google.com"},
+            None,
+        ),
+    ],
+)
+def test_proxies_environ(monkeypatch, url, env, expected):
+    for name, value in env.items():
+        monkeypatch.setenv(name, value)
+
+    client = httpx.AsyncClient()
+    dispatcher = client.dispatcher_for_url(httpx.URL(url))
+
+    if expected is None:
+        assert dispatcher == client.dispatch
+    else:
+        assert dispatcher.proxy_url == expected
diff --git a/tests/test_multipart.py b/tests/test_multipart.py
index a08ada8..3df19fe 100644
--- a/tests/test_multipart.py
+++ b/tests/test_multipart.py
@@ -99,8 +99,7 @@ def test_multipart_encode():
         "a": "1",
         "b": b"C",
         "c": ["11", "22", "33"],
-        "d": {"ff": ["1", b"2", "3"], "fff": ["11", b"22", "33"]},
-        "f": "",
+        "d": "",
     }
     files = {"file": ("name.txt", io.BytesIO(b"<file content>"))}
 
@@ -114,9 +113,7 @@ def test_multipart_encode():
             '--{0}\r\nContent-Disposition: form-data; name="c"\r\n\r\n11\r\n'
             '--{0}\r\nContent-Disposition: form-data; name="c"\r\n\r\n22\r\n'
             '--{0}\r\nContent-Disposition: form-data; name="c"\r\n\r\n33\r\n'
-            '--{0}\r\nContent-Disposition: form-data; name="d"\r\n\r\nff\r\n'
-            '--{0}\r\nContent-Disposition: form-data; name="d"\r\n\r\nfff\r\n'
-            '--{0}\r\nContent-Disposition: form-data; name="f"\r\n\r\n\r\n'
+            '--{0}\r\nContent-Disposition: form-data; name="d"\r\n\r\n\r\n'
             '--{0}\r\nContent-Disposition: form-data; name="file";'
             ' filename="name.txt"\r\n'
             "Content-Type: text/plain\r\n\r\n<file content>\r\n"