Repository: encode/httpx

Issue:
Mount API
Somewhat related to #927, #954 

We now have a really nicely defined Transport API. We also have the ability to mount proxies under particular schemes, or particular hosts.

Really we ought to be generalising that into a "mount a transport onto this scheme/domain/...".

Eg...

```python
client.mount('http:', ...)  # Mount a particular transport instance to handle all "http" scheme requests.
```

There's an equivalent API in requests... https://requests.readthedocs.io/en/master/user/advanced/#transport-adapters

```python
>>> s = requests.Session()
>>> s.mount('https://github.com/', MyAdapter())
```

Right now the closest we have is `proxies={<mount-point>: <Proxy or Transport>}`. However the set of mount points that are supported doesn't feel super well defined, and it *nominally* only supports proxies (even through they can actually be *any* transport instance).

Here's a bit of a proposal for how a more consistent, and generalised mount API might look...

* Support a `.transports = {'http': ..., 'https': ...}` read-only property on the client.
* The default `transport` sets `'http'` and `'https'`.
* Transports can only be mounted while a client is closed. We'll track that status in `.send`/`.close`.
* Closing a client closes all the transports.
* We'd support `.mount(prefix, transport)`.

Glossing over a bunch of stuff here since I just want to get the idea out there.

Ideally we'd also move *away* from having proxy transport instance be supported via `proxies = ...`, and instead only support the *simple* proxy config style for `proxies=...`, with *any* more specific transport mounting handled via the `.mount` interface so that there's only one way to do things.


Discussion:
Although this is new proposed API, I think it falls under the 1.0 cutoff, because it *impacts* stuff we want to resolve for 1.0.
Since #1098, #1099, #1103, this has now become more obvious.

I don't think we should support a `.mount()` method - that's awkward because really you only want to mount or unmount transports on instantiating the client, so that you can ensure that the transport lifespans match the client lifespan.

Instead the API ought to look like this...

```python
client = httpx.Client(mounts={"http://www.example.com": WSGITransport(...)})
client = httpx.Client(mounts={"http://": MyCustomTransport()})
client = httpx.Client(mounts={"http://": MyCustomTransport(), "https://": MyCustomTransport()})
client = httpx.Client(mounts={"all://": MyCustomTransport()})
```

Additionally we could extend our URL pattern matching to also support the following...

```python
client = httpx.Client(mounts={"http://www.example.com/some/path/prefix": WSGITransport(...)})
```

At some point we *might* also want to think about exposing our pattern -> transport info from the client, since that'll help developers better understand what proxies and mounts have been introduced, and get a better internal model of how transport lookups operate.

For example, perhaps we'd end up with something like this at some point...

```python
>>> os.environ["NO_PROXY"] = "example.com"
>>> os.environ["HTTP_PROXY"] = "localhost:1234"
>>> client = httpx.Client(mounts={"https://www.google.com/some/path": httpx.WSGITransport(...)})
>>> client.get_transports()
{
    <httpx.URLPattern("https://www.google.com/some/path")>: httpx.WSGITransport(...),
    <httpx.URLPattern("all://*example.com")>: httpx.SyncConnectionPool(...),
    <httpx.URLPattern("all://localhost:1234")>: httpx.SyncHTTPProxyl(...),
    <httpx.URLPattern("all://")>: httpx.SyncConnectionPool(...),
}
```

(Note that I've renamed `URLMatcher` in this example case, since I think `httpx.URLPattern` might be a better name)

Resolving patch:
diff --git a/httpx/_client.py b/httpx/_client.py
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -87,7 +87,7 @@ def __init__(
         cookies: CookieTypes = None,
         timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
-        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
+        event_hooks: typing.Mapping[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         trust_env: bool = True,
     ):
@@ -561,11 +561,12 @@ def __init__(
         cert: CertTypes = None,
         http2: bool = False,
         proxies: ProxiesTypes = None,
+        mounts: typing.Mapping[str, httpcore.SyncHTTPTransport] = None,
         timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG,
         limits: Limits = DEFAULT_LIMITS,
         pool_limits: Limits = None,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
-        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
+        event_hooks: typing.Mapping[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         transport: httpcore.SyncHTTPTransport = None,
         app: typing.Callable = None,
@@ -611,7 +612,7 @@ def __init__(
             app=app,
             trust_env=trust_env,
         )
-        self._proxies: typing.Dict[
+        self._mounts: typing.Dict[
             URLPattern, typing.Optional[httpcore.SyncHTTPTransport]
         ] = {
             URLPattern(key): None
@@ -626,7 +627,12 @@ def __init__(
             )
             for key, proxy in proxy_map.items()
         }
-        self._proxies = dict(sorted(self._proxies.items()))
+        if mounts is not None:
+            self._mounts.update(
+                {URLPattern(key): transport for key, transport in mounts.items()}
+            )
+
+        self._mounts = dict(sorted(self._mounts.items()))
 
     def _init_transport(
         self,
@@ -681,7 +687,7 @@ def _transport_for_url(self, url: URL) -> httpcore.SyncHTTPTransport:
         Returns the transport instance that should be used for a given URL.
         This will either be the standard connection pool, or a proxy.
         """
-        for pattern, transport in self._proxies.items():
+        for pattern, transport in self._mounts.items():
             if pattern.matches(url):
                 return self._transport if transport is None else transport
 
@@ -1109,17 +1115,17 @@ def close(self) -> None:
             self._state = ClientState.CLOSED
 
             self._transport.close()
-            for proxy in self._proxies.values():
-                if proxy is not None:
-                    proxy.close()
+            for transport in self._mounts.values():
+                if transport is not None:
+                    transport.close()
 
     def __enter__(self: T) -> T:
         self._state = ClientState.OPENED
 
         self._transport.__enter__()
-        for proxy in self._proxies.values():
-            if proxy is not None:
-                proxy.__enter__()
+        for transport in self._mounts.values():
+            if transport is not None:
+                transport.__enter__()
         return self
 
     def __exit__(
@@ -1131,9 +1137,9 @@ def __exit__(
         self._state = ClientState.CLOSED
 
         self._transport.__exit__(exc_type, exc_value, traceback)
-        for proxy in self._proxies.values():
-            if proxy is not None:
-                proxy.__exit__(exc_type, exc_value, traceback)
+        for transport in self._mounts.values():
+            if transport is not None:
+                transport.__exit__(exc_type, exc_value, traceback)
 
     def __del__(self) -> None:
         self.close()
@@ -1198,11 +1204,12 @@ def __init__(
         cert: CertTypes = None,
         http2: bool = False,
         proxies: ProxiesTypes = None,
+        mounts: typing.Mapping[str, httpcore.AsyncHTTPTransport] = None,
         timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG,
         limits: Limits = DEFAULT_LIMITS,
         pool_limits: Limits = None,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
-        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
+        event_hooks: typing.Mapping[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         transport: httpcore.AsyncHTTPTransport = None,
         app: typing.Callable = None,
@@ -1249,7 +1256,7 @@ def __init__(
             trust_env=trust_env,
         )
 
-        self._proxies: typing.Dict[
+        self._mounts: typing.Dict[
             URLPattern, typing.Optional[httpcore.AsyncHTTPTransport]
         ] = {
             URLPattern(key): None
@@ -1264,7 +1271,11 @@ def __init__(
             )
             for key, proxy in proxy_map.items()
         }
-        self._proxies = dict(sorted(self._proxies.items()))
+        if mounts is not None:
+            self._mounts.update(
+                {URLPattern(key): transport for key, transport in mounts.items()}
+            )
+        self._mounts = dict(sorted(self._mounts.items()))
 
     def _init_transport(
         self,
@@ -1319,7 +1330,7 @@ def _transport_for_url(self, url: URL) -> httpcore.AsyncHTTPTransport:
         Returns the transport instance that should be used for a given URL.
         This will either be the standard connection pool, or a proxy.
         """
-        for pattern, transport in self._proxies.items():
+        for pattern, transport in self._mounts.items():
             if pattern.matches(url):
                 return self._transport if transport is None else transport
 
@@ -1499,7 +1510,7 @@ async def _send_single_request(
         await timer.async_start()
 
         with map_exceptions(HTTPCORE_EXC_MAP, request=request):
-            (status_code, headers, stream, ext,) = await transport.arequest(
+            (status_code, headers, stream, ext) = await transport.arequest(
                 request.method.encode(),
                 request.url.raw,
                 headers=request.headers.raw,
@@ -1750,7 +1761,7 @@ async def aclose(self) -> None:
             self._state = ClientState.CLOSED
 
             await self._transport.aclose()
-            for proxy in self._proxies.values():
+            for proxy in self._mounts.values():
                 if proxy is not None:
                     await proxy.aclose()
 
@@ -1758,7 +1769,7 @@ async def __aenter__(self: U) -> U:
         self._state = ClientState.OPENED
 
         await self._transport.__aenter__()
-        for proxy in self._proxies.values():
+        for proxy in self._mounts.values():
             if proxy is not None:
                 await proxy.__aenter__()
         return self
@@ -1772,7 +1783,7 @@ async def __aexit__(
         self._state = ClientState.CLOSED
 
         await self._transport.__aexit__(exc_type, exc_value, traceback)
-        for proxy in self._proxies.values():
+        for proxy in self._mounts.values():
             if proxy is not None:
                 await proxy.__aexit__(exc_type, exc_value, traceback)
 


Test patch:
diff --git a/tests/client/test_async_client.py b/tests/client/test_async_client.py
--- a/tests/client/test_async_client.py
+++ b/tests/client/test_async_client.py
@@ -1,3 +1,4 @@
+import typing
 from datetime import timedelta
 
 import httpcore
@@ -188,15 +189,8 @@ async def __aexit__(self, *args):
             await super().__aexit__(*args)
             self.events.append("transport.__aexit__")
 
-    # Note that we're including 'proxies' here to *also* run through the
-    # proxy context management, although we can't easily test that at the
-    # moment, since we can't add proxies as transport instances.
-    #
-    # Once we have a more generalised Mount API we'll be able to remove this
-    # in favour of ensuring all mounts are context managed, which will
-    # also neccessarily include proxies.
     transport = Transport()
-    async with httpx.AsyncClient(transport=transport, proxies="http://www.example.com"):
+    async with httpx.AsyncClient(transport=transport):
         pass
 
     assert transport.events == [
@@ -206,6 +200,47 @@ async def __aexit__(self, *args):
     ]
 
 
+@pytest.mark.usefixtures("async_environment")
+async def test_context_managed_transport_and_mount():
+    class Transport(httpcore.AsyncHTTPTransport):
+        def __init__(self, name: str):
+            self.name: str = name
+            self.events: typing.List[str] = []
+
+        async def aclose(self):
+            # The base implementation of httpcore.AsyncHTTPTransport just
+            # calls into `.aclose`, so simple transport cases can just override
+            # this method for any cleanup, where more complex cases
+            # might want to additionally override `__aenter__`/`__aexit__`.
+            self.events.append(f"{self.name}.aclose")
+
+        async def __aenter__(self):
+            await super().__aenter__()
+            self.events.append(f"{self.name}.__aenter__")
+
+        async def __aexit__(self, *args):
+            await super().__aexit__(*args)
+            self.events.append(f"{self.name}.__aexit__")
+
+    transport = Transport(name="transport")
+    mounted = Transport(name="mounted")
+    async with httpx.AsyncClient(
+        transport=transport, mounts={"http://www.example.org": mounted}
+    ):
+        pass
+
+    assert transport.events == [
+        "transport.__aenter__",
+        "transport.aclose",
+        "transport.__aexit__",
+    ]
+    assert mounted.events == [
+        "mounted.__aenter__",
+        "mounted.aclose",
+        "mounted.__aexit__",
+    ]
+
+
 def hello_world(request):
     return httpx.Response(200, text="Hello, world!")
 
@@ -242,3 +277,28 @@ async def test_deleting_unclosed_async_client_causes_warning():
     await client.get("http://example.com")
     with pytest.warns(UserWarning):
         del client
+
+
+def unmounted(request: httpx.Request) -> httpx.Response:
+    data = {"app": "unmounted"}
+    return httpx.Response(200, json=data)
+
+
+def mounted(request: httpx.Request) -> httpx.Response:
+    data = {"app": "mounted"}
+    return httpx.Response(200, json=data)
+
+
+@pytest.mark.usefixtures("async_environment")
+async def test_mounted_transport():
+    transport = MockTransport(unmounted)
+    mounts = {"custom://": MockTransport(mounted)}
+
+    async with httpx.AsyncClient(transport=transport, mounts=mounts) as client:
+        response = await client.get("https://www.example.com")
+        assert response.status_code == 200
+        assert response.json() == {"app": "unmounted"}
+
+        response = await client.get("custom://www.example.com")
+        assert response.status_code == 200
+        assert response.json() == {"app": "mounted"}
diff --git a/tests/client/test_client.py b/tests/client/test_client.py
--- a/tests/client/test_client.py
+++ b/tests/client/test_client.py
@@ -1,3 +1,4 @@
+import typing
 from datetime import timedelta
 
 import httpcore
@@ -227,15 +228,8 @@ def __exit__(self, *args):
             super().__exit__(*args)
             self.events.append("transport.__exit__")
 
-    # Note that we're including 'proxies' here to *also* run through the
-    # proxy context management, although we can't easily test that at the
-    # moment, since we can't add proxies as transport instances.
-    #
-    # Once we have a more generalised Mount API we'll be able to remove this
-    # in favour of ensuring all mounts are context managed, which will
-    # also neccessarily include proxies.
     transport = Transport()
-    with httpx.Client(transport=transport, proxies="http://www.example.com"):
+    with httpx.Client(transport=transport):
         pass
 
     assert transport.events == [
@@ -245,6 +239,44 @@ def __exit__(self, *args):
     ]
 
 
+def test_context_managed_transport_and_mount():
+    class Transport(httpcore.SyncHTTPTransport):
+        def __init__(self, name: str):
+            self.name: str = name
+            self.events: typing.List[str] = []
+
+        def close(self):
+            # The base implementation of httpcore.SyncHTTPTransport just
+            # calls into `.close`, so simple transport cases can just override
+            # this method for any cleanup, where more complex cases
+            # might want to additionally override `__enter__`/`__exit__`.
+            self.events.append(f"{self.name}.close")
+
+        def __enter__(self):
+            super().__enter__()
+            self.events.append(f"{self.name}.__enter__")
+
+        def __exit__(self, *args):
+            super().__exit__(*args)
+            self.events.append(f"{self.name}.__exit__")
+
+    transport = Transport(name="transport")
+    mounted = Transport(name="mounted")
+    with httpx.Client(transport=transport, mounts={"http://www.example.org": mounted}):
+        pass
+
+    assert transport.events == [
+        "transport.__enter__",
+        "transport.close",
+        "transport.__exit__",
+    ]
+    assert mounted.events == [
+        "mounted.__enter__",
+        "mounted.close",
+        "mounted.__exit__",
+    ]
+
+
 def hello_world(request):
     return httpx.Response(200, text="Hello, world!")
 
@@ -300,3 +332,38 @@ def test_raw_client_header():
         ["User-Agent", f"python-httpx/{httpx.__version__}"],
         ["Example-Header", "example-value"],
     ]
+
+
+def unmounted(request: httpx.Request) -> httpx.Response:
+    data = {"app": "unmounted"}
+    return httpx.Response(200, json=data)
+
+
+def mounted(request: httpx.Request) -> httpx.Response:
+    data = {"app": "mounted"}
+    return httpx.Response(200, json=data)
+
+
+def test_mounted_transport():
+    transport = MockTransport(unmounted)
+    mounts = {"custom://": MockTransport(mounted)}
+
+    client = httpx.Client(transport=transport, mounts=mounts)
+
+    response = client.get("https://www.example.com")
+    assert response.status_code == 200
+    assert response.json() == {"app": "unmounted"}
+
+    response = client.get("custom://www.example.com")
+    assert response.status_code == 200
+    assert response.json() == {"app": "mounted"}
+
+
+def test_all_mounted_transport():
+    mounts = {"all://": MockTransport(mounted)}
+
+    client = httpx.Client(mounts=mounts)
+
+    response = client.get("https://www.example.com")
+    assert response.status_code == 200
+    assert response.json() == {"app": "mounted"}
diff --git a/tests/client/test_proxies.py b/tests/client/test_proxies.py
--- a/tests/client/test_proxies.py
+++ b/tests/client/test_proxies.py
@@ -41,12 +41,12 @@ def test_proxies_parameter(proxies, expected_proxies):
 
     for proxy_key, url in expected_proxies:
         pattern = URLPattern(proxy_key)
-        assert pattern in client._proxies
-        proxy = client._proxies[pattern]
+        assert pattern in client._mounts
+        proxy = client._mounts[pattern]
         assert isinstance(proxy, httpcore.SyncHTTPProxy)
         assert proxy.proxy_origin == url_to_origin(url)
 
-    assert len(expected_proxies) == len(client._proxies)
+    assert len(expected_proxies) == len(client._mounts)
 
 
 PROXY_URL = "http://[::1]"