Repository: encode/httpx

Issue:
Support for Event Hooks
I'm currently using [https://github.com/encode/requests-async](https://github.com/encode/requests-async) for one of my projects and now I wanted to switch to HTTPX, but I can't find out if it supports event hooks like this [https://requests.readthedocs.io/en/master/user/advanced/#event-hooks](https://requests.readthedocs.io/en/master/user/advanced/#event-hooks).

Or is there another easy way to add hooks onto an `AsyncClient` object to be called every request?


Discussion:
Right now it doesn’t have any API for that, no. It’s certainly something we want to consider, yes.

It’d be really helpful to start getting some user feedback to help us figure out if event hooks are what we’d like to use here, or if we should be considering a more general middleware API.

What’s your particular use case here?
We use hooks to collect metrics about all the requests, mostly for monitoring purposes. It would be nice if the  `AsyncClient` supported it similar to how `Session` in requests and requests_async does. So one could do something like this: 

```python3
def response_hook(response, *args, **kwargs): 
    # Log status of response, etc...
    ...

def get_client():
    client = httpx.AsyncClient()
    client.hooks["response"].append(response_hook) 
    return client
```
A more general middleware API could be a good idea and definitely solves the problem.

It's not a big deal to make a workaround for our specific case, but since the other libraries supported it I thought there might be another way of doing it in HTTPX which I couldn't find.  
IMO a general middleware API like the one drafted in #783 would be the most versatile way to provide these "on request/on response" hooks, as well as address a bunch of other use cases…

```python
class MonitoringMiddleware:
    def __init__(self, parent):
        self.parent = parent

    def __call__(self, request, scope, timeout):
        request_hook(request, ...)
        response = yield from self.parent(request, scope, timeout)
        response_hook(response, ...)
        return response

client = httpx.Client()
client.add_middleware(MonitoringMiddleware)
```
Though we might want to provide a more streamlined API, that might just happen to be implemented as middleware, e.g...

```python
def monitor_requests(request):
    # Process request...
    response = yield request
    # Process response...
    
client = httpx.Client(hooks=[monitor_requests])
```

(Somewhat inspired by https://github.com/encode/starlette/pull/799.)
Jumping on here instead of creating a new issue:

I've got some project code that patched both requests and the stdlib to allow for hooks that logged requests and redirects.  Went to go upstream it to requests and ended up here instead.  Is there enough interest here to make a proof of concept worthwhile?
Hey,

Yes, I think we could go for a PR drafting things out...

Forget the middleware stuff I mentioned above at the time, I’d agree a simpler and more integrated « register request/response callbacks » API is probably a better first step.

We could start off back from https://github.com/encode/httpx/issues/790#issuecomment-577284027, however I think we should consider a slightly different API design approach, in the form of registering lists of callbacks on client init (instead of mutating an attribute on the client). This is somewhat in line with Starlette’s event handlers API, and less fiddly than « expose a dict with arbitrary 'request'/'response' keys that might be mutated anytime etc ». So something like...

```python
import httpx

client = httpx.Client(
    on_request=[...],
    on_response=[...],
)
```
Okay, so I've been thinking about this for a bit, and it's time to get this down.
Here's what I'm thinking for the design here...

```python
# Setting on __init__.
client = httpx.Client(event_hooks={'request': ..., 'response': ...})

# Mutating the property.
client.event_hooks['request'] = ...
client.event_hooks['response'] = ...

# Should be included at the top-level API
httpx.get(..., event_hooks={...})
```

We want to support either none, one, or many hooks on each event type, so I think `event_hooks` should be a mapping type that always returns lists.

I think we want to support 4 event hooks, that will give us a nice comprehensive hook into the complete request/response flow...

* `request` - Signature: `func(request)`. Called after everything is prepared on the request, including any initial authentication.
* `auth` - Signature: `func(response, request)`. For most folks this is a corner case. Only called if the auth flow invokes additional requests beyond the original outgoing request. 
* `redirect` - Signature: `func(response, request)`. Called if the redirect flow invokes multiple requests. Not called for `allow_redirects=False`.
* `response` - Signature: `func(response)`. Called once we're done & happy. Obvs.

One thing I don't feel *super* absolutely clear on is for the async case what we should be expecting from the functions. I *think* that the sensible thing to do is always require regular callables for the sync case and always require async callables for the async case.

I don't really want to get into "allow either async/sync callables for `AsyncClient` and we'll magically introspect which you're using".

Sticking with `event_hooks` on AsyncClient being mandated as always being `async` function is at least simple & clear.

Aside: One neat pattern that event hooks would give us is this...

```python
client = httpx.Client(event_hooks={'response': lambda r: r.raise_for_status()})
```

Resolving patch:
diff --git a/httpx/_client.py b/httpx/_client.py
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -74,9 +74,12 @@ def __init__(
         cookies: CookieTypes = None,
         timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
+        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         trust_env: bool = True,
     ):
+        event_hooks = {} if event_hooks is None else event_hooks
+
         self._base_url = self._enforce_trailing_slash(URL(base_url))
 
         self._auth = self._build_auth(auth)
@@ -85,6 +88,10 @@ def __init__(
         self._cookies = Cookies(cookies)
         self._timeout = Timeout(timeout)
         self.max_redirects = max_redirects
+        self._event_hooks = {
+            "request": list(event_hooks.get("request", [])),
+            "response": list(event_hooks.get("response", [])),
+        }
         self._trust_env = trust_env
         self._netrc = NetRCInfo()
         self._is_closed = True
@@ -133,6 +140,19 @@ def timeout(self) -> Timeout:
     def timeout(self, timeout: TimeoutTypes) -> None:
         self._timeout = Timeout(timeout)
 
+    @property
+    def event_hooks(self) -> typing.Dict[str, typing.List[typing.Callable]]:
+        return self._event_hooks
+
+    @event_hooks.setter
+    def event_hooks(
+        self, event_hooks: typing.Dict[str, typing.List[typing.Callable]]
+    ) -> None:
+        self._event_hooks = {
+            "request": list(event_hooks.get("request", [])),
+            "response": list(event_hooks.get("response", [])),
+        }
+
     @property
     def auth(self) -> typing.Optional[Auth]:
         """
@@ -532,6 +552,7 @@ def __init__(
         limits: Limits = DEFAULT_LIMITS,
         pool_limits: Limits = None,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
+        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         transport: httpcore.SyncHTTPTransport = None,
         app: typing.Callable = None,
@@ -544,6 +565,7 @@ def __init__(
             cookies=cookies,
             timeout=timeout,
             max_redirects=max_redirects,
+            event_hooks=event_hooks,
             base_url=base_url,
             trust_env=trust_env,
         )
@@ -739,6 +761,13 @@ def send(
             finally:
                 response.close()
 
+        try:
+            for hook in self._event_hooks["response"]:
+                hook(response)
+        except Exception:
+            response.close()
+            raise
+
         return response
 
     def _send_handling_auth(
@@ -752,6 +781,9 @@ def _send_handling_auth(
         auth_flow = auth.sync_auth_flow(request)
         request = next(auth_flow)
 
+        for hook in self._event_hooks["request"]:
+            hook(request)
+
         while True:
             response = self._send_handling_redirects(
                 request,
@@ -1153,6 +1185,7 @@ def __init__(
         limits: Limits = DEFAULT_LIMITS,
         pool_limits: Limits = None,
         max_redirects: int = DEFAULT_MAX_REDIRECTS,
+        event_hooks: typing.Dict[str, typing.List[typing.Callable]] = None,
         base_url: URLTypes = "",
         transport: httpcore.AsyncHTTPTransport = None,
         app: typing.Callable = None,
@@ -1165,6 +1198,7 @@ def __init__(
             cookies=cookies,
             timeout=timeout,
             max_redirects=max_redirects,
+            event_hooks=event_hooks,
             base_url=base_url,
             trust_env=trust_env,
         )
@@ -1362,6 +1396,13 @@ async def send(
             finally:
                 await response.aclose()
 
+        try:
+            for hook in self._event_hooks["response"]:
+                await hook(response)
+        except Exception:
+            await response.aclose()
+            raise
+
         return response
 
     async def _send_handling_auth(
@@ -1375,6 +1416,9 @@ async def _send_handling_auth(
         auth_flow = auth.async_auth_flow(request)
         request = await auth_flow.__anext__()
 
+        for hook in self._event_hooks["request"]:
+            await hook(request)
+
         while True:
             response = await self._send_handling_redirects(
                 request,


Test patch:
diff --git a/tests/client/test_event_hooks.py b/tests/client/test_event_hooks.py
new file mode 100644
--- /dev/null
+++ b/tests/client/test_event_hooks.py
@@ -0,0 +1,189 @@
+import pytest
+
+import httpx
+from tests.utils import AsyncMockTransport, MockTransport
+
+
+def app(request: httpx.Request) -> httpx.Response:
+    if request.url.path == "/redirect":
+        return httpx.Response(303, headers={"server": "testserver", "location": "/"})
+    elif request.url.path.startswith("/status/"):
+        status_code = int(request.url.path[-3:])
+        return httpx.Response(status_code, headers={"server": "testserver"})
+
+    return httpx.Response(200, headers={"server": "testserver"})
+
+
+def test_event_hooks():
+    events = []
+
+    def on_request(request):
+        events.append({"event": "request", "headers": dict(request.headers)})
+
+    def on_response(response):
+        events.append({"event": "response", "headers": dict(response.headers)})
+
+    event_hooks = {"request": [on_request], "response": [on_response]}
+
+    with httpx.Client(event_hooks=event_hooks, transport=MockTransport(app)) as http:
+        http.get("http://127.0.0.1:8000/", auth=("username", "password"))
+
+    assert events == [
+        {
+            "event": "request",
+            "headers": {
+                "host": "127.0.0.1:8000",
+                "user-agent": f"python-httpx/{httpx.__version__}",
+                "accept": "*/*",
+                "accept-encoding": "gzip, deflate, br",
+                "connection": "keep-alive",
+                "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=",
+            },
+        },
+        {
+            "event": "response",
+            "headers": {"server": "testserver"},
+        },
+    ]
+
+
+def test_event_hooks_raising_exception(server):
+    def raise_on_4xx_5xx(response):
+        response.raise_for_status()
+
+    event_hooks = {"response": [raise_on_4xx_5xx]}
+
+    with httpx.Client(event_hooks=event_hooks, transport=MockTransport(app)) as http:
+        try:
+            http.get("http://127.0.0.1:8000/status/400")
+        except httpx.HTTPStatusError as exc:
+            assert exc.response.is_closed
+
+
+@pytest.mark.usefixtures("async_environment")
+async def test_async_event_hooks():
+    events = []
+
+    async def on_request(request):
+        events.append({"event": "request", "headers": dict(request.headers)})
+
+    async def on_response(response):
+        events.append({"event": "response", "headers": dict(response.headers)})
+
+    event_hooks = {"request": [on_request], "response": [on_response]}
+
+    async with httpx.AsyncClient(
+        event_hooks=event_hooks, transport=AsyncMockTransport(app)
+    ) as http:
+        await http.get("http://127.0.0.1:8000/", auth=("username", "password"))
+
+    assert events == [
+        {
+            "event": "request",
+            "headers": {
+                "host": "127.0.0.1:8000",
+                "user-agent": f"python-httpx/{httpx.__version__}",
+                "accept": "*/*",
+                "accept-encoding": "gzip, deflate, br",
+                "connection": "keep-alive",
+                "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=",
+            },
+        },
+        {
+            "event": "response",
+            "headers": {"server": "testserver"},
+        },
+    ]
+
+
+@pytest.mark.usefixtures("async_environment")
+async def test_async_event_hooks_raising_exception():
+    async def raise_on_4xx_5xx(response):
+        response.raise_for_status()
+
+    event_hooks = {"response": [raise_on_4xx_5xx]}
+
+    async with httpx.AsyncClient(
+        event_hooks=event_hooks, transport=AsyncMockTransport(app)
+    ) as http:
+        try:
+            await http.get("http://127.0.0.1:8000/status/400")
+        except httpx.HTTPStatusError as exc:
+            assert exc.response.is_closed
+
+
+def test_event_hooks_with_redirect():
+    """
+    A redirect request should not trigger a second 'request' event hook.
+    """
+
+    events = []
+
+    def on_request(request):
+        events.append({"event": "request", "headers": dict(request.headers)})
+
+    def on_response(response):
+        events.append({"event": "response", "headers": dict(response.headers)})
+
+    event_hooks = {"request": [on_request], "response": [on_response]}
+
+    with httpx.Client(event_hooks=event_hooks, transport=MockTransport(app)) as http:
+        http.get("http://127.0.0.1:8000/redirect", auth=("username", "password"))
+
+    assert events == [
+        {
+            "event": "request",
+            "headers": {
+                "host": "127.0.0.1:8000",
+                "user-agent": f"python-httpx/{httpx.__version__}",
+                "accept": "*/*",
+                "accept-encoding": "gzip, deflate, br",
+                "connection": "keep-alive",
+                "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=",
+            },
+        },
+        {
+            "event": "response",
+            "headers": {"server": "testserver"},
+        },
+    ]
+
+
+@pytest.mark.usefixtures("async_environment")
+async def test_async_event_hooks_with_redirect():
+    """
+    A redirect request should not trigger a second 'request' event hook.
+    """
+
+    events = []
+
+    async def on_request(request):
+        events.append({"event": "request", "headers": dict(request.headers)})
+
+    async def on_response(response):
+        events.append({"event": "response", "headers": dict(response.headers)})
+
+    event_hooks = {"request": [on_request], "response": [on_response]}
+
+    async with httpx.AsyncClient(
+        event_hooks=event_hooks, transport=AsyncMockTransport(app)
+    ) as http:
+        await http.get("http://127.0.0.1:8000/redirect", auth=("username", "password"))
+
+    assert events == [
+        {
+            "event": "request",
+            "headers": {
+                "host": "127.0.0.1:8000",
+                "user-agent": f"python-httpx/{httpx.__version__}",
+                "accept": "*/*",
+                "accept-encoding": "gzip, deflate, br",
+                "connection": "keep-alive",
+                "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=",
+            },
+        },
+        {
+            "event": "response",
+            "headers": {"server": "testserver"},
+        },
+    ]
diff --git a/tests/client/test_properties.py b/tests/client/test_properties.py
--- a/tests/client/test_properties.py
+++ b/tests/client/test_properties.py
@@ -49,3 +49,12 @@ def test_client_timeout():
     assert client.timeout.read == expected_timeout
     assert client.timeout.write == expected_timeout
     assert client.timeout.pool == expected_timeout
+
+
+def test_client_event_hooks():
+    def on_request(request):
+        pass  # pragma: nocover
+
+    client = httpx.Client()
+    client.event_hooks = {"request": [on_request]}
+    assert client.event_hooks == {"request": [on_request], "response": []}