Repository: encode/httpx

Issue:
Review logging.
Prompted by https://github.com/encode/httpx/discussions/2036

Since the significant `httpcore` redesign we've not got our trace logging anymore. I don't think we had a super consistent approach to what did or didn't go in there, so I'd like to review our `trace`, `debug`, and `info` level logging.

<div type='discussions-op-text'>

<sup>Originally posted by **jashandeep-sohi** January 18, 2022</sup>
Per the docs, https://www.python-httpx.org/environment_variables/, I should be able to set set `HTTPX_LOG_LEVEL=trace` and it should log detailed information about HTTP requests.

However this does not work (at least for the latest version `0.21.3`). After a little digging, it looks like the only place the `trace` extension is actually hooked into is the CLI client: https://github.com/encode/httpx/blob/83c81aa9b8ef0f7825df413a673e6d9300065f3a/httpx/_main.py#L479

Is this a bug or are the docs wrong?</div>


Discussion:
Also, it might be neat if this was also exposed in the command line client.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I've just bumped into this..

Have been trying to debug [a strange issue](https://github.com/encode/httpx/discussions/2130) with a difference in behaviour between httpx 0.20 and 0.21 involving a request hanging indefinitely, but after following the documentation found that I was hardly getting any output in the logs.

httpx 0.20
```
TRACE [2022-03-17 13:01:20] httpx._config - load_verify_locations cafile=/Users/human/Code/httpx-caching/venv/lib/python3.9/site-packages/certifi/cacert.pem
TRACE [2022-03-17 13:01:20] httpcore._async.connection_pool - get_connection_from_pool=(b'http', b'127.0.0.1', 60321)
TRACE [2022-03-17 13:01:20] httpcore._async.connection_pool - created connection=<AsyncHTTPConnection [Connecting]>
TRACE [2022-03-17 13:01:20] httpcore._async.connection_pool - adding connection to pool=<AsyncHTTPConnection [Connecting]>
TRACE [2022-03-17 13:01:20] httpcore._async.connection - open_socket origin=(b'http', b'127.0.0.1', 60321) timeout={}
TRACE [2022-03-17 13:01:20] httpcore._async.connection - create_connection socket=<httpcore._backends.anyio.SocketStream object at 0x1045bd2b0> http_version='HTTP/1.1'
TRACE [2022-03-17 13:01:20] httpcore._async.connection - connection.handle_async_request method=b'GET' url=(b'http', b'127.0.0.1', 60321, b'/conditional_get') headers=[(b'Host', b'127.0.0.1:60321'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.20.0')]
TRACE [2022-03-17 13:01:20] httpcore._async.http11 - send_request method=b'GET' url=(b'http', b'127.0.0.1', 60321, b'/conditional_get') headers=[(b'Host', b'127.0.0.1:60321'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.20.0')]
TRACE [2022-03-17 13:01:20] httpcore._async.http11 - send_data=Data(<0 bytes>)
TRACE [2022-03-17 13:01:20] httpcore._async.http11 - receive_event=EndOfMessage(headers=<Headers([])>)
TRACE [2022-03-17 13:01:20] httpcore._async.http11 - response_closed our_state=DONE their_state=DONE
DEBUG [2022-03-17 13:01:20] httpx._client - HTTP Request: GET http://127.0.0.1:60321/conditional_get "HTTP/1.1 304 Not Modified"
TRACE [2022-03-17 13:01:20] httpcore._async.http11 - response_closed our_state=IDLE their_state=IDLE
TRACE [2022-03-17 13:01:20] httpcore._async.connection_pool - removing connection from pool=<AsyncHTTPConnection [HTTP/1.1, CLOSED]>
```

httpx 0.22
```
TRACE [2022-03-17 12:54:09] httpx._config - load_ssl_context verify=True cert=None trust_env=True http2=False
TRACE [2022-03-17 12:54:09] httpx._config - load_verify_locations cafile=/Users/human/Code/httpx-caching/venv/lib/python3.9/site-packages/certifi/cacert.pem
DEBUG [2022-03-17 12:54:09] httpx._client - HTTP Request: GET http://127.0.0.1:58544/conditional_get "HTTP/1.1 304 Not Modified"
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This seems us really worth putting a bit of time into.

I think this would be a comprehensive logging output for `httpx`.

```
DEBUG: Client opened
DEBUG: Client closed

DEBUG: SSL config <...>

DEBUG: Request started <...>
DEBUG: Sent <...> bytes content
DEBUG: Request complete

DEBUG: Response started <...>
DEBUG: Received <...> bytes content
DEBUG: Response complete

DEBUG: Request error <...>

INFO: GET https://www.example.com "HTTP/1.1 200 OK"
```

There's all sorts of things we could include at the `httpcore` level, but I'd rather we just start by thinking about the `httpx` level, and then progress from there.

I don't think we really need a custom `TRACE` level here, if we can avoid it that'd probably be better.

Also, why are setting up a custom formatter at the moment? Could we avoid that and just rely on Python's default setup?
Actually, for `httpcore` we already know what we'd want to expose.

Everything that the `trace` extension shows is useful to us... https://www.encode.io/httpcore/extensions/#trace

We could either expose that using `DEBUG` level logging in `httpcore` directly, *or* we could expose it through `httpx`, by using the `trace` extension, and keeping a custom `TRACE` level.

So either this kind of thing...

```
DEBUG httpx: Request started <...>
DEBUG httpcore: connection.connect_tcp.started {'host': 'www.example.com', 'port': 443, 'local_address': None, 'timeout': None}
DEBUG httpcore: connection.connect_tcp.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f94d0>}
DEBUG httpcore: connection.start_tls.started {'ssl_context': <ssl.SSLContext object at 0x1093ee750>, 'server_hostname': b'www.example.com', 'timeout': None}
DEBUG httpcore: connection.start_tls.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f9450>}
DEBUG httpcore: http11.send_request_headers.started {'request': <Request [b'GET']>}
DEBUG httpcore: http11.send_request_headers.complete {'return_value': None}
DEBUG httpcore: http11.send_request_body.started {'request': <Request [b'GET']>}
DEBUG httpcore: http11.send_request_body.complete {'return_value': None}
DEBUG httpx: Sent <...> bytes content
DEBUG httpx: Request complete
```

Or this...

```
DEBUG httpx: Request started <...>
TRACE httpx: connection.connect_tcp.started {'host': 'www.example.com', 'port': 443, 'local_address': None, 'timeout': None}
TRACE httpx: connection.connect_tcp.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f94d0>}
TRACE httpx: connection.start_tls.started {'ssl_context': <ssl.SSLContext object at 0x1093ee750>, 'server_hostname': b'www.example.com', 'timeout': None}
TRACE httpx: connection.start_tls.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f9450>}
TRACE httpx: http11.send_request_headers.started {'request': <Request [b'GET']>}
TRACE httpx: http11.send_request_headers.complete {'return_value': None}
TRACE httpx: http11.send_request_body.started {'request': <Request [b'GET']>}
TRACE httpx: http11.send_request_body.complete {'return_value': None}
DEBUG httpx: Sent <...> bytes content
DEBUG httpx: Request complete
```


Resolving patch:
diff --git a/httpx/_client.py b/httpx/_client.py
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -1,5 +1,6 @@
 import datetime
 import enum
+import logging
 import typing
 import warnings
 from contextlib import asynccontextmanager, contextmanager
@@ -50,7 +51,6 @@
     Timer,
     URLPattern,
     get_environment_proxies,
-    get_logger,
     is_https_redirect,
     same_origin,
 )
@@ -84,7 +84,7 @@ class UseClientDefault:
 USE_CLIENT_DEFAULT = UseClientDefault()
 
 
-logger = get_logger(__name__)
+logger = logging.getLogger("httpx")
 
 USER_AGENT = f"python-httpx/{__version__}"
 ACCEPT_ENCODING = ", ".join(
@@ -1010,10 +1010,13 @@ def _send_single_request(self, request: Request) -> Response:
         self.cookies.extract_cookies(response)
         response.default_encoding = self._default_encoding
 
-        status = f"{response.status_code} {response.reason_phrase}"
-        response_line = f"{response.http_version} {status}"
-        logger.debug(
-            'HTTP Request: %s %s "%s"', request.method, request.url, response_line
+        logger.info(
+            'HTTP Request: %s %s "%s %d %s"',
+            request.method,
+            request.url,
+            response.http_version,
+            response.status_code,
+            response.reason_phrase,
         )
 
         return response
diff --git a/httpx/_config.py b/httpx/_config.py
--- a/httpx/_config.py
+++ b/httpx/_config.py
@@ -1,3 +1,4 @@
+import logging
 import os
 import ssl
 import sys
@@ -10,7 +11,7 @@
 from ._models import Headers
 from ._types import CertTypes, HeaderTypes, TimeoutTypes, URLTypes, VerifyTypes
 from ._urls import URL
-from ._utils import get_ca_bundle_from_env, get_logger
+from ._utils import get_ca_bundle_from_env
 
 DEFAULT_CIPHERS = ":".join(
     [
@@ -32,7 +33,7 @@
 )
 
 
-logger = get_logger(__name__)
+logger = logging.getLogger("httpx")
 
 
 class UnsetType:
@@ -75,12 +76,12 @@ def __init__(
         self.ssl_context = self.load_ssl_context()
 
     def load_ssl_context(self) -> ssl.SSLContext:
-        logger.trace(
-            f"load_ssl_context "
-            f"verify={self.verify!r} "
-            f"cert={self.cert!r} "
-            f"trust_env={self.trust_env!r} "
-            f"http2={self.http2!r}"
+        logger.debug(
+            "load_ssl_context verify=%r cert=%r trust_env=%r http2=%r",
+            self.verify,
+            self.cert,
+            self.trust_env,
+            self.http2,
         )
 
         if self.verify:
@@ -141,11 +142,13 @@ def load_ssl_context_verify(self) -> ssl.SSLContext:
             pass
 
         if ca_bundle_path.is_file():
-            logger.trace(f"load_verify_locations cafile={ca_bundle_path!s}")
-            context.load_verify_locations(cafile=str(ca_bundle_path))
+            cafile = str(ca_bundle_path)
+            logger.debug("load_verify_locations cafile=%r", cafile)
+            context.load_verify_locations(cafile=cafile)
         elif ca_bundle_path.is_dir():
-            logger.trace(f"load_verify_locations capath={ca_bundle_path!s}")
-            context.load_verify_locations(capath=str(ca_bundle_path))
+            capath = str(ca_bundle_path)
+            logger.debug("load_verify_locations capath=%r", capath)
+            context.load_verify_locations(capath=capath)
 
         self._load_client_certs(context)
 
diff --git a/httpx/_utils.py b/httpx/_utils.py
--- a/httpx/_utils.py
+++ b/httpx/_utils.py
@@ -1,10 +1,8 @@
 import codecs
 import email.message
-import logging
 import mimetypes
 import os
 import re
-import sys
 import time
 import typing
 from pathlib import Path
@@ -196,50 +194,6 @@ def obfuscate_sensitive_headers(
         yield k, v
 
 
-_LOGGER_INITIALIZED = False
-TRACE_LOG_LEVEL = 5
-
-
-class Logger(logging.Logger):
-    # Stub for type checkers.
-    def trace(self, message: str, *args: typing.Any, **kwargs: typing.Any) -> None:
-        ...  # pragma: no cover
-
-
-def get_logger(name: str) -> Logger:
-    """
-    Get a `logging.Logger` instance, and optionally
-    set up debug logging based on the HTTPX_LOG_LEVEL environment variable.
-    """
-    global _LOGGER_INITIALIZED
-
-    if not _LOGGER_INITIALIZED:
-        _LOGGER_INITIALIZED = True
-        logging.addLevelName(TRACE_LOG_LEVEL, "TRACE")
-
-        log_level = os.environ.get("HTTPX_LOG_LEVEL", "").upper()
-        if log_level in ("DEBUG", "TRACE"):
-            logger = logging.getLogger("httpx")
-            logger.setLevel(logging.DEBUG if log_level == "DEBUG" else TRACE_LOG_LEVEL)
-            handler = logging.StreamHandler(sys.stderr)
-            handler.setFormatter(
-                logging.Formatter(
-                    fmt="%(levelname)s [%(asctime)s] %(name)s - %(message)s",
-                    datefmt="%Y-%m-%d %H:%M:%S",
-                )
-            )
-            logger.addHandler(handler)
-
-    logger = logging.getLogger(name)
-
-    def trace(message: str, *args: typing.Any, **kwargs: typing.Any) -> None:
-        logger.log(TRACE_LOG_LEVEL, message, *args, **kwargs)
-
-    logger.trace = trace  # type: ignore
-
-    return typing.cast(Logger, logger)
-
-
 def port_or_default(url: "URL") -> typing.Optional[int]:
     if url.port is not None:
         return url.port


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,6 +1,8 @@
+import logging
 import os
 import random
 
+import certifi
 import pytest
 
 import httpx
@@ -14,7 +16,6 @@
     parse_header_links,
     same_origin,
 )
-from tests.utils import override_log_level
 
 from .common import TESTS_DIR
 
@@ -78,42 +79,59 @@ def test_parse_header_links(value, expected):
     assert parse_header_links(value) == expected
 
 
-@pytest.mark.anyio
-async def test_logs_debug(server, capsys):
-    with override_log_level("debug"):
-        async with httpx.AsyncClient() as client:
-            response = await client.get(server.url)
-            assert response.status_code == 200
-    stderr = capsys.readouterr().err
-    assert 'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"' in stderr
-
-
-@pytest.mark.anyio
-async def test_logs_trace(server, capsys):
-    with override_log_level("trace"):
-        async with httpx.AsyncClient() as client:
-            response = await client.get(server.url)
-            assert response.status_code == 200
-    stderr = capsys.readouterr().err
-    assert 'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"' in stderr
-
-
-@pytest.mark.anyio
-async def test_logs_redirect_chain(server, capsys):
-    with override_log_level("debug"):
-        async with httpx.AsyncClient(follow_redirects=True) as client:
-            response = await client.get(server.url.copy_with(path="/redirect_301"))
-            assert response.status_code == 200
-
-    stderr = capsys.readouterr().err.strip()
-    redirected_request_line, ok_request_line = stderr.split("\n")
-    assert redirected_request_line.endswith(
-        "HTTP Request: GET http://127.0.0.1:8000/redirect_301 "
-        '"HTTP/1.1 301 Moved Permanently"'
-    )
-    assert ok_request_line.endswith(
-        'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"'
-    )
+def test_logging_request(server, caplog):
+    caplog.set_level(logging.INFO)
+    with httpx.Client() as client:
+        response = client.get(server.url)
+        assert response.status_code == 200
+
+    assert caplog.record_tuples == [
+        (
+            "httpx",
+            logging.INFO,
+            'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"',
+        )
+    ]
+
+
+def test_logging_redirect_chain(server, caplog):
+    caplog.set_level(logging.INFO)
+    with httpx.Client(follow_redirects=True) as client:
+        response = client.get(server.url.copy_with(path="/redirect_301"))
+        assert response.status_code == 200
+
+    assert caplog.record_tuples == [
+        (
+            "httpx",
+            logging.INFO,
+            'HTTP Request: GET http://127.0.0.1:8000/redirect_301 "HTTP/1.1 301 Moved Permanently"',
+        ),
+        (
+            "httpx",
+            logging.INFO,
+            'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"',
+        ),
+    ]
+
+
+def test_logging_ssl(caplog):
+    caplog.set_level(logging.DEBUG)
+    with httpx.Client():
+        pass
+
+    cafile = certifi.where()
+    assert caplog.record_tuples == [
+        (
+            "httpx",
+            logging.DEBUG,
+            "load_ssl_context verify=True cert=None trust_env=True http2=False",
+        ),
+        (
+            "httpx",
+            logging.DEBUG,
+            f"load_verify_locations cafile='{cafile}'",
+        ),
+    ]
 
 
 def test_get_ssl_cert_file():
diff --git a/tests/utils.py b/tests/utils.py
deleted file mode 100644
--- a/tests/utils.py
+++ /dev/null
@@ -1,21 +0,0 @@
-import contextlib
-import logging
-import os
-import typing
-
-from httpx import _utils
-
-
-@contextlib.contextmanager
-def override_log_level(log_level: str) -> typing.Iterator[None]:
-    os.environ["HTTPX_LOG_LEVEL"] = log_level
-
-    # Force a reload on the logging handlers
-    _utils._LOGGER_INITIALIZED = False
-    _utils.get_logger("httpx")
-
-    try:
-        yield
-    finally:
-        # Reset the logger so we don't have verbose output in all unit tests
-        logging.getLogger("httpx").handlers = []