Repository: encode/httpx
Pull request #3178 (2024-04-25 01:53:59)

add socks5h proxy support
In some case socks5h is needed to resolve the hostname.

<!-- Thanks for contributing to HTTPX! 💚
Given this is a project maintained by volunteers, please read this template to not waste your time, or ours! 😁 -->

# Summary

add socks5h proxy support.
<!-- Write a small summary about what is happening here. -->

# Checklist

- [x] I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
- [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
- [x] I've updated the documentation accordingly.


T-256: `socks5` are resolving hostnames in remote side by default (socks5h).
Also, see [httpx-socks](https://github.com/romis2012/httpx-socks) where you can explicitly set `rdns=False` to locally resolving.

liubin: @T-256 Indeed I run into a complexity case that socks5 not work:

```bash
12:39:33-bin~/tmppp\ -> export https_proxy=socks5h://127.0.0.1:15666
12:39:49-bin~/tmppp\ -> curl -I https://twitter.com
HTTP/2 403 
date: Fri, 26 Apr 2024 04:39:51 GMT
perf: 7402827104
server: tsa_p
set-cookie: guest_id_marketing=v1%3A171410639188824760; Max-Age=63072000; Expires=Sun, 26 Apr 2026 04:39:51 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
set-cookie: guest_id_ads=v1%3A171410639188824760; Max-Age=63072000; Expires=Sun, 26 Apr 2026 04:39:51 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
set-cookie: personalization_id="v1_H5m2zAEnPFfEJv5Q0c4avQ=="; Max-Age=63072000; Expires=Sun, 26 Apr 2026 04:39:51 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
set-cookie: guest_id=v1%3A171410639188824760; Max-Age=63072000; Expires=Sun, 26 Apr 2026 04:39:51 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
x-powered-by: Express
... ...

12:39:51-bin~/tmppp\ -> export https_proxy=socks5://127.0.0.1:15666
12:39:56-bin~/tmppp\ -> curl -I https://twitter.com
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to twitter.com:443 
```

And my code with `requests` only work with socks5h too, if set to socks5, it will get an error:

```
  File "/Users/bin/Library/Python/3.9/lib/python/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: (MaxRetryError("SOCKSHTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /BAAI/bge-small-en-v1.5/resolve/main/config.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))"), '(Request ID: a0fdd8ed-dd99-4b38-bf53-47f133004e22)')
```

My code using both requests and httpx, it is not controlled by me, but be the packages that I'm using directly.

So let httpx to support socks5h is the simplest way for my case.
tomchristie: Thanks, yep we probably should have `socks5h` proxy support.

Have you tested the above? I don't think it'll work, although you'll be able to pass `socks5h://` through, support for it isn't actually implemented in `httpcore` yet.

Here's where you'll actually need to start on this...

* A draft PR for `httpcore` adding [documentation for `sock5h` support](https://www.encode.io/httpcore/proxies/#socks-proxy-support).
* Update the PR to [implement the functionality](https://github.com/encode/httpcore/blob/master/httpcore/_sync/socks_proxy.py) as described in the docs.

mateuszlitwin: Hello @tomchristie! I am also running into this issue, i.e. lack of `socks5h://` support. I cannot use `socks5://` because it is interpreted different by other tools (e.g. curl). I did test this PR + 1 line change in  https://github.com/encode/httpcore/blob/master/httpcore/_models.py#L294 to add default port sock5h and it works as expected - traffic is sent to the proxy without resolving hostname.
longouyang: Hi @tomchristie - firstly, thanks so much for all the work you've done on this project. We sure use the library a lot at $DAYJOB.

Also - I'd be happy to pitch in $250 support to the project if we could get socks5h support merged.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1d32e53..460e315 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,6 +16,7 @@ This release introduces an `httpx.SSLContext()` class and `ssl_context` paramete
 * Ensure JSON request bodies are compact. (#3363)
 * Review URL percent escape sets, based on WHATWG spec. (#3371, #3373)
 * Ensure `certifi` and `httpcore` are only imported if required. (#3377)
+* Treat `socks5h` as a valid proxy scheme. (#3178)
 
 ## 0.27.2 (27th August, 2024)
 
diff --git a/httpx/_config.py b/httpx/_config.py
index 5a1a98a..3fd5e1d 100644
--- a/httpx/_config.py
+++ b/httpx/_config.py
@@ -262,7 +262,7 @@ class Proxy:
         url = URL(url)
         headers = Headers(headers)
 
-        if url.scheme not in ("http", "https", "socks5"):
+        if url.scheme not in ("http", "https", "socks5", "socks5h"):
             raise ValueError(f"Unknown scheme for proxy URL {url!r}")
 
         if url.username or url.password:
diff --git a/httpx/_transports/default.py b/httpx/_transports/default.py
index 50ff910..85d0f5f 100644
--- a/httpx/_transports/default.py
+++ b/httpx/_transports/default.py
@@ -189,7 +189,7 @@ class HTTPTransport(BaseTransport):
                 http2=http2,
                 socket_options=socket_options,
             )
-        elif proxy.url.scheme == "socks5":
+        elif proxy.url.scheme in ("socks5", "socks5h"):
             try:
                 import socksio  # noqa
             except ImportError:  # pragma: no cover
@@ -215,7 +215,7 @@ class HTTPTransport(BaseTransport):
             )
         else:  # pragma: no cover
             raise ValueError(
-                "Proxy protocol must be either 'http', 'https', or 'socks5',"
+                "Proxy protocol must be either 'http', 'https', 'socks5', or 'socks5h',"
                 f" but got {proxy.url.scheme!r}."
             )
 
@@ -338,7 +338,7 @@ class AsyncHTTPTransport(AsyncBaseTransport):
                 http2=http2,
                 socket_options=socket_options,
             )
-        elif proxy.url.scheme == "socks5":
+        elif proxy.url.scheme in ("socks5", "socks5h"):
             try:
                 import socksio  # noqa
             except ImportError:  # pragma: no cover
@@ -364,7 +364,7 @@ class AsyncHTTPTransport(AsyncBaseTransport):
             )
         else:  # pragma: no cover
             raise ValueError(
-                "Proxy protocol must be either 'http', 'https', or 'socks5',"
+                "Proxy protocol must be either 'http', 'https', 'socks5', or 'socks5h',"
                 " but got {proxy.url.scheme!r}."
             )
 


Test patch:
diff --git a/tests/client/test_proxies.py b/tests/client/test_proxies.py
index 90a92f5..3e4090d 100644
--- a/tests/client/test_proxies.py
+++ b/tests/client/test_proxies.py
@@ -16,15 +16,16 @@ def url_to_origin(url: str) -> httpcore.URL:
 def test_socks_proxy():
     url = httpx.URL("http://www.example.com")
 
-    client = httpx.Client(proxy="socks5://localhost/")
-    transport = client._transport_for_url(url)
-    assert isinstance(transport, httpx.HTTPTransport)
-    assert isinstance(transport._pool, httpcore.SOCKSProxy)
+    for proxy in ("socks5://localhost/", "socks5h://localhost/"):
+        client = httpx.Client(proxy=proxy)
+        transport = client._transport_for_url(url)
+        assert isinstance(transport, httpx.HTTPTransport)
+        assert isinstance(transport._pool, httpcore.SOCKSProxy)
 
-    async_client = httpx.AsyncClient(proxy="socks5://localhost/")
-    async_transport = async_client._transport_for_url(url)
-    assert isinstance(async_transport, httpx.AsyncHTTPTransport)
-    assert isinstance(async_transport._pool, httpcore.AsyncSOCKSProxy)
+        async_client = httpx.AsyncClient(proxy=proxy)
+        async_transport = async_client._transport_for_url(url)
+        assert isinstance(async_transport, httpx.AsyncHTTPTransport)
+        assert isinstance(async_transport._pool, httpcore.AsyncSOCKSProxy)
 
 
 PROXY_URL = "http://[::1]"