Repository: python-telegram-bot/python-telegram-bot
Pull request: Rename Parameter `proxy_url` to `proxy` and Support `httpx.Proxy` as Input
Closes #3844

<!--
Hey! You're PRing? Cool!
Please be sure to check out our contribution guide (https://github.com/python-telegram-bot/python-telegram-bot/blob/master/.github/CONTRIBUTING.rst).
Especially, please have a look at the check list for PRs (https://github.com/python-telegram-bot/python-telegram-bot/blob/master/.github/CONTRIBUTING.rst#check-list-for-prs). Feel free to copy (parts of) the checklist to the PR description to remind you or the maintainers of open points or if you have questions on anything.
-->



Problem statement:
[FEATURE] SSL verification + custom proxies
### What kind of feature are you missing? Where do you notice a shortcoming of PTB?

There is no way to disable ssl verification for proxies and pass a custom httpx.Proxy, instead of just a string.

### Describe the solution you'd like

Some proxies work as a man-in-the-middle and may lead to errors, like: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006). 

Due to this reason, I think it's necessary to allow control over ssl verification in HTTPXRequest. Just add an argument like `ssl_verify: bool = True` and add `verify=ssl_verify` to `self._client_kwargs`.

Another thing I'd love to see, is to change the type of:
`proxy_url: Optional[str] = None,`
to
`proxy_url: Optional[Union[str, httpx.Proxy] = None,`

This is due to some proxies requiring authorization through headers, which could be passed as:
```
httpx.Proxy(
    url=proxy_url,
    headers={
        "x-secret-header": "secret value"
    },
)
```
(moreover it might be used for passing logging headers to the proxy and so on)


### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Hints from issue discussion:
1. I really dislike verify=False. It is better to add a custom CA cert, if and only if the user trusts the entity performing ssl mitm.
I would argue that we shouldn't easily allow disabling ssl verification as it compromises the security guarantees of Telegram. If someone really really want it then they should inherit from HTTPXRequest (or monkey patch it) and mangle whatever they want, but then the responsibility and the implications are on them. 

2. The suggested approach to setting a proxy seems like a solid solution. We should do it.
@tsnoam 

1 - In my specific case, there is simply no way to provide a custom CA cert, but I can agree that this might be a one-of-a-kind problem, so it shouldn't be implemented. However, I'd ask you to review the patch I'm currently using:
```
def _build_client(self) -> httpx.AsyncClient:
    self._client_kwargs['verify'] = False
    return httpx.AsyncClient(**self._client_kwargs)
```
2 - If I could be of any assistance, I'd be glad to help. I can make a pull request, but since this seems like a minor change, I'd probably spend more time reading guidelines.



1: Your snippet looks okay to me. Just be aware that we make no guarantees about the stability of protected methods :)

> 2: if we allow `httpx.Proxie` input, then the name "proxy_url" is no longer the best. I suggest deprecating that argument in favor of a new `proxy` argument that accepts both a string or such an object.
One more comment: httpx accepts multiple proxies, e.f.

```python
proxies = {
    # Route all traffic through a proxy by default...
    "all://": "http://localhost:8030",
    # But don't use proxies for HTTPS requests to "domain.io"...
    "https://domain.io": None,
    # And use another proxy for requests to "example.com" and its subdomains...
    "all://*example.com": "http://localhost:8031",
    # And yet another proxy if HTTP is used,
    # and the "internal" subdomain on port 5550 is requested...
    "http://internal.example.com:5550": "http://localhost:8032",
}
```

Since in our case we only have one target to query, I see no need to allow for such functionality. Cc @tsnoam 

@codereptile you are ofc very welcome to contribute if you are interested :) 
v0.25.0 of httpx adds support for https proxies is encode/httpx#2845. Would this feature resolve the verification problem mentioned in the original comment (possibly after exposing this feature appropriately via PTBs interface)? CC @tsnoam 
@Bibo-Joshi As far as I'm concerned, yes, you could set up the https proxy correctly, if PTB would expose the https proxy feature.
Seems like both tickets could be solved with a single PR.
@Bibo-Joshi I think that ptb already supports getting a `httpx.Proxy` object over `proxy_url` as there's no enforcement to it being `str`.

So if I read the code right, the problem is semantics & documentation. Nothing on the code itself.

So basically, in ideal world what could have been changed is:
1. rename `proxy_url` to `proxy`
2. change `proxy` type from `str` to `httpx.Proxy`.
and that's it.

again, I hope I got things right here...
@tsnoam Yes, you can actually just throw in the proxy and it'll work (and that's how I did), but the type/name change would be nice to avoid IDE/mypy errors

Commit message: Rename `proxy_url` to `proxy` and add type hint for `httpx.Proxy`

Patch to review:
diff --git a/telegram/request/_httpxrequest.py b/telegram/request/_httpxrequest.py
index 09898b65e71..09062a2ac56 100644
--- a/telegram/request/_httpxrequest.py
+++ b/telegram/request/_httpxrequest.py
@@ -17,16 +17,18 @@
 # You should have received a copy of the GNU Lesser Public License
 # along with this program.  If not, see [http://www.gnu.org/licenses/].
 """This module contains methods to make POST and GET requests using the httpx library."""
-from typing import Optional, Tuple
+from typing import Optional, Tuple, Union
 
 import httpx
 
 from telegram._utils.defaultvalue import DefaultValue
 from telegram._utils.logging import get_logger
 from telegram._utils.types import HTTPVersion, ODVInput
+from telegram._utils.warnings import warn
 from telegram.error import NetworkError, TimedOut
 from telegram.request._baserequest import BaseRequest
 from telegram.request._requestdata import RequestData
+from telegram.warnings import PTBDeprecationWarning
 
 # Note to future devs:
 # Proxies are currently only tested manually. The httpx development docs have a nice guide on that:
@@ -49,17 +51,10 @@ class HTTPXRequest(BaseRequest):
             Note:
                 Independent of the value, one additional connection will be reserved for
                 :meth:`telegram.Bot.get_updates`.
-        proxy_url (:obj:`str`, optional): The URL to the proxy server. For example
-            ``'http://127.0.0.1:3128'`` or ``'socks5://127.0.0.1:3128'``. Defaults to :obj:`None`.
+        proxy_url (:obj:`str`, optional): Legacy name for :paramref:`proxy`, kept for backward
+            compatibility. Defaults to :obj:`None`.
 
-            Note:
-                * The proxy URL can also be set via the environment variables ``HTTPS_PROXY`` or
-                  ``ALL_PROXY``. See `the docs of httpx`_ for more info.
-                * For Socks5 support, additional dependencies are required. Make sure to install
-                  PTB via :command:`pip install "python-telegram-bot[socks]"` in this case.
-                * Socks5 proxies can not be set via environment variables.
-
-            .. _the docs of httpx: https://www.python-httpx.org/environment_variables/#proxies
+            .. deprecated:: NEXT.VERSION
         read_timeout (:obj:`float` | :obj:`None`, optional): If passed, specifies the maximum
             amount of time (in seconds) to wait for a response from Telegram's server.
             This value is used unless a different value is passed to :meth:`do_request`.
@@ -91,6 +86,20 @@ class HTTPXRequest(BaseRequest):
 
             .. versionchanged:: 20.5
                 Accept ``"2"`` as a valid value.
+        proxy (:obj:`str` | ``httpx.Proxy``, optional): The URL to a proxy server or an
+            ``httpx.Proxy`` object. For example ``'http://127.0.0.1:3128'`` or
+            ``'socks5://127.0.0.1:3128'``. Defaults to :obj:`None`.
+
+            Note:
+                * The proxy URL can also be set via the environment variables ``HTTPS_PROXY`` or
+                  ``ALL_PROXY``. See `the docs of httpx`_ for more info.
+                * For Socks5 support, additional dependencies are required. Make sure to install
+                  PTB via :command:`pip install "python-telegram-bot[socks]"` in this case.
+                * Socks5 proxies can not be set via environment variables.
+
+            .. _the docs of httpx: https://www.python-httpx.org/environment_variables/#proxies
+
+            .. versionadded:: NEXT.VERSION
 
     """
 
@@ -99,13 +108,26 @@ class HTTPXRequest(BaseRequest):
     def __init__(
         self,
         connection_pool_size: int = 1,
-        proxy_url: Optional[str] = None,
+        proxy_url: Optional[Union[str, httpx.Proxy]] = None,
         read_timeout: Optional[float] = 5.0,
         write_timeout: Optional[float] = 5.0,
         connect_timeout: Optional[float] = 5.0,
         pool_timeout: Optional[float] = 1.0,
         http_version: HTTPVersion = "1.1",
+        proxy: Optional[Union[str, httpx.Proxy]] = None,
     ):
+        if proxy_url is not None and proxy is not None:
+            raise ValueError("The parameters `proxy_url` and `proxy` are mutually exclusive.")
+
+        if proxy_url is not None:
+            proxy = proxy_url
+            warn(
+                "The parameter `proxy_url` is deprecated since version NEXT.VERSION. Use `proxy` "
+                "instead.",
+                PTBDeprecationWarning,
+                stacklevel=2,
+            )
+
         self._http_version = http_version
         timeout = httpx.Timeout(
             connect=connect_timeout,
@@ -127,7 +149,7 @@ def __init__(
         # for why we need to use `dict()` here.
         self._client_kwargs = dict(  # pylint: disable=use-dict-literal  # noqa: C408
             timeout=timeout,
-            proxies=proxy_url,
+            proxies=proxy,
             limits=limits,
             http1=http1,
             http2=not http1,
diff --git a/tests/request/test_request.py b/tests/request/test_request.py
index 9334f7b769b..984b3ffaa07 100644
--- a/tests/request/test_request.py
+++ b/tests/request/test_request.py
@@ -42,6 +42,7 @@
     TimedOut,
 )
 from telegram.request._httpxrequest import HTTPXRequest
+from telegram.warnings import PTBDeprecationWarning
 from tests.auxil.envvars import TEST_WITH_OPT_DEPS
 from tests.auxil.slots import mro_slots
 
@@ -78,7 +79,7 @@ async def httpx_request():
 class TestNoSocksHTTP2WithoutRequest:
     async def test_init(self, bot):
         with pytest.raises(RuntimeError, match=r"python-telegram-bot\[socks\]"):
-            HTTPXRequest(proxy_url="socks5://foo")
+            HTTPXRequest(proxy="socks5://foo")
         with pytest.raises(RuntimeError, match=r"python-telegram-bot\[http2\]"):
             HTTPXRequest(http_version="2")
 
@@ -117,7 +118,7 @@ def __init__(self, *args, **kwargs):
 
         # Make sure that other exceptions are forwarded
         with pytest.raises(ImportError, match=r"Other Error Message"):
-            HTTPXRequest(proxy_url="socks5://foo")
+            HTTPXRequest(proxy="socks5://foo")
 
     def test_slot_behaviour(self):
         inst = HTTPXRequest()
@@ -380,18 +381,29 @@ class Client:
 
         request = HTTPXRequest(
             connection_pool_size=42,
-            proxy_url="proxy_url",
+            proxy="proxy",
             connect_timeout=43,
             read_timeout=44,
             write_timeout=45,
             pool_timeout=46,
         )
-        assert request._client.proxies == "proxy_url"
+        assert request._client.proxies == "proxy"
         assert request._client.limits == httpx.Limits(
             max_connections=42, max_keepalive_connections=42
         )
         assert request._client.timeout == httpx.Timeout(connect=43, read=44, write=45, pool=46)
 
+    def test_proxy_mutually_exclusive(self):
+        with pytest.raises(ValueError, match="mutually exclusive"):
+            HTTPXRequest(proxy="proxy", proxy_url="proxy_url")
+
+    def test_proxy_url_deprecation_warning(self, recwarn):
+        HTTPXRequest(proxy_url="http://127.0.0.1:3128")
+        assert len(recwarn) == 1
+        assert recwarn[0].category is PTBDeprecationWarning
+        assert "`proxy_url` is deprecated" in str(recwarn[0].message)
+        assert recwarn[0].filename == __file__, "incorrect stacklevel"
+
     async def test_multiple_inits_and_shutdowns(self, monkeypatch):
         self.test_flag = defaultdict(int)
 


Code review comments:
diff_hunk: @@ -91,6 +86,20 @@ class HTTPXRequest(BaseRequest):
 
             .. versionchanged:: 20.5
                 Accept ``"2"`` as a valid value.
+        proxy (:obj:`str` | ``httpx.Proxy``, optional): The URL to a proxy server or an
path: telegram/request/_httpxrequest.py
text: @user1:
Looking again at httpx code, the proxy parameter has a much wider definition:
```python
URLTypes = Union["URL", str]
ProxiesTypes = Union[URLTypes, "Proxy", Dict[URLTypes, Union[None, URLTypes, "Proxy"]]]
```

Since ptb doesn't need a dict, consider at least chaning to 
```suggestion
        proxy (:obj:`str` | ``httpx.Proxy`` | ``httpx.URL``, optional): The URL to a proxy server or an
```

@author:
I can add the URL type, yes. I'll skip the dict types though as discussed in #3844 :)