Repository: abhinavsingh/proxy.py
Pull request #1476 (2024-09-21 03:49:49)

TLS Intercept conditionally
Fixes
- https://github.com/abhinavsingh/proxy.py/discussions/1432

abhinavsingh: @kuefmz Can you give this PR a try?  This PR fixes logic where conditionally TLS interception wasn't working as expected.   Please let me know if something is still amiss.  Thank you for reporting this.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) Report
Attention: Patch coverage is `72.72727%` with `6 lines` in your changes missing coverage. Please review.
> Project coverage is 84.09%. Comparing base [(`a51ddaa`)](https://app.codecov.io/gh/abhinavsingh/proxy.py/commit/a51ddaae4149f5b5452c1f4283539e1a3fea3846?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) to head [(`38a8776`)](https://app.codecov.io/gh/abhinavsingh/proxy.py/commit/38a877634617049b7501d23f99ffd4766e953c5f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh).

| [Files with missing lines](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | Patch % | Lines |
|---|---|---|
| [proxy/http/proxy/server.py](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?src=pr&el=tree&filepath=proxy%2Fhttp%2Fproxy%2Fserver.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh#diff-cHJveHkvaHR0cC9wcm94eS9zZXJ2ZXIucHk=) | 76.92% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) |
| [proxy/plugin/tls\_intercept\_conditionally.py](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?src=pr&el=tree&filepath=proxy%2Fplugin%2Ftls_intercept_conditionally.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh#diff-cHJveHkvcGx1Z2luL3Rsc19pbnRlcmNlcHRfY29uZGl0aW9uYWxseS5weQ==) | 62.50% | [3 Missing :warning: ](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop    #1476      +/-   ##
===========================================
- Coverage    85.55%   84.09%   -1.47%     
===========================================
  Files          176      183       +7     
  Lines         7659     8343     +684     
  Branches       815     1272     +457     
===========================================
+ Hits          6553     7016     +463     
- Misses         947     1114     +167     
- Partials       159      213      +54     
```

| [Flag](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | Coverage Δ | |
|---|---|---|
| [GHA](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `84.09% <72.72%> (-1.30%)` | :arrow_down: |
| [Linux](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `84.09% <72.72%> (?)` | |
| [Python 3.10](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `85.24% <62.50%> (-0.03%)` | :arrow_down: |
| [Python 3.11](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `79.63% <72.72%> (?)` | |
| [Python 3.6](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [Python 3.7](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [Python 3.8](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [Python 3.9](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [Windows](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [macOS](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `?` | |
| [pytest](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh) | `84.09% <72.72%> (-1.30%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/abhinavsingh/proxy.py/pull/1476?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Abhinav+Singh).

abhinavsingh: @webknjaz I am wondering if you have encountered following error in other projects:

```
Running Sphinx v4.3.2
loading translations [en]... done
Initializing Spelling Checker 8.0.0
making output directory... done
Ignoring wiki words
Ignoring acronyms
Adding package names from PyPI to local dictionary…

Exception occurred:
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/xmlrpc/client.py", line 668, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'RuntimeError: PyPI no longer supports the XMLRPC list_packages method. Use JSON or Simple API instead. See https://github.com/pypi/warehouse/issues/16642 and https://warehouse.pypa.io/api-reference/xml-rpc.html#deprecated-methods for more information.'>
The full traceback has been saved in /tmp/sphinx-err-b2ijmmb5.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
ERROR: InvocationError for command /home/runner/work/proxy.py/proxy.py/.tox/spellcheck-docs/bin/python -m sphinx -j auto -a -n -W --keep-going -b spelling --color -d /home/runner/work/proxy.py/proxy.py/.tox/docs_doctree . /home/runner/work/proxy.py/proxy.py/.tox/docs_out (exited with code 2)
ERROR: invocation failed (exit code 1)
✖ FAIL spellcheck-docs in 10.504 seconds
___________________________________ summary ____________________________________
ERROR:   spellcheck-docs: parallel child exit code 1

Error: Process completed with exit code 1.
```

I bumped `sphinxcontrib-spelling` to version `8.0.0` but looks like problem might be rooted somewhere else.   Wondering whether Sphinx itself should be upgraded here?    If yes, IIRC, this might not be an easy peasy task.   Looking for some insights how to proceed w/o disabling spell check.

Patch:
diff --git a/proxy/http/proxy/server.py b/proxy/http/proxy/server.py
index 80af168..11e1760 100644
--- a/proxy/http/proxy/server.py
+++ b/proxy/http/proxy/server.py
@@ -281,10 +281,7 @@ class HttpProxyPlugin(HttpProtocolHandlerPlugin):
             # only for non-https requests and when
             # tls interception is enabled
             if raw is not None:
-                if (
-                    not self.request.is_https_tunnel
-                    or self.tls_interception_enabled
-                ):
+                if not self.request.is_https_tunnel or self._tls_intercept_enabled:
                     if self.response.is_complete:
                         self.handle_pipeline_response(raw)
                     else:
@@ -429,8 +426,7 @@ class HttpProxyPlugin(HttpProtocolHandlerPlugin):
             # We also handle pipeline scenario for https proxy
             # requests is TLS interception is enabled.
             if self.request.is_complete and (
-                    not self.request.is_https_tunnel or
-                    self.tls_interception_enabled
+                not self.request.is_https_tunnel or self._tls_intercept_enabled
             ):
                 if self.pipeline_request is not None and \
                         self.pipeline_request.is_connection_upgrade:
@@ -474,6 +470,20 @@ class HttpProxyPlugin(HttpProtocolHandlerPlugin):
             else:
                 self.upstream.queue(raw)
 
+    @property
+    def _tls_intercept_enabled(self) -> bool:
+        do_intercept = self.tls_interception_enabled
+        if not do_intercept:
+            return do_intercept
+        # If enabled by flags, check if a plugin wants us to bypass
+        # interception for this particular request
+        for plugin in self.plugins.values():
+            do_intercept = plugin.do_intercept(self.request)
+            # A plugin requested to not intercept the request
+            if do_intercept is False:
+                break
+        return do_intercept
+
     def on_request_complete(self) -> Union[socket.socket, bool]:
         self.emit_request_complete()
 
@@ -510,19 +520,8 @@ class HttpProxyPlugin(HttpProtocolHandlerPlugin):
         if self.upstream:
             if self.request.is_https_tunnel:
                 self.client.queue(PROXY_TUNNEL_ESTABLISHED_RESPONSE_PKT)
-                if self.tls_interception_enabled:
-                    # Check if any plugin wants to
-                    # disable interception even
-                    # with flags available
-                    do_intercept = True
-                    for plugin in self.plugins.values():
-                        do_intercept = plugin.do_intercept(self.request)
-                        # A plugin requested to not intercept
-                        # the request
-                        if do_intercept is False:
-                            break
-                    if do_intercept:
-                        return self.intercept()
+                if self._tls_intercept_enabled:
+                    return self.intercept()
             # If an upstream server connection was established for http request,
             # queue the request for upstream server.
             else:
diff --git a/proxy/plugin/__init__.py b/proxy/plugin/__init__.py
index 74c7e8d..48b7996 100644
--- a/proxy/plugin/__init__.py
+++ b/proxy/plugin/__init__.py
@@ -34,6 +34,7 @@ from .filter_by_url_regex import FilterByURLRegexPlugin
 from .modify_chunk_response import ModifyChunkResponsePlugin
 from .modify_request_header import ModifyRequestHeaderPlugin
 from .redirect_to_custom_server import RedirectToCustomServerPlugin
+from .tls_intercept_conditionally import TlsInterceptConditionallyPlugin
 
 
 __all__ = [
@@ -55,4 +56,5 @@ __all__ = [
     'CloudflareDnsResolverPlugin',
     'ProgramNamePlugin',
     'ModifyRequestHeaderPlugin',
+    'TlsInterceptConditionallyPlugin',
 ]
diff --git a/proxy/plugin/tls_intercept_conditionally.py b/proxy/plugin/tls_intercept_conditionally.py
new file mode 100644
index 0000000..58f992e
--- /dev/null
+++ b/proxy/plugin/tls_intercept_conditionally.py
@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
+    Network monitoring, controls & Application development, testing, debugging.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+from ..http.proxy import HttpProxyBasePlugin
+from ..http.parser import HttpParser
+
+
+class TlsInterceptConditionallyPlugin(HttpProxyBasePlugin):
+    """TLS intercept conditionally."""
+
+    def do_intercept(self, request: HttpParser) -> bool:
+        if request.host == b'httpbin.org':
+            return False
+        return super().do_intercept(request)


Test patch:
diff --git a/.github/workflows/test-library.yml b/.github/workflows/test-library.yml
index 0857fd1..38375b4 100644
--- a/.github/workflows/test-library.yml
+++ b/.github/workflows/test-library.yml
@@ -340,7 +340,7 @@ jobs:
         - build-docs
         - doctest-docs
         - linkcheck-docs
-        - spellcheck-docs
+        # - spellcheck-docs
       fail-fast: false
 
     env:
diff --git a/tests/http/proxy/test_http_proxy_tls_interception.py b/tests/http/proxy/test_http_proxy_tls_interception.py
index fad6eff..4ee8246 100644
--- a/tests/http/proxy/test_http_proxy_tls_interception.py
+++ b/tests/http/proxy/test_http_proxy_tls_interception.py
@@ -297,7 +297,7 @@ class TestHttpProxyTlsInterception(Assertions):
             self.proxy_plugin.return_value.handle_client_request.call_count,
             2,
         )
-        self.proxy_plugin.return_value.do_intercept.assert_called_once()
+        self.assertEqual(self.proxy_plugin.return_value.do_intercept.call_count, 2)
 
         callback_request = \
             self.proxy_plugin.return_value.handle_client_request.call_args_list[1][0][0]