Repository: abhinavsingh/proxy.py
Pull request #397 (2020-07-10 02:26:54)

Add plugin "FilterByURLRegexPlugin"
As-per issue #378, specifically https://github.com/abhinavsingh/proxy.py/issues/378#issuecomment-655320421.

I understand there are lots of commits in this pull request, so I'd suggest a squash. However, if you'd like me to tidy this up and resubmit, please let me know.

abhinavsingh: > I understand there are lots of commits in this pull request, so I'd suggest a squash. However, if you'd like me to tidy this up and resubmit, please let me know.

All PRs are squashed and merged in the develop branch.  So we are ok with several commits.

I'll recommend you to run `make autopep8` (for auto formatting) and `make` (for lint and tests) locally, otherwise CI will continue to fail.  Rest of the plugin code looks good to me to start with.

Thanks.

mikenye: I'm afraid I might be out of my depth here - I'm not sure how to address the last failing `flake8` checks... Can you help me out, maybe give me a hint? :)
codecov[bot]: # [Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=h1) Report
> Merging [#397](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=desc) into [develop](https://codecov.io/gh/abhinavsingh/proxy.py/commit/73bcf1d0577c52a7dbb97fc2cd4689e5dff7100a&el=desc) will **decrease** coverage by `0.44%`.
> The diff coverage is `45.94%`.

[![Impacted file tree graph](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397/graphs/tree.svg?width=650&height=150&src=pr&token=Zh9J7b4la2)](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop     #397      +/-   ##
===========================================
- Coverage    81.09%   80.65%   -0.45%     
===========================================
  Files           78       79       +1     
  Lines         2914     2951      +37     
===========================================
+ Hits          2363     2380      +17     
- Misses         551      571      +20     
```


| [Impacted Files](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [proxy/plugin/filter\_by\_url\_regex.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397/diff?src=pr&el=tree#diff-cHJveHkvcGx1Z2luL2ZpbHRlcl9ieV91cmxfcmVnZXgucHk=) | `44.44% <44.44%> (ø)` | |
| [proxy/plugin/\_\_init\_\_.py](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397/diff?src=pr&el=tree#diff-cHJveHkvcGx1Z2luL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=footer). Last update [73bcf1d...2254248](https://codecov.io/gh/abhinavsingh/proxy.py/pull/397?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

abhinavsingh: That's sweet.  Reduction in test coverage by <1% ideally shouldn't matter.  Unfortunately, `codecov` doesn't seem to respect threshold in `yaml` configuration.  We'll disable it and merge the PR.
mikenye: Awesome, thanks! I feel like a real developer! ;-)
abhinavsingh: LGTM.  Squash and merging PR now.  Once again, thank you for kick starting this plugin.  Let's make this plugin more modular and stable with time 👍 

Patch:
diff --git a/proxy/plugin/__init__.py b/proxy/plugin/__init__.py
index eca6f39..8450e8f 100644
--- a/proxy/plugin/__init__.py
+++ b/proxy/plugin/__init__.py
@@ -19,6 +19,7 @@ from .web_server_route import WebServerPlugin
 from .reverse_proxy import ReverseProxyPlugin
 from .proxy_pool import ProxyPoolPlugin
 from .filter_by_client_ip import FilterByClientIpPlugin
+from .filter_by_url_regex import FilterByURLRegexPlugin
 from .modify_chunk_response import ModifyChunkResponsePlugin
 
 __all__ = [
@@ -35,4 +36,5 @@ __all__ = [
     'ProxyPoolPlugin',
     'FilterByClientIpPlugin',
     'ModifyChunkResponsePlugin',
+    'FilterByURLRegexPlugin',
 ]
diff --git a/proxy/plugin/filter_by_url_regex.py b/proxy/plugin/filter_by_url_regex.py
new file mode 100644
index 0000000..a4106c9
--- /dev/null
+++ b/proxy/plugin/filter_by_url_regex.py
@@ -0,0 +1,136 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
+    Network monitoring, controls & Application development, testing, debugging.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+
+import logging
+
+from typing import Optional, List, Dict, Any
+
+from ..http.exception import HttpRequestRejected
+from ..http.parser import HttpParser
+from ..http.codes import httpStatusCodes
+from ..http.proxy import HttpProxyBasePlugin
+from ..common.utils import text_
+
+import re
+
+logger = logging.getLogger(__name__)
+
+
+class FilterByURLRegexPlugin(HttpProxyBasePlugin):
+    """
+        Drop traffic by inspecting request URL,
+        checking against a list of regular expressions,
+        then returning a HTTP status code.
+    """
+
+    FILTER_LIST: List[Dict[str, Any]] = [
+        {
+            'regex': b'tpc.googlesyndication.com/simgad/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google image ads',
+        },
+        {
+            'regex': b'tpc.googlesyndication.com/sadbundle/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google animated ad bundles',
+        },
+        {
+            'regex': b'pagead\\d+.googlesyndication.com/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google tracking',
+        },
+        {
+            'regex': b'(www){0,1}.google-analytics.com/r/collect\\?.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google tracking',
+        },
+        {
+            'regex': b'(www){0,1}.facebook.com/tr/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Facebook tracking',
+        },
+        {
+            'regex': b'tpc.googlesyndication.com/daca_images/simgad/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google image ads',
+        },
+        {
+            'regex': b'.*.2mdn.net/videoplayback/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Twitch.tv video ads',
+        },
+        {
+            'regex': b'(www.){0,1}google.com(.*)/pagead/.*',
+            'status_code': httpStatusCodes.NOT_FOUND,
+            'notes': 'Google ads',
+        },
+    ]
+
+    def before_upstream_connection(
+            self, request: HttpParser) -> Optional[HttpParser]:
+        return request
+
+    def handle_client_request(
+            self, request: HttpParser) -> Optional[HttpParser]:
+
+        # determine host
+        request_host = None
+        if request.host:
+            request_host = request.host
+        else:
+            if b'host' in request.headers:
+                request_host = request.header(b'host')
+
+        if not request_host:
+            logger.error("Cannot determine host")
+            return request
+
+        # build URL
+        url = b'%s%s' % (
+            request_host,
+            request.path,
+        )
+
+        # check URL against list
+        rule_number = 1
+        for blocked_entry in self.FILTER_LIST:
+
+            # if regex matches on URL
+            if re.search(text_(blocked_entry['regex']), text_(url)):
+
+                # log that the request has been filtered
+                logger.info("Blocked: %r with status_code '%r' by rule number '%r'" % (
+                    text_(url),
+                    blocked_entry['status_code'],
+                    rule_number,
+                ))
+
+                # close the connection with the status code from the filter
+                # list
+                raise HttpRequestRejected(
+                    status_code=blocked_entry['status_code'],
+                    headers={b'Connection': b'close'},
+                    reason=b'Blocked',
+                )
+
+                # stop looping through filter list
+                break
+
+            # increment rule number
+            rule_number += 1
+
+        return request
+
+    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
+        return chunk
+
+    def on_upstream_connection_close(self) -> None:
+        pass


Test patch:
diff --git a/tests/plugin/test_http_proxy_plugins.py b/tests/plugin/test_http_proxy_plugins.py
index 84ca5a6..51b3cf6 100644
--- a/tests/plugin/test_http_proxy_plugins.py
+++ b/tests/plugin/test_http_proxy_plugins.py
@@ -254,3 +254,30 @@ class TestHttpProxyPluginExamples(unittest.TestCase):
                 httpStatusCodes.OK,
                 reason=b'OK', body=b'Hello from man in the middle')
         )
+
+    @mock.patch('proxy.http.proxy.server.TcpServerConnection')
+    def test_filter_by_url_regex_plugin(
+            self, mock_server_conn: mock.Mock) -> None:
+        request = build_http_request(
+            b'GET', b'http://www.facebook.com/tr/',
+            headers={
+                b'Host': b'www.facebook.com',
+            }
+        )
+        self._conn.recv.return_value = request
+        self.mock_selector.return_value.select.side_effect = [
+            [(selectors.SelectorKey(
+                fileobj=self._conn,
+                fd=self._conn.fileno,
+                events=selectors.EVENT_READ,
+                data=None), selectors.EVENT_READ)], ]
+        self.protocol_handler.run_once()
+
+        self.assertEqual(
+            self.protocol_handler.client.buffer[0].tobytes(),
+            build_http_response(
+                status_code=httpStatusCodes.NOT_FOUND,
+                reason=b'Blocked',
+                headers={b'Connection': b'close'},
+            )
+        )
diff --git a/tests/plugin/utils.py b/tests/plugin/utils.py
index 0934367..3e58ff1 100644
--- a/tests/plugin/utils.py
+++ b/tests/plugin/utils.py
@@ -12,7 +12,7 @@ from typing import Type
 from proxy.http.proxy import HttpProxyBasePlugin
 
 from proxy.plugin import ModifyPostDataPlugin, ProposedRestApiPlugin, RedirectToCustomServerPlugin, \
-    FilterByUpstreamHostPlugin, CacheResponsesPlugin, ManInTheMiddlePlugin
+    FilterByUpstreamHostPlugin, CacheResponsesPlugin, ManInTheMiddlePlugin, FilterByURLRegexPlugin
 
 
 def get_plugin_by_test_name(test_name: str) -> Type[HttpProxyBasePlugin]:
@@ -29,4 +29,6 @@ def get_plugin_by_test_name(test_name: str) -> Type[HttpProxyBasePlugin]:
         plugin = CacheResponsesPlugin
     elif test_name == 'test_man_in_the_middle_plugin':
         plugin = ManInTheMiddlePlugin
+    elif test_name == 'test_filter_by_url_regex_plugin':
+        plugin = FilterByURLRegexPlugin
     return plugin