Repository: mitmproxy/mitmproxy

Issue:
v4 --replacements vs v5 --modify-headers
I'm trying to replace the `User-Agent` request header if it contains a certain string.

This works with "mitmproxy-4.0.4-linux":

```
./mitmproxy --replacements ":~hq User-Agent:Mozilla(.+):CUSTOMAGENT"
```

With "mitmproxy-5.2-linux", this at least replaces the `User-Agent`, but is missing my "certain string condition":

```
./mitmproxy --modify-headers "|~hq .+|User-Agent|CUSTOMAGENT"
```

How do I add my `Mozilla` condition in v5?

None of these work:

```
./mitmproxy --modify-headers "|~hq ^(.*?)Mozilla(.*?)$|User-Agent|CUSTOMAGENT"

./mitmproxy --modify-headers "/~hq .*?Mozilla.*?/User-Agent/CUSTOMAGENT"

./mitmproxy --modify-headers "|~hq Mozilla|User-Agent|CUSTOMAGENT"

./mitmproxy --modify-headers "|~hq User-Agent: Mozilla|User-Agent|CUSTOMAGENT"

./mitmproxy --modify-headers "|~hq \"^(.*?)Mozilla(.*?)$\"|User-Agent|CUSTOMAGENT"
```

I've been trying for hours, and I feel like I've tried every variation under the sun. There's a very small chance it's a bug, but most likely I'm just doing it wrong. If it matters, this system is Ubuntu 16.04.





Discussion:
I've been digging into this and this is definitely a bug in the implementation. Will follow up with details.
https://github.com/mitmproxy/mitmproxy/blob/a203b69ba97eab47302edbb9b378eff353550132/mitmproxy/addons/modifyheaders.py#L83-L99

Given your example line 85 to 87 remove the `user-agent` header. Then line 91 tries to match the flow against the filter. But the header is gone, so it cannot match.

I'm not sure why it's implemented this way, but I think both loops (which on the first two level share the exact same code) should be combined into one. Essentially moving the `pop` from the first loop into the else block here

https://github.com/mitmproxy/mitmproxy/blob/a203b69ba97eab47302edbb9b378eff353550132/mitmproxy/addons/modifyheaders.py#L98-L99

I'm working on a fix.

Resolving patch:
diff --git a/mitmproxy/addons/modifyheaders.py b/mitmproxy/addons/modifyheaders.py
--- a/mitmproxy/addons/modifyheaders.py
+++ b/mitmproxy/addons/modifyheaders.py
@@ -83,14 +83,21 @@ def response(self, flow):
         self.run(flow, flow.response.headers)
 
     def run(self, flow: http.HTTPFlow, hdrs: Headers) -> None:
-        # unset all specified headers
+        matches = []
+
+        # first check all the filters against the original, unmodified flow
         for spec in self.replacements:
-            if spec.matches(flow):
+            matches.append(spec.matches(flow))
+
+        # unset all specified headers
+        for i, spec in enumerate(self.replacements):
+            if matches[i]:
                 hdrs.pop(spec.subject, None)
 
         # set all specified headers if the replacement string is not empty
-        for spec in self.replacements:
-            if spec.matches(flow):
+
+        for i, spec in enumerate(self.replacements):
+            if matches[i]:
                 try:
                     replacement = spec.read_replacement()
                 except OSError as e:


Test patch:
diff --git a/test/mitmproxy/addons/test_modifyheaders.py b/test/mitmproxy/addons/test_modifyheaders.py
--- a/test/mitmproxy/addons/test_modifyheaders.py
+++ b/test/mitmproxy/addons/test_modifyheaders.py
@@ -113,6 +113,19 @@ def test_modify_headers(self):
             mh.response(f)
             assert "one" not in f.response.headers
 
+            # test modifying a header that is also part of the filter expression
+            # https://github.com/mitmproxy/mitmproxy/issues/4245
+            tctx.configure(
+                mh,
+                modify_headers=[
+                    "/~hq ^user-agent:.+Mozilla.+$/user-agent/Definitely not Mozilla ;)"
+                ]
+            )
+            f = tflow.tflow()
+            f.request.headers["user-agent"] = "Hello, it's me, Mozilla"
+            mh.request(f)
+            assert "Definitely not Mozilla ;)" == f.request.headers["user-agent"]
+
     @pytest.mark.parametrize("take", [True, False])
     def test_taken(self, take):
         mh = ModifyHeaders()