internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
Adopt Dependabot
I've only just learned that requires.io has been shut down. A big thank you to @omansion and @atabary for providing such a fantastic service over the years! ❤️

For mitmproxy this probably means we should migrate to Dependabot. This will probably mean a whole lot more PRs, let's see.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -75,10 +75,10 @@
     # It is not considered best practice to use install_requires to pin dependencies to specific versions.
     install_requires=[
         "aioquic_mitmproxy>=0.9.20,<0.10",
-        "asgiref>=3.2.10,<3.6",
+        "asgiref>=3.2.10,<3.7",
         "Brotli>=1.0,<1.1",
         "certifi>=2019.9.11",  # no semver here - this should always be on the last release!
-        "cryptography>=38.0,<38.1",
+        "cryptography>=38.0,<39.1",
         "flask>=1.1.1,<2.3",
         "h11>=0.11,<0.15",
         "h2>=4.1,<5",
@@ -89,7 +89,7 @@
         "msgpack>=1.0.0, <1.1.0",
         "passlib>=1.6.5, <1.8",
         "protobuf>=3.14,<5",
-        "pyOpenSSL>=22.1,<22.2",
+        "pyOpenSSL>=22.1,<23.1",
         "pyparsing>=2.4.2,<3.1",
         "pyperclip>=1.6.0,<1.9",
         "ruamel.yaml>=0.16,<0.18",
@@ -110,14 +110,14 @@
             "hypothesis>=5.8,<7",
             "parver>=0.1,<2.0",
             "pdoc>=4.0.0",
-            "pyinstaller==5.6.2",
+            "pyinstaller==5.7.0",
             "pytest-asyncio>=0.17,<0.21",
             "pytest-cov>=2.7.1,<4.1",
             "pytest-timeout>=1.3.3,<2.2",
-            "pytest-xdist>=2.1.0,<3.1",
+            "pytest-xdist>=2.1.0,<3.2",
             "pytest>=6.1.0,<8",
             "requests>=2.9.1,<3",
-            "tox>=3.5,<4",
+            "tox>=3.5,<5",
             "wheel>=0.36.2,<0.39",
         ],
     },


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
"raw" view is not raw, adds extra backslash
#### Problem Description

I just noticed during https://github.com/mitmproxy/mitmproxy/issues/5469#issuecomment-1191343747

#### Steps to reproduce the behavior:

http.txt

```
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 1

\
```

1. `ncat -l 1337 < http.txt`
2. `mitmproxy`
3. `curl --proxy localhost:8080 http://127.0.0.1:1337`

raw is `\\`:

![image](https://user-images.githubusercontent.com/679144/180199838-07c02668-5dab-4210-b5b6-aea78066e0fd.png)

hex:

![image](https://user-images.githubusercontent.com/679144/180199860-2cdd9455-7846-4921-b67c-6b9c3669b268.png)

#### System Information

```
Mitmproxy: 8.1.1 binary
Python:    3.10.5
OpenSSL:   OpenSSL 3.0.3 3 May 2022
Platform:  Linux-5.15.0-41-generic-x86_64-with-glibc2.35
```



Discussion:
Raw mode currently escapes unprintable characters as well as backslashes so that the representation is unambiguous. I do agree though that the example you are showing is all but optimal, maybe we should just switch to rendering unprintable characters as �.
> maybe we should just switch to rendering unprintable characters as �.

That sounds reasonable to me. Text bodies would then be rendered 100% correctly (which I think is very important) and binary bodies would contain � as one would kind of expect (I don't think a lot of people would inspect binary bodies for anything other than text they contain [we could additionally also add a `strings(1)` view :thinking: ])

Resolving patch:
diff --git a/mitmproxy/contentviews/raw.py b/mitmproxy/contentviews/raw.py
--- a/mitmproxy/contentviews/raw.py
+++ b/mitmproxy/contentviews/raw.py
@@ -1,12 +1,11 @@
 from . import base
-from mitmproxy.utils import strutils
 
 
 class ViewRaw(base.View):
     name = "Raw"
 
     def __call__(self, data, **metadata):
-        return "Raw", base.format_text(strutils.bytes_to_escaped_str(data, True))
+        return "Raw", base.format_text(data)
 
     def render_priority(self, data: bytes, **metadata) -> float:
         return 0.1 * float(bool(data))


Test patch:
diff --git a/test/mitmproxy/contentviews/test_raw.py b/test/mitmproxy/contentviews/test_raw.py
--- a/test/mitmproxy/contentviews/test_raw.py
+++ b/test/mitmproxy/contentviews/test_raw.py
@@ -5,6 +5,16 @@
 def test_view_raw():
     v = full_eval(raw.ViewRaw())
     assert v(b"foo")
+    # unicode
+    assert v("🫠".encode()) == (
+        "Raw",
+        [[("text", "🫠".encode())]],
+    )
+    # invalid utf8
+    assert v(b"\xFF") == (
+        "Raw",
+        [[("text", b"\xFF")]],
+    )
 
 
 def test_render_priority():
Repository: mitmproxy/mitmproxy

Issue:
Wrong message direction indicator in console raw TCP view
#### Problem Description

In the flow view of the console tool, for raw TCP flows the `⇒` and `⇐` direction indicators (from client and to client) always alternate and don't represent the correct direction.

#### Steps to reproduce the behavior:

1. Start a TCP server: `ncat --listen localhost 2000`
2. Start mitmproxy: `mitmproxy --mode=reverse:tcp://localhost:2000@2001 --tcp-hosts='.*'`
3. Connect to the reverse proxy: `ncat localhost 2000`. Enter multiple lines, e.g. `a`, press enter,  `b`, press enter.
4. In mitmproxy, open the flow view for the new flow. There the two messages have different direction indicators even though they were both sent by the client.

#### System Information

```
Mitmproxy: 9.0.1 binary
Python:    3.11.0
OpenSSL:   OpenSSL 3.0.7 1 Nov 2022
Platform:  Linux-6.1.9-arch1-2-x86_64-with-glibc2.37
```

#### Fix

The bug is in the function [`view_message_stream`](https://github.com/mitmproxy/mitmproxy/blob/main/mitmproxy/tools/console/flowview.py#L248) and was introduced in https://github.com/mitmproxy/mitmproxy/pull/3970/commits/fe75f14ea29d0654c3a688ecc4c3669a946ea2a8. The solution could be the same as in the similar function [`view_websocket_messages`](https://github.com/mitmproxy/mitmproxy/blob/main/mitmproxy/tools/console/flowview.py#L197).

I would like to contribute a fix for this issue.


Discussion:
Thanks for the clear description. And yes please, a fix would be very welcome! 😃 

Resolving patch:
diff --git a/mitmproxy/tools/console/flowview.py b/mitmproxy/tools/console/flowview.py
--- a/mitmproxy/tools/console/flowview.py
+++ b/mitmproxy/tools/console/flowview.py
@@ -255,21 +255,17 @@ def view_message_stream(self) -> urwid.Widget:
         viewmode = self.master.commands.call("console.flowview.mode")
 
         widget_lines = []
-
-        from_client = flow.messages[0].from_client
         for m in flow.messages:
             _, lines, _ = contentviews.get_message_content_view(viewmode, m, flow)
 
             for line in lines:
-                if from_client:
+                if m.from_client:
                     line.insert(0, self.FROM_CLIENT_MARKER)
                 else:
                     line.insert(0, self.TO_CLIENT_MARKER)
 
                 widget_lines.append(urwid.Text(line))
 
-            from_client = not from_client
-
         if flow.intercepted:
             markup = widget_lines[-1].get_text()[0]
             widget_lines[-1].set_text(("intercept", markup))


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Broken command argument type parsing
#### Problem Description

It seems like our command argument type parsing does not like `from __future__ import annotations`.

#### Steps to reproduce the behavior:

1. `mitmproxy`
2. `[n] [enter] [r]`


Discussion:

Resolving patch:
diff --git a/mitmproxy/command.py b/mitmproxy/command.py
--- a/mitmproxy/command.py
+++ b/mitmproxy/command.py
@@ -22,7 +22,7 @@
 
 
 def verify_arg_signature(f: Callable, args: Iterable[Any], kwargs: dict) -> None:
-    sig = inspect.signature(f)
+    sig = inspect.signature(f, eval_str=True)
     try:
         sig.bind(*args, **kwargs)
     except TypeError as v:
@@ -71,7 +71,7 @@ def __init__(self, manager: "CommandManager", name: str, func: Callable) -> None
         self.name = name
         self.manager = manager
         self.func = func
-        self.signature = inspect.signature(self.func)
+        self.signature = inspect.signature(self.func, eval_str=True)
 
         if func.__doc__:
             txt = func.__doc__.strip()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
UI: Navigation: Additionally allow Esc-key to navigate back.
#### Problem Description
First time using mitmproxy instead of mitmweb:
I intuitively tried navigating back to the flow list (initial view) after entering the view of a single flow using <kbd>Esc</kbd>.

#### Proposal
Additionally to <kbd>q</kbd> allow to use <kbd>Esc</kbd> to navigate back.



Discussion:
That's a good keybind suggestion. 👍 

Resolving patch:
diff --git a/mitmproxy/tools/console/defaultkeys.py b/mitmproxy/tools/console/defaultkeys.py
--- a/mitmproxy/tools/console/defaultkeys.py
+++ b/mitmproxy/tools/console/defaultkeys.py
@@ -17,6 +17,7 @@ def map(km: Keymap) -> None:
     km.add("E", "console.view.eventlog", ["commonkey", "global"], "View event log")
     km.add("Q", "console.exit", ["global"], "Exit immediately")
     km.add("q", "console.view.pop", ["commonkey", "global"], "Exit the current view")
+    km.add("esc", "console.view.pop", ["commonkey", "global"], "Exit the current view")
     km.add("-", "console.layout.cycle", ["global"], "Cycle to next layout")
     km.add("ctrl right", "console.panes.next", ["global"], "Focus next layout pane")
     km.add("ctrl left", "console.panes.prev", ["global"], "Focus previous layout pane")
diff --git a/mitmproxy/tools/console/keymap.py b/mitmproxy/tools/console/keymap.py
--- a/mitmproxy/tools/console/keymap.py
+++ b/mitmproxy/tools/console/keymap.py
@@ -1,5 +1,6 @@
 import logging
 import os
+from collections import defaultdict
 from collections.abc import Sequence
 from functools import cache
 
@@ -71,9 +72,7 @@ def sortkey(self):
 class Keymap:
     def __init__(self, master):
         self.executor = commandexecutor.CommandExecutor(master)
-        self.keys: dict[str, dict[str, Binding]] = {}
-        for c in Contexts:
-            self.keys[c] = {}
+        self.keys: dict[str, dict[str, Binding]] = defaultdict(dict)
         self.bindings = []
 
     def _check_contexts(self, contexts):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Toggling follow new option fails in single flow view.
#### Problem Description
Toggling follow new flow option via <kbd>F</kbd> does not work while in the view of a single flow.

#### Steps to reproduce the behavior:
1. launch mitmproxy
2. continually generate new requests
3. activate `Follow new` option
4. enter a flow
5. try to deactivate `Follow new` option
6. error: the view still follows the new flows

#### System Information
mitmproxy version 9.0.1



Discussion:
Given that we don't use `F` for something else at the moment, I would be fine with also assigning it to `Follow new` in the individual flow view. PRs welcome!

Resolving patch:
diff --git a/mitmproxy/tools/console/defaultkeys.py b/mitmproxy/tools/console/defaultkeys.py
--- a/mitmproxy/tools/console/defaultkeys.py
+++ b/mitmproxy/tools/console/defaultkeys.py
@@ -82,7 +82,12 @@ def map(km: Keymap) -> None:
         "Export this flow to file",
     )
     km.add("f", "console.command.set view_filter", ["flowlist"], "Set view filter")
-    km.add("F", "set console_focus_follow toggle", ["flowlist"], "Set focus follow")
+    km.add(
+        "F",
+        "set console_focus_follow toggle",
+        ["flowlist", "flowview"],
+        "Set focus follow",
+    )
     km.add(
         "ctrl l",
         "console.command cut.clip ",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Warn new users about the lazy creation of connections (when requests are expected to be served in the script fully and only)
#### Problem Description
The [example script](https://docs.mitmproxy.org/stable/addons-examples/#http-reply-from-proxy) for not sending any data to the server does not prevent mitmproxy from **establishing a connection** to the server.
For which reason is said connection established when no data has to be sent to this host right away and possibly never in the future?
I trusted mitmproxy to **not send _any_ data, as stated**, but I had to discover (the hard way) that **that's not the case**.
I used mitmproxy in an environment where it required to stay silent, but it wasn't compliant.

Could you please consider warning new users about this behavior?
<strike>Is there an easy way to prevent establishing connections?
Is it planned to do so on default in this case?</strike>
*EDIT*:  Trying to prevent connections by rerouting the connection to a closed port killed the flow for the client. Routing to a different host with invalid certificate worked though, warning me in the event log and suggesting setting connection strategy to lazy and it worked.

#### Steps to reproduce the behavior:
1. Load the example script
2. Have the client request examle.com
3. View the event log

#### System Information
Mitmproxy: 9.0.1
Python:    3.10.6
OpenSSL:   OpenSSL 3.0.7 1 Nov 2022
Platform:  Linux-5.15.0-71-generic-x86_64-with-glibc2.35




Discussion:
You can use `--set connection_strategy=lazy` to prevent this behavior, see https://docs.mitmproxy.org/stable/concepts-options/#connection_strategy for more details

> Could you please consider warning new users about this behavior?

I think we could add a comment to the example script mentioning this? Not sure where else you would expect this warning.

Resolving patch:
diff --git a/examples/addons/http-reply-from-proxy.py b/examples/addons/http-reply-from-proxy.py
--- a/examples/addons/http-reply-from-proxy.py
+++ b/examples/addons/http-reply-from-proxy.py
@@ -1,4 +1,4 @@
-"""Send a reply from the proxy without sending any data to the remote server."""
+"""Send a reply from the proxy without sending the request to the remote server."""
 from mitmproxy import http
 
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
`view.flows.add` command does not exist but the examples reference it
#### Problem Description

The `view.flows.add` command does not exist but the example `duplicate-modify-replay.py` shows this command being used.

`replay.client` seems to perform both the "add to view" and "replay" function.


Discussion:

Resolving patch:
diff --git a/examples/addons/duplicate-modify-replay.py b/examples/addons/duplicate-modify-replay.py
--- a/examples/addons/duplicate-modify-replay.py
+++ b/examples/addons/duplicate-modify-replay.py
@@ -10,6 +10,6 @@ def request(flow):
     # Only interactive tools have a view. If we have one, add a duplicate entry
     # for our flow.
     if "view" in ctx.master.addons:
-        ctx.master.commands.call("view.flows.add", [flow])
+        ctx.master.commands.call("view.flows.duplicate", [flow])
     flow.request.path = "/changed"
     ctx.master.commands.call("replay.client", [flow])


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Both Edit and Export share the same key binding
#### Problem Description

When in flow view, both `Edit` and `Export` commands are bound to the `e` key by default.

<img width="493" alt="imagen" src="https://github.com/mitmproxy/mitmproxy/assets/231822/362a928e-1634-4085-928a-b391b576fa37">

`Export` could use a different binding by default, like `x`.

#### System Information

Mitmproxy: 9.0.1
Python:    3.11.3
OpenSSL:   OpenSSL 1.1.1u  30 May 2023
Platform:  macOS-12.6.5-arm64-arm-64bit




Discussion:
Thanks for catching this! Definitely something we want to fix. Maybe we move export to `x` in all contexts? Does anyone have any other ideas/proposals?

Resolving patch:
diff --git a/mitmproxy/tools/console/consoleaddons.py b/mitmproxy/tools/console/consoleaddons.py
--- a/mitmproxy/tools/console/consoleaddons.py
+++ b/mitmproxy/tools/console/consoleaddons.py
@@ -375,7 +375,9 @@ def edit_focus_options(self) -> Sequence[str]:
         flow = self.master.view.focus.flow
         focus_options = []
 
-        if isinstance(flow, tcp.TCPFlow):
+        if flow is None:
+            raise exceptions.CommandError("No flow selected.")
+        elif isinstance(flow, tcp.TCPFlow):
             focus_options = ["tcp-message"]
         elif isinstance(flow, udp.UDPFlow):
             focus_options = ["udp-message"]
diff --git a/mitmproxy/tools/console/defaultkeys.py b/mitmproxy/tools/console/defaultkeys.py
--- a/mitmproxy/tools/console/defaultkeys.py
+++ b/mitmproxy/tools/console/defaultkeys.py
@@ -74,7 +74,7 @@ def map(km: Keymap) -> None:
         "D", "view.flows.duplicate @focus", ["flowlist", "flowview"], "Duplicate flow"
     )
     km.add(
-        "e",
+        "x",
         """
         console.choose.cmd Format export.formats
         console.command export.file {choice} @focus
@@ -156,7 +156,7 @@ def map(km: Keymap) -> None:
         console.choose.cmd Part console.edit.focus.options
         console.edit.focus {choice}
         """,
-        ["flowview"],
+        ["flowlist", "flowview"],
         "Edit a flow component",
     )
     km.add(
diff --git a/mitmproxy/tools/console/quickhelp.py b/mitmproxy/tools/console/quickhelp.py
--- a/mitmproxy/tools/console/quickhelp.py
+++ b/mitmproxy/tools/console/quickhelp.py
@@ -73,6 +73,7 @@ def make(
                     top_items["Unmark"] = "Toggle mark on this flow"
                 else:
                     top_items["Mark"] = "Toggle mark on this flow"
+                top_items["Edit"] = "Edit a flow component"
             if focused_flow.intercepted:
                 top_items["Resume"] = "Resume this intercepted flow"
             if focused_flow.modified():


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Outfile -w cannot be loaded
#### Problem Description
When the dump file is getting bigger, about 100mb it's not loaded anymore.

#### Steps to reproduce the behavior:
Make a big outfile and try to open it with an new instance of mitmweb.

#### System Information
Mitmweb Windows 10 6.0.2
mitmweb Not loading my saved flow
So I recorded some actions with mitmweb and saved the flow.

Then I closed mitmweb, and reopened it.  Then I went to open the saved flow file (which is 100 megabytes).  But when I open it, the requests and responses do not appear?



Discussion:
This report is not actionable. Which browser are you using? Are there any error messages, either in your browser's developer tools or in mitmweb's console window?
In order to investigate this, we need more details. How can this be reproduced? Do you have a flow sample you can share?

Resolving patch:
diff --git a/mitmproxy/tools/web/master.py b/mitmproxy/tools/web/master.py
--- a/mitmproxy/tools/web/master.py
+++ b/mitmproxy/tools/web/master.py
@@ -97,7 +97,9 @@ async def running(self):
         tornado.ioloop.IOLoop.current()
 
         # Add our web app.
-        http_server = tornado.httpserver.HTTPServer(self.app)
+        http_server = tornado.httpserver.HTTPServer(
+            self.app, max_buffer_size=2**32
+        )  # 4GB
         try:
             http_server.listen(self.options.web_port, self.options.web_host)
         except OSError as e:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Getting errors after terminating from mitmproxy console
#### Problem Description
Getting error log after terminating from the mitmproxy console when executing the io-write-file.py

#### Steps to reproduce the behavior:
1. Run the cmd : mitmproxy io-write-file.py "/path/to/dump/file"
2. After running a while then quit from mitmproxy console.
3. In terminal able to see errors. 

#### System Information
Mitmproxy: 10.1.5
Python:    3.11.5
OpenSSL:   OpenSSL 3.1.4 24 Oct 2023
Platform:  Windows-10-10.0.22631-SP0

![image](https://github.com/mitmproxy/mitmproxy/assets/48276907/f5b4787e-0321-46ca-8ac5-9bf3b6e413ac)



Discussion:

Resolving patch:
diff --git a/mitmproxy/proxy/mode_servers.py b/mitmproxy/proxy/mode_servers.py
--- a/mitmproxy/proxy/mode_servers.py
+++ b/mitmproxy/proxy/mode_servers.py
@@ -234,7 +234,10 @@ def listen_addrs(self) -> tuple[Address, ...]:
             if isinstance(s, mitmproxy_rs.UdpServer):
                 addrs.append(s.getsockname())
             else:
-                addrs.extend(sock.getsockname() for sock in s.sockets)
+                try:
+                    addrs.extend(sock.getsockname() for sock in s.sockets)
+                except OSError:  # pragma: no cover
+                    pass  # this can fail during shutdown, see https://github.com/mitmproxy/mitmproxy/issues/6529
         return tuple(addrs)
 
     async def _start(self) -> None:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
AttributeError: 'ClientPlayback' object has no attribute 'options'
#### Problem Description
Client playback does not seem to work anymore, since 10.2.0. 
On 10.1.6 it still works.

I am getting the following error:
>Traceback (most recent call last):
  File "mitmproxy/addons/clientplayback.py", line 177, in playback
AttributeError: 'ClientPlayback' object has no attribute 'options'

#### Steps to reproduce the behavior:
1. Save a recorded flow
2. Replay it using `mitmdump -C stored.flow` 
3. See error

#### System Information
Mitmproxy: 10.2.1 binary
Python:    3.12.1
OpenSSL:   OpenSSL 3.1.4 24 Oct 2023
Platform:  macOS-14.2.1-x86_64-i386-64bit



Discussion:

Resolving patch:
diff --git a/mitmproxy/addons/clientplayback.py b/mitmproxy/addons/clientplayback.py
--- a/mitmproxy/addons/clientplayback.py
+++ b/mitmproxy/addons/clientplayback.py
@@ -156,10 +156,10 @@ def __init__(self):
         self.replay_tasks = set()
 
     def running(self):
+        self.options = ctx.options
         self.playback_task = asyncio_utils.create_task(
             self.playback(), name="client playback"
         )
-        self.options = ctx.options
 
     async def done(self):
         if self.playback_task:


Test patch: