internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
Horizontal view: Pressing up/down modifies non-focused pane.
As the title says - we just need to patch that out of urwid.Pile, and take a look at the situation with urwid.Columns.
Horizontal view: Pressing up/down modifies non-focused pane.
As the title says - we just need to patch that out of urwid.Pile, and take a look at the situation with urwid.Columns.


Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/console/window.py b/mitmproxy/tools/console/window.py
--- a/mitmproxy/tools/console/window.py
+++ b/mitmproxy/tools/console/window.py
@@ -15,16 +15,30 @@
 from mitmproxy.tools.console import eventlog
 
 
-class Header(urwid.Frame):
+class StackWidget(urwid.Frame):
     def __init__(self, widget, title, focus):
-        super().__init__(
-            widget,
+        if title:
             header = urwid.AttrWrap(
                 urwid.Text(title),
                 "heading" if focus else "heading_inactive"
             )
+        else:
+            header = None
+        super().__init__(
+            widget,
+            header=header
         )
 
+    def keypress(self, size, key):
+        # Make sure that we don't propagate cursor events outside of the widget.
+        # Otherwise, in a horizontal layout, urwid's Pile would change the focused widget
+        # if we cannot scroll any further.
+        ret = super().keypress(size, key)
+        command = self._command_map[ret]  # awkward as they don't implement a full dict api
+        if command and command.startswith("cursor"):
+            return None
+        return ret
+
 
 class WindowStack:
     def __init__(self, master, base):
@@ -142,12 +156,16 @@ def refresh(self):
             self.pane = 0
 
         def wrapped(idx):
-            window = self.stacks[idx].top_window()
             widget = self.stacks[idx].top_widget()
-            if self.master.options.console_layout_headers and window.title:
-                return Header(widget, window.title, self.pane == idx)
+            if self.master.options.console_layout_headers:
+                title = self.stacks[idx].top_window().title
             else:
-                return widget
+                title = None
+            return StackWidget(
+                widget,
+                title,
+                self.pane == idx
+            )
 
         w = None
         if c == "single":


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Add command to edit flow body.
`console.edit.focus.options` currently provides no option to edit the flow body. I then tried the bodyview feature, but that is read-only. In general, there seems to be no option to edit a flow body?


Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/console/consoleaddons.py b/mitmproxy/tools/console/consoleaddons.py
--- a/mitmproxy/tools/console/consoleaddons.py
+++ b/mitmproxy/tools/console/consoleaddons.py
@@ -348,6 +348,8 @@ def edit_focus_options(self) -> typing.Sequence[str]:
             "reason",
             "request-headers",
             "response-headers",
+            "request-body",
+            "response-body",
             "status_code",
             "set-cookies",
             "url",
@@ -359,6 +361,11 @@ def edit_focus(self, part: str) -> None:
         """
             Edit a component of the currently focused flow.
         """
+        flow = self.master.view.focus.flow
+        # This shouldn't be necessary once this command is "console.edit @focus",
+        # but for now it is.
+        if not flow:
+            raise exceptions.CommandError("No flow selected.")
         if part == "cookies":
             self.master.switch_view("edit_focus_cookies")
         elif part == "form":
@@ -371,6 +378,21 @@ def edit_focus(self, part: str) -> None:
             self.master.switch_view("edit_focus_request_headers")
         elif part == "response-headers":
             self.master.switch_view("edit_focus_response_headers")
+        elif part in ("request-body", "response-body"):
+            if part == "request-body":
+                message = flow.request
+            else:
+                message = flow.response
+            if not message:
+                raise exceptions.CommandError("Flow has no {}.".format(part.split("-")[0]))
+            c = self.master.spawn_editor(message.get_content(strict=False) or b"")
+            # Fix an issue caused by some editors when editing a
+            # request/response body. Many editors make it hard to save a
+            # file without a terminating newline on the last line. When
+            # editing message bodies, this can cause problems. For now, I
+            # just strip the newlines off the end of the body when we return
+            # from an editor.
+            message.content = c.rstrip(b"\n")
         elif part == "set-cookies":
             self.master.switch_view("edit_focus_setcookies")
         elif part in ["url", "method", "status_code", "reason"]:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Transitive import of mitmproxy.version causes warning
Since #1837, we import `.script`, will imports `.flow`, which imports `.version`.
This causes the following warning in pytest:

```
test/mitmproxy/test_version.py::test_version
  /Users/kriechi/.pyenv/versions/3.5.3/lib/python3.5/runpy.py:125: 
RuntimeWarning: 'mitmproxy.version' found in sys.modules after import of package 
'mitmproxy', but prior to execution of 'mitmproxy.version'; this may result in 
unpredictable behaviour
    warn(RuntimeWarning(msg))

-- Docs: http://doc.pytest.org/en/latest/warnings.html
```

[Note](http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-double-import-trap)
> This next trap exists in all current versions of Python, including 3.3, and can be summed up in the following general guideline: “Never add a package directory, or any directory inside a package, directly to the Python path”.

> The reason this is problematic is that every module in that directory is now potentially accessible under two different names: as a top level module (since the directory is on sys.path) and as a submodule of the package (if the higher level directory containing the package itself is also on sys.path).

Maybe using the approach described [here](https://stackoverflow.com/questions/27947639/how-to-properly-create-a-pyinstaller-hook-or-maybe-hidden-import) works better?


Discussion:
To be honest I do not quite understand what the actual issue is? Is it the import?
Not sure myself, but I think the fact that `mitmproxy.version` is available (imported) BEFORE actually being imported is not "good".
Maybe this is related to https://github.com/mitmproxy/mitmproxy/blob/aa8969b240310f265640c92e14a4b7d6363e464b/setup.py#L15? Someone needs to remove this and see if CI still complains. If not, we should replace it with something like this: https://github.com/ffalcinelli/pydivert/blob/724bfdc0554ed6fe966eedda3b74197519526cd4/setup.py#L26-L28
Unfortunately the `RuntimeWarning` is still there.
Ah, now it's caused by the test_version execution test. Much easier to fix. :)

Resolving patch:
diff --git a/mitmproxy/version.py b/mitmproxy/version.py
--- a/mitmproxy/version.py
+++ b/mitmproxy/version.py
@@ -60,5 +60,5 @@ def get_version(dev: bool = False, build: bool = False, refresh: bool = False) -
     return mitmproxy_version
 
 
-if __name__ == "__main__":
+if __name__ == "__main__":  # pragma: no cover
     print(VERSION)


Test patch:
diff --git a/test/mitmproxy/test_version.py b/test/mitmproxy/test_version.py
--- a/test/mitmproxy/test_version.py
+++ b/test/mitmproxy/test_version.py
@@ -1,3 +1,4 @@
+import pathlib
 import runpy
 import subprocess
 from unittest import mock
@@ -6,7 +7,9 @@
 
 
 def test_version(capsys):
-    runpy.run_module('mitmproxy.version', run_name='__main__')
+    here = pathlib.Path(__file__).absolute().parent
+    version_file = here / ".." / ".." / "mitmproxy" / "version.py"
+    runpy.run_path(str(version_file), run_name='__main__')
     stdout, stderr = capsys.readouterr()
     assert len(stdout) > 0
     assert stdout.strip() == version.VERSION
Repository: mitmproxy/mitmproxy

Issue:
console: shift+tab is broken in WSL for some Windows terminals
See https://github.com/Microsoft/WSL/issues/1770. This seems to affect some but not all Windows terminals. 

We use shift+tab by default to switch to the next pane in the console app. We can:

- Say that this is not our problem, and wait for upstream to fix it.
- Find a different binding for next pane - which would be a shame, because shit+tab is very natural.

@mhils what say you?


Discussion:
This unfortunately affects the default terminal, which I guess the vast majority uses. 

I think the most pragmatic approach would be to bind to something else on Windows only as a temporary workaround, and unify to shift-tab once Microsoft has fixed it. How do you feel about this?

If you dislike this approach, #2676 would provide at least a somewhat convenient way to switch between panes.

Resolving patch:
diff --git a/mitmproxy/tools/console/defaultkeys.py b/mitmproxy/tools/console/defaultkeys.py
--- a/mitmproxy/tools/console/defaultkeys.py
+++ b/mitmproxy/tools/console/defaultkeys.py
@@ -11,6 +11,7 @@ def map(km):
     km.add("q", "console.view.pop", ["global"], "Exit the current view")
     km.add("-", "console.layout.cycle", ["global"], "Cycle to next layout")
     km.add("shift tab", "console.panes.next", ["global"], "Focus next layout pane")
+    km.add("ctrl right", "console.panes.next", ["global"], "Focus next layout pane")
     km.add("P", "console.view.flow @focus", ["global"], "View flow details")
 
     km.add("g", "console.nav.start", ["global"], "Go to start")


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Traceback appears in Status Bar, when trying to run Mitmproxy with --proxyauth ldap (ldaps) argument
##### Steps to reproduce the problem:

Run Mitmproxy with argument: `mitmproxy --proxyauth ldap` or `mitmproxy --proxyauth ldaps`

I am seeing:
![screenshot from 2018-01-06 15-41-10](https://user-images.githubusercontent.com/20267977/34640265-0d47714c-f2f8-11e7-91c6-9bfe3a0f9eb6.png)

##### Any other comments? What have you tried so far?
This issue is occured, because mitmproxy doesn't control the syntax of ldap authentication argument. 
https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/addons/proxyauth.py#L147-L152

##### System information

Mitmproxy: 3.0.0.dev1101 (commit d9d4d15) binary
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-104-generic-x86_64-with-debian-stretch-sid
  


Discussion:
Looks like the [check](https://github.com/mitmproxy/mitmproxy/blob/012b938254e1bf14d47447d5f3661d3299d8f582/mitmproxy/addons/proxyauth.py#L153-L156) needs to be moved right after the `split`.
Not sure why 5 parts are expected, maybe this should be changed to 4.

Resolving patch:
diff --git a/mitmproxy/addons/proxyauth.py b/mitmproxy/addons/proxyauth.py
--- a/mitmproxy/addons/proxyauth.py
+++ b/mitmproxy/addons/proxyauth.py
@@ -146,14 +146,14 @@ def configure(self, updated):
                         )
                 elif ctx.options.proxyauth.startswith("ldap"):
                     parts = ctx.options.proxyauth.split(':')
-                    security = parts[0]
-                    ldap_server = parts[1]
-                    dn_baseauth = parts[2]
-                    password_baseauth = parts[3]
                     if len(parts) != 5:
                         raise exceptions.OptionsError(
                             "Invalid ldap specification"
                         )
+                    security = parts[0]
+                    ldap_server = parts[1]
+                    dn_baseauth = parts[2]
+                    password_baseauth = parts[3]
                     if security == "ldaps":
                         server = ldap3.Server(ldap_server, use_ssl=True)
                     elif security == "ldap":


Test patch:
diff --git a/test/mitmproxy/addons/test_proxyauth.py b/test/mitmproxy/addons/test_proxyauth.py
--- a/test/mitmproxy/addons/test_proxyauth.py
+++ b/test/mitmproxy/addons/test_proxyauth.py
@@ -190,7 +190,7 @@ def test_configure(self):
             with pytest.raises(exceptions.OptionsError):
                 ctx.configure(up, proxyauth="ldap:test:test:test")
 
-            with pytest.raises(IndexError):
+            with pytest.raises(exceptions.OptionsError):
                 ctx.configure(up, proxyauth="ldap:fake_serveruid=?dc=example,dc=com:person")
 
             with pytest.raises(exceptions.OptionsError):
Repository: mitmproxy/mitmproxy

Issue:
Traceback appears, when trying to set mitmproxy's address as upstream server for reverse/upstream mode
##### Steps to reproduce the problem:

1. Run mitmproxy in **reverse** or **upstream** mode, using its own address as  upstream server address:
`mitmproxy --mode reverse:http://127.0.0.1:8080` or
`mitmproxy --mode upstream:http://127.0.0.1:8080`
2. Make a request using pathoc `pathoc 127.0.0.1:8080 "get:/"` or a browser.

I am seeing:
![screenshot from 2018-01-06 23-29-35](https://user-images.githubusercontent.com/20267977/34644136-7cac6222-f339-11e7-8995-2b30c6ab5c63.png)

##### Any other comments? What have you tried so far?
https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/proxy/protocol/base.py#L115
should be handled.

##### System information

Mitmproxy: 3.0.0.dev1101 (commit d9d4d15) binary
Python: 3.5.2
OpenSSL: OpenSSL 1.1.0g 2 Nov 2017
Platform: Linux-4.4.0-104-generic-x86_64-with-debian-stretch-sid



Discussion:
I believe we can fix this by moving [this line](https://github.com/mitmproxy/mitmproxy/blob/821d76df02f70ccca5623cbc65d02b80e3998704/mitmproxy/proxy/server.py#L117) into the try: block right afterwards. So that should be fairly simple, but we need a test! :-)

Resolving patch:
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -114,9 +114,9 @@ def _create_root_layer(self):
     def handle(self):
         self.log("clientconnect", "info")
 
-        root_layer = self._create_root_layer()
-
+        root_layer = None
         try:
+            root_layer = self._create_root_layer()
             root_layer = self.channel.ask("clientconnect", root_layer)
             root_layer()
         except exceptions.Kill:
@@ -151,7 +151,8 @@ def handle(self):
             print("Please lodge a bug report at: https://github.com/mitmproxy/mitmproxy", file=sys.stderr)
 
         self.log("clientdisconnect", "info")
-        self.channel.tell("clientdisconnect", root_layer)
+        if root_layer is not None:
+            self.channel.tell("clientdisconnect", root_layer)
         self.client_conn.finish()
 
     def log(self, msg, level):


Test patch:
diff --git a/test/mitmproxy/proxy/test_server.py b/test/mitmproxy/proxy/test_server.py
--- a/test/mitmproxy/proxy/test_server.py
+++ b/test/mitmproxy/proxy/test_server.py
@@ -511,6 +511,14 @@ def test_overridden_host_header(self):
         req = self.master.state.flows[0].request
         assert req.host_header == "127.0.0.1"
 
+    def test_selfconnection(self):
+        self.options.mode = "reverse:http://127.0.0.1:0"
+
+        p = self.pathoc()
+        with p.connect():
+            p.request("get:/")
+        assert self.master.has_log("The proxy shall not connect to itself.")
+
 
 class TestReverseSSL(tservers.ReverseProxyTest, CommonMixin, TcpMixin):
     reverse = True
Repository: mitmproxy/mitmproxy

Issue:
Traceback appears in Status Bar, when trying to replay live flow
##### Steps to reproduce the problem:

1. Run **pathod** : `pathod -a "/=200:p0,10"`
2. Run mitmproxy.
3. Send _get request_ to pathod through mitmproxy using **pathoc**: 
`pathoc -c localhost:9999 localhost:8080 'get:/'`
4. Try to replay the corresponding live flow in mitmproxy by pressing `r`.

I am seeing:
![screenshot from 2018-01-06 13-40-01](https://user-images.githubusercontent.com/20267977/34639350-30f9aed6-f2e7-11e7-8920-44979218178e.png)


##### Any other comments? What have you tried so far?
This issue is relevant for the situations, when server didn't have time to send a response yet, but a user tries to replay the corresponding flow.
I also faced this issue, when trying to replay `mitm.it` flow from onboardingapp.


##### System information

Mitmproxy: 3.0.0.dev1101 (commit d9d4d15) binary
Python: 3.5.2
OpenSSL: OpenSSL 1.1.0g 2 Nov 2017
Platform: Linux-4.4.0-104-generic-x86_64-with-debian-stretch-sid

  


Discussion:
We need to check in the `replay.client` command that the flow(s) we're about to replay are not live yet.

Resolving patch:
diff --git a/mitmproxy/addons/clientplayback.py b/mitmproxy/addons/clientplayback.py
--- a/mitmproxy/addons/clientplayback.py
+++ b/mitmproxy/addons/clientplayback.py
@@ -35,6 +35,9 @@ def start_replay(self, flows: typing.Sequence[flow.Flow]) -> None:
         """
             Replay requests from flows.
         """
+        for f in flows:
+            if f.live:
+                raise exceptions.CommandError("Can't replay live flow.")
         self.flows = list(flows)
         ctx.log.alert("Replaying %s flows." % len(self.flows))
         ctx.master.addons.trigger("update", [])


Test patch:
diff --git a/test/mitmproxy/addons/test_clientplayback.py b/test/mitmproxy/addons/test_clientplayback.py
--- a/test/mitmproxy/addons/test_clientplayback.py
+++ b/test/mitmproxy/addons/test_clientplayback.py
@@ -52,6 +52,10 @@ def test_playback(self):
             cp.stop_replay()
             assert not cp.flows
 
+            df = tflow.DummyFlow(tflow.tclient_conn(), tflow.tserver_conn(), True)
+            with pytest.raises(exceptions.CommandError, match="Can't replay live flow."):
+                cp.start_replay([df])
+
     def test_load_file(self, tmpdir):
         cp = clientplayback.ClientPlayback()
         with taddons.context():
Repository: mitmproxy/mitmproxy

Issue:
Traceback when viewing certain gifs 
Open the attached flow dump in mitmproxy console. When viewing the content, observe the traceback that flashes to screen. A number of issues here:

- [ ] Kaitai should be able to parse this gif. It's not technically correct, but I've seen this often, especially with single-pixel tracking gifs. 
- *Even if Kaitai fails*, we shouldn't crash - the user should see an "invalid image" message (or something more informative). (#2698)
- *Even if we crash*, the traceback shouldn't be flashed to screen and disappear. This is an issue with the way we handle super-long error events in console. (#2699)

[bar.log](https://github.com/mitmproxy/mitmproxy/files/1545955/bar.log)



Discussion:
@ujjwal96, do you want to take a look? 😃 
@mhils sure, will have a look at it soon :smiley: 
I see quite a few variations of this with other decoders, especially in the (very frequent) case where the content type doesn't match the content. 
FWIW, the problem with the image in the initial post is that there's an additional null-byte at the end of the file that shouldn't be there.
This issue is now just about fragile GIF parsing, the other bullet points have their own issues or PRs now. I think it's not strictly required to be on the 3.0 milestone now, so I'll remove that here. Nonetheless, we should fix this.
Ref: https://github.com/kaitai-io/kaitai_struct_formats/pull/73
@mhils The above PR handles the issue caused due to the attached GIF.

Resolving patch:
diff --git a/mitmproxy/contrib/kaitaistruct/gif.py b/mitmproxy/contrib/kaitaistruct/gif.py
--- a/mitmproxy/contrib/kaitaistruct/gif.py
+++ b/mitmproxy/contrib/kaitaistruct/gif.py
@@ -35,9 +35,11 @@ def __init__(self, _io, _parent=None, _root=None):
             self.global_color_table = self._root.ColorTable(io, self, self._root)
 
         self.blocks = []
-        while not self._io.is_eof():
-            self.blocks.append(self._root.Block(self._io, self, self._root))
-
+        while True:
+            _ = self._root.Block(self._io, self, self._root)
+            self.blocks.append(_)
+            if  ((self._io.is_eof()) or (_.block_type == self._root.BlockType.end_of_file)) :
+                break
 
     class ImageData(KaitaiStruct):
         def __init__(self, _io, _parent=None, _root=None):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Verbose mode not working in v3.0.0 RC2
##### Steps to reproduce the problem:

1. Start mitmdump with -v or --verbose flag
2. No DEBUG level logs prints on standard output

##### Any other comments? What have you tried so far?

In old stable version (2.0.2) the same steps produce desired output.

##### System information

Mitmproxy: 3.0.0.dev1136 (commit 15f525e)
Python:    3.6.3
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-3.16.0-5-amd64-x86_64-with-debian-8.9


Discussion:
Good catch, thanks! We apparently need to handle this here:

https://github.com/mitmproxy/mitmproxy/blob/da4c1ee625a3451b58c69454ecd5f4faf3cb67d4/mitmproxy/tools/main.py#L42-L48

Resolving patch:
diff --git a/mitmproxy/tools/main.py b/mitmproxy/tools/main.py
--- a/mitmproxy/tools/main.py
+++ b/mitmproxy/tools/main.py
@@ -44,8 +44,13 @@ def process_options(parser, opts, args):
         print(debug.dump_system_info())
         sys.exit(0)
     if args.quiet or args.options or args.commands:
+        # also reduce log verbosity if --options or --commands is passed,
+        # we don't want log messages from regular startup then.
         args.verbosity = 'error'
         args.flow_detail = 0
+    if args.verbose:
+        args.verbosity = 'debug'
+        args.flow_detail = 2
 
     adict = {}
     for n in dir(args):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Update socks mode access commandline + documentation (v3.0.2)
##### Steps to reproduce the problem:
1. mitmproxy --socks
"--socks is deprecated 
Please use '--set socks=value' instead"
2. Check online documentation at:
https://mitmproxy.org/docs/latest/concepts-modes/#socks-proxy
3. Check mitmproxy --help

##### Any other comments? What have you tried so far?
1. The advice given here doesn't appear to work (no combinations I tried were accepted).
2. The online documentation stops at Socks Proxy (no content)
3. The --help text shows the correct method (--mode socks5)

##### System information
Mitmproxy: 3.0.2
Python:    3.5.5rc1
OpenSSL:   OpenSSL 1.0.1f 6 Jan 2014
Platform:  Linux-2.6.39.4-kat124-ga627d40-armv7l-with-debian-jessie-sid

(Android: KatKiss Marshmallow hosting Linux Deploy: Ubuntu Trusty [armhf] using pyenv)

<!-- Please use the mitmproxy forums (https://discourse.mitmproxy.org/) for support/how-to questions. Thanks! :) -->



Discussion:

Resolving patch:
diff --git a/mitmproxy/utils/arg_check.py b/mitmproxy/utils/arg_check.py
--- a/mitmproxy/utils/arg_check.py
+++ b/mitmproxy/utils/arg_check.py
@@ -11,7 +11,6 @@
 --order
 --no-mouse
 --reverse
---socks
 --http2-priority
 --no-http2-priority
 --no-websocket
@@ -59,6 +58,7 @@
 -i
 -f
 --filter
+--socks
 """
 
 REPLACEMENTS = {
@@ -99,7 +99,8 @@
     "--replace": "--replacements",
     "-i": "--intercept",
     "-f": "--view-filter",
-    "--filter": "--view-filter"
+    "--filter": "--view-filter",
+    "--socks": "--mode socks5"
 }
 
 


Test patch: