internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump error logging is broken.
##### Steps to reproduce the problem:
1. `mitmdump`
2. `curl -x localhost:8080 example!!.com`
##### What is the expected behavior?

Proper console output
##### What went wrong?

```
λ mitmdump
127.0.0.1:56178: clientconnect
127.0.0.1:56178: request
  -> H
  -> T
  -> T
  -> P
  ->
  -> p
  -> r
  -> o
  -> t
  -> o
  -> c
  -> o
  -> l
  ->
  -> e
  -> r
  -> r
  -> o
  -> r
  ->
  -> i
  -> n
  ->
  -> c
  -> l
  -> i
  -> e
  -> n
  -> t
  ->
  -> r
  -> e
  -> q
  -> u
  -> e
  -> s
  -> t
  -> :
  ->
  -> B
  -> a
  -> d
  ->
  -> H
  -> T
  -> T
  -> P
  ->
  -> r
  -> e
  -> q
  -> u
  -> e
  -> s
  -> t
  ->
  -> l
  -> i
  -> n
  -> e
  -> :
  ->
  -> b
  -> '
  -> G
  -> E
  -> T
  ->
  -> H
  -> T
  -> T
  -> P
  -> :
  -> /
  -> /
  -> e
  -> x
  -> a
  -> m
  -> p
  -> l
  -> e
  -> !
  -> !
  -> .
  -> c
  -> o
  -> m
  -> /
  ->
  -> H
  -> T
  -> T
  -> P
  -> /
  -> 1
  -> .
  -> 1
  -> '
127.0.0.1:56178: clientdisconnect
```

---

Mitmproxy Version: master
Operating System: Win10 x64



Discussion:

Resolving patch:
diff --git a/mitmproxy/exceptions.py b/mitmproxy/exceptions.py
--- a/mitmproxy/exceptions.py
+++ b/mitmproxy/exceptions.py
@@ -27,6 +27,10 @@ class Kill(ProxyException):
 
 
 class ProtocolException(ProxyException):
+    """
+    ProtocolExceptions are caused by invalid user input, unavailable network resources,
+    or other events that are outside of our influence.
+    """
     pass
 
 
diff --git a/mitmproxy/protocol/http.py b/mitmproxy/protocol/http.py
--- a/mitmproxy/protocol/http.py
+++ b/mitmproxy/protocol/http.py
@@ -153,12 +153,13 @@ def __call__(self):
                 # We optimistically guess there might be an HTTP client on the
                 # other end
                 self.send_error_response(400, repr(e))
-                self.log(
-                    "request",
-                    "warn",
-                    "HTTP protocol error in client request: %s" % e
+                six.reraise(
+                    exceptions.ProtocolException,
+                    exceptions.ProtocolException(
+                        "HTTP protocol error in client request: {}".format(e)
+                    ),
+                    sys.exc_info()[2]
                 )
-                return
 
             self.log("request", "debug", [repr(request)])
 
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -132,7 +132,7 @@ def handle(self):
                 self.log(str(e), "warn")
                 self.log("Invalid certificate, closing connection. Pass --insecure to disable validation.", "warn")
             else:
-                self.log(repr(e), "warn")
+                self.log(str(e), "warn")
 
                 self.log(traceback.format_exc(), "debug")
             # If an error propagates to the topmost level,


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
pathoc does not accept `:pa,f` to pause forever at end of message
##### Steps to reproduce the problem:

`pathoc www.example.com 'get:/:pa,f'`
##### What is the expected behavior?

Send request, but pause forever after sending.
##### What went wrong?

I get a stack trace with "a float is required".

```
$ pathoc www.example.com 'get:/:pa,f'
08-09-16 16:59:41: >> 'GET':/:pa,f
Traceback (most recent call last):
  File "/usr/local/bin/pathoc", line 11, in <module>
    sys.exit(go_pathoc())
  File "/usr/local/lib/python2.7/dist-packages/pathod/pathoc_cmdline.py", line 226, in go_pathoc
    pathoc.main(args)
  File "/usr/local/lib/python2.7/dist-packages/pathod/pathoc.py", line 522, in main
    ret = p.request(spec)
  File "/usr/local/lib/python2.7/dist-packages/pathod/pathoc.py", line 452, in request
    return self.http(r)
  File "/usr/local/lib/python2.7/dist-packages/pathod/pathoc.py", line 432, in http
    return resp
  File "/usr/local/lib/python2.7/dist-packages/pathod/pathoc.py", line 411, in http
    req = language.serve(r, self.wfile, self.settings)
  File "/usr/local/lib/python2.7/dist-packages/pathod/language/__init__.py", line 105, in serve
    disconnect = writer.write_values(fp, vals, actions[:])
  File "/usr/local/lib/python2.7/dist-packages/pathod/language/writer.py", line 61, in write_values
    time.sleep(a[2])
TypeError: a float is required
```
##### Any other comments? What have you tried so far?

All other combinations of pause flags work as expected:

```
$ pathoc www.example.com 'get:/:p2,5'
08-09-16 17:05:07: >> 'GET':/:p2,5
<< 200 OK: 1270 bytes
$ pathoc www.example.com 'get:/:pr,5'
08-09-16 17:05:21: >> 'GET':/:pr,5
<< 200 OK: 1270 bytes
$ pathoc www.example.com 'get:/:pa,5'
08-09-16 17:05:41: >> 'GET':/:pa,5
<< 200 OK: 1270 bytes
$ pathoc www.example.com 'get:/:p2,f'
^C08-09-16 17:04:46: >> 'GET':/:p2,f
$ pathoc www.example.com 'get:/:pr,f'
^C08-09-16 17:04:55: >> 'GET':/:pr,f
```

---

pathoc version: 0.17
Operating System: Debian Linux 8.5 "Jessie" x64



Discussion:

Resolving patch:
diff --git a/pathod/language/writer.py b/pathod/language/writer.py
--- a/pathod/language/writer.py
+++ b/pathod/language/writer.py
@@ -57,7 +57,9 @@ def write_values(fp, vals, actions, sofar=0, blocksize=BLOCKSIZE):
         while actions:
             a = actions.pop()
             if a[1] == "pause":
-                time.sleep(a[2])
+                time.sleep(
+                    FOREVER if a[2] == "f" else a[2]
+                )
             elif a[1] == "disconnect":
                 return True
             elif a[1] == "inject":


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump doesn't seem to work when bound to localhost
##### Steps to reproduce the problem:


```
$ mitmdump -b 127.0.0.1 -R https://example.com -p 1234 -s "script.py arg"
Proxy server listening at http://127.0.0.1:1234
127.0.0.1:49415: clientconnect
127.0.0.1:49415: GET https://example.com/page
 << Server connection to example.com:443 failed: Error connecting to "example.com": [Errno 49] Can't assign requested address
```

##### Any other comments? What have you tried so far?

Works when bind argument isn't specified. Doesn't work on Windows either.

##### System information

Mitmproxy version: 0.18.2
Python version: 2.7.11
Platform: Darwin-15.6.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.11.6 ('', '', '') x86_64


Discussion:

Resolving patch:
diff --git a/mitmproxy/cmdline.py b/mitmproxy/cmdline.py
--- a/mitmproxy/cmdline.py
+++ b/mitmproxy/cmdline.py
@@ -208,6 +208,11 @@ def get_common_options(args):
     if args.quiet:
         args.verbose = 0
 
+    if args.addr in ("localhost", "127.0.0.1", "::1"):
+        upstream_bind_address = ""
+    else:
+        upstream_bind_address = args.addr
+
     return dict(
         app=args.app,
         app_host=args.app_host,
@@ -251,6 +256,7 @@ def get_common_options(args):
         ignore_hosts = args.ignore_hosts,
         listen_host = args.addr,
         listen_port = args.port,
+        upstream_bind_address = upstream_bind_address,
         mode = mode,
         no_upstream_cert = args.no_upstream_cert,
         spoof_source_address = args.spoof_source_address,
diff --git a/mitmproxy/options.py b/mitmproxy/options.py
--- a/mitmproxy/options.py
+++ b/mitmproxy/options.py
@@ -67,6 +67,7 @@ def __init__(
             ignore_hosts = (),  # type: Sequence[str]
             listen_host = "",  # type: str
             listen_port = LISTEN_PORT,  # type: int
+            upstream_bind_address = "",  # type: str
             mode = "regular",  # type: str
             no_upstream_cert = False,  # type: bool
             rawtcp = False,  # type: bool
@@ -127,6 +128,7 @@ def __init__(
         self.ignore_hosts = ignore_hosts
         self.listen_host = listen_host
         self.listen_port = listen_port
+        self.upstream_bind_address = upstream_bind_address
         self.mode = mode
         self.no_upstream_cert = no_upstream_cert
         self.rawtcp = rawtcp
diff --git a/mitmproxy/protocol/base.py b/mitmproxy/protocol/base.py
--- a/mitmproxy/protocol/base.py
+++ b/mitmproxy/protocol/base.py
@@ -121,7 +121,7 @@ def __init__(self, server_address=None):
                 server_address, (self.ctx.client_conn.address.host, 0), True)
         else:
             self.server_conn = models.ServerConnection(
-                server_address, (self.config.options.listen_host, 0))
+                server_address, (self.config.options.upstream_bind_address, 0))
 
         self.__check_self_connect()
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
PyInstaller: OSX builds have mitmproxy_main executable, not mitmproxy
##### Steps to reproduce the problem:

1. Download snapshot from https://snapshots.mitmproxy.org/v0.19/



Discussion:

Resolving patch:
diff --git a/release/rtool.py b/release/rtool.py
--- a/release/rtool.py
+++ b/release/rtool.py
@@ -196,7 +196,7 @@ def make_bdist():
                     executable += ".exe"
 
                 # Remove _main suffix from mitmproxy executable
-                if executable.startswith("mitmproxy_main"):
+                if "_main" in executable:
                     shutil.move(
                         executable,
                         executable.replace("_main", "")


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Inline script errors not thrown in eventlog
##### Steps to reproduce the problem:

1. edit any inline script and add an error (example sslstrip-test.py attached with function nothing())
[sslstrip-test.txt](https://github.com/mitmproxy/mitmproxy/files/608581/sslstrip-test.txt)
2. mitmproxy --script sslstrip-test.py --eventlog
3. nothing in eventlog (see screenshot) 
![mitmproxy-error-not-shown](https://cloud.githubusercontent.com/assets/19553457/20568123/ab76f120-b19b-11e6-836b-dd3800a8c6d4.png)



##### Any other comments? What have you tried so far?
Remember that it was working with v0.17.1, Maximilian said it was fixed from v0.18 to v0.18.2, so am i missing something here (a debug mode?)? Mitmproxy is still working, ignoring the script.

Thanks a lot,

##### System information
Mitmproxy version: 0.19
Python version: 3.5.2+
Platform: Linux-4.8.0-kali1-amd64-x86_64-with-Kali-kali-rolling-kali-rolling
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Linux distro: Kali kali-rolling kali-rolling

<!--
    Cut and paste the output of "mitmdump --sysinfo".

    If you're using an older version if mitmproxy, please specify the version
    and OS.
-->



Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/console/master.py b/mitmproxy/tools/console/master.py
--- a/mitmproxy/tools/console/master.py
+++ b/mitmproxy/tools/console/master.py
@@ -38,6 +38,11 @@
 EVENTLOG_SIZE = 500
 
 
+class Logger:
+    def log(self, evt):
+        signals.add_log(evt.msg, evt.level)
+
+
 class ConsoleMaster(master.Master):
     palette = []
 
@@ -58,6 +63,7 @@ def __init__(self, options, server):
         signals.replace_view_state.connect(self.sig_replace_view_state)
         signals.push_view_state.connect(self.sig_push_view_state)
         signals.sig_add_log.connect(self.sig_add_log)
+        self.addons.add(Logger())
         self.addons.add(*addons.default_addons())
         self.addons.add(intercept.Intercept(), self.view)
 
@@ -447,7 +453,3 @@ def tcp_message(self, f):
             direction=direction,
         ), "info")
         signals.add_log(strutils.bytes_to_escaped_str(message.content), "debug")
-
-    @controller.handler
-    def log(self, evt):
-        signals.add_log(evt.msg, evt.level)


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
FlowView is not updating
##### Steps to reproduce the problem:

1. Open Flow Reponse View
2. Replay Flow (`r`)

##### System information

Mitmproxy version: 0.19
Python version: 3.5.2
Platform: Windows-10-10.0.14393-SP0
SSL version: OpenSSL 1.0.2h  3 May 2016
Windows version: 10 10.0.14393 SP0 Multiprocessor Free
FlowView is not updating
##### Steps to reproduce the problem:

1. Open Flow Reponse View
2. Replay Flow (`r`)

##### System information

Mitmproxy version: 0.19
Python version: 3.5.2
Platform: Windows-10-10.0.14393-SP0
SSL version: OpenSSL 1.0.2h  3 May 2016
Windows version: 10 10.0.14393 SP0 Multiprocessor Free


Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/console/master.py b/mitmproxy/tools/console/master.py
--- a/mitmproxy/tools/console/master.py
+++ b/mitmproxy/tools/console/master.py
@@ -49,6 +49,7 @@ class ConsoleMaster(master.Master):
     def __init__(self, options, server):
         super().__init__(options, server)
         self.view = view.View()  # type: view.View
+        self.view.sig_view_update.connect(signals.flow_change.send)
         self.stream_path = None
         # This line is just for type hinting
         self.options = self.options  # type: Options


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Gracefully handle underscores in hostnames
We need to handle invalid hostnames more gracefully. See the discussion here:

https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-urls-which-have-under-scores/
##### Steps to reproduce the problem:
1. mitmdump -vv
2. env http_proxy=http://localhost:8080 curl "http://foo_bar.com"

My inclination here is to make the host matching more liberal, and pass this sort of error upstream. Basically, if I'm using mitmproxy in an analysis situation, and the client is exhibiting this behaviour, I want to see what the server does with it rather than see an error.

Either way, we certainly shouldn't be spewing a user-visible traceback to the logs for this kind of protocol error. 



Discussion:

Resolving patch:
diff --git a/mitmproxy/net/check.py b/mitmproxy/net/check.py
--- a/mitmproxy/net/check.py
+++ b/mitmproxy/net/check.py
@@ -1,6 +1,7 @@
 import re
 
-_label_valid = re.compile(b"(?!-)[A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
+# Allow underscore in host name
+_label_valid = re.compile(b"(?!-)[A-Z\d\-_]{1,63}(?<!-)$", re.IGNORECASE)
 
 
 def is_valid_host(host: bytes) -> bool:


Test patch:
diff --git a/test/mitmproxy/net/test_check.py b/test/mitmproxy/net/test_check.py
--- a/test/mitmproxy/net/test_check.py
+++ b/test/mitmproxy/net/test_check.py
@@ -8,3 +8,5 @@ def test_is_valid_host():
     assert check.is_valid_host(b"one.two")
     assert not check.is_valid_host(b"one" * 255)
     assert check.is_valid_host(b"one.two.")
+    # Allow underscore
+    assert check.is_valid_host(b"one_two")
Repository: mitmproxy/mitmproxy

Issue:
Change view._store from dictionary to OrderedDict
##### Steps to reproduce the problem:
 
1. Mark multiple flows in the flow list using `m` key  
2. Toggle to show-marked-only view using `M`
3. mitmproxy rearranges flow list when `M` is toggled


##### Any other comments? What have you tried so far?

This is an issue with the flow list mentioned earlier in #1960. The marked flows can get randomly rearranged whenever M or Z is pressed, as it iterates through `view._store` which is a dictionary, to refresh the view.

##### System information

Mitmproxy version: 2.0.0 (v1.0.1-45-g0022c81)
Python version: 3.5.3
Platform: Linux-4.8.2-c9-x86_64-with-debian-jessie-sid
SSL version: OpenSSL 1.0.1f 6 Jan 2014
Linux distro: debian jessie/sid 
<!--
    Cut and paste the output of "mitmproxy --version".

    If you're using an older version if mitmproxy, please specify the version
    and OS.
-->



Discussion:

Resolving patch:
diff --git a/mitmproxy/addons/view.py b/mitmproxy/addons/view.py
--- a/mitmproxy/addons/view.py
+++ b/mitmproxy/addons/view.py
@@ -102,7 +102,7 @@ def generate(self, f: mitmproxy.flow.Flow) -> int:
 class View(collections.Sequence):
     def __init__(self):
         super().__init__()
-        self._store = {}
+        self._store = collections.OrderedDict()
         self.filter = matchall
         # Should we show only marked flows?
         self.show_marked = False


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Provide a better error message for proxy mode misconfigurations
##### Steps to reproduce the problem:
1. `mitmproxy -T`
2. `curl -x localhost:8080 example.com`
##### What is the expected behavior?

mitmproxy should provide a warning that you want to run either transparent mode or configure an explicit proxy.
##### What went wrong?

mitmproxy complains about "Invalid HTTP request form (expected: relative, got: authority)"
##### Any other comments? What have you tried so far?

We should just detect this very specific case (expected: relative, got: authority) and provide a better error message for it.

See #1579 for a confused user. :stuck_out_tongue:

---

Mitmproxy Version: master
Operating System: Ubuntu 16.04



Discussion:
Seems like one just needs to throw in another "if" check and raise with a more appropriate exception? If so, I can take care of this.

I would like to work on this issue.

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py
--- a/mitmproxy/proxy/protocol/http.py
+++ b/mitmproxy/proxy/protocol/http.py
@@ -131,7 +131,7 @@ class HTTPMode(enum.Enum):
 # At this point, we see only a subset of the proxy modes
 MODE_REQUEST_FORMS = {
     HTTPMode.regular: ("authority", "absolute"),
-    HTTPMode.transparent: ("relative"),
+    HTTPMode.transparent: ("relative",),
     HTTPMode.upstream: ("authority", "absolute"),
 }
 
@@ -143,9 +143,16 @@ def validate_request_form(mode, request):
         )
     allowed_request_forms = MODE_REQUEST_FORMS[mode]
     if request.first_line_format not in allowed_request_forms:
-        err_message = "Invalid HTTP request form (expected: %s, got: %s)" % (
-            " or ".join(allowed_request_forms), request.first_line_format
-        )
+        if mode == HTTPMode.transparent:
+            err_message = (
+                "Mitmproxy received an {} request even though it is not running in regular mode. "
+                "This usually indicates a misconfiguration, please see "
+                "http://docs.mitmproxy.org/en/stable/modes.html for details."
+            ).format("HTTP CONNECT" if request.first_line_format == "authority" else "absolute-form")
+        else:
+            err_message = "Invalid HTTP request form (expected: %s, got: %s)" % (
+                " or ".join(allowed_request_forms), request.first_line_format
+            )
         raise exceptions.HttpException(err_message)
 
 


Test patch:
diff --git a/test/mitmproxy/protocol/test_http1.py b/test/mitmproxy/protocol/test_http1.py
--- a/test/mitmproxy/protocol/test_http1.py
+++ b/test/mitmproxy/protocol/test_http1.py
@@ -30,6 +30,16 @@ def test_relative_request(self):
         assert b"Invalid HTTP request form" in r.content
 
 
+class TestProxyMisconfiguration(tservers.TransparentProxyTest):
+
+    def test_absolute_request(self):
+        p = self.pathoc()
+        with p.connect():
+            r = p.request("get:'http://localhost:%d/p/200'" % self.server.port)
+        assert r.status_code == 400
+        assert b"misconfiguration" in r.content
+
+
 class TestExpectHeader(tservers.HTTPProxyTest):
 
     def test_simple(self):
Repository: mitmproxy/mitmproxy

Issue:
Remove pypy support
Officially mitmproxy still claims to support pypy. However, I can't remember seeing any test runs - let alone successful ones. mitmproxy requires Python 3.5+, but pypy currently only offers a Python3.3 compatible release.

We should therefore remove pypy-related things from `setup.py` and `.travis.yml`.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -37,7 +37,6 @@
         "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: Implementation :: CPython",
-        "Programming Language :: Python :: Implementation :: PyPy",
         "Topic :: Security",
         "Topic :: Internet",
         "Topic :: Internet :: WWW/HTTP",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Add PyInstaller indicator to `mitmproxy --version`
We currently cannot distinguish if users use our precompiled binaries or if they installed mitmproxy using pip/brew/$packagemanager. It would be very useful to output if we are running the precompiled PyInstaller binary. 

https://pythonhosted.org/PyInstaller/runtime-information.html


Discussion:
Use of precompiled binaries can be detected (I'll send a PR by tomorrow) but how would it distinguish between the other cases (system package / pip / source), or it just wouldn't be necessary?
As first step, identifying pyinstaller builds is good enough.
We already distinguish between release versions and git-based installations based on `git describe` and tags.
Sure, thanks!

Resolving patch:
diff --git a/mitmproxy/utils/debug.py b/mitmproxy/utils/debug.py
--- a/mitmproxy/utils/debug.py
+++ b/mitmproxy/utils/debug.py
@@ -37,8 +37,12 @@ def dump_system_info():
         except:
             pass
 
+    bin_indicator = ""  # PyInstaller builds indicator, if using precompiled binary
+    if getattr(sys, 'frozen', False):
+        bin_indicator = "Precompiled Binary"
+
     data = [
-        "Mitmproxy version: {} ({})".format(version.VERSION, git_describe),
+        "Mitmproxy version: {} ({}) {}".format(version.VERSION, git_describe, bin_indicator),
         "Python version: {}".format(platform.python_version()),
         "Platform: {}".format(platform.platform()),
         "SSL version: {}".format(SSL.SSLeay_version(SSL.SSLEAY_VERSION).decode()),


Test patch:
diff --git a/test/mitmproxy/utils/test_debug.py b/test/mitmproxy/utils/test_debug.py
--- a/test/mitmproxy/utils/test_debug.py
+++ b/test/mitmproxy/utils/test_debug.py
@@ -1,11 +1,13 @@
 import io
 import subprocess
+import sys
 from unittest import mock
 
 from mitmproxy.utils import debug
 
 
 def test_dump_system_info():
+    setattr(sys, 'frozen', True)
     assert debug.dump_system_info()
 
     with mock.patch('subprocess.check_output') as m: