internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
Support absolute-form HTTP requests with IPv6 addresses
##### Steps to reproduce the problem:

1. MITMDump proxy IPv6 flow
2. Log
```
172.17.15.1:53074: HTTP protocol error in client request: Bad HTTP request line: b'GET http://[::ffff:180.97.8.37]/mmsns/9KavCVwReibwDKBMmibrWUdVZZbHCQ0bV3R89mboKO6QDls7Sxcl4tfbHvLIHFbj3NASftTH2VAGw/150?tp=wxpc&length=2208&width=1242&idx=1&token=WSEN6qDsKwV8A02w3onOGQYfxnkibdqSOkmHhZGNB4DGicdGyTltMQXCTF7lr4IJR8Jz4lKQBBW47EV1CP33SGjg HTTP/1.1'
172.17.15.1:53075: HTTP protocol error in client request: Bad HTTP request line: b'GET http://[::ffff:b461:819]/mmcrhead/Q3auHgzwzM606QEH0kXoF60vMh5Iiay7B3DiauET3kCpbBwEfgzhNqOSeJ6y4geORGPxEcKf36Totd4sHQcwvBEg/0 HTTP/1.1'
```


##### Any other comments? What have you tried so far?
No


##### System information
```
Mitmproxy version: 1.0.2
Python version: 3.6.0
Platform: Darwin-15.6.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.11.6 ('', '', '') x86_6
```




Discussion:
Thanks for the report!

Resolving patch:
diff --git a/mitmproxy/net/check.py b/mitmproxy/net/check.py
--- a/mitmproxy/net/check.py
+++ b/mitmproxy/net/check.py
@@ -1,3 +1,4 @@
+import ipaddress
 import re
 
 # Allow underscore in host name
@@ -6,17 +7,26 @@
 
 def is_valid_host(host: bytes) -> bool:
     """
-        Checks if a hostname is valid.
+    Checks if the passed bytes are a valid DNS hostname or an IPv4/IPv6 address.
     """
     try:
         host.decode("idna")
     except ValueError:
         return False
+    # RFC1035: 255 bytes or less.
     if len(host) > 255:
         return False
     if host and host[-1:] == b".":
         host = host[:-1]
-    return all(_label_valid.match(x) for x in host.split(b"."))
+    # DNS hostname
+    if all(_label_valid.match(x) for x in host.split(b".")):
+        return True
+    # IPv4/IPv6 address
+    try:
+        ipaddress.ip_address(host.decode('idna'))
+        return True
+    except ValueError:
+        return False
 
 
 def is_valid_port(port):


Test patch:
diff --git a/test/mitmproxy/net/test_check.py b/test/mitmproxy/net/test_check.py
--- a/test/mitmproxy/net/test_check.py
+++ b/test/mitmproxy/net/test_check.py
@@ -11,3 +11,4 @@ def test_is_valid_host():
     assert check.is_valid_host(b"one.two.")
     # Allow underscore
     assert check.is_valid_host(b"one_two")
+    assert check.is_valid_host(b"::1")
Repository: mitmproxy/mitmproxy

Issue:
@concurrent annotation doesn't work in the OOP script method
I.E:

```
class SomeClass:
    @concurrent [doesn't work, 2 args]
    def request(self, flow):
        pass

```
@concurrent annotation doesn't work in the OOP script method
I.E:

```
class SomeClass:
    @concurrent [doesn't work, 2 args]
    def request(self, flow):
        pass

```


Discussion:
Good catch - thanks!
Good catch - thanks!

Resolving patch:
diff --git a/mitmproxy/script/concurrent.py b/mitmproxy/script/concurrent.py
--- a/mitmproxy/script/concurrent.py
+++ b/mitmproxy/script/concurrent.py
@@ -29,4 +29,8 @@ def run():
             "script.concurrent (%s)" % fn.__name__,
             target=run
         ).start()
-    return _concurrent
+    # Support @concurrent for class-based addons
+    if "." in fn.__qualname__:
+        return staticmethod(_concurrent)
+    else:
+        return _concurrent


Test patch:
diff --git a/test/mitmproxy/data/addonscripts/concurrent_decorator_class.py b/test/mitmproxy/data/addonscripts/concurrent_decorator_class.py
new file mode 100644
--- /dev/null
+++ b/test/mitmproxy/data/addonscripts/concurrent_decorator_class.py
@@ -0,0 +1,13 @@
+import time
+from mitmproxy.script import concurrent
+
+
+class ConcurrentClass:
+
+    @concurrent
+    def request(flow):
+        time.sleep(0.1)
+
+
+def start():
+    return ConcurrentClass()
diff --git a/test/mitmproxy/script/test_concurrent.py b/test/mitmproxy/script/test_concurrent.py
--- a/test/mitmproxy/script/test_concurrent.py
+++ b/test/mitmproxy/script/test_concurrent.py
@@ -44,3 +44,21 @@ def test_concurrent_err(self):
             )
             sc.start()
             assert "decorator not supported" in tctx.master.event_log[0][1]
+
+    def test_concurrent_class(self):
+            with taddons.context() as tctx:
+                sc = script.Script(
+                    tutils.test_data.path(
+                        "mitmproxy/data/addonscripts/concurrent_decorator_class.py"
+                    )
+                )
+                sc.start()
+
+                f1, f2 = tflow.tflow(), tflow.tflow()
+                tctx.cycle(sc, f1)
+                tctx.cycle(sc, f2)
+                start = time.time()
+                while time.time() - start < 5:
+                    if f1.reply.state == f2.reply.state == "committed":
+                        return
+                raise ValueError("Script never acked")
Repository: mitmproxy/mitmproxy

Issue:
mitmweb: --host Not Working
##### Steps to reproduce the problem:

mitmweb is not respecting --host. All hosts are ips, never domain names. "showhost" is displayed at bottom of mitmweb web page.

##### Any other comments? What have you tried so far?

I had reported this long ago. The issue was switches weren't implemented. Now they are but --host isn't working anyway.

##### System information

Mitmproxy version: 1.0.0
Python version: 3.5.2
Platform: Windows-10-10.0.14393
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Windows version: 10 10.0.14393  Multiprocessor Free



Discussion:
As always, thanks for the great report. We really appreciate having someone who consistently reports good Windows and mitmweb issues 😊

Regarding the actual issue: This needs to be implemented [here](https://github.com/mitmproxy/mitmproxy/blob/27353388157adda3d22000fcf146fe599f4f9a6f/web/src/js/flow/utils.js#L58).
Hi, I'd like to work on this. I'm still going through the source code though so some more time would be appreciated. Thanks!

Resolving patch:
diff --git a/mitmproxy/tools/web/app.py b/mitmproxy/tools/web/app.py
--- a/mitmproxy/tools/web/app.py
+++ b/mitmproxy/tools/web/app.py
@@ -65,6 +65,7 @@ def flow_to_json(flow: mitmproxy.flow.Flow) -> dict:
                 "timestamp_start": flow.request.timestamp_start,
                 "timestamp_end": flow.request.timestamp_end,
                 "is_replay": flow.request.is_replay,
+                "pretty_host": flow.request.pretty_host,
             }
         if flow.response:
             if flow.response.raw_content:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump -nr does not exit automatically
##### Steps to reproduce the problem:

1. Use `mitmdump -nr foo.mitm` to print some flows.
2. Mitmdump should exit automatically after printing, but it doesn't.

##### System information

Mitmproxy version: 3.0.0 (2.0.0dev0136-0x05e1154)
Python version: 3.5.2
Platform: Linux-3.4.0+-x86_64-with-Ubuntu-14.04-trusty
SSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
Linux distro: Ubuntu 14.04 trusty
mitmdump -nr does not exit automatically
##### Steps to reproduce the problem:

1. Use `mitmdump -nr foo.mitm` to print some flows.
2. Mitmdump should exit automatically after printing, but it doesn't.

##### System information

Mitmproxy version: 3.0.0 (2.0.0dev0136-0x05e1154)
Python version: 3.5.2
Platform: Linux-3.4.0+-x86_64-with-Ubuntu-14.04-trusty
SSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
Linux distro: Ubuntu 14.04 trusty


Discussion:

Resolving patch:
diff --git a/mitmproxy/options.py b/mitmproxy/options.py
--- a/mitmproxy/options.py
+++ b/mitmproxy/options.py
@@ -78,7 +78,7 @@ def __init__(self, **kwargs) -> None:
             "Kill extra requests during replay."
         )
         self.add_option(
-            "keepserving", bool, True,
+            "keepserving", bool, False,
             "Continue serving after client playback or file read."
         )
         self.add_option(


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Can't filter by hostname when --host is used
##### Steps to reproduce the problem:

1. mitmproxy -T --host
2. Capture something
3. Press f and filter by something like `~d domain.com` or `~u domain com`

##### Any other comments? What have you tried so far?

Going to detail view to get the resolved IP address and using it for the filter works, but it's uncomfortable when you want to match things like "google.com" (lots of IP ranges, lots of unrelated domains, but many mentioning google)

##### System information

Mitmproxy version: 2.0.0 (release version)
Python version: 3.6.0
Platform: Linux-4.9.11-1-ARCH-x86_64-with-arch
SSL version: OpenSSL 1.0.2k  26 Jan 2017
Linux distro: arch


Discussion:
Thanks for the report. It's probably a good idea to also iterate over the host headers for `~d`. This needs to be fixed in flowfilter.py and filt.peg.

Resolving patch:
diff --git a/mitmproxy/flowfilter.py b/mitmproxy/flowfilter.py
--- a/mitmproxy/flowfilter.py
+++ b/mitmproxy/flowfilter.py
@@ -319,10 +319,14 @@ class FDomain(_Rex):
     code = "d"
     help = "Domain"
     flags = re.IGNORECASE
+    is_binary = False
 
     @only(http.HTTPFlow)
     def __call__(self, f):
-        return bool(self.re.search(f.request.data.host))
+        return bool(
+            self.re.search(f.request.host) or
+            self.re.search(f.request.pretty_host)
+        )
 
 
 class FUrl(_Rex):
@@ -339,7 +343,7 @@ def make(klass, s, loc, toks):
 
     @only(http.HTTPFlow)
     def __call__(self, f):
-        return self.re.search(f.request.url)
+        return self.re.search(f.request.pretty_url)
 
 
 class FSrc(_Rex):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
cryptography dependency update?
The template here wasn't relevant so I cut it, apologies.

`mitmproxy` currently mandates `cryptography <1.8` which when built from source on macOS/OS X El Capitan (10.11) using Xcode 8 results in `Symbol not found: _getentropy` errors. Reported and explained in more detail [here](https://github.com/pyca/cryptography/issues/3332), hence my short summary. These issues were fixed in https://github.com/pyca/cryptography/pull/3354, which was released in the 1.8 branch.

This is also currently causing some "fun" for downstream package managers having to carry their own somewhat hacky patches like [this](https://github.com/Homebrew/homebrew-core/blob/37abcfc55099f635d0e187657a55a1eed36b5ccf/Formula/mitmproxy.rb#L193-L202). The gist of my request is whether `mitmproxy` has plans to/would consider update/updating [cryptography](https://github.com/mitmproxy/mitmproxy/blob/e723a58af5dc4fc7a46958aa9ce8c386a7387450/setup.py#L67) to use the latest release without these issues.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -64,7 +64,7 @@
         "click>=6.2, <7",
         "certifi>=2015.11.20.1",  # no semver here - this should always be on the last release!
         "construct>=2.8, <2.9",
-        "cryptography>=1.3, <1.8",
+        "cryptography>=1.3, <1.9",
         "cssutils>=1.0.1, <1.1",
         "h2>=2.5.1, <3",
         "html2text>=2016.1.8, <=2016.9.19",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmweb -n: Proxy server listening at http://d:u/
##### Steps to reproduce the problem:

1. `mitmweb -n`
2. Incorrect information is displayed on the console: "Proxy server listening at http://d:u/" 

##### Any other comments? What have you tried so far?

mitmweb should use the termstatus addon for printing this as well.

##### System information


Mitmproxy version: 3.0.0 (2.0.0dev0203-0xc088249)
Python version: 3.5.2
Platform: Linux-3.4.0+-x86_64-with-Ubuntu-14.04-trusty
SSL version: OpenSSL 1.0.2g-fips  1 Mar 2016
Linux distro: Ubuntu 14.04 trusty


Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/web/app.py b/mitmproxy/tools/web/app.py
--- a/mitmproxy/tools/web/app.py
+++ b/mitmproxy/tools/web/app.py
@@ -421,6 +421,7 @@ def get(self):
             contentViews=[v.name.replace(' ', '_') for v in contentviews.views],
             listen_host=self.master.options.listen_host,
             listen_port=self.master.options.listen_port,
+            server=self.master.options.server,
         ))
 
     def put(self):
diff --git a/mitmproxy/tools/web/master.py b/mitmproxy/tools/web/master.py
--- a/mitmproxy/tools/web/master.py
+++ b/mitmproxy/tools/web/master.py
@@ -9,6 +9,7 @@
 from mitmproxy.addons import intercept
 from mitmproxy.addons import termlog
 from mitmproxy.addons import view
+from mitmproxy.addons import termstatus
 from mitmproxy.options import Options  # noqa
 from mitmproxy.tools.web import app
 
@@ -35,7 +36,7 @@ def __init__(self, options, server, with_termlog=True):
             self.events,
         )
         if with_termlog:
-            self.addons.add(termlog.TermLog())
+            self.addons.add(termlog.TermLog(), termstatus.TermStatus())
         self.app = app.Application(
             self, self.options.web_debug
         )
@@ -99,11 +100,6 @@ def run(self):  # pragma: no cover
         iol.add_callback(self.start)
         tornado.ioloop.PeriodicCallback(lambda: self.tick(timeout=0), 5).start()
 
-        self.add_log(
-            "Proxy server listening at http://{}:{}/".format(self.server.address[0], self.server.address[1]),
-            "info"
-        )
-
         web_url = "http://{}:{}/".format(self.options.web_iface, self.options.web_port)
         self.add_log(
             "Web   server listening at {}".format(web_url),


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump: unrecognized arguments: -dd
##### Steps to reproduce the problem:

1. `mitmdump -d -nr foo.mitm`

```
mitmdump: error: unrecognized arguments: -d
```

##### System information

Mitmproxy version: 3.0.0 (2.0.0dev0136-0x05e1154)
Python version: 3.5.2
Platform: Linux-3.4.0+-x86_64-with-Ubuntu-14.04-trusty
SSL version: OpenSSL 1.0.2g-fips  1 Mar 2016
Linux distro: Ubuntu 14.04 trusty


Discussion:
This is apparently `--flow-detail` now. `mitmdump --help` explains that I should use `--flow-detail LEVEL`, but does not indicate what a valid value for LEVEL would be.

Resolving patch:
diff --git a/mitmproxy/options.py b/mitmproxy/options.py
--- a/mitmproxy/options.py
+++ b/mitmproxy/options.py
@@ -433,7 +433,13 @@ def __init__(self, **kwargs) -> None:
         )
         self.add_option(
             "flow_detail", int, 1,
-            "Flow detail display level."
+            """
+            The display detail level for flows in mitmdump: 0 (almost quiet) to 3 (very verbose).
+              0: shortened request URL, response status code, WebSocket and TCP message notifications.
+              1: full request URL with response status code
+              2: 1 + HTTP headers
+              3: 2 + full response content, content of WebSocket and TCP messages.
+            """
         )
 
         self.update(**kwargs)


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Flaky Coverage for http2.py
See [codecov](https://codecov.io/gh/mitmproxy/mitmproxy/compare/ab2fcbef8dceb5abc97f7e73638629ded606daa6...82ecc5448d4e3eb53a947154f49d431148f6508a/src/mitmproxy/proxy/protocol/http2.py) for https://github.com/mitmproxy/mitmproxy/pull/2200. Apparently that line is not triggered consistently in tests.


Discussion:
Here's another one: https://codecov.io/gh/mitmproxy/mitmproxy/compare/c6a16e95e856c859b147e72a484feefe96c37ad9...4201fc3073d2e683ff1471a1aedc18aea923cf38/src/mitmproxy/proxy/protocol/http2.py

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/http2.py b/mitmproxy/proxy/protocol/http2.py
--- a/mitmproxy/proxy/protocol/http2.py
+++ b/mitmproxy/proxy/protocol/http2.py
@@ -62,7 +62,7 @@ def safe_send_body(self, raise_zombie: Callable, stream_id: int, chunks: List[by
                 raise_zombie(self.lock.release)
                 max_outbound_frame_size = self.max_outbound_frame_size
                 frame_chunk = chunk[position:position + max_outbound_frame_size]
-                if self.local_flow_control_window(stream_id) < len(frame_chunk):
+                if self.local_flow_control_window(stream_id) < len(frame_chunk):  # pragma: no cover
                     self.lock.release()
                     time.sleep(0.1)
                     continue
@@ -362,7 +362,7 @@ def __call__(self):
             self._kill_all_streams()
 
 
-def detect_zombie_stream(func):
+def detect_zombie_stream(func):  # pragma: no cover
     @functools.wraps(func)
     def wrapper(self, *args, **kwargs):
         self.raise_zombie()
@@ -454,7 +454,7 @@ def data_finished(self):
         else:
             return self.request_data_finished
 
-    def raise_zombie(self, pre_command=None):
+    def raise_zombie(self, pre_command=None):  # pragma: no cover
         connection_closed = self.h2_connection.state_machine.state == h2.connection.ConnectionState.CLOSED
         if self.zombie is not None or connection_closed:
             if pre_command is not None:
@@ -626,7 +626,7 @@ def run(self):
             self.log(repr(e), "info")
         except exceptions.SetServerNotAllowedException as e:  # pragma: no cover
             self.log("Changing the Host server for HTTP/2 connections not allowed: {}".format(e), "info")
-        except exceptions.Kill:
+        except exceptions.Kill:  # pragma: no cover
             self.log("Connection killed", "info")
 
         self.kill()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Private key is present in mitmproxy-ca-cert.p12
##### Steps to reproduce the problem:

Private key is inserted in .p12 file, although filename says "mitmproxy-ca-**cert**.p12"
Output can be verified with:
openssl pkcs12 -in mitmproxy-ca-cert.p12 -nocerts -nodes

I don't have a Windows machine to test if a .p12 file would work without the private key inside.
We can generate a .p12 file without key with:
openssl pkcs12 -export -nokeys -in mitmproxy-ca-cert.pem -out mitmproxy-ca-cert.p12

-> If private key is needed, file could be renamed "mitmproxy-ca.p12" for better understanding
-> If private key is not needed, .p12 generation should be modified



##### System information


Mitmproxy version: 2.0.1 (release version) Precompiled Binary
Python version: 3.5.2
Platform: Darwin-16.6.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.12.5 ('', '', '') x86_64




Discussion:
Thanks for the useful report. I just tested this and found that the private key is not needed. Fixing this in [certs.py](https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/certs.py) should be quite easy, so I'll leave this open for new contributors! :)

Resolving patch:
diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py
--- a/mitmproxy/certs.py
+++ b/mitmproxy/certs.py
@@ -264,6 +264,12 @@ def create_store(path, basename, o=None, cn=None, expiry=DEFAULT_EXP):
 
         # Dump the certificate in PKCS12 format for Windows devices
         with open(os.path.join(path, basename + "-ca-cert.p12"), "wb") as f:
+            p12 = OpenSSL.crypto.PKCS12()
+            p12.set_certificate(ca)
+            f.write(p12.export())
+
+        # Dump the certificate and key in a PKCS12 format for Windows devices
+        with open(os.path.join(path, basename + "-ca.p12"), "wb") as f:
             p12 = OpenSSL.crypto.PKCS12()
             p12.set_certificate(ca)
             p12.set_privatekey(key)


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Script Reloader does not unload old script
##### Steps to reproduce the problem:

1. mitmdump -s script.py
2. Modify script on disk.

##### System information

Mitmproxy version: 3.0.0 (2.0.0dev0407-0x315daa0)
Python version: 3.5.2
Platform: Linux-4.4.0-43-Microsoft-x86_64-with-Ubuntu-16.04-xenial
SSL version: OpenSSL 1.0.2g  1 Mar 2016
Linux distro: Ubuntu 16.04 xenial


Discussion:
Also: We fail badly if the script is deleted during execution.

Resolving patch:
diff --git a/mitmproxy/addons/script.py b/mitmproxy/addons/script.py
--- a/mitmproxy/addons/script.py
+++ b/mitmproxy/addons/script.py
@@ -52,11 +52,19 @@ def addons(self):
 
     def tick(self):
         if time.time() - self.last_load > self.ReloadInterval:
-            mtime = os.stat(self.fullpath).st_mtime
+            try:
+                mtime = os.stat(self.fullpath).st_mtime
+            except FileNotFoundError:
+                scripts = ctx.options.scripts
+                scripts.remove(self.path)
+                ctx.options.update(scripts=scripts)
+                return
+
             if mtime > self.last_mtime:
                 ctx.log.info("Loading script: %s" % self.path)
                 if self.ns:
                     ctx.master.addons.remove(self.ns)
+                    del sys.modules[self.ns.__name__]
                 self.ns = load_script(ctx, self.fullpath)
                 if self.ns:
                     # We're already running, so we have to explicitly register and


Test patch:
diff --git a/test/mitmproxy/addons/test_script.py b/test/mitmproxy/addons/test_script.py
--- a/test/mitmproxy/addons/test_script.py
+++ b/test/mitmproxy/addons/test_script.py
@@ -1,6 +1,7 @@
 import traceback
 import sys
 import time
+import os
 import pytest
 
 from unittest import mock
@@ -183,6 +184,20 @@ def test_dupes(self):
                     scripts = ["one", "one"]
                 )
 
+    def test_script_deletion(self):
+        tdir = tutils.test_data.path("mitmproxy/data/addonscripts/")
+        with open(tdir + "/dummy.py", 'w') as f:
+            f.write("\n")
+        with taddons.context() as tctx:
+            sl = script.ScriptLoader()
+            tctx.master.addons.add(sl)
+            tctx.configure(sl, scripts=[tutils.test_data.path("mitmproxy/data/addonscripts/dummy.py")])
+
+            os.remove(tutils.test_data.path("mitmproxy/data/addonscripts/dummy.py"))
+            tctx.invoke(sl, "tick")
+            assert not tctx.options.scripts
+            assert not sl.addons
+
     def test_order(self):
         rec = tutils.test_data.path("mitmproxy/data/addonscripts/recorder")
         sc = script.ScriptLoader()