internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
Postpone establishing connection with server
Hi, I use mitmproxy in upstream mode as a flow replayer by loading a couple scripts and resource files. It's never intended to deliver a request to remote server. Everything is ok except that, although each of the canned responses (loaded from a file on the disk) is correctly returned to the client, it actually makes connection to remote server on each of the flow via the upstream proxy server I configured in the cmd line, which is totally unnecessary and time-wasting in my senario. 
Would it be possible to postpone the establishing of connection with server (after the request() hook has been revoked) so that my upstream mitmproxy server can run with a dummy -U flag, which don't have to be a real proxy server at all, as it never should really forward a request/establish a connection to remote server?


Discussion:
Thanks for raising this. As you never intend to send a request upstream, have you tried just using mitmproxy in regular mode instead? That should give you the desired behaviour.

In any case, we need to make the strategy for upstream connections user-decidable. I tracked this in #1775.
Interesting, yes, as you suggested, it works in regular mode, even if machine the mitmproxy resides on does not have internet connection at all.

However, most of my other client apps would require features of upstream forwarding. Would it be possible for me to server both client apps with one single mitmproxy instance (as they require shared features that are implemented as inline scripts already)?
I'm not sure if I get your question correctly. You can only run one proxy mode at the same time. If you need to mitmproxy instances, I would suggest to just run them on different ports.

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py
--- a/mitmproxy/proxy/protocol/http.py
+++ b/mitmproxy/proxy/protocol/http.py
@@ -217,16 +217,19 @@ def handle_regular_connect(self, f):
         return False
 
     def handle_upstream_connect(self, f):
-        self.establish_server_connection(
-            f.request.host,
-            f.request.port,
-            f.request.scheme
-        )
-        self.send_request(f.request)
-        f.response = self.read_response_headers()
-        f.response.data.content = b"".join(
-            self.read_response_body(f.request, f.response)
-        )
+        # if the user specifies a response in the http_connect hook, we do not connect upstream here.
+        # https://github.com/mitmproxy/mitmproxy/pull/2473
+        if not f.response:
+            self.establish_server_connection(
+                f.request.host,
+                f.request.port,
+                f.request.scheme
+            )
+            self.send_request(f.request)
+            f.response = self.read_response_headers()
+            f.response.data.content = b"".join(
+                self.read_response_body(f.request, f.response)
+            )
         self.send_response(f.response)
         if is_ok(f.response.status_code):
             layer = UpstreamConnectLayer(self, f.request)


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Unable to use upstream proxy and proxy authentification
##### Steps to reproduce the problem:

1. git clone https://github.com/mitmproxy/mitmproxy.git
2. cd mitmproxy
3. ./dev.sh 
4. . venv/bin/activate
5. mitmdump --proxyauth test:test -m upstream:http://10.7.0.225:8080/ --ssl-insecure

On 10.7.0.225, I have a VM that runs mitmproxy :
6. mitmdump --insecure

##### Any other comments? What have you tried so far?
It seems that each time you have a clientdisconnect/clientconnect, you get a new prompt to login (proxyauth). Basically for each domain you have to login to the proxyauth, here test/test. 
I wild guess would be that the information in the header to pass by the upstreamproxy somehow is not compatible with the header auth information 

##### System information
HOST : 
```
Mitmproxy version: 3.0.0 (2.0.0dev0435-0x1c6b33f) 
Python version: 3.6.0
Platform: Linux-4.9.0-0.bpo.2-amd64-x86_64-with-debian-8.7
SSL version: OpenSSL 1.1.0e  16 Feb 2017
Linux distro: debian 8.7 
```

VM (10.7.0.225):
```
Mitmproxy version : 2.0.0 (release version)
Python version : 3.5.2
Platform : Linux-4.4.0-75-generic-x86_64-with-Ubuntu-16.04-xenial
SSL version: OpenSSL 1.0.2g 1 Mar 2016
Linux distro: Ubuntu 16.04 xenial
```



Discussion:
Thanks for the report. [This](https://github.com/mitmproxy/mitmproxy/blob/69c5a0b6993f70340a0a4b99159086c638d77103/mitmproxy/addons/proxyauth.py#L64) is faulty, because the mode is now `upstream:...`, not just `upstream`.
I fixed it with just a minor change, show I create a pull request ?
```

-        return ctx.options.mode in ("regular", "upstream")
+        if ctx.options.mode == "regular" or "upstream" in ctx.options.mode:
+            return True
+        else:
+            return False
```
Go ahead please!

Resolving patch:
diff --git a/mitmproxy/addons/proxyauth.py b/mitmproxy/addons/proxyauth.py
--- a/mitmproxy/addons/proxyauth.py
+++ b/mitmproxy/addons/proxyauth.py
@@ -61,7 +61,7 @@ def is_proxy_auth(self) -> bool:
             - True, if authentication is done as if mitmproxy is a proxy
             - False, if authentication is done as if mitmproxy is a HTTP server
         """
-        return ctx.options.mode in ("regular", "upstream")
+        return ctx.options.mode == "regular" or ctx.options.mode.startswith("upstream:")
 
     def which_auth_header(self) -> str:
         if self.is_proxy_auth():


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Keyerror in compat 0.17 - peer_address
I get this error when loading an old 0.17 dump file:

```
Traceback (most recent call last):
  File "/usr/local/Cellar/mitmproxy/2.0.2/bin/mitmdump", line 11, in <module>
    load_entry_point('mitmproxy==2.0.2', 'console_scripts', 'mitmdump')()
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/tools/main.py", line 120, in mitmdump
    master = dump.DumpMaster(dump_options, server)
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/tools/dump.py", line 55, in __init__
    self.load_flows_file(options.rfile)
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/master.py", line 182, in load_flows_file
    return self.load_flows(freader)
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/master.py", line 164, in load_flows
    for i in fr.stream():
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/io.py", line 42, in stream
    data = io_compat.migrate_flow(data)
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/io_compat.py", line 152, in migrate_flow
    flow_data = converters[flow_version[:2]](flow_data)
  File "/usr/local/Cellar/mitmproxy/2.0.2/libexec/lib/python3.6/site-packages/mitmproxy/io_compat.py", line 61, in convert_017_018
    data["server_conn"]["ip_address"] = data["server_conn"].pop("peer_address")
KeyError: 'peer_address'
```

##### Steps to reproduce the problem:

1. Create an outfile with mitmproxy 0.17
2. Open with mitmdump 2.0.2
3. Get a keyerror on "peer_address"


##### Any other comments? What have you tried so far?

Was told on slack to create an issue, however I can't supply the test files because they contain sensitive data.


##### System information

$ /usr/local/Cellar/mitmproxy/2.0.2/bin/mitmdump --version
Mitmproxy version: 2.0.2 (release version)
Python version: 3.6.2
Platform: Darwin-16.7.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0f  25 May 2017
Mac version: 10.12.6 ('', '', '') x86_64



Discussion:
Commenting from my actual account to keep track of it - accidentally created the issue on the wrong account.
0.17 is old enough that we won't maintain this ourselves anymore, but I'd certainly accept a PR that fixes this, which should be easy.

Resolving patch:
diff --git a/mitmproxy/io/compat.py b/mitmproxy/io/compat.py
--- a/mitmproxy/io/compat.py
+++ b/mitmproxy/io/compat.py
@@ -58,7 +58,7 @@ def convert_017_018(data):
     # convert_unicode needs to be called for every dual release and the first py3-only release
     data = convert_unicode(data)
 
-    data["server_conn"]["ip_address"] = data["server_conn"].pop("peer_address")
+    data["server_conn"]["ip_address"] = data["server_conn"].pop("peer_address", None)
     data["marked"] = False
     data["version"] = (0, 18)
     return data


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
xss_scanner.py issue
hi
in the function test_query_injection the body is converted to lower case (line 153) but in the get_SQLi_data function it is compared with upper case words (line 187) that results in no match




##### System information



Discussion:
Thanks for the report! Would you mind sending a pull request that fixes this? 😃 
sure, also there are issuse with "decode" in log_SQLi_data function(line 176-179) that causes to error.
Im working on a new sqli example that also test post data for sqli and save vulnerable hosts to file.

Resolving patch:
diff --git a/examples/complex/xss_scanner.py b/examples/complex/xss_scanner.py
--- a/examples/complex/xss_scanner.py
+++ b/examples/complex/xss_scanner.py
@@ -198,7 +198,7 @@ def get_SQLi_data(new_body: str, original_body: str, request_URL: str, injection
     }
     for dbms, regexes in DBMS_ERRORS.items():
         for regex in regexes:
-            if re.search(regex, new_body) and not re.search(regex, original_body):
+            if re.search(regex, new_body, re.IGNORECASE) and not re.search(regex, original_body, re.IGNORECASE):
                 return SQLiData(request_URL,
                                 injection_point,
                                 regex,


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
MIT incompatible with GPL3
I was pleased to see mitmproxy is licensed under MIT. At least one dependency, html2text, is licensed under GPL3. This is a problem as per https://opensource.stackexchange.com/questions/1640/if-im-using-a-gpl-3-library-in-my-project-can-i-license-my-project-under-mit-l. Can we resolve this incompatible licensing issue? html2text is barely used. I think it could be removed.


Discussion:

Resolving patch:
diff --git a/mitmproxy/contentviews/__init__.py b/mitmproxy/contentviews/__init__.py
--- a/mitmproxy/contentviews/__init__.py
+++ b/mitmproxy/contentviews/__init__.py
@@ -22,7 +22,7 @@
 from mitmproxy.net import http
 from mitmproxy.utils import strutils
 from . import (
-    auto, raw, hex, json, xml_html, html_outline, wbxml, javascript, css,
+    auto, raw, hex, json, xml_html, wbxml, javascript, css,
     urlencoded, multipart, image, query, protobuf
 )
 from .base import View, VIEW_CUTOFF, KEY_MAX, format_text, format_dict, TViewResult
@@ -168,7 +168,6 @@ def get_content_view(viewmode: View, data: bytes, **metadata):
 add(json.ViewJSON())
 add(xml_html.ViewXmlHtml())
 add(wbxml.ViewWBXML())
-add(html_outline.ViewHTMLOutline())
 add(javascript.ViewJavaScript())
 add(css.ViewCSS())
 add(urlencoded.ViewURLEncoded())
diff --git a/mitmproxy/contentviews/html_outline.py b/mitmproxy/contentviews/html_outline.py
deleted file mode 100644
--- a/mitmproxy/contentviews/html_outline.py
+++ /dev/null
@@ -1,17 +0,0 @@
-import html2text
-
-from mitmproxy.contentviews import base
-
-
-class ViewHTMLOutline(base.View):
-    name = "HTML Outline"
-    prompt = ("html outline", "o")
-    content_types = ["text/html"]
-
-    def __call__(self, data, **metadata):
-        data = data.decode("utf-8", "replace")
-        h = html2text.HTML2Text(baseurl="")
-        h.ignore_images = True
-        h.body_width = 0
-        outline = h.handle(data)
-        return "HTML Outline", base.format_text(outline)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -66,7 +66,6 @@
         "click>=6.2, <7",
         "cryptography>=2.0,<2.1",
         "h2>=3.0, <4",
-        "html2text>=2016.1.8, <=2016.9.19",
         "hyperframe>=5.0, <6",
         "kaitaistruct>=0.7, <0.8",
         "ldap3>=2.2.0, <2.4",


Test patch:
diff --git a/test/mitmproxy/contentviews/test_html_outline.py b/test/mitmproxy/contentviews/test_html_outline.py
deleted file mode 100644
--- a/test/mitmproxy/contentviews/test_html_outline.py
+++ /dev/null
@@ -1,9 +0,0 @@
-from mitmproxy.contentviews import html_outline
-from test.mitmproxy.contentviews import full_eval
-
-
-def test_view_html_outline():
-    v = full_eval(html_outline.ViewHTMLOutline())
-    s = b"<html><br><br></br><p>one</p></html>"
-    assert v(s)
-    assert v(b'\xfe')
Repository: mitmproxy/mitmproxy

Issue:
[requires.io] dependency update on master branch



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -85,7 +85,7 @@
             "pydivert>=2.0.3,<2.2",
         ],
         'dev': [
-            "flake8>=3.2.1, <3.5",
+            "flake8>=3.5, <3.6",
             "Flask>=0.10.1, <0.13",
             "mypy>=0.530,<0.541",
             "pytest-cov>=2.2.1, <3",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump does not apply filter to saved data
##### Steps to reproduce the problem:

1. I captured some traffic, and ran the following to filter it:

```
$ mitmdump -r traffic.mitm -w out.mitm '~u main.css'
Proxy server listening at http://[::]:8080
172.16.122.1:51049: GET https://www.sjoerdlangkemper.nl/css/main.css
                 << 304 Not Modified 0b
$
```

It displays only the matched URL, but it saves all traffic. When done, out.mitm contains the same requests and responses as traffic.mitm. I.e. `mitmproxy -r out.mitm` shows a lot of requests, where I would expect only the request for main.css.


##### Any other comments? What have you tried so far?

I tried this with release 2.0.2, and there it worked as expected. This issue seems to be similar to #1089.

##### System information

```
$ mitmdump --version
Mitmproxy version: 3.0.0 (2.0.0dev0965-0x168c72a) 
Python version: 3.5.2
Platform: Linux-4.4.0-98-generic-x86_64-with-Ubuntu-16.04-xenial
SSL version: OpenSSL 1.1.0f  25 May 2017
Linux distro: Ubuntu 16.04 xenial
```



Discussion:

Resolving patch:
diff --git a/mitmproxy/addons/save.py b/mitmproxy/addons/save.py
--- a/mitmproxy/addons/save.py
+++ b/mitmproxy/addons/save.py
@@ -42,7 +42,7 @@ def configure(self, updated):
                     )
             else:
                 self.filt = None
-        if "save_stream_file" in updated:
+        if "save_stream_file" in updated or "save_stream_filter" in updated:
             if self.stream:
                 self.done()
             if ctx.options.save_stream_file:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Bug in expected_http_body_size
##### Steps to reproduce the problem:

1. Any response with more than one content-length header


##### Any other comments? What have you tried so far?

headers is MultiDict so this:
size = int(headers["content-length"])
can be change to:
size = int(headers.get_all("content-length")[0])


Discussion:
Thanks. Would you like to file a PR that fixes that? We should use `get_all` and verify that all returned values are equal as per https://tools.ietf.org/html/rfc7230#section-3.3.2. Maybe like that:
```python
# Handle multiple content-length headers as per https://tools.ietf.org/html/rfc7230#section-3.3.2
sizes = headers.get_all("content-length")
different_content_length_headers = any(x != sizes[0] for x in sizes)
if different_content_length_headers:
    raise exceptions.HttpSyntaxException("Conflicting Content Length Headers")
size = int(sizes[0])
```
Yes, i'll do it today.

Resolving patch:
diff --git a/mitmproxy/net/http/http1/read.py b/mitmproxy/net/http/http1/read.py
--- a/mitmproxy/net/http/http1/read.py
+++ b/mitmproxy/net/http/http1/read.py
@@ -210,7 +210,11 @@ def expected_http_body_size(request, response=None):
         return None
     if "content-length" in headers:
         try:
-            size = int(headers["content-length"])
+            sizes = headers.get_all("content-length")
+            different_content_length_headers = any(x != sizes[0] for x in sizes)
+            if different_content_length_headers:
+                raise exceptions.HttpSyntaxException("Conflicting Content Length Headers")
+            size = int(sizes[0])
             if size < 0:
                 raise ValueError()
             return size


Test patch:
diff --git a/test/mitmproxy/net/http/http1/test_read.py b/test/mitmproxy/net/http/http1/test_read.py
--- a/test/mitmproxy/net/http/http1/test_read.py
+++ b/test/mitmproxy/net/http/http1/test_read.py
@@ -194,6 +194,17 @@ def test_expected_http_body_size():
         treq(headers=Headers(content_length="42"))
     ) == 42
 
+    # more than 1 content-length headers with same value
+    assert expected_http_body_size(
+        treq(headers=Headers([(b'content-length', b'42'), (b'content-length', b'42')]))
+    ) == 42
+
+    # more than 1 content-length headers with conflicting value
+    with pytest.raises(exceptions.HttpSyntaxException):
+        expected_http_body_size(
+            treq(headers=Headers([(b'content-length', b'42'), (b'content-length', b'45')]))
+        )
+
     # no length
     assert expected_http_body_size(
         treq(headers=Headers())
Repository: mitmproxy/mitmproxy

Issue:
Transparent mode fail with looking up failure.
##### Steps to reproduce the problem:

1. Launch Wifi Access Point(OS X)
2. Setup pfctl configuration so that http packet will be forwarded.
3. Launch mitmproxy ( `sudo mitmproxy -p 8080 -m transparent --showhost` )
4. Access web page after connecting to AP which launched before.
5. See event log.


##### Any other comments? What have you tried so far?

When I tried to use transparent mode with OS X(10.11.6).
RuntimeError("Could not resolve original destination.") raised.

I investigated this bug.
And I found that this is caused by difference between AF_INET's and AF_INET6's peername.
https://github.com/mitmproxy/mitmproxy/blob/de006ea8adc08b9a8a6aa94eda2b30468727c307/mitmproxy/net/tcp.py#L567

If we use AF_INET, getpeername() return string like `"192.168.2.5:45670"`.
But if we use AF_INET6, getpeername() return string like `"::ffff:192.168.2.5:45670"`.

`pfctl -s state` 's result is like below.
`ALL tcp 192.168.2.5:45670 -> xx.xx.xx.xx:33291 -> xx.xx.xx.xx:443       ESTABLISHED:ESTABLISHED`

As you see, `::ffff:` doesn't exist.

So [lookup](https://github.com/mitmproxy/mitmproxy/blob/f17c0fdac636f7269f4885294e2a8d2c52c23590/mitmproxy/platform/pf.py#L4) function raises RuntimeError() because `spec in i` condition won't become true.

##### System information

Mitmproxy version: 3.0.0 (release version)
Python version: 3.6.2
Platform: Darwin-15.6.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2l  25 May 2017
Mac version: 10.11.6 ('', '', '') x86_64


Discussion:
Thanks for the awesome report! :smiley: 🍰

Resolving patch:
diff --git a/mitmproxy/platform/pf.py b/mitmproxy/platform/pf.py
--- a/mitmproxy/platform/pf.py
+++ b/mitmproxy/platform/pf.py
@@ -1,3 +1,4 @@
+import re
 import sys
 
 
@@ -8,6 +9,9 @@ def lookup(address, port, s):
 
         Returns an (address, port) tuple, or None.
     """
+    # We may get an ipv4-mapped ipv6 address here, e.g. ::ffff:127.0.0.1.
+    # Those still appear as "127.0.0.1" in the table, so we need to strip the prefix.
+    address = re.sub("^::ffff:(?=\d+.\d+.\d+.\d+$)", "", address)
     s = s.decode()
     spec = "%s:%s" % (address, port)
     for i in s.split("\n"):


Test patch:
diff --git a/test/mitmproxy/platform/test_pf.py b/test/mitmproxy/platform/test_pf.py
--- a/test/mitmproxy/platform/test_pf.py
+++ b/test/mitmproxy/platform/test_pf.py
@@ -15,6 +15,7 @@ def test_simple(self):
             d = f.read()
 
         assert pf.lookup("192.168.1.111", 40000, d) == ("5.5.5.5", 80)
+        assert pf.lookup("::ffff:192.168.1.111", 40000, d) == ("5.5.5.5", 80)
         with pytest.raises(Exception, match="Could not resolve original destination"):
             pf.lookup("192.168.1.112", 40000, d)
         with pytest.raises(Exception, match="Could not resolve original destination"):
Repository: mitmproxy/mitmproxy

Issue:
command console.choose.cmd crashes when given no arguments
Enter mitmproxy console, issue a command like:

    console.choose.cmd

... and press return. 


Discussion:

Resolving patch:
diff --git a/mitmproxy/command.py b/mitmproxy/command.py
--- a/mitmproxy/command.py
+++ b/mitmproxy/command.py
@@ -190,10 +190,19 @@ def parsearg(manager: CommandManager, spec: str, argtype: type) -> typing.Any:
         raise exceptions.CommandError("Unsupported argument type: %s" % argtype)
 
 
+def verify_arg_signature(f: typing.Callable, args: list, kwargs: dict) -> None:
+    sig = inspect.signature(f)
+    try:
+        sig.bind(*args, **kwargs)
+    except TypeError as v:
+        raise exceptions.CommandError("Argument mismatch: %s" % v.args[0])
+
+
 def command(path):
     def decorator(function):
         @functools.wraps(function)
         def wrapper(*args, **kwargs):
+            verify_arg_signature(function, args, kwargs)
             return function(*args, **kwargs)
         wrapper.__dict__["command_path"] = path
         return wrapper


Test patch:
diff --git a/test/mitmproxy/test_command.py b/test/mitmproxy/test_command.py
--- a/test/mitmproxy/test_command.py
+++ b/test/mitmproxy/test_command.py
@@ -163,3 +163,10 @@ def test_decorator():
     with taddons.context() as tctx:
         tctx.master.addons.add(a)
         assert tctx.master.commands.call("cmd1 bar") == "ret bar"
+
+
+def test_verify_arg_signature():
+    with pytest.raises(exceptions.CommandError):
+        command.verify_arg_signature(lambda: None, [1, 2], {})
+        print('hello there')
+    command.verify_arg_signature(lambda a, b: None, [1, 2], {})
\ No newline at end of file
Repository: mitmproxy/mitmproxy

Issue:
command "console.bodyview" crashes when given an invalid argument
Triggered by entering a command like:

    console.bodyview @focus foo
command "console.bodyview" crashes when given an invalid argument
Triggered by entering a command like:

    console.bodyview @focus foo


Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/console/consoleaddons.py b/mitmproxy/tools/console/consoleaddons.py
--- a/mitmproxy/tools/console/consoleaddons.py
+++ b/mitmproxy/tools/console/consoleaddons.py
@@ -329,9 +329,9 @@ def bodyview(self, f: flow.Flow, part: str) -> None:
             correct viewier, and fall back to the programs in $PAGER or $EDITOR
             if necessary.
         """
-        fpart = getattr(f, part)
+        fpart = getattr(f, part, None)
         if not fpart:
-            raise exceptions.CommandError("Could not view part %s." % part)
+            raise exceptions.CommandError("Part must be either request or response, not %s." % part)
         t = fpart.headers.get("content-type")
         content = fpart.get_content(strict=False)
         if not content:


Test patch: