internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
[tests actually pass] Remove harextractor example
If nobody objects, this PR (temporarily) removes the harextractor example, which depends on harparser, a Python2-only library. It's probably worth the time to rewrite this without any dependencies, but first we want to get a Python 3 release out of the door.



Discussion:

Resolving patch:
diff --git a/examples/har_extractor.py b/examples/har_extractor.py
--- a/examples/har_extractor.py
+++ b/examples/har_extractor.py
@@ -60,6 +60,12 @@ def start(context):
         suit your actual needs of HAR generation. As it will probably be
         necessary to cluster logs by IPs or reset them from time to time.
     """
+    if sys.version_info >= (3, 0):
+        raise RuntimeError(
+            "har_extractor.py does not work on Python 3. "
+            "Please check out https://github.com/mitmproxy/mitmproxy/issues/1320 "
+            "if you want to help making this work again."
+        )
     context.dump_file = None
     if len(sys.argv) > 1:
         context.dump_file = sys.argv[1]


Test patch:
diff --git a/test/mitmproxy/test_examples.py b/test/mitmproxy/test_examples.py
--- a/test/mitmproxy/test_examples.py
+++ b/test/mitmproxy/test_examples.py
@@ -1,6 +1,7 @@
 import glob
 import json
 import os
+import sys
 from contextlib import contextmanager
 
 from mitmproxy import script
@@ -133,6 +134,12 @@ def test_redirect_requests():
 
 
 def test_har_extractor():
+    if sys.version_info >= (3, 0):
+        with tutils.raises("does not work on Python 3"):
+            with example("har_extractor.py -"):
+                pass
+        return
+
     with tutils.raises(script.ScriptException):
         with example("har_extractor.py"):
             pass
Repository: mitmproxy/mitmproxy

Issue:
Snapshots have weird filename due to py3
##### Steps to reproduce the problem:
1. Look at https://snapshots.mitmproxy.org/v0.18/

```
mitmproxy-0.18dev0636-0xg588dad1-py2-none-any.whl  26-Jun-2016 21:55    985K
mitmproxy-0.18dev0759-0xg22c0db3-win32.zip         09-Jul-2016 20:45     35M
mitmproxy-0.18dev0775-0xb'g6bb267c'-osx.tar.gz     10-Jul-2016 09:29     35M
mitmproxy-0.18dev0775-0xb'g6bb267c'-py2.py3-non..> 10-Jul-2016 09:28    986K
pathod-0.18dev0759-0xg22c0db3-win32.zip            09-Jul-2016 20:46     12M
pathod-0.18dev0775-0xb'g6bb267c'-osx.tar.gz        10-Jul-2016 09:28     13M
```
##### What is the expected behavior?

Have proper filename.
##### What went wrong?

Some files contain e.g. `0xb'g6bb267c'`, others contain `0xg22c0db3`. The `0x` part is ok, but the other thing looks like a bytes vs. str problem introduced by creating snapshots with Python 3.



Discussion:

Resolving patch:
diff --git a/release/rtool.py b/release/rtool.py
--- a/release/rtool.py
+++ b/release/rtool.py
@@ -76,7 +76,7 @@ def get_snapshot_version():
         return "{version}dev{tag_dist:04}-0x{commit}".format(
             version=get_version(),  # this should already be the next version
             tag_dist=tag_dist,
-            commit=commit
+            commit=commit.decode()
         )
 
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Error when killing HTTP/2 Flow
##### Steps to reproduce the problem:
1. `mitmproxy`
2. Add catch-all interception filter
3. Visit HTTP/2 URL
4. Press `X`
##### What is the expected behavior?

Flow should be killed.
##### What went wrong?

![image](https://cloud.githubusercontent.com/assets/1019198/17043891/eeb34af8-4f6f-11e6-8169-3f2f26e67ed7.png)
##### Any other comments? What have you tried so far?

When fixing this, let's make sure that killing responses works as well.

---

Mitmproxy Version: master
Operating System: Ubuntu 14.04 x64



Discussion:
Duplicate of #1193 and #1284 - but finally we know an easy way to reproduce it. 


Resolving patch:
diff --git a/mitmproxy/protocol/http2.py b/mitmproxy/protocol/http2.py
--- a/mitmproxy/protocol/http2.py
+++ b/mitmproxy/protocol/http2.py
@@ -584,6 +584,8 @@ def run(self):
         except exceptions.ProtocolException as e:  # pragma: no cover
             self.log(repr(e), "info")
             self.log(traceback.format_exc(), "debug")
+        except exceptions.Kill:
+            self.log("Connection killed", "info")
 
         if not self.zombie:
             self.zombie = time.time()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Attempting to delete a flow that has finished crashes mitmproxy
I often hit `d` as a way to navigate downward, a behavior instilled by `less` and my browser's navigation. However, when I do so I get `ControlException: Message already acked` and mitmproxy shuts down. Don't feel like I should be punished so badly. `d` should let me know less forcefully.

Mitmproxy Version: Fresh from the press.

---

I'll get around to looking into this.



Discussion:

Resolving patch:
diff --git a/mitmproxy/console/flowlist.py b/mitmproxy/console/flowlist.py
--- a/mitmproxy/console/flowlist.py
+++ b/mitmproxy/console/flowlist.py
@@ -188,7 +188,8 @@ def keypress(self, xxx_todo_changeme, key):
             self.flow.accept_intercept(self.master)
             signals.flowlist_change.send(self)
         elif key == "d":
-            self.flow.kill(self.master)
+            if not self.flow.reply.acked:
+                self.flow.kill(self.master)
             self.state.delete_flow(self.flow)
             signals.flowlist_change.send(self)
         elif key == "D":
@@ -255,7 +256,8 @@ def keypress(self, xxx_todo_changeme, key):
                 callback = self.save_flows_prompt,
             )
         elif key == "X":
-            self.flow.kill(self.master)
+            if not self.flow.reply.acked:
+                self.flow.kill(self.master)
         elif key == "enter":
             if self.flow.request:
                 self.master.view_flow(self.flow)
diff --git a/mitmproxy/console/flowview.py b/mitmproxy/console/flowview.py
--- a/mitmproxy/console/flowview.py
+++ b/mitmproxy/console/flowview.py
@@ -569,7 +569,8 @@ def keypress(self, size, key):
             else:
                 self.view_next_flow(self.flow)
             f = self.flow
-            f.kill(self.master)
+            if not f.reply.acked:
+                f.kill(self.master)
             self.state.delete_flow(f)
         elif key == "D":
             f = self.master.duplicate_flow(self.flow)
diff --git a/mitmproxy/web/app.py b/mitmproxy/web/app.py
--- a/mitmproxy/web/app.py
+++ b/mitmproxy/web/app.py
@@ -230,7 +230,8 @@ def post(self, flow_id):
 class FlowHandler(RequestHandler):
 
     def delete(self, flow_id):
-        self.flow.kill(self.master)
+        if not self.flow.reply.acked:
+            self.flow.kill(self.master)
         self.state.delete_flow(self.flow)
 
     def put(self, flow_id):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
[PY3] Search does not work
##### Steps to reproduce the problem:
1. View a flow
2. Press `/`
3. 
##### What is the expected behavior?

No crash.
##### What went wrong?

```
Traceback (most recent call last):
  File "/media/sf_git/mitmproxy/mitmproxy/console/master.py", line 498, in run
    self.loop.run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 376, in _run
    self.event_loop.run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 682, in run
    self._loop()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 719, in _loop
    self._watch_files[fd]()
  File "/usr/local/lib/python3.5/dist-packages/urwid/raw_display.py", line 393, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/usr/local/lib/python3.5/dist-packages/urwid/raw_display.py", line 493, in parse_input
    callback(processed, processed_codes)
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 403, in _update
    self.process_input(keys)
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 503, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/media/sf_git/mitmproxy/mitmproxy/console/window.py", line 42, in keypress
    k = super(self.__class__, self).keypress(size, k)
  File "/usr/local/lib/python3.5/dist-packages/urwid/container.py", line 1116, in keypress
    return self.footer.keypress((maxcol,),key)
  File "/media/sf_git/mitmproxy/mitmproxy/console/statusbar.py", line 134, in keypress
    return self.master.ab.keypress(*args, **kwargs)
  File "/media/sf_git/mitmproxy/mitmproxy/console/statusbar.py", line 89, in keypress
    self.prompt_execute(self._w.get_edit_text())
  File "/media/sf_git/mitmproxy/mitmproxy/console/statusbar.py", line 112, in prompt_execute
    msg = p(txt, *args)
  File "/media/sf_git/mitmproxy/mitmproxy/console/searchable.py", line 52, in set_search
    self.find_next(False)
  File "/media/sf_git/mitmproxy/mitmproxy/console/searchable.py", line 83, in find_next
    rng = xrange(1, len(self.body) + 1)
NameError: name 'xrange' is not defined
```

---

Mitmproxy Version: master
Operating System: Ubuntu 14.04 x64



Discussion:

Resolving patch:
diff --git a/mitmproxy/console/searchable.py b/mitmproxy/console/searchable.py
--- a/mitmproxy/console/searchable.py
+++ b/mitmproxy/console/searchable.py
@@ -78,9 +78,9 @@ def find_next(self, backwards):
                 return
         # Start search at focus + 1
         if backwards:
-            rng = xrange(len(self.body) - 1, -1, -1)
+            rng = range(len(self.body) - 1, -1, -1)
         else:
-            rng = xrange(1, len(self.body) + 1)
+            rng = range(1, len(self.body) + 1)
         for i in rng:
             off = (self.focus_position + i) % len(self.body)
             w = self.body[off]


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Cannot clear flows
##### Steps to reproduce the problem:
1. Start mitmproxy
2. Press z
##### What is the expected behavior?

No crash
##### What went wrong?

```
Traceback (most recent call last):
  File "/media/sf_git/mitmproxy/mitmproxy/console/master.py", line 515, in run
    self.loop.run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 376, in _run
    self.event_loop.run()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 682, in run
    self._loop()
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 719, in _loop
    self._watch_files[fd]()
  File "/usr/local/lib/python3.5/dist-packages/urwid/raw_display.py", line 393, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/usr/local/lib/python3.5/dist-packages/urwid/raw_display.py", line 493, in parse_input
    callback(processed, processed_codes)
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 403, in _update
    self.process_input(keys)
  File "/usr/local/lib/python3.5/dist-packages/urwid/main_loop.py", line 503, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/media/sf_git/mitmproxy/mitmproxy/console/window.py", line 42, in keypress
    k = super(self.__class__, self).keypress(size, k)
  File "/usr/local/lib/python3.5/dist-packages/urwid/container.py", line 1128, in keypress
    return self.body.keypress( (maxcol, remaining), key )
  File "/media/sf_git/mitmproxy/mitmproxy/console/flowlist.py", line 361, in keypress
    self.master.clear_flows()
  File "/media/sf_git/mitmproxy/mitmproxy/console/master.py", line 686, in clear_flows
    self.state.clear()
  File "/media/sf_git/mitmproxy/mitmproxy/console/master.py", line 185, in clear
    marked_flows = [f for f in self.state.view if f.marked]
AttributeError: 'ConsoleState' object has no attribute 'state'
```

@dufferzafar, can you fix this? :smiley:

Mitmproxy Version: master
Operating System: Ubuntu 14.04 x64



Discussion:

Resolving patch:
diff --git a/mitmproxy/console/master.py b/mitmproxy/console/master.py
--- a/mitmproxy/console/master.py
+++ b/mitmproxy/console/master.py
@@ -182,7 +182,7 @@ def disable_marked_filter(self):
         self.mark_filter = False
 
     def clear(self):
-        marked_flows = [f for f in self.state.view if f.marked]
+        marked_flows = [f for f in self.view if f.marked]
         super(ConsoleState, self).clear()
 
         for f in marked_flows:


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Scripts: "done" hook is not executed.
##### Steps to reproduce the problem:
1. `touch foo`
2. `mitmdump -r foo -s examples\stub.py`
##### What is the expected behavior?

```
start
configure
done
```
##### What went wrong?

```
start
configure
```
##### Any other comments? What have you tried so far?

Somewhat related: Rename it to `stop` ?

---

Mitmproxy Version: master
Operating System: Win10 x64



Discussion:

Resolving patch:
diff --git a/mitmproxy/dump.py b/mitmproxy/dump.py
--- a/mitmproxy/dump.py
+++ b/mitmproxy/dump.py
@@ -118,5 +118,6 @@ def request(self, f):
 
     def run(self):  # pragma: no cover
         if self.options.rfile and not self.options.keepserving:
+            self.addons.done()
             return
         super(DumpMaster, self).run()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Status bar state confusion
##### Steps to reproduce the problem:
1. `mitmproxy`
2. `n g enter e`
##### What is the expected behavior?

Event log should show.
##### What went wrong?

I'm also getting a `Quit (yes, no)?` prompt.

---

Mitmproxy Version: master
Operating System: Ubuntu 14.04 x64



Discussion:
You don't even have to press `n g enter`. I see the Quit prompt when I press `e`.

And when I press `n` on the Quit prompt - I get into the `Create new request` prompt.


Resolving patch:
diff --git a/mitmproxy/console/master.py b/mitmproxy/console/master.py
--- a/mitmproxy/console/master.py
+++ b/mitmproxy/console/master.py
@@ -257,6 +257,7 @@ def __init__(self, server, options):
 
         signals.call_in.connect(self.sig_call_in)
         signals.pop_view_state.connect(self.sig_pop_view_state)
+        signals.replace_view_state.connect(self.sig_replace_view_state)
         signals.push_view_state.connect(self.sig_push_view_state)
         signals.sig_add_log.connect(self.sig_add_log)
         self.addons.add(options, *builtins.default_addons())
@@ -295,7 +296,19 @@ def cb(*_):
             return callback(*args)
         self.loop.set_alarm_in(seconds, cb)
 
+    def sig_replace_view_state(self, sender):
+        """
+            A view has been pushed onto the stack, and is intended to replace
+            the current view rather tha creating a new stack entry.
+        """
+        if len(self.view_stack) > 1:
+            del self.view_stack[1]
+
     def sig_pop_view_state(self, sender):
+        """
+            Pop the top view off the view stack. If no more views will be left
+            after this, prompt for exit.
+        """
         if len(self.view_stack) > 1:
             self.view_stack.pop()
             self.loop.widget = self.view_stack[-1]
@@ -311,6 +324,9 @@ def sig_pop_view_state(self, sender):
             )
 
     def sig_push_view_state(self, sender, window):
+        """
+            Push a new view onto the view stack.
+        """
         self.view_stack.append(window)
         self.loop.widget = window
         self.loop.draw_screen()
@@ -351,8 +367,8 @@ def run_script_once(self, command, f):
 
     def toggle_eventlog(self):
         self.options.eventlog = not self.options.eventlog
-        signals.pop_view_state.send(self)
         self.view_flowlist()
+        signals.replace_view_state.send(self)
 
     def _readflows(self, path):
         """
diff --git a/mitmproxy/console/signals.py b/mitmproxy/console/signals.py
--- a/mitmproxy/console/signals.py
+++ b/mitmproxy/console/signals.py
@@ -43,3 +43,4 @@ def add_log(e, level):
 # Pop and push view state onto a stack
 pop_view_state = blinker.Signal()
 push_view_state = blinker.Signal()
+replace_view_state = blinker.Signal()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Make `.*` match the whole body for replacements.
See https://discourse.mitmproxy.org/t/emulate-charles-proxys-map-local/116 for context - specifying `.*` as a replacement pattern currently replaces every single line in the message body. It feels more natural if that would replace the whole body altogether. We can accomplish this behaviour by passing `re.DOTALL` as a flag for `FBod/FBodRequest/FBodResponse` in `mitmproxy/filt.py`.

This issue should be pretty straightforward to implement, so I'll leave this for newcomers who are interested in contributing to mitmproxy. Feel free to ask here or on Slack if you have any further questions! :relaxed: 



Discussion:

Resolving patch:
diff --git a/mitmproxy/builtins/replace.py b/mitmproxy/builtins/replace.py
--- a/mitmproxy/builtins/replace.py
+++ b/mitmproxy/builtins/replace.py
@@ -36,9 +36,9 @@ def execute(self, f):
         for rex, s, cpatt in self.lst:
             if cpatt(f):
                 if f.response:
-                    f.response.replace(rex, s)
+                    f.response.replace(rex, s, flags=re.DOTALL)
                 else:
-                    f.request.replace(rex, s)
+                    f.request.replace(rex, s, flags=re.DOTALL)
 
     def request(self, flow):
         if not flow.reply.has_message:
diff --git a/mitmproxy/filt.py b/mitmproxy/filt.py
--- a/mitmproxy/filt.py
+++ b/mitmproxy/filt.py
@@ -244,6 +244,7 @@ def __call__(self, f):
 class FBod(_Rex):
     code = "b"
     help = "Body"
+    flags = re.DOTALL
 
     @only(HTTPFlow, TCPFlow)
     def __call__(self, f):
@@ -264,6 +265,7 @@ def __call__(self, f):
 class FBodRequest(_Rex):
     code = "bq"
     help = "Request body"
+    flags = re.DOTALL
 
     @only(HTTPFlow, TCPFlow)
     def __call__(self, f):
@@ -280,6 +282,7 @@ def __call__(self, f):
 class FBodResponse(_Rex):
     code = "bs"
     help = "Response body"
+    flags = re.DOTALL
 
     @only(HTTPFlow, TCPFlow)
     def __call__(self, f):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Divide by Zero error
Its in `netlib/strutils.py`

This line # around 126 :

``` python
    for i in six.iterbytes(s[:100])
) / len(s[:100]) > 0.3
```

if s is empty, it gives this error in the mitmproxy, (doesn't crash though due to recent improvements in mitmproxy i guess..)



Discussion:
Thanks. This looks easy to fix, so if anyone wants to get started on mitmproxy development, I'd be happy to see a PR with fix + added test in test/netlib/test_strutils.py! :smiley: 


Resolving patch:
diff --git a/netlib/strutils.py b/netlib/strutils.py
--- a/netlib/strutils.py
+++ b/netlib/strutils.py
@@ -121,6 +121,9 @@ def escaped_str_to_bytes(data):
 
 def is_mostly_bin(s):
     # type: (bytes) -> bool
+    if not s or len(s) == 0:
+        return False
+
     return sum(
         i < 9 or 13 < i < 32 or 126 < i
         for i in six.iterbytes(s[:100])


Test patch:
diff --git a/test/netlib/test_strutils.py b/test/netlib/test_strutils.py
--- a/test/netlib/test_strutils.py
+++ b/test/netlib/test_strutils.py
@@ -85,6 +85,7 @@ def test_escaped_str_to_bytes():
 def test_is_mostly_bin():
     assert not strutils.is_mostly_bin(b"foo\xFF")
     assert strutils.is_mostly_bin(b"foo" + b"\xFF" * 10)
+    assert not strutils.is_mostly_bin("")
 
 
 def test_is_xml():
Repository: mitmproxy/mitmproxy

Issue:
Wait for new cryptography release
For the next mitmproxy release, we want to recommend Python 3.5 by default. This requires a cryptography release which includes https://github.com/pyca/cryptography/pull/3063.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -66,7 +66,7 @@
         "certifi>=2015.11.20.1",  # no semver here - this should always be on the last release!
         "configargparse>=0.10, <0.11",
         "construct>=2.5.2, <2.6",
-        "cryptography>=1.3, <1.5",
+        "cryptography>=1.3, <1.6",
         "cssutils>=1.0.1, <1.1",
         "Flask>=0.10.1, <0.12",
         "h2>=2.4.1, <3",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
mitmdump should display "ready" message after start.
##### Steps to reproduce the problem:
1. `mitmdump`
##### What is the expected behavior?

mitmdump should print some notification that it is running and ready to receive traffic.
##### What went wrong?

mitmdump just displays nothing, it looks like it hangs.
##### Any other comments? What have you tried so far?

Ideally we'd only show that if mitmdump is invoked in a tty?

---

Mitmproxy Version: 0.18
Operating System: Win10 x64



Discussion:
ref: #302


Resolving patch:
diff --git a/mitmproxy/dump.py b/mitmproxy/dump.py
--- a/mitmproxy/dump.py
+++ b/mitmproxy/dump.py
@@ -48,6 +48,10 @@ def __init__(self, server, options):
         self.options = self.options  # type: Options
         self.set_stream_large_bodies(options.stream_large_bodies)
 
+        print("Proxy server listening at http://%s:%d" % (
+            (options.listen_host or "0.0.0.0"),
+            options.listen_port))
+
         if self.server and self.options.http2 and not tcp.HAS_ALPN:  # pragma: no cover
             print("ALPN support missing (OpenSSL 1.0.2+ required)!\n"
                   "HTTP/2 is disabled. Use --no-http2 to silence this warning.",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Add API to programatically create new requests.
Original discussion: https://discourse.mitmproxy.org/t/are-there-any-script-examples-for-creating-a-get-post-put/134

It would be great if we'd have a dedicated API to make new requests from scripts, e.g. `master.new_request()` or something along those lines. This would probably just call `new_request` and then `replay_request`, but this seems useful enough to spell out explicitly.



Discussion:

Resolving patch:
diff --git a/mitmproxy/flow/master.py b/mitmproxy/flow/master.py
--- a/mitmproxy/flow/master.py
+++ b/mitmproxy/flow/master.py
@@ -187,6 +187,10 @@ def replay_request(self, f, block=False):
             rt.join()
         return rt
 
+    def new_request(self, method, scheme, host, port, path, block=False):
+        f = self.create_request(method, scheme, host, port, path)
+        return self.replay_request(f, block)
+
     @controller.handler
     def log(self, l):
         self.add_log(l.msg, l.level)


Test patch:
diff --git a/test/mitmproxy/test_flow.py b/test/mitmproxy/test_flow.py
--- a/test/mitmproxy/test_flow.py
+++ b/test/mitmproxy/test_flow.py
@@ -463,6 +463,15 @@ def test_duplicate_flow(self):
         assert s.flow_count() == 2
         assert s.index(f2) == 1
 
+    def test_new_request(self):
+        s = flow.State()
+        fm = flow.FlowMaster(
+            options.Options(),
+            DummyServer(ProxyConfig(options.Options())),
+            s
+        )
+        assert fm.new_request("GET", "http", "example.com", 80, "/")
+
     def test_create_flow(self):
         s = flow.State()
         fm = flow.FlowMaster(None, None, s)