internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
New Table UI has no replay indicator
#### Problem Description

The new table UI doesn't show that a flow has been replayed:

![image](https://user-images.githubusercontent.com/1019198/68954563-90a5f080-07c4-11ea-9d73-3abf9ce674b3.png)

The two-line UI still has an indicator - `[r]` in my case as my terminal is detected to not support UTF8:

![image](https://user-images.githubusercontent.com/1019198/68954680-cb0f8d80-07c4-11ea-91d0-af110277d143.png)


#### Steps to reproduce the behavior:
1. Add a flow, e.g. by pressing `n` `enter`
2. Press `r` to replay.

#### System Information
Mitmproxy: 5.0.0.dev (+356, commit a5412ab)
Python:    3.8.0
OpenSSL:   OpenSSL 1.1.1  11 Sep 2018
Platform:  Linux-4.4.0-18362-Microsoft-x86_64-with-glibc2.27


Discussion:
@Jessonsotoventura: You are the expert here - would you mind taking a look? 😊 
Yes - I'll take a look at this issue
The issue seems to be here:

<https://github.com/mitmproxy/mitmproxy/blob/ac22aee2f512cefd298fe0d9563a5de0ef7e42ee/mitmproxy/tools/console/common.py#L371-L380>

2 things that caught my attention:

The line ` if f.get('resp_is_replay', False):` should probably be ` if f.get('req_is_replay', False):`

And even if that is true, the last if there `if f['focus']:` overwrites the cursor with a `>` if there is only 1 flow in the table (after all you will be focusing it).

The replay symbol should be drawn on the second column, not the first.

Hopefully this is helpful

Resolving patch:
diff --git a/mitmproxy/tools/console/common.py b/mitmproxy/tools/console/common.py
--- a/mitmproxy/tools/console/common.py
+++ b/mitmproxy/tools/console/common.py
@@ -369,18 +369,15 @@ def raw_format_table(f):
     req = []
 
     cursor = [' ', 'focus']
-    if f.get('resp_is_replay', False):
-        cursor[0] = SYMBOL_REPLAY
-        cursor[1] = 'replay'
-    if f['marked']:
-        if cursor[0] == ' ':
-            cursor[0] = SYMBOL_MARK
-        cursor[1] = 'mark'
     if f['focus']:
         cursor[0] = '>'
-
     req.append(fcol(*cursor))
 
+    if f.get('resp_is_replay', False) or f.get('req_is_replay', False):
+        req.append(fcol(SYMBOL_REPLAY, 'replay'))
+    if f['marked']:
+        req.append(fcol(SYMBOL_MARK, 'mark'))
+
     if f["two_line"]:
         req.append(TruncatedText(f["req_url"], colorize_url(f["req_url"]), 'left'))
         pile.append(urwid.Columns(req, dividechars=1))


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
AttributeError: 'tuple' object has no attribute 'find'
#### Problem Description
I tried to run successfully but when I hit ESC, below error displayed.

#### Steps to reproduce the behavior:
1. Run `$ mitmproxy -p 9999 --mode socks5 --showhost`
```
Traceback (most recent call last):
  File "/Users/binhho/Documents/t-mobile/mitmproxy/mitmproxy/master.py", line 86, in run_loop
    loop()
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/main_loop.py", line 286, in run
    self._run()
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/main_loop.py", line 1484, in run
    reraise(*exc_info)
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/raw_display.py", line 404, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/raw_display.py", line 475, in parse_input
    codes, wait_for_more)
  File "/Users/binhho/Documents/t-mobile/mitmproxy/venv/lib/python3.7/site-packages/urwid/escape.py", line 387, in process_keyqueue
    if run[0] == "esc" or run[0].find("meta ") >= 0:
AttributeError: 'tuple' object has no attribute 'find'
```
#### System Information
Paste the output of "mitmproxy --version" here.
```
Mitmproxy: 6.0.0.dev (+17, commit 406f8ac)
Python:    3.7.5
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Darwin-19.2.0-x86_64-i386-64bit
```



Discussion:
This should be fixed in the recent urwid version 2.1.0: https://github.com/urwid/urwid/pull/304
We need to test and fix incompatibilities with this new dependency.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -81,7 +81,7 @@
         "ruamel.yaml>=0.16,<0.17",
         "sortedcontainers>=2.1.0,<2.2",
         "tornado>=4.3,<7",
-        "urwid>=2.0.1,<2.1",
+        "urwid>=2.1.0,<2.2",
         "wsproto>=0.14.0,<0.15.0",
         "publicsuffix2>=2.20190812,<3",
         "zstandard>=0.11.0,<0.13.0",


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Headers replace bug when count is set
#### Problem Description
Headers matching for replace are fully removed, if count=1

#### Steps to reproduce the behavior:
1. create header instance: `headers = Headers(host='foo.com', referrer='foo.com')`
2. replace `foo` with `bar` and `count=1`: `headers.replace('foo', 'bar', count=1)`
3. result is empty headers: `Headers[]`

```
>>> headers = Headers(host='foo.com', referrer='foo.com')
>>> headers.replace('foo', 'bar', count=1)
1
>>> headers
Headers[]
```

#### System Information
Mitmproxy: 5.0.1
Python:    3.7.6
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Darwin-19.3.0-x86_64-i386-64bit



Discussion:
Thanks for providing super easy steps to reproduce this! :smiley:
I'll try to fix it.

Resolving patch:
diff --git a/mitmproxy/net/http/headers.py b/mitmproxy/net/http/headers.py
--- a/mitmproxy/net/http/headers.py
+++ b/mitmproxy/net/http/headers.py
@@ -162,8 +162,12 @@ def replace(self, pattern, repl, flags=0, count=0):
         pattern = re.compile(pattern, flags)
         replacements = 0
         flag_count = count > 0
+        count_reached = False
         fields = []
         for name, value in self.fields:
+            if count_reached:
+                fields.append((name, value))
+                continue
             line, n = pattern.subn(repl, name + b": " + value, count=count)
             try:
                 name, value = line.split(b": ", 1)
@@ -173,10 +177,12 @@ def replace(self, pattern, repl, flags=0, count=0):
                 pass
             else:
                 replacements += n
+                fields.append((name, value))
                 if flag_count:
                     count -= n
                     if count == 0:
-                        break
+                        count_reached = True
+                continue
             fields.append((name, value))
         self.fields = tuple(fields)
         return replacements


Test patch:
diff --git a/test/mitmproxy/net/http/test_headers.py b/test/mitmproxy/net/http/test_headers.py
--- a/test/mitmproxy/net/http/test_headers.py
+++ b/test/mitmproxy/net/http/test_headers.py
@@ -88,6 +88,8 @@ def test_replace_with_count(self):
         headers = Headers(Host="foobarfoo.com", Accept="foo/bar")
         replacements = headers.replace("foo", "bar", count=1)
         assert replacements == 1
+        assert headers["Host"] == "barbarfoo.com"
+        assert headers["Accept"] == "foo/bar"
 
 
 def test_parse_content_type():
Repository: mitmproxy/mitmproxy

Issue:
ServerConnection.ip_address is not populated inside `serverconnect` event
#### Problem Description

Inside the `serverconnect` event the `ip_address` property is not populated, it is `None`. The `address` property is correctly populated. Since the connection is open I would expect the IP address to be known and the property to be available.

#### Steps to reproduce the behavior:

`mitmdump --scripts conn.py`

```python
def serverconnect(conn):
  print('-------connect--------------')
  print(conn.id)
  print(conn.address)
  print(conn.ip_address)
  print('-------/connect--------------')

def serverdisconnect(conn):
  print('----------disconnect-----------')
  print(conn.id)
  print(conn.address)
  print(conn.ip_address)
  print('----------/disconnect-----------')
```

Output

```
-------connect--------------
ab301a58-2366-4025-bab9-77c2bdfe74e6
('prinzhorn.it', 80)
None
-------/connect--------------

----------disconnect-----------
ab301a58-2366-4025-bab9-77c2bdfe74e6
('prinzhorn.it', 80)
('87.98.150.101', 80)
----------/disconnect-----------
```

#### System Information

```
Mitmproxy: 5.0.1
Python:    3.7.5
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Linux-5.3.0-45-generic-x86_64-with-Ubuntu-19.10-eoan
```



Discussion:
I assume this is the place

https://github.com/mitmproxy/mitmproxy/blob/3046a628fd0b719ff587eac5f5fd6965cd5eac89/mitmproxy/proxy/protocol/base.py#L164-L166

If this is expected behavior then the docs need to be updated. It says

> Mitmproxy **has** connected to a server.
`conn.connected()` is `False` in both cases. Either it needs to be `True` inside `serverconnect` or `True` inside `serverdisconnect`. It is not symmetrical right now.

https://github.com/mitmproxy/mitmproxy/blob/3046a628fd0b719ff587eac5f5fd6965cd5eac89/mitmproxy/proxy/protocol/base.py#L148-L149

Resolving patch:
diff --git a/mitmproxy/proxy/protocol/base.py b/mitmproxy/proxy/protocol/base.py
--- a/mitmproxy/proxy/protocol/base.py
+++ b/mitmproxy/proxy/protocol/base.py
@@ -160,10 +160,10 @@ def connect(self):
         """
         if not self.server_conn.address:
             raise exceptions.ProtocolException("Cannot connect to server, no server address given.")
-        self.log("serverconnect", "debug", [repr(self.server_conn.address)])
-        self.channel.ask("serverconnect", self.server_conn)
         try:
             self.server_conn.connect()
+            self.log("serverconnect", "debug", [repr(self.server_conn.address)])
+            self.channel.ask("serverconnect", self.server_conn)
         except exceptions.TcpException as e:
             raise exceptions.ProtocolException(
                 "Server connection to {} failed: {}".format(


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Increase WebSocket message timestamp precision
#### Is your feature request related to a problem? Please describe.
Timestamps for WebSocket messages are rounded to whole seconds. For at least some protocols or analyses, a higher precision may be useful or even necessary for productive use; one example is measuring the performance impact of mitmproxy.

#### Describe the solution you'd like
Avoid converting the timestamp to an integer and use a float instead:

https://github.com/mitmproxy/mitmproxy/blob/7fdcbb09e6034ab1f76724965cfdf45f3d775129/mitmproxy/websocket.py#L28

Basic support for higher-precision timestamps seems to be as simple as changing the type of `WebSocketMessage.timestamp` from `int` to `float` and dropping the `int()` call around `time.time()`. I'm not sure if more is needed e.g. to ensure backward compatibility, but I was able to read a previous dump just fine with that modification (which makes sense since integers in the relevant range are strictly a subset of double-precision floats).

#### Describe alternatives you've considered
If keeping the `timestamp` an `int` is unavoidable, the Linux approach of storing the microseconds in a second integer (between 0 and 999999, inclusive) could be used. I don't think this is a good idea though.

#### Additional context
None


Discussion:
Thanks! You are absolutely right, this should just be a float. Happy to accept a PR that fixes this. :)

Resolving patch:
diff --git a/mitmproxy/websocket.py b/mitmproxy/websocket.py
--- a/mitmproxy/websocket.py
+++ b/mitmproxy/websocket.py
@@ -17,7 +17,7 @@ class WebSocketMessage(serializable.Serializable):
     """
 
     def __init__(
-        self, type: int, from_client: bool, content: bytes, timestamp: Optional[int]=None, killed: bool=False
+        self, type: int, from_client: bool, content: bytes, timestamp: Optional[float]=None, killed: bool=False
     ) -> None:
         self.type = Opcode(type)  # type: ignore
         """indicates either TEXT or BINARY (from wsproto.frame_protocol.Opcode)."""
@@ -25,7 +25,7 @@ def __init__(
         """True if this messages was sent by the client."""
         self.content = content
         """A byte-string representing the content of this message."""
-        self.timestamp: int = timestamp or int(time.time())
+        self.timestamp: float = timestamp or time.time()
         """Timestamp of when this message was received or created."""
         self.killed = killed
         """True if this messages was killed and should not be sent to the other endpoint."""


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Mitmweb fails with addons/options-configure.py example.
I am new to learn it, but i follow official [demo][1], it can't working?
```python
Proxy server listening at http://*:8888
ERROR:tornado.application:Uncaught exception GET /options.json (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:8081', method='GET', uri='/options.json', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\tornado\web.py", line 1697, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\tools\web\app.py", line 453, in get
    self.write(optmanager.dump_dicts(self.master.options))
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\optmanager.py", line 469, in dump_dicts
    t = typecheck.typespec_to_str(o.typespec)
  File "c:\users\jekoie\appdata\local\programs\python\python37-32\lib\site-packages\mitmproxy\utils\typecheck.py", line 85, in typespec_to_str
    raise NotImplementedError
NotImplementedError
ERROR:tornado.access:500 GET /options.json (127.0.0.1) 3.91ms
````

[1]: https://docs.mitmproxy.org/stable/addons-options/#handling-configuration-updates


Discussion:
Thanks for reporting this. It's is a bug in mitmweb only, mitmproxy and mitmdump are unaffected.

Repro:
```
mitmweb -s examples/addons/options-configure.py
```

Resolving patch:
diff --git a/mitmproxy/utils/typecheck.py b/mitmproxy/utils/typecheck.py
--- a/mitmproxy/utils/typecheck.py
+++ b/mitmproxy/utils/typecheck.py
@@ -81,6 +81,8 @@ def typespec_to_str(typespec: typing.Any) -> str:
         t = 'optional str'
     elif typespec == typing.Sequence[str]:
         t = 'sequence of str'
+    elif typespec == typing.Optional[int]:
+        t = 'optional int'
     else:
         raise NotImplementedError
     return t


Test patch:
diff --git a/test/mitmproxy/utils/test_typecheck.py b/test/mitmproxy/utils/test_typecheck.py
--- a/test/mitmproxy/utils/test_typecheck.py
+++ b/test/mitmproxy/utils/test_typecheck.py
@@ -72,6 +72,7 @@ def test_typesec_to_str():
     assert(typecheck.typespec_to_str(str)) == "str"
     assert(typecheck.typespec_to_str(typing.Sequence[str])) == "sequence of str"
     assert(typecheck.typespec_to_str(typing.Optional[str])) == "optional str"
+    assert(typecheck.typespec_to_str(typing.Optional[int])) == "optional int"
     with pytest.raises(NotImplementedError):
         typecheck.typespec_to_str(dict)
Repository: mitmproxy/mitmproxy

Issue:
mitmproxy  dictionary changed size during iteration
#### Problem Description
A clear and concise description of what the bug is.
在使用mitmproxy作为二级代理时，出现了错误。
 flow.live.change_upstream_proxy_server((proxy, 8080))
#### Steps to reproduce the behavior:
1. mitmproxy has crashed!
Please lodge a bug report at: https://github.com/mitmproxy/mitmproxy
 Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/server.py", line 121, in handle
    root_layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/modes/http_proxy.py", line 23, in __call__
    layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 286, in __call__
    layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http1.py", line 83, in __call__
    layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 188, in __call__
    if not self._process_flow(flow):
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 262, in _process_flow
    return self.handle_upstream_connect(f)
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 236, in handle_upstream_connect
    return layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 87, in __call__
    layer()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 279, in __call__
    self._establish_tls_with_client_and_server()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 365, in _establish_tls_with_client_and_server
    self._establish_tls_with_client()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 369, in _establish_tls_with_client
    cert, key, chain_file = self._find_cert()
  File "/usr/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 501, in _find_cert
    return self.config.certstore.get_cert(host, list(sans))
  File "/usr/lib/python3.6/site-packages/mitmproxy/certs.py", line 326, in get_cert
    self.expire(entry)
  File "/usr/lib/python3.6/site-packages/mitmproxy/certs.py", line 156, in expire
    for k, v in list(self.certs.items()):
RuntimeError: dictionary changed size during iteration


#### System Information
Paste the output of "mitmproxy --version" here.
Mitmproxy: 4.0.4
Python: 3.6.3
OpenSSL: OpenSSL 1.0.2o 27 Mar 2018
Platform: Linux-3.10.0-1062.el7.x86_64-with



Discussion:
Hi, how can reproduce this?

Resolving patch:
diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py
--- a/mitmproxy/certs.py
+++ b/mitmproxy/certs.py
@@ -173,9 +173,7 @@ def expire(self, entry):
         self.expire_queue.append(entry)
         if len(self.expire_queue) > self.STORE_CAP:
             d = self.expire_queue.pop(0)
-            for k, v in list(self.certs.items()):
-                if v == d:
-                    del self.certs[k]
+            self.certs = {k: v for k, v in self.certs.items() if v != d}
 
     @staticmethod
     def load_dhparam(path):


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
replay.client.stop crashes the client
#### Problem Description
Running replay.client.stop (with or without pending replays) will crash the client and will require a full restart.

Versions:
MacOS 10.15.5 (19F96)
Python 3.7.7 (default, Mar 10 2020, 15:43:33)

The same also occurred on Ubuntu 20.04.

#### Steps to reproduce the behavior:
1. Launch mitmproxy with given version
2. Run :replay.client.stop
3. Whole client crashes

#### System Information
Mitmproxy: 5.1.1
Python:    3.7.7
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  Darwin-19.5.0-x86_64-i386-64bit




Discussion:
Thanks for the straightforward repro!
@mhils No worries. I'll see if I can get to it in the near future (still a newbie to open source though). 

Resolving patch:
diff --git a/mitmproxy/addons/clientplayback.py b/mitmproxy/addons/clientplayback.py
--- a/mitmproxy/addons/clientplayback.py
+++ b/mitmproxy/addons/clientplayback.py
@@ -180,7 +180,7 @@ def stop_replay(self) -> None:
             self.q.queue.clear()
             for f in lst:
                 f.revert()
-            ctx.master.addons.trigger("update", lst)
+        ctx.master.addons.trigger("update", lst)
         ctx.log.alert("Client replay queue cleared.")
 
     @command.command("replay.client")


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
View response in Firefox gives file not found
#### Problem Description
Hitting `v` on a flow opens up firefox pointing to a temp mitmproxy file but the file doesn't exist

#### Steps to reproduce the behavior:
1. Capture a request 
2. Press v in the flow view of request and 2 for viewing response
3. Firefox opens up to view the response html page but only displays file not found e.g. `Firefox can’t find the file at /tmp/mproxynqb8uytn.html`. (Hitting `e` and editing the response works correctly and opens up the response as a tmp file in my editor)

#### System Information
Mitmproxy: 5.1.1
Python:    3.8.3
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  Linux-5.6.19-2-MANJARO-x86_64-with-glibc2.2.5



Discussion:
Thanks! The problem is likely in https://github.com/mitmproxy/mitmproxy/blob/408bd7fd4c5ed147ce2951428bfdd6206e1eb8b8/mitmproxy/tools/console/master.py#L139-L174. I'd guess that the process we're calling exits before your Firefox renders and we've then already unlinked. Not sure what the best approach is, but I guess moving the unlink to a thread with a timeout should fix this main issue.

Resolving patch:
diff --git a/mitmproxy/tools/console/master.py b/mitmproxy/tools/console/master.py
--- a/mitmproxy/tools/console/master.py
+++ b/mitmproxy/tools/console/master.py
@@ -11,6 +11,7 @@
 import tempfile
 import typing  # noqa
 import contextlib
+import threading
 
 import urwid
 
@@ -171,7 +172,9 @@ def spawn_external_viewer(self, data, contenttype):
                 signals.status_message.send(
                     message="Can't start external viewer: %s" % " ".join(c)
                 )
-        os.unlink(name)
+        # add a small delay before deletion so that the file is not removed before being loaded by the viewer
+        t = threading.Timer(1.0, os.unlink, args=[name])
+        t.start()
 
     def set_palette(self, opts, updated):
         self.ui.register_palette(


Test patch: