internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mitmproxy/mitmproxy

Issue:
It is difficult to get information from request/response/detail tab, if url is very-very long
##### Steps to reproduce the problem:

1. Run mitmproxy.
2. Make a request with pathoc: `pathoc -c example.com:80 localhost:8080 'get:/SOMETHING VERY LONG'`
3. Enter the corresponding flow.
You won't be able to see request/response/detail of the flow without changing size of your window. 
![image](https://user-images.githubusercontent.com/20267977/35508608-a7e26636-04f9-11e8-96e6-2627d0b8d1f0.png)

But if we decide to make a request with url over for example 10k length, even changing the size of window won't help you to see request/response/detail tab.

##### Any other comments? What have you tried so far?

About maximum length of a URL
https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers


##### System information

Mitmproxy: 3.0.0.dev64 (commit 6dd336f) 
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-112-generic-x86_64-with-Ubuntu-16.04-xenial



Discussion:
Maybe I can work on this? My initial thoughts would be to have a maximum URL length, and allow user to select it to see the whole content.
@tran-tien-dat: Sure, feel free to. :-)
I could not find a way to display the extra-extra long url with user's consent without breaking the accessibility of the UI. But to restore it currently, i have wrapped the url  upto allowed maximum columns.

Resolving patch:
diff --git a/mitmproxy/tools/console/flowview.py b/mitmproxy/tools/console/flowview.py
--- a/mitmproxy/tools/console/flowview.py
+++ b/mitmproxy/tools/console/flowview.py
@@ -30,12 +30,14 @@ def __init__(
         self.focus_changed()
 
     def focus_changed(self):
+        cols, _ = self.master.ui.get_cols_rows()
         if self.master.view.focus.flow:
             self._w = common.format_flow(
                 self.master.view.focus.flow,
                 False,
                 extended=True,
-                hostheader=self.master.options.showhost
+                hostheader=self.master.options.showhost,
+                max_url_len=cols,
             )
         else:
             self._w = urwid.Pile([])


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Add exption handling for hasattr because it could throw
Fixed https://github.com/mitmproxy/mitmproxy/issues/2849


Discussion:
I will update it and add test
Hi, any update on this? :)

Resolving patch:
diff --git a/mitmproxy/command.py b/mitmproxy/command.py
--- a/mitmproxy/command.py
+++ b/mitmproxy/command.py
@@ -1,5 +1,5 @@
 """
-    This module manges and invokes typed commands.
+    This module manages and invokes typed commands.
 """
 import inspect
 import types
@@ -131,8 +131,13 @@ def collect_commands(self, addon):
         for i in dir(addon):
             if not i.startswith("__"):
                 o = getattr(addon, i)
-                if hasattr(o, "command_path"):
-                    self.add(o.command_path, o)
+                try:
+                    is_command = hasattr(o, "command_path")
+                except Exception:
+                    pass  # hasattr may raise if o implements __getattr__.
+                else:
+                    if is_command:
+                        self.add(o.command_path, o)
 
     def add(self, path: str, func: typing.Callable):
         self.commands[path] = Command(self, path, func)


Test patch:
diff --git a/test/mitmproxy/test_command.py b/test/mitmproxy/test_command.py
--- a/test/mitmproxy/test_command.py
+++ b/test/mitmproxy/test_command.py
@@ -309,6 +309,31 @@ def empty(self) -> None:
         pass
 
 
+class TAttr:
+    def __getattr__(self, item):
+        raise IOError
+
+
+class TCmds(TAttr):
+    def __init__(self):
+        self.TAttr = TAttr()
+
+    @command.command("empty")
+    def empty(self) -> None:
+        pass
+
+
+def test_collect_commands():
+    """
+        This tests for the error thrown by hasattr()
+    """
+    with taddons.context() as tctx:
+        c = command.CommandManager(tctx.master)
+        a = TCmds()
+        c.collect_commands(a)
+        assert "empty" in c.commands
+
+
 def test_decorator():
     with taddons.context() as tctx:
         c = command.CommandManager(tctx.master)
Repository: mitmproxy/mitmproxy

Issue:
Keybinding ‘M’ erroneously highlighted
##### Steps to reproduce the problem:

1. run `mitmproxy`
2. type `O` to open the options editor
3. move the cursor down to `default_contentview` and press Enter
4. type `4` to select `json`
5. type `q` to return to the flows list
6. the status bar now says “[M:json]” with the ‘M’ highlighted, as if it were a shortcut
7. however, typing `M` invokes “Toggle viewing marked flows” instead

##### Any other comments? What have you tried so far?

This is similar to #2953.

##### System information

Mitmproxy: 3.0.3 binary
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-116-generic-x86_64-with-debian-stretch-sid



Discussion:
Thanks for reporting these with fantastic instructions, this is really helpful! For now let's just remove the binding, we can think about adding it back later. 

If someone wants to tackle the "adding back" part in a second PR, it's a bit more difficult now that shortcuts are user-configurable. We'd need to have a function that returns the current keybinding for a given command, and catch the case where no keybinding is configured (in which case we probably need to display something else, e.g. `[mode:json]`). 

Resolving patch:
diff --git a/mitmproxy/tools/console/statusbar.py b/mitmproxy/tools/console/statusbar.py
--- a/mitmproxy/tools/console/statusbar.py
+++ b/mitmproxy/tools/console/statusbar.py
@@ -228,10 +228,8 @@ def get_status(self):
             r.append("[")
             r.append(("heading_key", "u"))
             r.append(":%s]" % self.master.options.stickyauth)
-        if self.master.options.console_default_contentview != "auto":
-            r.append("[")
-            r.append(("heading_key", "M"))
-            r.append(":%s]" % self.master.options.console_default_contentview)
+        if self.master.options.console_default_contentview != 'auto':
+            r.append("[contentview:%s]" % (self.master.options.console_default_contentview))
         if self.master.options.has_changed("view_order"):
             r.append("[")
             r.append(("heading_key", "o"))


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
No shortcut for replacements editor
##### Steps to reproduce the problem:

1. run `mitmproxy`
2. type `O` to open the options editor
3. move the cursor down to `replacements` and press Enter
4. type `a` then `/~s/foo/bar` to add a replacement, and press Esc to commit
5. type `q` and again `q` to return to the flows list
6. the status bar now says “[Replacing]” with the ‘R’ highlighted, as if it were a shortcut
7. however, typing `R` doesn’t do anything

##### Any other comments? What have you tried so far?

It seems like `R` was intended to be a shortcut for the replacements editor (which would be very convenient), but left out. It’s not listed in the online help, either.

If it wasn’t intended to be a shortcut, it shouldn’t be highlighted in the status bar.

##### System information

Mitmproxy: 3.0.3 binary
Python:    3.5.2
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-116-generic-x86_64-with-debian-stretch-sid


Discussion:
Hi! If someone isn't working on this already, I would like to work on this.

So from the above discussion, I am not sure that whether or not would we be needing R as a shortcut . Also, if we need to make it a shortcut, do I need to add a relevant entry here: https://github.com/mitmproxy/mitmproxy/blob/957a630bb5d2fcd6a8ad7092863d19e38307f090/mitmproxy/tools/console/overlay.py#L67

Thanks!
Hello, I am not working on this, you would like to ask others as well.
@kajojify @fenilgandhi 
I had a look into it but hit the same dead end. There is a decision to be made here and needs someone more  familiar with the code as the wrong one would add inconsistency to the code. 

@proishan11  Also if you want to you can start looking into it or wait for @kajojify  to reply.
But if the decision comes in favor of removing the highlight , it can be done here. 
https://github.com/mitmproxy/mitmproxy/blob/957a630bb5d2fcd6a8ad7092863d19e38307f090/mitmproxy/tools/console/statusbar.py#L162
I am not working on it. Feel free to take it :) 
Good luck! 

Thanks to your help @fenilgandhi, I am having no problem in removing the highlight.
Also, I want to ask that is there any documentation to get familiar with the code?
hey @proishan11, are you working on it or can i take this one ?


@fenilgandhi  I am not working on this. Feel free to take it :)

Resolving patch:
diff --git a/mitmproxy/tools/console/statusbar.py b/mitmproxy/tools/console/statusbar.py
--- a/mitmproxy/tools/console/statusbar.py
+++ b/mitmproxy/tools/console/statusbar.py
@@ -191,9 +191,7 @@ def get_status(self):
             r.append(("heading_key", "H"))
             r.append("eaders]")
         if len(self.master.options.replacements):
-            r.append("[")
-            r.append(("heading_key", "R"))
-            r.append("eplacing]")
+            r.append("[%d replacements]" % len(self.master.options.replacements))
         if creplay.count():
             r.append("[")
             r.append(("heading_key", "cplayback"))


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
[mitmweb] asyncio has no attribute call_soon
##### Steps to reproduce the problem:

1.Launch mitmweb
2.Save some flow
3.Load those flows

![asyncio_attribute_error](https://user-images.githubusercontent.com/15815157/38856635-48c7130c-4227-11e8-9019-0caf8f22038c.JPG)


##### Any other comments? What have you tried so far?

https://github.com/mitmproxy/mitmproxy/blob/0fa1280daa94729defa8411d86266bd2b52ad0b6/mitmproxy/tools/web/app.py#L238-L239

I replaced this line with `asyncio.ensure_future(self.master.load_flows(i))` to make the loading works.


##### System information

<!-- Paste the output of "mitmproxy --version" here. -->
![mitmdump_version](https://user-images.githubusercontent.com/15815157/38857072-93f56a8a-4228-11e8-8fbb-fed4e11c2745.JPG)


<!-- Please use the mitmproxy forums (https://discourse.mitmproxy.org/) for support/how-to questions. Thanks! :) -->



Discussion:
Want to submit a PR for this? Your solution is correct. 

Resolving patch:
diff --git a/mitmproxy/tools/web/app.py b/mitmproxy/tools/web/app.py
--- a/mitmproxy/tools/web/app.py
+++ b/mitmproxy/tools/web/app.py
@@ -235,7 +235,7 @@ def post(self):
         self.view.clear()
         bio = BytesIO(self.filecontents)
         for i in io.FlowReader(bio).stream():
-            asyncio.call_soon(self.master.load_flow, i)
+            asyncio.ensure_future(self.master.load_flow(i))
         bio.close()
 
 


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
duplicate and replay example error
Hello I am trying to understand duplicate and replay process so I tried to run dup_and_replay.py but there is something that I keep getting this error

Addon error: Traceback (most recent call last):
  File "dup.py", line 6, in request
    ctx.master.view.add(f)
AttributeError: 'DumpMaster' object has no attribute 'view'

Could you please help me 


Discussion:

Resolving patch:
diff --git a/examples/complex/dup_and_replay.py b/examples/complex/dup_and_replay.py
--- a/examples/complex/dup_and_replay.py
+++ b/examples/complex/dup_and_replay.py
@@ -2,7 +2,13 @@
 
 
 def request(flow):
-    f = flow.copy()
-    ctx.master.view.add(f)
-    f.request.path = "/changed"
-    ctx.master.replay_request(f, block=True)
+    # Avoid an infinite loop by not replaying already replayed requests
+    if flow.request.is_replay:
+        return
+    flow = flow.copy()
+    # Only interactive tools have a view. If we have one, add a duplicate entry
+    # for our flow.
+    if "view" in ctx.master.addons:
+        ctx.master.commands.call("view.add", [flow])
+    flow.request.path = "/changed"
+    ctx.master.commands.call("replay.client", [flow])
diff --git a/mitmproxy/addons/view.py b/mitmproxy/addons/view.py
--- a/mitmproxy/addons/view.py
+++ b/mitmproxy/addons/view.py
@@ -293,6 +293,7 @@ def clear_not_marked(self):
         self._refilter()
         self.sig_store_refresh.send(self)
 
+    @command.command("view.marked.toggle")
     def add(self, flows: typing.Sequence[mitmproxy.flow.Flow]) -> None:
         """
             Adds a flow to the state. If the flow already exists, it is


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Client replay doesn't update request timestamps
##### Steps to reproduce the problem:

1. Save flows to a file, `saved.mitm`
2. Wait a few minutes
3. Use  a command like `mitmdump -n -C saved.mitm -s myscript.py -w replayed.mtim` to replay the saved flows. 
4. Then observe the flows with `mitmproxy -n -r replayed.mitm` and see that the request-response time of the flows are inaccurate because it seems to be calculated using the start time of the requests in `saved.mitm`.

##### Any other comments? What have you tried so far?
If such behavior is intentional, what's the recommended way to set the request timestamps so that they reflect the time when the requests are actually replayed rather than the previously saved timestamps?

As a workaround, I've added these two lines after the `server.wfile.flush()` line in `addons/clientplayback.py` for my use case:

```
                r.timestamp_end = time.time()
                r.timestamp_start = server.wfile.first_byte_timestamp
```
It seems reasonable but I'm not sure if it's correct.

##### System information
Mitmproxy: 4.0.4
Python:    3.7.0
OpenSSL:   OpenSSL 1.0.2p  14 Aug 2018
Platform:  Darwin-17.5.0-x86_64-i386-64bit




Discussion:
Thanks for the report. I agree - we should probably do a simple
```
request.timestamp_start = request.timestamp_end = time.time()
```
if/before we do client replay. Do you want to submit a PR for this? :)

Resolving patch:
diff --git a/mitmproxy/addons/clientplayback.py b/mitmproxy/addons/clientplayback.py
--- a/mitmproxy/addons/clientplayback.py
+++ b/mitmproxy/addons/clientplayback.py
@@ -1,6 +1,7 @@
 import queue
 import threading
 import typing
+import time
 
 from mitmproxy import log
 from mitmproxy import controller
@@ -94,6 +95,7 @@ def replay(self, f):  # pragma: no cover
 
                 server.wfile.write(http1.assemble_request(r))
                 server.wfile.flush()
+                r.timestamp_start = r.timestamp_end = time.time()
 
                 if f.server_conn:
                     f.server_conn.close()


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Confusing confdir argument
My tests are messy cause I used mitmproxy and mitmdump, both of them in 4.0.1 and 4.0.3 and now I mixed all cases.

At some point, I think I had an error saying that --confdir was deprecated and had to use "--set confdir=" (I can't reproduce this case though with mitmproxy or mitmdump...)

I spent some time to try to make "--set confdir=" work in a weird bash script and arguments with quotes and maybe it failed due to me.
But I realized --confdir was still working eventually for mitmdump in 4.0.3.

Question to sum up:
Is it attended to have both "--confdir" and "--set confdir=" working at the same time for mitmproxy & mitmdump?
If yes, help (-h) should specify it clearly with something like: "--confdir PATH, --set confdir=PATH"
If not, one of them should be deleted.


Discussion:
@tomlabaude Hello! 
I checked your case for both mitmproxy and mitmdump (latest version from the master branch). `--confdir` is deprecated for newest mitmproxy as well as for newest mitmdump and only `--set confdir=` is available.
You can read more about mitmproxy options here -> https://docs.mitmproxy.org/stable/concepts-options/

In response to your question. Since deprecation list has `confdir` inside (https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/utils/arg_check.py#L4), then no, it isn't meant to have both `--confdir` and `--set confdir=` - only `--set confdir`.
And yes, we should delete `--confdir` from `--help` output. Thanks :) 

Resolving patch:
diff --git a/mitmproxy/tools/_main.py b/mitmproxy/tools/_main.py
--- a/mitmproxy/tools/_main.py
+++ b/mitmproxy/tools/_main.py
@@ -87,7 +87,7 @@ def run(
         arg_check.check()
         sys.exit(1)
     try:
-        opts.confdir = args.confdir
+        opts.set(*args.setoptions, defer=True)
         optmanager.load_paths(
             opts,
             os.path.join(opts.confdir, OPTIONS_FILE_NAME),
@@ -110,7 +110,6 @@ def run(
         if args.commands:
             master.commands.dump()
             sys.exit(0)
-        opts.set(*args.setoptions, defer=True)
         if extra:
             opts.update(**extra(args))
 
diff --git a/mitmproxy/tools/cmdline.py b/mitmproxy/tools/cmdline.py
--- a/mitmproxy/tools/cmdline.py
+++ b/mitmproxy/tools/cmdline.py
@@ -20,12 +20,6 @@ def common_options(parser, opts):
         action='store_true',
         help="Show all commands and their signatures",
     )
-    parser.add_argument(
-        "--confdir",
-        type=str, dest="confdir", default=core.CONF_DIR,
-        metavar="PATH",
-        help="Path to the mitmproxy config directory"
-    )
     parser.add_argument(
         "--set",
         type=str, dest="setoptions", default=[],


Test patch:
Repository: mitmproxy/mitmproxy

Issue:
Server replay corrupts Date header
##### Steps to reproduce the problem:

1. Start `mitmproxy`.
2. Run a request through it and observe that the `Date` header of the response is correct:

       $ curl -six http://localhost:8080 http://httpbin.org/status/200 | grep Date:
       Date: Thu, 21 Mar 2019 12:45:00 GMT

3. In mitmproxy, start server replay:

       : replay.server @all
       
4. Run a request again and observe that the `Date` header now has `-0000` instead of `GMT`:

       $ curl -six http://localhost:8080 http://httpbin.org/status/200 | grep Date:
       Date: Thu, 21 Mar 2019 12:46:58 -0000


##### Any other comments? What have you tried so far?

Per [RFC 7231 § 7.1.1](https://tools.ietf.org/html/rfc7231#section-7.1.1), a correct HTTP date always has `GMT`.


##### System information

    Mitmproxy: 4.0.4 binary
    Python:    3.6.3
    OpenSSL:   OpenSSL 1.1.0h  27 Mar 2018
    Platform:  Linux-4.15.0-46-generic-x86_64-with-debian-buster-sid



Discussion:
Thanks! The culprit is here somewhere:
https://github.com/mitmproxy/mitmproxy/blob/8353f4a55afeec9d30727d91d642e8b8af4040f8/mitmproxy/addons/serverplayback.py#L182
https://github.com/mitmproxy/mitmproxy/blob/8353f4a55afeec9d30727d91d642e8b8af4040f8/mitmproxy/net/http/response.py#L168

Resolving patch:
diff --git a/mitmproxy/net/http/cookies.py b/mitmproxy/net/http/cookies.py
--- a/mitmproxy/net/http/cookies.py
+++ b/mitmproxy/net/http/cookies.py
@@ -304,7 +304,7 @@ def refresh_set_cookie_header(c: str, delta: int) -> str:
             e = email.utils.parsedate_tz(attrs["expires"])
             if e:
                 f = email.utils.mktime_tz(e) + delta
-                attrs.set_all("expires", [email.utils.formatdate(f)])
+                attrs.set_all("expires", [email.utils.formatdate(f, usegmt=True)])
             else:
                 # This can happen when the expires tag is invalid.
                 # reddit.com sends a an expires tag like this: "Thu, 31 Dec
diff --git a/mitmproxy/net/http/response.py b/mitmproxy/net/http/response.py
--- a/mitmproxy/net/http/response.py
+++ b/mitmproxy/net/http/response.py
@@ -186,7 +186,7 @@ def refresh(self, now=None):
                 d = parsedate_tz(self.headers[i])
                 if d:
                     new = mktime_tz(d) + delta
-                    self.headers[i] = formatdate(new)
+                    self.headers[i] = formatdate(new, usegmt=True)
         c = []
         for set_cookie_header in self.headers.get_all("set-cookie"):
             try:


Test patch:
diff --git a/test/mitmproxy/net/http/test_response.py b/test/mitmproxy/net/http/test_response.py
--- a/test/mitmproxy/net/http/test_response.py
+++ b/test/mitmproxy/net/http/test_response.py
@@ -148,7 +148,7 @@ def test_set_cookies(self):
     def test_refresh(self):
         r = tresp()
         n = time.time()
-        r.headers["date"] = email.utils.formatdate(n)
+        r.headers["date"] = email.utils.formatdate(n, usegmt=True)
         pre = r.headers["date"]
         r.refresh(946681202)
         assert pre == r.headers["date"]
Repository: mitmproxy/mitmproxy

Issue:
Command view.focus.[next|prev] causes mitmproxy to crash 
#### Problem Description
`mitmproxy` crashes if one tries to run the command `view.focus.next` or `view.focus.prev` when there are no flows in it yet

#### Steps to reproduce the behavior:
1. Open mitmproxy making sure there are no clients using it
2. In the main screen press `:` and then enter the command `view.focus.next`
3. Press `Return` and notice that the application crashes

#### System Information
$ mitmproxy --version
Mitmproxy: 4.0.4
Python:    3.7.3
OpenSSL:   OpenSSL 1.1.1b  26 Feb 2019
Platform:  Linux-5.0.0-31-generic-x86_64-with-Ubuntu-19.04-disco




Discussion:

Resolving patch:
diff --git a/mitmproxy/addons/view.py b/mitmproxy/addons/view.py
--- a/mitmproxy/addons/view.py
+++ b/mitmproxy/addons/view.py
@@ -238,18 +238,24 @@ def focus_next(self) -> None:
         """
             Set focus to the next flow.
         """
-        idx = self.focus.index + 1
-        if self.inbounds(idx):
-            self.focus.flow = self[idx]
+        if self.focus.index is not None:
+            idx = self.focus.index + 1
+            if self.inbounds(idx):
+                self.focus.flow = self[idx]
+        else:
+            pass
 
     @command.command("view.focus.prev")
     def focus_prev(self) -> None:
         """
             Set focus to the previous flow.
         """
-        idx = self.focus.index - 1
-        if self.inbounds(idx):
-            self.focus.flow = self[idx]
+        if self.focus.index is not None:
+            idx = self.focus.index - 1
+            if self.inbounds(idx):
+                self.focus.flow = self[idx]
+        else:
+            pass
 
     # Order
     @command.command("view.order.options")


Test patch: