Repository: mitmproxy/mitmproxy

Issue:
Dumper Addon: make the number of printed lines configurable when using `flow_detail = 3`
I try to dump full traffic on stdout using mitmdump, but when response body has more than about 80 or 90 lines on stdout appear (cut off) and all other lines are lost,

I use this command:
*mitmdump  --set flow_detail=3 --set content_view_lines_cutoff=2000* <br><br>

The exact behavior is described [here](https://discourse.mitmproxy.org/t/mitmdump-cutting-off-responses/1714) on old mitmproxy forum: unfortunately there was no response

@debian-10:~$ Scaricati/mitmdump --version
Mitmproxy: 6.0.2 binary
Python:    3.9.1
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Linux-4.19.0-14-amd64-x86_64-with-glibc2.28



Discussion:
I'm not sure why it's that way, but it happens here

https://github.com/mitmproxy/mitmproxy/blob/8178f6c77b715cd342f8be161d5ca5da8938645c/mitmproxy/addons/dumper.py#L110-L113

but all calls to `_echo_message` are already guarded with `if ctx.options.flow_detail >= 3:` so the `else` will _never_ be reached. Edit: ok, technically if `flow_details` was `4` or something :roll_eyes: 

It's been this way since the beginning https://github.com/mitmproxy/mitmproxy/commit/b0b3b19ad644afeb353cf6e02bd4aac61f2774c8

`content_view_lines_cutoff` is only for mitmproxy ui, I guess we need to introduce a new option for the dumper and default to `0` (dump everything) I guess?

In either case `70` is currently a magic number. Magic number bad. Make programmer angry.

Docs say "3: 2 + **full response content**, content of WebSocket and TCP messages.". Liars.
I changed my mind, sort of.

`flow_detail = 4` is simply undocumented :thinking: . You can use it ~as a workaround~ :+1: 
I've made a PR for the docs and updated this to be a feature request to make the `70` configurable when using flow_detail 3
> `content_view_lines_cutoff` is only for mitmproxy ui, I guess we need to introduce a new option for the dumper and default to `0` (dump everything) I guess?

We should make it an option for all tools then and reuse it in mitmdump. :)

Resolving patch:
diff --git a/mitmproxy/addons/dumper.py b/mitmproxy/addons/dumper.py
--- a/mitmproxy/addons/dumper.py
+++ b/mitmproxy/addons/dumper.py
@@ -18,6 +18,7 @@
 from mitmproxy import http
 from mitmproxy.contrib import click as miniclick
 from mitmproxy.net.dns import response_codes
+from mitmproxy.options import CONTENT_VIEW_LINES_CUTOFF
 from mitmproxy.tcp import TCPFlow
 from mitmproxy.tcp import TCPMessage
 from mitmproxy.udp import UDPFlow
@@ -54,12 +55,12 @@ def load(self, loader):
             "flow_detail",
             int,
             1,
-            """
+            f"""
             The display detail level for flows in mitmdump: 0 (quiet) to 4 (very verbose).
               0: no output
               1: shortened request URL with response status code
               2: full request URL with response status code and HTTP headers
-              3: 2 + truncated response content, content of WebSocket and TCP messages
+              3: 2 + truncated response content, content of WebSocket and TCP messages (content_view_lines_cutoff: {CONTENT_VIEW_LINES_CUTOFF})
               4: 3 + nothing is truncated
             """,
         )
@@ -125,7 +126,9 @@ def _echo_message(
             logging.debug(error)
 
         if ctx.options.flow_detail == 3:
-            lines_to_echo = itertools.islice(lines, 70)
+            lines_to_echo = itertools.islice(
+                lines, ctx.options.content_view_lines_cutoff
+            )
         else:
             lines_to_echo = lines
 


Test patch:
diff --git a/test/mitmproxy/addons/test_dumper.py b/test/mitmproxy/addons/test_dumper.py
--- a/test/mitmproxy/addons/test_dumper.py
+++ b/test/mitmproxy/addons/test_dumper.py
@@ -97,7 +97,7 @@ def test_simple():
 def test_echo_body():
     f = tflow.tflow(resp=True)
     f.response.headers["content-type"] = "text/html"
-    f.response.content = b"foo bar voing\n" * 100
+    f.response.content = b"foo bar voing\n" * 600
 
     sio = io.StringIO()
     d = dumper.Dumper(sio)
@@ -108,6 +108,21 @@ def test_echo_body():
         assert "cut off" in t
 
 
+def test_echo_body_custom_cutoff():
+    f = tflow.tflow(resp=True)
+    f.response.headers["content-type"] = "text/html"
+    f.response.content = b"foo bar voing\n" * 4
+
+    sio = io.StringIO()
+    d = dumper.Dumper(sio)
+    with taddons.context(d) as ctx:
+        ctx.configure(d, flow_detail=3)
+        ctx.configure(d, content_view_lines_cutoff=3)
+        d._echo_message(f.response, f)
+        t = sio.getvalue()
+        assert "cut off" in t
+
+
 def test_echo_trailer():
     sio = io.StringIO()
     d = dumper.Dumper(sio)
@@ -118,7 +133,7 @@ def test_echo_trailer():
         f.request.headers["content-type"] = "text/html"
         f.request.headers["transfer-encoding"] = "chunked"
         f.request.headers["trailer"] = "my-little-request-trailer"
-        f.request.content = b"some request content\n" * 100
+        f.request.content = b"some request content\n" * 600
         f.request.trailers = Headers(
             [(b"my-little-request-trailer", b"foobar-request-trailer")]
         )