Repository: mitmproxy/mitmproxy

Issue:
mitmweb: Add support for syntax highlighting / coloring
#### Problem Description
When selecting "View: msgpack" on a msgpack request or response, the resultant JSON is displayed using single-quotes (`'`), which is not valid. It should use double-quotes (`"`) instead.

#### Steps to reproduce the behavior:
1. Have a request/response in `mitmweb` and click Replace and use the file contained in this archive: [get_resource_version.zip](https://github.com/mitmproxy/mitmproxy/files/9692381/get_resource_version.zip)
2. Select "View: msgpack" next to the Replace button
3. Notice that the resultant JSON contains single-quoted properties and strings:
```json
{
    'data_headers': {
        'result_code': 1
    },
    'data': {
        'resource_version': 'y2XM6giU6zz56wCm'
    }
}
```

I am not an expert on the JSON standard, but pasting the output into a [JSON validator](https://duckduckgo.com/?q=json+validator&t=ffab&ia=answer) throws up errors which can be fixed by using double quotes (and I assume that's what the red highlighting means, since the above code block starts with ```json)

#### System Information
```
>mitmproxy --version
Mitmproxy: 8.1.1
Python:    3.10.6
OpenSSL:   OpenSSL 1.1.1n  15 Mar 2022
Platform:  Windows-10-10.0.22622-SP0
```

#### P.S.
I know a bit of Python but am unfamiliar with this codebase, however I would be happy to try fixing this myself and submitting a PR if that would be easier? It seems like something that would be simple enough for a first time contributor to attempt.


Discussion:
I don't know if this is _supposed_ to be valid JSON. First and foremost this is a human readable version of msgpack that happens to look somewhat like JSON.

The code is here

https://github.com/mitmproxy/mitmproxy/blob/a1631154d722ea4c82dcb72e0c84ad38c6991de5/mitmproxy/contentviews/msgpack.py#L18-L30

I don't know if everything that msgpack does _can_ be represented in JSON (e.g. bytes). If so, then this code could basically just use `json.dumps`.

@tasn can you comment on that please?
So yeah, msgpack is _not_ JSON compatible. E.g. map keys can have _any_ type. I'm closing, feel free to re-open if I'm missing something.
Ah, that makes sense about them not being the same especially with the bytes. 

My use case was more pasting them into syntax highlighting things which expect the double quotes, rather than parsing the json (there are msgpack deserialisers for that). My msgpack data usually only used primitive types. I can just continue using a simple find-and-replace though, so I don’t have any objection to you closing.
If you're interested in contributing, I'm sure adding coloring to the mspack view would be a great addition.

https://github.com/mitmproxy/mitmproxy/blob/a1631154d722ea4c82dcb72e0c84ad38c6991de5/mitmproxy/contentviews/msgpack.py#L18-L30

Here's how this works for JSON

https://github.com/mitmproxy/mitmproxy/blob/a1631154d722ea4c82dcb72e0c84ad38c6991de5/mitmproxy/contentviews/json.py#L20-L38

You essentially need to use `TViewLine` and mostly in the last else add tokens for the corresponding types instead of treating all the same:

https://github.com/mitmproxy/mitmproxy/blob/a1631154d722ea4c82dcb72e0c84ad38c6991de5/mitmproxy/contentviews/msgpack.py#L29-L30

But mitmweb is currently missing the colors that the terminal uses https://github.com/mitmproxy/mitmproxy/blob/cd4a74fae7cbd8119afc3900597f798ec1604db7/mitmproxy/tools/console/palettes.py#L149-L225
That should be somewhat easy to add as well, but it can be a separate step. (TIL mitmproxy has themes?)

I'm going to re-open this but as a feature request to add syntax highlighting support to mitmweb. Everything is already there (e.g. it adds things like `class="json_string"`) but it doesn't do anything yet. So if the classes/colors where there it would already work for JSON. For msgpack it lacks it in the contentview.
@Prinzhorn, yeah, adding coloring would indeed be a great addition!

As for JSON: yeah, msgpack is more complex than JSON and you can't represent it as valid json.
I’m happy to contribute syntax highlighting as described. Thank you for taking the time to provide the relevant code snippets. I have a fair bit on my plate right now so it might be a while before I have a chance to look at it. 

I think I understand roughly what needs to be done, but one question I had was how this would be tested — can you actually get data from the content view with colour information inside of it? Or would it be a screenshot test?
> how this would be tested — can you actually get data from the content view with colour information inside of it? Or would it be a screenshot test?

Check out the tests for other contentviews https://github.com/mitmproxy/mitmproxy/tree/a1631154d722ea4c82dcb72e0c84ad38c6991de5/test/mitmproxy/contentviews

From what I can tell none of them test the coloring @mhils ?

If you wanted to test the coloring, you should be able to test the output of `pretty` for different inputs and make sure the expected parts have the expected tokens (strings, numbers, etc.)

Also again I think there are two separate things to be done:

1. Add coloring to the msgpack view by using `TViewLine` and adding the appropriate tokens to the list (strings, numbers, etc.)
2. Make coloring work in mitmweb. In general mitmweb already adds `<span>`s with the token classes already. But I don't know what's the best way to get the colors into mitmweb without doing it manually.
> From what I can tell none of them test the coloring @mhils ?

I don't think so, but I haven't checked. :)

> Make coloring work in mitmweb. In general mitmweb already adds <span>s with the token classes already. But I don't know what's the best way to get the colors into mitmweb without doing it manually.

Common tokens have CSS class equivalents already (and we can add more there). Let me know if that doesn't make sense! :)

Resolving patch:
diff --git a/mitmproxy/contentviews/json.py b/mitmproxy/contentviews/json.py
--- a/mitmproxy/contentviews/json.py
+++ b/mitmproxy/contentviews/json.py
@@ -28,11 +28,14 @@ def format_json(data: Any) -> Iterator[base.TViewLine]:
             yield current_line
             current_line = []
         if re.match(r'\s*"', chunk):
-            current_line.append(("json_string", chunk))
+            if len(current_line) == 1 and current_line[0][0] == "text" and current_line[0][1].isspace():
+                current_line.append(("Token_Name_Tag", chunk))
+            else:
+                current_line.append(("Token_Literal_String", chunk))
         elif re.match(r"\s*\d", chunk):
-            current_line.append(("json_number", chunk))
+            current_line.append(("Token_Literal_Number", chunk))
         elif re.match(r"\s*(true|null|false)", chunk):
-            current_line.append(("json_boolean", chunk))
+            current_line.append(("Token_Keyword_Constant", chunk))
         else:
             current_line.append(("text", chunk))
     yield current_line
diff --git a/mitmproxy/contentviews/msgpack.py b/mitmproxy/contentviews/msgpack.py
--- a/mitmproxy/contentviews/msgpack.py
+++ b/mitmproxy/contentviews/msgpack.py
@@ -15,23 +15,62 @@ def parse_msgpack(s: bytes) -> Any:
         return PARSE_ERROR
 
 
-def pretty(value, htchar="    ", lfchar="\n", indent=0):
-    nlch = lfchar + htchar * (indent + 1)
-    if type(value) is dict:
-        items = [
-            nlch + repr(key) + ": " + pretty(value[key], htchar, lfchar, indent + 1)
-            for key in value
-        ]
-        return "{%s}" % (",".join(items) + lfchar + htchar * indent)
-    elif type(value) is list:
-        items = [nlch + pretty(item, htchar, lfchar, indent + 1) for item in value]
-        return "[%s]" % (",".join(items) + lfchar + htchar * indent)
-    else:
-        return repr(value)
+def format_msgpack(data: Any, output = None, indent_count: int = 0) -> list[base.TViewLine]:
+    if output is None:
+        output = [[]]
+
+    indent = ("text", "    " * indent_count)
+
+    if type(data) is str:
+        token = [("Token_Literal_String", f"\"{data}\"")]
+        output[-1] += token
+
+        # Need to return if single value, but return is discarded in dict/list loop
+        return output
+
+    elif type(data) is float or type(data) is int:
+        token = [("Token_Literal_Number", repr(data))]
+        output[-1] += token
+
+        return output
+
+    elif type(data) is bool:
+        token = [("Token_Keyword_Constant", repr(data))]
+        output[-1] += token
+
+        return output
+
+    elif type(data) is dict:
+        output[-1] += [("text", "{")]
+        for key in data:
+            output.append([indent, ("text", "    "), ("Token_Name_Tag", f'"{key}"'), ("text", ": ")])
+            format_msgpack(data[key], output, indent_count + 1)
 
+            if key != list(data)[-1]:
+                output[-1] += [("text", ",")]
+
+        output.append([indent, ("text", "}")])
+
+        return output
+
+    elif type(data) is list:
+        output[-1] += [("text", "[")]
+        for item in data:
+            output.append([indent, ("text", "    ")])
+            format_msgpack(item, output, indent_count + 1)
+
+            if item != data[-1]:
+                output[-1] += [("text", ",")]
+
+        output.append([indent, ("text", "]")])
+
+        return output
+
+    else:
+        token = [("text", repr(data))]
+        output[-1] += token
 
-def format_msgpack(data):
-    return base.format_text(pretty(data))
+        return output
 
 
 class ViewMsgPack(base.View):
diff --git a/mitmproxy/tools/console/palettes.py b/mitmproxy/tools/console/palettes.py
--- a/mitmproxy/tools/console/palettes.py
+++ b/mitmproxy/tools/console/palettes.py
@@ -69,10 +69,11 @@ class Palette:
         "mark",
         # Hex view
         "offset",
-        # JSON view
-        "json_string",
-        "json_number",
-        "json_boolean",
+        # JSON/msgpack view
+        "Token_Name_Tag",
+        "Token_Literal_String",
+        "Token_Literal_Number",
+        "Token_Keyword_Constant",
         # TCP flow details
         "from_client",
         "to_client",
@@ -207,10 +208,11 @@ class LowDark(Palette):
         mark=("light red", "default"),
         # Hex view
         offset=("dark cyan", "default"),
-        # JSON view
-        json_string=("dark blue", "default"),
-        json_number=("light magenta", "default"),
-        json_boolean=("dark magenta", "default"),
+        # JSON/msgpack view
+        Token_Name_Tag=("dark green", "default"),
+        Token_Literal_String=("dark blue", "default"),
+        Token_Literal_Number=("light magenta", "default"),
+        Token_Keyword_Constant=("dark magenta", "default"),
         # TCP flow details
         from_client=("light blue", "default"),
         to_client=("light red", "default"),
@@ -306,10 +308,11 @@ class LowLight(Palette):
         mark=("dark red", "default"),
         # Hex view
         offset=("dark blue", "default"),
-        # JSON view
-        json_string=("dark blue", "default"),
-        json_number=("light magenta", "default"),
-        json_boolean=("dark magenta", "default"),
+        # JSON/msgpack view
+        Token_Name_Tag=("dark green", "default"),
+        Token_Literal_String=("dark blue", "default"),
+        Token_Literal_Number=("light magenta", "default"),
+        Token_Keyword_Constant=("dark magenta", "default"),
         # TCP flow details
         from_client=("dark blue", "default"),
         to_client=("dark red", "default"),
@@ -427,10 +430,11 @@ class SolarizedLight(LowLight):
         ),
         # Hex view
         offset=(sol_cyan, "default"),
-        # JSON view
-        json_string=(sol_cyan, "default"),
-        json_number=(sol_blue, "default"),
-        json_boolean=(sol_magenta, "default"),
+        # JSON/msgpack view
+        Token_Name_Tag=(sol_green, "default"),
+        Token_Literal_String=(sol_cyan, "default"),
+        Token_Literal_Number=(sol_blue, "default"),
+        Token_Keyword_Constant=(sol_magenta, "default"),
         # TCP flow details
         from_client=(sol_blue, "default"),
         to_client=(sol_red, "default"),
@@ -506,10 +510,11 @@ class SolarizedDark(LowDark):
         ),
         # Hex view
         offset=(sol_cyan, "default"),
-        # JSON view
-        json_string=(sol_cyan, "default"),
-        json_number=(sol_blue, "default"),
-        json_boolean=(sol_magenta, "default"),
+        # JSON/msgpack view
+        Token_Name_Tag=(sol_green, "default"),
+        Token_Literal_String=(sol_cyan, "default"),
+        Token_Literal_Number=(sol_blue, "default"),
+        Token_Keyword_Constant=(sol_magenta, "default"),
         # TCP flow details
         from_client=(sol_blue, "default"),
         to_client=(sol_red, "default"),


Test patch:
diff --git a/test/mitmproxy/contentviews/test_json.py b/test/mitmproxy/contentviews/test_json.py
--- a/test/mitmproxy/contentviews/test_json.py
+++ b/test/mitmproxy/contentviews/test_json.py
@@ -17,6 +17,32 @@ def test_parse_json():
 
 def test_format_json():
     assert list(json.format_json({"data": ["str", 42, True, False, None, {}, []]}))
+    assert list(json.format_json({"string": "test"})) == [
+        [('text', '{'), ('text', '')],
+        [('text', '    '), ('Token_Name_Tag', '"string"'), ('text', ': '), ('Token_Literal_String', '"test"'), ('text', '')],
+        [('text', ''), ('text', '}')]]
+    assert list(json.format_json({"num": 4})) == [
+        [('text', '{'), ('text', '')],
+        [('text', '    '), ('Token_Name_Tag', '"num"'), ('text', ': '), ('Token_Literal_Number', '4'), ('text', '')],
+        [('text', ''), ('text', '}')]]
+    assert list(json.format_json({"bool": True})) == [
+        [('text', '{'), ('text', '')],
+        [('text', '    '), ('Token_Name_Tag', '"bool"'), ('text', ': '), ('Token_Keyword_Constant', 'true'), ('text', '')],
+        [('text', ''), ('text', '}')]]
+    assert list(json.format_json({"object": {"int": 1}})) == [
+        [('text', '{'), ('text', '')],
+        [('text', '    '), ('Token_Name_Tag', '"object"'), ('text', ': '), ('text', '{'), ('text', '')],
+        [('text', '        '), ('Token_Name_Tag', '"int"'), ('text', ': '), ('Token_Literal_Number', '1'), ('text', '')],
+        [('text', '    '), ('text', '}'), ('text', '')],
+        [('text', ''), ('text', '}')]]
+    assert list(json.format_json({"list": ["string", 1, True]})) == [
+        [('text', '{'), ('text', '')],
+        [('text', '    '), ('Token_Name_Tag', '"list"'), ('text', ': '), ('text', '[')],
+        [('Token_Literal_String', '        "string"'), ('text', ',')],
+        [('Token_Literal_Number', '        1'), ('text', ',')],
+        [('Token_Keyword_Constant', '        true'), ('text', '')],
+        [('text', '    '), ('text', ']'), ('text', '')],
+        [('text', ''), ('text', '}')]]
 
 
 def test_view_json():
diff --git a/test/mitmproxy/contentviews/test_msgpack.py b/test/mitmproxy/contentviews/test_msgpack.py
--- a/test/mitmproxy/contentviews/test_msgpack.py
+++ b/test/mitmproxy/contentviews/test_msgpack.py
@@ -18,9 +18,32 @@ def test_parse_msgpack():
 
 
 def test_format_msgpack():
-    assert list(
-        msgpack.format_msgpack({"data": ["str", 42, True, False, None, {}, []]})
-    )
+    assert list(msgpack.format_msgpack({"string": "test", "int": 1, "float": 1.44, "bool": True})) == [
+        [('text', '{')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"string"'), ('text', ': '), ('Token_Literal_String', '"test"'), ('text', ',')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"int"'), ('text', ': '), ('Token_Literal_Number', '1'), ('text', ',')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"float"'), ('text', ': '), ('Token_Literal_Number', '1.44'), ('text', ',')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"bool"'), ('text', ': '), ('Token_Keyword_Constant', 'True')],
+        [('text', ''), ('text', '}')]
+    ]
+
+    assert list(msgpack.format_msgpack({"object": {"key": "value"}, "list": [1]})) == [
+        [('text', '{')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"object"'), ('text', ': '), ('text', '{')],
+        [('text', '    '), ('text', '    '), ('Token_Name_Tag', '"key"'), ('text', ': '), ('Token_Literal_String', '"value"')],
+        [('text', '    '), ('text', '}'), ('text', ',')],
+        [('text', ''), ('text', '    '), ('Token_Name_Tag', '"list"'), ('text', ': '), ('text', '[')],
+        [('text', '    '), ('text', '    '), ('Token_Literal_Number', '1')],
+        [('text', '    '), ('text', ']')],
+        [('text', ''), ('text', '}')]]
+
+    assert list(msgpack.format_msgpack('string')) == [[('Token_Literal_String', '"string"')]]
+
+    assert list(msgpack.format_msgpack(1.2)) == [[('Token_Literal_Number', '1.2')]]
+
+    assert list(msgpack.format_msgpack(True)) == [[('Token_Keyword_Constant', 'True')]]
+
+    assert list(msgpack.format_msgpack(b'\x01\x02\x03')) == [[('text', "b'\\x01\\x02\\x03'")]]
 
 
 def test_view_msgpack():