Repository: zulip/zulip

Issue:
Slack Incoming integration ignores fields
I am trying to use a Slack integration from Uptime Kuma and when the message comes in it seems to post only the main text and none of the blocks. I decided to try the example from Slack's website and, in-particular, it seems that the `fields` portion is getting skipped over.

Here is the example JSON I'm using

```json
{
  "text": "Message",
  "blocks": [
    {
      "type": "header",
      "text": {
        "type": "plain_text",
        "text": "New request"
      }
    },
    {
      "type": "section",
      "fields": [
        {
          "type": "mrkdwn",
          "text": "*Type:*\nPaid Time Off"
        },
        {
          "type": "mrkdwn",
          "text": "*Created by:*\n<example.com|Fred Enriquez>"
        }
      ]
    },
    {
      "type": "section",
      "fields": [
        {
          "type": "mrkdwn",
          "text": "*When:*\nAug 10 - Aug 13"
        }
      ]
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "<https://example.com|View request>"
      }
    }
  ]
}
```

The message it posts is simply this

<img width="191" alt="CleanShot 2022-06-09 at 07 44 16@2x" src="https://user-images.githubusercontent.com/125599/172838959-09217b8e-1eef-4d19-9883-c88227dafdf6.png">

It's only getting the last section where it has a `text` set. All the previous sections where it includes `fields` don't show up. I tried to convert the prior section from this

```json
    {
      "type": "section",
      "fields": [
        {
          "type": "mrkdwn",
          "text": "*When:*\nAug 10 - Aug 13"
        }
      ]
    },
```

to this

```json
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*When:*\nAug 10 - Aug 13"
      }
    },
```

and it properly posted that text as well

<img width="120" alt="CleanShot 2022-06-09 at 07 45 41@2x" src="https://user-images.githubusercontent.com/125599/172839156-eeb3c427-441f-400a-ab6c-292fb5ba7d16.png">

So it feels like those with `fields` are getting skipped over. Uptime Kuma is posting all the relevant information in sections with fields, so I'm not getting much of a message when it tries to post.

```javascript
{
    "type": "section",
    "fields": [{
        "type": "mrkdwn",
        "text": "*Message*\n" + msg,
    },
    {
        "type": "mrkdwn",
        "text": "*Time (UTC)*\n" + time,
    }],
}],
```

Let me know if I can provide more information or where I can help. Thanks.


Discussion:
Zulip currently does skip over `fields`, only handling `text` and `accessory` pieces: https://github.com/zulip/zulip/blob/d0c5389d60c7232405d3e01710983f46ab245ff5/zerver/webhooks/slack_incoming/view.py#L76-L87

I pushed #22258 recently, which adds more supported syntax, but still doesn't address `fields`.

The difficulty with the [`fields` of the `section` type](https://api.slack.com/reference/block-kit/blocks#section) is that they're laid out two-columns wide, side-by-side.  We can't easily replicate that layout with Zulip's markdown.  We can render them sequentially, but that means that people that are using them as _columns_ will get nonsense.  That is, if the user intends:

| *Severity* | *Component* |
|-----------|---------------|
| Critical | Web |

rendering them sequentially will yield the garbled:
> *Severity*
> *Component*
> Critical
> Web

And it's impossible to know, a priori, if the webhook is trying to read across or down.
Actually, the help popup says you can make tables in Zulip markdown.
Zulip does support markdown tables, which are an extension to Markdown.  Unfortunately, [Python's Markdown tables](https://python-markdown.github.io/extensions/tables/) _require_ a header, which means that if the user is intending the _left_ column to be the headers, the rendering will be misleading and wrong.  You can see this with how Github renders the table, in that the first row is bold, and centered:

| Severity: | Critical |
|-----------|---------------|
| Which component is affected: | Rending in desktop clients |

You can hack around this by leaving the "header" row blank:
| | |
|-----------|---------------|
| Severity: | Critical |
| Which component is affected: | Rending in desktop clients |

But the other issue is that line breaks are not possible in Markdown tables.  So while Slack supports [this](https://api.slack.com/tools/block-kit-builder?blocks=%5B%7B%22type%22:%22section%22,%22fields%22:%5B%7B%22type%22:%22mrkdwn%22,%22text%22:%22*Severity*%22%7D,%7B%22type%22:%22mrkdwn%22,%22text%22:%22*Bad*%5CnAlso,%20worse%5CnAnd%20verry%20verry%20sad%22%7D,%7B%22type%22:%22mrkdwn%22,%22text%22:%22*Component*%22%7D,%7B%22type%22:%22mrkdwn%22,%22text%22:%22Web%22%7D%5D%7D%5D):
![Screen Shot 2022-06-20 at 3 48 37 PM](https://user-images.githubusercontent.com/28347/174686186-2026ba20-763a-4a29-9c12-a458be5d0f4a.png)


You can't do the equivalent in Markdown -- note that we need to pad the left column with empty cells, effectively, which gets complicated pretty quickly.
| | |
|-----------|---------------|
| *Severity* | *Bad* |
|                   | Also, worse |
|                   | And very, very sad |
| *Component* | Rending in desktop clients |

Yes, I see these are all valid concerns - but honestly I'd much prefer to have the information there at all, because currently the webhook is almost useless, and worry about the layout later.
Hello @zulip/server-integrations members, this issue was labeled with the "area: integrations" label, so you may want to check it out!

<!-- areaLabelAddition -->

This issue has worsened: Now we just get "Uptime Kuma Alert" without _any_ information :/
Hm, sounds like it would be worth implementing something here, even if the formatting is flawed.

@alexmv Would a table with a blank header + replacing `\n` with a space be an acceptable workaround, do you think?

Resolving patch:
diff --git a/zerver/webhooks/slack_incoming/view.py b/zerver/webhooks/slack_incoming/view.py
--- a/zerver/webhooks/slack_incoming/view.py
+++ b/zerver/webhooks/slack_incoming/view.py
@@ -1,6 +1,7 @@
 # Webhooks for external integrations.
 import re
-from typing import Optional
+from itertools import zip_longest
+from typing import Literal, Optional, TypedDict, cast
 
 from django.http import HttpRequest, HttpResponse
 from django.utils.translation import gettext as _
@@ -14,6 +15,7 @@
     WildValue,
     check_dict,
     check_int,
+    check_list,
     check_string,
     check_string_in,
     check_url,
@@ -98,17 +100,17 @@ def render_block(block: WildValue) -> str:
             if element_type == "image":
                 pieces.append(render_block_element(element))
             else:
-                pieces.append(element.tame(check_text_block()))
+                pieces.append(element.tame(check_text_block())["text"])
         return "\n\n".join(piece.strip() for piece in pieces if piece.strip() != "")
     elif block_type == "divider":
         return "----"
     elif block_type == "header":
-        return "## " + block["text"].tame(check_text_block(plain_text_only=True))
+        return "## " + block["text"].tame(check_text_block(plain_text_only=True))["text"]
     elif block_type == "image":
         image_url = block["image_url"].tame(check_url)
         alt_text = block["alt_text"].tame(check_string)
         if "title" in block:
-            alt_text = block["title"].tame(check_text_block(plain_text_only=True))
+            alt_text = block["title"].tame(check_text_block(plain_text_only=True))["text"]
         return f"[{alt_text}]({image_url})"
     elif block_type == "input":
         # Unhandled
@@ -116,30 +118,50 @@ def render_block(block: WildValue) -> str:
     elif block_type == "section":
         pieces = []
         if "text" in block:
-            pieces.append(block["text"].tame(check_text_block()))
+            pieces.append(block["text"].tame(check_text_block())["text"])
 
         if "accessory" in block:
             pieces.append(render_block_element(block["accessory"]))
 
         if "fields" in block:
-            # TODO -- these should be rendered in two columns,
-            # left-to-right.  We could render them sequentially,
-            # except some may be Title1 / Title2 / value1 / value2,
-            # which would be nonsensical when rendered sequentially.
-            pass
+            fields = block["fields"].tame(check_list(check_text_block()))
+            if len(fields) == 1:
+                # Special-case a single field to display a bit more
+                # nicely, without extraneous borders and limitations
+                # on its contents.
+                pieces.append(fields[0]["text"])
+            else:
+                # It is not possible to have newlines in a table, nor
+                # escape the pipes that make it up; replace them with
+                # whitespace.
+                field_text = [f["text"].replace("\n", " ").replace("|", " ") for f in fields]
+                # Because Slack formats this as two columns, but not
+                # necessarily a table with a bold header, we emit a
+                # blank header row first.
+                table = "| | |\n|-|-|\n"
+                # Then take the fields two-at-a-time to make the table
+                iters = [iter(field_text)] * 2
+                for left, right in zip_longest(*iters, fillvalue=""):
+                    table += f"| {left} | {right} |\n"
+                pieces.append(table)
 
         return "\n\n".join(piece.strip() for piece in pieces if piece.strip() != "")
 
     return ""
 
 
-def check_text_block(plain_text_only: bool = False) -> Validator[str]:
+class TextField(TypedDict):
+    text: str
+    type: Literal["plain_text", "mrkdwn"]
+
+
+def check_text_block(plain_text_only: bool = False) -> Validator[TextField]:
     if plain_text_only:
         type_validator = check_string_in(["plain_text"])
     else:
-        type_validator = check_string
+        type_validator = check_string_in(["plain_text", "mrkdwn"])
 
-    def f(var_name: str, val: object) -> str:
+    def f(var_name: str, val: object) -> TextField:
         block = check_dict(
             [
                 ("type", type_validator),
@@ -147,15 +169,7 @@ def f(var_name: str, val: object) -> str:
             ],
         )(var_name, val)
 
-        # We can't use `value_validator=check_string` above to let
-        # mypy know this is a str, because there's an optional boolean
-        # `emoji` key which can appear -- hence the assert.
-        text = block["text"]
-        assert isinstance(text, str)
-
-        # Ideally we would escape the content if it was plain text,
-        # but out flavor of Markdown doesn't support escapes. :(
-        return text
+        return cast(TextField, block)
 
     return f
 


Test patch:
diff --git a/zerver/webhooks/slack_incoming/tests.py b/zerver/webhooks/slack_incoming/tests.py
--- a/zerver/webhooks/slack_incoming/tests.py
+++ b/zerver/webhooks/slack_incoming/tests.py
@@ -74,6 +74,9 @@ def test_message_with_actions(self) -> None:
 [Overlook Hotel](https://google.com) \n :star: \n Doors had too many axe holes, guest in room 237 was far too rowdy, whole place felt stuck in the 1920s.
 
 [Haunted hotel image](https://is5-ssl.mzstatic.com/image/thumb/Purple3/v4/d3/72/5c/d3725c8f-c642-5d69-1904-aa36e4297885/source/256x256bb.jpg)
+
+**Average Rating**
+1.0
 """.strip()
 
         self.check_webhook(
@@ -90,6 +93,9 @@ def test_message_with_blocks(self) -> None:
 [Overlook Hotel](https://example.com) \n :star: \n Doors had too many axe holes, guest in room 237 was far too rowdy, whole place felt stuck in the 1920s.
 
 [Haunted hotel image](https://is5-ssl.mzstatic.com/image/thumb/Purple3/v4/d3/72/5c/d3725c8f-c642-5d69-1904-aa36e4297885/source/256x256bb.jpg)
+
+**Average Rating**
+1.0
 """.strip()
 
         self.check_webhook(
@@ -175,6 +181,12 @@ def test_attachment_blocks(self) -> None:
 [cute cat](https://pbs.twimg.com/profile_images/625633822235693056/lNGUneLX_400x400.jpg)
 
 This is a section block with a button.
+
+| | |
+|-|-|
+| one | two |
+| three | four |
+| five |  |
         """.strip()
 
         self.check_webhook(