Repository: zulip/zulip

Issue:
Unclear message authors in notification emails
Background: a self-hosted version updated sometime in 2020. Notification emails on missed messages enabled.

The problem: text/plain alternative doesn't adequately mark authors. Example based on a real email with 3 messages, adjusted with random text and names to maintain privacy:

```
Person1: It looks like this is your first time opening an issue in this project!  Be sure to review the contributing guidelines and code of conduct.
Person2: @_**Person1** https://github.com/zulip/zulip/issues/new:
quote
Reporting a security vulnerability?

Check out the project's security policy.
@_**Person3** Remember, contributions to this repository should follow its contributing guidelines and code of conduct. 
Person4: https://github.com/zulip/zulip/pulls
```

It's easy to miss that there were 3 people in the discussion (I thought it was 2 until I started editing the text), because the author names (`Person1:`, `Person2:`, `Person4`) are all plain.

Suggested solution: send each message as a separate MIME part - preferably separate email, because I'm not sure how well email clients deal with multiple inline text parts in a single email. The reason is that messages can carry arbitrary text, so there's almost always a way to craft a message that appears as multiple ones in the notification email.

Suggested workaround: To solve a lesser version of this (messages accidentally illegible, rather than purposely spoofed), there should be some delineation between separate messages. An additional space should do the trick, or a couple dashes as a visual separator. Perhaps author's name can be combined with an indentation, a symbol, or a timestamp like IRC does.


Discussion:
An elegant solution would be prefixing every line in each message with `>` which email clients automatically render as quotes, so e.g.

```
Person 1:
> Hello world

Person 2:
> Something
> Something more
```

It however wouldn't prevent spoofing because 1. emails can entirely be spoofed and 2. Zulip allows multiple users to have the same name (see #15688).
Hello @zulip/server-development members, this issue was labeled with the "area: emails" label, so you may want to check it out!

<!-- areaLabelAddition -->
When I mentioned spoofing, I didn't mean crafting an email, but crafting a Zulip message to appear as something else than it is.

Prefixing with ">" would make the contents unspoofable: there's no way to create a Zulip message that would end up without ">" when inside an email.

">" itself, on the other hand, would make the UX just a little bit worse by turning the notifications into a quote delivery mechanism. Emails are meant to contain the content, not just quotes (except mailing list digests, which those notifications don't seem to be). Related: the email sender is always the same, even when the email only contains one message. This makes the emails ever so slightly less useful.

Also related, but kinda rant: placing multiple messages into a single email is like trying to push a square peg through a round hole. The result loses something in relation to a forwarded message (message separation, search by From), but it's not quite a mailing list digest either (notifications coming within minutes of each other, often with just one message inside).
@zulipbot claim.

Resolving patch:
diff --git a/zerver/lib/email_notifications.py b/zerver/lib/email_notifications.py
--- a/zerver/lib/email_notifications.py
+++ b/zerver/lib/email_notifications.py
@@ -170,6 +170,19 @@ def fix_spoilers_in_text(content: str, language: str) -> str:
     return "\n".join(output)
 
 
+def add_quote_prefix_in_text(content: str) -> str:
+    """
+    We add quote prefix ">" to each line of the message in plain text
+    format, such that email clients render the message as quote.
+    """
+    lines = content.split("\n")
+    output = []
+    for line in lines:
+        quoted_line = f"> {line}"
+        output.append(quoted_line)
+    return "\n".join(output)
+
+
 def build_message_list(
     user: UserProfile,
     messages: List[Message],
@@ -197,7 +210,7 @@ def fix_plaintext_image_urls(content: str) -> str:
     def prepend_sender_to_message(
         message_plain: str, message_html: str, sender: str
     ) -> Tuple[str, str]:
-        message_plain = f"{sender}: {message_plain}"
+        message_plain = f"{sender}:\n{message_plain}"
         message_soup = BeautifulSoup(message_html, "html.parser")
         sender_name_soup = BeautifulSoup(f"<b>{sender}</b>: ", "html.parser")
         first_tag = message_soup.find()
@@ -219,6 +232,7 @@ def build_message_payload(message: Message, sender: Optional[str] = None) -> Dic
         # plain text.
         plain = re.sub(r"/user_uploads/(\S*)", user.realm.uri + r"/user_uploads/\1", plain)
         plain = fix_spoilers_in_text(plain, user.default_language)
+        plain = add_quote_prefix_in_text(plain)
 
         assert message.rendered_content is not None
         fragment = lxml.html.fragment_fromstring(message.rendered_content, create_parent=True)


Test patch:
diff --git a/zerver/tests/test_email_notifications.py b/zerver/tests/test_email_notifications.py
--- a/zerver/tests/test_email_notifications.py
+++ b/zerver/tests/test_email_notifications.py
@@ -517,7 +517,7 @@ def _extra_context_in_missed_stream_messages_mention(
 
         if show_message_content:
             verify_body_include = [
-                "Othello, the Moor of Venice: 1 2 3 4 5 6 7 8 9 10 @**King Hamlet** -- ",
+                "Othello, the Moor of Venice: > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > 9 > 10 > @**King Hamlet** -- ",
                 "You are receiving this because you were mentioned in Zulip Dev.",
             ]
             email_subject = "#Denmark > test"
@@ -560,7 +560,7 @@ def _extra_context_in_missed_stream_messages_wildcard_mention(
 
         if show_message_content:
             verify_body_include = [
-                "Othello, the Moor of Venice: 1 2 3 4 5 @**all** -- ",
+                "Othello, the Moor of Venice: > 1 > 2 > 3 > 4 > 5 > @**all** -- ",
                 "You are receiving this because you were mentioned in Zulip Dev.",
             ]
             email_subject = "#Denmark > test"
@@ -598,7 +598,7 @@ def _extra_context_in_missed_stream_messages_email_notify(self, send_as_user: bo
         self.send_stream_message(self.example_user("othello"), "Denmark", "11", topic_name="test2")
         msg_id = self.send_stream_message(self.example_user("othello"), "denmark", "12")
         verify_body_include = [
-            "Othello, the Moor of Venice: 1 2 3 4 5 6 7 8 9 10 12 -- ",
+            "Othello, the Moor of Venice: > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > 9 > 10 > 12 -- ",
             "You are receiving this because you have email notifications enabled for this stream.",
         ]
         email_subject = "#Denmark > test"
@@ -618,7 +618,7 @@ def _extra_context_in_missed_stream_messages_mention_two_senders(
             self.example_user("othello"), "Denmark", "@**King Hamlet**"
         )
         verify_body_include = [
-            "Cordelia, Lear's daughter: 0 1 2 Othello, the Moor of Venice: @**King Hamlet** -- ",
+            "Cordelia, Lear's daughter: > 0 > 1 > 2 Othello, the Moor of Venice: > @**King Hamlet** -- ",
             "You are receiving this because you were mentioned in Zulip Dev.",
         ]
         email_subject = "#Denmark > test"
@@ -626,7 +626,7 @@ def _extra_context_in_missed_stream_messages_mention_two_senders(
             msg_id, verify_body_include, email_subject, send_as_user, trigger="mentioned"
         )
 
-    def _extra_context_in_personal_missed_stream_messages(
+    def _extra_context_in_missed_personal_messages(
         self,
         send_as_user: bool,
         show_message_content: bool = True,
@@ -640,7 +640,7 @@ def _extra_context_in_personal_missed_stream_messages(
         )
 
         if show_message_content:
-            verify_body_include = ["Extremely personal message!"]
+            verify_body_include = ["> Extremely personal message!"]
             email_subject = "PMs with Othello, the Moor of Venice"
             verify_body_does_not_include: List[str] = []
         else:
@@ -675,7 +675,7 @@ def _extra_context_in_personal_missed_stream_messages(
             verify_body_does_not_include=verify_body_does_not_include,
         )
 
-    def _reply_to_email_in_personal_missed_stream_messages(self, send_as_user: bool) -> None:
+    def _reply_to_email_in_missed_personal_messages(self, send_as_user: bool) -> None:
         msg_id = self.send_personal_message(
             self.example_user("othello"),
             self.example_user("hamlet"),
@@ -685,7 +685,7 @@ def _reply_to_email_in_personal_missed_stream_messages(self, send_as_user: bool)
         email_subject = "PMs with Othello, the Moor of Venice"
         self._test_cases(msg_id, verify_body_include, email_subject, send_as_user)
 
-    def _reply_warning_in_personal_missed_stream_messages(self, send_as_user: bool) -> None:
+    def _reply_warning_in_missed_personal_messages(self, send_as_user: bool) -> None:
         msg_id = self.send_personal_message(
             self.example_user("othello"),
             self.example_user("hamlet"),
@@ -695,7 +695,7 @@ def _reply_warning_in_personal_missed_stream_messages(self, send_as_user: bool)
         email_subject = "PMs with Othello, the Moor of Venice"
         self._test_cases(msg_id, verify_body_include, email_subject, send_as_user)
 
-    def _extra_context_in_huddle_missed_stream_messages_two_others(
+    def _extra_context_in_missed_huddle_messages_two_others(
         self, send_as_user: bool, show_message_content: bool = True
     ) -> None:
         msg_id = self.send_huddle_message(
@@ -708,7 +708,9 @@ def _extra_context_in_huddle_missed_stream_messages_two_others(
         )
 
         if show_message_content:
-            verify_body_include = ["Othello, the Moor of Venice: Group personal message! -- Reply"]
+            verify_body_include = [
+                "Othello, the Moor of Venice: > Group personal message! -- Reply"
+            ]
             email_subject = "Group PMs with Iago and Othello, the Moor of Venice"
             verify_body_does_not_include: List[str] = []
         else:
@@ -735,9 +737,7 @@ def _extra_context_in_huddle_missed_stream_messages_two_others(
             verify_body_does_not_include=verify_body_does_not_include,
         )
 
-    def _extra_context_in_huddle_missed_stream_messages_three_others(
-        self, send_as_user: bool
-    ) -> None:
+    def _extra_context_in_missed_huddle_messages_three_others(self, send_as_user: bool) -> None:
         msg_id = self.send_huddle_message(
             self.example_user("othello"),
             [
@@ -748,15 +748,13 @@ def _extra_context_in_huddle_missed_stream_messages_three_others(
             "Group personal message!",
         )
 
-        verify_body_include = ["Othello, the Moor of Venice: Group personal message! -- Reply"]
+        verify_body_include = ["Othello, the Moor of Venice: > Group personal message! -- Reply"]
         email_subject = (
             "Group PMs with Cordelia, Lear's daughter, Iago, and Othello, the Moor of Venice"
         )
         self._test_cases(msg_id, verify_body_include, email_subject, send_as_user)
 
-    def _extra_context_in_huddle_missed_stream_messages_many_others(
-        self, send_as_user: bool
-    ) -> None:
+    def _extra_context_in_missed_huddle_messages_many_others(self, send_as_user: bool) -> None:
         msg_id = self.send_huddle_message(
             self.example_user("othello"),
             [
@@ -768,7 +766,7 @@ def _extra_context_in_huddle_missed_stream_messages_many_others(
             "Group personal message!",
         )
 
-        verify_body_include = ["Othello, the Moor of Venice: Group personal message! -- Reply"]
+        verify_body_include = ["Othello, the Moor of Venice: > Group personal message! -- Reply"]
         email_subject = "Group PMs with Cordelia, Lear's daughter, Iago, and 2 others"
         self._test_cases(msg_id, verify_body_include, email_subject, send_as_user)
 
@@ -784,7 +782,7 @@ def _deleted_message_in_missed_stream_messages(self, send_as_user: bool) -> None
         handle_missedmessage_emails(hamlet.id, [{"message_id": msg_id}])
         self.assert_length(mail.outbox, 0)
 
-    def _deleted_message_in_personal_missed_stream_messages(self, send_as_user: bool) -> None:
+    def _deleted_message_in_missed_personal_messages(self, send_as_user: bool) -> None:
         msg_id = self.send_personal_message(
             self.example_user("othello"),
             self.example_user("hamlet"),
@@ -798,7 +796,7 @@ def _deleted_message_in_personal_missed_stream_messages(self, send_as_user: bool
         handle_missedmessage_emails(hamlet.id, [{"message_id": msg_id}])
         self.assert_length(mail.outbox, 0)
 
-    def _deleted_message_in_huddle_missed_stream_messages(self, send_as_user: bool) -> None:
+    def _deleted_message_in_missed_huddle_messages(self, send_as_user: bool) -> None:
         msg_id = self.send_huddle_message(
             self.example_user("othello"),
             [
@@ -850,7 +848,7 @@ def test_smaller_user_group_mention_priority(self) -> None:
         )
 
         expected_email_include = [
-            "Othello, the Moor of Venice: @*hamlet_only* @*hamlet_and_cordelia* -- ",
+            "Othello, the Moor of Venice: > @*hamlet_only* > @*hamlet_and_cordelia* -- ",
             "You are receiving this because @hamlet_only was mentioned in Zulip Dev.",
         ]
 
@@ -890,7 +888,7 @@ def test_personal_over_user_group_mention_priority(self) -> None:
         )
 
         expected_email_include = [
-            "Othello, the Moor of Venice: @*hamlet_and_cordelia* @**King Hamlet** -- ",
+            "Othello, the Moor of Venice: > @*hamlet_and_cordelia* > @**King Hamlet** -- ",
             "You are receiving this because you were mentioned in Zulip Dev.",
         ]
 
@@ -924,13 +922,11 @@ def test_message_content_disabled_in_missed_message_notifications(self) -> None:
             False, show_message_content=False
         )
         mail.outbox = []
-        self._extra_context_in_personal_missed_stream_messages(
+        self._extra_context_in_missed_personal_messages(
             False, show_message_content=False, message_content_disabled_by_user=True
         )
         mail.outbox = []
-        self._extra_context_in_huddle_missed_stream_messages_two_others(
-            False, show_message_content=False
-        )
+        self._extra_context_in_missed_huddle_messages_two_others(False, show_message_content=False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
     def test_extra_context_in_missed_stream_messages_as_user(self) -> None:
@@ -953,8 +949,8 @@ def test_extra_context_in_missed_stream_messages_as_user_two_senders(self) -> No
     def test_extra_context_in_missed_stream_messages_two_senders(self) -> None:
         self._extra_context_in_missed_stream_messages_mention_two_senders(False)
 
-    def test_reply_to_email_in_personal_missed_stream_messages(self) -> None:
-        self._reply_to_email_in_personal_missed_stream_messages(False)
+    def test_reply_to_email_in_missed_personal_messages(self) -> None:
+        self._reply_to_email_in_missed_personal_messages(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
     def test_extra_context_in_missed_stream_messages_email_notify_as_user(self) -> None:
@@ -964,36 +960,36 @@ def test_extra_context_in_missed_stream_messages_email_notify(self) -> None:
         self._extra_context_in_missed_stream_messages_email_notify(False)
 
     @override_settings(EMAIL_GATEWAY_PATTERN="")
-    def test_reply_warning_in_personal_missed_stream_messages(self) -> None:
-        self._reply_warning_in_personal_missed_stream_messages(False)
+    def test_reply_warning_in_missed_personal_messages(self) -> None:
+        self._reply_warning_in_missed_personal_messages(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_extra_context_in_personal_missed_stream_messages_as_user(self) -> None:
-        self._extra_context_in_personal_missed_stream_messages(True)
+    def test_extra_context_in_missed_personal_messages_as_user(self) -> None:
+        self._extra_context_in_missed_personal_messages(True)
 
-    def test_extra_context_in_personal_missed_stream_messages(self) -> None:
-        self._extra_context_in_personal_missed_stream_messages(False)
+    def test_extra_context_in_missed_personal_messages(self) -> None:
+        self._extra_context_in_missed_personal_messages(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_extra_context_in_huddle_missed_stream_messages_two_others_as_user(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_two_others(True)
+    def test_extra_context_in_missed_huddle_messages_two_others_as_user(self) -> None:
+        self._extra_context_in_missed_huddle_messages_two_others(True)
 
-    def test_extra_context_in_huddle_missed_stream_messages_two_others(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_two_others(False)
+    def test_extra_context_in_missed_huddle_messages_two_others(self) -> None:
+        self._extra_context_in_missed_huddle_messages_two_others(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_extra_context_in_huddle_missed_stream_messages_three_others_as_user(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_three_others(True)
+    def test_extra_context_in_missed_huddle_messages_three_others_as_user(self) -> None:
+        self._extra_context_in_missed_huddle_messages_three_others(True)
 
-    def test_extra_context_in_huddle_missed_stream_messages_three_others(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_three_others(False)
+    def test_extra_context_in_missed_huddle_messages_three_others(self) -> None:
+        self._extra_context_in_missed_huddle_messages_three_others(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_extra_context_in_huddle_missed_stream_messages_many_others_as_user(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_many_others(True)
+    def test_extra_context_in_missed_huddle_messages_many_others_as_user(self) -> None:
+        self._extra_context_in_missed_huddle_messages_many_others(True)
 
-    def test_extra_context_in_huddle_missed_stream_messages_many_others(self) -> None:
-        self._extra_context_in_huddle_missed_stream_messages_many_others(False)
+    def test_extra_context_in_missed_huddle_messages_many_others(self) -> None:
+        self._extra_context_in_missed_huddle_messages_many_others(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
     def test_deleted_message_in_missed_stream_messages_as_user(self) -> None:
@@ -1003,18 +999,18 @@ def test_deleted_message_in_missed_stream_messages(self) -> None:
         self._deleted_message_in_missed_stream_messages(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_deleted_message_in_personal_missed_stream_messages_as_user(self) -> None:
-        self._deleted_message_in_personal_missed_stream_messages(True)
+    def test_deleted_message_in_missed_personal_messages_as_user(self) -> None:
+        self._deleted_message_in_missed_personal_messages(True)
 
-    def test_deleted_message_in_personal_missed_stream_messages(self) -> None:
-        self._deleted_message_in_personal_missed_stream_messages(False)
+    def test_deleted_message_in_missed_personal_messages(self) -> None:
+        self._deleted_message_in_missed_personal_messages(False)
 
     @override_settings(SEND_MISSED_MESSAGE_EMAILS_AS_USER=True)
-    def test_deleted_message_in_huddle_missed_stream_messages_as_user(self) -> None:
-        self._deleted_message_in_huddle_missed_stream_messages(True)
+    def test_deleted_message_in_missed_huddle_messages_as_user(self) -> None:
+        self._deleted_message_in_missed_huddle_messages(True)
 
-    def test_deleted_message_in_huddle_missed_stream_messages(self) -> None:
-        self._deleted_message_in_huddle_missed_stream_messages(False)
+    def test_deleted_message_in_missed_huddle_messages(self) -> None:
+        self._deleted_message_in_missed_huddle_messages(False)
 
     def test_realm_message_content_allowed_in_email_notifications(self) -> None:
         user = self.example_user("hamlet")
@@ -1029,14 +1025,14 @@ def test_realm_message_content_allowed_in_email_notifications(self) -> None:
             user, "message_content_in_email_notifications", True, acting_user=None
         )
         mail.outbox = []
-        self._extra_context_in_personal_missed_stream_messages(False, show_message_content=True)
+        self._extra_context_in_missed_personal_messages(False, show_message_content=True)
 
         # Emails don't have missed message content when message content is disabled by the user
         do_change_user_setting(
             user, "message_content_in_email_notifications", False, acting_user=None
         )
         mail.outbox = []
-        self._extra_context_in_personal_missed_stream_messages(
+        self._extra_context_in_missed_personal_messages(
             False, show_message_content=False, message_content_disabled_by_user=True
         )
 
@@ -1050,7 +1046,7 @@ def test_realm_message_content_allowed_in_email_notifications(self) -> None:
             user, "message_content_in_email_notifications", True, acting_user=None
         )
         mail.outbox = []
-        self._extra_context_in_personal_missed_stream_messages(
+        self._extra_context_in_missed_personal_messages(
             False, show_message_content=False, message_content_disabled_by_realm=True
         )
 
@@ -1058,7 +1054,7 @@ def test_realm_message_content_allowed_in_email_notifications(self) -> None:
             user, "message_content_in_email_notifications", False, acting_user=None
         )
         mail.outbox = []
-        self._extra_context_in_personal_missed_stream_messages(
+        self._extra_context_in_missed_personal_messages(
             False,
             show_message_content=False,
             message_content_disabled_by_user=True,
@@ -1137,7 +1133,7 @@ def test_sender_name_in_missed_message(self) -> None:
 
         assert isinstance(mail.outbox[0], EmailMultiAlternatives)
         assert isinstance(mail.outbox[0].alternatives[0][0], str)
-        self.assertIn("Iago: @**King Hamlet**\n\n--\nYou are", mail.outbox[0].body)
+        self.assertIn("Iago:\n> @**King Hamlet**\n\n--\nYou are", mail.outbox[0].body)
         # If message content starts with <p> tag the sender name is appended inside the <p> tag.
         self.assertIn(
             '<p><b>Iago</b>: <span class="user-mention"', mail.outbox[0].alternatives[0][0]
@@ -1145,7 +1141,7 @@ def test_sender_name_in_missed_message(self) -> None:
 
         assert isinstance(mail.outbox[1], EmailMultiAlternatives)
         assert isinstance(mail.outbox[1].alternatives[0][0], str)
-        self.assertIn("Iago: * 1\n *2\n\n--\nYou are receiving", mail.outbox[1].body)
+        self.assertIn("Iago:\n> * 1\n>  *2\n\n--\nYou are receiving", mail.outbox[1].body)
         # If message content does not starts with <p> tag sender name is appended before the <p> tag
         self.assertIn(
             "       <b>Iago</b>: <div><ul>\n<li>1<br/>\n *2</li>\n</ul></div>\n",
@@ -1154,7 +1150,7 @@ def test_sender_name_in_missed_message(self) -> None:
 
         assert isinstance(mail.outbox[2], EmailMultiAlternatives)
         assert isinstance(mail.outbox[2].alternatives[0][0], str)
-        self.assertEqual("Hello\n\n--\n\nReply", mail.outbox[2].body[:16])
+        self.assertEqual("> Hello\n\n--\n\nReply", mail.outbox[2].body[:18])
         # Sender name is not appended to message for PM missed messages
         self.assertIn(
             ">\n                    \n                        <div><p>Hello</p></div>\n",