Repository: zulip/zulip

Issue:
Server sending mark-unread for already-unread messages
To reproduce:
* Open up the web app in a dev server running https://github.com/zulip/zulip/pull/21477 .
  * (That provides handy UI for exercising the issue; but the behavior is something the server itself shouldn't allow, and that PR doesn't touch the server, so it's already a bug in main.)
* Watch the events the server emits.
  * One way to do this is in the "Network" tab of Chrome DevTools, while viewing the web app; look at "Preview" for each "events" request.
  * Another is to run a dev build of the mobile app, and use the debug console there.
* Go to some conversation with several messages.
* Mark the first message as unread, with `m` or the three-dots menu.
* Mark that message as unread again.

Expected:
* Each of the `update_message_flags`/`remove`/`read` events should list just the messages for which the `read` flag was removed. As [the API docs](https://zulip.com/api/get-events#update_message_flags-remove) say:
  > `messages`: (integer)[]
  > Array containing the IDs of the messages from which the flag was removed.

Actual:
* Each of those messages lists all of the messages in the conversation, even though most of them are already unread and therefore cannot have the `read` flag removed.
  * (In my experiments, typically one of the messages will be promptly marked as read after marking the whole thread as unread -- either the one at the top where my cursor is, or the next one just after it. That seems like a bug in #21477; certainly the inconsistency where it's sometimes one and sometimes another seems like a bug. Given that web client behavior, that one message has the `read` flag and so that one message is available to participate in a later mark-as-unread event.

    But regardless of the client's behavior, the server is responsible for its event stream: three messages get marked as unread, then just one gets marked as read, then all three get marked again as unread.)

---

With the current version of https://github.com/zulip/zulip-mobile/pull/4790 -- which I was just about to merge -- this behavior causes the unread counter for the conversation (in the inbox view) to grow larger and larger if you keep doing it, because we accumulate more and more duplicate references to the same message. I might add a workaround in the client before merging that. But we really shouldn't have to work around broken server behavior; it should be fixed in the server.



Discussion:
Can I work on this @gnprice? If yes, please assign this to me.


I think probably this should be claimed by someone more experienced with the codebase; @amanagr can you debug this one?
I think the client will likely want to add a workaround in any case -- since probably an appropriate workaround would also help generate correct behavior in bugs/corner cases involving races, event reordering (say, an unread, read, unread sequence for a given message ID gets somehow reordered to read, unread, unread).
Reasonable. Can we agree that we'll fix this bug before merging any UI to exercise the feature?

The difference is:
* If we fix the bug before there's UI to exercise the feature, then I can go ahead and merge https://github.com/zulip/zulip-mobile/pull/4790 . It's been open a long time, is all ready as far as its code is concerned, and would be great to get in.

  Then adding that workaround can be a polish followup, alongside adding an implementation for group PMs (as https://github.com/zulip/zulip-mobile/pull/4790 covers stream messages and 1:1 PMs.) As I mention at https://github.com/zulip/zulip-mobile/pull/4790#issuecomment-1143009113 , the natural way to fix it should also make the code more efficient, too.

  In principle the client would be susceptible to getting these buggy events from a server that doesn't have the fix -- but with no UI to exercise the feature, it'll be rare to nonexistent.

* If we end up with UI to exercise the feature while it still has this bug, then there will be a contingent of servers that could be sending these buggy events for a long time. The client will have to deal with it as a blocker, and we can't really merge https://github.com/zulip/zulip-mobile/pull/4790 until we first write the workaround.


> event reordering (say, an unread, read, unread sequence for a given message ID gets somehow reordered to read, unread, unread).

This sort of bug could easily cause broken behavior regardless of any workaround, though: just reorder it to unread, unread, read instead. In general it's an imporant invariant of the event system that the events come in order, so that the resulting state they describe is consistent with the eventual state in the database.


Resolving patch:
diff --git a/zerver/actions/message_flags.py b/zerver/actions/message_flags.py
--- a/zerver/actions/message_flags.py
+++ b/zerver/actions/message_flags.py
@@ -1,7 +1,8 @@
 from collections import defaultdict
 from dataclasses import asdict, dataclass, field
-from typing import List, Optional, Set
+from typing import List, Optional, Set, Tuple
 
+from django.db import transaction
 from django.db.models import F
 from django.utils.timezone import now as timezone_now
 from django.utils.translation import gettext as _
@@ -229,7 +230,7 @@ def do_clear_mobile_push_notifications_for_ids(
 
 def do_update_message_flags(
     user_profile: UserProfile, operation: str, flag: str, messages: List[int]
-) -> int:
+) -> Tuple[int, List[int]]:
     valid_flags = [item for item in UserMessage.flags if item not in UserMessage.NON_API_FLAGS]
     if flag not in valid_flags:
         raise JsonableError(_("Invalid flag: '{}'").format(flag))
@@ -250,18 +251,33 @@ def do_update_message_flags(
 
     # And then create historical UserMessage records.  See the called function for more context.
     create_historical_user_messages(user_id=user_profile.id, message_ids=historical_message_ids)
-
-    if operation == "add":
-        count = msgs.update(flags=F("flags").bitor(flagattr))
-    elif operation == "remove":
-        count = msgs.update(flags=F("flags").bitand(~flagattr))
-
+    with transaction.atomic():
+        if operation == "add":
+            msgs = (
+                msgs.select_for_update()
+                .order_by("message_id")
+                .extra(where=[UserMessage.where_flag_is_absent(flagattr)])
+            )
+            updated_message_ids = [um.message_id for um in msgs]
+            msgs.filter(message_id__in=updated_message_ids).update(flags=F("flags").bitor(flagattr))
+        elif operation == "remove":
+            msgs = (
+                msgs.select_for_update()
+                .order_by("message_id")
+                .extra(where=[UserMessage.where_flag_is_present(flagattr)])
+            )
+            updated_message_ids = [um.message_id for um in msgs]
+            msgs.filter(message_id__in=updated_message_ids).update(
+                flags=F("flags").bitand(~flagattr)
+            )
+
+    count = len(updated_message_ids)
     event = {
         "type": "update_message_flags",
         "op": operation,
         "operation": operation,
         "flag": flag,
-        "messages": messages,
+        "messages": updated_message_ids,
         "all": False,
     }
 
@@ -270,14 +286,14 @@ def do_update_message_flags(
         # unread), extend the event with an additional object with
         # details on the messages required to update the client's
         # `unread_msgs` data structure.
-        raw_unread_data = get_raw_unread_data(user_profile, messages)
+        raw_unread_data = get_raw_unread_data(user_profile, updated_message_ids)
         event["message_details"] = format_unread_message_details(user_profile.id, raw_unread_data)
 
     send_event(user_profile.realm, event, [user_profile.id])
 
     if flag == "read" and operation == "add":
         event_time = timezone_now()
-        do_clear_mobile_push_notifications_for_ids([user_profile.id], messages)
+        do_clear_mobile_push_notifications_for_ids([user_profile.id], updated_message_ids)
 
         do_increment_logging_stat(
             user_profile, COUNT_STATS["messages_read::hour"], None, event_time, increment=count
@@ -290,4 +306,4 @@ def do_update_message_flags(
             increment=min(1, count),
         )
 
-    return count
+    return count, updated_message_ids
diff --git a/zerver/models.py b/zerver/models.py
--- a/zerver/models.py
+++ b/zerver/models.py
@@ -25,7 +25,7 @@
 import orjson
 import re2
 from bitfield import BitField
-from bitfield.types import BitHandler
+from bitfield.types import Bit, BitHandler
 from django.conf import settings
 from django.contrib.auth.models import (
     AbstractBaseUser,
@@ -3181,14 +3181,7 @@ class Meta:
         unique_together = ("user_profile", "message")
 
     @staticmethod
-    def where_unread() -> str:
-        # Use this for Django ORM queries to access unread message.
-        # This custom SQL plays nice with our partial indexes.  Grep
-        # the code for example usage.
-        return "flags & 1 = 0"
-
-    @staticmethod
-    def where_starred() -> str:
+    def where_flag_is_present(flagattr: Bit) -> str:
         # Use this for Django ORM queries to access starred messages.
         # This custom SQL plays nice with our partial indexes.  Grep
         # the code for example usage.
@@ -3196,12 +3189,27 @@ def where_starred() -> str:
         # The key detail is that e.g.
         #   UserMessage.objects.filter(user_profile=user_profile, flags=UserMessage.flags.starred)
         # will generate a query involving `flags & 2 = 2`, which doesn't match our index.
-        return "flags & 2 <> 0"
+        return f"flags & {1 << flagattr.number} <> 0"
+
+    @staticmethod
+    def where_flag_is_absent(flagattr: Bit) -> str:
+        return f"flags & {1 << flagattr.number} = 0"
+
+    @staticmethod
+    def where_unread() -> str:
+        return AbstractUserMessage.where_flag_is_absent(getattr(AbstractUserMessage.flags, "read"))
+
+    @staticmethod
+    def where_starred() -> str:
+        return AbstractUserMessage.where_flag_is_present(
+            getattr(AbstractUserMessage.flags, "starred")
+        )
 
     @staticmethod
     def where_active_push_notification() -> str:
-        # See where_starred for documentation.
-        return "flags & 4096 <> 0"
+        return AbstractUserMessage.where_flag_is_present(
+            getattr(AbstractUserMessage.flags, "active_mobile_push_notification")
+        )
 
     def flags_list(self) -> List[str]:
         flags = int(self.flags)
diff --git a/zerver/views/message_flags.py b/zerver/views/message_flags.py
--- a/zerver/views/message_flags.py
+++ b/zerver/views/message_flags.py
@@ -38,13 +38,13 @@ def update_message_flags(
     request_notes = RequestNotes.get_notes(request)
     assert request_notes.log_data is not None
 
-    count = do_update_message_flags(user_profile, operation, flag, messages)
+    count, updated_message_ids = do_update_message_flags(user_profile, operation, flag, messages)
 
     target_count_str = str(len(messages))
     log_data_str = f"[{operation} {flag}/{target_count_str}] actually {count}"
     request_notes.log_data["extra"] = log_data_str
 
-    return json_success(request, data={"messages": messages})
+    return json_success(request, data={"messages": updated_message_ids})
 
 
 @has_request_variables


Test patch:
diff --git a/zerver/tests/test_events.py b/zerver/tests/test_events.py
--- a/zerver/tests/test_events.py
+++ b/zerver/tests/test_events.py
@@ -651,12 +651,27 @@ def test_update_message_flags(self) -> None:
             state_change_expected=True,
         )
         check_update_message_flags_add("events[0]", events[0])
+        self.assert_length(events[0]["messages"], 1)
+        # No message_id is returned from the server if the flag is already preset.
+        events = self.verify_action(
+            lambda: do_update_message_flags(user_profile, "add", "starred", [message]),
+            state_change_expected=False,
+        )
+        self.assert_length(events[0]["messages"], 0)
 
         events = self.verify_action(
             lambda: do_update_message_flags(user_profile, "remove", "starred", [message]),
             state_change_expected=True,
         )
         check_update_message_flags_remove("events[0]", events[0])
+        self.assert_length(events[0]["messages"], 1)
+
+        # No message_id is returned from the server if the flag is already absent.
+        events = self.verify_action(
+            lambda: do_update_message_flags(user_profile, "remove", "starred", [message]),
+            state_change_expected=False,
+        )
+        self.assert_length(events[0]["messages"], 0)
 
     def test_update_read_flag_removes_unread_msg_ids(self) -> None: