Repository: zulip/zulip

Issue:
Relax restrictions on moving messages to other streams
Following #6427 and #14498, we should be able to make the feature to move messages to new streams more accessible.  There's several directions I'd except this to take:

* [x] Allowing administrators (eventually perhaps stream owners) to move messages to-and-from private streams.  We'll need to take some care about how to do the announcement messages about these moves (From a security perspective, it perhaps should say "To a private stream" or "From a private stream" in automated the tombstone messages, rather than naming which stream it went to?  Not sure.)
* [ ] Allowing non-administrators to use this feature.  I'm guessing we'd extend the options for "who can edit message topics" to include values for who can move messages between streams.
* [x] Offering the option to move a topic to another stream in the main message-edit UI on a message (#13912 primarily implements an out-of-band version right now).

At the moment, I'd like to use this issue thread for feedback on what the semantics/options for this feature should be.


Discussion:
Hello @zulip/server-message-view members, this issue was labeled with the "area: message-editing" label, so you may want to check it out!

<!-- areaLabelAddition -->
Comment on 1)

I don't see a need for special care here:
If a topic is moved from a private stream to a public one its content is anyway becoming public so why hide where this content came from? Same the other way around: If a topic was public before, i.e. everybody has seen it anyway, what's the point in hiding this fact once it's available only to a subset of those same people?
Comment on 2)

It would indeed be nice to allow administrators to decide whether normal users can use this feature or not. I could imagine some users are less careless/knowledgeable and might move topics by accident, not knowing the implications. Some administrators may decide to educate their users but others may want to simply disable the functionality altogether/keep it in their hands.
> If a topic is moved from a private stream to a public one its content is anyway becoming public so why hide where this content came from? Same the other way around: If a topic was public before, i.e. everybody has seen it anyway, what's the point in hiding this fact once it's available only to a subset of those same people?

@andreas-bulling - just because the content of a message is public doesn't mean that you'd want to make the name of a private stream public. I think the "to/from a private stream" approach in the tombstones makes sense- it minimizes information leakage.
I agree that the scenario "name of a topic is more private than its content" is possible in theory - but highly unlikely in practice I think (we are talking about moving a private topic to a public one, just to be on the same page. Moving from public to private is not critical I believe we can easily agree). 

The question is whether this arguably small likelihood of practical relevance warrants making the UI more complex but this is not up to me to decide. I simply think few people will ever use this feature so as a user I'd prefer a leaner UI.
Another idea would be to make the tombstones optional (with a checkbox for whether to create tombstone messages or not).  I think there are cases where the topic only has one message that's a quick question and the tombstones might feel like excessive overhead (Or where someone is doing an administrative merge of two streams).  

I think making the tombstone messages say "from a private stream" would be reasonable.
I've been looking through the pull requests a bit.
How far is the implementation of this feature? Especially 1) would be very helpful for a workflow we're trying to build up right now.

To give a bit of feedback on the ideas: I think the newest idea (by tim) summarizes this the best: Tombstones should be optional and having them say "from/to a private stream" seems like a good idea to prevent the potential leaking of confidential information
This issue is a bit out of date; we've already implemented making the tombstones optional in the 3.0 release.  Though we haven't added the "to a private stream" text change.

The much more complicated issue involves UserMessage objects (which track which users actually received the message, and is used in private streams without shared history to determine which users should have access to which messages).  A wrong implementation would be the naive one that preserves those objects; the end result would be that users who were on the public stream when messages were sent could still access them via the API even though they were moved to a private stream (because they'd still be part of that user's private history).  We resolved the analog of that issue for guest users in 3.0, but need to resolve the similar issue for private streams.  

I think given the work we've already done, doing the groundwork to allow moving messages TO private streams won't actually be very hard; @amanagr are you up for making this happen?

(And then we can think about the FROM side as well -- that is simpler in many ways assuming we're OK with the security implications)



Sure! I will try to pick this up soon.
Any new developments on this?
@Lithimlin thanks for pinging on this issue, I can try some positive work done on it this week, seems simple enough.
This private stream feature was implemented via https://github.com/zulip/zulip/issues/16284.  I think the last bit of this will likely be that we'll include stream administrators in who can move messages between streams.  
Can this be used to rename a topic as well?

Resolving patch:
diff --git a/zerver/lib/actions.py b/zerver/lib/actions.py
--- a/zerver/lib/actions.py
+++ b/zerver/lib/actions.py
@@ -143,6 +143,7 @@
     access_stream_by_id,
     access_stream_for_send_message,
     can_access_stream_user_ids,
+    check_stream_access_based_on_stream_post_policy,
     check_stream_name,
     create_stream_if_needed,
     get_default_value_for_history_public_to_subscribers,
@@ -2802,7 +2803,7 @@ def check_update_message(
     if stream_id is not None:
         if not message.is_stream_message():
             raise JsonableError(_("Message must be a stream message"))
-        if not user_profile.is_realm_admin:
+        if not user_profile.can_move_messages_between_streams():
             raise JsonableError(_("You don't have permission to move this message"))
         try:
             access_stream_by_id(user_profile, message.recipient.type_id)
@@ -2816,6 +2817,7 @@ def check_update_message(
             raise JsonableError(_("Cannot change message content while changing stream"))
 
         new_stream = access_stream_by_id(user_profile, stream_id, require_active=True)[0]
+        check_stream_access_based_on_stream_post_policy(user_profile, new_stream)
 
     number_changed = do_update_message(
         user_profile,


Test patch:
diff --git a/frontend_tests/node_tests/settings_data.js b/frontend_tests/node_tests/settings_data.js
--- a/frontend_tests/node_tests/settings_data.js
+++ b/frontend_tests/node_tests/settings_data.js
@@ -107,117 +107,58 @@ run_test("user_can_change_logo", () => {
     assert.equal(can_change_logo(), false);
 });
 
-run_test("user_can_invite_others_to_realm", () => {
-    const can_invite_others_to_realm = settings_data.user_can_invite_others_to_realm;
-
-    page_params.is_admin = true;
-    page_params.realm_invite_to_realm_policy =
-        settings_config.common_policy_values.by_admins_only.code;
-    assert.equal(can_invite_others_to_realm(), true);
-
-    page_params.is_admin = false;
-    assert.equal(can_invite_others_to_realm(), false);
-
-    page_params.is_moderator = true;
-    page_params.realm_invite_to_realm_policy =
-        settings_config.common_policy_values.by_moderators_only.code;
-    assert.equal(can_invite_others_to_realm(), true);
-
-    page_params.is_moderator = false;
-    assert.equal(can_invite_others_to_realm(), false);
-
-    page_params.is_guest = true;
-    page_params.realm_invite_to_realm_policy = settings_config.common_policy_values.by_members.code;
-    assert.equal(can_invite_others_to_realm(), false);
-
-    page_params.is_guest = false;
-    assert.equal(can_invite_others_to_realm(), true);
-
-    page_params.realm_invite_to_realm_policy =
-        settings_config.common_policy_values.by_full_members.code;
-    page_params.user_id = 30;
-    people.add_active_user(isaac);
-    isaac.date_joined = new Date(Date.now());
-    page_params.realm_waiting_period_threshold = 10;
-    assert.equal(can_invite_others_to_realm(), false);
-
-    isaac.date_joined = new Date(Date.now() - 20 * 86400000);
-    assert.equal(can_invite_others_to_realm(), true);
-});
-
-run_test("user_can_subscribe_other_users", () => {
-    const can_subscribe_other_users = settings_data.user_can_subscribe_other_users;
-
-    page_params.is_admin = true;
-    page_params.realm_invite_to_stream_policy =
-        settings_config.common_policy_values.by_admins_only.code;
-    assert.equal(can_subscribe_other_users(), true);
-
-    page_params.is_admin = false;
-    assert.equal(can_subscribe_other_users(), false);
-
-    page_params.is_moderator = true;
-    page_params.realm_invite_to_stream_policy =
-        settings_config.common_policy_values.by_moderators_only.code;
-    assert.equal(can_subscribe_other_users(), true);
-
-    page_params.is_moderator = false;
-    assert.equal(can_subscribe_other_users(), false);
-
-    page_params.is_guest = true;
-    page_params.realm_invite_to_stream_policy =
-        settings_config.common_policy_values.by_members.code;
-    assert.equal(can_subscribe_other_users(), false);
-
-    page_params.is_guest = false;
-    assert.equal(can_subscribe_other_users(), true);
-
-    page_params.realm_invite_to_stream_policy =
-        settings_config.common_policy_values.by_full_members.code;
-    page_params.user_id = 30;
-    people.add_active_user(isaac);
-    isaac.date_joined = new Date(Date.now());
-    page_params.realm_waiting_period_threshold = 10;
-    assert.equal(can_subscribe_other_users(), false);
-
-    isaac.date_joined = new Date(Date.now() - 20 * 86400000);
-    assert.equal(can_subscribe_other_users(), true);
-});
-
-run_test("user_can_create_streams", () => {
-    const can_create_streams = settings_data.user_can_create_streams;
-
-    page_params.is_admin = true;
-    page_params.realm_create_stream_policy =
-        settings_config.common_policy_values.by_admins_only.code;
-    assert.equal(can_create_streams(), true);
-
-    page_params.is_admin = false;
-    assert.equal(can_create_streams(), false);
-
-    page_params.is_moderator = true;
-    page_params.realm_create_stream_policy =
-        settings_config.common_policy_values.by_moderators_only.code;
-    assert.equal(can_create_streams(), true);
-
-    page_params.is_moderator = false;
-    assert.equal(can_create_streams(), false);
-
-    page_params.is_guest = true;
-    page_params.realm_create_stream_policy = settings_config.common_policy_values.by_members.code;
-    assert.equal(can_create_streams(), false);
-
-    page_params.is_guest = false;
-    assert.equal(can_create_streams(), true);
-
-    page_params.realm_create_stream_policy =
-        settings_config.common_policy_values.by_full_members.code;
-    page_params.user_id = 30;
-    people.add_active_user(isaac);
-    isaac.date_joined = new Date(Date.now());
-    page_params.realm_waiting_period_threshold = 10;
-    assert.equal(can_create_streams(), false);
-
-    isaac.date_joined = new Date(Date.now() - 20 * 86400000);
-    assert.equal(can_create_streams(), true);
-});
+function test_policy(label, policy, validation_func) {
+    run_test(label, () => {
+        page_params.is_admin = true;
+        page_params[policy] = settings_config.common_policy_values.by_admins_only.code;
+        assert.equal(validation_func(), true);
+
+        page_params.is_admin = false;
+        assert.equal(validation_func(), false);
+
+        page_params.is_moderator = true;
+        page_params[policy] = settings_config.common_policy_values.by_moderators_only.code;
+        assert.equal(validation_func(), true);
+
+        page_params.is_moderator = false;
+        assert.equal(validation_func(), false);
+
+        page_params.is_guest = true;
+        page_params[policy] = settings_config.common_policy_values.by_members.code;
+        assert.equal(validation_func(), false);
+
+        page_params.is_guest = false;
+        assert.equal(validation_func(), true);
+
+        page_params[policy] = settings_config.common_policy_values.by_full_members.code;
+        page_params.user_id = 30;
+        people.add_active_user(isaac);
+        isaac.date_joined = new Date(Date.now());
+        page_params.realm_waiting_period_threshold = 10;
+        assert.equal(validation_func(), false);
+
+        isaac.date_joined = new Date(Date.now() - 20 * 86400000);
+        assert.equal(validation_func(), true);
+    });
+}
+
+test_policy(
+    "user_can_create_streams",
+    "realm_create_stream_policy",
+    settings_data.user_can_create_streams,
+);
+test_policy(
+    "user_can_subscribe_other_users",
+    "realm_invite_to_stream_policy",
+    settings_data.user_can_subscribe_other_users,
+);
+test_policy(
+    "user_can_invite_others_to_realm",
+    "realm_invite_to_realm_policy",
+    settings_data.user_can_invite_others_to_realm,
+);
+test_policy(
+    "user_can_move_messages_between_streams",
+    "realm_move_messages_between_streams_policy",
+    settings_data.user_can_move_messages_between_streams,
+);
diff --git a/zerver/tests/test_message_edit.py b/zerver/tests/test_message_edit.py
--- a/zerver/tests/test_message_edit.py
+++ b/zerver/tests/test_message_edit.py
@@ -1,6 +1,6 @@
 import datetime
 from operator import itemgetter
-from typing import Any, Dict, List, Tuple
+from typing import Any, Dict, List, Optional, Tuple
 from unittest import mock
 
 import orjson
@@ -8,6 +8,8 @@
 from django.http import HttpResponse
 
 from zerver.lib.actions import (
+    do_change_stream_post_policy,
+    do_change_user_role,
     do_set_realm_property,
     do_update_message,
     get_topic_messages,
@@ -17,7 +19,7 @@
 from zerver.lib.test_classes import ZulipTestCase
 from zerver.lib.test_helpers import cache_tries_captured, queries_captured
 from zerver.lib.topic import LEGACY_PREV_TOPIC, TOPIC_NAME
-from zerver.models import Message, Stream, UserMessage, UserProfile, get_realm, get_stream
+from zerver.models import Message, Realm, Stream, UserMessage, UserProfile, get_realm, get_stream
 
 
 class EditMessageTest(ZulipTestCase):
@@ -1214,6 +1216,58 @@ def test_move_message_realm_admin_cant_move_from_private_stream_without_subscrip
             "You don't have permission to move this message due to missing access to its stream",
         )
 
+    def test_move_message_from_private_stream_message_access_checks(
+        self,
+    ) -> None:
+        hamlet = self.example_user("hamlet")
+        user_profile = self.example_user("iago")
+        self.assertEqual(user_profile.role, UserProfile.ROLE_REALM_ADMINISTRATOR)
+        self.login("iago")
+
+        private_stream = self.make_stream(
+            "privatestream", invite_only=True, history_public_to_subscribers=False
+        )
+        self.subscribe(hamlet, "privatestream")
+        original_msg_id = self.send_stream_message(hamlet, "privatestream", topic_name="test123")
+        self.subscribe(user_profile, "privatestream")
+        new_msg_id = self.send_stream_message(user_profile, "privatestream", topic_name="test123")
+
+        # Now we unsub and hamlet sends a new message (we won't have access to it even after re-subbing!)
+        self.unsubscribe(user_profile, "privatestream")
+        new_inaccessible_msg_id = self.send_stream_message(
+            hamlet, "privatestream", topic_name="test123"
+        )
+
+        # Re-subscribe and send another message:
+        self.subscribe(user_profile, "privatestream")
+        newest_msg_id = self.send_stream_message(
+            user_profile, "privatestream", topic_name="test123"
+        )
+
+        verona = get_stream("Verona", user_profile.realm)
+
+        result = self.client_patch(
+            "/json/messages/" + str(new_msg_id),
+            {
+                "message_id": new_msg_id,
+                "stream_id": verona.id,
+                "propagate_mode": "change_all",
+            },
+        )
+
+        self.assert_json_success(result)
+        self.assertEqual(Message.objects.get(id=new_msg_id).recipient_id, verona.recipient_id)
+        self.assertEqual(Message.objects.get(id=newest_msg_id).recipient_id, verona.recipient_id)
+        # The original message and the new, inaccessible message weren't moved,
+        # because user_profile doesn't have access to them.
+        self.assertEqual(
+            Message.objects.get(id=original_msg_id).recipient_id, private_stream.recipient_id
+        )
+        self.assertEqual(
+            Message.objects.get(id=new_inaccessible_msg_id).recipient_id,
+            private_stream.recipient_id,
+        )
+
     def test_move_message_cant_move_private_message(
         self,
     ) -> None:
@@ -1268,26 +1322,174 @@ def test_move_message_to_stream_change_later(self) -> None:
             f"This topic was moved here from #**test move stream>test** by @_**Iago|{user_profile.id}**",
         )
 
-    def test_move_message_to_stream_no_allowed(self) -> None:
+    def test_move_message_between_streams_policy_setting(self) -> None:
         (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
-            "aaron", "test move stream", "new stream", "test"
+            "othello", "old_stream_1", "new_stream_1", "test"
         )
 
-        result = self.client_patch(
-            "/json/messages/" + str(msg_id),
-            {
-                "message_id": msg_id,
-                "stream_id": new_stream.id,
-                "propagate_mode": "change_all",
-            },
+        def check_move_message_according_to_policy(role: int, expect_fail: bool = False) -> None:
+            do_change_user_role(user_profile, role, acting_user=None)
+
+            result = self.client_patch(
+                "/json/messages/" + str(msg_id),
+                {
+                    "message_id": msg_id,
+                    "stream_id": new_stream.id,
+                    "propagate_mode": "change_all",
+                },
+            )
+
+            if expect_fail:
+                self.assert_json_error(result, "You don't have permission to move this message")
+                messages = get_topic_messages(user_profile, old_stream, "test")
+                self.assertEqual(len(messages), 3)
+                messages = get_topic_messages(user_profile, new_stream, "test")
+                self.assertEqual(len(messages), 0)
+            else:
+                self.assert_json_success(result)
+                messages = get_topic_messages(user_profile, old_stream, "test")
+                self.assertEqual(len(messages), 1)
+                messages = get_topic_messages(user_profile, new_stream, "test")
+                self.assertEqual(len(messages), 4)
+
+        # Check sending messages when policy is Realm.POLICY_ADMINS_ONLY.
+        do_set_realm_property(
+            user_profile.realm,
+            "move_messages_between_streams_policy",
+            Realm.POLICY_ADMINS_ONLY,
+            acting_user=None,
+        )
+        check_move_message_according_to_policy(UserProfile.ROLE_MODERATOR, expect_fail=True)
+        check_move_message_according_to_policy(UserProfile.ROLE_REALM_ADMINISTRATOR)
+
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_2", "new_stream_2", "test"
+        )
+        # Check sending messages when policy is Realm.POLICY_MODERATORS_ONLY.
+        do_set_realm_property(
+            user_profile.realm,
+            "move_messages_between_streams_policy",
+            Realm.POLICY_MODERATORS_ONLY,
+            acting_user=None,
         )
-        self.assert_json_error(result, "You don't have permission to move this message")
+        check_move_message_according_to_policy(UserProfile.ROLE_MEMBER, expect_fail=True)
+        check_move_message_according_to_policy(UserProfile.ROLE_MODERATOR)
 
-        messages = get_topic_messages(user_profile, old_stream, "test")
-        self.assertEqual(len(messages), 3)
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_3", "new_stream_3", "test"
+        )
+        # Check sending messages when policy is Realm.POLICY_FULL_MEMBERS_ONLY.
+        do_set_realm_property(
+            user_profile.realm,
+            "move_messages_between_streams_policy",
+            Realm.POLICY_FULL_MEMBERS_ONLY,
+            acting_user=None,
+        )
+        do_set_realm_property(
+            user_profile.realm, "waiting_period_threshold", 100000, acting_user=None
+        )
+        check_move_message_according_to_policy(UserProfile.ROLE_MEMBER, expect_fail=True)
 
-        messages = get_topic_messages(user_profile, new_stream, "test")
-        self.assertEqual(len(messages), 0)
+        do_set_realm_property(user_profile.realm, "waiting_period_threshold", 0, acting_user=None)
+        check_move_message_according_to_policy(UserProfile.ROLE_MEMBER)
+
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_4", "new_stream_4", "test"
+        )
+        # Check sending messages when policy is Realm.POLICY_MEMBERS_ONLY.
+        do_set_realm_property(
+            user_profile.realm,
+            "move_messages_between_streams_policy",
+            Realm.POLICY_MEMBERS_ONLY,
+            acting_user=None,
+        )
+        check_move_message_according_to_policy(UserProfile.ROLE_GUEST, expect_fail=True)
+        check_move_message_according_to_policy(UserProfile.ROLE_MEMBER)
+
+    def test_move_message_to_stream_based_on_stream_post_policy(self) -> None:
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_1", "new_stream_1", "test"
+        )
+        do_set_realm_property(
+            user_profile.realm,
+            "move_messages_between_streams_policy",
+            Realm.POLICY_MEMBERS_ONLY,
+            acting_user=None,
+        )
+
+        def check_move_message_to_stream(role: int, error_msg: Optional[str] = None) -> None:
+            do_change_user_role(user_profile, role, acting_user=None)
+
+            result = self.client_patch(
+                "/json/messages/" + str(msg_id),
+                {
+                    "message_id": msg_id,
+                    "stream_id": new_stream.id,
+                    "propagate_mode": "change_all",
+                },
+            )
+
+            if error_msg is not None:
+                self.assert_json_error(result, error_msg)
+                messages = get_topic_messages(user_profile, old_stream, "test")
+                self.assertEqual(len(messages), 3)
+                messages = get_topic_messages(user_profile, new_stream, "test")
+                self.assertEqual(len(messages), 0)
+            else:
+                self.assert_json_success(result)
+                messages = get_topic_messages(user_profile, old_stream, "test")
+                self.assertEqual(len(messages), 1)
+                messages = get_topic_messages(user_profile, new_stream, "test")
+                self.assertEqual(len(messages), 4)
+
+        # Check when stream_post_policy is STREAM_POST_POLICY_ADMINS.
+        do_change_stream_post_policy(new_stream, Stream.STREAM_POST_POLICY_ADMINS)
+        error_msg = "Only organization administrators can send to this stream."
+        check_move_message_to_stream(UserProfile.ROLE_MODERATOR, error_msg)
+        check_move_message_to_stream(UserProfile.ROLE_REALM_ADMINISTRATOR)
+
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_2", "new_stream_2", "test"
+        )
+
+        # Check when stream_post_policy is STREAM_POST_POLICY_MODERATORS.
+        do_change_stream_post_policy(new_stream, Stream.STREAM_POST_POLICY_MODERATORS)
+        error_msg = "Only organization administrators and moderators can send to this stream."
+        check_move_message_to_stream(UserProfile.ROLE_MEMBER, error_msg)
+        check_move_message_to_stream(UserProfile.ROLE_MODERATOR)
+
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_3", "new_stream_3", "test"
+        )
+
+        # Check when stream_post_policy is STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS.
+        do_change_stream_post_policy(new_stream, Stream.STREAM_POST_POLICY_RESTRICT_NEW_MEMBERS)
+        error_msg = "New members cannot send to this stream."
+
+        do_set_realm_property(
+            user_profile.realm, "waiting_period_threshold", 100000, acting_user=None
+        )
+        check_move_message_to_stream(UserProfile.ROLE_MEMBER, error_msg)
+
+        do_set_realm_property(user_profile.realm, "waiting_period_threshold", 0, acting_user=None)
+        check_move_message_to_stream(UserProfile.ROLE_MEMBER)
+
+        (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(
+            "othello", "old_stream_4", "new_stream_4", "test"
+        )
+
+        # Check when stream_post_policy is STREAM_POST_POLICY_EVERYONE.
+        # In this case also, guest is not allowed as we do not allow guest to move
+        # messages between streams in any case, so stream_post_policy of new stream does
+        # not matter.
+        do_change_stream_post_policy(new_stream, Stream.STREAM_POST_POLICY_EVERYONE)
+        do_set_realm_property(
+            user_profile.realm, "waiting_period_threshold", 100000, acting_user=None
+        )
+        check_move_message_to_stream(
+            UserProfile.ROLE_GUEST, "You don't have permission to move this message"
+        )
+        check_move_message_to_stream(UserProfile.ROLE_MEMBER)
 
     def test_move_message_to_stream_with_content(self) -> None:
         (user_profile, old_stream, new_stream, msg_id, msg_id_later) = self.prepare_move_topics(