Repository: matrix-org/synapse

Issue:
Homeserver configuration options to tweak user directory behaviour
I'd like an option, perhaps in homeserver.yaml, to provide a list of regex patterns for the homeserver to exclude from user directory search results.


Discussion:
my feeling is that if a user's displayname shouldn't be presented in the directory, we probably don't want it in rooms either?
We should consider if a config regiex is the best approach, or if an API removing specific usernames addresses the same problem?
could also shove this through the spam checker module to let arbitrary decisions to be made.
I'm happy so long as it isn't in homeserver.yaml (which would require redeploying from ansible and restarting synapse)
We had a brief discussion about this issue:

> It's to help us with abuse management and essentially we need to be able to block certain user ids from appearing in the directory
> there are a few ideas on how to do this, but we want to avoid any in the homeserver.yaml
> The most promising approach is to extend our current spam checking module - this is part of broader tool called mjolnir. 
> But I think if you can get it to do two things then the issue will be solved:
> * Allow for give regexes to be excluded (I think mjolnir acts on globs)
> * Allow for specific users to be excluded
>
> This will need to apply to the mxid and display name.

I think the first steps here are to understand how spam checking works in Synapse (and Mjolnir) and the connection between those. I'll put some notes in here as I go through it.
From taking a look at [Mjolnir](https://github.com/matrix-org/mjolnir/blob/master/synapse_antispam/) and the [spam check code in Synapse](https://github.com/matrix-org/synapse/blob/f0561fcffd172cb0dfe035dcc78f51bdd451c010/synapse/events/spamcheck.py). I believe the work would entail:

1. Synapse:
    1. Extended the interface of `SpamChecker` to include a new method
        1. (`user_appears_in_directory_results` or something, it would take a user ID and display name as parameters)
        2. To be backwards compatible we could have the `SpamChecker` not require this method on the actual spam checking implementation (this actually might be a nice feature for all of the spam checking methods so you don't end up with pass-through code, e.g. [this in Mjolnir](https://github.com/matrix-org/mjolnir/blob/5d8fea07fc2aa70d2a58f21e0b675bf439d0beb5/synapse_antispam/mjolnir/antispam.py#L109-L116)).
    2. Modify `UserDirectoryHandler` to use the above method to filter results before returning from `search_users` (this might be tricky based on the handling of `limit`, unsure).
2. Mjolnir:
    1. Extend the code that builds the ban list to handle a new type (see [`build`](https://github.com/matrix-org/mjolnir/blob/5d8fea07fc2aa70d2a58f21e0b675bf439d0beb5/synapse_antispam/mjolnir/ban_list.py#L32-L73)).
    2. Add the implementation of the method which Synapse will call.

Unclear bits to me:
* Do the commands to modify the banlist need to be modified? (I haven't looked deeply into what commands exist already, will need to talk to @lampholder about what his expected experience is here.)
* Does it make sense to store this data as events, like the ban list is stored?
* Does any of this need any sort of spec changes? (In particular I saw event types of both `org.matrix.room.rule` and `m.room.rule`)

Overall, this seems pretty doable, will be curious for feedback from @richvdh and @turt2live about the architectural changes to Synapse & Mjolnir.

@lampholder would interacting with this as part of Mjolnir meet your needs? Would you expect the same user ban list to be applied or would it need to be a separate list?
Mjolnir's role in this is probably best tracked elsewhere imo. The Synapse side really only needs to worry about supplying the function and calling it at the right time - how the spam checker deals with that is its problem.

To actually answer your questions though:
* Do the commands to modify the banlist need to be modified?
  * No, the commands stay the same.
* Does it make sense to store this data as events, like the ban list is stored?
  * Probably not? 
* Does any of this need any sort of spec changes?
  * No, Mjolnir is already handling the spec changes. And again, Synapse shouldn't concern itself with the details of how Mjolnir operates.

Mjolnir's changes would be minimal. Synapse's changes would also be relatively minimal and generic.

Resolving patch:
diff --git a/synapse/events/spamcheck.py b/synapse/events/spamcheck.py
--- a/synapse/events/spamcheck.py
+++ b/synapse/events/spamcheck.py
@@ -15,6 +15,7 @@
 # limitations under the License.
 
 import inspect
+from typing import Dict
 
 from synapse.spam_checker_api import SpamCheckerApi
 
@@ -125,3 +126,29 @@ def user_may_publish_room(self, userid, room_id):
             return True
 
         return self.spam_checker.user_may_publish_room(userid, room_id)
+
+    def check_username_for_spam(self, user_profile: Dict[str, str]) -> bool:
+        """Checks if a user ID or display name are considered "spammy" by this server.
+
+        If the server considers a username spammy, then it will not be included in
+        user directory results.
+
+        Args:
+            user_profile: The user information to check, it contains the keys:
+                * user_id
+                * display_name
+                * avatar_url
+
+        Returns:
+            True if the user is spammy.
+        """
+        if self.spam_checker is None:
+            return False
+
+        # For backwards compatibility, if the method does not exist on the spam checker, fallback to not interfering.
+        checker = getattr(self.spam_checker, "check_username_for_spam", None)
+        if not checker:
+            return False
+        # Make a copy of the user profile object to ensure the spam checker
+        # cannot modify it.
+        return checker(user_profile.copy())
diff --git a/synapse/handlers/user_directory.py b/synapse/handlers/user_directory.py
--- a/synapse/handlers/user_directory.py
+++ b/synapse/handlers/user_directory.py
@@ -52,6 +52,7 @@ def __init__(self, hs):
         self.is_mine_id = hs.is_mine_id
         self.update_user_directory = hs.config.update_user_directory
         self.search_all_users = hs.config.user_directory_search_all_users
+        self.spam_checker = hs.get_spam_checker()
         # The current position in the current_state_delta stream
         self.pos = None
 
@@ -65,7 +66,7 @@ def __init__(self, hs):
             # we start populating the user directory
             self.clock.call_later(0, self.notify_new_event)
 
-    def search_users(self, user_id, search_term, limit):
+    async def search_users(self, user_id, search_term, limit):
         """Searches for users in directory
 
         Returns:
@@ -82,7 +83,16 @@ def search_users(self, user_id, search_term, limit):
                     ]
                 }
         """
-        return self.store.search_user_dir(user_id, search_term, limit)
+        results = await self.store.search_user_dir(user_id, search_term, limit)
+
+        # Remove any spammy users from the results.
+        results["results"] = [
+            user
+            for user in results["results"]
+            if not self.spam_checker.check_username_for_spam(user)
+        ]
+
+        return results
 
     def notify_new_event(self):
         """Called when there may be more deltas to process


Test patch:
diff --git a/tests/handlers/test_user_directory.py b/tests/handlers/test_user_directory.py
--- a/tests/handlers/test_user_directory.py
+++ b/tests/handlers/test_user_directory.py
@@ -147,6 +147,98 @@ def test_private_room(self):
         s = self.get_success(self.handler.search_users(u1, "user3", 10))
         self.assertEqual(len(s["results"]), 0)
 
+    def test_spam_checker(self):
+        """
+        A user which fails to the spam checks will not appear in search results.
+        """
+        u1 = self.register_user("user1", "pass")
+        u1_token = self.login(u1, "pass")
+        u2 = self.register_user("user2", "pass")
+        u2_token = self.login(u2, "pass")
+
+        # We do not add users to the directory until they join a room.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 0)
+
+        room = self.helper.create_room_as(u1, is_public=False, tok=u1_token)
+        self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
+        self.helper.join(room, user=u2, tok=u2_token)
+
+        # Check we have populated the database correctly.
+        shares_private = self.get_users_who_share_private_rooms()
+        public_users = self.get_users_in_public_rooms()
+
+        self.assertEqual(
+            self._compress_shared(shares_private), set([(u1, u2, room), (u2, u1, room)])
+        )
+        self.assertEqual(public_users, [])
+
+        # We get one search result when searching for user2 by user1.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 1)
+
+        # Configure a spam checker that does not filter any users.
+        spam_checker = self.hs.get_spam_checker()
+
+        class AllowAll(object):
+            def check_username_for_spam(self, user_profile):
+                # Allow all users.
+                return False
+
+        spam_checker.spam_checker = AllowAll()
+
+        # The results do not change:
+        # We get one search result when searching for user2 by user1.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 1)
+
+        # Configure a spam checker that filters all users.
+        class BlockAll(object):
+            def check_username_for_spam(self, user_profile):
+                # All users are spammy.
+                return True
+
+        spam_checker.spam_checker = BlockAll()
+
+        # User1 now gets no search results for any of the other users.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 0)
+
+    def test_legacy_spam_checker(self):
+        """
+        A spam checker without the expected method should be ignored.
+        """
+        u1 = self.register_user("user1", "pass")
+        u1_token = self.login(u1, "pass")
+        u2 = self.register_user("user2", "pass")
+        u2_token = self.login(u2, "pass")
+
+        # We do not add users to the directory until they join a room.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 0)
+
+        room = self.helper.create_room_as(u1, is_public=False, tok=u1_token)
+        self.helper.invite(room, src=u1, targ=u2, tok=u1_token)
+        self.helper.join(room, user=u2, tok=u2_token)
+
+        # Check we have populated the database correctly.
+        shares_private = self.get_users_who_share_private_rooms()
+        public_users = self.get_users_in_public_rooms()
+
+        self.assertEqual(
+            self._compress_shared(shares_private), set([(u1, u2, room), (u2, u1, room)])
+        )
+        self.assertEqual(public_users, [])
+
+        # Configure a spam checker.
+        spam_checker = self.hs.get_spam_checker()
+        # The spam checker doesn't need any methods, so create a bare object.
+        spam_checker.spam_checker = object()
+
+        # We get one search result when searching for user2 by user1.
+        s = self.get_success(self.handler.search_users(u1, "user2", 10))
+        self.assertEqual(len(s["results"]), 1)
+
     def _compress_shared(self, shared):
         """
         Compress a list of users who share rooms dicts to a list of tuples.