Repository: zulip/zulip

Issue:
Sync LDAP e-Mail distributor with user groups
Hi together,

currently we are evaluating zulip. While setting up our test instance i noticed that there is no way in syncing our ldap groups with the zulip user-groups.

In our LDAP we have some e-mail groups which work as distributor, i'd like to use these as the base for zulip groups. In example, when someone is in the e-mail distributor "it@company.com" i'd like to have them in the zulip group "it".


Sync LDAP e-Mail distributor with user groups
Hi together,

currently we are evaluating zulip. While setting up our test instance i noticed that there is no way in syncing our ldap groups with the zulip user-groups.

In our LDAP we have some e-mail groups which work as distributor, i'd like to use these as the base for zulip groups. In example, when someone is in the e-mail distributor "it@company.com" i'd like to have them in the zulip group "it".




Discussion:
@H3npi thanks for the suggestion.  This should be doable with the Zulip API, though it's probably worth adding a quick little management command to do this; it shouldn't be hard.
@timabbott, how can this be done via the API? I do not see an API call for setting/managing user groups
@H3npi there are API calls similar to https://zulipchat.com/api/get-user-groups for editing them; they just aren't documented yet.  
(If you want to spelunk how to do it, `zerver/views/user_groups.py` in this project has the code)
Hi,
is the *group syncing with LDAP* actively being worked on or planned to include in a future release? I believe that together with https://github.com/zulip/zulip/issues/9480 this is a very much desired feature. A scenario could be, that a new user is added on the LDAP backend, can login to Zulip and automatically joins the streams which fit to its LDAP groups.
@lreiher would a better solution for your use case involve just syncing initial stream subscriptions from some LDAP groups?  
I am not so sure if I understand your comment right. Ideally, every LDAP group in a specific search scope would create a new user group in Zulip. Specific streams could then be made accessible to this user group by #9480. I guess that whenever Zulip syncs user data, it should then also sync group data to reflect the most recent changes to those groups.
Hello @zulip/server-authentication, @zulip/server-settings members, this issue was labeled with the "area: authentication", "area: settings (admin/org)" labels, so you may want to check it out!

<!-- areaLabelAddition -->
>(If you want to spelunk how to do it, zerver/views/user_groups.py in this project has the code)

@timabbott How would I go about using the API calls/functions in this file? They don't seem to be included in the `zulip` python package.
@jjblack just to check, is what you're looking for syncing Zulip streams with LDAP, or user groups (which are just groups of people you can mention) with LDAP?  This issue is just for the "user groups" feature.
@timabbott I am looking for a way to sync Zulip user groups with LDAP groups.
Hi,
I am also looking for this feature - currently we add users to groups manually in my org, LDAP group integration would be really useful.

@timabbott I suggest it might be the best to:
a) sync LDAP groups to Zulip groups (so they can be @mentioned),
and
b) allow adding entire Zulip groups to streams (this way the feature will work with other auth methods, not only with LDAP)
We are also looking for this feature. 
It would be great if an LDAP group is synced with the stream and/or mapped with groups. In any case it would be necessary to synchronize stream user and LDAP groups. 

Furthermore a support for group sync for SSO via SAML or Oauth would then be the perfect solution. 
@mateuszmandera we should discuss the right model for this after the 2.1 release work is complete.
Please also include the options to map ldap groups to zulip admins. :-)

For the rest: zulip LDAP implementation is really nice, and specifically: so much better than the one from rocket! It's the reason we prefer zulip over it! It just needs these group features, to eliminate the need to configure things locally in zulip.
Thanks for making zulip available! :-)
> Please also include the options to map ldap groups to zulip admins. :-)

That feature is already available. In settings.py you can do this:
```
AUTH_LDAP_USER_FLAGS_BY_GROUP = {
        "is_realm_admin": "cn=YourAdminGroup,ou=groups,dc=example,dc=com",
        "is_staff": "cn=YourStaffGroup,ou=groups,dc=example,dc=com",
}
```

But I am still missing the possibility to map LDAP groups to user groups in Zulip.
I need this functionality as well! Is there any news on when this could get implemented? It kind of feels like the first thing you would want after LDAP login.
We also need this functionality and it would go hand in hand with #9480 in my opinion - LDAP sync of the groups and members, then assign those groups to streams. This would greatly simplify group and stream management in Zulip.
Any update on a target fix for the groups?  We can manually update these, however it will make pushing for zulip to be our production product with out group changes in LDAP changing what streams people can see.
This is now a priority for the project, since it will be an important part of the overall vision detailed for using user groups for all permissions in Zulip detailed in #19525.

@mateuszmandera can I add this to your TODO list?
Retagged this as a Zulip 6.0 release goal, since that's the release we're targeting #19525 for.
I know it's already moving, but just another +1 for this feature :)  Glad to see it's right around the corner.
Any update ? Apparently It has not been included in Zulip 6.0 :/
> Any update ? Apparently It has not been included in Zulip 6.0 :/

I am working on partly implementing this feature, but it might take a while until it is merged. Maybe one or two more months.
The PR is almost done.

Just fyi, there is a follow up PR to complete Rocketchats sync feature:
https://github.com/zulip/zulip/pull/24490
(for anyone interested)
@H3npi thanks for the suggestion.  This should be doable with the Zulip API, though it's probably worth adding a quick little management command to do this; it shouldn't be hard.
@timabbott, how can this be done via the API? I do not see an API call for setting/managing user groups
@H3npi there are API calls similar to https://zulipchat.com/api/get-user-groups for editing them; they just aren't documented yet.  
(If you want to spelunk how to do it, `zerver/views/user_groups.py` in this project has the code)
Hi,
is the *group syncing with LDAP* actively being worked on or planned to include in a future release? I believe that together with https://github.com/zulip/zulip/issues/9480 this is a very much desired feature. A scenario could be, that a new user is added on the LDAP backend, can login to Zulip and automatically joins the streams which fit to its LDAP groups.
@lreiher would a better solution for your use case involve just syncing initial stream subscriptions from some LDAP groups?  
I am not so sure if I understand your comment right. Ideally, every LDAP group in a specific search scope would create a new user group in Zulip. Specific streams could then be made accessible to this user group by #9480. I guess that whenever Zulip syncs user data, it should then also sync group data to reflect the most recent changes to those groups.
Hello @zulip/server-authentication, @zulip/server-settings members, this issue was labeled with the "area: authentication", "area: settings (admin/org)" labels, so you may want to check it out!

<!-- areaLabelAddition -->
>(If you want to spelunk how to do it, zerver/views/user_groups.py in this project has the code)

@timabbott How would I go about using the API calls/functions in this file? They don't seem to be included in the `zulip` python package.
@jjblack just to check, is what you're looking for syncing Zulip streams with LDAP, or user groups (which are just groups of people you can mention) with LDAP?  This issue is just for the "user groups" feature.
@timabbott I am looking for a way to sync Zulip user groups with LDAP groups.
Hi,
I am also looking for this feature - currently we add users to groups manually in my org, LDAP group integration would be really useful.

@timabbott I suggest it might be the best to:
a) sync LDAP groups to Zulip groups (so they can be @mentioned),
and
b) allow adding entire Zulip groups to streams (this way the feature will work with other auth methods, not only with LDAP)
We are also looking for this feature. 
It would be great if an LDAP group is synced with the stream and/or mapped with groups. In any case it would be necessary to synchronize stream user and LDAP groups. 

Furthermore a support for group sync for SSO via SAML or Oauth would then be the perfect solution. 
@mateuszmandera we should discuss the right model for this after the 2.1 release work is complete.
Please also include the options to map ldap groups to zulip admins. :-)

For the rest: zulip LDAP implementation is really nice, and specifically: so much better than the one from rocket! It's the reason we prefer zulip over it! It just needs these group features, to eliminate the need to configure things locally in zulip.
Thanks for making zulip available! :-)
> Please also include the options to map ldap groups to zulip admins. :-)

That feature is already available. In settings.py you can do this:
```
AUTH_LDAP_USER_FLAGS_BY_GROUP = {
        "is_realm_admin": "cn=YourAdminGroup,ou=groups,dc=example,dc=com",
        "is_staff": "cn=YourStaffGroup,ou=groups,dc=example,dc=com",
}
```

But I am still missing the possibility to map LDAP groups to user groups in Zulip.
I need this functionality as well! Is there any news on when this could get implemented? It kind of feels like the first thing you would want after LDAP login.
We also need this functionality and it would go hand in hand with #9480 in my opinion - LDAP sync of the groups and members, then assign those groups to streams. This would greatly simplify group and stream management in Zulip.
Any update on a target fix for the groups?  We can manually update these, however it will make pushing for zulip to be our production product with out group changes in LDAP changing what streams people can see.
This is now a priority for the project, since it will be an important part of the overall vision detailed for using user groups for all permissions in Zulip detailed in #19525.

@mateuszmandera can I add this to your TODO list?
Retagged this as a Zulip 6.0 release goal, since that's the release we're targeting #19525 for.
I know it's already moving, but just another +1 for this feature :)  Glad to see it's right around the corner.
Any update ? Apparently It has not been included in Zulip 6.0 :/
> Any update ? Apparently It has not been included in Zulip 6.0 :/

I am working on partly implementing this feature, but it might take a while until it is merged. Maybe one or two more months.
The PR is almost done.

Just fyi, there is a follow up PR to complete Rocketchats sync feature:
https://github.com/zulip/zulip/pull/24490
(for anyone interested)

Resolving patch:
diff --git a/zproject/backends.py b/zproject/backends.py
--- a/zproject/backends.py
+++ b/zproject/backends.py
@@ -80,6 +80,10 @@
 
 from zerver.actions.create_user import do_create_user, do_reactivate_user
 from zerver.actions.custom_profile_fields import do_update_user_custom_profile_data_if_changed
+from zerver.actions.user_groups import (
+    bulk_add_members_to_user_groups,
+    bulk_remove_members_from_user_groups,
+)
 from zerver.actions.user_settings import do_regenerate_api_key
 from zerver.actions.users import do_deactivate_user
 from zerver.lib.avatar import avatar_url, is_avatar_new
@@ -105,6 +109,8 @@
     PreregistrationRealm,
     PreregistrationUser,
     Realm,
+    UserGroup,
+    UserGroupMembership,
     UserProfile,
     custom_profile_fields_for_realm,
     get_realm,
@@ -910,6 +916,78 @@ def sync_custom_profile_fields_from_ldap(
         except SyncUserError as e:
             raise ZulipLDAPError(str(e)) from e
 
+    def sync_groups_from_ldap(self, user_profile: UserProfile, ldap_user: _LDAPUser) -> None:
+        """
+        For the groups set up for syncing for the realm in LDAP_SYNCHRONIZED_GROUPS_BY_REALM:
+
+        (1) Makes sure the user has membership in the Zulip UserGroups corresponding
+            to the LDAP groups ldap_user belongs to.
+        (2) Makes sure the user doesn't have membership in the Zulip UserGroups corresponding
+            to the LDAP groups ldap_user doesn't belong to.
+        """
+
+        if user_profile.realm.string_id not in settings.LDAP_SYNCHRONIZED_GROUPS_BY_REALM:
+            # no groups to sync for this realm
+            return
+
+        configured_ldap_group_names_for_sync = set(
+            settings.LDAP_SYNCHRONIZED_GROUPS_BY_REALM[user_profile.realm.string_id]
+        )
+
+        try:
+            ldap_logger.debug("Syncing groups for user: %s", user_profile.id)
+            intended_group_name_set_for_user = set(ldap_user.group_names).intersection(
+                configured_ldap_group_names_for_sync
+            )
+
+            existing_group_name_set_for_user = set(
+                UserGroupMembership.objects.filter(
+                    user_group__realm=user_profile.realm,
+                    user_group__name__in=set(
+                        settings.LDAP_SYNCHRONIZED_GROUPS_BY_REALM[user_profile.realm.string_id]
+                    ),
+                    user_profile=user_profile,
+                ).values_list("user_group__name", flat=True)
+            )
+
+            ldap_logger.debug(
+                "intended groups: %s; zulip groups: %s",
+                repr(intended_group_name_set_for_user),
+                repr(existing_group_name_set_for_user),
+            )
+
+            new_groups = UserGroup.objects.filter(
+                name__in=intended_group_name_set_for_user.difference(
+                    existing_group_name_set_for_user
+                ),
+                realm=user_profile.realm,
+            )
+            if new_groups:
+                ldap_logger.debug(
+                    "add %s to %s", user_profile.id, [group.name for group in new_groups]
+                )
+                bulk_add_members_to_user_groups(new_groups, [user_profile.id], acting_user=None)
+
+            group_names_for_membership_deletion = existing_group_name_set_for_user.difference(
+                intended_group_name_set_for_user
+            )
+            groups_for_membership_deletion = UserGroup.objects.filter(
+                name__in=group_names_for_membership_deletion, realm=user_profile.realm
+            )
+
+            if group_names_for_membership_deletion:
+                ldap_logger.debug(
+                    "removing groups %s from %s",
+                    group_names_for_membership_deletion,
+                    user_profile.id,
+                )
+                bulk_remove_members_from_user_groups(
+                    groups_for_membership_deletion, [user_profile.id], acting_user=None
+                )
+
+        except Exception as e:
+            raise ZulipLDAPError(str(e)) from e
+
 
 class ZulipLDAPAuthBackend(ZulipLDAPAuthBackendBase):
     REALM_IS_NONE_ERROR = 1
@@ -1136,6 +1214,7 @@ def get_or_build_user(
         self.sync_avatar_from_ldap(user, ldap_user)
         self.sync_full_name_from_ldap(user, ldap_user)
         self.sync_custom_profile_fields_from_ldap(user, ldap_user)
+        self.sync_groups_from_ldap(user, ldap_user)
         return (user, built)
 
 
diff --git a/zproject/default_settings.py b/zproject/default_settings.py
--- a/zproject/default_settings.py
+++ b/zproject/default_settings.py
@@ -2,6 +2,8 @@
 from email.headerregistry import Address
 from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple
 
+from django_auth_ldap.config import GroupOfUniqueNamesType, LDAPGroupType
+
 from scripts.lib.zulip_tools import deport
 from zproject.settings_types import JwtAuthKey, OIDCIdPConfigDict, SAMLIdPConfigDict
 
@@ -69,6 +71,8 @@
 FAKE_LDAP_MODE: Optional[str] = None
 FAKE_LDAP_NUM_USERS = 8
 AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL: Optional[Dict[str, Any]] = None
+LDAP_SYNCHRONIZED_GROUPS_BY_REALM: Dict[str, List[str]] = {}
+AUTH_LDAP_GROUP_TYPE: LDAPGroupType = GroupOfUniqueNamesType()
 
 # Social auth; we support providing values for some of these
 # settings in zulip-secrets.conf instead of settings.py in development.
diff --git a/zproject/prod_settings_template.py b/zproject/prod_settings_template.py
--- a/zproject/prod_settings_template.py
+++ b/zproject/prod_settings_template.py
@@ -266,6 +266,20 @@
 #    ]
 # }
 
+
+## LDAP group sync configuration.
+## See: https://zulip.readthedocs.io/en/latest/production/authentication-methods.html#synchronizing-groups
+# AUTH_LDAP_GROUP_TYPE = GroupOfUniqueNamesType()
+# AUTH_LDAP_GROUP_SEARCH = LDAPSearch(
+#     "ou=groups,dc=www,dc=example,dc=com", ldap.SCOPE_SUBTREE, "(objectClass=groupOfUniqueNames)"
+# )
+# LDAP_SYNCHRONIZED_GROUPS_BY_REALM = {
+#   "subdomain1" : [
+#       "group1",
+#       "group2",
+#   ]
+# }
+
 ########
 ## Google OAuth.
 ##


Test patch:
diff --git a/zerver/tests/fixtures/ldap/directory.json b/zerver/tests/fixtures/ldap/directory.json
--- a/zerver/tests/fixtures/ldap/directory.json
+++ b/zerver/tests/fixtures/ldap/directory.json
@@ -77,5 +77,24 @@
         "uid": ["user2_with_shared_email"],
         "sn": ["shortname"],
         "mail": ["shared_email@zulip.com"]
+    },
+
+    "ou=groups,dc=zulip,dc=com": {
+        "ou": "groups"
+    },
+    "cn=cool_test_group,ou=groups,dc=zulip,dc=com": {
+        "objectClass": ["groupOfUniqueNames"],
+        "cn": ["cool_test_group"],
+        "uniqueMember": [
+            "uid=hamlet,ou=users,dc=zulip,dc=com"
+        ]
+    },
+    "cn=another_test_group,ou=groups,dc=zulip,dc=com": {
+        "objectClass": ["groupOfUniqueNames"],
+        "cn": ["another_test_group"],
+        "uniqueMember": [
+            "uid=hamlet,ou=users,dc=zulip,dc=com",
+            "uid=cordelia,ou=users,dc=zulip,dc=com"
+        ]
     }
 }
diff --git a/zerver/tests/test_auth_backends.py b/zerver/tests/test_auth_backends.py
--- a/zerver/tests/test_auth_backends.py
+++ b/zerver/tests/test_auth_backends.py
@@ -63,6 +63,10 @@
     do_set_realm_authentication_methods,
     do_set_realm_property,
 )
+from zerver.actions.user_groups import (
+    bulk_add_members_to_user_groups,
+    create_user_group_in_database,
+)
 from zerver.actions.user_settings import do_change_password, do_change_user_setting
 from zerver.actions.users import change_user_is_active, do_deactivate_user
 from zerver.lib.avatar import avatar_url
@@ -89,6 +93,7 @@
 )
 from zerver.lib.types import Validator
 from zerver.lib.upload.base import DEFAULT_AVATAR_SIZE, MEDIUM_AVATAR_SIZE, resize_avatar
+from zerver.lib.user_groups import is_user_in_group
 from zerver.lib.users import get_all_api_keys, get_api_key, get_raw_user_data
 from zerver.lib.utils import assert_is_not_none
 from zerver.lib.validator import (
@@ -109,6 +114,7 @@
     Realm,
     RealmDomain,
     Stream,
+    UserGroup,
     UserProfile,
     clear_supported_auth_backends_cache,
     get_realm,
@@ -7248,5 +7254,136 @@ def test_invalid_realm_for_user_failure(self) -> None:
             self.assert_json_error_contains(result, "Invalid subdomain", 404)
 
 
+class LDAPGroupSyncTest(ZulipTestCase):
+    @override_settings(AUTHENTICATION_BACKENDS=("zproject.backends.ZulipLDAPAuthBackend",))
+    def test_ldap_group_sync(self) -> None:
+        self.init_default_ldap_database()
+
+        hamlet = self.example_user("hamlet")
+        with self.settings(LDAP_APPEND_DOMAIN="zulip.com"):
+            result = sync_user_from_ldap(hamlet, mock.Mock())
+            self.assertTrue(result)
+        self.assertTrue(hamlet.is_active)
+
+        realm = get_realm("zulip")
+
+        with self.settings(
+            AUTH_LDAP_GROUP_SEARCH=LDAPSearch(
+                "ou=groups,dc=zulip,dc=com",
+                ldap.SCOPE_ONELEVEL,
+                "(objectClass=groupOfUniqueNames)",
+            ),
+            LDAP_SYNCHRONIZED_GROUPS_BY_REALM={
+                "zulip": [
+                    "cool_test_group",
+                ]
+            },
+            LDAP_APPEND_DOMAIN="zulip.com",
+        ), self.assertLogs("zulip.ldap", "DEBUG") as zulip_ldap_log:
+            self.assertFalse(UserGroup.objects.filter(realm=realm, name="cool_test_group").exists())
+
+            create_user_group_in_database(
+                "cool_test_group", [], realm, acting_user=None, description="Created by LDAP sync"
+            )
+
+            self.assertTrue(UserGroup.objects.filter(realm=realm, name="cool_test_group").exists())
+
+            user_group = UserGroup.objects.get(realm=realm, name="cool_test_group")
+
+            self.assertFalse(
+                is_user_in_group(
+                    user_group,
+                    hamlet,
+                    direct_member_only=True,
+                )
+            )
+
+            sync_user_from_ldap(hamlet, mock.Mock())
+            self.assertTrue(
+                is_user_in_group(
+                    user_group,
+                    hamlet,
+                    direct_member_only=True,
+                )
+            )
+
+            # Add a user to a Zulip group that they are not member of in ldap.
+            # This implies that they should be deleted from the Zulip group
+            # upon the next sync.
+            cordelia = self.example_user("cordelia")
+            bulk_add_members_to_user_groups(
+                [user_group],
+                [cordelia.id],
+                acting_user=None,
+            )
+
+            self.assertTrue(
+                is_user_in_group(
+                    UserGroup.objects.get(realm=realm, name="cool_test_group"),
+                    cordelia,
+                    direct_member_only=True,
+                )
+            )
+
+            # This should remove cordelia from cool_test_group
+            sync_user_from_ldap(cordelia, mock.Mock())
+
+            self.assertFalse(
+                is_user_in_group(
+                    UserGroup.objects.get(realm=realm, name="cool_test_group"),
+                    cordelia,
+                    direct_member_only=True,
+                )
+            )
+
+        hamlet = self.example_user("hamlet")
+        cordelia = self.example_user("cordelia")
+
+        self.assertEqual(
+            zulip_ldap_log.output,
+            [
+                f"DEBUG:zulip.ldap:Syncing groups for user: {hamlet.id}",
+                "DEBUG:zulip.ldap:intended groups: {'cool_test_group'}; zulip groups: set()",
+                f"DEBUG:zulip.ldap:add {hamlet.id} to ['cool_test_group']",
+                f"DEBUG:zulip.ldap:Syncing groups for user: {cordelia.id}",
+                "DEBUG:zulip.ldap:intended groups: set(); zulip groups: {'cool_test_group'}",
+                f"DEBUG:zulip.ldap:removing groups {{'cool_test_group'}} from {cordelia.id}",
+            ],
+        )
+
+        # Test an exception using a malformed ldap group search setting.
+        with self.settings(
+            AUTH_LDAP_GROUP_SEARCH=LDAPSearch(
+                "ou=groups,dc=zulip,dc=com",
+                ldap.SCOPE_ONELEVEL,
+                "(objectClass=groupOfUniqueNames",  # this is malformed, missing ")"
+            ),
+            LDAP_SYNCHRONIZED_GROUPS_BY_REALM={
+                "zulip": [
+                    "cool_test_group",
+                ]
+            },
+            LDAP_APPEND_DOMAIN="zulip.com",
+        ), self.assertLogs("django_auth_ldap", "WARN") as django_ldap_log, self.assertLogs(
+            "zulip.ldap", "DEBUG"
+        ) as zulip_ldap_log:
+            with self.assertRaisesRegex(
+                ZulipLDAPError,
+                "search_s.*",
+            ):
+                sync_user_from_ldap(cordelia, mock.Mock())
+
+        self.assertEqual(
+            zulip_ldap_log.output,
+            [f"DEBUG:zulip.ldap:Syncing groups for user: {cordelia.id}"],
+        )
+        self.assertEqual(
+            django_ldap_log.output,
+            [
+                'WARNING:django_auth_ldap:search_s("ou=groups,dc=zulip,dc=com", 1, "(&(objectClass=groupOfUniqueNames(uniqueMember=uid=cordelia,ou=users,dc=zulip,dc=com))", "None", 0) while authenticating cordelia',
+            ],
+        )
+
+
 # Don't load the base class as a test: https://bugs.python.org/issue17519.
 del SocialAuthBase