Repository: privacyidea/privacyidea

Issue:
The Serverpool in LDAP resolver is not persistent
The serverpool in the ldap resolver is not persistent but rather created on each ``checkPass``.

As far as normal quering is concerned, it is persistent if the bind is still active. Nevertheless we should move the serverpool into a member ``self.serverpool``.


Discussion:
First idea was to make the serverpool global per-process. However, this doesn't really help, because ldap3 stores the server pool *state* separately for *each connection object*! So with two servers A and B we have the following problem: Even if connection 1 learns that server A is down and uses B instead, connection 2 does not have this information and might even try A again!

We *could* use the ldap3 REUSABLE strategy ([see here](https://ldap3.readthedocs.io/connection.html)), which keeps a connection pool, but it isn't well-documented and it is asynchronous, which means we would have to restructure the LDAP resolver code.

We could maybe write our own ServerPool implementation?
Some ideas for an alternate ServerPool implementation:
* We could have a per-process server pool that holds the information which servers are online and which are offline. We would have to implement the server pool in a thread-safe manner.
* In order to check if a pool server is available, the ``ldap3.ServerPool`` opens and closes a TCP socket. This is often enough, but doesn't cover the case that a pool server accepts connections, but does not answer LDAP requests. Our alternate pool implementation could attempt a LDAP Bind with the service account credentials to also cover this case. If the Bind fails, the server is considered offline.
* We could implement several strategies to select servers from the pool (analogously to ldap3): Strategies like "round robin" or "random" could attempt to distribute LDAP connections evenly on all available pool servers. A "failover" strategy could send all LDAP connections to the first server in the pool and only use the other servers in case the first server goes down.
Starting with ldap3 2.6, the ``ServerPool`` class now uses a single state instead of one state per connection connection (see https://github.com/cannatag/ldap3/issues/649). This enables us to implement a proper *per-process* server pool. So I'm reopening this and assigning it to 3.1

Resolving patch:
diff --git a/privacyidea/lib/machines/ldap.py b/privacyidea/lib/machines/ldap.py
--- a/privacyidea/lib/machines/ldap.py
+++ b/privacyidea/lib/machines/ldap.py
@@ -65,8 +65,8 @@ def __init__(self, name, config=None):
 
     def _bind(self):
         if not self.i_am_bound:
-            server_pool = IdResolver.get_serverpool(self.uri, self.timeout,
-                                                    tls_context=self.tls_context)
+            server_pool = IdResolver.create_serverpool(self.uri, self.timeout,
+                                                       tls_context=self.tls_context)
             self.l = IdResolver.create_connection(authtype=self.authtype,
                                                   server=server_pool,
                                                   user=self.binddn,
@@ -312,10 +312,10 @@ def testconnection(params):
         else:
             tls_context = None
         try:
-            server_pool = IdResolver.get_serverpool(ldap_uri,
-                                                    float(params.get(
+            server_pool = IdResolver.create_serverpool(ldap_uri,
+                                                       float(params.get(
                                                         "TIMEOUT", 5)),
-                                                    tls_context=tls_context)
+                                                       tls_context=tls_context)
             l = IdResolver.create_connection(authtype=\
                                                  params.get("AUTHTYPE",
                                                             AUTHTYPE.SIMPLE),
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -58,6 +58,7 @@
 
 import logging
 import yaml
+import threading
 import functools
 
 from .UserIdResolver import UserIdResolver
@@ -77,6 +78,7 @@
 import binascii
 from privacyidea.lib.crypto import urandom, geturandom
 from privacyidea.lib.utils import is_true
+from privacyidea.lib.framework import get_app_local_store
 import datetime
 
 from privacyidea.lib import _
@@ -110,6 +112,33 @@
     DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
 
 
+class LockingServerPool(ldap3.ServerPool):
+    """
+    A ``ServerPool`` subclass that uses a RLock to synchronize invocations of
+    ``initialize``, ``get_server`` and ``get_current_server``.
+
+    We synchronize invocations to rule out race conditions when multiple threads
+    try to manipulate the server pool state concurrently.
+
+    We use a ``RLock`` instead of a simple ``Lock`` to avoid locking ourselves.
+    """
+    def __init__(self, *args, **kwargs):
+        ldap3.ServerPool.__init__(self, *args, **kwargs)
+        self._lock = threading.RLock()
+
+    def initialize(self, connection):
+        with self._lock:
+            return ldap3.ServerPool.initialize(self, connection)
+
+    def get_server(self, connection):
+        with self._lock:
+            return ldap3.ServerPool.get_server(self, connection)
+
+    def get_current_server(self, connection):
+        with self._lock:
+            return ldap3.ServerPool.get_current_server(self, connection)
+
+
 def get_ad_timestamp_now():
     """
     returns the current UTC time as it is used in Active Directory in the
@@ -273,6 +302,7 @@ def __init__(self):
         self.cache_timeout = 120
         self.tls_context = None
         self.start_tls = False
+        self.serverpool_persistent = False
         self.serverpool_rounds = SERVERPOOL_ROUNDS
         self.serverpool_skip = SERVERPOOL_SKIP
         self.serverpool = None
@@ -298,11 +328,7 @@ def checkPass(self, uid, password):
             bind_user = self._getDN(uid)
 
         if not self.serverpool:
-            self.serverpool = self.get_serverpool(self.uri, self.timeout,
-                                                  get_info=ldap3.NONE,
-                                                  tls_context=self.tls_context,
-                                                  rounds=self.serverpool_rounds,
-                                                  exhaust=self.serverpool_skip)
+            self.serverpool = self.get_serverpool_instance(get_info=ldap3.NONE)
 
         try:
             log.debug("Authtype: {0!r}".format(self.authtype))
@@ -445,11 +471,7 @@ def _getDN(self, userId):
     def _bind(self):
         if not self.i_am_bound:
             if not self.serverpool:
-                self.serverpool = self.get_serverpool(self.uri, self.timeout,
-                                              get_info=self.get_info,
-                                              tls_context=self.tls_context,
-                                              rounds=self.serverpool_rounds,
-                                              exhaust=self.serverpool_skip)
+                self.serverpool = self.get_serverpool_instance(self.get_info)
             self.l = self.create_connection(authtype=self.authtype,
                                             server=self.serverpool,
                                             user=self.binddn,
@@ -753,6 +775,7 @@ def loadConfig(self, config):
                                    ca_certs_file=self.tls_ca_file)
         else:
             self.tls_context = None
+        self.serverpool_persistent = is_true(config.get("SERVERPOOL_PERSISTENT", False))
         self.serverpool_rounds = int(config.get("SERVERPOOL_ROUNDS") or SERVERPOOL_ROUNDS)
         self.serverpool_skip = int(config.get("SERVERPOOL_SKIP") or SERVERPOOL_SKIP)
         # The configuration might have changed. We reset the serverpool
@@ -803,8 +826,8 @@ def split_uri(uri):
         return server, port, ssl
 
     @classmethod
-    def get_serverpool(cls, urilist, timeout, get_info=None, tls_context=None, rounds=SERVERPOOL_ROUNDS,
-                       exhaust=SERVERPOOL_SKIP):
+    def create_serverpool(cls, urilist, timeout, get_info=None, tls_context=None, rounds=SERVERPOOL_ROUNDS,
+                          exhaust=SERVERPOOL_SKIP, pool_cls=ldap3.ServerPool):
         """
         This create the serverpool for the ldap3 connection.
         The URI from the LDAP resolver can contain a comma separated list of
@@ -826,13 +849,14 @@ def get_serverpool(cls, urilist, timeout, get_info=None, tls_context=None, round
             before giving up
         :param exhaust: The seconds, for how long a non-reachable server should be
             removed from the serverpool
+        :param pool_cls: ``ldap3.ServerPool`` subclass that should be instantiated
         :return: Server Pool
-        :rtype: LDAP3 Server Pool Instance
+        :rtype: serverpool_cls
         """
         get_info = get_info or ldap3.SCHEMA
-        server_pool = ldap3.ServerPool(None, ldap3.ROUND_ROBIN,
-                                       active=rounds,
-                                       exhaust=exhaust)
+        server_pool = pool_cls(None, ldap3.ROUND_ROBIN,
+                               active=rounds,
+                               exhaust=exhaust)
         for uri in urilist.split(","):
             uri = uri.strip()
             host, port, ssl = cls.split_uri(uri)
@@ -845,6 +869,56 @@ def get_serverpool(cls, urilist, timeout, get_info=None, tls_context=None, round
             log.debug("Added {0!s}, {1!s}, {2!s} to server pool.".format(host, port, ssl))
         return server_pool
 
+    def get_serverpool_instance(self, get_info=None):
+        """
+        Return a ``ServerPool`` instance that should be used. If ``SERVERPOOL_PERSISTENT``
+        is enabled, invoke ``get_persistent_serverpool`` to retrieve a per-process
+        server pool instance. If it is not enabled, invoke ``create_serverpool``
+        to retrieve a per-request server pool instance.
+        :param get_info: one of ldap3.SCHEMA, ldap3.NONE, ldap3.ALL
+        :return: a ``ServerPool``/``LockingServerPool`` instance
+        """
+        if self.serverpool_persistent:
+            return self.get_persistent_serverpool(get_info)
+        else:
+            return self.create_serverpool(self.uri, self.timeout, get_info,
+                                          self.tls_context, self.serverpool_rounds, self.serverpool_skip)
+
+    def get_persistent_serverpool(self, get_info=None):
+        """
+        Return a process-level instance of ``LockingServerPool`` for the current LDAP resolver
+        configuration. Retrieve it from the app-local store. If such an instance does not exist
+        yet, create one.
+        :param get_info: one of ldap3.SCHEMA, ldap3.NONE, ldap3.ALL
+        :return: a ``LockingServerPool`` instance
+        """
+        if not get_info:
+            get_info = ldap3.SCHEMA
+        pools = get_app_local_store().setdefault('ldap_server_pools', {})
+        # Create a hashable tuple that describes the current server pool configuration
+        pool_description = (self.uri,
+                            self.timeout,
+                            get_info,
+                            repr(self.tls_context),  # this is the string representation of the TLS context
+                            self.serverpool_rounds,
+                            self.serverpool_skip)
+        if pool_description not in pools:
+            log.debug("Creating a persistent server pool instance for {!r} ...".format(pool_description))
+            # Create a suitable instance of ``LockingServerPool``
+            server_pool = self.create_serverpool(self.uri, self.timeout, get_info,
+                                                 self.tls_context, self.serverpool_rounds, self.serverpool_skip,
+                                                 pool_cls=LockingServerPool)
+            # It may happen that another thread tries to add an instance to the dictionary concurrently.
+            # However, only one of them will win, and the other ``LockingServerPool`` instance will be
+            # garbage-collected eventually.
+            return pools.setdefault(pool_description, server_pool)
+        else:
+            # If there is already a ``LockingServerPool`` instance, return it.
+            # We never remove instances from the dictionary, so a ``KeyError`` cannot occur.
+            # As a side effect, when we change the LDAP Id resolver configuration,
+            # outdated ``LockingServerPool`` instances will survive until the next server restart.
+            return pools[pool_description]
+
     @classmethod
     def getResolverClassDescriptor(cls):
         """
@@ -881,6 +955,7 @@ def getResolverClassDescriptor(cls):
                                 'CACHE_TIMEOUT': 'int',
                                 'SERVERPOOL_ROUNDS': 'int',
                                 'SERVERPOOL_SKIP': 'int',
+                                'SERVERPOOL_PERSISTENT': 'bool',
                                 'OBJECT_CLASSES': 'string',
                                 'DN_TEMPLATE': 'string'}
         return {typ: descriptor}
@@ -920,11 +995,11 @@ def testconnection(cls, param):
             tls_context = None
         get_info = get_info_configuration(is_true(param.get("NOSCHEMAS")))
         try:
-            server_pool = cls.get_serverpool(ldap_uri, timeout,
-                                             tls_context=tls_context,
-                                             get_info=get_info,
-                                             rounds=serverpool_rounds,
-                                             exhaust=serverpool_skip)
+            server_pool = cls.create_serverpool(ldap_uri, timeout,
+                                                tls_context=tls_context,
+                                                get_info=get_info,
+                                                rounds=serverpool_rounds,
+                                                exhaust=serverpool_skip)
             l = cls.create_connection(authtype=param.get("AUTHTYPE",
                                                           AUTHTYPE.SIMPLE),
                                       server=server_pool,


Test patch:
diff --git a/tests/test_lib_resolver.py b/tests/test_lib_resolver.py
--- a/tests/test_lib_resolver.py
+++ b/tests/test_lib_resolver.py
@@ -14,11 +14,12 @@
 from ldap3.core.exceptions import LDAPOperationResult
 from ldap3.core.results import RESULT_SIZE_LIMIT_EXCEEDED
 import mock
+import ldap3
 import responses
 import datetime
 import uuid
 import pytest
-from privacyidea.lib.resolvers.LDAPIdResolver import IdResolver as LDAPResolver
+from privacyidea.lib.resolvers.LDAPIdResolver import IdResolver as LDAPResolver, LockingServerPool
 from privacyidea.lib.resolvers.SQLIdResolver import IdResolver as SQLResolver
 from privacyidea.lib.resolvers.SCIMIdResolver import IdResolver as SCIMResolver
 from privacyidea.lib.resolvers.UserIdResolver import UserIdResolver
@@ -1128,28 +1129,39 @@ def test_06_split_uri(self):
     def test_07_get_serverpool(self):
         timeout = 5
         urilist = "ldap://themis"
-        server_pool = LDAPResolver.get_serverpool(urilist, timeout)
+        server_pool = LDAPResolver.create_serverpool(urilist, timeout)
         self.assertEqual(len(server_pool), 1)
         self.assertEqual(server_pool.active, SERVERPOOL_ROUNDS)
         self.assertEqual(server_pool.exhaust, SERVERPOOL_SKIP)
         self.assertEqual(server_pool.strategy, "ROUND_ROBIN")
 
         urilist = "ldap://themis, ldap://server2"
-        server_pool = LDAPResolver.get_serverpool(urilist, timeout)
+        server_pool = LDAPResolver.create_serverpool(urilist, timeout)
         self.assertEqual(len(server_pool), 2)
         self.assertEqual(server_pool.servers[0].name, "ldap://themis:389")
         self.assertEqual(server_pool.servers[1].name, "ldap://server2:389")
 
         urilist = "ldap://themis, ldaps://server2"
-        server_pool = LDAPResolver.get_serverpool(urilist, timeout)
+        server_pool = LDAPResolver.create_serverpool(urilist, timeout)
         self.assertEqual(len(server_pool), 2)
         self.assertEqual(server_pool.servers[0].name, "ldap://themis:389")
         self.assertEqual(server_pool.servers[1].name, "ldaps://server2:636")
 
         urilist = "ldap://themis, ldaps://server2"
-        server_pool = LDAPResolver.get_serverpool(urilist, timeout,
-                                                  rounds=5,
-                                                  exhaust=60)
+        server_pool = LDAPResolver.create_serverpool(urilist, timeout,
+                                                     rounds=5,
+                                                     exhaust=60)
+        self.assertEqual(len(server_pool), 2)
+        self.assertEqual(server_pool.active, 5)
+        self.assertEqual(server_pool.exhaust, 60)
+        self.assertEqual(server_pool.strategy, "ROUND_ROBIN")
+
+        urilist = "ldap://themis, ldaps://server2"
+        server_pool = LDAPResolver.create_serverpool(urilist, timeout,
+                                                     rounds=5,
+                                                     exhaust=60,
+                                                     pool_cls=LockingServerPool)
+        self.assertIs(type(server_pool), LockingServerPool)
         self.assertEqual(len(server_pool), 2)
         self.assertEqual(server_pool.active, 5)
         self.assertEqual(server_pool.exhaust, 60)
@@ -2037,6 +2049,61 @@ def test_34_censored_tests(self):
         r = delete_resolver("testname1")
         self.assertTrue(r)
 
+    @ldap3mock.activate
+    def test_35_persistent_serverpool(self):
+        ldap3mock.setLDAPDirectory(LDAPDirectory)
+        params = {'LDAPURI': 'ldap://localhost, ldap://127.0.0.1, ldap://127.0.1.1',
+                      'LDAPBASE': 'o=test',
+                      'BINDDN': 'cn=manager,ou=example,o=test',
+                      'BINDPW': 'ldaptest',
+                      'LOGINNAMEATTRIBUTE': 'cn',
+                      'LDAPSEARCHFILTER': '(cn=*)',
+                      'USERINFO': '{ "username": "cn", "phone": "telephoneNumber", '
+                                  '"mobile" : "mobile", "email": "mail", '
+                                  '"surname" : "sn", "givenname": "givenName" }',
+                      'UIDTYPE': 'DN',
+                      'CACHE_TIMEOUT': '0', # to disable the per-process cache
+                      'resolver': 'testpool',
+                      'type': 'ldapresolver'}
+        y1 = LDAPResolver()
+        y1.loadConfig(params)
+        y2 = LDAPResolver()
+        y2.loadConfig(params)
+        # Make a query, so that a ServerPool is instantiated
+        y1.getUserId('bob')
+        y2.getUserId('bob')
+        # We haven't configured a persistent serverpool, so every resolver has its own ServerPool
+        self.assertIs(type(y1.serverpool), ldap3.ServerPool)
+        self.assertIs(type(y2.serverpool), ldap3.ServerPool)
+        self.assertIsNot(y1.serverpool, y2.serverpool)
+        # Now, we configure a persistent serverpool
+        params["SERVERPOOL_PERSISTENT"] = "true"
+        y3 = LDAPResolver()
+        y3.loadConfig(params)
+        y4 = LDAPResolver()
+        y4.loadConfig(params)
+        y3.getUserId('bob')
+        y4.getUserId('bob')
+        # The resolvers share a ServerPool
+        self.assertIs(type(y3.serverpool), LockingServerPool)
+        self.assertIs(type(y4.serverpool), LockingServerPool)
+        self.assertIs(y3.serverpool, y4.serverpool)
+
+    def test_36_locking_serverpool(self):
+        # check that the LockingServerPool correctly forwards all relevant method calls
+        pool = LockingServerPool()
+        pool.add(ldap3.Server('server1'))
+        pool.add(ldap3.Server('server2'))
+        with mock.patch('ldap3.ServerPool.initialize') as mock_method:
+            pool.initialize(None)
+            mock_method.assert_called_once()
+        with mock.patch('ldap3.ServerPool.get_server') as mock_method:
+            pool.get_server(None)
+            mock_method.assert_called_once()
+        with mock.patch('ldap3.ServerPool.get_current_server') as mock_method:
+            pool.get_current_server(None)
+            mock_method.assert_called_once()
+
 class BaseResolverTestCase(MyTestCase):
 
     def test_00_basefunctions(self):