Repository: privacyidea/privacyidea

Issue:
LdapServerPoolExhaustedError on Authentications
Webserver - Uwsgi/Nginx

Hello,

I am seeing the LDAPServerPoolExhaustedError on /validate/check.

From the [source code](https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/resolvers/LDAPIdResolver.py#L87) I see the default is 2 cycles (active) of 30s (exhaust) period to keep an inactive LDAP server from the pool. The LDAP library will throw the exception if no server is hit during or after the completion of this cycle. 

A networking issue could have occurred and it resolved itself, is there a more graceful handle on the exception? Or is this something safe to ignore until the problematic server comes online?

I am looking forward to your input.

Here is the stacktrace:

<pre>
ERROR in app: Exception on /validate/check [POST]
Traceback (most recent call last):
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
  response = self.full_dispatch_request()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
  rv = self.handle_user_exception(e)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
  reraise(exc_type, exc_value, tb)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/flask/app.py", line 1610, in full_dispatch_request
  rv = self.preprocess_request()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/flask/app.py", line 1831, in preprocess_request
  rv = func()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/api/validate.py", line 117, in before_request
  request.User = get_user_from_param(request.all_data)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/log.py", line 154, in log_wrapper
  return func(*args, **kwds)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 576, in get_user_from_param
  resolver=param.get("resolver"))
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/log.py", line 154, in log_wrapper
  return func(*args, **kwds)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 98, in __init__
  self._get_user_from_userstore()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/usercache.py", line 63, in cache_wrapper
  return wrapped_function(*args, **kwds)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 106, in _get_user_from_userstore
  self._get_resolvers()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 207, in _get_resolvers
  if self._locate_user_in_resolver(resolvername):
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/user.py", line 227, in _locate_user_in_resolver
  uid = y.getUserId(self.login)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/resolvers/LDAPIdResolver.py", line 163, in cache_wrapper
  f_result = func(self, *args, **kwds)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/resolvers/LDAPIdResolver.py", line 463, in getUserId
  self._bind()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/privacyidea/lib/resolvers/LDAPIdResolver.py", line 374, in _bind
  if not self.l.bind():
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/core/connection.py", line 513, in bind
  self.open(read_server_info=False)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/strategy/sync.py", line 56, in open
  BaseStrategy.open(self, reset_usage, read_server_info)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/strategy/base.py", line 116, in open
  new_server = self.connection.server_pool.get_server(self.connection) # get a server from the server_pool if available
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/core/pooling.py", line 294, in get_server
  return self.pool_states[connection].get_server()
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/core/pooling.py", line 84, in get_server
  self.last_used_server = self.find_active_server(self.last_used_server + 1)
 File "/opt/privacyidea/local/lib/python2.7/dist-packages/ldap3/core/pooling.py", line 179, in find_active_server
  raise LDAPServerPoolExhaustedError('no active server available in server pool after maximum number of tries') LDAPServerPoolExhaustedError: no active server available in server pool after maximum number of tries
</pre>


Discussion:
Can you tell, what occurred in your situation?

The LDAP request itself has a timeout, which you can configure. The challenge is, to find nice working values.
If an LDAP server is down for a longer time, than it will be added to the pool after the what you call exhaust period. If it is put back in the pool to early, you might run into the timeout very quickly again and thus slowing down your authentication requests. 
Of course the downside is, that if you only have a network issue, which is **longer** than the timeout but **shorter** than the exhaust time, then you complete LDAP server will be not responsive - only for the *exhaust* time.

Again - I think it is difficult to find perfect values, and probably there are no values, which match for all users/all scenarios.

Would it be of help for you, if we make the *cylcles* and the *exhaust time* configurable?
I probably would do this in ``pi.cfg`` instead of adding these explicitly to the resolver - what do you think?
Agreed, it is difficult to find those goldilocks.

Often times, we are in a situation where there are brief networking issues due to maintenance and the pool will become exhausted if we try to preform an authentication. These issues are quickly resolved, but we will be in the complete the LDAP server will be not responsive scenario.

I think it will be more user customizable if we allow these fields to be configurable.
Make 

    #  pool
    SERVERPOOL_ROUNDS = 2
    # The number of seconds a non-responding server is removed from the server pool
    SERVERPOOL_SKIP = 30

configurable in pi.cfg. (Nope: I think resolvers can not read pi.cfg. So we need to add this as a resolver parameter) :-/
The parameters ``SERVERPOOL_ROUNDS`` and ``SERVERPOOL_SKIP`` can be added to the UI underneath the Server-URI.

![image](https://user-images.githubusercontent.com/1908620/31650174-6d64dc26-b316-11e7-995f-c9cb9cbc7c42.png)

I was thinking we could only display the values if the server-URI contains a comma. But even with only one server, the resolver creates a Serverpool. Thus, if an administrator would configure a list of servers with comma, change the serverpool values and then remove the list and run only one server (the values would be hidden) he would have configured values, which he does not see anymore.

So maybe it is the best, to always display these values? (But I do not like this, this seems to complicated for the admin in 90% of the cases)

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -754,7 +754,12 @@ def getResolverClassDescriptor(cls):
                                 'AUTHTYPE': 'string',
                                 'TLS_VERIFY': 'bool',
                                 'TLS_CA_FILE': 'string',
-                                'START_TLS': 'bool'}
+                                'START_TLS': 'bool',
+                                'CACHE_TIMEOUT': 'int',
+                                'SERVERPOOL_ROUNDS': 'int',
+                                'SERVERPOOL_SKIP': 'int',
+                                'OBJECT_CLASSES': 'string',
+                                'DN_TEMPLATE': 'string'}
         return {typ: descriptor}
 
     @classmethod


Test patch:
diff --git a/tests/test_lib_resolver.py b/tests/test_lib_resolver.py
--- a/tests/test_lib_resolver.py
+++ b/tests/test_lib_resolver.py
@@ -866,6 +866,15 @@ def test_07_get_serverpool(self):
         self.assertEqual(server_pool.servers[0].name, "ldap://themis:389")
         self.assertEqual(server_pool.servers[1].name, "ldaps://server2:636")
 
+        urilist = "ldap://themis, ldaps://server2"
+        server_pool = LDAPResolver.get_serverpool(urilist, timeout,
+                                                  rounds=5,
+                                                  exhaust=60)
+        self.assertEqual(len(server_pool), 2)
+        self.assertEqual(server_pool.active, 5)
+        self.assertEqual(server_pool.exhaust, 60)
+        self.assertEqual(server_pool.strategy, "ROUND_ROBIN")
+
     @ldap3mock.activate
     def test_08_trimresult(self):
         ldap3mock.setLDAPDirectory(LDAPDirectory)