Repository: privacyidea/privacyidea

Issue:
Need to untangle  extended policy conditions
The extended condition checking in policies bears the problem that in some cases to be *checked information* is missing because it is not available and not sensible in this situtation (case A) 

and in other cases it is missing even if it should be there (case B)

(Maybe there are furthes cases)

There are different possible reactions in regards to a missing *checked information*

* the condition can be ignored, 
* a *checked information* could be treated as 0 or an empty string, 
* a *checked information* could cause the policy check to fail. (More on this later)

# Example Situations

## POST /auth

depends whether it is a user (then a user_obj exists) or if it is an admin (then no user_obj exists)

## GET /realm 

is sometimes used in the API with admins and can lead to unexpected beahviour

# Internal matching methods

Till now I identified ``PolicyClass.match_policies`` as a source. This is sometimes used in the UI by ``ui_get_rights`` and by ``ui_get_enroll_tokentypes`` to clarify, which rights a user should have in the UI. Problems arise from an admin logging in, when the user_obj is not available. But problems can also arise from extended conditions on tokens (when the token attributes is not available) or tokeninfo (when the checked token info is not available).
In thises case we often only want to know, if the admin is allowed to do something "at all" (without more specific checks), so that we can adapt the UI accordingly.

For same cases sometimes ``Match.generic`` is a source. This happens when we want to display the Realm Dropdown in the UI login screen (function ``single_page_application()`` or in the get API of the realms (``get_realms_api()``).
In this case we only want to know, policies for which realms exist "at all", e.g. to show an admin the corresponding realms.

# Possible solutions

The code location where a policy method is called "knows" what to do. So the call to the policy method should decide, whether the extended conditions need to be checked in this situation or not. We could do this

* by dedicated policy method calls (the method name indicates, that extended conditions are not checked)
* or by paramters passing to the lower levels.

## Using dedicated method names

A method name like ``match`` imho could indicate, that we currently do a matching on this very situation.
A method name containing ``list``could indicate, that we do not care about extended conditions.

## Using Parameters

From the places where ``PolicyClass.match_policies`` or ``Match.generic`` is called we can pass down the information, if we want to ignore extended conditions or if we want to check for extended conditions and then raise an Exception.
The calling location *knows* the best, if a certain *checked information* needs to be available or not.

Do we skip the check completely? Or do we only change the output (not raising an exception but allow the policy silentl)?

Here is an example of skipping completely:

**pass the information to the lower level**:

~~~~python
policies = Match.generic(g, ...,
                             active=True, check_extended_conditions=False).policies()
~~~~
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/webui/login.py#L106 to determine the realm dropdown
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/api/realm.py#L199 to determine the visible realms

and
~~~~python
typepols = self.match_policies(scope=role, ...
                                               check_extended_conditions=False)
~~~~
at
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/policy.py#L1124 in ui_get_enroll_tokentypes
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/policy.py#L1048 in ui_get_rights

**skip the checks completely**

~~~~python
if check_extended_conditions:
            reduced_policies = self.filter_policies_by_conditions(reduced_policies, user_object, request_headers,
                                                                  serial)
~~~~
at https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/policy.py#L730

**Note**: This can be different for different condition types. E.g. during login we know, that a user_obj would be available, if a normal user logs in (do not ignore). But when the user logs in, a tokeninfo would *not* be available.
While if an admin logs in, neither a user_obj nor a tokeninfo is available.

*Do we need to differentiate between different extended condition types?*


**possible actions**

Possible actions on the extended conditions could be

* ``do_not_check_at_all``, ignore the extended condition, without checking it in the first place
* ``check_and...``
   - ``check_and_raise_exception_on_missing`` (default, current behaviour)
   - ``check_and_ignore_condition_if_missing`` (this may come in handy when logging in as admin or as user)
   - ``check_and_compare_as_empty_if_missing`` (this sounds like something, that an administrator would want to configure and would not be used in the code)

# Important logging output

(Already done)

In the methods ``_policy_matches_***_condition``  in the ``PolicyClass`` we need to produce a more detailed output, so that we know in the future, if there is anything to adapt.

In the last else-statement, that logs an error, when the condition can not be checked we should log a stack trace:

~~~~python
 else:
            log.warning(u"Policy {!r} has condition on {!s} {!r}, but the according object"
                        u" is not available.".format(policy['name'], type, key
            ))
            log.error("Possible programming error: {0!s}".format(''.join(traceback.format_stack())))  # <----- HERE
            # If the policy specifies a userinfo or tokeninfo condition, but no object is available,
            # the policy is misconfigured. We have to raise a PolicyError to ensure that
            # the privacyIDEA server does not silently misbehave.
            raise PolicyError(
                u"Policy {!r} has condition on {!s}, but an according object is not available".format(
                    policy['name'], type
                ))
~~~~

## Notes

Also see #2514.


Discussion:

Resolving patch:
diff --git a/privacyidea/api/realm.py b/privacyidea/api/realm.py
--- a/privacyidea/api/realm.py
+++ b/privacyidea/api/realm.py
@@ -58,6 +58,7 @@
 from ..lib.utils import reduce_realms
 from flask import g
 from privacyidea.lib.auth import ROLE
+from privacyidea.lib.policy import CONDITION_CHECK
 from flask_babel import gettext as _
 import logging
 
@@ -199,7 +200,8 @@ def get_realms_api():
     policies = Match.generic(g, scope=luser.get("role", ROLE.ADMIN),
                              adminrealm=luser.get("realm"),
                              adminuser=luser.get("username"),
-                             active=True).policies()
+                             active=True,
+                             extended_condition_check=CONDITION_CHECK.DO_NOT_CHECK_AT_ALL).policies()
     realms = reduce_realms(all_realms, policies)
 
     return send_result(realms)
diff --git a/privacyidea/lib/policy.py b/privacyidea/lib/policy.py
--- a/privacyidea/lib/policy.py
+++ b/privacyidea/lib/policy.py
@@ -452,6 +452,13 @@ class CONDITION_SECTION(object):
     HTTP_REQUEST_HEADER = "HTTP Request header"
 
 
+class CONDITION_CHECK(object):
+    __doc__ = """The available check methods for extended conditions"""
+    DO_NOT_CHECK_AT_ALL = 1
+    ONLY_CHECK_USERINFO = [CONDITION_SECTION.USERINFO]
+    CHECK_AND_RAISE_EXCEPTION_ON_MISSING = None
+
+
 class PolicyClass(object):
     """
     A policy object can be used to query the current set of policies.
@@ -670,7 +677,8 @@ def list_policies(self, name=None, scope=None, realm=None, active=None,
     def match_policies(self, name=None, scope=None, realm=None, active=None,
                        resolver=None, user=None, user_object=None, pinode=None,
                        client=None, action=None, adminrealm=None, adminuser=None, time=None,
-                       sort_by_priority=True, audit_data=None, request_headers=None, serial=None):
+                       sort_by_priority=True, audit_data=None, request_headers=None, serial=None,
+                       extended_condition_check=None):
         """
         Return all policies matching the given context.
         Optionally, write the matching policies to the audit log.
@@ -739,9 +747,10 @@ def match_policies(self, name=None, scope=None, realm=None, active=None,
         log.debug("Policies after matching time: {0!s}".format([p.get("name") for p in reduced_policies]))
 
         # filter policies by the policy conditions
-        reduced_policies = self.filter_policies_by_conditions(reduced_policies, user_object, request_headers,
-                                                              serial)
-        log.debug("Policies after matching conditions".format([p.get("name") for p in reduced_policies]))
+        if extended_condition_check != CONDITION_CHECK.DO_NOT_CHECK_AT_ALL:
+            reduced_policies = self.filter_policies_by_conditions(reduced_policies, user_object, request_headers,
+                                                                  serial, extended_condition_check)
+            log.debug("Policies after matching conditions".format([p.get("name") for p in reduced_policies]))
 
         if audit_data is not None:
             for p in reduced_policies:
@@ -749,7 +758,8 @@ def match_policies(self, name=None, scope=None, realm=None, active=None,
 
         return reduced_policies
 
-    def filter_policies_by_conditions(self, policies, user_object=None, request_headers=None, serial=None):
+    def filter_policies_by_conditions(self, policies, user_object=None, request_headers=None, serial=None,
+                                      extended_condition_check=None):
         """
         Given a list of policy dictionaries and a current user object (if any),
         return a list of all policies whose conditions match the given user object.
@@ -758,6 +768,7 @@ def filter_policies_by_conditions(self, policies, user_object=None, request_head
         :param user_object: a User object, or None if there is no current user
         :param request_headers: The HTTP headers
         :type request_headers: Request object
+        :param extended_condition_check: A list of sections to check or None.
         :return: generates a list of policy dictionaries
         """
         reduced_policies = []
@@ -766,35 +777,39 @@ def filter_policies_by_conditions(self, policies, user_object=None, request_head
         for policy in policies:
             include_policy = True
             for section, key, comparator, value, active in policy['conditions']:
-                if active:
-                    if section == CONDITION_SECTION.USERINFO:
-                        if not self._policy_matches_info_condition(policy, key, comparator, value,
-                                                                   CONDITION_SECTION.USERINFO,
-                                                                   user_object=user_object):
-                            include_policy = False
-                            break
-                    elif section == CONDITION_SECTION.TOKENINFO:
-                        dbtoken = dbtoken or Token.query.filter(Token.serial == serial).first() if serial else None
-                        if not self._policy_matches_info_condition(policy, key, comparator, value,
-                                                                   CONDITION_SECTION.TOKENINFO,
-                                                                   dbtoken=dbtoken):
-                            include_policy = False
-                            break
-                    elif section == CONDITION_SECTION.TOKEN:
-                        dbtoken = dbtoken or Token.query.filter(Token.serial == serial).first() if serial else None
-                        if not self._policy_matches_token_condition(policy, key, comparator, value, dbtoken):
-                            include_policy = False
-                            break
-                    elif section == CONDITION_SECTION.HTTP_REQUEST_HEADER:
-                        if not self._policy_matches_request_header_condition(policy, key, comparator, value,
-                                                                             request_headers):
-                            include_policy = False
-                            break
-                    else:
-                        log.warning(u"Policy {!r} has condition with unknown section: {!r}".format(
-                            policy['name'], section
-                        ))
-                        raise PolicyError(u"Policy {!r} has condition with unknown section".format(policy['name']))
+                if (extended_condition_check is CONDITION_CHECK.CHECK_AND_RAISE_EXCEPTION_ON_MISSING
+                        or section in extended_condition_check):
+                    # We check conditions, either if we are supposed to check everything or if
+                    # the section is contained in the extended condition check
+                    if active:
+                        if section == CONDITION_SECTION.USERINFO:
+                            if not self._policy_matches_info_condition(policy, key, comparator, value,
+                                                                       CONDITION_SECTION.USERINFO,
+                                                                       user_object=user_object):
+                                include_policy = False
+                                break
+                        elif section == CONDITION_SECTION.TOKENINFO:
+                            dbtoken = dbtoken or Token.query.filter(Token.serial == serial).first() if serial else None
+                            if not self._policy_matches_info_condition(policy, key, comparator, value,
+                                                                       CONDITION_SECTION.TOKENINFO,
+                                                                       dbtoken=dbtoken):
+                                include_policy = False
+                                break
+                        elif section == CONDITION_SECTION.TOKEN:
+                            dbtoken = dbtoken or Token.query.filter(Token.serial == serial).first() if serial else None
+                            if not self._policy_matches_token_condition(policy, key, comparator, value, dbtoken):
+                                include_policy = False
+                                break
+                        elif section == CONDITION_SECTION.HTTP_REQUEST_HEADER:
+                            if not self._policy_matches_request_header_condition(policy, key, comparator, value,
+                                                                                 request_headers):
+                                include_policy = False
+                                break
+                        else:
+                            log.warning(u"Policy {!r} has condition with unknown section: {!r}".format(
+                                policy['name'], section
+                            ))
+                            raise PolicyError(u"Policy {!r} has condition with unknown section".format(policy['name']))
             if include_policy:
                 reduced_policies.append(policy)
         return reduced_policies
@@ -1094,11 +1109,19 @@ def ui_get_rights(self, scope, realm, username, client=None):
             admin_user = username
             admin_realm = realm
             user_object = None
+            # During login of the admin there is no token, no tokeninfo and no user info available.
+            # Also, the http header is only passed down to the policy Match-class, but not in the get_rights method.
+            # Thus we can not check any extended conditions for admins at this point.
+            extended_condition_check = CONDITION_CHECK.DO_NOT_CHECK_AT_ALL
         elif scope == SCOPE.USER:
-            # If the logged-in user is a user, we pass a user object to allow matching for userinfo attributes
             admin_user = None
             admin_realm = None
+            # If the logged-in user is a user, we pass a user object to allow matching for userinfo attributes
             user_object = User(username, realm)
+            # During login of the admin there is no token and no tokeninfo available.
+            # Also, the http header is only passed down to the policy Match-class, but not in the get_rights method.
+            # Thus we can only check the extended condition "userinfo" for users at this point.
+            extended_condition_check = CONDITION_CHECK.ONLY_CHECK_USERINFO
         else:
             raise PolicyError(u"Unknown scope: {}".format(scope))
         pols = self.match_policies(scope=scope,
@@ -1106,7 +1129,8 @@ def ui_get_rights(self, scope, realm, username, client=None):
                                    adminrealm=admin_realm,
                                    adminuser=admin_user,
                                    active=True,
-                                   client=client)
+                                   client=client,
+                                   extended_condition_check=extended_condition_check)
         for pol in pols:
             for action, action_value in pol.get("action").items():
                 if action_value:
@@ -1171,6 +1195,10 @@ def ui_get_enroll_tokentypes(self, client, logged_in_user):
                 enroll_types[tokenclass.get_class_type()] = \
                     tokenclass.get_class_info("description")
 
+        if role == SCOPE.ADMIN:
+            extended_condition_check = CONDITION_CHECK.DO_NOT_CHECK_AT_ALL
+        else:
+            extended_condition_check = CONDITION_CHECK.ONLY_CHECK_USERINFO
         if pols:
             # admin policies or user policies are set, so we need to
             # test, which tokens are allowed to be enrolled for this user
@@ -1184,7 +1212,8 @@ def ui_get_enroll_tokentypes(self, client, logged_in_user):
                                                active=True,
                                                action="enroll"+tokentype.upper(),
                                                adminrealm=adminrealm,
-                                               adminuser=adminuser)
+                                               adminuser=adminuser,
+                                               extended_condition_check=extended_condition_check)
                 if typepols:
                     # If there is no policy allowing the enrollment of this
                     # tokentype, it is deleted.
@@ -2754,7 +2783,7 @@ def admin_or_user(cls, g, action, user_obj):
     @classmethod
     def generic(cls, g, scope=None, realm=None, resolver=None, user=None, user_object=None,
                 client=None, action=None, adminrealm=None, adminuser=None, time=None,
-                active=True, sort_by_priority=True, serial=None):
+                active=True, sort_by_priority=True, serial=None, extended_condition_check=None):
         """
         Low-level legacy policy matching interface: Search for active policies and return
         them sorted by priority. All parameters that should be used for matching have to
@@ -2772,7 +2801,7 @@ def generic(cls, g, scope=None, realm=None, resolver=None, user=None, user_objec
                    resolver=resolver, user=user, user_object=user_object,
                    client=client, action=action, adminrealm=adminrealm,
                    adminuser=adminuser, time=time, serial=serial,
-                   sort_by_priority=sort_by_priority)
+                   sort_by_priority=sort_by_priority, extended_condition_check=extended_condition_check)
 
 
 def get_allowed_custom_attributes(g, user_obj):


Test patch:
diff --git a/tests/test_api_roles.py b/tests/test_api_roles.py
--- a/tests/test_api_roles.py
+++ b/tests/test_api_roles.py
@@ -1389,6 +1389,34 @@ def test_02_enroll_rights(self):
             # only HOTP tokens
             self.assertEqual(set(result['value'].keys()), {'hotp'})
 
+        # We set a new policy condition for users (bob), to be allowed to enroll TOTP
+        # tokens only in special cases. This condition will not match for certain tokens.
+        with self.app.test_request_context('/policy/totp_enroll',
+                                           json={'action': "enrollTOTP",
+                                                 'scope': SCOPE.USER,
+                                                 'realm': '',
+                                                 'active': True,
+                                                 'conditions': [
+                                                     ["userinfo", "username", "equals", "bob", True],
+                                                     ["token", "count", "equals", "1", True],
+                                                 ]
+                                                 },
+                                           method='POST',
+                                           headers={'Authorization': self.at}):
+            res = self.app.full_dispatch_request()
+            self.assertEqual(res.status_code, 200)
+
+        # check the rights of bob. The TOTP enroll right will basically be there
+        with self.app.test_request_context('/auth/rights',
+                                           method='GET',
+                                           headers={'Authorization': bob_token}):
+            res = self.app.full_dispatch_request()
+            self.assertEqual(res.status_code, 200)
+            result = res.json.get("result")
+            # HOTP and TOTP tokens
+            self.assertIn('hotp', set(result['value'].keys()))
+            self.assertIn('totp', set(result['value'].keys()))
+
         # check the rights of manager
         with self.app.test_request_context('/auth/rights',
                                            method='GET',
@@ -1412,11 +1440,11 @@ def _try_enroll(auth_token, token_type):
         status_code, result = _try_enroll(bob_token, 'hotp')
         self.assertEqual(status_code, 200)
         self.assertEqual(result, {"status": True, "value": True})
-        # ... but not TOTP ...
+        # ... but not TOTP, since the extended condition does not match
         status_code, result = _try_enroll(bob_token, 'totp')
         self.assertEqual(status_code, 403)
         self.assertFalse(result['status'])
-        self.assertIn("not allowed to enroll this token type", result['error']['message'])
+        self.assertIn("has conditions on tokens, but a token object is not available", result['error']['message'])
         # ... and certainly not SPASS
         status_code, result = _try_enroll(bob_token, 'spass')
         self.assertEqual(status_code, 403)