Repository: privacyidea/privacyidea

Issue:
privacyidea-token-janitor delete action does not delete all matching tokens
I just stumbled upon this problem, which can be reproduced as follows:
* Have a lot of tokens
* Run ``privacyidea-token-janitor find --action delete --chunksize 10``, which should delete all tokens
* Token janitor deletes a lot of tokens and exits successfully after a while
* Run ``privacyidea-token-janitor find --action delete --chunksize 10``
* The token janitor still finds some tokens to delete!

This is probably related to the chunksize feature we added in #1224 and #1264 by using the pagination feature of ``get_tokens``:
* Assume we have 100 tokens numbered from 0 to 99
* The token janitor fetches the first chunk of 10 tokens to delete with ``get_tokens(..., page=1)``, i.e. tokens 0 to 9
* It deletes these tokens
* The token janitor now fetches the next chunk of 10 tokens to delete with ``get_tokens(..., page=2)``.  As tokens 0 to 9 have been deleted, page 1 now contains tokens 10 to 19, so page 2 is now tokens 20 to 29!
* Because of this, tokens 10 to 19 are never deleted.

The underlying problem is that we modify the database while doing a paginated search, and this is not only the case for token deletion, but also for things like unassign, disable :-/
privacyidea-token-janitor delete action does not delete all matching tokens
I just stumbled upon this problem, which can be reproduced as follows:
* Have a lot of tokens
* Run ``privacyidea-token-janitor find --action delete --chunksize 10``, which should delete all tokens
* Token janitor deletes a lot of tokens and exits successfully after a while
* Run ``privacyidea-token-janitor find --action delete --chunksize 10``
* The token janitor still finds some tokens to delete!

This is probably related to the chunksize feature we added in #1224 and #1264 by using the pagination feature of ``get_tokens``:
* Assume we have 100 tokens numbered from 0 to 99
* The token janitor fetches the first chunk of 10 tokens to delete with ``get_tokens(..., page=1)``, i.e. tokens 0 to 9
* It deletes these tokens
* The token janitor now fetches the next chunk of 10 tokens to delete with ``get_tokens(..., page=2)``.  As tokens 0 to 9 have been deleted, page 1 now contains tokens 10 to 19, so page 2 is now tokens 20 to 29!
* Because of this, tokens 10 to 19 are never deleted.

The underlying problem is that we modify the database while doing a paginated search, and this is not only the case for token deletion, but also for things like unassign, disable :-/


Discussion:
Looking about this some more, it turns out to be quite nasty :-/

The bug only happens if the number of tokens is greater than the chunksize (otherwise, pagination has no effect).

``privacyidea-token-janitor`` misses tokens for invocations in which (1) the action influences the tokens found by the filter criterion, and (2) the filter criterion is translated into a SQL query:
* ``--action delete`` is a problem for any filter criterion: It deletes tokens, so they don't match the filter criterion anymore, which causes the above problem
* ``--active true --action disable`` disables active tokens, so they don't match the filter criterion anymore, same problem 

But the bug doesn't occur in the following cases:
* ``--active false --action unassign`` unassigns inactive tokens. But as they are still inactive, they still match the filter criterion, so the bug does not occur!
* For ``--orphaned 1 --action unassign``, the check for orphaned tokens is not part of the SQL query.  So from the SQL perspective, unassigned tokens still match the query, so the bug does not occur!

For the cases in which the action interferes with the filter criterion, we would need to use a solution like in ``delete_chunked`` for audit rotation:
https://github.com/privacyidea/privacyidea/blob/4da61fa77746f45d10f079c66db99deb9223510d/privacyidea/lib/sqlutils.py#L81
Here, we do not use the pagination feature, but just delete matching tokens until no tokens are found anymore.

We could fix the bug by checking ahead-of-time if the filter criterion and action will interfere, and using a special implementation in these cases. But checking for such conditions seems pretty hard and error-prone to me.

We originally introduced the feature in #1224 to speed up the search for orphaned tokens. So I have the following proposal: We only use paginated search for actions which do not change the database *at all* (i.e. CSV export, ``listuser``, ``export``), and refer to good-old non-paginated search for all other actions. This will slow down these actions, and maybe cause problems for big queries. But *if* the queries go through, the resulting database state will be correct, which I think is much better than the current behavior! :-)
Or: we could save the total number of matching tokens in the *first* invocation of ``get_tokens``. For all further invocations, we check if the new total number of matching tokens is smaller, and if that is the case, we restart the query at page 1.

But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.
> We could fix the bug by checking ahead-of-time if the filter criterion and action will interfere, and using a special implementation in these cases. But checking for such conditions seems pretty hard and error-prone to me.

could You explain this a little more? We check if the `action` is `delete` and use the non-paginated query. The filter should be the same, or not?

> But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.

is the database locked during the pagination requests? Otherwise this problem would arise there, too.

> could You explain this a little more? We check if the `action` is `delete` and use the non-paginated query. The filter should be the same, or not?

I'll try :) You're right, for ``delete`` the case is easy because we just use the ``DeleteLimit`` query from lib/sqlutils.py regardless of the filter criterion. But it's slightly harder for, e.g., ``disable``: In that case, it depends on the filter criterion whether we can use pagination or not:

* For ``--active true``, we must not use pagination, because active tokens will be disabled and won't match the SQL query anymore
* For ``--orphaned 1``, we can use pagination, because the orphaned token check is not part of the SQL query

We could think of falling back to non-paginated queries for cases in which we aren't sure, but this could be complicated too :)

> 
> > But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.
> 
> is the database locked during the pagination requests? Otherwise this problem would arise there, too.

Good point! No, the database is not locked, so all sorts of funny things could happen anyway.
Looking about this some more, it turns out to be quite nasty :-/

The bug only happens if the number of tokens is greater than the chunksize (otherwise, pagination has no effect).

``privacyidea-token-janitor`` misses tokens for invocations in which (1) the action influences the tokens found by the filter criterion, and (2) the filter criterion is translated into a SQL query:
* ``--action delete`` is a problem for any filter criterion: It deletes tokens, so they don't match the filter criterion anymore, which causes the above problem
* ``--active true --action disable`` disables active tokens, so they don't match the filter criterion anymore, same problem 

But the bug doesn't occur in the following cases:
* ``--active false --action unassign`` unassigns inactive tokens. But as they are still inactive, they still match the filter criterion, so the bug does not occur!
* For ``--orphaned 1 --action unassign``, the check for orphaned tokens is not part of the SQL query.  So from the SQL perspective, unassigned tokens still match the query, so the bug does not occur!

For the cases in which the action interferes with the filter criterion, we would need to use a solution like in ``delete_chunked`` for audit rotation:
https://github.com/privacyidea/privacyidea/blob/4da61fa77746f45d10f079c66db99deb9223510d/privacyidea/lib/sqlutils.py#L81
Here, we do not use the pagination feature, but just delete matching tokens until no tokens are found anymore.

We could fix the bug by checking ahead-of-time if the filter criterion and action will interfere, and using a special implementation in these cases. But checking for such conditions seems pretty hard and error-prone to me.

We originally introduced the feature in #1224 to speed up the search for orphaned tokens. So I have the following proposal: We only use paginated search for actions which do not change the database *at all* (i.e. CSV export, ``listuser``, ``export``), and refer to good-old non-paginated search for all other actions. This will slow down these actions, and maybe cause problems for big queries. But *if* the queries go through, the resulting database state will be correct, which I think is much better than the current behavior! :-)
Or: we could save the total number of matching tokens in the *first* invocation of ``get_tokens``. For all further invocations, we check if the new total number of matching tokens is smaller, and if that is the case, we restart the query at page 1.

But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.
> We could fix the bug by checking ahead-of-time if the filter criterion and action will interfere, and using a special implementation in these cases. But checking for such conditions seems pretty hard and error-prone to me.

could You explain this a little more? We check if the `action` is `delete` and use the non-paginated query. The filter should be the same, or not?

> But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.

is the database locked during the pagination requests? Otherwise this problem would arise there, too.

> could You explain this a little more? We check if the `action` is `delete` and use the non-paginated query. The filter should be the same, or not?

I'll try :) You're right, for ``delete`` the case is easy because we just use the ``DeleteLimit`` query from lib/sqlutils.py regardless of the filter criterion. But it's slightly harder for, e.g., ``disable``: In that case, it depends on the filter criterion whether we can use pagination or not:

* For ``--active true``, we must not use pagination, because active tokens will be disabled and won't match the SQL query anymore
* For ``--orphaned 1``, we can use pagination, because the orphaned token check is not part of the SQL query

We could think of falling back to non-paginated queries for cases in which we aren't sure, but this could be complicated too :)

> 
> > But this could have unexpected results if some user just deletes or disables some token via the WebUI in parallel.
> 
> is the database locked during the pagination requests? Otherwise this problem would arise there, too.

Good point! No, the database is not locked, so all sorts of funny things could happen anyway.

Resolving patch:
diff --git a/privacyidea/lib/token.py b/privacyidea/lib/token.py
--- a/privacyidea/lib/token.py
+++ b/privacyidea/lib/token.py
@@ -273,12 +273,52 @@ def _create_token_query(tokentype=None, realm=None, assigned=None, user=None,
     return sql_query
 
 
+def get_tokens_paginated_generator(tokentype=None, realm=None, assigned=None, user=None,
+                                   serial_wildcard=None, active=None, resolver=None, rollout_state=None,
+                                   revoked=None, locked=None, tokeninfo=None, maxfail=None, psize=1000):
+    """
+    Fetch chunks of ``psize`` tokens that match the filter criteria from the database and generate
+    lists of token objects.
+    See ``get_tokens`` for information on the arguments.
+
+    Note that individual lists may contain less than ``psize`` elements if
+    a token entry has an invalid type.
+
+    :param psize: Maximum size of chunks that are fetched from the database
+    :return: This is a generator that generates non-empty lists of token objects.
+    """
+    main_sql_query = _create_token_query(tokentype=tokentype, realm=realm,
+                                         assigned=assigned, user=user,
+                                         serial_wildcard=serial_wildcard,
+                                         active=active, resolver=resolver,
+                                         rollout_state=rollout_state,
+                                         revoked=revoked, locked=locked,
+                                         tokeninfo=tokeninfo, maxfail=maxfail).order_by(Token.id)
+    # Fetch the first ``psize`` tokens
+    sql_query = main_sql_query.limit(psize)
+    while True:
+        entries = sql_query.all()
+        if entries:
+            token_objects = []
+            for token in entries:
+                token_obj = create_tokenclass_object(token)
+                if isinstance(token_obj, TokenClass):
+                    token_objects.append(token_obj)
+            yield token_objects
+            if len(entries) < psize:
+                break
+            # Fetch the next ``psize`` tokens, starting with the ID *after* the ID of the last returned token.
+            # ``token`` is defined because we have ensured that ``entries`` has at least one entry.
+            sql_query = main_sql_query.filter(Token.id > token.id).limit(psize)
+        else:
+            break
+
 @log_with(log)
 #@cache.memoize(10)
 def get_tokens(tokentype=None, realm=None, assigned=None, user=None,
                serial=None, serial_wildcard=None, active=None, resolver=None, rollout_state=None,
                count=False, revoked=None, locked=None, tokeninfo=None,
-               maxfail=None, psize=None, page=1):
+               maxfail=None):
     """
     (was getTokensOfType)
     This function returns a list of token objects of a
@@ -324,14 +364,7 @@ def get_tokens(tokentype=None, realm=None, assigned=None, user=None,
     :type tokeninfo: dict
     :param maxfail: If only tokens should be returned, which failcounter
         reached maxfail
-    :param psize: The number of returned tokens can be restricted by using the parameter psize.
-        Pagination then is used.
-    :type psize: int
-    :param page: If pagination is used, this is the page to get
-    :type page: int
-
     :return: A list of tokenclasses (lib.tokenclass).
-        In case of pagination a tuple of count, prev, next, token_list
     :rtype: list
     """
     token_list = []
@@ -353,25 +386,6 @@ def get_tokens(tokentype=None, realm=None, assigned=None, user=None,
     # Decide, what we are supposed to return
     if count is True:
         ret = sql_query.count()
-    elif psize:
-        # We return a paginated list of token objects.
-        pagination = sql_query.paginate(page, per_page=psize,
-                                        error_out=False)
-        count = pagination.total
-        prev = None
-        if pagination.has_prev:
-            prev = page - 1
-        next = None
-        if pagination.has_next:
-            next = page + 1
-        for token in pagination.items:
-            tokenobject = create_tokenclass_object(token)
-            if isinstance(tokenobject, TokenClass):
-                # A database token, that has a non existing type, will
-                # return None, and not a TokenClass. We do not want to
-                # add None to our list
-                token_list.append(tokenobject)
-        ret = count, prev, next, token_list
     else:
         # Return a simple, flat list of tokenobjects
         for token in sql_query.all():


Test patch:
diff --git a/tests/test_lib_token.py b/tests/test_lib_token.py
--- a/tests/test_lib_token.py
+++ b/tests/test_lib_token.py
@@ -53,7 +53,8 @@
                                    get_tokens_paginate,
                                    set_validity_period_end,
                                    set_validity_period_start, remove_token, delete_tokeninfo,
-                                   import_token, get_one_token, get_tokens_from_serial_or_user)
+                                   import_token, get_one_token, get_tokens_from_serial_or_user,
+                                   get_tokens_paginated_generator)
 
 from privacyidea.lib.error import (TokenAdminError, ParameterError,
                                    privacyIDEAError, ResourceNotFoundError)
@@ -185,19 +186,6 @@ def test_02_get_tokens(self):
         tokenobject_list = get_tokens(serial_wildcard="*")
         self.assertEqual(len(tokenobject_list), 4)
 
-        # test pagination
-        count, prev, next, tokenobject_list = get_tokens(serial_wildcard="*", psize=2, page=1)
-        self.assertEqual(count, 4)
-        self.assertEqual(prev, None)
-        self.assertEqual(next, 2)
-        self.assertEqual(len(tokenobject_list), 2)
-
-        count, prev, next, tokenobject_list = get_tokens(serial_wildcard="*", psize=2, page=2)
-        self.assertEqual(count, 4)
-        self.assertEqual(prev, 1)
-        self.assertEqual(next, None)
-        self.assertEqual(len(tokenobject_list), 2)
-
     def test_03_get_token_type(self):
         ttype = get_token_type("hotptoken")
         self.assertTrue(ttype == "hotp", ttype)
@@ -1399,6 +1387,54 @@ def test_54_helper_functions(self):
             get_tokens_from_serial_or_user(serial="S1", user=shadow)
         unassign_token(serial=None, user=user)
 
+    def test_55_get_tokens_paginated_generator(self):
+        def flatten_tokens(l):
+            return [token.token.id for l2 in l for token in l2]
+
+        # serial72 token has invalid type. Check behavior and remove it.
+        self.assertEquals(list(get_tokens_paginated_generator(serial_wildcard="serial*")), [[]])
+        Token.query.filter_by(serial="serial72").delete()
+
+        all_matching_tokens = get_tokens(serial_wildcard="S*")
+        lists1 = list(get_tokens_paginated_generator(serial_wildcard="S*"))
+        self.assertEquals(len(lists1), 1)
+        self.assertEquals(len(lists1[0]), 6)
+        lists2 = list(get_tokens_paginated_generator(serial_wildcard="S*", psize=2))
+        self.assertEquals(len(lists2), 3)
+        self.assertEquals(len(lists2[0]), 2)
+        self.assertEquals(len(lists2[1]), 2)
+        self.assertEquals(len(lists2[2]), 2)
+        lists3 = list(get_tokens_paginated_generator(serial_wildcard="S*", psize=3))
+        self.assertEquals(len(lists3), 2)
+        self.assertEquals(len(lists3[0]), 3)
+        self.assertEquals(len(lists3[1]), 3)
+        lists4 = list(get_tokens_paginated_generator(serial_wildcard="S*", psize=4))
+        self.assertEquals(len(lists4), 2)
+        self.assertEquals(len(lists4[0]), 4)
+        self.assertEquals(len(lists4[1]), 2)
+        lists5 = list(get_tokens_paginated_generator(serial_wildcard="S*", psize=6))
+        self.assertEquals(len(lists5), 1)
+        self.assertEquals(len(lists5[0]), 6)
+        self.assertEquals(set([t.token.id for t in all_matching_tokens]), set(flatten_tokens(lists1)))
+        self.assertEquals(flatten_tokens(lists1), flatten_tokens(lists2))
+        self.assertEquals(flatten_tokens(lists2), flatten_tokens(lists3))
+        self.assertEquals(flatten_tokens(lists3), flatten_tokens(lists4))
+        self.assertEquals(flatten_tokens(lists4), flatten_tokens(lists5))
+
+        lists6 = list(get_tokens_paginated_generator(serial_wildcard="*DOESNOTEXIST*"))
+        self.assertEquals(lists6, [])
+
+    def test_56_get_tokens_paginated_generator_removal(self):
+        all_serials = set(t.token.serial for t in get_tokens(serial_wildcard="S*"))
+        # Test proper behavior if a matching token is deleted while paginating
+        gen = get_tokens_paginated_generator(serial_wildcard="S*", psize=3)
+        list1 = next(gen)
+        remove_token(list1[0].token.serial)
+        list2 = next(gen)
+        # Check that we did not miss any tokens
+        self.assertEquals(set(t.token.serial for t in list1 + list2), all_serials)
+
+
 class TokenFailCounterTestCase(MyTestCase):
     """
     Test the lib.token on an interface level