Repository: privacyidea/privacyidea

Issue:
Check: Rotate_audit log could get into a dead lock with ~800.000 entries
When rotating with highwatermark and lowwatermark the rotation takes a long time and could run into a dead lock.


Discussion:
Note: This also seems to happen if we use ``--age <...>``.

In case one has to delete a lot of rows, it generally seems to be a good idea to delete them in chunks (see [here](http://msutic.blogspot.com/2017/09/delete-large-amounts-of-data-on-galera.html), [here](https://stackoverflow.com/a/1318975)), because this will improve performance. I think this would also decrease the probability of deadlocks.
As it turns out, it isn't that easy to delete audit rows in chunks in a way that is supported by all our database backends.

MySQL has a ``DELETE FROM pidea_audit WHERE timestamp < X LIMIT 1000`` syntax, which we could simply execute in a loop until no entries are deleted anymore. However, this isn't supported by SQLAlchemy.

For SQLite, we could use ``DELETE FROM pidea_audit WHERE id IN (SELECT id FROM pidea_audit WHERE timestamp < X LIMIT 1000)``, but this isn't supported by MySQL (at least it isn't supported by my 5.7.23 server).
Did you reproduce that?
Would we not dead lock, if we would delete each audit row seperately? (Which of course would take a vast amount of time - just to narrow down things)
I haven't reproduced any deadlocks yet, no :-/ I suspect they can only really be reproduced with a galera setup.

I'm not sure, but I think yes, having a lot of ``DELETE FROM pidea_audit WHERE id = X`` queries should reduce the risk of deadlocks. But it would probably be extremely slow if a lot of rows are deleted.
I reproduced the deadlock with two privacyIDEA nodes and a 3-node galera cluster setup. Steps:

* Generate 800000 (fake) entries in the audit log
* Use ``siege -c4`` to generate a lot of authentication requests on both privacyIDEA nodes
* Run ``pi-manage rotate_audit --age 10`` on one node

Result:
```
Cleaning up with age: 10. mysql://galera:galera@192.168.0.241/pi
Deleting entries older than 2018-09-22 09:53:11.086138
Traceback (most recent call last):
  File "/opt/venv/bin/pi-manage", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/opt/privacyidea/pi-manage", line 1164, in <module>
    manager.run()
  File "/opt/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/opt/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/opt/venv/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/opt/privacyidea/pi-manage", line 676, in rotate_audit
    session.commit()
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 471, in commit
    t[1].commit()
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1643, in commit
    self._do_commit()
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1674, in _do_commit
    self.connection._commit_impl()
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/opt/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 1792, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/e3q8)
```

If we don't generate any load, the deadlock does not happen.

However, it seems like this isn't really reproducible in a reliable way :-/ Sometimes, the audit rotation works, but blocks the concurrent authentications during the rotation!

Resolving patch:
diff --git a/privacyidea/lib/sqlutils.py b/privacyidea/lib/sqlutils.py
new file mode 100644
--- /dev/null
+++ b/privacyidea/lib/sqlutils.py
@@ -0,0 +1,118 @@
+# -*- coding: utf-8 -*-
+#  2018-10-02 Friedrich Weber <friedrich.weber@netknights.it>
+#             Add chunked deletions
+#
+# This code is free software; you can redistribute it and/or
+# modify it under the terms of the GNU AFFERO GENERAL PUBLIC LICENSE
+# License as published by the Free Software Foundation; either
+# version 3 of the License, or any later version.
+#
+# This code is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU AFFERO GENERAL PUBLIC LICENSE for more details.
+#
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+#
+
+from sqlalchemy import select, text
+from sqlalchemy.ext.compiler import compiles
+from sqlalchemy.sql import ClauseElement, Delete
+
+
+class DeleteLimit(Delete, ClauseElement):
+    """
+    A modified DELETE clause element that allows to specify a limit
+    for the number of rows that should be deleted.
+
+    Deleting a large number of rows via just one SQL statement
+    may cause deadlocks in a replicated setup. This clause can
+    be used to split the one big DELETE statement into multiple
+    smaller statements ("chunks") which reduces the probability
+    of deadlocks.
+
+    A dedicated clause element is needed because different
+    databases use different notation for limits on DELETE statements.
+    """
+    def __init__(self, table, filter, limit=1000):
+        Delete.__init__(self, table)
+        self.table = table
+        self.filter = filter
+        if not isinstance(limit, int):
+            raise RuntimeError('limit must be an integer')
+        if limit <= 0:
+            raise RuntimeError('limit must be positive')
+        self.limit = limit
+
+
+@compiles(DeleteLimit)
+def visit_delete_limit(element, compiler, **kw):
+    """
+    Default compiler for the DeleteLimit clause element.
+    This compiles to a DELETE statement with a SELECT subquery which
+    has a limit set::
+
+        DELETE FROM ... WHERE id IN
+        (SELECT id FROM ... WHERE ... LIMIT ...)
+
+    However, this syntax is not supported by MySQL.
+    """
+    select_stmt = select([element.table.c.id]).where(element.filter).limit(element.limit)
+    delete_stmt = element.table.delete().where(element.table.c.id.in_(select_stmt))
+    return compiler.process(delete_stmt)
+
+
+@compiles(DeleteLimit, 'mysql')
+def visit_delete_limit_mysql(element, compiler, **kw):
+    """
+    Special compiler for the DeleteLimit clause element
+    for MySQL dialects. This compiles to a DELETE element
+    with a LIMIT::
+
+        DELETE FROM pidea_audit WHERE ... LIMIT ...
+    """
+    return 'DELETE FROM {} WHERE {} LIMIT {:d}'.format(
+        compiler.process(element.table, asfrom=True),
+        compiler.process(element.filter), element.limit)
+
+
+def delete_chunked(session, table, filter, limit=1000):
+    """
+    Delete all rows matching a given filter criterion from a table,
+    but only delete *limit* rows at a time. Commit after each DELETE.
+
+    :param session: SQLAlchemy session object
+    :param table: SQLAlchemy table object (e.g. ``LogEntry.__table__``)
+    :param filter: A filter criterion (e.g. ``LogEntry.age < now``)
+    :param limit: Number of rows to delete in one chunk
+    :return: total number of deleted rows
+    """
+    deleted = 0
+    statement = DeleteLimit(table, filter, limit)
+    while True:
+        result = session.execute(statement)
+        deleted += result.rowcount
+        session.commit()
+        if result.rowcount < limit:
+            return deleted
+
+
+def delete_matching_rows(session, table, filter, chunksize=None):
+    """
+    Delete all rows matching a given filter criterion from a table,
+    using chunked deletes if *chunksize* is not None.
+
+    :param session: session object
+    :param table: table object
+    :param filter: filter criterion
+    :param chunksize: An integer (a chunksize), or None.
+    :return: total number of deleted rows
+    """
+    if chunksize is None:
+        result = session.execute(table.delete().where(filter))
+        session.commit()
+        return result.rowcount
+    else:
+        return delete_chunked(session, table, filter, chunksize)


Test patch:
diff --git a/tests/test_lib_sqlutils.py b/tests/test_lib_sqlutils.py
new file mode 100644
--- /dev/null
+++ b/tests/test_lib_sqlutils.py
@@ -0,0 +1,80 @@
+"""
+This file contains the tests for lib/sqlutils.py
+"""
+from datetime import datetime
+
+from mock import MagicMock
+from sqlalchemy.orm import Session
+from sqlalchemy.testing import AssertsCompiledSQL, AssertsExecutionResults
+from sqlalchemy.testing.assertsql import CompiledSQL
+
+from privacyidea.lib.sqlutils import DeleteLimit, delete_matching_rows
+from privacyidea.models import Audit as LogEntry
+from .base import MyTestCase
+
+
+class SQLUtilsCompilationTestCase(MyTestCase, AssertsCompiledSQL):
+    def test_01_delete_limit(self):
+        stmt = DeleteLimit(LogEntry.__table__, LogEntry.id < 100)
+        self.assertEqual(stmt.limit, 1000)
+        stmt = DeleteLimit(LogEntry.__table__, LogEntry.id < 100, 1234)
+        self.assertEqual(stmt.limit, 1234)
+        with self.assertRaises(RuntimeError):
+            DeleteLimit(LogEntry.__table__, LogEntry.id < 100, None)
+        with self.assertRaises(RuntimeError):
+            DeleteLimit(LogEntry.__table__, LogEntry.id < 100, "not an integer")
+        with self.assertRaises(RuntimeError):
+            DeleteLimit(LogEntry.__table__, LogEntry.id < 100, 0)
+        with self.assertRaises(RuntimeError):
+            DeleteLimit(LogEntry.__table__, LogEntry.id < 100, -10)
+
+    def test_02_compile_delete_limit(self):
+        now = datetime.now()
+        stmt_age = DeleteLimit(LogEntry.__table__, LogEntry.date < now)
+        self.assert_compile(stmt_age,
+                            "DELETE FROM pidea_audit WHERE pidea_audit.id IN "
+                            "(SELECT pidea_audit.id FROM pidea_audit WHERE pidea_audit.date < :date_1 LIMIT :param_1)",
+                            checkparams={"date_1": now, "param_1": 1000},
+                            dialect='default')
+        self.assert_compile(stmt_age,
+                            "DELETE FROM pidea_audit WHERE pidea_audit.date < %s LIMIT 1000",
+                            checkpositional=(now,),
+                            dialect='mysql')
+
+        stmt_id = DeleteLimit(LogEntry.__table__, LogEntry.id < 1000, 1234)
+        self.assert_compile(stmt_id,
+                            "DELETE FROM pidea_audit WHERE pidea_audit.id IN "
+                            "(SELECT pidea_audit.id FROM pidea_audit WHERE pidea_audit.id < :id_1 LIMIT :param_1)",
+                            checkparams={"id_1": 1000, "param_1": 1234},
+                            dialect='default')
+        self.assert_compile(stmt_id,
+                            "DELETE FROM pidea_audit WHERE pidea_audit.id < %s LIMIT 1234",
+                            checkpositional=(1000,),
+                            dialect='mysql')
+
+    def test_03_delete(self):
+        session = MagicMock()
+
+        fake_results = [1000, 1000, 500]
+        def fake_execute_delete1(stmt):
+            result = MagicMock()
+            result.rowcount = fake_results.pop(0)
+            return result
+
+        session.execute.side_effect = fake_execute_delete1
+        # delete chunked
+        result = delete_matching_rows(session, LogEntry.__table__, LogEntry.id < 1234, 1000)
+        # was called three times to delete 2500 entries
+        self.assertEqual(len(session.execute.mock_calls), 3)
+        self.assertEqual(result, 2500)
+
+        session.execute.reset_mock()
+        def fake_execute_delete2(stmt):
+            result = MagicMock()
+            result.rowcount = 2500
+            return result
+        session.execute.side_effect = fake_execute_delete2
+        # delete in one statement
+        result = delete_matching_rows(session, LogEntry.__table__, LogEntry.id < 1234)
+        self.assertEqual(len(session.execute.mock_calls), 1)
+        self.assertEqual(result, 2500)
\ No newline at end of file