Repository: learningequality/kolibri

Issue:
In Kolibri deployments, multiple frontend instances cause duplicate processing of tasks
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
In Kolibri cloud deployments, such as Kolibri Data Portal (KDP), we start multiple instances of Kolibri, each within an individual Kubernetes pod. Starting several instances of Kolibri means we also start up separate task workers within each pod. Doing so creates the opportunity for the duplication of task processing, since Kolibri's task processing does not protect against this happening.

## Errors and logs
In KDP, we receive errors similar to the following, which are raised in the asynchronous processing of Kolibri syncs:
```
django.db.utils.IntegrityError: duplicate key value violates unique constraint "morango_databasemaxcounter_instance_id_partition_99e4f1fb_uniq"
```
which leads to
```
kolibri_data_portal.management.commands.sync_proceed_to.SyncProceedToError: Failed to finalize <transfer session ID>
```
This would appear to be an issue, but isn't because when you look at the Morango `TransferSession` referenced by the error, you can see that it did complete successfully and didn't actually "fail to finalize". In addition, you can also see that the transfer was completed prior to the `duplicate key value violates unique constraint` error, which indicates that there was another worker trying to process the same task, hence the reason for `duplicate key` error.

## Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
Kolibri should implement a distributed locking mechanism which prevents Kolibri task workers from picking up tasks which have already been picked up and "locked" by a worker. It may be useful to annotate which worker picked up the task on the job record, and have workers filter for tasks using that; for instance the combination of worker hostname and process ID should work appropriately in deployments like KDP and would be useful for debugging too. 

## User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->
This can cause significant database bottlenecks on KDP's side since these tasks can retry their operations when they encounter transaction serialization errors due to their more aggressive transaction isolation. Also, the flood of these errors obscure real errors that need to be resolved.

## Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Kolibri 0.14+0.15
Linux + Google Cloud




Discussion:
@rtibbles 

Resolving patch:
diff --git a/kolibri/core/tasks/job.py b/kolibri/core/tasks/job.py
--- a/kolibri/core/tasks/job.py
+++ b/kolibri/core/tasks/job.py
@@ -47,6 +47,9 @@ class State(object):
     QUEUED means the Job has been sent to the workers for running, but has not
     been run yet (to our knowledge).
 
+    SELECTED means the Job has been picked up by the worker controller as a next
+    job to pass to its workers, but that it has not actually started running yet.
+
     RUNNING means that one of the workers has started running the job, but is not
     complete yet. If the job has been set to track progress, then the job's progress
     and total_progress fields should be continuously updated.
@@ -66,6 +69,7 @@ class State(object):
     PENDING = "PENDING"
     SCHEDULED = "SCHEDULED"
     QUEUED = "QUEUED"
+    SELECTED = "SELECTED"
     RUNNING = "RUNNING"
     FAILED = "FAILED"
     CANCELING = "CANCELING"
diff --git a/kolibri/core/tasks/storage.py b/kolibri/core/tasks/storage.py
--- a/kolibri/core/tasks/storage.py
+++ b/kolibri/core/tasks/storage.py
@@ -6,6 +6,7 @@
 from sqlalchemy import func
 from sqlalchemy import or_
 from sqlalchemy import String
+from sqlalchemy import update
 from sqlalchemy.exc import OperationalError
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker
@@ -148,6 +149,50 @@ def mark_job_as_canceling(self, job_id):
         """
         self._update_job(job_id, State.CANCELING)
 
+    def _postgres_next_queued_job(self, session, priority):
+        """
+        For postgres we are doing our best to ensure that the selected job
+        is not then also selected by another potentially concurrent worker controller
+        process. We do this by doing a select for update within a subquery.
+        This should work as long as our connection uses the default isolation level
+        of READ_COMMITTED.
+        More details here: https://dba.stackexchange.com/a/69497
+        For SQLAlchemy details here: https://stackoverflow.com/a/25943713
+        """
+        subquery = (
+            session.query(ORMJob.id)
+            .filter(ORMJob.state == State.QUEUED)
+            .filter(ORMJob.priority == priority)
+            .order_by(ORMJob.time_created)
+            .limit(1)
+            .with_for_update(skip_locked=True)
+        )
+        return self.engine.execute(
+            update(ORMJob)
+            .values(state=State.SELECTED)
+            .where(ORMJob.id == subquery.as_scalar())
+            .returning(ORMJob.saved_job)
+        ).fetchone()
+
+    def _sqlite_next_queued_job(self, session, priority):
+        """
+        Due to the difficulty in appropriately locking the task row
+        we do not support multiple task runners potentially duelling
+        to lock tasks for SQLite, so here we just do a minimal
+        best effort to mark the job as selected for running.
+        """
+        orm_job = (
+            session.query(ORMJob)
+            .filter(ORMJob.state == State.QUEUED)
+            .filter(ORMJob.priority == priority)
+            .order_by(ORMJob.time_created)
+            .first()
+        )
+        if orm_job:
+            orm_job.state = State.SELECTED
+            session.add(orm_job)
+        return orm_job
+
     def get_next_queued_job(self, priority_order=Priority.PriorityOrder):
         """
         Looks for the oldest queued job with priorities provided in `priority_order`. It
@@ -161,15 +206,14 @@ def get_next_queued_job(self, priority_order=Priority.PriorityOrder):
         :return: job if found else None.
         """
         with self.session_scope() as s:
-            q = s.query(ORMJob).filter(ORMJob.state == State.QUEUED)
-
             orm_job = None
+            method = (
+                self._sqlite_next_queued_job
+                if self.engine.dialect.name == "sqlite"
+                else self._postgres_next_queued_job
+            )
             for priority in priority_order:
-                orm_job = (
-                    q.filter(ORMJob.priority == priority)
-                    .order_by(ORMJob.time_created)
-                    .first()
-                )
+                orm_job = method(s, priority)
                 if orm_job:
                     break
 


Test patch:
diff --git a/kolibri/core/tasks/test/taskrunner/test_worker.py b/kolibri/core/tasks/test/taskrunner/test_worker.py
--- a/kolibri/core/tasks/test/taskrunner/test_worker.py
+++ b/kolibri/core/tasks/test/taskrunner/test_worker.py
@@ -7,6 +7,7 @@
 from kolibri.core.tasks.job import Job
 from kolibri.core.tasks.job import State
 from kolibri.core.tasks.test.base import connection
+from kolibri.core.tasks.test.taskrunner.test_job_running import EventProxy
 from kolibri.core.tasks.worker import Worker
 
 QUEUE = "pytest"
@@ -23,6 +24,21 @@ def error_func():
     raise TypeError(error_text)
 
 
+@pytest.fixture
+def flag():
+    e = EventProxy()
+    yield e
+    e.clear()
+
+
+def toggle_flag(flag_id):
+    evt = EventProxy(event_id=flag_id)
+    if evt.is_set():
+        evt.clear()
+    else:
+        evt.set()
+
+
 @pytest.fixture
 def worker():
     with connection() as c:
@@ -33,6 +49,16 @@ def worker():
         b.shutdown()
 
 
+@pytest.fixture
+def worker2():
+    with connection() as c:
+        b = Worker(c, regular_workers=1, high_workers=1)
+        b.storage.clear(force=True)
+        yield b
+        b.storage.clear(force=True)
+        b.shutdown()
+
+
 @pytest.mark.django_db
 class TestWorker:
     def test_enqueue_job_runs_job(self, worker):
@@ -52,6 +78,25 @@ def test_enqueue_job_runs_job(self, worker):
 
         assert job.state == State.COMPLETED
 
+    def test_enqueue_job_runs_job_once(self, worker, worker2, flag):
+
+        job = Job(toggle_flag, flag.event_id)
+        worker.storage.enqueue_job(job, QUEUE)
+
+        while job.state != State.COMPLETED:
+            job = worker.storage.get_job(job.job_id)
+            time.sleep(0.5)
+        try:
+            # Get the future, or pass if it has already been cleaned up.
+            future = worker.future_job_mapping[job.job_id]
+
+            future.result()
+        except KeyError:
+            pass
+
+        assert job.state == State.COMPLETED
+        assert flag.is_set()
+
     def test_can_handle_unicode_exceptions(self, worker):
         # Make sure task exception info is not an object, but is either a string or None.
         # See Storage.mark_job_as_failed in kolibri.core.tasks.storage for more details on why we do this.