Repository: pulp/pulpcore

Issue:
Add a scheduled cleanup task to keep the tasks table reasonable
We have seen substancial performance degredation on tasks tables that are several million rows long. While this is probably not uncommon in a usual Pulp installation it is a tedious task for the admin to call purge tasks regularly.
We should just schedule it. To give admins the control they are used to, this should be guarded by a setting (maybe down to different tasks states) for how long tasks are supposed to be finished before vacuumed.

Question: Are we confident to add a reasonable time as default, or does this feature need to be off by default?


Discussion:
I just want to point out that the purge is opened not only to admin but also to regular users too.

> Question: Are we confident to add a reasonable time as default, or does this feature need to be off by default?

I would pitch to have this feature off by default just because it is very difficult to come up with a 'reasonable' one size-fits-all value for the (a) task interval and (b) for how long to keep tasks since they are in their final state.

> should be guarded by a setting (maybe down to different tasks states) 

I would not distinguish between the task states and purge all of them ( final states).



Resolving patch:
diff --git a/pulpcore/app/settings.py b/pulpcore/app/settings.py
--- a/pulpcore/app/settings.py
+++ b/pulpcore/app/settings.py
@@ -256,8 +256,11 @@
 
 # how long to protect ephemeral items in minutes
 ORPHAN_PROTECTION_TIME = 24 * 60
-# if set to 0, upload and tmpfile cleanup task is disabled
+
+# Custom cleaup intervals
+# for the following, if set to 0, the corresponding cleanup task is disabled
 UPLOAD_PROTECTION_TIME = 0
+TASK_PROTECTION_TIME = 0
 TMPFILE_PROTECTION_TIME = 0
 
 REMOTE_USER_ENVIRON_NAME = "REMOTE_USER"
diff --git a/pulpcore/app/tasks/purge.py b/pulpcore/app/tasks/purge.py
--- a/pulpcore/app/tasks/purge.py
+++ b/pulpcore/app/tasks/purge.py
@@ -1,7 +1,9 @@
 from gettext import gettext as _
 from logging import getLogger
 
+from django.conf import settings
 from django.db.models.deletion import ProtectedError
+from django.utils import timezone
 
 from pulpcore.app.models import (
     ProgressReport,
@@ -9,7 +11,7 @@
 )
 from pulpcore.app.role_util import get_objects_for_user
 from pulpcore.app.util import get_domain, get_current_authenticated_user
-from pulpcore.constants import TASK_STATES
+from pulpcore.constants import TASK_STATES, TASK_FINAL_STATES
 
 log = getLogger(__name__)
 
@@ -55,7 +57,7 @@ def _details_reporting(current_reports, current_details, totals_pb):
     return current_reports
 
 
-def purge(finished_before, states):
+def purge(finished_before=None, states=None):
     """
     This task purges from the database records of tasks which finished prior to the specified time.
 
@@ -70,10 +72,15 @@ def purge(finished_before, states):
     by deleting a Task.
 
     Args:
-        finished_before (DateTime): Earliest finished-time to **NOT** purge.
-        states (List[str]): List of task-states we want to purge.
+        finished_before (Optional[DateTime]): Earliest finished-time to **NOT** purge.
+        states (Optional[List[str]]): List of task-states we want to purge.
 
     """
+    if finished_before is None:
+        assert settings.TASK_PROTECTION_TIME > 0
+        finished_before = timezone.now() - timezone.timedelta(minutes=settings.TASK_PROTECTION_TIME)
+    if states is None:
+        states = TASK_FINAL_STATES
     current_user = get_current_authenticated_user()
     domain = get_domain()
     # Tasks, prior to the specified date, in the specified state, owned by the current-user, in the
diff --git a/pulpcore/app/util.py b/pulpcore/app/util.py
--- a/pulpcore/app/util.py
+++ b/pulpcore/app/util.py
@@ -379,6 +379,7 @@ def configure_cleanup():
             "pulpcore.app.tasks.orphan.tmpfile_cleanup",
             settings.TMPFILE_PROTECTION_TIME,
         ),
+        ("tasks", "pulpcore.app.tasks.purge.purge", settings.TASK_PROTECTION_TIME),
     ]:
         if protection_time > 0:
             dispatch_interval = timedelta(minutes=protection_time)


Test patch: