Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Duplicate tasks pile up if the task-queue is offline
### Discussed in https://github.com/paperless-ngx/paperless-ngx/discussions/2593

<div type='discussions-op-text'>

<sup>Originally posted by **hashworks** February  3, 2023</sup>
### Description

Recent versions added a new service to run, paperless-task-queue, otherwise known as the Celery Workers.

They handle tasks that get queued in the redis DB. This works fine for normal system operation where all required processes are running fine.

If the task-queue service goes down for a while, the webserver, the consumer and the scheduler service don't mind at all - they will happily queue new tasks in the redis DB since their job is done. This is a problem when one restarts the task-queue service days later:

```
[2023-02-03 13:08:34,827] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[8bcc8e12-dd69-4c91-826b-1f3388f4b1ec] received
[2023-02-03 13:08:35,220] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[c7c52433-7a1a-4dc6-8895-92ff5e90e152] received
[2023-02-03 13:09:16,771] [INFO] [celery.app.trace] Task paperless_mail.tasks.process_mail_accounts[e839b305-63a3-4617-99a5-fc83985194ab] succeeded in 41.953297510015545s: 'No new documents were added.'
[2023-02-03 13:09:16,833] [INFO] [celery.app.trace] Task paperless_mail.tasks.process_mail_accounts[d1e5fe78-9b16-4f93-b4eb-accfe6031d8e] succeeded in 41.921987653011456s: 'No new documents were added.'
[2023-02-03 13:09:16,839] [INFO] [celery.app.trace] Task paperless_mail.tasks.process_mail_accounts[6c80a09b-0ab3-470c-8305-712193ec902b] succeeded in 42.04609970000456s: 'No new documents were added.'
[2023-02-03 13:09:17,062] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[3bf543ea-189e-406e-b45c-9c2dbfa4037f] received
[2023-02-03 13:09:17,063] [INFO] [celery.worker.strategy] Task documents.tasks.index_optimize[689bf96f-7077-47bb-a9c7-76c3362b1279] received
[2023-02-03 13:09:17,107] [INFO] [celery.app.trace] Task documents.tasks.train_classifier[cab4d3ff-2809-4570-85f9-3339f33a645b] succeeded in 0.02393642399692908s: None
[2023-02-03 13:09:17,109] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[d2ae11b2-0c9c-4920-abe2-599f6843eda6] received
[2023-02-03 13:09:19,266] [INFO] [celery.worker.strategy] Task documents.tasks.train_classifier[49f73466-33e0-4d76-8956-a1ac02ab3e19] received
```

The task-queue will then try to handle all queued up tasks, including classifiers and mail account checks that are normally only run every few minutes. This results in multiple hours of mail checking where no new documents are handled.

Tasks that are queued by a cron job should have a reasonable timeout. If they are added every 10 minutes, they should be discarded if they are older than that.

### Steps to reproduce

1. Use tasks that are added by a cron job, like mail checking
2. Stop the celery workers that handle the task queue
3. Wait a few days
4. Restart celery workers
5. See that one needs to wait a few hours for the celery workers to handle all piled up tasks

### Paperless-ngx version

1.12.2

### Host OS

Arch Linux

### Installation method

Bare metal

### Other

A related issue is #2549. It was closed by recommending a redis reset, but I think a task timeout would be a better solution.</div>


Discussion:

Resolving patch:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -109,6 +109,16 @@ def _parse_redis_url(env_redis: Optional[str]) -> Tuple[str]:
 
 
 def _parse_beat_schedule() -> Dict:
+    """
+    Configures the scheduled tasks, according to default or
+    environment variables.  Task expiration is configured so the task will
+    expire (and not run), shortly before the default frequency will put another
+    of the same task into the queue
+
+
+    https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html#beat-entries
+    https://docs.celeryq.dev/en/latest/userguide/calling.html#expiration
+    """
     schedule = {}
     tasks = [
         {
@@ -117,6 +127,11 @@ def _parse_beat_schedule() -> Dict:
             # Default every ten minutes
             "env_default": "*/10 * * * *",
             "task": "paperless_mail.tasks.process_mail_accounts",
+            "options": {
+                # 1 minute before default schedule sends again
+                "expires": 9.0
+                * 60.0,
+            },
         },
         {
             "name": "Train the classifier",
@@ -124,6 +139,11 @@ def _parse_beat_schedule() -> Dict:
             # Default hourly at 5 minutes past the hour
             "env_default": "5 */1 * * *",
             "task": "documents.tasks.train_classifier",
+            "options": {
+                # 1 minute before default schedule sends again
+                "expires": 59.0
+                * 60.0,
+            },
         },
         {
             "name": "Optimize the index",
@@ -131,6 +151,12 @@ def _parse_beat_schedule() -> Dict:
             # Default daily at midnight
             "env_default": "0 0 * * *",
             "task": "documents.tasks.index_optimize",
+            "options": {
+                # 1 hour before default schedule sends again
+                "expires": 23.0
+                * 60.0
+                * 60.0,
+            },
         },
         {
             "name": "Perform sanity check",
@@ -138,6 +164,13 @@ def _parse_beat_schedule() -> Dict:
             # Default Sunday at 00:30
             "env_default": "30 0 * * sun",
             "task": "documents.tasks.sanity_check",
+            "options": {
+                # 1 hour before default schedule sends again
+                "expires": 7.0
+                * 23.0
+                * 60.0
+                * 60.0,
+            },
         },
     ]
     for task in tasks:
@@ -151,9 +184,11 @@ def _parse_beat_schedule() -> Dict:
         #   - five time-and-date fields
         #   - separated by at least one blank
         minute, hour, day_month, month, day_week = value.split(" ")
+
         schedule[task["name"]] = {
             "task": task["task"],
             "schedule": crontab(minute, hour, day_week, day_month, month),
+            "options": task["options"],
         }
 
     return schedule
@@ -561,22 +596,21 @@ def _parse_beat_schedule() -> Dict:
 # Task queue                                                                  #
 ###############################################################################
 
-TASK_WORKERS = __get_int("PAPERLESS_TASK_WORKERS", 1)
-
-WORKER_TIMEOUT: Final[int] = __get_int("PAPERLESS_WORKER_TIMEOUT", 1800)
+# https://docs.celeryq.dev/en/stable/userguide/configuration.html
 
 CELERY_BROKER_URL = _CELERY_REDIS_URL
 CELERY_TIMEZONE = TIME_ZONE
 
 CELERY_WORKER_HIJACK_ROOT_LOGGER = False
-CELERY_WORKER_CONCURRENCY = TASK_WORKERS
+CELERY_WORKER_CONCURRENCY: Final[int] = __get_int("PAPERLESS_TASK_WORKERS", 1)
+TASK_WORKERS = CELERY_WORKER_CONCURRENCY
 CELERY_WORKER_MAX_TASKS_PER_CHILD = 1
 CELERY_WORKER_SEND_TASK_EVENTS = True
-
+CELERY_TASK_SEND_SENT_EVENT = True
 CELERY_SEND_TASK_SENT_EVENT = True
 
 CELERY_TASK_TRACK_STARTED = True
-CELERY_TASK_TIME_LIMIT = WORKER_TIMEOUT
+CELERY_TASK_TIME_LIMIT: Final[int] = __get_int("PAPERLESS_WORKER_TIMEOUT", 1800)
 
 CELERY_RESULT_EXTENDED = True
 CELERY_RESULT_BACKEND = "django-db"
@@ -608,7 +642,7 @@ def default_threads_per_worker(task_workers) -> int:
 
 THREADS_PER_WORKER = os.getenv(
     "PAPERLESS_THREADS_PER_WORKER",
-    default_threads_per_worker(TASK_WORKERS),
+    default_threads_per_worker(CELERY_WORKER_CONCURRENCY),
 )
 
 ###############################################################################


Test patch:
diff --git a/src/paperless/tests/test_settings.py b/src/paperless/tests/test_settings.py
--- a/src/paperless/tests/test_settings.py
+++ b/src/paperless/tests/test_settings.py
@@ -149,6 +149,11 @@ def test_redis_socket_parsing(self):
 
 
 class TestCeleryScheduleParsing(TestCase):
+    MAIL_EXPIRE_TIME = 9.0 * 60.0
+    CLASSIFIER_EXPIRE_TIME = 59.0 * 60.0
+    INDEX_EXPIRE_TIME = 23.0 * 60.0 * 60.0
+    SANITY_EXPIRE_TIME = 7.0 * 23.0 * 60.0 * 60.0
+
     def test_schedule_configuration_default(self):
         """
         GIVEN:
@@ -165,18 +170,22 @@ def test_schedule_configuration_default(self):
                 "Check all e-mail accounts": {
                     "task": "paperless_mail.tasks.process_mail_accounts",
                     "schedule": crontab(minute="*/10"),
+                    "options": {"expires": self.MAIL_EXPIRE_TIME},
                 },
                 "Train the classifier": {
                     "task": "documents.tasks.train_classifier",
                     "schedule": crontab(minute="5", hour="*/1"),
+                    "options": {"expires": self.CLASSIFIER_EXPIRE_TIME},
                 },
                 "Optimize the index": {
                     "task": "documents.tasks.index_optimize",
                     "schedule": crontab(minute=0, hour=0),
+                    "options": {"expires": self.INDEX_EXPIRE_TIME},
                 },
                 "Perform sanity check": {
                     "task": "documents.tasks.sanity_check",
                     "schedule": crontab(minute=30, hour=0, day_of_week="sun"),
+                    "options": {"expires": self.SANITY_EXPIRE_TIME},
                 },
             },
             schedule,
@@ -203,18 +212,22 @@ def test_schedule_configuration_changed(self):
                 "Check all e-mail accounts": {
                     "task": "paperless_mail.tasks.process_mail_accounts",
                     "schedule": crontab(minute="*/50", day_of_week="mon"),
+                    "options": {"expires": self.MAIL_EXPIRE_TIME},
                 },
                 "Train the classifier": {
                     "task": "documents.tasks.train_classifier",
                     "schedule": crontab(minute="5", hour="*/1"),
+                    "options": {"expires": self.CLASSIFIER_EXPIRE_TIME},
                 },
                 "Optimize the index": {
                     "task": "documents.tasks.index_optimize",
                     "schedule": crontab(minute=0, hour=0),
+                    "options": {"expires": self.INDEX_EXPIRE_TIME},
                 },
                 "Perform sanity check": {
                     "task": "documents.tasks.sanity_check",
                     "schedule": crontab(minute=30, hour=0, day_of_week="sun"),
+                    "options": {"expires": self.SANITY_EXPIRE_TIME},
                 },
             },
             schedule,
@@ -238,14 +251,17 @@ def test_schedule_configuration_disabled(self):
                 "Check all e-mail accounts": {
                     "task": "paperless_mail.tasks.process_mail_accounts",
                     "schedule": crontab(minute="*/10"),
+                    "options": {"expires": self.MAIL_EXPIRE_TIME},
                 },
                 "Train the classifier": {
                     "task": "documents.tasks.train_classifier",
                     "schedule": crontab(minute="5", hour="*/1"),
+                    "options": {"expires": self.CLASSIFIER_EXPIRE_TIME},
                 },
                 "Perform sanity check": {
                     "task": "documents.tasks.sanity_check",
                     "schedule": crontab(minute=30, hour=0, day_of_week="sun"),
+                    "options": {"expires": self.SANITY_EXPIRE_TIME},
                 },
             },
             schedule,