Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] File task remains in "Started" after hard timeout
### Description

While processing a large PDF document, paperless hits the hard time limit but the file task remains in the state "Started"  instead of changing to "Failed" signaling the processing error.


### Steps to reproduce

1. Add a large PDF to paperless
2. Check the File task is in state "Started"
3. Check the server log for "TimeLimitExceeded(1800)"
4. Check the File Task it now should be "Failed" but remains in "Started"

### Webserver logs

```bash
[2023-04-07 17:50:04,883] [INFO] [ocrmypdf._pipeline] with existing rotation ⇨, page is facing ⇧, confidence 34.43 - rotation appears correct
[2023-04-07 17:51:14,392] [INFO] [ocrmypdf._pipeline] with existing rotation ⇨, page is facing ⇧, confidence 30.87 - rotation appears correct
[2023-04-07 17:51:24,016] [ERROR] [celery.worker.request] Task handler raised error: TimeLimitExceeded(1800)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(1800,)
[2023-04-07 17:51:24,018] [ERROR] [celery.worker.request] Hard time limit (1800s) exceeded for documents.tasks.consume_file[6add6b72-3f87-463d-8614-feb3f9c24645]
2023-04-07 15:51:25,020 INFO reaped unknown pid 1448 (terminated by SIGPIPE)
[2023-04-07 17:51:26,832: ERROR/MainProcess] Process 'ForkPoolWorker-11' pid:541 exited with 'signal 9 (SIGKILL)'
```


### Browser logs

_No response_

### Paperless-ngx version

Paperless-ngx 1.13.0

### Host OS

Docker on Synology NAS

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I can confirm this - processes that run into timeouts are not properly marked as „failed“ and are stuck in the queue forever. 

Resolving patch:
diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py
--- a/src/documents/signals/handlers.py
+++ b/src/documents/signals/handlers.py
@@ -4,6 +4,7 @@
 
 from celery import states
 from celery.signals import before_task_publish
+from celery.signals import task_failure
 from celery.signals import task_postrun
 from celery.signals import task_prerun
 from django.conf import settings
@@ -591,3 +592,29 @@ def task_postrun_handler(
         # Don't let an exception in the signal handlers prevent
         # a document from being consumed.
         logger.exception("Updating PaperlessTask failed")
+
+
+@task_failure.connect
+def task_failure_handler(
+    sender=None,
+    task_id=None,
+    exception=None,
+    args=None,
+    traceback=None,
+    **kwargs,
+):
+    """
+    Updates the result of a failed PaperlessTask.
+
+    https://docs.celeryq.dev/en/stable/userguide/signals.html#task-failure
+    """
+    try:
+        task_instance = PaperlessTask.objects.filter(task_id=task_id).first()
+
+        if task_instance is not None and task_instance.result is None:
+            task_instance.status = states.FAILURE
+            task_instance.result = traceback
+            task_instance.date_done = timezone.now()
+            task_instance.save()
+    except Exception:  # pragma: no cover
+        logger.exception("Updating PaperlessTask failed")


Test patch:
diff --git a/src/documents/tests/test_task_signals.py b/src/documents/tests/test_task_signals.py
--- a/src/documents/tests/test_task_signals.py
+++ b/src/documents/tests/test_task_signals.py
@@ -9,6 +9,7 @@
 from documents.signals.handlers import before_task_publish_handler
 from documents.signals.handlers import task_postrun_handler
 from documents.signals.handlers import task_prerun_handler
+from documents.signals.handlers import task_failure_handler
 from documents.tests.test_consumer import fake_magic_from_file
 from documents.tests.utils import DirectoriesMixin
 
@@ -146,3 +147,44 @@ def test_task_postrun_handler(self):
         task = PaperlessTask.objects.get()
 
         self.assertEqual(celery.states.SUCCESS, task.status)
+
+    def test_task_failure_handler(self):
+        """
+        GIVEN:
+            - A celery task is started via the consume folder
+        WHEN:
+            - Task failed execution
+        THEN:
+            - The task is marked as failed
+        """
+        headers = {
+            "id": str(uuid.uuid4()),
+            "task": "documents.tasks.consume_file",
+        }
+        body = (
+            # args
+            (
+                ConsumableDocument(
+                    source=DocumentSource.ConsumeFolder,
+                    original_file="/consume/hello-9.pdf",
+                ),
+                None,
+            ),
+            # kwargs
+            {},
+            # celery stuff
+            {"callbacks": None, "errbacks": None, "chain": None, "chord": None},
+        )
+        self.util_call_before_task_publish_handler(
+            headers_to_use=headers,
+            body_to_use=body,
+        )
+
+        task_failure_handler(
+            task_id=headers["id"],
+            exception="Example failure",
+        )
+
+        task = PaperlessTask.objects.get()
+
+        self.assertEqual(celery.states.FAILURE, task.status)