Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Increase PAPERLESS_WORKER_TIMEOUT results in a django warning
**Describe the bug**
When I increase PAPERLESS_WORKER_TIMEOUT and change the value from 1800 to 3600, I will get Warnings on Docker-Log.
Occurs since the following PR https://github.com/paperless-ngx/paperless-ngx/pull/37

**To Reproduce**
Steps to reproduce the behavior:
1. Change Config PAPERLESS_WORKER_TIMEOUT  to 3600
2. See warning messages in docker-logs

**Expected behavior**
No warnings from django


**Webserver logs**
 ```
paperless_1          |         /usr/local/lib/python3.9/site-packages/django_q/conf.py:139: UserWarning: Retry and timeout are misconfigured. Set retry larger than timeout,
paperless_1          |         failure to do so will cause the tasks to be retriggered before completion.
paperless_1          |         See https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
```

**Relevant information**
 - Host OS of the machine running paperless: Debian 11.2
 - Docker Image: ghcr.io/paperless-ngx/paperless-ngx:latest
 - Version: 1.6.0
 - Installation method: docker



Discussion:
I'm not sure I see what the bug is here.  This seems like expected behavior given the setting of the timeout value.  What's the use case for the larger worker timeout?
For example a 4 MB File with > 10 pages has a duration of 45 minutes on my server. So it's run into a timeout and the dokument doesn't imported.
I see.  It should be simple enough to include configuration of the retry value for cases like this.  That will silence the warning.

Resolving patch:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -3,6 +3,7 @@
 import multiprocessing
 import os
 import re
+from typing import Final
 
 from concurrent_log_handler.queue import setup_logging_queues
 from django.utils.translation import gettext_lazy as _
@@ -29,7 +30,7 @@
 os.environ["OMP_THREAD_LIMIT"] = "1"
 
 
-def __get_boolean(key, default="NO"):
+def __get_boolean(key: str, default: str = "NO") -> bool:
     """
     Return a boolean value based on whatever the user has supplied in the
     environment based on whether the value "looks like" it's True or not.
@@ -37,6 +38,13 @@ def __get_boolean(key, default="NO"):
     return bool(os.getenv(key, default).lower() in ("yes", "y", "1", "t", "true"))
 
 
+def __get_int(key: str, default: int) -> int:
+    """
+    Return an integer value based on the environment variable or a default
+    """
+    return int(os.getenv(key, default))
+
+
 # NEVER RUN WITH DEBUG IN PRODUCTION.
 DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO")
 
@@ -395,7 +403,7 @@ def __get_boolean(key, default="NO"):
 # in total.
 
 
-def default_task_workers():
+def default_task_workers() -> int:
     # always leave one core open
     available_cores = max(multiprocessing.cpu_count(), 1)
     try:
@@ -406,20 +414,29 @@ def default_task_workers():
         return 1
 
 
-TASK_WORKERS = int(os.getenv("PAPERLESS_TASK_WORKERS", default_task_workers()))
+TASK_WORKERS = __get_int("PAPERLESS_TASK_WORKERS", default_task_workers())
+
+PAPERLESS_WORKER_TIMEOUT: Final[int] = __get_int("PAPERLESS_WORKER_TIMEOUT", 1800)
+
+# Per django-q docs, timeout must be smaller than retry
+# We default retry to 10s more than the timeout
+PAPERLESS_WORKER_RETRY: Final[int] = __get_int(
+    "PAPERLESS_WORKER_RETRY",
+    PAPERLESS_WORKER_TIMEOUT + 10,
+)
 
 Q_CLUSTER = {
     "name": "paperless",
     "catch_up": False,
     "recycle": 1,
-    "retry": 1800,
-    "timeout": int(os.getenv("PAPERLESS_WORKER_TIMEOUT", 1800)),
+    "retry": PAPERLESS_WORKER_RETRY,
+    "timeout": PAPERLESS_WORKER_TIMEOUT,
     "workers": TASK_WORKERS,
     "redis": os.getenv("PAPERLESS_REDIS", "redis://localhost:6379"),
 }
 
 
-def default_threads_per_worker(task_workers):
+def default_threads_per_worker(task_workers) -> int:
     # always leave one core open
     available_cores = max(multiprocessing.cpu_count(), 1)
     try:


Test patch: