Repository: paperless-ngx/paperless-ngx

Issue:
Reduce PAPERLESS_TASK_WORKERS and  PAPERLESS_WEBSERVER_WORKERS default values
### Description

# Problem with memory and CPU usage
In connection with #1084, reducing the `PAPERLESS_TASK_WORKERS` and  `PAPERLESS_WEBSERVER_WORKERS` default values to `1` helps not only with reducing the memory footprint (from approx. 400mb to approx. 200mb) but because the Django-Q's internals, the CPU usage goes downs as well (each worker checks in every second, which creates a Redis hit; the more workers, the more check-ins, the more Redis hits, the higher the idle CPU usage).

# Typical paperless use-case
I argue paperless is mostly used by hobbyists to store simple invoices, notes, etc. However, by and large, the instance is idling rather than running multicore, CPU heavy processes (see typical use-cases on [/r/selfhosted](https://www.reddit.com/r/selfhosted/search/?q=paperless))

# Change the defaults to match the typical use-case
If the instance is mostly idling and every once in a while processing a new document, it is more reasonable than not to limit the default number of workers to `1`. This is still configurable via env variables, so if anyone ever needs more workers, they can change it, but most users, simply starting the docker container, would benefit from the reduced memory and CPU usage.

Any thoughts, objections?

@stumpylog , I'm happy to create a PR to change the defaults and update the [documentation](https://paperless-ngx.readthedocs.io/en/latest/setup.html?highlight=PAPERLESS_TASK_WORKERS#considerations-for-less-powerful-devices) as well if we are happy with reducing the defaults to preserve memory and CPU.





# Affected code

On any modern CPU (including slow, Atom CPUs) paperless creates [unnecessary amount of task workers based on core-count](https://github.com/paperless-ngx/paperless-ngx/blob/9a1bd9637cbef0e7318a3c61cae14805c6510cea/src/paperless/settings.py#L438) and [exactly 2 web workers](https://github.com/paperless-ngx/paperless-ngx/blob/9a1bd9637cbef0e7318a3c61cae14805c6510cea/gunicorn.conf.py#L4) for historical (?) reasons

### Webserver logs

_No response_

### Paperless-ngx version

1.7.1

### Host OS

Ubuntu

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Obviously appreciate all your thoughts, why is this a separate issue from #1084?
@shamoon , this helps both with idle CPU consumption, and memory reduction as well; however, the most drastic effect is on memory.
Sometimes people are funny about changing defaults, so I prefer to have a separate thread/bug/conversation around it rather than mixing everything in #1084 which would eventually become un-closable due to the mixed topics.
As all of these settings look to be already documented as options to tweak for better performance and/or less resource usage, I'm not super inclined to change the default away from the algorithm used to determine worker and thread counts.

My preference remains looking further into the `django-q` code in hopes to make it less polling, more event driven.  This is something #1014 will be able to assist with, as we'll have our own, updatable version to experiment with.
I agree, @stumpylog it is documented. However, the existence of the documentation may not has to do a lot with sensible default values?

Let me explain.

**History of adding workers**

The first time a CPU core/thread based worker calculation was [added 2 years ago in the **original** paperless-ng project](https://github.com/jonaswinkler/paperless-ng/blame/c7c98c623de1ae809d9c63890a1bd9361651bbc1/src/paperless/settings.py#L341).

This change did not have any special justification. It seems like it was copy-pasted from another project or sample code (with the best intentions, of course), but these defaults were not well thought-out for paperless specifically. Consequently, *just because it is how it is now*, it should not mean we cannot improve the codebase (i.e., improve the default values) - and I'm sure you agree this applies to all code in the codebase?

**Assumptions**
- Most hobbyists (the main users) of the project are only somewhat technical at best
- Most of them run multiple docker instances on their home servers
- Most of them would not to trawl through all documentation of all  the projects they are using 
- Most of them would have a hard time understanding how changing the number of web and tasks workers would affect things (would something break or get better? how would they know?)
- Importantly, most users run their dockers with default settings

**Conclusions from the history and assumptions**

Given the worker-count was introduced without much discussion or justification and the assumptions above, I think *the default settings should be re-targeted at the most typical use-case*: hobbyist, single user, running on low-spec home servers. 

They (we) would all love and benefit from lower memory and CPU consumption, wouldn't you agree?

Conversely, I'm not sure who and how benefits from the current defaults? 
Given the project's goals, I'm having a hard time justifying when/who would ever *need* to have more workers than 1 - but even if some rare cases they do, it would remain configurable, so the rare case can be catered for; yet, the default settings would (and *should!*) serve better the majority of the users.
Not being a web related developer at all, does reducing `PAPERLESS_WEBSERVER_WORKERS` affect the responsiveness of the browsing experience?  I assume each page load makes multiple API requests to gather data, does only 1 worker take longer to respond?

For other worker count, I could see reducing the counts further, essentially to 1 process for low core counts, with it only starting to increase at 16+ cores.  Along with a small tweak to the threads per worker count as well.  Something like this (for some likely common core counts).  The original value on the left, new calculation on the right.

Cores: 1, workers: 1 -> 1, threads: 1 -> 1
Cores: 2, workers: 2 -> 1, threads: 1 -> 1
Cores: 4, workers: 2 -> 1, threads: 2 -> 1
Cores: 8, workers: 2 -> 1, threads: 4 -> 2
Cores: 16, workers: 4 -> 2, threads: 4 -> 2
Cores: 32, workers: 5 -> 2, threads: 6 -> 4
Cores: 64, workers: 8 -> 2, threads: 8 -> 8
Cores: 128, workers: 11 -> 3, threads: 11 -> 10

The idea being fewer workers on lower hardware, but starting to increase as the core count (and so presumable resources) increases.

> does reducing `PAPERLESS_WEBSERVER_WORKERS` affect the responsiveness of the browsing experience?

In practice, no (I changed both web and worker count to 1, haven't noticed any differences apart from lower resource usage).

However, I just checked the performance metrics of the whole page load, and was staggering to see how big the thumbnails of the scans are.

The thumbnail loading time is so long and inefficient compared to everything else that this whole current argument is moot: the thumbnails are saved as PNG and are **humongous**. 

For instance, the thumbnail (i.e., the `/api/documents/2/thumb/` image) of a simple scanned invoice I have is
- 2.6M (!!!), which loads every time I open the `/documents` page
- If I convert it to JPG, it goes down to 274K (10.3% of the original PNG thumbnail size)
- If I convert it to WebP, it goes down to 89K (a mere 3.3% of the original PNG "thumbnail" size)

I created a separate bug to track this, should be an easy fix (#1124)

>  1 process for low core counts, with it only starting to increase at 16+ cores

On face value it seems like a good idea; however, it is saying that "if you have more cores, you'll *need* to add more memory and more idling CPU time to paperless". 

Is this the case though? Is this true that just because paperless is running on a large server, potentially shared among many services and users, we *need* to allocate more resources?

I'm not sure a larger core count justifies assigning more resources to paperless. Just because we could, it does not mean we should. What are your thoughts?
I also dont really notice a difference in testing (the frontend) but TBH I dont have much expertise in the ramifications of this, though in reading a bit I get the sense that in setups with a small amount of users (i.e. concurrent requests) its unlikely that the webserver would be taxed to the point that it was beneficial to have > 1. See https://docs.gunicorn.org/en/stable/design.html#how-many-workers

Again, defer to others if someone has more insight.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Given that there isn't any downsides making this change, the arguments I put forward and the reference from @shamoon, could we please make this change for the greater good?
I also don't fully understand the ramifications, but given #1127 is now merged (presumably reducing the load on each individual worker [described here](https://github.com/paperless-ngx/paperless-ngx/issues/1098#issuecomment-1151991273)) I think reducing the number of workers is a broad solution to the performance issues we've seen lately.

I agree it should be dealt with in either `django-q` or some other dynamic scaling, but until that's implemented this might be a reliable solution.

Resolving patch:
diff --git a/gunicorn.conf.py b/gunicorn.conf.py
--- a/gunicorn.conf.py
+++ b/gunicorn.conf.py
@@ -1,7 +1,7 @@
 import os
 
 bind = f'[::]:{os.getenv("PAPERLESS_PORT", 8000)}'
-workers = int(os.getenv("PAPERLESS_WEBSERVER_WORKERS", 2))
+workers = int(os.getenv("PAPERLESS_WEBSERVER_WORKERS", 1))
 worker_class = "paperless.workers.ConfigurableWorker"
 timeout = 120
 
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -425,27 +425,7 @@ def __get_float(key: str, default: float) -> float:
 # Task queue                                                                  #
 ###############################################################################
 
-
-# Sensible defaults for multitasking:
-# use a fair balance between worker processes and threads epr worker so that
-# both consuming many documents in parallel and consuming large documents is
-# reasonably fast.
-# Favors threads per worker on smaller systems and never exceeds cpu_count()
-# in total.
-
-
-def default_task_workers() -> int:
-    # always leave one core open
-    available_cores = max(multiprocessing.cpu_count(), 1)
-    try:
-        if available_cores < 4:
-            return available_cores
-        return max(math.floor(math.sqrt(available_cores)), 1)
-    except NotImplementedError:
-        return 1
-
-
-TASK_WORKERS = __get_int("PAPERLESS_TASK_WORKERS", default_task_workers())
+TASK_WORKERS = __get_int("PAPERLESS_TASK_WORKERS", 1)
 
 PAPERLESS_WORKER_TIMEOUT: Final[int] = __get_int("PAPERLESS_WORKER_TIMEOUT", 1800)
 


Test patch:
diff --git a/src/documents/tests/test_settings.py b/src/documents/tests/test_settings.py
deleted file mode 100644
--- a/src/documents/tests/test_settings.py
+++ /dev/null
@@ -1,35 +0,0 @@
-import logging
-from unittest import mock
-
-from django.test import TestCase
-from paperless.settings import default_task_workers
-from paperless.settings import default_threads_per_worker
-
-
-class TestSettings(TestCase):
-    @mock.patch("paperless.settings.multiprocessing.cpu_count")
-    def test_single_core(self, cpu_count):
-        cpu_count.return_value = 1
-
-        default_workers = default_task_workers()
-
-        default_threads = default_threads_per_worker(default_workers)
-
-        self.assertEqual(default_workers, 1)
-        self.assertEqual(default_threads, 1)
-
-    def test_workers_threads(self):
-        for i in range(1, 64):
-            with mock.patch(
-                "paperless.settings.multiprocessing.cpu_count",
-            ) as cpu_count:
-                cpu_count.return_value = i
-
-                default_workers = default_task_workers()
-
-                default_threads = default_threads_per_worker(default_workers)
-
-                self.assertTrue(default_workers >= 1)
-                self.assertTrue(default_threads >= 1)
-
-                self.assertTrue(default_workers * default_threads <= i, f"{i}")
diff --git a/src/paperless/tests/test_settings.py b/src/paperless/tests/test_settings.py
--- a/src/paperless/tests/test_settings.py
+++ b/src/paperless/tests/test_settings.py
@@ -1,7 +1,9 @@
 import datetime
+from unittest import mock
 from unittest import TestCase
 
 from paperless.settings import _parse_ignore_dates
+from paperless.settings import default_threads_per_worker
 
 
 class TestIgnoreDateParsing(TestCase):
@@ -56,3 +58,27 @@ def test_single_ignore_dates_set(self):
         ]
 
         self._parse_checker(test_cases)
+
+    def test_workers_threads(self):
+        """
+        GIVEN:
+            - Certain CPU counts
+        WHEN:
+            - Threads per worker is calculated
+        THEN:
+            - Threads per worker less than or equal to CPU count
+            - At least 1 thread per worker
+        """
+        default_workers = 1
+
+        for i in range(1, 64):
+            with mock.patch(
+                "paperless.settings.multiprocessing.cpu_count",
+            ) as cpu_count:
+                cpu_count.return_value = i
+
+                default_threads = default_threads_per_worker(default_workers)
+
+                self.assertGreaterEqual(default_threads, 1)
+
+                self.assertLessEqual(default_workers * default_threads, i)