Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Constant moderate CPU usage when idle (aka. paperless is contributing to global warming)
### Description

paperless ngx relies on redis to move data between the processes. The problem is that somehow something somewhere is not configured correctly, resulting on a permanent 1-3% CPU usage (see below)

It may seem not much but given that
- it is idling, as in, not doing anything meaningful, so CPU usage should be 0%
- it is using CPU on everyone's instances all the time (thousands and thousands instances)
- multiply these two together (instanceCount * Watts), and we are wasting useful energy to do... absolutely nothing (or contribute to climate change, depending how we obtain the energy to do nothing)

### Steps to reproduce

Start up the docker image `linuxserver/paperless-ngx` with the embedded redis.

This is the typical usage

```
top - 06:49:06 up 39 days,  7:55,  0 users,  load average: 0.01, 0.02, 0.01
Tasks:  19 total,   1 running,  18 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us,  1.0 sy,  0.0 ni, 97.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3820.6 total,    644.3 free,   1353.6 used,   1822.7 buff/cache
MiB Swap:   2048.0 total,   1244.2 free,    803.8 used.   2175.0 avail Mem 
  scroll coordinates: y = 1/19 (tasks), x = 1/12 (fields)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                
    401 abc       20   0   50188   4204   3400 S   0.7   0.1   1:42.51 redis-server                                                                                                           
    459 abc       20   0  248844  62192   8032 S   0.7   1.6   2:02.26 python3                                                                                                                
    405 abc       20   0  181140  65540  13416 S   0.3   1.7   0:06.16 python3                                                                                                                
    434 abc       20   0 1091836 128732  22712 S   0.3   3.3   1:32.53 gunicorn                                                                                                               
    435 abc       20   0 1161544 140780  24080 S   0.3   3.6   1:44.55 gunicorn                                                                                                               
   2891 root      20   0    9124   3688   3180 R   0.3   0.1   0:01.57 top                                                                                                                    
      1 root      20   0     196      0      0 S   0.0   0.0   0:00.07 s6-svscan                                                                                                              
     35 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-supervise                                                                                                           
    396 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-supervise                                                                                                           
    397 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-supervise                                                                                                           
    398 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-supervise                                                                                                           
    399 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-supervise            
```

### Webserver logs

_No response_

### Paperless-ngx version

1.7.1

### Host OS

Ubuntu

### Installation method

Docker

### Browser

None

### Configuration changes

_No response_

### Other

_No response_


Discussion:
It seems like it's related to this https://github.com/Koed00/django-q/issues/359 bug, which, by default, causes Django-Q to hammer Redis every 0.5s.

```
1654246140.654145 [0 127.0.0.1:47442] "BLPOP" "django_q:paperless:q" "1"
1654246141.058557 [0 127.0.0.1:47440] "SET" "django_q:paperless:cluster:762b58c7-fd8e-441e-8d75-9ce080f49354" ".eJxVUb9PWzEQztNLk9eEXxWVysScqlJUBApiYYElQm8pRWJBT06eS0wc-2GfW4hUCYa0UnXtQK5b_x8GJNiZmDp36IqEnQgJbJ18nz-d777PZy9-_4tKk4Xz-RFThzo7blpg4CxhedcnNKa35_SVGlj9ok2fG0sH1EgH9VTVOcagO4RJzoCDGPCn6Zi2atW_lThqzCbDV9RLMAn3Q60C5-HLAHMugXm8Uwr7O32gkY9vPnDOcKG6zCgGQitLO5sYFyKn9CLCWlc6C9xkHmPZucmxt9fefhwXY6GAfix8HP8_u_vjfi5v_7q_3H_32XYwsVyBUFxSehVh5VHtm6n-pgMhbfOTU93QlknC15lkw9OsMPrkNHOqEN2-DBoauPSsBgxTVk6mJawecgB-EvzDcjv3BaMwVtJxQvruvmFswXgWgtZ67m3JjjMrhtx7geWetkA402LsfWttfXWDt9b8mwOtBGhD6XWEdWC2_6SkUjjb4567iabWemGDgnq16TckK1Hp1vu-GLx1neYDdUTGgA:1nx2zN:TTmAJm639WDkWFBkPjnRiU6xoS8IVYQ1KNyXelLb9Ps" "EX" "3"
1654246141.563877 [0 127.0.0.1:47440] "SET" "django_q:paperless:cluster:762b58c7-fd8e-441e-8d75-9ce080f49354" ".eJxVUU1LXDEU9fHGmeeM_ZAKunI9pTAoypRu3OhG5G2qgpvyyMyLvnQyyTO5qTpQsAsryG0LznXX_9Gf4KKgf8GV6y66FUxmEDThkntyuLn3nJxMXv6LJkYLX-WfmdrX2UHLAgNnCStbPqEhvf1GX6mJtUNtetxY-kTNtN9IVYNjDLpDmOQMOIg-f5oOaa1eu6vGUfNFMpihIsEk3A-0CpyHUwHmXALzeHMi7DP6SKc-vvvAl4YL1WVGMRBaWdpcxbgUOaUXEda70lngJvMYK86Njp2djfXHcTEWCuj89fbw_8ntb_djYf3n_dXuuy-2g4nlCoTiktK_EVYf1c6N9bccCGlbe051Q1smCWczyQbHWWn00XHmVCm6PRk0NHH-WQ0YpqwcTUtY2-cA_Cj4h5WN3BechrGSjhPSd_cNYwvGsxC0NnJvS3aQWTHg3gusFNoC4XSbscX2yvvlD7y94t_sayVAG0qvI2wAs70nJdXS2YJ77iYaW-uF9Usq6uNvSJai5NcfKt4Eb12n9QBztMZh:1nx2zN:PLOuANzXkIhMWmqTasQzI_8iK6OfLapBbsNk0bpIdB0" "EX" "3"
1654246141.660220 [0 127.0.0.1:47442] "BLPOP" "django_q:paperless:q" "1"
1654246142.069069 [0 127.0.0.1:47440] "SET" "django_q:paperless:cluster:762b58c7-fd8e-441e-8d75-9ce080f49354" ".eJxVUc9LHDEUdjrb3XG3PxSFerLXlcLSUtnixYteRObSKniRIbsTnXSzyZi8tLog6EGF8uqh-7z5_3gQ9F_w5NmD10KTXQSb8Mj78vHy3vfl6OXFQzQxWvg2_87Urs72WhYYOEtY-eYTGtLCMR1SE2s_telxY2mbmmm_kaoGxxh0hzDJGXAQff48HdJKvXZfjaPm62QwTUWCSbgfaBU4DycDzLkE5vH6RNhn9JVOfJz6wDeGC9VlRjEQWllaX8a4FDmlfyKsd6WzwE3mMVacGx2bm2urT-NiLBTQr6mN4ePR3aX7Pb96_vdq68MP28HEcgVCcUnpdYTVJ7XvxvpbDoS0rR2nuqEtk4SzmWSDg6w0ev8gc6oU3Z4MGpo4918NGKasHE1LWNvlAHw_-IeVtdwXnISxko4T0nf3DWMLxrMQtDZyb0u2l1kx4N4LrBTaAuGrNmMf24tfPi_x9qJ_s6-VAG0ovYmwAcz2npVUS2cL7rnbaGytF9YvqaiPvyH59CKK31MxE7x1ndY_Zk_FNw:1nx2zO:L9xloEIhnSkzzlK-SGweg_YC-hYrODWjXJn0zf9TGWI" "EX" "3"
```

A quick-fix would be to add `guard_cycle` with a higher value to `settings.py` in [Q_CLUSTER](https://github.com/paperless-ngx/paperless-ngx/blob/main/src/paperless/settings.py#L449)

Another is to use a more [lightweight broker instead of Redis](https://django-q.readthedocs.io/en/latest/brokers.html)(quite frankly, this whole Django-Q+Redis setup is a massive overkill, both in terms of development environment and memory requirement when running paperless).

Will keep digging
# The quick fix
Adding a longer `quard_cycle` (from the default 0.2 to 4s) reduced the average load immediately (see below)

I'll create a PR.

![image](https://user-images.githubusercontent.com/3745745/171968242-57a547f5-5b6f-4e51-999d-d1db8c716489.png)

# Also helps

I also reduced the `PAPERLESS_TASK_WORKERS` as each worker checks in to Redis [every second](https://github.com/Koed00/django-q/blob/4f70793cdffecf3863b321de80343066dd5cdddb/django_q/brokers/redis_broker.py#L21), creating unnecessary load. That's another problem with Django-Q - why did they use a blocking call with a 1s timeout instead of longer...? Sigh




Resolving patch:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -458,6 +458,7 @@ def default_task_workers() -> int:
 
 Q_CLUSTER = {
     "name": "paperless",
+    "guard_cycle": 5,
     "catch_up": False,
     "recycle": 1,
     "retry": PAPERLESS_WORKER_RETRY,


Test patch: