Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Document consumer process causes permanent 100% CPU usage
### Description

Paperless document consumer causes excessive CPU load. In this screenshot, it consumed 330 hrs CPU time within ~334 hrs of system uptime. This is not related to actually performing document consumption work.
It causes load, even if the system is idle with no documents being uploaded and starts directly after launching paperless-ngx.
![Screenshot_20230920_162142](https://github.com/paperless-ngx/paperless-ngx/assets/2143820/bfa8e4d2-5f03-4deb-a69e-66513c73c707)

```
● paperless-consumer.service - Paperless consumer service
     Loaded: loaded (/usr/lib/systemd/system/paperless-consumer.service; disabled; preset: disabled)
     Active: active (running) since Wed 2023-09-06 17:42:31 CEST; 1 week 6 days ago
   Main PID: 420 (python3)
      Tasks: 13 (limit: 2105)
     Memory: 81.6M
        CPU: 1w 6d 18h 7min 15.826s
     CGroup: /system.slice/paperless-consumer.service
             └─420 python3 /usr/bin/paperless-manage document_consumer

Sep 06 17:42:31 pi3ap systemd[1]: Started Paperless consumer service.
Sep 06 17:43:02 pi3ap paperless-manage[420]: [2023-09-06 15:43:02,635] [INFO] [paperless.management.consumer] Polling directory for changes: /home/paperless/consume
```

Killing the consumer process mitigates the issue for me and seems to not pose any issues. My paperless instance runs on a headless server, where I don't use the document consumption via the consume directory.

### Steps to reproduce

1. Start paperless-ngx service via `systemctl start paperless.target`


### Webserver logs

This might be related to
```
[2023-09-20 15:23:37,910] [INFO] [paperless.management.consumer] Polling directory for changes: /home/paperless/consume
```
if this indicates a zero-delay polling loop.  
Log file: [paperless.log](https://github.com/paperless-ngx/paperless-ngx/files/12675168/paperless.log)

### Browser logs

_No response_

### Paperless-ngx version

1.16.5, 1.17.3

### Host OS

Arch Linux ARM aarch64 on Raspberry Pi 4

### Installation method

Bare metal via Arch AUR package from https://aur.archlinux.org/packages/paperless-ngx

### Browser

_No response_

### Configuration changes

`PAPERLESS_EMAIL_TASK_CRON=disable`. 
Set all data paths to subdirs of `/home/paperless`, which is $HOME of the `paperless` user




Discussion:
Why use polling at all?  I assume a local install would support inotify anyway.
That's the out-of-the-box experience and a good question.

It should use inotify, which is installed. The package has [`python-inotify-simple`](https://archlinux.org/packages/extra/any/python-inotify-simple/) and [`python-inotifyrecursive`](https://aur.archlinux.org/packages/python-inotifyrecursive) packages marked as a hard dependency, and both are installed on the base system.
That's not the default.  To use polling, `PAPERLESS_CONSUMER_POLLING` must be set in the environment.
That's not defined in the environment
```
$ sudo -u paperless env
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
TERM=xterm-256color
LANG=de_DE.UTF-8
LANGUAGE=de_DE:de:en_US:en:C
LS_COLORS=*#=90:*.7z=1;31:*.aac=36:*.ace=1;31:*.alz=1;31:*.arc=1;31:*.arj=1;31:*.asf=1;35:*.au=36:*.avi=1;35:*.avif=1;35:*.bak=90:*.bmp=1;35:*.bz=1;31:*.bz2=1;31:*.cab=1;31:*.cgm=1;35:*.cpio=1;31:*.deb=1;31:*.dl=1;35:*.dpkg-dist=90:*.dpkg-old=90:*.dwm=1;31:*.dz=1;31:*.ear=1;31:*.emf=1;35:*.esd=1;31:*.flac=36:*.flc=1;35:*.fli=1;35:*.flv=1;35:*.gif=1;35:*.gl=1;35:*.gz=1;31:*.jar=1;31:*.jpeg=1;35:*.jpg=1;35:*.lha=1;31:*.lrz=1;31:*.lz=1;31:*.lz4=1;31:*.lzh=1;31:*.lzma=1;31:*.lzo=1;31:*.m2v=1;35:*.m4a=36:*.m4v=1;35:*.mid=36:*.midi=36:*.mjpeg=1;35:*.mjpg=1;35:*.mka=36:*.mkv=1;35:*.mng=1;35:*.mov=1;35:*.mp3=36:*.mp4=1;35:*.mp4v=1;35:*.mpc=36:*.mpeg=1;35:*.mpg=1;35:*.nuv=1;35:*.oga=36:*.ogg=36:*.ogm=1;35:*.ogv=1;35:*.ogx=1;35:*.old=90:*.opus=36:*.orig=90:*.part=90:*.pbm=1;35:*.pcx=1;35:*.pgm=1;35:*.png=1;35:*.ppm=1;35:*.qt=1;35:*.ra=36:*.rar=1;31:*.rej=90:*.rm=1;35:*.rmvb=1;35:*.rpm=1;31:*.rpmnew=90:*.rpmorig=90:*.rpmsave=90:*.rz=1;31:*.sar=1;31:*.spx=36:*.svg=1;35:*.svgz=1;35:*.swm=1;31:*.swp=90:*.t7z=1;31:*.tar=1;31:*.taz=1;31:*.tbz=1;31:*.tbz2=1;31:*.tga=1;35:*.tgz=1;31:*.tif=1;35:*.tiff=1;35:*.tlz=1;31:*.tmp=90:*.txz=1;31:*.tz=1;31:*.tzo=1;31:*.tzst=1;31:*.ucf-dist=90:*.ucf-new=90:*.ucf-old=90:*.vob=1;35:*.war=1;31:*.wav=36:*.webm=1;35:*.webp=1;35:*.wim=1;31:*.wmv=1;35:*.xbm=1;35:*.xcf=1;35:*.xpm=1;35:*.xspf=36:*.xwd=1;35:*.xz=1;31:*.yuv=1;35:*.z=1;31:*.zip=1;31:*.zoo=1;31:*.zst=1;31:*~=90:bd=40;1;33:ca=0:cd=40;1;33:di=1;34:do=1;35:ex=1;32:ln=1;36:mh=0:mi=0:or=40;1;31:ow=34;42:pi=40;33:rs=0:sg=30;43:so=1;35:st=37;44:su=37;41:tw=30;42
MAIL=/var/mail/paperless
LOGNAME=paperless
USER=paperless
HOME=/home/paperless
SHELL=/usr/bin/nologin
SUDO_COMMAND=/usr/bin/env
SUDO_USER=root
SUDO_UID=0
SUDO_GID=0
```
And not in the config file
```
$ grep "PAPERLESS_CONSUMER_POLLING" /etc/paperless.conf
#PAPERLESS_CONSUMER_POLLING=10
```

If you run `python3` as the paperless user, or the equivalent for aur, does `from inotifyrecursive import INotify` and `from inotifyrecursive import flags` work?
Ok, that package was broken. Sorry for the noise.

`ModuleNotFoundError: No module named 'inotifyrecursive'`

I guess this issue could be converted to an enhancement request: It should be logged as a warning, if importing things fail in a non-fatal way and fall-backs are used.

The current fall-back seems to use a zero-delay busy waiting loop, which is sub-optimal at best.

Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -245,6 +245,8 @@ def handle(self, *args, **options):
         if settings.CONSUMER_POLLING == 0 and INotify:
             self.handle_inotify(directory, recursive, options["testing"])
         else:
+            if INotify is None and settings.CONSUMER_POLLING == 0:  # pragma: no cover
+                logger.warn("Using polling as INotify import failed")
             self.handle_polling(directory, recursive, options["testing"])
 
         logger.debug("Consumer exiting.")
@@ -257,8 +259,14 @@ def handle_polling(self, directory, recursive, is_testing: bool):
             timeout = self.testing_timeout_s
             logger.debug(f"Configuring timeout to {timeout}s")
 
+        polling_interval = settings.CONSUMER_POLLING
+        if polling_interval == 0:  # pragma: no cover
+            # Only happens if INotify failed to import
+            logger.warn("Using polling of 10s, consider settng this")
+            polling_interval = 10
+
         with ThreadPoolExecutor(max_workers=4) as pool:
-            observer = PollingObserver(timeout=settings.CONSUMER_POLLING)
+            observer = PollingObserver(timeout=polling_interval)
             observer.schedule(Handler(pool), directory, recursive=recursive)
             observer.start()
             try:


Test patch: