Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Some documents fail to be consumed due to "Device or resource busy"
Hello,

i'm a really fresh user to paperless and installed it just some days ago via docker-compose. 
I would like to reference an issue from the before repository from jonas because i think i have the same problem:
https://github.com/jonaswinkler/paperless-ng/issues/1406

My scanner is an HP OfficeJet Pro 9022e with smb-scan function. And mostly files that are still not finished written gets queued from paperless but the job run into a `[Errno 16] Device or resource busy: `failure. 
This happens for example with files where i add pages manually by scanning and so the files are still handled by the scanner.

In Logs it looks like this:
```
[2022-03-18 14:00:43,759] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/scan_20220318_20220318135855.pdf to the task queue.
14:00:43 [Q] INFO Enqueued 1
14:00:43 [Q] INFO Process-1:174 processing [scan_20220318_20220318135855.pdf]
14:00:43 [Q] INFO Process-1:174 stopped doing work
14:00:43 [Q] ERROR Failed [scan_20220318_20220318135855.pdf] - [Errno 16] Device or resource busy: '/usr/src/paperless/src/../consume/scan_20220318_20220318135855.pdf' : Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django_q/cluster.py", line 432, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 70, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 199, in try_consume_file
    self.pre_check_duplicate()
  File "/usr/src/paperless/src/documents/consumer.py", line 90, in pre_check_duplicate
    with open(self.path, "rb") as f:
OSError: [Errno 16] Device or resource busy: '/usr/src/paperless/src/../consume/scan_20220318_20220318135855.pdf'

14:00:44 [Q] INFO recycled worker Process-1:174
14:00:44 [Q] INFO Process-1:176 ready for work at 24155
```

What i've done already was to define `PAPERLESS_CONSUMER_POLLING: 60` via docker-compose which canceled some warnings in paperless-log that i had before but didn't solved the main problem.

To solve the problem in specific case is to go to admin-page and restart the failed jobs 
or the more brutal way: to restart the paperless-container.

**Relevant information**
 - Host OS of the machine running paperless: Ubuntu 20.04 LTS
 - Browser not necessary
 - Version: 1.6.0
 - Installation method: docker-compose
```
version: "3.4"
services:
  paperless-redis:
    image: redis:alpine
    container_name: paperless-redis
    restart: unless-stopped
    networks:
      - my-bridge
    volumes:
      - paperless-redis_data:/data

  paperless-db:
    image: postgres:13-alpine
    container_name: paperless-db
    restart: unless-stopped
    networks:
      - my-bridge
    volumes:
      - paperless-db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: ...
      POSTGRES_PASSWORD: ...

  paperless-app:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    container_name: paperless-app
    restart: unless-stopped
    depends_on:
      - paperless-redis
      - paperless-db
    ports:
      - 8010:8000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    networks:
      - my-bridge
    volumes:
      - paperless_data:/usr/src/paperless/data
      - paperless_media:/usr/src/paperless/media
      - paperless_export:/usr/src/paperless/export
      - paperless_incoming:/usr/src/paperless/consume
    environment:
      PAPERLESS_REDIS: redis://paperless-redis:6379
      PAPERLESS_DBHOST: paperless-db
      USERMAP_UID: 1000
      USERMAP_GID: 100
      #PAPERLESS_OCR_LANGUAGES: tur ces
      PAPERLESS_SECRET_KEY: ...
      PAPERLESS_TIME_ZONE: Europe/Berlin
      PAPERLESS_OCR_LANGUAGE: deu
      PAPERLESS_FILENAME_FORMAT: "{created_year}/{correspondent}/{created}_{title}"
      PAPERLESS_CONSUMER_POLLING: 60

volumes:
  paperless_data:
    external: true
  paperless_media:
    external: true
  paperless_incoming:
    external: true
  paperless_export:
    external: true
  paperless-db_data:
    external: true
  paperless-redis_data:
    external: true

networks:
  my-bridge:
    external: true
```

[BUG] multiple consumer working on file when using consume folder as network share


**Describe the bug**
If i share the consume folder to my network and fill in files from a network attached computer i the files were consumed multiple times including errors

**To Reproduce**
Steps to reproduce the behavior:
1. make /consume/ a network shared folder
2. fill in files from network
3. files are consumed multiple times

**Expected behavior**
i was expecting to have just one file workin on

**logs**
```
[2022-03-22 10:51:20,386] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,393] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,573] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,586] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,593] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,598] [INFO] [paperless.consumer] Consuming Test.pdf

[2022-03-22 10:51:20,603] [DEBUG] [paperless.consumer] Detected mime type: application/pdf

[2022-03-22 10:51:20,614] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser

[2022-03-22 10:51:20,620] [DEBUG] [paperless.consumer] Parsing Test.pdf...

[2022-03-22 10:51:20,621] [INFO] [paperless.consumer] Consuming Test.pdf

[2022-03-22 10:51:20,625] [DEBUG] [paperless.consumer] Detected mime type: application/pdf

[2022-03-22 10:51:20,637] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser

[2022-03-22 10:51:20,642] [DEBUG] [paperless.consumer] Parsing Test.pdf...

[2022-03-22 10:51:22,008] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:51:22,029] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:51:22,194] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-z04o2u17/archive.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-z04o2u17/sidecar.txt'}

[2022-03-22 10:51:22,213] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-2a786d6a/archive.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-2a786d6a/sidecar.txt'}

[2022-03-22 10:52:08,511] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:08,512] [WARNING] [paperless.parsing.tesseract] Encountered an error while running OCR: No text was found in the original document. Attempting force OCR to get the text.

[2022-03-22 10:52:08,512] [DEBUG] [paperless.parsing.tesseract] Fallback: Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-2a786d6a/archive-fallback.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'force_ocr': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-2a786d6a/sidecar-fallback.txt'}

[2022-03-22 10:52:08,599] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:08,600] [WARNING] [paperless.parsing.tesseract] Encountered an error while running OCR: No text was found in the original document. Attempting force OCR to get the text.

[2022-03-22 10:52:08,600] [DEBUG] [paperless.parsing.tesseract] Fallback: Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-z04o2u17/archive-fallback.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'force_ocr': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-z04o2u17/sidecar-fallback.txt'}

[2022-03-22 10:52:54,579] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:54,580] [WARNING] [paperless.parsing.tesseract] No text was found in /usr/src/paperless/src/../consume/Test.pdf, the content will be empty.

[2022-03-22 10:52:54,580] [DEBUG] [paperless.consumer] Generating thumbnail for Test.pdf...

[2022-03-22 10:52:54,587] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-2a786d6a/archive.pdf[0] /tmp/paperless/paperless-2a786d6a/convert.png

[2022-03-22 10:52:54,746] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:54,747] [WARNING] [paperless.parsing.tesseract] No text was found in /usr/src/paperless/src/../consume/Test.pdf, the content will be empty.

[2022-03-22 10:52:54,747] [DEBUG] [paperless.consumer] Generating thumbnail for Test.pdf...

[2022-03-22 10:52:54,753] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-z04o2u17/archive.pdf[0] /tmp/paperless/paperless-z04o2u17/convert.png

[2022-03-22 10:52:56,759] [DEBUG] [paperless.parsing.tesseract] Execute: optipng -silent -o5 /tmp/paperless/paperless-2a786d6a/convert.png -out /tmp/paperless/paperless-2a786d6a/thumb_optipng.png

[2022-03-22 10:52:56,925] [DEBUG] [paperless.parsing.tesseract] Execute: optipng -silent -o5 /tmp/paperless/paperless-z04o2u17/convert.png -out /tmp/paperless/paperless-z04o2u17/thumb_optipng.png

[2022-03-22 10:53:07,858] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.

[2022-03-22 10:53:07,864] [DEBUG] [paperless.consumer] Saving record to database

[2022-03-22 10:53:07,968] [DEBUG] [paperless.consumer] Deleting file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:53:07,992] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-2a786d6a

[2022-03-22 10:53:07,993] [INFO] [paperless.consumer] Document 2022-02-23 Test consumption finished

[2022-03-22 10:53:08,037] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.

[2022-03-22 10:53:08,050] [ERROR] [paperless.consumer] The following error occured while consuming Test.pdf: [Errno 2] No such file or directory: '/usr/src/paperless/src/../consume/Test.pdf'

Traceback (most recent call last):

  File "/usr/src/paperless/src/documents/consumer.py", line 283, in try_consume_file

    document = self._store(text=text, date=date, mime_type=mime_type)

  File "/usr/src/paperless/src/documents/consumer.py", line 361, in _store

    stats = os.stat(self.path)

FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/src/../consume/Test.pdf'

[2022-03-22 10:53:08,051] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-z04o2u17

[2022-03-22 10:53:08,635] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.

[2022-03-22 10:53:09,130] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.

[2022-03-22 10:53:09,644] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.
```

**Relevant information**
 - Host OS of the machine running paperless: [e.g. Synology NAS / docker]
 - Browser [Chrome]
 - Version [latest]
 - Installation method: [docker]



Discussion:
This came up a lot in -ng, I think. Is the issue that the scanner isnt "done" creating documents or are these docs that have already been completely scanned and saved?

Maybe duplicate of #431
I think the issue is that the scanner isn't ready with the complete scan process when the files appears in the smb-share.
I also think that the scanner completely does the job inside the share instead doing everthing localy and copying the finished file to the share.

So thats why inotify also does not work.

I think it is not a problem at all that paperless can't process the uncompleted file but paperless should start a new try after a while or pause the cue for some seconds and restart then. Or it should start a defined number of retrys until it stops to import the specific file at all. At least in case of Error 16...

Does that help?
If needed i could do some tests in any verbose mode if you tell me how or what to do.

Thank!


Oh yes and it looks like that the issue is duplicate of #431
I wonder if adding some sort of check either of mtime vs crime or maybe st_size stops changing would alleviate this?

Basically some way to wait until a file is "done" before really turning the consumer loose on it.  Or maybe the inotify flags could be used. I'll have to look at it.
We can also look at the error message. If the file is busy (Error 16 from the python file handler), reschedule the file in 1 Minute or so. I think there will be other use cases for this too. For Example, somebody creates an office document and saves it into the customer folder but leaves the application running. This will create a busy file too.

EDIT: Now I have seen that https://github.com/paperless-ngx/paperless-ngx/issues/432#issuecomment-1072699418 suggests the same 🙈
If the file is still busy being written, paperless should anyway receive another inotify event when the next write occurs or it's closed.

So maybe it would be enough to ignore the file as long as it is reporting as busy.


Btw. Had the same issue with the original paperless and an hp scanner...
i made an inotify-log today to show you what the scanner is doing:
```
19:12:07 scan_20220320_20220320191034.pdf CREATE
19:12:07 scan_20220320_20220320191034.pdf OPEN
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY
19:12:08 scan_20220320_20220320191034.pdf MODIFY

19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY
19:12:35 scan_20220320_20220320191034.pdf MODIFY

19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:12 scan_20220320_20220320191034.pdf MODIFY
19:13:13 scan_20220320_20220320191034.pdf MODIFY
19:13:13 scan_20220320_20220320191034.pdf MODIFY

19:13:28 scan_20220320_20220320191034.pdf MODIFY
19:13:28 scan_20220320_20220320191034.pdf CLOSE_WRITE,CLOSE
19:13:28 scan_20220320_20220320191034.pdf ATTRIB
```
In this case is scanned 3 pages manually. Be aware of the times.
Maybe that helps to improve the process. 

The problem is that i think it is not possible for me to use paperless with inotify. I tryied to do the log on the paperless-driving ubuntu-machine that maps the comsume-folder by smb-cifs and nothing was logged. So i made this log above on the NAS where the share is hosted.
@paperless-ngx/backend does somebody of you have an idea, of how to solve this issue with all its edge cases? From my Point of view, the idea with the rescheduling if it fails with error code 16 can be tested.


Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -3,7 +3,9 @@
 from pathlib import Path
 from pathlib import PurePath
 from threading import Thread
+from time import monotonic
 from time import sleep
+from typing import Final
 
 from django.conf import settings
 from django.core.management.base import BaseCommand
@@ -53,6 +55,25 @@ def _consume(filepath):
         logger.warning(f"Not consuming file {filepath}: Unknown file extension.")
         return
 
+    # Total wait time: up to 500ms
+    os_error_retry_count: Final[int] = 50
+    os_error_retry_wait: Final[float] = 0.01
+
+    read_try_count = 0
+    file_open_ok = False
+
+    while (read_try_count < os_error_retry_count) and not file_open_ok:
+        try:
+            with open(filepath, "rb"):
+                file_open_ok = True
+        except OSError:
+            read_try_count += 1
+            sleep(os_error_retry_wait)
+
+    if read_try_count >= os_error_retry_count:
+        logger.warning(f"Not consuming file {filepath}: OS reports file as busy still")
+        return
+
     tag_ids = None
     try:
         if settings.CONSUMER_SUBDIRS_AS_TAGS:
@@ -81,19 +102,23 @@ def _consume_wait_unmodified(file):
 
     logger.debug(f"Waiting for file {file} to remain unmodified")
     mtime = -1
+    size = -1
     current_try = 0
     while current_try < settings.CONSUMER_POLLING_RETRY_COUNT:
         try:
-            new_mtime = os.stat(file).st_mtime
+            stat_data = os.stat(file)
+            new_mtime = stat_data.st_mtime
+            new_size = stat_data.st_size
         except FileNotFoundError:
             logger.debug(
                 f"File {file} moved while waiting for it to remain " f"unmodified.",
             )
             return
-        if new_mtime == mtime:
+        if new_mtime == mtime and new_size == size:
             _consume(file)
             return
         mtime = new_mtime
+        size = new_size
         sleep(settings.CONSUMER_POLLING_DELAY)
         current_try += 1
 
@@ -182,14 +207,32 @@ def handle_inotify(self, directory, recursive):
             descriptor = inotify.add_watch(directory, inotify_flags)
 
         try:
+
+            inotify_debounce: Final[float] = 0.5
+            notified_files = {}
+
             while not self.stop_flag:
+
                 for event in inotify.read(timeout=1000):
                     if recursive:
                         path = inotify.get_path(event.wd)
                     else:
                         path = directory
                     filepath = os.path.join(path, event.name)
-                    _consume(filepath)
+                    notified_files[filepath] = monotonic()
+
+                # Check the files against the timeout
+                still_waiting = {}
+                for filepath in notified_files:
+                    # Time of the last inotify event for this file
+                    last_event_time = notified_files[filepath]
+                    if (monotonic() - last_event_time) > inotify_debounce:
+                        _consume(filepath)
+                    else:
+                        still_waiting[filepath] = last_event_time
+                # These files are still waiting to hit the timeout
+                notified_files = still_waiting
+
         except KeyboardInterrupt:
             pass
 


Test patch:
diff --git a/src/documents/tests/test_management_consumer.py b/src/documents/tests/test_management_consumer.py
--- a/src/documents/tests/test_management_consumer.py
+++ b/src/documents/tests/test_management_consumer.py
@@ -260,6 +260,21 @@ def test_is_ignored(self):
                 f'_is_ignored("{file_path}") != {expected_ignored}',
             )
 
+    @mock.patch("documents.management.commands.document_consumer.open")
+    def test_consume_file_busy(self, open_mock):
+
+        # Calling this mock always raises this
+        open_mock.side_effect = OSError
+
+        self.t_start()
+
+        f = os.path.join(self.dirs.consumption_dir, "my_file.pdf")
+        shutil.copy(self.sample_file, f)
+
+        self.wait_for_task_mock_call()
+
+        self.task_mock.assert_not_called()
+
 
 @override_settings(
     CONSUMER_POLLING=1,