Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] multiple consumer working on file when using consume folder as network share


**Describe the bug**
If i share the consume folder to my network and fill in files from a network attached computer i the files were consumed multiple times including errors

**To Reproduce**
Steps to reproduce the behavior:
1. make /consume/ a network shared folder
2. fill in files from network
3. files are consumed multiple times

**Expected behavior**
i was expecting to have just one file workin on

**logs**
```
[2022-03-22 10:51:20,386] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,393] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,573] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,586] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,593] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/src/../consume/Test.pdf to the task queue.

[2022-03-22 10:51:20,598] [INFO] [paperless.consumer] Consuming Test.pdf

[2022-03-22 10:51:20,603] [DEBUG] [paperless.consumer] Detected mime type: application/pdf

[2022-03-22 10:51:20,614] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser

[2022-03-22 10:51:20,620] [DEBUG] [paperless.consumer] Parsing Test.pdf...

[2022-03-22 10:51:20,621] [INFO] [paperless.consumer] Consuming Test.pdf

[2022-03-22 10:51:20,625] [DEBUG] [paperless.consumer] Detected mime type: application/pdf

[2022-03-22 10:51:20,637] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser

[2022-03-22 10:51:20,642] [DEBUG] [paperless.consumer] Parsing Test.pdf...

[2022-03-22 10:51:22,008] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:51:22,029] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:51:22,194] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-z04o2u17/archive.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-z04o2u17/sidecar.txt'}

[2022-03-22 10:51:22,213] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-2a786d6a/archive.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-2a786d6a/sidecar.txt'}

[2022-03-22 10:52:08,511] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:08,512] [WARNING] [paperless.parsing.tesseract] Encountered an error while running OCR: No text was found in the original document. Attempting force OCR to get the text.

[2022-03-22 10:52:08,512] [DEBUG] [paperless.parsing.tesseract] Fallback: Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-2a786d6a/archive-fallback.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'force_ocr': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-2a786d6a/sidecar-fallback.txt'}

[2022-03-22 10:52:08,599] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:08,600] [WARNING] [paperless.parsing.tesseract] Encountered an error while running OCR: No text was found in the original document. Attempting force OCR to get the text.

[2022-03-22 10:52:08,600] [DEBUG] [paperless.parsing.tesseract] Fallback: Calling OCRmyPDF with args: {'input_file': '/usr/src/paperless/src/../consume/Test.pdf', 'output_file': '/tmp/paperless/paperless-z04o2u17/archive-fallback.pdf', 'use_threads': True, 'jobs': 2, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'force_ocr': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-z04o2u17/sidecar-fallback.txt'}

[2022-03-22 10:52:54,579] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:54,580] [WARNING] [paperless.parsing.tesseract] No text was found in /usr/src/paperless/src/../consume/Test.pdf, the content will be empty.

[2022-03-22 10:52:54,580] [DEBUG] [paperless.consumer] Generating thumbnail for Test.pdf...

[2022-03-22 10:52:54,587] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-2a786d6a/archive.pdf[0] /tmp/paperless/paperless-2a786d6a/convert.png

[2022-03-22 10:52:54,746] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file

[2022-03-22 10:52:54,747] [WARNING] [paperless.parsing.tesseract] No text was found in /usr/src/paperless/src/../consume/Test.pdf, the content will be empty.

[2022-03-22 10:52:54,747] [DEBUG] [paperless.consumer] Generating thumbnail for Test.pdf...

[2022-03-22 10:52:54,753] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-z04o2u17/archive.pdf[0] /tmp/paperless/paperless-z04o2u17/convert.png

[2022-03-22 10:52:56,759] [DEBUG] [paperless.parsing.tesseract] Execute: optipng -silent -o5 /tmp/paperless/paperless-2a786d6a/convert.png -out /tmp/paperless/paperless-2a786d6a/thumb_optipng.png

[2022-03-22 10:52:56,925] [DEBUG] [paperless.parsing.tesseract] Execute: optipng -silent -o5 /tmp/paperless/paperless-z04o2u17/convert.png -out /tmp/paperless/paperless-z04o2u17/thumb_optipng.png

[2022-03-22 10:53:07,858] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.

[2022-03-22 10:53:07,864] [DEBUG] [paperless.consumer] Saving record to database

[2022-03-22 10:53:07,968] [DEBUG] [paperless.consumer] Deleting file /usr/src/paperless/src/../consume/Test.pdf

[2022-03-22 10:53:07,992] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-2a786d6a

[2022-03-22 10:53:07,993] [INFO] [paperless.consumer] Document 2022-02-23 Test consumption finished

[2022-03-22 10:53:08,037] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.

[2022-03-22 10:53:08,050] [ERROR] [paperless.consumer] The following error occured while consuming Test.pdf: [Errno 2] No such file or directory: '/usr/src/paperless/src/../consume/Test.pdf'

Traceback (most recent call last):

  File "/usr/src/paperless/src/documents/consumer.py", line 283, in try_consume_file

    document = self._store(text=text, date=date, mime_type=mime_type)

  File "/usr/src/paperless/src/documents/consumer.py", line 361, in _store

    stats = os.stat(self.path)

FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/src/../consume/Test.pdf'

[2022-03-22 10:53:08,051] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-z04o2u17

[2022-03-22 10:53:08,635] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.

[2022-03-22 10:53:09,130] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.

[2022-03-22 10:53:09,644] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/src/../consume/Test.pdf: File not found.
```

**Relevant information**
 - Host OS of the machine running paperless: [e.g. Synology NAS / docker]
 - Browser [Chrome]
 - Version [latest]
 - Installation method: [docker]



Discussion:
`ionotify` listens on two different flags for changes on files in the customer folder: https://github.com/paperless-ngx/paperless-ngx/blob/main/src/documents/management/commands/document_consumer.py#L177

It seems that both triggered a consuming task multiple times.

@paperless-ngx/backend maybe it is possible to add some kind of destictUntilChanged or debounce (e.g. with https://ninmesara.github.io/RxPY/api/operators/distinct.html#rx.Observable.distinct_until_changed) to the consumer to handle this behavior. This can also reduce the attempts for consuming a busy file (#432)
Thanks @FrankStrieter for pointing it out.. My actual workaround is to use a POLLING_INTERVAL with 60, but a fix to use ionotify again would be great!
With 1.7.0, there is now some additional checking and waiting around file consumption for both inotify and polling based setups.  These changes should resolve the issue.
Hm... is it only me or are others still seeing this issue on 1.7.0?
> Hm... is it only me or are others still seeing this issue on 1.7.0?

If you're still experiencing this issue, could you provide some logs of when it occurs?
Here's my log:

```
[2022-05-06 13:56:29,162] [INFO] [paperless.management.consumer] Using inotify to watch directory for changes: /scan
[2022-05-06 14:02:05,634] [DEBUG] [paperless.management.consumer] Not consuming file /scan/tmp.txt: File has moved.
[2022-05-06 14:02:31,200] [INFO] [paperless.management.consumer] Adding /scan/SCN_0001.pdf to the task queue.
[2022-05-06 14:02:32,217] [INFO] [paperless.consumer] Consuming SCN_0001.pdf
[2022-05-06 14:02:32,220] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2022-05-06 14:02:32,240] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-05-06 14:02:32,246] [DEBUG] [paperless.consumer] Parsing SCN_0001.pdf...
[2022-05-06 14:02:32,395] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /scan/SCN_0001.pdf
[2022-05-06 14:02:33,288] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/scan/SCN_0001.pdf', 'output_file': '/tmp/paperless/paperless-200d20f8/archive.pdf', 'use_threads': True, 'jobs': 1, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-200d20f8/sidecar.txt'}
[2022-05-06 14:02:37,603] [INFO] [paperless.management.consumer] Adding /scan/SCN_0001.pdf to the task queue.
[2022-05-06 14:02:37,977] [INFO] [paperless.consumer] Consuming SCN_0001.pdf
[2022-05-06 14:02:37,980] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2022-05-06 14:02:37,984] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-05-06 14:02:37,990] [DEBUG] [paperless.consumer] Parsing SCN_0001.pdf...
[2022-05-06 14:02:38,066] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /scan/SCN_0001.pdf
[2022-05-06 14:02:38,242] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/scan/SCN_0001.pdf', 'output_file': '/tmp/paperless/paperless-4vhiqh3m/archive.pdf', 'use_threads': True, 'jobs': 1, 'language': 'eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': '/tmp/paperless/paperless-4vhiqh3m/sidecar.txt'}
[2022-05-06 14:02:54,424] [DEBUG] [paperless.parsing.tesseract] Using text from sidecar file
[2022-05-06 14:02:54,425] [DEBUG] [paperless.consumer] Generating thumbnail for SCN_0001.pdf...
[2022-05-06 14:02:54,440] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-200d20f8/archive.pdf[0] /tmp/paperless/paperless-200d20f8/convert.png
[2022-05-06 14:02:56,967] [DEBUG] [paperless.parsing.tesseract] Execute: optipng -silent -o5 /tmp/paperless/paperless-200d20f8/convert.png -out /tmp/paperless/paperless-200d20f8/thumb_optipng.png
[2022-05-06 14:03:05,911] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.
[2022-05-06 14:03:05,918] [DEBUG] [paperless.consumer] Saving record to database
[2022-05-06 14:03:06,038] [DEBUG] [paperless.consumer] Deleting file /scan/SCN_0001.pdf
[2022-05-06 14:03:06,419] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-200d20f8
[2022-05-06 14:03:06,421] [INFO] [paperless.consumer] Document 2022-03-25 SCN_0001 consumption finished
[2022-05-06 14:03:09,332] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-4vhiqh3m
[2022-05-06 14:03:09,341] [ERROR] [paperless.consumer] Error while consuming document SCN_0001.pdf: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ocrmypdf.io.yhk3zbv0/origin.pdf'
Traceback (most recent call last):
  File "/app/paperless/src/paperless_tesseract/parsers.py", line 261, in parse
    ocrmypdf.ocr(**args)
  File "/usr/local/lib/python3.8/dist-packages/ocrmypdf/api.py", line 337, in ocr
    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)
  File "/usr/local/lib/python3.8/dist-packages/ocrmypdf/_sync.py", line 385, in run_pipeline
    exec_concurrent(context, executor)
  File "/usr/local/lib/python3.8/dist-packages/ocrmypdf/_sync.py", line 302, in exec_concurrent
    pdf = post_process(pdf, context, executor)
  File "/usr/local/lib/python3.8/dist-packages/ocrmypdf/_sync.py", line 235, in post_process
    pdf_out = metadata_fixup(pdf_out, context)
  File "/usr/local/lib/python3.8/dist-packages/ocrmypdf/_pipeline.py", line 798, in metadata_fixup
    with pikepdf.open(context.origin) as original, pikepdf.open(working_file) as pdf:
  File "/usr/local/lib/python3.8/dist-packages/pikepdf/_methods.py", line 923, in open
    pdf = Pdf._open(
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ocrmypdf.io.yhk3zbv0/origin.pdf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/paperless/src/documents/consumer.py", line 261, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/app/paperless/src/paperless_tesseract/parsers.py", line 314, in parse
    raise ParseError(f"{e.__class__.__name__}: {str(e)}")
documents.parsers.ParseError: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ocrmypdf.io.yhk3zbv0/origin.pdf'
[2022-05-06 14:06:06,824] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.
[2022-05-06 14:06:29,897] [DEBUG] [paperless.classifier] Document classification model does not exist (yet), not performing automatic matching.
```
```
[2022-05-06 14:02:31,200] [INFO] [paperless.management.consumer] Adding /scan/SCN_0001.pdf to the task queue.
...
[2022-05-06 14:02:37,603] [INFO] [paperless.management.consumer] Adding /scan/SCN_0001.pdf to the task queue.
````

So we would have gotten 2 inotify events here, ~6s apart.  That's much more than I expected would be ever seen.  If possible, could you confirm this directly using the below?

`inotifywait --monitor -e moved_to -e close_write --timefmt '%Y-%m-%dT%H:%M:%S' --format '%T %w %f %e' <actual scan folder here>`


OK, what I did was first copying SCN_2022_05_06_15_36_33_00.pdf to the network share via a Windows PC.
After consuming was finished I scanned a document and let my scanner copy the document (SCN_0001.pdf) via CIFS to the SMB folder, indeed inotifywait noticed two events for the same file.

```
root@linuxserver-paperless-ngx:/# inotifywait --monitor -e moved_to -e close_write --timefmt '%Y-%m-%dT%H:%M:%S' --format 
'%T %w %f %e' /scan                                                                                                       
Setting up watches.                                                                                                       
Watches established.                                                                                                      
2022-05-06T16:38:32 /scan/ SCN_2022_05_06_15_36_33_00.pdf CLOSE_WRITE,CLOSE                                               
2022-05-06T16:41:38 /scan/ SCN_0001.pdf CLOSE_WRITE,CLOSE                                                                 
2022-05-06T16:41:45 /scan/ SCN_0001.pdf CLOSE_WRITE,CLOSE 
```
Roughly 7 seconds that time.  I wonder if some scanners create the file, then scan to it, before finally closing again.

I guess we could make it configurable how long to wait.  I'll have to think if there's other solutions to such a delay.

Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -216,7 +216,7 @@ def handle_inotify(self, directory, recursive):
 
         try:
 
-            inotify_debounce: Final[float] = 0.5
+            inotify_debounce: Final[float] = settings.CONSUMER_INOTIFY_DELAY
             notified_files = {}
 
             while not self.stop_flag:
@@ -234,10 +234,23 @@ def handle_inotify(self, directory, recursive):
                 for filepath in notified_files:
                     # Time of the last inotify event for this file
                     last_event_time = notified_files[filepath]
-                    if (monotonic() - last_event_time) > inotify_debounce:
+
+                    # Current time - last time over the configured timeout
+                    waited_long_enough = (
+                        monotonic() - last_event_time
+                    ) > inotify_debounce
+
+                    # Also make sure the file exists still, some scanners might write a
+                    # temporary file first
+                    file_still_exists = os.path.exists(filepath) and os.path.isfile(
+                        filepath,
+                    )
+
+                    if waited_long_enough and file_still_exists:
                         _consume(filepath)
-                    else:
+                    elif file_still_exists:
                         still_waiting[filepath] = last_event_time
+
                 # These files are still waiting to hit the timeout
                 notified_files = still_waiting
 
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -48,6 +48,13 @@ def __get_int(key: str, default: int) -> int:
     return int(os.getenv(key, default))
 
 
+def __get_float(key: str, default: float) -> float:
+    """
+    Return an integer value based on the environment variable or a default
+    """
+    return float(os.getenv(key, default))
+
+
 # NEVER RUN WITH DEBUG IN PRODUCTION.
 DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO")
 
@@ -485,6 +492,11 @@ def default_threads_per_worker(task_workers) -> int:
     os.getenv("PAPERLESS_CONSUMER_POLLING_RETRY_COUNT", 5),
 )
 
+CONSUMER_INOTIFY_DELAY: Final[float] = __get_float(
+    "PAPERLESS_CONSUMER_INOTIFY_DELAY",
+    0.5,
+)
+
 CONSUMER_DELETE_DUPLICATES = __get_boolean("PAPERLESS_CONSUMER_DELETE_DUPLICATES")
 
 CONSUMER_RECURSIVE = __get_boolean("PAPERLESS_CONSUMER_RECURSIVE")


Test patch:
diff --git a/src/documents/tests/test_management_consumer.py b/src/documents/tests/test_management_consumer.py
--- a/src/documents/tests/test_management_consumer.py
+++ b/src/documents/tests/test_management_consumer.py
@@ -98,6 +98,9 @@ def slow_write_file(self, target, incomplete=False):
             print("file completed.")
 
 
+@override_settings(
+    CONSUMER_INOTIFY_DELAY=0.01,
+)
 class TestConsumer(DirectoriesMixin, ConsumerMixin, TransactionTestCase):
     def test_consume_file(self):
         self.t_start()
@@ -286,7 +289,7 @@ class TestConsumerPolling(TestConsumer):
     pass
 
 
-@override_settings(CONSUMER_RECURSIVE=True)
+@override_settings(CONSUMER_INOTIFY_DELAY=0.01, CONSUMER_RECURSIVE=True)
 class TestConsumerRecursive(TestConsumer):
     # just do all the tests with recursive
     pass