Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Polling: File is marked for consumption multiple times, when modified by a PRE_CONSUME_SCRIPT. Additional jobs fail with "Cannot consume ...: File not found."
### Description

When polling is activated and a file gets modified by a PRE_CONSUME_SCRIPT, the modified version of the file gets added to the consume queue too: Now two versions of the file are on the queue, ready to get processed (unsure, if this is also the case for INofify).

This causes the consume queue to
1) When allowing only one consumption job at a time: The second job failing (my case), since the first job already moved the file to its archive location (Cannot consume ...: File not found.)
1) When allowing more consumption jobs at a time: Repeating the error by spawning a third job and so on (this is just a wild guess though).

**Suggested Fix:**
Before applying PRE_CONSUME, already "consume" the file by moving it to a temporary folder, that is not under polling/iNotify observation. Then feed that file to PRE_CONSUME_SCRIPT and run all other paperless-tasks on that moved file.

### Steps to reproduce

- Enable polling for the consumer. Set the interval to 5s
- Add a PRE_CONSUME_SCRIPT to your setup (possibly one, that takes some time > 5s, as this is the case for me)
- Make that script modify the input file (e.g. replacing it by another file via "mv")
- Once the PRE_CONSUME_SCRIPT finished modifying the file and paperless putting it into the system, a second job will spawn and immediately fail with "File not found."

### Webserver logs

```bash
[2023-01-13 23:14:39,481] [DEBUG] [paperless.management.consumer] Waiting for file /usr/src/paperless/consume/2023-01-13-2313.pdf to remain unmodified
[2023-01-13 23:14:44,486] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/2023-01-13-2313.pdf to the task queue.
[2023-01-13 23:14:44,606] [INFO] [paperless.consumer] Consuming 2023-01-13-2313.pdf
[2023-01-13 23:14:44,614] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2023-01-13 23:14:44,615] [INFO] [paperless.consumer] Executing pre-consume script /scripts/pre_consume.sh
[2023-01-13 23:15:25,210] [INFO] [paperless.consumer] /scripts/pre_consume.sh exited 0
[2023-01-13 23:15:25,211] [INFO] [paperless.consumer] Script stdout:
[2023-01-13 23:15:25,211] [INFO] [paperless.consumer] PRE_CONSUME: (PDF) Extract PDF->PPM
[2023-01-13 23:15:25,211] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-4.ppm
[2023-01-13 23:15:25,211] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-4.ppm
[2023-01-13 23:15:25,212] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-4.ppm.merged -> /tmp/pdfimg-5708/im-4.ppm.unpaper
[2023-01-13 23:15:25,212] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-4.ppm: 99.8457
[2023-01-13 23:15:25,212] [INFO] [paperless.consumer] PRE_CONSUME: Discarding blank page /tmp/pdfimg-5708/im-4.ppm
[2023-01-13 23:15:25,212] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-6.ppm
[2023-01-13 23:15:25,213] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-6.ppm
[2023-01-13 23:15:25,213] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-6.ppm.merged -> /tmp/pdfimg-5708/im-6.ppm.unpaper
[2023-01-13 23:15:25,213] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-6.ppm: 99.8246
[2023-01-13 23:15:25,213] [INFO] [paperless.consumer] PRE_CONSUME: Discarding blank page /tmp/pdfimg-5708/im-6.ppm
[2023-01-13 23:15:25,214] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-5.ppm
[2023-01-13 23:15:25,214] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-5.ppm
[2023-01-13 23:15:25,214] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-5.ppm.merged -> /tmp/pdfimg-5708/im-5.ppm.unpaper
[2023-01-13 23:15:25,214] [INFO] [paperless.consumer] out of deviation range - NO ROTATING
[2023-01-13 23:15:25,215] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-5.ppm: 97.9447
[2023-01-13 23:15:25,215] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-2.ppm
[2023-01-13 23:15:25,215] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-2.ppm
[2023-01-13 23:15:25,216] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-2.ppm.merged -> /tmp/pdfimg-5708/im-2.ppm.unpaper
[2023-01-13 23:15:25,216] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-2.ppm: 99.838
[2023-01-13 23:15:25,216] [INFO] [paperless.consumer] PRE_CONSUME: Discarding blank page /tmp/pdfimg-5708/im-2.ppm
[2023-01-13 23:15:25,217] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-3.ppm
[2023-01-13 23:15:25,217] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-3.ppm
[2023-01-13 23:15:25,218] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-3.ppm.merged -> /tmp/pdfimg-5708/im-3.ppm.unpaper
[2023-01-13 23:15:25,218] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-3.ppm: 95.9085
[2023-01-13 23:15:25,218] [INFO] [paperless.consumer] PRE_CONSUME: Masking /tmp/pdfimg-5708/im-1.ppm
[2023-01-13 23:15:25,218] [INFO] [paperless.consumer] PRE_CONSUME: Applying unpaper to /tmp/pdfimg-5708/im-1.ppm
[2023-01-13 23:15:25,219] [INFO] [paperless.consumer] Processing sheet #1: /tmp/pdfimg-5708/im-1.ppm.merged -> /tmp/pdfimg-5708/im-1.ppm.unpaper
[2023-01-13 23:15:25,220] [INFO] [paperless.consumer] out of deviation range - NO ROTATING
[2023-01-13 23:15:25,220] [INFO] [paperless.consumer] PRE_CONSUME: White percentage of /tmp/pdfimg-5708/im-1.ppm: 96.2587
[2023-01-13 23:15:25,220] [INFO] [paperless.consumer] PRE_CONSUME: OCR of /tmp/pdfimg-5708/im-3.ppm.unpaper
[2023-01-13 23:15:25,221] [INFO] [paperless.consumer] PRE_CONSUME: OCR of /tmp/pdfimg-5708/im-1.ppm.unpaper
[2023-01-13 23:15:25,221] [INFO] [paperless.consumer] PRE_CONSUME: OCR of /tmp/pdfimg-5708/im-5.ppm.unpaper
[2023-01-13 23:15:25,221] [INFO] [paperless.consumer] PRE_CONSUME: Merging PDF
[2023-01-13 23:15:25,221] [INFO] [paperless.consumer] PRE_CONSUME: (PDF) Backing up high-quality original for POST_CONSUME and replacing input file with OCRed version
[2023-01-13 23:15:25,222] [INFO] [paperless.consumer] PRE_CONSUME: (PDF) hash: 1ca02e4c2ec7fe118db5a62353de19f8 (/tmp/5708.pdf)
[2023-01-13 23:15:25,222] [WARNING] [paperless.consumer] Script stderr:
[2023-01-13 23:15:25,222] [WARNING] [paperless.consumer] [ppm_pipe @ 0x556086facf00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,223] [WARNING] [paperless.consumer] [image2 @ 0x556086fb2a80] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,223] [WARNING] [paperless.consumer] [image2 @ 0x556086fb2a80] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,223] [WARNING] [paperless.consumer] [ppm_pipe @ 0x560f8ce09f00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,224] [WARNING] [paperless.consumer] [image2 @ 0x560f8ce0fa80] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,224] [WARNING] [paperless.consumer] [image2 @ 0x560f8ce0fa80] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,224] [WARNING] [paperless.consumer] [ppm_pipe @ 0x5648aa1d6f00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,225] [WARNING] [paperless.consumer] [image2 @ 0x5648aa1dccc0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,225] [WARNING] [paperless.consumer] [image2 @ 0x5648aa1dccc0] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,225] [WARNING] [paperless.consumer] [ppm_pipe @ 0x55dbfb999f00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,226] [WARNING] [paperless.consumer] [image2 @ 0x55dbfb99fa80] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,226] [WARNING] [paperless.consumer] [image2 @ 0x55dbfb99fa80] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,226] [WARNING] [paperless.consumer] [ppm_pipe @ 0x556e3c6a8f00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,227] [WARNING] [paperless.consumer] [image2 @ 0x556e3c6af140] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,227] [WARNING] [paperless.consumer] [image2 @ 0x556e3c6af140] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,227] [WARNING] [paperless.consumer] [ppm_pipe @ 0x558f70566f00] Stream #0: not enough frames to estimate rate; consider increasing probesize
[2023-01-13 23:15:25,228] [WARNING] [paperless.consumer] [image2 @ 0x558f7056ccc0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[2023-01-13 23:15:25,228] [WARNING] [paperless.consumer] [image2 @ 0x558f7056ccc0] Encoder did not produce proper pts, making some up.
[2023-01-13 23:15:25,228] [WARNING] [paperless.consumer] Input file is not a PDF, checking if it is an image...
[2023-01-13 23:15:25,229] [WARNING] [paperless.consumer] Input file is an image
[2023-01-13 23:15:25,229] [WARNING] [paperless.consumer] Input image has no ICC profile, assuming sRGB
[2023-01-13 23:15:25,229] [WARNING] [paperless.consumer] Image seems valid. Try converting to PDF...
[2023-01-13 23:15:25,229] [WARNING] [paperless.consumer] Successfully converted to PDF, processing...
[2023-01-13 23:15:25,230] [WARNING] [paperless.consumer]     1 [tesseract] libpng warning: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG
[2023-01-13 23:15:25,230] [WARNING] [paperless.consumer] Postprocessing...
[2023-01-13 23:15:25,230] [WARNING] [paperless.consumer] Optimize ratio: 1.62 savings: 38.1%
[2023-01-13 23:15:25,231] [WARNING] [paperless.consumer] Output file is a PDF/A-2B (as expected)
[2023-01-13 23:15:25,231] [WARNING] [paperless.consumer] Input file is not a PDF, checking if it is an image...
[2023-01-13 23:15:25,231] [WARNING] [paperless.consumer] Input file is an image
[2023-01-13 23:15:25,232] [WARNING] [paperless.consumer] Input image has no ICC profile, assuming sRGB
[2023-01-13 23:15:25,232] [WARNING] [paperless.consumer] Image seems valid. Try converting to PDF...
[2023-01-13 23:15:25,232] [WARNING] [paperless.consumer] Successfully converted to PDF, processing...
[2023-01-13 23:15:25,232] [WARNING] [paperless.consumer]     1 [tesseract] libpng warning: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG
[2023-01-13 23:15:25,233] [WARNING] [paperless.consumer]     1 [tesseract] lots of diacritics - possibly poor OCR
[2023-01-13 23:15:25,233] [WARNING] [paperless.consumer] Postprocessing...
[2023-01-13 23:15:25,233] [WARNING] [paperless.consumer] Optimize ratio: 1.51 savings: 33.7%
[2023-01-13 23:15:25,234] [WARNING] [paperless.consumer] Output file is a PDF/A-2B (as expected)
[2023-01-13 23:15:25,234] [WARNING] [paperless.consumer] Input file is not a PDF, checking if it is an image...
[2023-01-13 23:15:25,234] [WARNING] [paperless.consumer] Input file is an image
[2023-01-13 23:15:25,235] [WARNING] [paperless.consumer] Input image has no ICC profile, assuming sRGB
[2023-01-13 23:15:25,235] [WARNING] [paperless.consumer] Image seems valid. Try converting to PDF...
[2023-01-13 23:15:25,235] [WARNING] [paperless.consumer] Successfully converted to PDF, processing...
[2023-01-13 23:15:25,236] [WARNING] [paperless.consumer]     1 [tesseract] libpng warning: iCCP: profile 'ICC Profile': 'RGB ': RGB color space not permitted on grayscale PNG
[2023-01-13 23:15:25,236] [WARNING] [paperless.consumer] Postprocessing...
[2023-01-13 23:15:25,236] [WARNING] [paperless.consumer] Optimize ratio: 1.96 savings: 49.1%
[2023-01-13 23:15:25,236] [WARNING] [paperless.consumer] Output file is a PDF/A-2B (as expected)
[2023-01-13 23:15:25,238] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2023-01-13 23:15:25,246] [DEBUG] [paperless.consumer] Parsing 2023-01-13-2313.pdf...
[2023-01-13 23:15:25,526] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /usr/src/paperless/consume/2023-01-13-2313.pdf
[2023-01-13 23:15:26,047] [DEBUG] [paperless.parsing.tesseract] Detected language de
[2023-01-13 23:15:26,141] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/usr/src/paperless/consume/2023-01-13-2313.pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-uh2eb_jw/archive.pdf'), 'use_threads': True, 'jobs': 6, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'sidecar': PosixPath('/tmp/paperless/paperless-uh2eb_jw/sidecar.txt'), 'tesseract_timeout': 0}
[2023-01-13 23:15:29,506] [DEBUG] [paperless.management.consumer] Waiting for file /usr/src/paperless/consume/2023-01-13-2313.pdf to remain unmodified
[2023-01-13 23:15:32,151] [DEBUG] [paperless.parsing.tesseract] Incomplete sidecar file: discarding.
[2023-01-13 23:15:33,269] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-uh2eb_jw/archive.pdf
[2023-01-13 23:15:33,290] [DEBUG] [paperless.parsing.tesseract] Detected language de
[2023-01-13 23:15:33,290] [DEBUG] [paperless.consumer] Generating thumbnail for 2023-01-13-2313.pdf...
[2023-01-13 23:15:33,303] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-uh2eb_jw/archive.pdf[0] /tmp/paperless/paperless-uh2eb_jw/convert.webp
[2023-01-13 23:15:34,514] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/2023-01-13-2313.pdf to the task queue.
[2023-01-13 23:15:41,051] [DEBUG] [paperless.consumer] Saving record to database
[2023-01-13 23:15:41,051] [DEBUG] [paperless.consumer] Creation date from parse_date: 2022-01-21 00:00:00+01:00
[2023-01-13 23:15:41,368] [INFO] [paperless.handlers] Assigning document type Wohnung to 2022-01-21 2023-01-13-2313
[2023-01-13 23:15:41,383] [INFO] [paperless.handlers] Tagging "2022-01-21 2023-01-13-2313" with "Nebenkosten"
[2023-01-13 23:15:41,408] [INFO] [paperless.handlers] Assigning storage path Klaus to 2022-01-21 2023-01-13-2313
[2023-01-13 23:15:41,484] [DEBUG] [paperless.filehandling] Document has storage_path 1 (klaus/{document_type}/{created_year}-{created_month}_{correspondent}_{title}_{tag_list}) set
[2023-01-13 23:15:41,505] [DEBUG] [paperless.filehandling] Document has storage_path 1 (klaus/{document_type}/{created_year}-{created_month}_{correspondent}_{title}_{tag_list}) set
[2023-01-13 23:15:41,510] [DEBUG] [paperless.consumer] Deleting file /usr/src/paperless/consume/2023-01-13-2313.pdf
[2023-01-13 23:15:41,528] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-uh2eb_jw
[2023-01-13 23:15:41,529] [INFO] [paperless.consumer] Executing post-consume script /scripts/post_consume.sh
[2023-01-13 23:15:41,606] [INFO] [paperless.consumer] /scripts/post_consume.sh exited 0
[2023-01-13 23:15:41,606] [INFO] [paperless.consumer] Script stdout:
[2023-01-13 23:15:41,607] [INFO] [paperless.consumer] POST_CONSUME: (HASH):  (/usr/src/paperless/media/documents/originals/klaus/Wohnung/2022-01_none_2023-01-13-2313_Nebenkosten,New.pdf)
[2023-01-13 23:15:41,607] [INFO] [paperless.consumer] POST_CONSUME: (PDF) Copying high-quality 'original file' to paperless.
[2023-01-13 23:15:41,608] [INFO] [paperless.consumer] Document 2022-01-21 2023-01-13-2313 consumption finished
[2023-01-13 23:15:42,021] [ERROR] [paperless.consumer] Cannot consume /usr/src/paperless/consume/2023-01-13-2313.pdf: File not found.
```
```


### Browser logs

_No response_

### Paperless-ngx version

1.11.3

### Host OS

Linux Debian 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

Too many to fit into a single line of text, yet, not relevant to the issue

### Other

PRE/POST consume scripts do the pre/post processing and OCR. OCR in paperless has been disabled. Polling required, since "consume" lives on a samba share


Discussion:
Presumably your script is rebuilding the pdf _somewhere_ and moving it back to the same name?
> Presumably your script is rebuilding the pdf _somewhere_ and moving it back to the same name?

Yes, exactly. My script builds intermediate files on a tmpfs at /tmp and once everything is done, it uses "mv /tmp/thenewpdf.pdf $1" to replace the original file with the one, that my script has built.
Yes, that's going to cause this.  Polling and inotify are looking for file move events, which this would cause.  I would think writing to the existing file shouldn't trigger anything
That could indeed do the trick.
According to this https://github.com/paperless-ngx/paperless-ngx/blob/a16e8324bef12eed7ecab64d5b814aa514cf9b3c/src/documents/management/commands/document_consumer.py#L135-L140 , polling only looks for created or moved files.

If the inode of that file does not change (why should it?), when one directly (over)writes content in it, then this code here https://github.com/gorakhargosh/watchdog/blob/cf0a19589ed8e3f8587733febf54fad82b09614e/src/watchdog/utils/dirsnapshot.py#L99-L114 should not be triggered.

I will give this a try later and report back.
See the same behavior in #668
Duplicate of #2462, the mechanism causing it and solution will essentially be the same.

Resolving patch:
diff --git a/src/documents/consumer.py b/src/documents/consumer.py
--- a/src/documents/consumer.py
+++ b/src/documents/consumer.py
@@ -1,7 +1,10 @@
 import datetime
 import hashlib
 import os
+import shutil
+import tempfile
 import uuid
+from pathlib import Path
 from subprocess import CompletedProcess
 from subprocess import run
 from typing import Optional
@@ -94,7 +97,8 @@ def _fail(
 
     def __init__(self):
         super().__init__()
-        self.path = None
+        self.path: Optional[Path] = None
+        self.original_path: Optional[Path] = None
         self.filename = None
         self.override_title = None
         self.override_correspondent_id = None
@@ -167,16 +171,18 @@ def run_pre_consume_script(self):
 
         self.log("info", f"Executing pre-consume script {settings.PRE_CONSUME_SCRIPT}")
 
-        filepath_arg = os.path.normpath(self.path)
+        working_file_path = str(self.path)
+        original_file_path = str(self.original_path)
 
         script_env = os.environ.copy()
-        script_env["DOCUMENT_SOURCE_PATH"] = filepath_arg
+        script_env["DOCUMENT_SOURCE_PATH"] = original_file_path
+        script_env["DOCUMENT_WORKING_PATH"] = working_file_path
 
         try:
             completed_proc = run(
                 args=[
                     settings.PRE_CONSUME_SCRIPT,
-                    filepath_arg,
+                    original_file_path,
                 ],
                 env=script_env,
                 capture_output=True,
@@ -195,7 +201,7 @@ def run_pre_consume_script(self):
                 exception=e,
             )
 
-    def run_post_consume_script(self, document):
+    def run_post_consume_script(self, document: Document):
         if not settings.POST_CONSUME_SCRIPT:
             return
 
@@ -285,8 +291,8 @@ def try_consume_file(
         Return the document object if it was successfully created.
         """
 
-        self.path = path
-        self.filename = override_filename or os.path.basename(path)
+        self.path = Path(path).resolve()
+        self.filename = override_filename or self.path.name
         self.override_title = override_title
         self.override_correspondent_id = override_correspondent_id
         self.override_document_type_id = override_document_type_id
@@ -311,6 +317,15 @@ def try_consume_file(
 
         self.log("info", f"Consuming {self.filename}")
 
+        # For the actual work, copy the file into a tempdir
+        self.original_path = self.path
+        tempdir = tempfile.TemporaryDirectory(
+            prefix="paperless-ngx",
+            dir=settings.SCRATCH_DIR,
+        )
+        self.path = Path(tempdir.name) / Path(self.filename)
+        shutil.copy(self.original_path, self.path)
+
         # Determine the parser class.
 
         mime_type = magic.from_file(self.path, mime=True)
@@ -453,11 +468,12 @@ def progress_callback(current_progress, max_progress):
                 # Delete the file only if it was successfully consumed
                 self.log("debug", f"Deleting file {self.path}")
                 os.unlink(self.path)
+                self.original_path.unlink()
 
                 # https://github.com/jonaswinkler/paperless-ng/discussions/1037
                 shadow_file = os.path.join(
-                    os.path.dirname(self.path),
-                    "._" + os.path.basename(self.path),
+                    os.path.dirname(self.original_path),
+                    "._" + os.path.basename(self.original_path),
                 )
 
                 if os.path.isfile(shadow_file):
@@ -474,6 +490,7 @@ def progress_callback(current_progress, max_progress):
             )
         finally:
             document_parser.cleanup()
+            tempdir.cleanup()
 
         self.run_post_consume_script(document)
 


Test patch:
diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py
--- a/src/documents/tests/test_consumer.py
+++ b/src/documents/tests/test_consumer.py
@@ -833,7 +833,8 @@ def test_pre_consume_script(self, m):
         with tempfile.NamedTemporaryFile() as script:
             with override_settings(PRE_CONSUME_SCRIPT=script.name):
                 c = Consumer()
-                c.path = "path-to-file"
+                c.original_path = "path-to-file"
+                c.path = "/tmp/somewhere/path-to-file"
                 c.run_pre_consume_script()
 
                 m.assert_called_once()
@@ -841,10 +842,19 @@ def test_pre_consume_script(self, m):
                 args, kwargs = m.call_args
 
                 command = kwargs["args"]
+                environment = kwargs["env"]
 
                 self.assertEqual(command[0], script.name)
                 self.assertEqual(command[1], "path-to-file")
 
+                self.assertDictContainsSubset(
+                    {
+                        "DOCUMENT_SOURCE_PATH": c.original_path,
+                        "DOCUMENT_WORKING_PATH": c.path,
+                    },
+                    environment,
+                )
+
     @mock.patch("documents.consumer.Consumer.log")
     def test_script_with_output(self, mocked_log):
         """
@@ -961,9 +971,10 @@ def test_post_consume_script_with_correspondent(self, m):
 
                 m.assert_called_once()
 
-                args, kwargs = m.call_args
+                _, kwargs = m.call_args
 
                 command = kwargs["args"]
+                environment = kwargs["env"]
 
                 self.assertEqual(command[0], script.name)
                 self.assertEqual(command[1], str(doc.pk))
@@ -972,6 +983,17 @@ def test_post_consume_script_with_correspondent(self, m):
                 self.assertEqual(command[7], "my_bank")
                 self.assertCountEqual(command[8].split(","), ["a", "b"])
 
+                self.assertDictContainsSubset(
+                    {
+                        "DOCUMENT_ID": str(doc.pk),
+                        "DOCUMENT_DOWNLOAD_URL": f"/api/documents/{doc.pk}/download/",
+                        "DOCUMENT_THUMBNAIL_URL": f"/api/documents/{doc.pk}/thumb/",
+                        "DOCUMENT_CORRESPONDENT": "my_bank",
+                        "DOCUMENT_TAGS": "a,b",
+                    },
+                    environment,
+                )
+
     def test_script_exit_non_zero(self):
         """
         GIVEN: