Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] File altered at DOCUMENT_WORKING_PATH by PAPERLESS_PRE_CONSUME_SCRIPT is not consumed properly
### Description

I am decrypting pdfs with PRE_CONSUME_SCRIPT. It was working before 2.1.1 (but not sure of the exact moment). Currently, it partially does not behave as expected.

Decrypted file is presented correctly in thumbnail:

![Screenshot from 2023-12-30 19-09-17](https://github.com/paperless-ngx/paperless-ngx/assets/125857607/810c1587-1fd9-4ca1-a8aa-52cf137be0ce)

But the same file incorrectly remain encrypted in viewer.

![Screenshot from 2023-12-30 19-06-56](https://github.com/paperless-ngx/paperless-ngx/assets/125857607/bdbe67e8-9eb1-49d2-8f6f-9df2645ea425)

I am guessing that thumbnail is created after decryption... but the file itself is archived before decryption.



### Steps to reproduce

More info @ https://github.com/lukasz-lobocki/pdf_decrypt_retrieve_attachments

```python
def unlock_pdf(s_file_path: str, w_file_path: str) -> None:
    passw = None
    print("Reading passwords from {pf}".format(pf=pass_file_path))
    with open(pass_file_path, "r") as f:
        passwords = f.readlines()
    for p in passwords:
        passw = p.strip()
        try:
            with pikepdf.open(w_file_path, password=passw, allow_overwriting_input=True) as pdf:
                print("Unlocked succesfully with password {f}***{l}".format(f=passw[0], l=passw[-1]))
                pdf.save(w_file_path)
                #pdf.save(s_file_path)
                print("Unlocked working file replaced {}".format(w_file_path))
                break
        except pikepdf.PasswordError:
            print("Password {f}***{l} is not working".format(f=passw[0], l=passw[-1]))
            continue
    if passw is None:
        print("Empty password file {pf}".format(pf=pass_file_path))
```

### Webserver logs

```
[2023-12-30 18:57:48,263] [INFO] [paperless.consumer] Decrypting file /tmp/paperless/paperless-ngxlys5ck3z/2019-02-28 Drive100.pdf

[2023-12-30 18:57:48,264] [INFO] [paperless.consumer] Reading passwords from /usr/src/paperless/scripts/passwords.txt

[2023-12-30 18:57:48,265] [INFO] [paperless.consumer] Unlocked succesfully with password D***!

[2023-12-30 18:57:48,265] [INFO] [paperless.consumer] Unlocked working file replaced /tmp/paperless/paperless-ngxlys5ck3z/2019-02-28 Drive100.pdf
```


### Browser logs

_No response_

### Paperless-ngx version

2.2.1

### Host OS

Debian 6.1.67-1 (2023-12-12) x86_64

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

in docker-compose.yml volume: `- /home/la_lukasz/paperless-ngx/scripts:/usr/src/paperless/scripts` added. In docker-compose.env `PAPERLESS_PRE_CONSUME_SCRIPT=/usr/src/paperless/scripts/pre-consumption.py` added.

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
I dont think anything changed with respect to using a pre-consume script recently, but thats not so much my end of things. 

From your screenshot above you show the popup preview, one thing that changed there is it now uses the "built-in" PDF viewer, whereas before the popup always used the browser preview. Indeed, the lock icon indicates a password-protected PDF (for the sake of usability the popup doesnt let you put in a password at all). So I just want to make sure, in the preview area of the "edit" page (/documents/ID/) it's also prompting you for a password?

Again, Im not the best person to comment on the script part, just want to get an idea if the frontend is at all relevant here.
If anything, we fixed an issue here.  We copy the _original_ file into paperless, but parse from the _working copy_.  The intent is always to archive the _original_, so that is what has been done here.
This change was made in #4781, the original path is now stored, not an maybe altered version of the original.  I don't see this as a bug, but rather the storage of a possibly altered file.

The workaround is decryption before consumption, either manually or watching another directory and then moving to the consumption directory.
Thank you.

So this part of https://docs.paperless-ngx.com/advanced_usage/#pre-consume-script is misleading. Modification is no longer an option, contrary to what the below paragraph suggests.

> Pre-consume scripts which modify the document should only change the DOCUMENT_WORKING_PATH file or a second consume task may be triggered, leading to failures as two tasks work on the same document path

Another idea, based on your @stumpylog suggestion to decrypt before consumption. Is it _allowed_ to delete `DOCUMENT_SOURCE_PATH` file or `DOCUMENT_WORKING_PATH` file from within `PAPERLESS_PRE_CONSUME_SCRIPT`? In order to "sneaky" drop the original file and substitute it with new decrypted one...

PS. Yes @shamoon , the preview area of the "edit" page (/documents/ID/) it's also prompting for password.
There really isn't a good way for a pre or post script to modify the file.  Changing the original triggers a new consume, which then fails is interesting ways when the original consume completes.  Changing a document in post means the checksum no longer matches, which produces warnings.  Changing the working copy allows parsing to happen, but it is ultimately discarded.

But we want to store the original document, since this is ultimately trying to be an archive solution.  I'll keep this open and think some more about a solution.
@stumpylog I had a usecase, where my `PAPERLESS_PRE_CONSUME_SCRIPT` did cut white space from a jpeg to make the pdf smaller and more pleasant to look at. I do not use this at the moment, but I would like to know that I could again use it in the future. Just as before, `PAPERLESS_PRE_CONSUME_SCRIPT` should be able to alter a copy of the original file which is then archived. If my understanding is correct, I would therefore second @lukasz-lobocki, in that he described an actual bug, not a feature. Also, the previous functionality is still mentioned in the documentation..

Resolving patch:
diff --git a/src/documents/consumer.py b/src/documents/consumer.py
--- a/src/documents/consumer.py
+++ b/src/documents/consumer.py
@@ -135,24 +135,24 @@ def pre_check_file_exists(self):
         """
         Confirm the input file still exists where it should
         """
-        if not os.path.isfile(self.path):
+        if not os.path.isfile(self.original_path):
             self._fail(
                 ConsumerStatusShortMessage.FILE_NOT_FOUND,
-                f"Cannot consume {self.path}: File not found.",
+                f"Cannot consume {self.original_path}: File not found.",
             )
 
     def pre_check_duplicate(self):
         """
         Using the MD5 of the file, check this exact file doesn't already exist
         """
-        with open(self.path, "rb") as f:
+        with open(self.original_path, "rb") as f:
             checksum = hashlib.md5(f.read()).hexdigest()
         existing_doc = Document.objects.filter(
             Q(checksum=checksum) | Q(archive_checksum=checksum),
         )
         if existing_doc.exists():
             if settings.CONSUMER_DELETE_DUPLICATES:
-                os.unlink(self.path)
+                os.unlink(self.original_path)
             self._fail(
                 ConsumerStatusShortMessage.DOCUMENT_ALREADY_EXISTS,
                 f"Not consuming {self.filename}: It is a duplicate of"
@@ -211,7 +211,7 @@ def run_pre_consume_script(self):
 
         self.log.info(f"Executing pre-consume script {settings.PRE_CONSUME_SCRIPT}")
 
-        working_file_path = str(self.path)
+        working_file_path = str(self.working_copy)
         original_file_path = str(self.original_path)
 
         script_env = os.environ.copy()
@@ -343,8 +343,8 @@ def try_consume_file(
         Return the document object if it was successfully created.
         """
 
-        self.path = Path(path).resolve()
-        self.filename = override_filename or self.path.name
+        self.original_path = Path(path).resolve()
+        self.filename = override_filename or self.original_path.name
         self.override_title = override_title
         self.override_correspondent_id = override_correspondent_id
         self.override_document_type_id = override_document_type_id
@@ -377,17 +377,16 @@ def try_consume_file(
         self.log.info(f"Consuming {self.filename}")
 
         # For the actual work, copy the file into a tempdir
-        self.original_path = self.path
         tempdir = tempfile.TemporaryDirectory(
             prefix="paperless-ngx",
             dir=settings.SCRATCH_DIR,
         )
-        self.path = Path(tempdir.name) / Path(self.filename)
-        copy_file_with_basic_stats(self.original_path, self.path)
+        self.working_copy = Path(tempdir.name) / Path(self.filename)
+        copy_file_with_basic_stats(self.original_path, self.working_copy)
 
         # Determine the parser class.
 
-        mime_type = magic.from_file(self.path, mime=True)
+        mime_type = magic.from_file(self.working_copy, mime=True)
 
         self.log.debug(f"Detected mime type: {mime_type}")
 
@@ -406,7 +405,7 @@ def try_consume_file(
 
         document_consumption_started.send(
             sender=self.__class__,
-            filename=self.path,
+            filename=self.working_copy,
             logging_group=self.logging_group,
         )
 
@@ -444,7 +443,7 @@ def progress_callback(current_progress, max_progress):  # pragma: no cover
                 ConsumerStatusShortMessage.PARSING_DOCUMENT,
             )
             self.log.debug(f"Parsing {self.filename}...")
-            document_parser.parse(self.path, mime_type, self.filename)
+            document_parser.parse(self.working_copy, mime_type, self.filename)
 
             self.log.debug(f"Generating thumbnail for {self.filename}...")
             self._send_progress(
@@ -454,7 +453,7 @@ def progress_callback(current_progress, max_progress):  # pragma: no cover
                 ConsumerStatusShortMessage.GENERATING_THUMBNAIL,
             )
             thumbnail = document_parser.get_thumbnail(
-                self.path,
+                self.working_copy,
                 mime_type,
                 self.filename,
             )
@@ -527,7 +526,7 @@ def progress_callback(current_progress, max_progress):  # pragma: no cover
 
                     self._write(
                         document.storage_type,
-                        self.original_path,
+                        self.working_copy,
                         document.source_path,
                     )
 
@@ -560,9 +559,9 @@ def progress_callback(current_progress, max_progress):  # pragma: no cover
                 document.save()
 
                 # Delete the file only if it was successfully consumed
-                self.log.debug(f"Deleting file {self.path}")
-                os.unlink(self.path)
+                self.log.debug(f"Deleting file {self.working_copy}")
                 self.original_path.unlink()
+                self.working_copy.unlink()
 
                 # https://github.com/jonaswinkler/paperless-ng/discussions/1037
                 shadow_file = os.path.join(
@@ -735,7 +734,7 @@ def _store(
             )[:127],
             content=text,
             mime_type=mime_type,
-            checksum=hashlib.md5(self.original_path.read_bytes()).hexdigest(),
+            checksum=hashlib.md5(self.working_copy.read_bytes()).hexdigest(),
             created=create_date,
             modified=create_date,
             storage_type=storage_type,
diff --git a/src/documents/parsers.py b/src/documents/parsers.py
--- a/src/documents/parsers.py
+++ b/src/documents/parsers.py
@@ -322,7 +322,9 @@ def __init__(self, logging_group, progress_callback=None):
         self.logging_group = logging_group
         self.settings = self.get_settings()
         os.makedirs(settings.SCRATCH_DIR, exist_ok=True)
-        self.tempdir = tempfile.mkdtemp(prefix="paperless-", dir=settings.SCRATCH_DIR)
+        self.tempdir = Path(
+            tempfile.mkdtemp(prefix="paperless-", dir=settings.SCRATCH_DIR),
+        )
 
         self.archive_path = None
         self.text = None
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -90,16 +90,18 @@ def has_alpha(self, image) -> bool:
         with Image.open(image) as im:
             return im.mode in ("RGBA", "LA")
 
-    def remove_alpha(self, image_path: str):
+    def remove_alpha(self, image_path: str) -> Path:
+        no_alpha_image = Path(self.tempdir) / "image-no-alpha"
         subprocess.run(
             [
                 settings.CONVERT_BINARY,
                 "-alpha",
                 "off",
                 image_path,
-                image_path,
+                no_alpha_image,
             ],
         )
+        return no_alpha_image
 
     def get_dpi(self, image) -> Optional[int]:
         try:
@@ -251,7 +253,8 @@ def construct_ocrmypdf_parameters(
                     f"Removing alpha layer from {input_file} "
                     "for compatibility with img2pdf",
                 )
-                self.remove_alpha(input_file)
+                # Replace the input file with the non-alpha
+                ocrmypdf_args["input_file"] = self.remove_alpha(input_file)
 
             if dpi:
                 self.log.debug(f"Detected DPI for image {input_file}: {dpi}")


Test patch:
diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py
--- a/src/documents/tests/test_consumer.py
+++ b/src/documents/tests/test_consumer.py
@@ -928,7 +928,7 @@ def setUp(self) -> None:
     @override_settings(PRE_CONSUME_SCRIPT=None)
     def test_no_pre_consume_script(self, m):
         c = Consumer()
-        c.path = "path-to-file"
+        c.working_copy = "path-to-file"
         c.run_pre_consume_script()
         m.assert_not_called()
 
@@ -938,7 +938,7 @@ def test_no_pre_consume_script(self, m):
     def test_pre_consume_script_not_found(self, m, m2):
         c = Consumer()
         c.filename = "somefile.pdf"
-        c.path = "path-to-file"
+        c.working_copy = "path-to-file"
         self.assertRaises(ConsumerError, c.run_pre_consume_script)
 
     @mock.patch("documents.consumer.run")
@@ -947,7 +947,7 @@ def test_pre_consume_script(self, m):
             with override_settings(PRE_CONSUME_SCRIPT=script.name):
                 c = Consumer()
                 c.original_path = "path-to-file"
-                c.path = "/tmp/somewhere/path-to-file"
+                c.working_copy = "/tmp/somewhere/path-to-file"
                 c.task_id = str(uuid.uuid4())
                 c.run_pre_consume_script()
 
@@ -963,7 +963,7 @@ def test_pre_consume_script(self, m):
 
                 subset = {
                     "DOCUMENT_SOURCE_PATH": c.original_path,
-                    "DOCUMENT_WORKING_PATH": c.path,
+                    "DOCUMENT_WORKING_PATH": c.working_copy,
                     "TASK_ID": c.task_id,
                 }
                 self.assertDictEqual(environment, {**environment, **subset})
@@ -991,7 +991,7 @@ def test_script_with_output(self):
             with override_settings(PRE_CONSUME_SCRIPT=script.name):
                 with self.assertLogs("paperless.consumer", level="INFO") as cm:
                     c = Consumer()
-                    c.path = "path-to-file"
+                    c.working_copy = "path-to-file"
 
                     c.run_pre_consume_script()
                     self.assertIn(
@@ -1024,7 +1024,7 @@ def test_script_exit_non_zero(self):
 
             with override_settings(PRE_CONSUME_SCRIPT=script.name):
                 c = Consumer()
-                c.path = "path-to-file"
+                c.working_copy = "path-to-file"
                 self.assertRaises(
                     ConsumerError,
                     c.run_pre_consume_script,