Repository: paperless-ngx/paperless-ngx

Issue:
Cannot import pdf
### Description

Paperless does not accept pdf.

### Steps to reproduce

1. Import via GUI
2. Error message with name of pdf
3. Error Message: Not sure how to handle PDF image of this type
Or
1. Import via "consume"
2. Error message without file name
3. Error Message: Not sure how to handle PDF image of this type


### Webserver logs

```bash
[2022-11-09 11:51:42,534] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/535161001.pdf to the task queue.
[2022-11-09 11:51:42,550] [ERROR] [paperless.handlers] Creating PaperlessTask failed: connection already closed
11:51:42 [Q] INFO Enqueued 1
11:51:42 [Q] INFO Process-1:3687 processing [535161001.pdf]
11:51:42 [Q] INFO Process-1:3687 stopped doing work
11:51:43 [Q] ERROR Failed [535161001.pdf] - Not sure how to handle PDF image of this type : Traceback (most recent call last):
  File "/usr/src/paperless/src/src/django-q/django_q/cluster.py", line 454, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 99, in consume_file
    pdf_filepath, separators = barcodes.scan_file_for_separating_barcodes(path)
  File "/usr/src/paperless/src/documents/barcodes.py", line 126, in scan_file_for_separating_barcodes
    pillow_img = pdfimage.as_pil_image()
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 742, in as_pil_image
    im = self._extract_transcoded()
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 606, in _extract_transcoded
    if self.mode in {'DeviceN', 'Separation'}:
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 301, in mode
    raise NotImplementedError(
NotImplementedError: Not sure how to handle PDF image of this type
11:51:46 [Q] INFO recycled worker Process-1:3687
11:51:46 [Q] INFO Process-1:3689 ready for work at 18466
```


### Browser logs

```bash
Not sure how to handle PDF image of this type : Traceback (most recent call last):
  File "/usr/src/paperless/src/src/django-q/django_q/cluster.py", line 454, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 99, in consume_file
    pdf_filepath, separators = barcodes.scan_file_for_separating_barcodes(path)
  File "/usr/src/paperless/src/documents/barcodes.py", line 126, in scan_file_for_separating_barcodes
    pillow_img = pdfimage.as_pil_image()
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 742, in as_pil_image
    im = self._extract_transcoded()
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 606, in _extract_transcoded
    if self.mode in {'DeviceN', 'Separation'}:
  File "/usr/local/lib/python3.9/site-packages/pikepdf/models/image.py", line 301, in mode
    raise NotImplementedError(
NotImplementedError: Not sure how to handle PDF image of this type
```


### Paperless-ngx version

1.9.2

### Host OS

Synology DSM 7.1.1-42962

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Its not very easy to look into this without a pdf. Most likely the PDF is malformed, you can try re-saving / re-generating it and trying again.
The origin of the pdf is from a bank (WISE). Therefore, I cannot attach it. It was generated by Text 2.1.0 (by lowagie.com). However, I assume that  bank can produce valid pdfs.
Looks like a duplicate to #1743 to me. 

I've had similar problems with some PDF (which could be opened in viewers without problems). My workaround was to open them in Adobe Reader and print them again to the PDF file printer of Windows 11. The resulting file could be consumed thereafter.

That's not nice but since it seems to be bug #1743 which already has been fixed and only waits for the next release it probably won't be needed very long anymore.
the PDF itself is probably valid, but as with #1743, pikepdf doesn't handle every image type (and not nearly as many as I'd like, but it's hard to support every format).

Resolving patch:
diff --git a/src/documents/barcodes.py b/src/documents/barcodes.py
--- a/src/documents/barcodes.py
+++ b/src/documents/barcodes.py
@@ -10,12 +10,10 @@
 import magic
 from django.conf import settings
 from pdf2image import convert_from_path
-from pdf2image.exceptions import PDFPageCountError
 from pikepdf import Page
 from pikepdf import PasswordError
 from pikepdf import Pdf
 from pikepdf import PdfImage
-from pikepdf.models.image import HifiPrintImageNotTranscodableError
 from PIL import Image
 from PIL import ImageSequence
 from pyzbar import pyzbar
@@ -101,7 +99,7 @@ def convert_from_tiff_to_pdf(filepath: str) -> str:
                 images[0].save(newpath)
             else:
                 images[0].save(newpath, save_all=True, append_images=images[1:])
-        except OSError as e:
+        except OSError as e:  # pragma: no cover
             logger.warning(
                 f"Could not save the file as pdf. Error: {str(e)}",
             )
@@ -122,13 +120,16 @@ def _pikepdf_barcode_scan(pdf_filepath: str):
                 for image_key in page.images:
                     pdfimage = PdfImage(page.images[image_key])
 
+                    # This type is known to have issues:
+                    # https://github.com/pikepdf/pikepdf/issues/401
                     if "/CCITTFaxDecode" in pdfimage.filters:
                         raise BarcodeImageFormatError(
                             "Unable to decode CCITTFaxDecode images",
                         )
 
                     # Not all images can be transcoded to a PIL image, which
-                    # is what pyzbar expects to receive
+                    # is what pyzbar expects to receive, so this may
+                    # raise an exception, triggering fallback
                     pillow_img = pdfimage.as_pil_image()
 
                     detected_barcodes = barcode_reader(pillow_img)
@@ -155,29 +156,23 @@ def _pdf2image_barcode_scan(pdf_filepath: str):
         if mime_type == "image/tiff":
             pdf_filepath = convert_from_tiff_to_pdf(filepath)
 
-        # Chose the scanner
-        if settings.CONSUMER_USE_LEGACY_DETECTION:
-            logger.debug("Using pdf2image for barcodes")
-            scanner_function = _pdf2image_barcode_scan
-        else:
-            logger.debug("Using pikepdf for barcodes")
-            scanner_function = _pikepdf_barcode_scan
-
-        # Run the scanner
+        # Always try pikepdf first, it's usually fine, faster and
+        # uses less memory
         try:
-            scanner_function(pdf_filepath)
-        # Neither method can handle password protected PDFs without it being
-        # provided.  Log it and continue
-        except (PasswordError, PDFPageCountError) as e:
+            _pikepdf_barcode_scan(pdf_filepath)
+        # Password protected files can't be checked
+        except PasswordError as e:
             logger.warning(
-                f"File is likely password protected, not splitting: {e}",
+                f"File is likely password protected, not checking for barcodes: {e}",
             )
-        # Handle pikepdf related image decoding issues with a fallback
-        except (BarcodeImageFormatError, HifiPrintImageNotTranscodableError) as e:
+        # Handle pikepdf related image decoding issues with a fallback to page
+        # by page conversion to images in a temporary directory
+        except Exception as e:
             logger.warning(
                 f"Falling back to pdf2image because: {e}",
             )
             try:
+                # Clear the list in case some processing worked
                 separator_page_numbers = []
                 _pdf2image_barcode_scan(pdf_filepath)
             # This file is really borked, allow the consumption to continue
@@ -186,11 +181,6 @@ def _pdf2image_barcode_scan(pdf_filepath: str):
                 logger.warning(
                     f"Exception during barcode scanning: {e}",
                 )
-        # We're not sure what happened, but allow the consumption to continue
-        except Exception as e:  # pragma: no cover
-            logger.warning(
-                f"Exception during barcode scanning: {e}",
-            )
 
     else:
         logger.warning(
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -573,11 +573,6 @@ def default_threads_per_worker(task_workers) -> int:
     "PAPERLESS_CONSUMER_BARCODE_TIFF_SUPPORT",
 )
 
-CONSUMER_USE_LEGACY_DETECTION: Final[bool] = __get_boolean(
-    "PAPERLESS_CONSUMER_USE_LEGACY_DETECTION",
-    "NO",
-)
-
 CONSUMER_BARCODE_STRING: Final[str] = os.getenv(
     "PAPERLESS_CONSUMER_BARCODE_STRING",
     "PATCHT",


Test patch:
diff --git a/src/documents/tests/test_barcodes.py b/src/documents/tests/test_barcodes.py
--- a/src/documents/tests/test_barcodes.py
+++ b/src/documents/tests/test_barcodes.py
@@ -468,41 +468,6 @@ def test_barcode_splitter(self):
         self.assertTrue(os.path.isfile(target_file1))
         self.assertTrue(os.path.isfile(target_file2))
 
-    @override_settings(CONSUMER_USE_LEGACY_DETECTION=True)
-    def test_barcode_splitter_legacy_fallback(self):
-        """
-        GIVEN:
-            - File containing barcode
-            - Legacy method of detection is enabled
-        WHEN:
-            - File is scanned for barcodes
-        THEN:
-            - Barcodes are properly detected
-        """
-        test_file = os.path.join(
-            self.BARCODE_SAMPLE_DIR,
-            "patch-code-t-middle.pdf",
-        )
-        tempdir = tempfile.mkdtemp(prefix="paperless-", dir=settings.SCRATCH_DIR)
-
-        pdf_file, separator_page_numbers = barcodes.scan_file_for_separating_barcodes(
-            test_file,
-        )
-
-        self.assertEqual(test_file, pdf_file)
-        self.assertTrue(len(separator_page_numbers) > 0)
-
-        document_list = barcodes.separate_pages(test_file, separator_page_numbers)
-        self.assertTrue(document_list)
-        for document in document_list:
-            barcodes.save_to_dir(document, target_dir=tempdir)
-
-        target_file1 = os.path.join(tempdir, "patch-code-t-middle_document_0.pdf")
-        target_file2 = os.path.join(tempdir, "patch-code-t-middle_document_1.pdf")
-
-        self.assertTrue(os.path.isfile(target_file1))
-        self.assertTrue(os.path.isfile(target_file2))
-
     @override_settings(CONSUMER_ENABLE_BARCODES=True)
     def test_consume_barcode_file(self):
         test_file = os.path.join(
@@ -586,7 +551,7 @@ def test_consume_barcode_supported_no_extension_file(self):
         with mock.patch("documents.tasks.async_to_sync"):
             self.assertEqual(tasks.consume_file(dst), "File successfully split")
 
-    def test_scan_file_for_separating_barcodes_password_pikepdf(self):
+    def test_scan_file_for_separating_barcodes_password(self):
         """
         GIVEN:
             - Password protected PDF
@@ -603,22 +568,3 @@ def test_scan_file_for_separating_barcodes_password_pikepdf(self):
 
         self.assertEqual(pdf_file, test_file)
         self.assertListEqual(separator_page_numbers, [])
-
-    @override_settings(CONSUMER_USE_LEGACY_DETECTION=True)
-    def test_scan_file_for_separating_barcodes_password_pdf2image(self):
-        """
-        GIVEN:
-            - Password protected PDF
-            - pdf2image based scanning
-        WHEN:
-            - File is scanned for barcode
-        THEN:
-            - Scanning handle the exception without exception
-        """
-        test_file = os.path.join(self.SAMPLE_DIR, "password-is-test.pdf")
-        pdf_file, separator_page_numbers = barcodes.scan_file_for_separating_barcodes(
-            test_file,
-        )
-
-        self.assertEqual(pdf_file, test_file)
-        self.assertListEqual(separator_page_numbers, [])