Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] DOUBLE Patch-T document seperator always leads to a failed task
### Description

Using the "patch-T" document seperator always leads to a failed task. Documents are seperated and stored properly. The failed tasks drag attention even though it's not necessary.


Seperator is [Patch-T](https://www.alliancegroup.co.uk/downloads/patches-for-printing-on-a4-paper.pdf).

![image](https://user-images.githubusercontent.com/113503248/190173386-e8604473-484a-4ba4-b18a-6a9bbbbf000f.png)


### Steps to reproduce

Source document format is pdf , it has 5 pages, page 3 is "patch-t" seperator.


### Webserver logs

```bash
Unable to get page count.
Syntax Error: Invalid page count 0
Command Line Error: Wrong page range given: the first page (1) can not be after the last page (0).
 : Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 479, in pdfinfo_from_path
    raise ValueError
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/paperless/src/docker/src/django-q/django_q/cluster.py", line 434, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 109, in consume_file
    separators = barcodes.scan_file_for_separating_barcodes(file_to_process)
  File "/usr/src/paperless/src/documents/barcodes.py", line 110, in scan_file_for_separating_barcodes
    pages_from_path = convert_from_path(filepath, output_folder=path)
  File "/usr/local/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 98, in convert_from_path
    page_count = pdfinfo_from_path(pdf_path, userpw, poppler_path=poppler_path)["Pages"]
  File "/usr/local/lib/python3.9/site-packages/pdf2image/pdf2image.py", line 488, in pdfinfo_from_path
    raise PDFPageCountError(
pdf2image.exceptions.PDFPageCountError: Unable to get page count.
Syntax Error: Invalid page count 0
Command Line Error: Wrong page range given: the first page (1) can not be after the last page (0).
```


### Paperless-ngx version

1.8

### Host OS

Linux 3.10.108 #42661 SMP (x86_64)

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Can you provide a sample file?  At a glance, it looks like a malformed document, where `pdfinfo` isn't able to get valid pages out from the document.
That's what the scanner generated:
[0166_220914170108_SEP.pdf](https://github.com/paperless-ngx/paperless-ngx/files/9567661/0166_220914170108_SEP.pdf)

Paperless splits this into 3 files. 1 and 3 are consumed correctly. File 2 seems to be the "seperator file" which contains no pages and leads to the error. Attached here.
[0166_220914170108_SEP_document_1.pdf](https://github.com/paperless-ngx/paperless-ngx/files/9567697/0166_220914170108_SEP_document_1.pdf)


Ah, I see.  The back to back patch-t barcodes are probably what's tripping it up.  The assumption was probably a single patch-t seperator, not 2 end to end.
Same issue with single patch-t page
That's surprising, we have quite good coverage of the more normal cases.  Where is the patch code in this document?  Maybe the end or middle?
The error seems to be in getting the page count of the PDF.

Can you provide a sample PDF where the error occurs?

I suspect, the error lies in an unusual PDF format. 
We test with standard PDFs and as @stumpylog said, we have a good coverage.

Edit: did you scan the seperator page, or just uploaded the pdf from the source? 

Ah I just took a look at your scan.

It could be the case, that two seperator pages right behind each other are the problem. It tries to get the document between the seperator pages and it obviously can't find it.

Please try with just a single seperator page
I was wrong, sorry! :/

The error only occurs having 2 or more seperator in a row. The PDF between the seperators has no content. That's clear.
As I always scan double sided pages, i created this double-sided seperator.
Otherwise the blank backside of the seperator will be added to the second document as first page. That's what I tried to avoid because of the blank thumbnail in this case....


Oh boy, the mobile github app seemed to have swallowed the other answers. Sorry.

Could you upload the document where the error occurs with the single seperator page?
@gador , I was wrong. Look at my last [post](https://github.com/paperless-ngx/paperless-ngx/issues/1590#issuecomment-1247035126). So I can't provide a document with a single seperator that leads to an error because there is none. Sorry for that! 
I renamed this issue.
Hi @Skram-de thanks for your feedback!

I'll look into it sometime later, when I have access to my laptop again 👍

Resolving patch:
diff --git a/src/documents/barcodes.py b/src/documents/barcodes.py
--- a/src/documents/barcodes.py
+++ b/src/documents/barcodes.py
@@ -8,6 +8,7 @@
 import magic
 from django.conf import settings
 from pdf2image import convert_from_path
+from pikepdf import Page
 from pikepdf import Pdf
 from PIL import Image
 from PIL import ImageSequence
@@ -122,47 +123,56 @@ def separate_pages(filepath: str, pages_to_split_on: List[int]) -> List[str]:
     Returns a list of (temporary) filepaths to consume.
     These will need to be deleted later.
     """
+
+    document_paths = []
+
+    if not pages_to_split_on:
+        logger.warning("No pages to split on!")
+        return document_paths
+
     os.makedirs(settings.SCRATCH_DIR, exist_ok=True)
     tempdir = tempfile.mkdtemp(prefix="paperless-", dir=settings.SCRATCH_DIR)
     fname = os.path.splitext(os.path.basename(filepath))[0]
     pdf = Pdf.open(filepath)
-    document_paths = []
-    logger.debug(f"Temp dir is {str(tempdir)}")
-    if not pages_to_split_on:
-        logger.warning("No pages to split on!")
-    else:
-        # go from the first page to the first separator page
+
+    # A list of documents, ie a list of lists of pages
+    documents: List[List[Page]] = []
+    # A single document, ie a list of pages
+    document: List[Page] = []
+
+    for idx, page in enumerate(pdf.pages):
+        # Keep building the new PDF as long as it is not a
+        # separator index
+        if idx not in pages_to_split_on:
+            document.append(page)
+            # Make sure to append the very last document to the documents
+            if idx == (len(pdf.pages) - 1):
+                documents.append(document)
+                document = []
+        else:
+            # This is a split index, save the current PDF pages, and restart
+            # a new destination page listing
+            logger.debug(f"Starting new document at idx {idx}")
+            documents.append(document)
+            document = []
+
+    documents = [x for x in documents if len(x)]
+
+    logger.debug(f"Split into {len(documents)} new documents")
+
+    # Write the new documents out
+    for doc_idx, document in enumerate(documents):
         dst = Pdf.new()
-        for n, page in enumerate(pdf.pages):
-            if n < pages_to_split_on[0]:
-                dst.pages.append(page)
-        output_filename = f"{fname}_document_0.pdf"
+        dst.pages.extend(document)
+
+        output_filename = f"{fname}_document_{doc_idx}.pdf"
+
+        logger.debug(f"pdf no:{doc_idx} has {len(dst.pages)} pages")
         savepath = os.path.join(tempdir, output_filename)
         with open(savepath, "wb") as out:
             dst.save(out)
-        document_paths = [savepath]
-
-        # iterate through the rest of the document
-        for count, page_number in enumerate(pages_to_split_on):
-            logger.debug(f"Count: {str(count)} page_number: {str(page_number)}")
-            dst = Pdf.new()
-            try:
-                next_page = pages_to_split_on[count + 1]
-            except IndexError:
-                next_page = len(pdf.pages)
-            # skip the first page_number. This contains the barcode page
-            for page in range(page_number + 1, next_page):
-                logger.debug(
-                    f"page_number: {str(page_number)} next_page: {str(next_page)}",
-                )
-                dst.pages.append(pdf.pages[page])
-            output_filename = f"{fname}_document_{str(count + 1)}.pdf"
-            logger.debug(f"pdf no:{str(count)} has {str(len(dst.pages))} pages")
-            savepath = os.path.join(tempdir, output_filename)
-            with open(savepath, "wb") as out:
-                dst.save(out)
-            document_paths.append(savepath)
-    logger.debug(f"Temp files are {str(document_paths)}")
+        document_paths.append(savepath)
+
     return document_paths
 
 


Test patch:
diff --git a/src/documents/tests/samples/barcodes/patch-code-t-double.pdf b/src/documents/tests/samples/barcodes/patch-code-t-double.pdf
new file mode 100644
Binary files /dev/null and b/src/documents/tests/samples/barcodes/patch-code-t-double.pdf differ
diff --git a/src/documents/tests/test_barcodes.py b/src/documents/tests/test_barcodes.py
--- a/src/documents/tests/test_barcodes.py
+++ b/src/documents/tests/test_barcodes.py
@@ -287,6 +287,26 @@ def test_separate_pages(self):
             "patch-code-t-middle.pdf",
         )
         pages = barcodes.separate_pages(test_file, [1])
+
+        self.assertEqual(len(pages), 2)
+
+    def test_separate_pages_double_code(self):
+        """
+        GIVEN:
+            - Input PDF with two patch code pages in a row
+        WHEN:
+            - The input file is split
+        THEN:
+            - Only two files are output
+        """
+        test_file = os.path.join(
+            os.path.dirname(__file__),
+            "samples",
+            "barcodes",
+            "patch-code-t-double.pdf",
+        )
+        pages = barcodes.separate_pages(test_file, [1, 2])
+
         self.assertEqual(len(pages), 2)
 
     def test_separate_pages_no_list(self):