Repository: paperless-ngx/paperless-ngx
Pull request: Refactor: performance and storage optimization of barcode scanning
<!--
Note: All PRs with code changes should be targeted to the `dev` branch, pure documentation changes can target `main`
-->

## Proposed change

In issue #7467 I described the problem that paperless will require a lot of hard disk storage while scanning for barcodes. The current behavior requires about 100MB disk space per page. For larger documents this could result in paperless running out of disk space. Also scanning for barcodes in large documents can last very long, but for large documents it is unlikely to have barcodes since they cannot be scanned with many pages on most scanners.

This pull request implements both solution ideas I gave in that issue.

First the new implementation will not convert all pages to images (which uses the storage) and then start scanning but instead will convert one page after the other, scan for barcodes and delete the temporary file. Therefore at most one tempfile is simultaneously present. This reduces the disk space requirements to that of one page. In my example which cased my initial issue it reduces the required disk space from 35GB to 100MB.

The second is a new option `PAPERLESS_CONSUMER_BARCODE_MAX_PAGES` which will limit the search for barcodes to the given number of first pages. The limit can be disabled by setting it to 0, which also is the default. This option can help to reduce the time spent on barcode scanning in cases where this does not make sense.

<!--
Please include a summary of the change and which issue is fixed (if any) and any relevant motivation / context. List any dependencies that are required for this change. If appropriate, please include an explanation of how your proposed change can be tested. Screenshots and / or videos can also be helpful if appropriate.
-->

<!--
⚠️ Important: Pull requests that implement a new feature or enhancement *should almost always target an existing feature request* with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. If that is not currently the case, please open a feature request instead of this PR to gather feedback from both users and the project maintainers.
-->

Closes #7467

## Type of change

<!--
What type of change does your PR introduce to Paperless-ngx?
NOTE: Please check only one box!
-->

- [ ] Bug fix: non-breaking change which fixes an issue.
- [ ] New feature / Enhancement: non-breaking change which adds functionality. _Please read the important note above._
- [ ] Breaking change: fix or feature that would cause existing functionality to not work as expected.
- [ ] Documentation only.
- [x] Other. Please explain: performance improvement

## Checklist:

<!--
NOTE: PRs that do not address the following will not be merged, please do not skip any relevant items.
-->

- [x] I have read & agree with the [contributing guidelines](https://github.com/paperless-ngx/paperless-ngx/blob/main/CONTRIBUTING.md).
- [x] If applicable, I have included testing coverage for new code in this PR, for [backend](https://docs.paperless-ngx.com/development/#testing) and / or [front-end](https://docs.paperless-ngx.com/development/#testing-and-code-style) changes.
- [x] If applicable, I have tested my code for new features & regressions on both mobile & desktop devices, using the latest version of major browsers.
- [x] If applicable, I have checked that all tests pass, see [documentation](https://docs.paperless-ngx.com/development/#back-end-development).
- [x] I have run all `pre-commit` hooks, see [documentation](https://docs.paperless-ngx.com/development/#code-formatting-with-pre-commit-hooks).
- [x] I have made corresponding changes to the documentation as needed.
- [x] I have checked my modifications for any breaking changes.

If you have further questions, or need additional changes please let me know. I am happy to make required changes to fit into paperless.

Problem statement:
[BUG] Importing of large PDFs fails caused by excessive Disk usage when scanning Barcodes.
### Description

I tried to import a large (300+ Pages PDF) into Paperless. I observed that the processing for Barcode scanning uses Excessive Disk Space (40 GB +) which makes the import fail due to missing Disk Space.

I will add an additional Comment with my configuration, since the field for that is only One Line.
I will also add a comment with my analysis of the problem in the paperless source code and an Idea on how to solve it.

### Steps to reproduce

1. Put a large PDF (for me it was 300+ Pages) in the Consumer Folder
2. Observe the Disk space usage of Paperless

### Webserver logs

```bash
There is nothing relevant here. The observed Disk usage is relevant.
```


### Browser logs

_No response_

### Paperless-ngx version

2.11.4

### Host OS

Ubuntu 22.04

### Installation method

Bare metal

### System status

```json
{
    "pngx_version": "2.11.4",
    "server_os": "Linux-5.15.0-118-generic-x86_64-with-glibc2.35",
    "install_type": "bare-metal",
    "storage": {
        "total": 844362571776,
        "available": 232367833088
    },
    "database": {
        "type": "sqlite",
        "url": "/srv/paperless/data/db.sqlite3",
        "status": "OK",
        "error": null,
        "migration_status": {
            "latest_migration": "paperless_mail.0001_initial_squashed_0009_mailrule_assign_tags",
            "unapplied_migrations": []
        }
    },
    "tasks": {
        "redis_url": "redis://localhost:6379",
        "redis_status": "OK",
        "redis_error": null,
        "celery_status": "OK",
        "index_status": "OK",
        "index_last_modified": "2024-08-14T00:00:31.761945+02:00",
        "index_error": null,
        "classifier_status": "OK",
        "classifier_last_trained": "2024-08-14T08:05:01.549154Z",
        "classifier_error": null
    }
}
```


### Browser

_No response_

### Configuration changes

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.

Hints from issue discussion:
My config looks like to following:
```
# Required services
PAPERLESS_REDIS=redis://localhost:6379

# Paths and folders
PAPERLESS_CONSUMPTION_DIR=/srv/scanner
PAPERLESS_DATA_DIR=/srv/paperless/data
PAPERLESS_MEDIA_ROOT=/srv/paperless/media

# Security and hosting
PAPERLESS_SECRET_KEY=secret
PAPERLESS_URL=https://paperless.intranet

# OCR settings
PAPERLESS_OCR_LANGUAGE=deu+eng
PAPERLESS_OCR_MODE=skip
PAPERLESS_OCR_OUTPUT_TYPE=pdfa
PAPERLESS_ENABLE_NLTK=false

# Software tweaks
PAPERLESS_TASK_WORKERS=1
PAPERLESS_THREADS_PER_WORKER=1
PAPERLESS_TIME_ZONE=Europe/Berlin

# Date settings
PAPERLESS_FILENAME_DATE_ORDER=DMY
PAPERLESS_NUMBER_OF_SUGGESTED_DATES=5

# Enable double sided documents
PAPERLESS_CONSUMER_RECURSIVE=true
PAPERLESS_CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED=true
PAPERLESS_CONSUMER_COLLATE_DOUBLE_SIDED_SUBDIR_NAME=double-sided

# Enable Barcode handling
PAPERLESS_CONSUMER_ENABLE_BARCODES=true
PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE=true
PAPERLESS_CONSUMER_ASN_BARCODE_PREFIX=ASN
PAPERLESS_CONSUMER_BARCODE_STRING=883ef982-4166-49bf-a269-dbb4315e07ac
PAPERLESS_CONSUMER_BARCODE_SCANNER=ZXING
PAPERLESS_CONSUMER_BARCODE_UPSCALE=2
PAPERLESS_CONSUMER_BARCODE_DPI=600
```
My analysis of the problem:

Root cause seems to be that in order to scan barcodes, paperless will convert the pdf pages to (uncompressed) image files. This makes sense since it is required for the barcode scanning libs. But Paperless will first convert all Pages to images and then process one image after the other. That means that paperless will require disk Space for all files simultaneously, and since they are uncompressed for larger documents that leads to Disk usage above 40 GB. This will then cause a Denial of Service if the Server does not have a really large Disk.

I have two ideas which could improve the situation:

# Only convert one page after the other to an image

Here https://github.com/paperless-ngx/paperless-ngx/blob/fea7b0ec8ce0c78ef66443baf3c4e3df60fad82e/src/documents/barcodes.py#L234 paperless will convert all pages to images. After that is finished one page after the other is processed. It would be possible to move the conversion to image into the for loop processing all images. There only one page could be converted and after processing the file can be deleted again.

To implement this nicely one would have to read the number of pages from the PDF before the for loop. The used conversion library has a function for that. https://github.com/Belval/pdf2image/blob/3bfb7f75e82f47e7ce0f09a05e12cda4d4a4fcca/pdf2image/pdf2image.py#L525

Then in the for loop the pages must be individually converted. This can be achieved by the `first_page` and `last_page` arguments of the `convert_from_path` function.

This change would be very simple and is extremely effective and fixes the problem completely.

# Add a configurable limit for how many pages are barcode processed

One could also add a Setting where a maximum number of pages for barcode processing can be set. For me this seems plausible since my scanner can only read 20 pages in one go, so more pages than that are most probably an imported PDF which has no barcodes. Therefore limiting the search to the first 20 pages would be sufficient. This will limit the disk space requirements, however it is still not optimal, since more than one page is stored while it is not required from the program logic.

This could be simply implemented by setting the `last_page` argument of `convert_from_path` to the value from the config.

It makes sense to implement this in combination with the first approach. This will give us the minimal disk space usage from Idea 1, but also improves processing time, since for long pdfs not every page must be processed.

# Remarks
The limit should be optional, also no limit must still be an option. The first idea does not pose any disadvantages, it can therefore be the default behaviour.

If you have any question, or I can help with getting this fixed please let me know.
At best this is a feature request.  A 300 page PDF is massively oversized for normal usage.  I doubt ocrmypdf will handle it will either.
300 pages is just normal for documents like user manual which I rather archive in Paperless than putting them in a analog folder. Since the document was obtained as a pdf ocrmypdf is not an issue here.

Another normal use case would be adding a lot of preexisting documents to paperless using a scanner that is capable of scanning a few hundred pages in one go and splitting them by barcode. Here ocrmypdf also would not be an issue since it will receive the documents after the split where each document is then smaller.

I think this issue should be addressed, and as I have outlined before a fix is really simple to realize. It will also protect against accidential Denial of Service attacks.

Would it be possible to contribute the fixes via a pull request. I want to have that clarified before I start working on it.
1. I agree, requesting that behavior is changed from how it currently works is in fact, a change request.
2. Manuals often have diagrams, pictures that are probably exaggerating this.
3. You could just disable barcodes before ingesting this doc, no?
4. Ocrmypdf is run against all pdfs by default.
5. Since this is not something most (any?) users are running into it would a bit of a niche change. We ask that PRs are linked to discussions that demonstrate some amount of community support. In general the idea sounds fine to me but you would need to make sure it isn’t eg much slower. But stumpylog is our main backend dev so I’d defer to him on the specifics 
1. Okay I can see that depending on definition this is a Feature Request, Bugreport or a Performance Improvement requests. For me it was a bug since there is no limitation on the number of pages and I do not get an error with something like "This large PDFs are unsupported" and also I found no limitation in the documentation. There are lots of warnings that on slower devices this or that feature can be a problem, but that multiple GB ob disk space for tmp data is required was not specified. However I would consider that Requests as a mixture of Bug (as explained), Feature Request (especially the possibility to limit barcode searching to the first n pages), and just a Request to improve the Resource consumption of the Software, which is not really a feature, but also not a bug but rather a software quality improvment.

2. Diagrams and pictures should not add to the filesize of the images or? As far as my understanding goes the images are stored as raw PIL image data, which is not compressed so the size only depends on the Page size and resolution. Even the resize factor does not change the situation here, since resizing is done one page at a time, the same as my suggestion for the conversion to image is.

3. Yes disabling barcode scanning would fix the issue. But changing the config and restarting the the server for one document is cumbersome. There could also be a subfolder in the input folder that skips barcode recognition same as the subfolder for double sided documents. It would fix that particular issue for me, but the waste of diskspace could also pose a problem for people running paperless on a raspberry pi with limited sd card space. From my calculation we can estimate 1GB of disk required per 10 pages. Implementing my first solution approach would reduce the space requirement to a max of 100 MB for any number ob pages. For me also a configurable TMP folder could fix the issue and I could put all that data on a hard disk instead of my system drive SSD but this would also slow down the process and would not fix the resouce wasting, but just throwing hardware at the problem.

4. I know this, but from my understanding it runs after the barcode spliting takes place. Therefore when someone would take a stack of 200 existing documents, equip them with ASN Stickers and scan that as one pdf in the input folder the barcode splitting logic would split the one pdf in a couple of smaller ones, before handing it to ocrmypdf. Therefore  @stumpylog s remark if OCRmypdf will handle that is not applicable to that case. And for my 300 page manual ocrmypdf did not have performance issues after I made sure that enough disk space is available. With setting PAPERLESS_OCR_MODE=skip this should not be a problem for normal Downloaded PDFs which have a text layer.

5. I am not sure if no other user ran into it, I had problems importing some files since some months but was not able to pinpoint it the the use of disk space. So finding the cause for my problems to make a helpful report took me some hours. Maybe other users just see strange errors, but overlook that their paperless ran out of diskspace. In the worst case they file bugreports which can not be resolved since the error is not simple to debug.

From what I read now in the sourcecode implementing both of my ideas can be done in a short time but I would need some time to setup a development setup of paperless. I am also able to make a pull request, but I do not waste time on a fix that is never merged upstream. Just maintaining my personal fork for that reason seems nonsentical, and is not really in the spirit of open source software. From my point of view improvements should be given back to the community.
Yea you’re right, if the PR is a performance improvement I agree a feature request discussion doesn’t make all the a much sense.

Again I defer on the specifics, but in general, yes, if a PR improves some things and doesn’t break other things it will be accepted. Indeed that is the open source spirit.
Okay I try to make a pull request when I find the time for that. I just looked in your development documentation and that seems very good. I hope that it suffices for me to get a development setup going.

The only thing unclear for me is currently the following excerpt from the documentation:

> Run pytest in the src/ directory to execute all tests. This also generates a HTML coverage report. When runnings test, paperless.conf is loaded as well. However, the tests rely on the default configuration. This is not ideal. But for now, make sure no settings except for DEBUG are overridden when testing.

Does that mean all tests run with default config. That means barcode recognition is deactivated and it won't test a configuration with barcode scanning?
No there are many tests specific to the barcode logic
I found that tests that is helpful for development. I can see that the config is overriden for the tests. Therefore I assume that hint only means that for test running paperless.conf must be at default state.

Commit message: Logging adjustments

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>

Patch to review:
diff --git a/docs/configuration.md b/docs/configuration.md
index 7172afcb3c5..d8ec27d2c83 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1289,6 +1289,15 @@ combination with PAPERLESS_CONSUMER_BARCODE_UPSCALE bigger than 1.0.
 
     Defaults to "300"
 
+#### [`PAPERLESS_CONSUMER_BARCODE_MAX_PAGES=<int>`](#PAPERLESS_CONSUMER_BARCODE_MAX_PAGES) {#PAPERLESS_CONSUMER_BARCODE_MAX_PAGES}
+
+: Because barcode detection is a computationally-intensive operation, this setting
+limits the detection of barcodes to a number of first pages. If your scanner has
+a limit for the number of pages that can be scanned it would be sensible to set this
+as the limit here.
+
+    Defaults to "0", allowing all pages to be checked for barcodes.
+
 #### [`PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE=<bool>`](#PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE) {#PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE}
 
 : Enables the detection of barcodes in the scanned document and
diff --git a/src/documents/barcodes.py b/src/documents/barcodes.py
index 2e290a61b68..151ce73e69d 100644
--- a/src/documents/barcodes.py
+++ b/src/documents/barcodes.py
@@ -1,4 +1,5 @@
 import logging
+import os
 import re
 import tempfile
 from dataclasses import dataclass
@@ -7,6 +8,7 @@
 
 from django.conf import settings
 from pdf2image import convert_from_path
+from pdf2image import pdfinfo_from_path
 from pdf2image.exceptions import PDFPageCountError
 from pikepdf import Page
 from pikepdf import Pdf
@@ -231,13 +233,40 @@ def detect(self) -> None:
             logger.debug("Scanning for barcodes using ZXING")
 
         try:
-            pages_from_path = convert_from_path(
-                self.pdf_file,
-                dpi=settings.CONSUMER_BARCODE_DPI,
-                output_folder=self.temp_dir.name,
+            # Read number of pages from pdf
+            num_of_pages = pdfinfo_from_path(self.pdf_file)["Pages"]
+            logger.debug(f"PDF has {num_of_pages} pages")
+
+            # Get limit from configuration
+ barcode_max_pages = (
+                num_of_pages
+                if settings.CONSUMER_BARCODE_MAX_PAGES == 0
+                else settings.CONSUMER_BARCODE_MAX_PAGES
             )
 
-            for current_page_number, page in enumerate(pages_from_path):
+            if barcode_max_pages < num_of_pages:
+                logger.debug(
+                    f"Barcodes detection will be limited to the first {barcode_max_pages} pages",
+                )
+
+            # Loop al page
+            for current_page_number in range(min(num_of_pages, barcode_max_pages)):
+                logger.debug(f"Processing page {current_page_number}")
+
+                # Convert page to image
+                page = convert_from_path(
+                    self.pdf_file,
+                    dpi=settings.CONSUMER_BARCODE_DPI,
+                    output_folder=self.temp_dir.name,
+                    first_page=current_page_number + 1,
+                    last_page=current_page_number + 1,
+                )[0]
+
+                # Remember filename, since it is lost by upscaling
+                page_filename = page.filename
+                logger.debug(f"Image is at {page_filename}")
+
+                # Upscale image if configured
                 factor = settings.CONSUMER_BARCODE_UPSCALE
                 if factor > 1.0:
                     logger.debug(
@@ -248,11 +277,15 @@ def detect(self) -> None:
                         (int(round(x * factor)), (int(round(y * factor)))),
                     )
 
+                # Detect barcodes
                 for barcode_value in reader(page):
                     self.barcodes.append(
                         Barcode(current_page_number, barcode_value),
                     )
 
+                # Delete temporary image file
+                os.remove(page_filename)
+
         # Password protected files can't be checked
         # This is the exception raised for those
         except PDFPageCountError as e:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
index ee6110732f1..ebe64ba9ec1 100644
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -925,6 +925,11 @@ def default_threads_per_worker(task_workers) -> int:
 
 CONSUMER_BARCODE_DPI: Final[int] = __get_int("PAPERLESS_CONSUMER_BARCODE_DPI", 300)
 
+CONSUMER_BARCODE_MAX_PAGES: Final[int] = __get_int(
+    "PAPERLESS_CONSUMER_BARCODE_MAX_PAGES",
+    0,
+)
+
 CONSUMER_ENABLE_TAG_BARCODE: Final[bool] = __get_boolean(
     "PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE",
 )


Code review comments:
diff_hunk: @@ -231,13 +233,40 @@ def detect(self) -> None:
             logger.debug("Scanning for barcodes using ZXING")
 
         try:
-            pages_from_path = convert_from_path(
-                self.pdf_file,
-                dpi=settings.CONSUMER_BARCODE_DPI,
-                output_folder=self.temp_dir.name,
+            # Read number of pages from pdf
+            num_of_pages = pdfinfo_from_path(self.pdf_file)["Pages"]
+            logger.debug(f"PDF has {num_of_pages} pages")
+
+            # Get limit from configuration
+ barcode_max_pages = (
+                num_of_pages
+                if settings.CONSUMER_BARCODE_MAX_PAGES == 0
+                else settings.CONSUMER_BARCODE_MAX_PAGES
             )
 
-            for current_page_number, page in enumerate(pages_from_path):
+            if barcode_max_pages < num_of_pages:
path: src/documents/barcodes.py
text: @user1:
```suggestion
            if barcode_max_pages < num_of_pages:  # pragma: no cover
```

This would be fine for the log line