Repository: paperless-ngx/paperless-ngx

Issue:
SubprocessOutputError: Ghostscript PDF/A rendering failed
### Description

Since 1.17.0 i get SubprocessOutputError: Ghostscript PDF/A rendering failed when i upload a new file. 1.16.5 Docker image works. 1.17.2 not fix this problem

### Steps to reproduce

...File-Upload

### Webserver logs

```bash
Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 320, in parse
    ocrmypdf.ocr(**args)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/api.py", line 347, in ocr
    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 391, in run_pipeline
    optimize_messages = exec_concurrent(context, executor)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 308, in exec_concurrent
    pdf, messages = post_process(pdf, context, executor)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 239, in post_process
    pdf_out = convert_to_pdfa(pdf_out, ps_stub_out, context)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_pipeline.py", line 789, in convert_to_pdfa
    context.plugin_manager.hook.generate_pdfa(
  File "/usr/local/lib/python3.9/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 116, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/builtin_plugins/ghostscript.py", line 77, in generate_pdfa
    ghostscript.generate_pdfa(
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_exec/ghostscript.py", line 246, in generate_pdfa
    raise SubprocessOutputError('Ghostscript PDF/A rendering failed') from e
ocrmypdf.exceptions.SubprocessOutputError: Ghostscript PDF/A rendering failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 349, in main_wrap
    raise exc_info[1]
  File "/usr/src/paperless/src/documents/consumer.py", line 425, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 377, in parse
    raise ParseError(f"{e.__class__.__name__}: {e!s}") from e
documents.parsers.ParseError: SubprocessOutputError: Ghostscript PDF/A rendering failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/paperless/src/documents/tasks.py", line 156, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 455, in try_consume_file
    self._fail(
  File "/usr/src/paperless/src/documents/consumer.py", line 109, in _fail
    raise ConsumerError(f"{self.filename}: {log_message or message}") from exception
```


### Browser logs

```bash
Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 320, in parse
    ocrmypdf.ocr(**args)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/api.py", line 347, in ocr
    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 391, in run_pipeline
    optimize_messages = exec_concurrent(context, executor)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 308, in exec_concurrent
    pdf, messages = post_process(pdf, context, executor)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_sync.py", line 239, in post_process
    pdf_out = convert_to_pdfa(pdf_out, ps_stub_out, context)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_pipeline.py", line 789, in convert_to_pdfa
    context.plugin_manager.hook.generate_pdfa(
  File "/usr/local/lib/python3.9/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 116, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/builtin_plugins/ghostscript.py", line 77, in generate_pdfa
    ghostscript.generate_pdfa(
  File "/usr/local/lib/python3.9/site-packages/ocrmypdf/_exec/ghostscript.py", line 246, in generate_pdfa
    raise SubprocessOutputError('Ghostscript PDF/A rendering failed') from e
ocrmypdf.exceptions.SubprocessOutputError: Ghostscript PDF/A rendering failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 349, in main_wrap
    raise exc_info[1]
  File "/usr/src/paperless/src/documents/consumer.py", line 425, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 377, in parse
    raise ParseError(f"{e.__class__.__name__}: {e!s}") from e
documents.parsers.ParseError: SubprocessOutputError: Ghostscript PDF/A rendering failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/paperless/src/documents/tasks.py", line 156, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 455, in try_consume_file
    self._fail(
  File "/usr/src/paperless/src/documents/consumer.py", line 109, in _fail
    raise ConsumerError(f"{self.filename}: {log_message or message}") from exception
```


### Paperless-ngx version

1.17.2

### Host OS

arm64

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
This problem is now multiple times reported since version 1.17. Unfortunately the paperless devs says more often "It is an upstream problem, we can't do anything." From a user perspective this is a bad approach, because less people will use a product if it doesn't work.

Why can't we at least have a dependency rollback or freeze for the official docker image? If there are no security issues which were fixed by the "faulty" dependency packages, this should be no problem.
Can we manually downgrade affected components inside docker-environment?
#4056 #3933 #4004

I've asked for people who can reproduce it [to try commands](https://github.com/paperless-ngx/paperless-ngx/issues/3933#issuecomment-1668801085), but have never gotten a response.
People uploaded this file for you to reproduce:
https://github.com/paperless-ngx/paperless-ngx/files/12289624/Allgemeine.Versicherungsbedingungen.pdf

https://github.com/paperless-ngx/paperless-ngx/issues/3933#issuecomment-1667995519

https://github.com/paperless-ngx/paperless-ngx/issues/3933#issuecomment-1666888667
different example file

> People uploaded this file for you to reproduce: https://github.com/paperless-ngx/paperless-ngx/files/12289624/Allgemeine.Versicherungsbedingungen.pdf

I can reproduce the error with this file on Paperless-NGX 1.17.2
I tried to set the following option inside docker-compose file:
PAPERLESS_OCR_USER_ARGS=: '{"unpaper_args": "--continue-on-soft-render-error"}'
This doesn't have any effect. Is this option correct?
Ghostscript reports:
```
The following warnings were encountered at least once while processing this file:
        bad trailer dictionary
        Name tree node missing required Limits entry


 This file had errors that were repaired or ignored.

 The file was produced by:

 >>>> Adobe PDF Library 16.0.7 <<<<

 Please notify the author of the software that produced this

 file that it does not conform to Adobe's published PDF

 specification.
```

qpdf reports:
```
File is not encrypted
File is linearized
WARNING: input.pdf: linearized file contains an uncompressed object after a compressed one in a cross-reference stream
WARNING: input.pdf: first shared object offset mismatch: hint table = 425894; computed = 425181
WARNING: input.pdf: page 0 has shared identifier entries
WARNING: input.pdf: page 0: shared object 3289: in hint table but not computed list
WARNING: input.pdf: incorrect offset in outlines table: hint table = 474775; computed = 474062
qpdf: operation succeeded with warnings
```

Setting `PAPERLESS_OCR_USER_ARGS: '{"continue_on_soft_render_error": true}'` allows the file to process, but I'm not inclined to make it a default, given it is telling Ghostscript to forge ahead even when the output is likely to be malformed/not render right.
After setting the env-var 
```
PAPERLESS_OCR_USER_ARGS: '{"continue_on_soft_render_error": true}'
``` 
the file can be imported. So what should I do - let this option defined? What are the consequences of this definition?
> The PDF interpreter normally tries to repair, or ignore, all problems encountered in PDF files. Setting -dPDFSTOPONERROR instead causes the interpreter to signal an error and stop processing the PDF file, instead of printing a warning.

So Ghostscript will try to work around rendering issues, which it may not do correctly, so the archive file may have issue with images, text, layout, etc, but will otherwise be produced.

Resolving patch:
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -304,6 +304,7 @@ def parse(self, document_path: Path, mime_type, file_name=None):
         import ocrmypdf
         from ocrmypdf import EncryptedPdfError
         from ocrmypdf import InputFileError
+        from ocrmypdf import SubprocessOutputError
 
         archive_path = Path(os.path.join(self.tempdir, "archive.pdf"))
         sidecar_file = Path(os.path.join(self.tempdir, "sidecar.txt"))
@@ -333,6 +334,13 @@ def parse(self, document_path: Path, mime_type, file_name=None):
             )
             if original_has_text:
                 self.text = text_original
+        except SubprocessOutputError as e:
+            if "Ghostscript PDF/A rendering" in str(e):
+                self.log.warning(
+                    "Ghostscript PDF/A rendering failed, consider setting "
+                    "PAPERLESS_OCR_USER_ARGS: '{\"continue_on_soft_render_error\": true}'",  # noqa: E501
+                )
+            raise e
         except (NoTextFoundException, InputFileError) as e:
             self.log.warning(
                 f"Encountered an error while running OCR: {e!s}. "


Test patch: