Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] OCR not working for documents in Hebrew
### Description

When a document in Hebrew is uploaded, OCR results are empty

### Steps to reproduce

1. Document in Hebrew uploaded
2. Content field is empty
3. Example image for OCR attached

### Webserver logs

```bash
[2022-12-30 20:17:51,803] [INFO] [paperless.consumer] Consuming test.jpg
[2022-12-30 20:17:51,804] [DEBUG] [paperless.consumer] Detected mime type: image/jpeg
[2022-12-30 20:17:51,804] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-12-30 20:17:51,806] [DEBUG] [paperless.consumer] Parsing test.jpg...
[2022-12-30 20:17:51,858] [DEBUG] [paperless.parsing.tesseract] Estimated DPI 205 based on image width 1701
[2022-12-30 20:17:51,858] [DEBUG] [paperless.parsing.tesseract] Detected DPI for image /tmp/paperless/paperless-upload-8m7_wlf1: 120
[2022-12-30 20:17:51,858] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-upload-8m7_wlf1'), 'output_file': '/tmp/paperless/paperless-exhki7iq/archive.pdf', 'use_threads': True, 'jobs': '1', 'language': 'heb+eng', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean_final': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'pages': '1-10', 'image_dpi': 120}
[2022-12-30 20:17:54,171] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-exhki7iq/archive.pdf
[2022-12-30 20:17:54,383] [DEBUG] [paperless.parsing.tesseract] Detected language he
[2022-12-30 20:17:54,383] [WARNING] [paperless.parsing.tesseract] Detected RTL language he
[2022-12-30 20:17:54,383] [WARNING] [paperless.parsing.tesseract] Encountered an error while running OCR: No text was found in the original document. Attempting force OCR to get the text.
[2022-12-30 20:17:54,384] [DEBUG] [paperless.parsing.tesseract] Estimated DPI 205 based on image width 1701
[2022-12-30 20:17:54,384] [DEBUG] [paperless.parsing.tesseract] Detected DPI for image /tmp/paperless/paperless-upload-8m7_wlf1: 120
[2022-12-30 20:17:54,384] [DEBUG] [paperless.parsing.tesseract] Fallback: Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-upload-8m7_wlf1'), 'output_file': '/tmp/paperless/paperless-exhki7iq/archive-fallback.pdf', 'use_threads': True, 'jobs': '1', 'language': 'heb+eng', 'output_type': 'pdfa', 'progress_bar': False, 'force_ocr': True, 'clean_final': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'pages': '1-10', 'image_dpi': 120}
[2022-12-30 20:17:56,679] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-exhki7iq/archive-fallback.pdf
[2022-12-30 20:17:56,682] [DEBUG] [paperless.parsing.tesseract] Detected language he
[2022-12-30 20:17:56,682] [WARNING] [paperless.parsing.tesseract] Detected RTL language he
[2022-12-30 20:17:56,682] [WARNING] [paperless.parsing.tesseract] No text was found in /tmp/paperless/paperless-upload-8m7_wlf1, the content will be empty.
[2022-12-30 20:17:56,682] [DEBUG] [paperless.consumer] Generating thumbnail for test.jpg...
[2022-12-30 20:17:56,685] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient /tmp/paperless/paperless-exhki7iq/archive.pdf[0] /tmp/paperless/paperless-exhki7iq/convert.webp
[2022-12-30 20:17:57,731] [DEBUG] [paperless.consumer] Saving record to database
[2022-12-30 20:17:57,732] [DEBUG] [paperless.consumer] Creation date from st_mtime: 2022-12-30 20:17:51+02:00
[2022-12-30 20:17:57,739] [INFO] [paperless.handlers] Assigning document type receipts to 2022-12-30 test
[2022-12-30 20:17:57,764] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-upload-8m7_wlf1
[2022-12-30 20:17:57,771] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-exhki7iq
[2022-12-30 20:17:57,771] [INFO] [paperless.consumer] Document 2022-12-30 test consumption finished
```


### Browser logs

_No response_

### Paperless-ngx version

1.11.1

### Host OS

Fedora 35 x86_64

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

PAPERLESS_OCR_LANGUAGES=eng heb
APERLESS_OCR_LANGUAGE=heb+eng
PAPERLESS_FILENAME_FORMAT={document_type}/{created_year}/{created_month}/{tag_list}_{title}
PAPERLESS_OCR_MODE=skip
PAPERLESS_OCR_CLEAN=clean-final
PAPERLESS_OCR_PAGES=10
PAPERLESS_IGNORE_DATES="31-12-1975"
PAPERLESS_TASK_WORKERS=1
PAPERLESS_THREADS_PER_WORKER=1
PAPERLESS_WABSERVER_WORKERS=1
PAPERLESS_FILENAME_DATE_ORDER="DMY"
PAPERLESS_NUMBER_OF_SUGGESTED_DATES=5
PAPERLESS_DATE_ORDER="DMY"
PAPERLESS_ENABLE_UPDATE_CHECK=true

### Other

_No response_

![test](https://user-images.githubusercontent.com/9785479/210101161-af79856d-3632-44d3-ab46-3e0edb37e1f2.jpg)



Discussion:
Based on the logs, tesseract is not finding any text when doing the OCR against the file.

It's no doubt related to the attempt to fix RTL languages which are digitally born, as the text from those comes out not RTL.
> Based on the logs, tesseract is not finding any text when doing the OCR against the file.
> 
> It's no doubt related to the attempt to fix RTL languages which are digitally born, as the text from those comes out not RTL.

@stumpylog Not sure what you mean, but according to documentation tesseract supports Hebrew. So what is the problem?
I don't know why tesseract might be reporting no text found in one line.  Do you have a sample file I can experiment with?
@stumpylog Yes. Picture is attached to the original post.
Oh, so that attached image is an actual document.  I thought it was a screenshot of paperless
Matching your OCR settings above, notably `skip`, I'm getting lots of content when I upload the image attached.  I can't tell if it's 100% correct content, but it generally looks correct and the RTL nature of the text also appears correct.

Resolving patch:
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -1,6 +1,8 @@
 import json
 import os
 import re
+from pathlib import Path
+from typing import Optional
 
 from django.conf import settings
 from documents.parsers import DocumentParser
@@ -99,7 +101,7 @@ def calculate_a4_dpi(self, image):
             self.log("warning", f"Error while calculating DPI for image {image}: {e}")
             return None
 
-    def extract_text(self, sidecar_file, pdf_file):
+    def extract_text(self, sidecar_file: Optional[Path], pdf_file: Path):
         # When re-doing OCR, the sidecar contains ONLY the new text, not
         # the whole text, so do not utilize it in that case
         if (
@@ -139,11 +141,15 @@ def extract_text(self, sidecar_file, pdf_file):
 
             self.log("debug", f"Detected language {lang}")
 
-            if lang in {
-                "ar",  # Arabic
-                "he",  # Hebrew,
-                "fa",  # Persian
-            }:
+            if (
+                lang
+                in {
+                    "ar",  # Arabic
+                    "he",  # Hebrew,
+                    "fa",  # Persian
+                }
+                and pdf_file.name != "archive-fallback.pdf"
+            ):
                 raise RtlLanguageException()
             return stripped
         except RtlLanguageException:
@@ -275,7 +281,7 @@ def construct_ocrmypdf_parameters(
 
         return ocrmypdf_args
 
-    def parse(self, document_path, mime_type, file_name=None):
+    def parse(self, document_path: Path, mime_type, file_name=None):
         # This forces tesseract to use one core per page.
         os.environ["OMP_THREAD_LIMIT"] = "1"
 
@@ -300,8 +306,8 @@ def parse(self, document_path, mime_type, file_name=None):
         import ocrmypdf
         from ocrmypdf import InputFileError, EncryptedPdfError
 
-        archive_path = os.path.join(self.tempdir, "archive.pdf")
-        sidecar_file = os.path.join(self.tempdir, "sidecar.txt")
+        archive_path = Path(os.path.join(self.tempdir, "archive.pdf"))
+        sidecar_file = Path(os.path.join(self.tempdir, "sidecar.txt"))
 
         args = self.construct_ocrmypdf_parameters(
             document_path,
@@ -335,8 +341,12 @@ def parse(self, document_path, mime_type, file_name=None):
                 f"Attempting force OCR to get the text.",
             )
 
-            archive_path_fallback = os.path.join(self.tempdir, "archive-fallback.pdf")
-            sidecar_file_fallback = os.path.join(self.tempdir, "sidecar-fallback.txt")
+            archive_path_fallback = Path(
+                os.path.join(self.tempdir, "archive-fallback.pdf"),
+            )
+            sidecar_file_fallback = Path(
+                os.path.join(self.tempdir, "sidecar-fallback.txt"),
+            )
 
             # Attempt to run OCR with safe settings.
 


Test patch: