Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Hebrew not treated as RTL when there is a text in the PDF
### Description

When PDF containing Hebrew text is OCRed the text in "Content" window is right-to-left. (Although aligned to left. Would be nice if this will be fixed and text will be aligned to right)  
When PDF already contains text and this text is taken as-is, the text is not treated as RTL, i.e. all characters in words are reversed.  
This causes major problems as the search will not find properly typed words. To use search I need to reverse the order of characters :(  
Attaching PDF in Hebrew as example
[mapa.pdf](https://github.com/paperless-ngx/paperless-ngx/files/10328954/mapa.pdf)


### Steps to reproduce

1. Upload a PDF containing text in Hebrew
2. Go to "Content" tab
3. Characters order reversed

### Webserver logs

```bash
Not relevant
```


### Browser logs

_No response_

### Paperless-ngx version

1.11.2

### Host OS

Fedora 35 x86_64

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Ah.

Well one immediate problem here is that the UI currently has no knowledge of the fact that the document is RTL. I suppose we will need to update the /documents API endpoint with a field for this, which I was about to do but actually, the document itself doesnt even retain this info. We could `langdetect` the content but this seems costly, I presume it would be better to retain the info during the consumption process, but Im not sure of the best approach to that, anyone else have thoughts?

If the API can return something like 'is_rtl' (or the language of course) fixing the frontend part is trivial.
@shamoon Why langdetect is costly? I understand you need only a small portion of the text for it. And this could be an optional feature as well (turned on only by users who know they have documents in RTL language).  
Anyway. I would guess the simplest solution is to add in the configuration (env or GUI) option to specify that the default text is RTL. Then in the document editing GUI add an option to overwrite this.  
But I would prefer an automatic solution very much.
@shamoon So tesseract already analyzes text in PDF and detects language properly.  It seems like you only need to save this information to the database and then use it in GUI?
Yes, and I've come back around to that being the only viable solution here, see my [comment above](https://github.com/paperless-ngx/paperless-ngx/issues/2301#issuecomment-1368473087)

Will wait to see if folks who work more on the backend have thoughts about how best to achieve this.

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -29,6 +29,7 @@
 from django.views.generic import TemplateView
 from django_filters.rest_framework import DjangoFilterBackend
 from documents.tasks import consume_file
+from langdetect import detect
 from packaging import version as packaging_version
 from paperless import version
 from paperless.db import GnuPG
@@ -325,6 +326,13 @@ def metadata(self, request, pk=None):
             "original_filename": doc.original_filename,
         }
 
+        lang = "en"
+        try:
+            lang = detect(doc.content)
+        except Exception:
+            pass
+        meta["lang"] = lang
+
         if doc.has_archive_version:
             meta["archive_size"] = self.get_filesize(doc.archive_path)
             meta["archive_metadata"] = self.get_metadata(
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -2,6 +2,7 @@
 import os
 import re
 import subprocess
+import tempfile
 from pathlib import Path
 from typing import Optional
 
@@ -137,36 +138,27 @@ def extract_text(self, sidecar_file: Optional[Path], pdf_file: Path):
         if not os.path.isfile(pdf_file):
             return None
 
-        from pdfminer.high_level import extract_text as pdfminer_extract_text
-
         try:
-            stripped = post_process_text(pdfminer_extract_text(pdf_file))
-
-            self.log("debug", f"Extracted text from PDF file {pdf_file}")
-
-            # pdfminer.six does not handle RTL text
-            # as a hack, for some languages, return no text, to force
-            # OCRMyPdf/Tesseract do handle this correctly
-            from langdetect import detect
-
-            lang = detect(stripped)
-
-            self.log("debug", f"Detected language {lang}")
-
-            if (
-                lang
-                in {
-                    "ar",  # Arabic
-                    "he",  # Hebrew,
-                    "fa",  # Persian
-                }
-                and pdf_file.name != "archive-fallback.pdf"
-            ):
-                raise RtlLanguageException()
-            return stripped
-        except RtlLanguageException:
-            self.log("warning", f"Detected RTL language {lang}")
-            return None
+            text = None
+            with tempfile.NamedTemporaryFile(
+                mode="w+",
+                dir=self.tempdir,
+            ) as tmp:
+                subprocess.run(
+                    [
+                        "pdftotext",
+                        "-q",
+                        "-layout",
+                        "-enc",
+                        "UTF-8",
+                        pdf_file,
+                        tmp.name,
+                    ],
+                )
+                text = tmp.read()
+
+            return post_process_text(text)
+
         except Exception:
             # TODO catch all for various issues with PDFminer.six.
             #  If PDFminer fails, fall back to OCR.
@@ -342,7 +334,7 @@ def parse(self, document_path: Path, mime_type, file_name=None):
             )
             if original_has_text:
                 self.text = text_original
-        except (NoTextFoundException, RtlLanguageException, InputFileError) as e:
+        except (NoTextFoundException, InputFileError) as e:
             self.log(
                 "warning",
                 f"Encountered an error while running OCR: {str(e)}. "


Test patch:
diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py
--- a/src/paperless_tesseract/tests/test_parser.py
+++ b/src/paperless_tesseract/tests/test_parser.py
@@ -661,28 +661,14 @@ def test_rtl_language_detection(self):
             - Text from the document is extracted
         """
         parser = RasterisedDocumentParser(None)
-        with mock.patch.object(
-            parser,
-            "construct_ocrmypdf_parameters",
-            wraps=parser.construct_ocrmypdf_parameters,
-        ) as wrapped:
 
-            parser.parse(
-                os.path.join(self.SAMPLE_FILES, "rtl-test.pdf"),
-                "application/pdf",
-            )
-
-            # There isn't a good way to actually check this working, with RTL correctly return
-            #  as it would require tesseract-ocr-ara installed for everyone running the
-            #  test suite.  This test does provide the coverage though and attempts to ensure
-            # the force OCR happens
-            self.assertIsNotNone(parser.get_text())
+        parser.parse(
+            os.path.join(self.SAMPLE_FILES, "rtl-test.pdf"),
+            "application/pdf",
+        )
 
-            self.assertEqual(parser.construct_ocrmypdf_parameters.call_count, 2)
-            # Check the last call kwargs
-            self.assertTrue(
-                parser.construct_ocrmypdf_parameters.call_args.kwargs["safe_fallback"],
-            )
+        # Copied from the PDF to here.  Don't even look at it
+        self.assertIn("ةﯾﻠﺧﺎدﻻ ةرازو", parser.get_text())
 
 
 class TestParserFileTypes(DirectoriesMixin, TestCase):