Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] RTL Import is reversed
### Description

When importing an RTL (arabic) document, the content is reversed. 

I.e, if it is the following:
 إعادة اصدار لوحة المركبة
Paperless imports it as:
ﺔﺒﻛﺮﻤﻟﺍ ﺔﺣﻮﻟ ﺭﺍﺪﺻﺍ ﺓﺩﺎﻋﺇ

I believe this could potentially be related to the following issue from the original paperless-ng repo: https://github.com/jonaswinkler/paperless-ng/pull/627

### Steps to reproduce

1. Import an RTL (Arabic in my case) document
2. Check the content within Paperless
3. Will be reversed

### Webserver logs

_No response_

### Paperless-ngx version

1.7.1

### Host OS

Unraid 6.10.3

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Reviewing that linked issue it seems like not every doc is reversed, that it might be dependent on how the pdf was authored. Is that the case for you? That PR is just for tests but at the end it looks like there was some brainstorming about a possible solution. Not in my wheelhouse though
> Reviewing that linked issue it seems like not every doc is reversed, that it might be dependent on how the pdf was authored. Is that the case for you? That PR is just for tests but at the end it looks like there was some brainstorming about a possible solution. Not in my wheelhouse though

I'm not entirely sure, will test it a bit more though, but it appears when there is existing OCR / it's a digitally created document, the "extraction" of that text is when the issue appears. (In that issue, pdfminer.six)

When the document has no text, just a pure scan and OCR needs to be triggered, it appears to work as intended. I'll do some more testing in a few and try and confirm my theory
Sorry, misclicked the close button 
> Is that the case for you?

Seems to be any digitally created RTL documents. Scanned ones work fine.
Do you have a sample document we could use for testing?  Or a way to generate one?  I'm not at all familiar with RTL, but I'm guessing it is something about the sidecar file which probably exists for generated vs scanned documents.

> Do you have a sample document we could use for testing? Or a way to generate one? I'm not at all familiar with RTL, but I'm guessing it is something about the sidecar file which probably exists for generated vs scanned documents.

I've attached a sample document. 

I have tested a bit, and the behavior is also duplicated if you create a word file with RTL content, and save that as PDF."

[test.pdf](https://github.com/paperless-ngx/paperless-ngx/files/9131710/test.pdf)


Thank you.  It certainly looks reversed to me.  Interestingly, this document appears to be OCRd with tesseract.  We'll try to find a fix, though it could always be somewhere in a dependency.
Hello, any ETA on a fix?
Unfortunately, the best fix requires a library to update: pdfminer/pdfminer.six/issues/515

It doesn't appear to be being work sadly, so it may not happen for some time.

Resolving patch:
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -13,6 +13,10 @@ class NoTextFoundException(Exception):
     pass
 
 
+class RtlLanguageException(Exception):
+    pass
+
+
 class RasterisedDocumentParser(DocumentParser):
     """
     This parser uses Tesseract to try and get some text out of a rasterised
@@ -125,7 +129,26 @@ def extract_text(self, sidecar_file, pdf_file):
             stripped = post_process_text(pdfminer_extract_text(pdf_file))
 
             self.log("debug", f"Extracted text from PDF file {pdf_file}")
+
+            # pdfminer.six does not handle RTL text
+            # as a hack, for some languages, return no text, to force
+            # OCRMyPdf/Tesseract do handle this correctly
+            from langdetect import detect
+
+            lang = detect(stripped)
+
+            self.log("debug", f"Detected language {lang}")
+
+            if lang in {
+                "ar",  # Arabic
+                "he",  # Hebrew,
+                "fa",  # Persian
+            }:
+                raise RtlLanguageException()
             return stripped
+        except RtlLanguageException:
+            self.log("warning", f"Detected RTL language {lang}")
+            return None
         except Exception:
             # TODO catch all for various issues with PDFminer.six.
             #  If PDFminer fails, fall back to OCR.
@@ -305,7 +328,7 @@ def parse(self, document_path, mime_type, file_name=None):
             )
             if original_has_text:
                 self.text = text_original
-        except (NoTextFoundException, InputFileError) as e:
+        except (NoTextFoundException, RtlLanguageException, InputFileError) as e:
             self.log(
                 "warning",
                 f"Encountered an error while running OCR: {str(e)}. "


Test patch:
diff --git a/src/paperless_tesseract/tests/samples/rtl-test.pdf b/src/paperless_tesseract/tests/samples/rtl-test.pdf
new file mode 100755
Binary files /dev/null and b/src/paperless_tesseract/tests/samples/rtl-test.pdf differ
diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py
--- a/src/paperless_tesseract/tests/test_parser.py
+++ b/src/paperless_tesseract/tests/test_parser.py
@@ -588,6 +588,39 @@ def test_ocrmypdf_parameters(self):
             params = parser.construct_ocrmypdf_parameters("", "", "", "")
             self.assertNotIn("deskew", params)
 
+    def test_rtl_language_detection(self):
+        """
+        GIVEN:
+            - File with text in an RTL language
+        WHEN:
+            - Document is parsed
+        THEN:
+            - Text from the document is extracted
+        """
+        parser = RasterisedDocumentParser(None)
+        with mock.patch.object(
+            parser,
+            "construct_ocrmypdf_parameters",
+            wraps=parser.construct_ocrmypdf_parameters,
+        ) as wrapped:
+
+            parser.parse(
+                os.path.join(self.SAMPLE_FILES, "rtl-test.pdf"),
+                "application/pdf",
+            )
+
+            # There isn't a good way to actually check this working, with RTL correctly return
+            #  as it would require tesseract-ocr-ara installed for everyone running the
+            #  test suite.  This test does provide the coverage though and attempts to ensure
+            # the force OCR happens
+            self.assertIsNotNone(parser.get_text())
+
+            self.assertEqual(parser.construct_ocrmypdf_parameters.call_count, 2)
+            # Check the last call kwargs
+            self.assertTrue(
+                parser.construct_ocrmypdf_parameters.call_args.kwargs["safe_fallback"],
+            )
+
 
 class TestParserFileTypes(DirectoriesMixin, TestCase):