Repository: paperless-ngx/paperless-ngx

Issue:
issue UnicodeEncodeError: 'utf-8' codec can't encode character '\udcb7' in position 18: surrogates not allowed
### Description

My Docker-compose file is:
```
root@rene:~# cat paperless-compose.yml
version: "3.4"
services:
  broker:
    image: docker.io/library/redis:7
    restart: unless-stopped
    volumes:
      - redisdata:/data

  db:
    image: docker.io/library/postgres:13
    restart: unless-stopped
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: paperless
      POSTGRES_PASSWORD: paperless

  webserver:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    restart: unless-stopped
    depends_on:
      - db
      - broker
    ports:
      - "8010:8000"
    healthcheck:
      test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - data:/usr/src/paperless/data
      - media:/usr/src/paperless/media
      - ./export:/usr/src/paperless/export
      - /mnt/paperless_smb:/usr/src/paperless/consume
    environment:
      PAPERLESS_REDIS: redis://broker:6379
      PAPERLESS_DBHOST: db
      USERMAP_UID: 1000
      USERMAP_GID: 100
      PAPERLESS_TIME_ZONE: Europe/Berlin
      PAPERLESS_SECRET_KEY: "a-secret-key"
      PAPERLESS_URL: http://localhost:8010
      PAPERLESS_CSRF_TRUSTED_ORIGINS: http://localhost:8010
      PAPERLESS_CORS_ALLOWED_HOSTS: http://localhost:8010
      PAPERLESS_ADMIN_USER: admin
      PAPERLESS_ADMIN_PASSWORD: admin
      PAPERLESS_ADMIN_MAIL: my-mail@domain.de
volumes:
  data:
  media:
  pgdata:
  redisdata:
```

on  - /mnt/paperless_smb is my nas content mountebd about fstab:
# UNCONFIGURED FSTAB FOR BASE SYSTEM
//192.168.178.146/home/Drive/Dokumente /mnt/paperless_smb cifs credentials=/etc/smb_credentials,uid=100

paperless didt search recursive, cant activate it

### Steps to reproduce

after each restart the issue on log

### Webserver logs

```bash
[2023-05-07 11:45:08,428] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/Scan 07 Apr. 23 083731.pdf to the task queue.

[2023-05-07 11:45:08,567] [ERROR] [paperless.handlers] Creating PaperlessTask failed

Traceback (most recent call last):

  File "/usr/src/paperless/src/documents/signals/handlers.py", line 535, in before_task_publish_handler

    PaperlessTask.objects.create(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 671, in create

    obj.save(force_insert=True, using=self.db)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 812, in save

    self.save_base(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 863, in save_base

    updated = self._save_table(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1006, in _save_table

    results = self._do_insert(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1047, in _do_insert

    return manager._insert(

  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method

    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1791, in _insert

    return query.get_compiler(using=using).execute_sql(returning_fields)

  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql

    cursor.execute(sql, params)

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute

    return self._execute_with_wrappers(

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers

    return executor(sql, params, many, context)

  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute

    return self.cursor.execute(sql, params)

UnicodeEncodeError: 'utf-8' codec can't encode character '\udcb7' in position 18: surrogates not allowed

[2023-05-07 11:45:08,575] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/ER.xlsx: Unknown file extension.

[2023-05-07 11:45:08,575] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/Mappe.xlsx: Unknown file extension.

[2023-05-07 11:45:08,576] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/Dokument (1).docx: Unknown file extension.

[2023-05-07 11:45:08,576] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/Dokument.docx: Unknown file extension.

[2023-05-07 11:45:08,576] [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/consume/Kndigung.docx: Unknown file extension.

[2023-05-07 11:45:08,581] [INFO] [paperless.management.consumer] Using inotify to watch directory for changes: /usr/src/paperless/consume
```


### Browser logs

_No response_

### Paperless-ngx version

lastest

### Host OS

Ubuntu WSL2

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

see above docker compose

### Other

_No response_


Discussion:
Highly likely to be something about using WSL.  But without a sample for generating the error to look into it more, there's not much I can think of
I don't know what information you need? paperless-ngx runs in a docker container on the wsl2. Whether it's wsl2 or a root server doesn't matter, does it? my paperless-ngx doesn't support .doc files either. I can't find a config file in the container under /usr/ to enable .doc support either. It is a bit confusing
A sample file that produces the error
It is not so easy to find a file. i have 1464 errors on import.
for example hundreds of errors:
'utf-8' codec can't encode character
the file is a .txt file....

Also many "normal" .pdf files can not be read.

![image](https://github.com/paperless-ngx/paperless-ngx/assets/10163536/7cbe96b2-2cba-4604-a302-e669ee0f974e)
[bildquellen.txt](https://github.com/paperless-ngx/paperless-ngx/files/11455831/bildquellen.txt)



![image](https://github.com/paperless-ngx/paperless-ngx/assets/10163536/f9f944b1-d9c5-44fc-b5a2-07ecce700b0c)

This can't be normal. So many .pdf files and .doc files and .txt files. How can it have problems? How can I retrofit this? Is there something wrong with the compose file?
The same file you provided isn't valid UTF-8.  For example on lines 5, 48 and 59, a text editor will show a ? symbol, owing to the partial symbol.  It looks like this:
![image](https://github.com/paperless-ngx/paperless-ngx/assets/797416/134fddba-f84f-4055-95b5-206846b27b47)

Looking at the bytes of the file, it looks like `Pantothens\xe4ure`, which isn't valid.

At a guess, whatever is producing this files is attempting to insert some sort of symbol, but only doing part of it, resulting in an invalid string.

Do you happen to have a PDF which also exhibits the error?

Resolving patch:
diff --git a/src/documents/parsers.py b/src/documents/parsers.py
--- a/src/documents/parsers.py
+++ b/src/documents/parsers.py
@@ -7,6 +7,7 @@
 import subprocess
 import tempfile
 from functools import lru_cache
+from pathlib import Path
 from typing import Iterator
 from typing import Match
 from typing import Optional
@@ -319,6 +320,18 @@ def progress(self, current_progress, max_progress):
         if self.progress_callback:
             self.progress_callback(current_progress, max_progress)
 
+    def read_file_handle_unicode_errors(self, filepath: Path) -> str:
+        """
+        Helper utility for reading from a file, and handling a problem with its
+        unicode, falling back to ignoring the error to remove the invalid bytes
+        """
+        try:
+            text = filepath.read_text(encoding="utf-8")
+        except UnicodeDecodeError as e:
+            self.log("warning", f"Unicode error during text reading, continuing: {e}")
+            text = filepath.read_bytes().decode("utf-8", errors="replace")
+        return text
+
     def extract_metadata(self, document_path, mime_type):
         return []
 
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -122,8 +122,7 @@ def extract_text(self, sidecar_file: Optional[Path], pdf_file: Path):
             and os.path.isfile(sidecar_file)
             and settings.OCR_MODE != "redo"
         ):
-            with open(sidecar_file) as f:
-                text = f.read()
+            text = self.read_file_handle_unicode_errors(sidecar_file)
 
             if "[OCR skipped on page" not in text:
                 # This happens when there's already text in the input file.
@@ -155,7 +154,7 @@ def extract_text(self, sidecar_file: Optional[Path], pdf_file: Path):
                         tmp.name,
                     ],
                 )
-                text = tmp.read()
+                text = self.read_file_handle_unicode_errors(Path(tmp.name))
 
             return post_process_text(text)
 
diff --git a/src/paperless_text/parsers.py b/src/paperless_text/parsers.py
--- a/src/paperless_text/parsers.py
+++ b/src/paperless_text/parsers.py
@@ -16,11 +16,7 @@ class TextDocumentParser(DocumentParser):
     logging_name = "paperless.parsing.text"
 
     def get_thumbnail(self, document_path, mime_type, file_name=None):
-        def read_text():
-            with open(document_path) as src:
-                lines = [line.strip() for line in src.readlines()]
-                text = "\n".join(lines[:50])
-                return text
+        text = self.read_file_handle_unicode_errors(document_path)
 
         img = Image.new("RGB", (500, 700), color="white")
         draw = ImageDraw.Draw(img)
@@ -29,7 +25,7 @@ def read_text():
             size=20,
             layout_engine=ImageFont.Layout.BASIC,
         )
-        draw.text((5, 5), read_text(), font=font, fill="black")
+        draw.text((5, 5), text, font=font, fill="black")
 
         out_path = os.path.join(self.tempdir, "thumb.webp")
         img.save(out_path, format="WEBP")
@@ -37,5 +33,4 @@ def read_text():
         return out_path
 
     def parse(self, document_path, mime_type, file_name=None):
-        with open(document_path) as f:
-            self.text = f.read()
+        self.text = self.read_file_handle_unicode_errors(document_path)


Test patch:
diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py
--- a/src/paperless_tesseract/tests/test_parser.py
+++ b/src/paperless_tesseract/tests/test_parser.py
@@ -2,6 +2,7 @@
 import shutil
 import tempfile
 import uuid
+from pathlib import Path
 from typing import ContextManager
 from unittest import mock
 
@@ -39,7 +40,7 @@ def __enter__(self):
 
 
 class TestParser(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
-    SAMPLE_FILES = os.path.join(os.path.dirname(__file__), "samples")
+    SAMPLE_FILES = Path(__file__).resolve().parent / "samples"
 
     def assertContainsStrings(self, content, strings):
         # Asserts that all strings appear in content, in the given order.
@@ -77,7 +78,7 @@ def test_get_text_from_pdf(self):
         parser = RasterisedDocumentParser(uuid.uuid4())
         text = parser.extract_text(
             None,
-            os.path.join(self.SAMPLE_FILES, "simple-digital.pdf"),
+            self.SAMPLE_FILES / "simple-digital.pdf",
         )
 
         self.assertContainsStrings(text.strip(), ["This is a test document."])
diff --git a/src/paperless_text/tests/samples/decode_error.txt b/src/paperless_text/tests/samples/decode_error.txt
new file mode 100644
--- /dev/null
+++ b/src/paperless_text/tests/samples/decode_error.txt
@@ -0,0 +1 @@
+Pantothens�ure
diff --git a/src/paperless_text/tests/test_parser.py b/src/paperless_text/tests/test_parser.py
--- a/src/paperless_text/tests/test_parser.py
+++ b/src/paperless_text/tests/test_parser.py
@@ -1,4 +1,4 @@
-import os
+from pathlib import Path
 
 from django.test import TestCase
 
@@ -8,12 +8,14 @@
 
 
 class TestTextParser(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
+    SAMPLE_DIR = Path(__file__).resolve().parent / "samples"
+
     def test_thumbnail(self):
         parser = TextDocumentParser(None)
 
         # just make sure that it does not crash
         f = parser.get_thumbnail(
-            os.path.join(os.path.dirname(__file__), "samples", "test.txt"),
+            self.SAMPLE_DIR / "test.txt",
             "text/plain",
         )
         self.assertIsFile(f)
@@ -22,9 +24,29 @@ def test_parse(self):
         parser = TextDocumentParser(None)
 
         parser.parse(
-            os.path.join(os.path.dirname(__file__), "samples", "test.txt"),
+            self.SAMPLE_DIR / "test.txt",
             "text/plain",
         )
 
         self.assertEqual(parser.get_text(), "This is a test file.\n")
         self.assertIsNone(parser.get_archive_path())
+
+    def test_parse_invalid_bytes(self):
+        """
+        GIVEN:
+            - Text file which contains invalid UTF bytes
+        WHEN:
+            - The file is parsed
+        THEN:
+            - Parsing continues
+            - Invalid bytes are removed
+        """
+        parser = TextDocumentParser(None)
+
+        parser.parse(
+            self.SAMPLE_DIR / "decode_error.txt",
+            "text/plain",
+        )
+
+        self.assertEqual(parser.get_text(), "Pantothens�ure\n")
+        self.assertIsNone(parser.get_archive_path())