Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Imported email with broken UTF-8 encoded subject line is not displayed correctly
### Description

If the subject line of an email is incorrectly encoded in UTF-8, Paperless-ngx cannot display it correctly. The UI reports an "Error retrieving metadata"[^1] and the preview pane keeps displaying the "Loading..." message.

By clicking the download button you download the archived PDF, which looks correct, instead of the EML file. The dropdown with "Download original" is missing.

[redacted_example.eml.txt](https://github.com/paperless-ngx/paperless-ngx/files/14223011/redacted_example.eml.txt)

[^1]:  `
{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":500,"statusText":"Internal Server Error","url":"http://dms/api/documents/2034/metadata/","ok":false,"name":"HttpErrorResponse","message":"Http failure response for http://dms/api/documents/2034/metadata/: 500 Internal Server Error","error":"\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Server Error (500)</title>\n</head>\n<body>\n  <h1>Server Error (500)</h1><p></p>\n</body>\n</html>\n"}
`

### Steps to reproduce

1. Remove the .txt suffix from the supplied email file and import it
2. Open the imported document

### Webserver logs

```bash
webserver_1  | [2024-02-09 15:32:41,572] [ERROR] [django.request] Internal Server Error: /api/documents/2034/metadata/
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
webserver_1  |     raise exc_info[1]
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
webserver_1  |     response = await get_response(request)
webserver_1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
webserver_1  |     raise exc_info[1]
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 284, in _get_response_async
webserver_1  |     response = await sync_to_async(
webserver_1  |                ^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
webserver_1  |     ret: _R = await loop.run_in_executor(
webserver_1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
webserver_1  |     result = self.fn(*self.args, **self.kwargs)
webserver_1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
webserver_1  |     return func(*args, **kwargs)
webserver_1  |            ^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/django/template/response.py", line 114, in render
webserver_1  |     self.content = self.rendered_content
webserver_1  |                    ^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/rest_framework/response.py", line 70, in rendered_content
webserver_1  |     ret = renderer.render(self.data, accepted_media_type, context)
webserver_1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
webserver_1  |   File "/usr/local/lib/python3.11/site-packages/rest_framework/renderers.py", line 109, in render
webserver_1  |     return ret.encode()
webserver_1  |            ^^^^^^^^^^^^
webserver_1  | UnicodeEncodeError: 'utf-8' codec can't encode characters in position 870-871: surrogates not allowed
```


### Browser logs

```bash
XHRGET
http://dms/api/documents/2034/metadata/
[HTTP/1.1 500 Internal Server Error 107ms]

	
GET
	http://dms/api/documents/2034/metadata/
Status
500
Internal Server Error
VersionHTTP/1.1
Transferred531 B (145 B size)
Referrer Policysame-origin
DNS ResolutionSystem

    	
    content-language
    	en-gb
    content-length
    	145
    content-type
    	text/html; charset=utf-8
    cross-origin-opener-policy
    	same-origin
    date
    	Fri, 09 Feb 2024 14:38:10 GMT
    referrer-policy
    	same-origin
    server
    	uvicorn
    vary
    	Accept-Language, origin, Cookie
    x-api-version
    	4
    x-content-type-options
    	nosniff
    x-frame-options
    	SAMEORIGIN
    x-version
    	2.4.3
    	
    Accept
    	application/json; version=4
    Accept-Encoding
    	gzip, deflate
    Accept-Language
    	en-GB,en;q=0.5
    Connection
    	keep-alive
    Cookie
    	csrftoken=8y3o6cSaQtFvYbbepAW3QKVklg0Uj81q; sessionid=6yltak2aawuoxrqb15zjb7pmrebunbvj; CSRF-Token-UMYOGMB=yNoCxdogNEZWXccsXaEQZCzVWyZ2Pi9A; CSRF-Token-PE74D3O=ebmqeMcWZFyU9WKedhg6UH2cjeGkdppK; CSRF-Token-7TUI43L=zHsKFoW9yvtyKberxjfmvAqA9kZQr2HY; CSRF-Token-D2IMSJQ=9YMs3pb7A7f9GvycSsauehZ2M9QXwwLe; CSRF-Token-LDLNTBR=C7zJa9TRwgXXtwNUPR3C7jQW6hc4ajSQ; CSRF-Token-EZRGVQZ=jn6KW5TGJxxvrmdZdAanTEsro3tpE2xR
    Host
    	dms
    Referer
    	http://dms/documents/2034/details
    User-Agent
    	Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0
    X-CSRFToken
    	8y3o6cSaQtFvYbbepAW3QKVklg0Uj81q
```


### Paperless-ngx version

2.4.3

### Host OS

Ubuntu 22.04.3

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:
I mean, not sure what else you would expect?  It's not encoded as valid UTF-8, what else would we do with it?
`str.encode()` offers multiple ways to deal with invalid UTF-8[^1]. You already process the subject line in a more robust way when extracting the title (when importing the email via IMAP) and when creating the PDF file.

[^1]: https://docs.python.org/3/library/codecs.html#error-handlers
You may notice none of the lines above are actually in paperless.  The actual failing line is in DRF, where we have no control
If you believe the bug is in a library you use, can you please figure out what it is and report it so they can fix it? As a user who is unfamiliar with your codebase, I cannot determine whether you use DRF incorrectly or if it has a bug, let alone what that bug is, just by looking at the backtrace. From a user perspective, this is a bug in Paperless-ngx no matter what the root cause is.
I don't think there's anything we can do.  If a file is broken, that's going to cause problems no matter what.

Resolving patch:
diff --git a/src/paperless_mail/parsers.py b/src/paperless_mail/parsers.py
--- a/src/paperless_mail/parsers.py
+++ b/src/paperless_mail/parsers.py
@@ -69,6 +69,11 @@ def extract_metadata(self, document_path: Path, mime_type: str):
 
         for key, value in mail.headers.items():
             value = ", ".join(i for i in value)
+            try:
+                value.encode("utf-8")
+            except UnicodeEncodeError as e:  # pragma: no cover
+                self.log.debug(f"Skipping header {key}: {e}")
+                continue
 
             result.append(
                 {
diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py
--- a/src/paperless_tesseract/parsers.py
+++ b/src/paperless_tesseract/parsers.py
@@ -55,11 +55,21 @@ def extract_metadata(self, document_path, mime_type):
                 value = str(value)
                 try:
                     m = namespace_pattern.match(key)
+                    if m is None:  # pragma: no cover
+                        continue
+                    namespace = m.group(1)
+                    key_value = m.group(2)
+                    try:
+                        namespace.encode("utf-8")
+                        key_value.encode("utf-8")
+                    except UnicodeEncodeError as e:  # pragma: no cover
+                        self.log.debug(f"Skipping metadata key {key}: {e}")
+                        continue
                     result.append(
                         {
-                            "namespace": m.group(1),
-                            "prefix": meta.REVERSE_NS[m.group(1)],
-                            "key": m.group(2),
+                            "namespace": namespace,
+                            "prefix": meta.REVERSE_NS[namespace],
+                            "key": key_value,
                             "value": value,
                         },
                     )


Test patch: