Repository: paperless-ngx/paperless-ngx

Issue:
Import via document_importer compains about lacking files with umlauts that are referenced in the manifest.json
### Description

I tried to export my documents via document_export; some files with umlauts in their names.
when trying to import them back into the clean database, paperless complains about file in the manifest.json but not in the source directory and exits without having imported anything.

exact error is: 
`The manifest file refers to "2023-07-19 Überlassung Businessbike.pdf" which does not appear to be in the source directory.
`


### Steps to reproduce

export a database with german umlauts

try to import them into a new database

### Webserver logs

```bash
the web server does not log anything then running that command.
```


### Browser logs

_No response_

### Paperless-ngx version

1.17.4

### Host OS

rasbian os bookworm 64 Bit 

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Perhaps this is OS specific, but no issues here

```bash
python3 manage.py document_exporter ../export
100%|███████████████████████████████████████| 1/1 [00:00<00:00, 443.89it/s]
```

```bash
ls -a ../export
2023-11-01 Überlassung Businessbike.pdf				manifest.json
2023-11-01 Überlassung Businessbike.pdf-archive.pdf		version.json
2023-11-01 Überlassung Businessbike.pdf-thumbnail.webp
```

...

```bash
document_importer ../export
Checking the manifest
Installed 152 object(s) from 1 fixture(s)
Copy files into paperless...
100%|███████████████████████████████████████| 1/1 [00:00<00:00, 237.30it/s]
Updating search index...
100%|███████████████████████████████████████| 1/1 [00:00<00:00, 198.53it/s]
```

```bash
ls ../media/documents/originals
2023-Überlassung Businessbike.pdf
```

Did you check if `"2023-07-19 Überlassung Businessbike.pdf"` is in the export folder?
it sure is, as many other similiar documnets with Umlauts in their name which do not get imported...
Maybe the problem is not the umlauts, but anything else? I assume the manifest.json and the documents are supposed to be in the same directory, and if the manifest is readable, the docs also should be - so a permission issue seems unlikely....

Adding a Document by web UI works, so paperless-ngx is functional.

I thought about replacing the escaped umlauts in the manifest  through their original- maybe it's the way they are converted  in the manifest and re-escaped ?

```
root@rpi3:/opt/paperless_ngx/paperless-ngx/export# ls -la 2023-07-19\ Ü*   
-rwxrwxrwx 1 root martin 311899 Nov  2 01:56 '2023-07-19 Überlassung Businessbike.pdf'
-rwxrwxrwx 1 root martin 295084 Nov  2 01:56 '2023-07-19 Überlassung Businessbike.pdf-archive.pdf'
-rwxrwxrwx 1 root martin  28432 Nov  2 01:56 '2023-07-19 Überlassung Businessbike.pdf-thumbnail.webp'  
```

the manifest.json references as
```
    "__exported_file_name__": "2023-07-19 U\u0308berlassung Businessbike.pdf",
    "__exported_thumbnail_name__": "2023-07-19 U\u0308berlassung Businessbike.pdf-thumbnail.webp",
    "__exported_archive_name__": "2023-07-19 U\u0308berlassung Businessbike.pdf-archive.pdf"
```
There is a Unicode character class of multibyte characters for example the combining diaeresis (U+0308) which can be used in combination with another character e.g., a simple “U” to create an umlaut or similar instead of using the actual Unicode character in this case Ü (Latin Capital Letter U with Diaeresis, U+00DC). 
It might be one of the used libraries that uses the multicharacter encoding approach in favor of the single character representation. It could also be a combination of the browser and/or the file system or even the database encoding. More information needed.
The previous posts in this issue essentially describe the same situation I encountered on an macOS system which hosted a single user paperless instance. The manifest contains the multicharacter variant of the umlaut “Ü” while the filenames use the singlecharacter variant. You will see the same thing if you compare the umlaut in the manifest error line in https://github.com/paperless-ngx/paperless-ngx/issues/4485#issue-1973338982 with the umlauts in the ls output in https://github.com/paperless-ngx/paperless-ngx/issues/4485#issuecomment-1790373885. You can copy the respective umlaut and paste it into a textarea and press backspace once to notice the difference.
The aforementioned macOS instance doesn't exist anymore, so I cannot inquire further, but maybe this can be of help.
thanks, @NoxObscuro, you pointed me onto the right track. after changing the umlaut-encoding in two of my filenames in the manifest.json (AND after deleting the user those documents where saved) I coud reimport all my 184 documents.
I leave that open here - i have no idea why in two files the program chose that encoding (and in many others, id didn't) - but I sure prevents from restoring the backup, and it's sure worth a change in the document_exporter module.
I had exactly the same problem yesterday and solved it by removing the umlaut from the document title and re-exporting the documents.

Just for the record ;)

Resolving patch:
diff --git a/src/documents/management/commands/document_exporter.py b/src/documents/management/commands/document_exporter.py
--- a/src/documents/management/commands/document_exporter.py
+++ b/src/documents/management/commands/document_exporter.py
@@ -396,20 +396,31 @@ def dump(self, progress_bar_disable=False):
                         notes,
                     ),
                 )
-                manifest_name.write_text(json.dumps(content, indent=2))
+                manifest_name.write_text(
+                    json.dumps(content, indent=2, ensure_ascii=False),
+                    encoding="utf-8",
+                )
                 if manifest_name in self.files_in_export_dir:
                     self.files_in_export_dir.remove(manifest_name)
 
         # 4.1 write manifest to target folder
         manifest_path = (self.target / Path("manifest.json")).resolve()
-        manifest_path.write_text(json.dumps(manifest, indent=2))
+        manifest_path.write_text(
+            json.dumps(manifest, indent=2, ensure_ascii=False),
+            encoding="utf-8",
+        )
         if manifest_path in self.files_in_export_dir:
             self.files_in_export_dir.remove(manifest_path)
 
         # 4.2 write version information to target folder
         version_path = (self.target / Path("version.json")).resolve()
         version_path.write_text(
-            json.dumps({"version": version.__full_version_str__}, indent=2),
+            json.dumps(
+                {"version": version.__full_version_str__},
+                indent=2,
+                ensure_ascii=False,
+            ),
+            encoding="utf-8",
         )
         if version_path in self.files_in_export_dir:
             self.files_in_export_dir.remove(version_path)


Test patch: