Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Text encoding issue when using the MariaDB backend
### Description

Certain documents return errors depending on their content when scanned. I believe that this is because of certain equation-like expressions within the document.

This does not occur when using the SQLite backend. I have yet to test PostgreSQL.

### Steps to reproduce

The following is an example of such a document:
https://www.ukmt.org.uk/sites/default/files/ukmt/SMC_2021_paper.pdf

Import the document when using the mariadb backend.

### Webserver logs

```bash
SMC_2021_paper.pdf: The following error occurred while consuming SMC_2021_paper.pdf: (1366, "Incorrect string value: '\\xF0\\x9D\\x91\\xA5\\xC2\\xB0...' for column `paperless`.`documents_document`.`content` at row 1") : Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 320, in _query
    self._do_get_result(db)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 145, in _do_get_result
    self._result = result = self._get_result()
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 352, in _get_result
    return self._get_db().store_result()
MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9D\\x91\\xA5\\xC2\\xB0...' for column `paperless`.`documents_document`.`content` at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 280, in main_wrap
    raise exc_info[1]
  File "/usr/src/paperless/src/documents/consumer.py", line 361, in try_consume_file
    document = self._store(text=text, date=date, mime_type=mime_type)
  File "/usr/src/paperless/src/documents/consumer.py", line 471, in _store
    document = Document.objects.create(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 514, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 806, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 857, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1000, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1041, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1434, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1621, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 320, in _query
    self._do_get_result(db)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 145, in _do_get_result
    self._result = result = self._get_result()
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 352, in _get_result
    return self._get_db().store_result()
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9D\\x91\\xA5\\xC2\\xB0...' for column `paperless`.`documents_document`.`content` at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/src/django-q/django_q/cluster.py", line 454, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 154, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 423, in try_consume_file
    self._fail(
  File "/usr/src/paperless/src/documents/consumer.py", line 90, in _fail
    raise ConsumerError(f"{self.filename}: {log_message or message}") from exception
documents.consumer.ConsumerError: SMC_2021_paper.pdf: The following error occurred while consuming SMC_2021_paper.pdf: (1366, "Incorrect string value: '\\xF0\\x9D\\x91\\xA5\\xC2\\xB0...' for column `paperless`.`documents_document`.`content` at row 1")
```


### Paperless-ngx version

1.9

### Host OS

Alpine Linux

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

This should be a relatively easy fix, hopefully just a swap of the encoding type when using mariadb, however I am not experienced enough with SQL to resolve this on my own.


Discussion:

Resolving patch:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -319,6 +319,7 @@ def __get_path(key: str, default: str) -> str:
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
         "NAME": os.path.join(DATA_DIR, "db.sqlite3"),
+        "OPTIONS": {},
     },
 }
 
@@ -340,21 +341,18 @@ def __get_path(key: str, default: str) -> str:
     # Leave room for future extensibility
     if os.getenv("PAPERLESS_DBENGINE") == "mariadb":
         engine = "django.db.backends.mysql"
-        options = {"read_default_file": "/etc/mysql/my.cnf"}
+        options = {"read_default_file": "/etc/mysql/my.cnf", "charset": "utf8mb4"}
     else:  # Default to PostgresDB
         engine = "django.db.backends.postgresql_psycopg2"
         options = {"sslmode": os.getenv("PAPERLESS_DBSSLMODE", "prefer")}
 
     DATABASES["default"]["ENGINE"] = engine
-    for key, value in options.items():
-        DATABASES["default"]["OPTIONS"][key] = value
+    DATABASES["default"]["OPTIONS"].update(options)
 
 if os.getenv("PAPERLESS_DB_TIMEOUT") is not None:
-    _new_opts = {"timeout": float(os.getenv("PAPERLESS_DB_TIMEOUT"))}
-    if "OPTIONS" in DATABASES["default"]:
-        DATABASES["default"]["OPTIONS"].update(_new_opts)
-    else:
-        DATABASES["default"]["OPTIONS"] = _new_opts
+    DATABASES["default"]["OPTIONS"].update(
+        {"timeout": float(os.getenv("PAPERLESS_DB_TIMEOUT"))},
+    )
 
 DEFAULT_AUTO_FIELD = "django.db.models.AutoField"
 


Test patch: