Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] MySQL Server has gone away
### Description

Hi team! The issue I am having is with the database. if the service is left running and not consuming new documents, after some time when I go back and try to add new documents to be consumed, I get the error that mysql server is not available.

The document gets processed nevertheless. If I restart the docker container, it starts working again... up until left hanging and not processing for some time and the error reappears.

DB server is containerized as well and it is `linuxserver/mariadb:latest` docker image. Many other services are using the same db w/o any issues.

I am not even sure what the consequences are of this behaviour, can you share that information? And any ideas on how to fix this? Thanks!

### Steps to reproduce

1. start docker container
2. do not push any documents for processing for about 5 minutes
3. push a document to be processed
4. error is logged but document gets processed

### Webserver logs

```bash
[2023-09-25 15:19:10,531] [ERROR] [paperless.handlers] Creating PaperlessTask failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 255, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (2006, 'Server has gone away')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/documents/signals/handlers.py", line 537, in before_task_publish_handler
    PaperlessTask.objects.create(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 671, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1006, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1047, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1791, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 255, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (2006, 'Server has gone away')
```


### Browser logs

_No response_

### Paperless-ngx version

1.17.4

### Host OS

Debian 12

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Duplicate of #1783.  This is never something we can reproduce.  Generally, I would suggest making sure Docker is up to date, re-creating images and the network.  As you can see from the traceback, this is not something paperless controls directly.  There's some related Django bugs, but they've all be closed as invalid or wontfix unfortunatly.

It doesn't actually affect anything, the document is still consumed.

Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -9,6 +9,7 @@
 from time import sleep
 from typing import Final
 
+from django import db
 from django.conf import settings
 from django.core.management.base import BaseCommand
 from django.core.management.base import CommandError
@@ -38,6 +39,7 @@ def _tags_from_path(filepath) -> set[Tag]:
 
     Returns set of Tag models
     """
+    db.close_old_connections()
     tag_ids = set()
     path_parts = Path(filepath).relative_to(settings.CONSUMPTION_DIR).parent.parts
     for part in path_parts:
diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py
--- a/src/documents/signals/handlers.py
+++ b/src/documents/signals/handlers.py
@@ -14,6 +14,7 @@
 from django.contrib.auth.models import User
 from django.contrib.contenttypes.models import ContentType
 from django.db import DatabaseError
+from django.db import close_old_connections
 from django.db import models
 from django.db.models import Q
 from django.dispatch import receiver
@@ -529,6 +530,8 @@ def before_task_publish_handler(sender=None, headers=None, body=None, **kwargs):
         return
 
     try:
+        close_old_connections()
+
         task_args = body[0]
         input_doc, _ = task_args
 
@@ -560,6 +563,7 @@ def task_prerun_handler(sender=None, task_id=None, task=None, **kwargs):
     https://docs.celeryq.dev/en/stable/userguide/signals.html#task-prerun
     """
     try:
+        close_old_connections()
         task_instance = PaperlessTask.objects.filter(task_id=task_id).first()
 
         if task_instance is not None:
@@ -587,6 +591,7 @@ def task_postrun_handler(
     https://docs.celeryq.dev/en/stable/userguide/signals.html#task-postrun
     """
     try:
+        close_old_connections()
         task_instance = PaperlessTask.objects.filter(task_id=task_id).first()
 
         if task_instance is not None:
@@ -615,6 +620,7 @@ def task_failure_handler(
     https://docs.celeryq.dev/en/stable/userguide/signals.html#task-failure
     """
     try:
+        close_old_connections()
         task_instance = PaperlessTask.objects.filter(task_id=task_id).first()
 
         if task_instance is not None and task_instance.result is None:


Test patch: