Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Impossible to connect to redis using a unix socket
### Description

On my setup, paperless connects to redis using a unix socket. Since 1.10.0 this does not work any more. The cause seems to be the format of the PAPERLESS_REDIS variable in the `paperless.conf` config file.
If I want celery to connect to redis successfully using a socket, I [need to use this format](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html): `PAPERLESS_REDIS='redis+socket:///run/redis/redis.sock'`. This is the only way to get celery to connect successfully.
Unfortunately, the rest of paperless does not seem to understand this format and gives the following error:
```
Nov 30 18:34:48 test celery[17178]: [2022-11-30 18:34:48,696] [INFO] [celery.worker.strategy] Task documents.tasks.consume_file[40c8ef18-4be7-4a1f-98d9-f1da403ac1f2] received
Nov 30 18:34:48 test celery[17215]: [2022-11-30 18:34:48,897] [ERROR] [celery.app.trace] Task documents.tasks.consume_file[40c8ef18-4be7-4a1f-98d9-f1da403ac1f2] raised unexpected: ValueError('Redis URL must specify one of the following schemes (redis://, rediss://, unix://)')
Nov 30 18:34:48 test celery[17215]: Traceback (most recent call last):
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
Nov 30 18:34:48 test celery[17215]:     R = retval = fun(*args, **kwargs)
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
Nov 30 18:34:48 test celery[17215]:     return self.run(*args, **kwargs)
Nov 30 18:34:48 test celery[17215]:   File "/usr/share/paperless/src/documents/tasks.py", line 177, in consume_file
Nov 30 18:34:48 test celery[17215]:     document = Consumer().try_consume_file(
Nov 30 18:34:48 test celery[17215]:   File "/usr/share/paperless/src/documents/consumer.py", line 272, in try_consume_file
Nov 30 18:34:48 test celery[17215]:     self._send_progress(0, 100, "STARTING", MESSAGE_NEW_FILE)
Nov 30 18:34:48 test celery[17215]:   File "/usr/share/paperless/src/documents/consumer.py", line 77, in _send_progress
Nov 30 18:34:48 test celery[17215]:     async_to_sync(self.channel_layer.group_send)(
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/asgiref/sync.py", line 218, in __call__
Nov 30 18:34:48 test celery[17215]:     return call_result.result()
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
Nov 30 18:34:48 test celery[17215]:     return self.__get_result()
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
Nov 30 18:34:48 test celery[17215]:     raise self._exception
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/asgiref/sync.py", line 284, in main_wrap
Nov 30 18:34:48 test celery[17215]:     result = await self.awaitable(*args, **kwargs)
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/channels_redis/core.py", line 546, in group_send
Nov 30 18:34:48 test celery[17215]:     connection = self.connection(self.consistent_hash(group))
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/channels_redis/core.py", line 738, in connection
Nov 30 18:34:48 test celery[17215]:     self.pools[index] = self.create_pool(index)
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/channels_redis/core.py", line 133, in create_pool
Nov 30 18:34:48 test celery[17215]:     return aioredis.ConnectionPool.from_url(host["address"])
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/redis/asyncio/connection.py", line 1409, in from_url
Nov 30 18:34:48 test celery[17215]:     url_options = parse_url(url)
Nov 30 18:34:48 test celery[17215]:   File "/usr/lib/python3.10/site-packages/redis/asyncio/connection.py", line 1345, in parse_url
Nov 30 18:34:48 test celery[17215]:     raise ValueError(
Nov 30 18:34:48 test celery[17215]: ValueError: Redis URL must specify one of the following schemes (redis://, rediss://, unix://)
```

### Steps to reproduce

1. Set redis to listen to socket
2. Configure paperless to connect to redis using the socket
3. Import a file
4. Does not work. 

### Webserver logs

```bash
nothing
```


### Browser logs

_No response_

### Paperless-ngx version

1.10.0

### Host OS

Archlinux

### Installation method

Bare metal

### Browser

Firefox

### Configuration changes

PAPERLESS_REDIS='redis+socket:///run/redis/redis.sock'

### Other

Would be awesome if paperless would also support using the [virtual_host URL parameter](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html) when connecting to redis using a unix socket.


Discussion:
For a Unix socket, the channels documentation says it should be unix:///path/to/redis.sock.  So that's the compatibility issue
Yes, but if I set this configuration format, celery does not start any more because it can't connect to redis.

Resolving patch:
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -8,6 +8,7 @@
 from typing import Final
 from typing import Optional
 from typing import Set
+from typing import Tuple
 from urllib.parse import urlparse
 
 from celery.schedules import crontab
@@ -65,6 +66,43 @@ def __get_path(key: str, default: str) -> str:
     return os.path.abspath(os.path.normpath(os.environ.get(key, default)))
 
 
+def _parse_redis_url(env_redis: Optional[str]) -> Tuple[str]:
+    """
+    Gets the Redis information from the environment or a default and handles
+    converting from incompatible django_channels and celery formats.
+
+    Returns a tuple of (celery_url, channels_url)
+    """
+
+    # Not set, return a compatible default
+    if env_redis is None:
+        return ("redis://localhost:6379", "redis://localhost:6379")
+
+    _, path = env_redis.split(":")
+
+    if "unix" in env_redis.lower():
+        # channels_redis socket format, looks like:
+        # "unix:///path/to/redis.sock"
+        if "?db=" in env_redis:
+            path, number = path.split("?db=")
+            return (f"redis+socket:{path}?virtual_host={number}", env_redis)
+        else:
+            return (f"redis+socket:{path}", env_redis)
+
+    elif "+socket" in env_redis.lower():
+        # celery socket style, looks like:
+        # "redis+socket:///path/to/redis.sock"
+        if "?virtual_host=" in env_redis:
+            # Virtual host (aka db number)
+            path, number = path.split("?virtual_host=")
+            return (env_redis, f"unix:{path}?db={number}")
+        else:
+            return (env_redis, f"unix:{path}")
+
+    # Not a socket
+    return (env_redis, env_redis)
+
+
 # NEVER RUN WITH DEBUG IN PRODUCTION.
 DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO")
 
@@ -182,7 +220,9 @@ def __get_path(key: str, default: str) -> str:
 STATIC_URL = os.getenv("PAPERLESS_STATIC_URL", BASE_URL + "static/")
 WHITENOISE_STATIC_PREFIX = "/static/"
 
-_REDIS_URL = os.getenv("PAPERLESS_REDIS", "redis://localhost:6379")
+_CELERY_REDIS_URL, _CHANNELS_REDIS_URL = _parse_redis_url(
+    os.getenv("PAPERLESS_REDIS", None),
+)
 
 # TODO: what is this used for?
 TEMPLATES = [
@@ -205,7 +245,7 @@ def __get_path(key: str, default: str) -> str:
     "default": {
         "BACKEND": "channels_redis.core.RedisChannelLayer",
         "CONFIG": {
-            "hosts": [_REDIS_URL],
+            "hosts": [_CHANNELS_REDIS_URL],
             "capacity": 2000,  # default 100
             "expiry": 15,  # default 60
         },
@@ -468,7 +508,7 @@ def __get_path(key: str, default: str) -> str:
 
 WORKER_TIMEOUT: Final[int] = __get_int("PAPERLESS_WORKER_TIMEOUT", 1800)
 
-CELERY_BROKER_URL = _REDIS_URL
+CELERY_BROKER_URL = _CELERY_REDIS_URL
 CELERY_TIMEZONE = TIME_ZONE
 
 CELERY_WORKER_HIJACK_ROOT_LOGGER = False
@@ -513,7 +553,7 @@ def __get_path(key: str, default: str) -> str:
 CACHES = {
     "default": {
         "BACKEND": "django.core.cache.backends.redis.RedisCache",
-        "LOCATION": _REDIS_URL,
+        "LOCATION": _CHANNELS_REDIS_URL,
     },
 }
 


Test patch:
diff --git a/src/paperless/tests/test_settings.py b/src/paperless/tests/test_settings.py
--- a/src/paperless/tests/test_settings.py
+++ b/src/paperless/tests/test_settings.py
@@ -3,6 +3,7 @@
 from unittest import TestCase
 
 from paperless.settings import _parse_ignore_dates
+from paperless.settings import _parse_redis_url
 from paperless.settings import default_threads_per_worker
 
 
@@ -82,3 +83,54 @@ def test_workers_threads(self):
                 self.assertGreaterEqual(default_threads, 1)
 
                 self.assertLessEqual(default_workers * default_threads, i)
+
+    def test_redis_socket_parsing(self):
+        """
+        GIVEN:
+            - Various Redis connection URI formats
+        WHEN:
+            - The URI is parsed
+        THEN:
+            - Socket based URIs are translated
+            - Non-socket URIs are unchanged
+            - None provided uses default
+        """
+
+        for input, expected in [
+            # Nothing is set
+            (None, ("redis://localhost:6379", "redis://localhost:6379")),
+            # celery style
+            (
+                "redis+socket:///run/redis/redis.sock",
+                (
+                    "redis+socket:///run/redis/redis.sock",
+                    "unix:///run/redis/redis.sock",
+                ),
+            ),
+            # redis-py / channels-redis style
+            (
+                "unix:///run/redis/redis.sock",
+                (
+                    "redis+socket:///run/redis/redis.sock",
+                    "unix:///run/redis/redis.sock",
+                ),
+            ),
+            # celery style with db
+            (
+                "redis+socket:///run/redis/redis.sock?virtual_host=5",
+                (
+                    "redis+socket:///run/redis/redis.sock?virtual_host=5",
+                    "unix:///run/redis/redis.sock?db=5",
+                ),
+            ),
+            # redis-py / channels-redis style with db
+            (
+                "unix:///run/redis/redis.sock?db=10",
+                (
+                    "redis+socket:///run/redis/redis.sock?virtual_host=10",
+                    "unix:///run/redis/redis.sock?db=10",
+                ),
+            ),
+        ]:
+            result = _parse_redis_url(input)
+            self.assertTupleEqual(expected, result)