Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Paperless does not try to reconnect to Redis if it fails on startup
### Description

If Redis is not booted up yet or the container is restarting (especially in Kubernetes), paperless does not try to reconnect to it and therefore can't process any scheduled tasks like mail fetching.

### Expected behavior

paperless is trying to (re-)connect to Redis on startup until it can reach Redis or kills itself if it fails too often.

### Steps to reproduce

1. Start a paperless instance without Redis
2. You're done. Paperless seems to work (at least UI etc.) but task processing will not work.

### Webserver logs

```bash
➜  ~ k logs -n paperless-prod paperless-prod-0 --tail=50
Executing /usr/local/bin/supervisord -c /etc/supervisord.conf
2022-04-20 11:46:56,761 INFO Set uid to user 0 succeeded
2022-04-20 11:46:56,769 INFO supervisord started with pid 1
2022-04-20 11:46:57,772 INFO spawned: 'consumer' with pid 290
2022-04-20 11:46:57,775 INFO spawned: 'gunicorn' with pid 291
2022-04-20 11:46:57,779 INFO spawned: 'scheduler' with pid 292
[2022-04-20 13:46:58 +0200] [291] [INFO] Starting gunicorn 20.1.0
[2022-04-20 13:46:58 +0200] [291] [INFO] Listening at: http://0.0.0.0:8000 (291)
[2022-04-20 13:46:58 +0200] [291] [INFO] Using worker: paperless.workers.ConfigurableWorker
[2022-04-20 13:46:58 +0200] [291] [INFO] Server is ready. Spawning workers
13:46:59 [Q] INFO Q Cluster hotel-cat-kentucky-tennessee starting.
2022-04-20 11:46:59,590 INFO success: consumer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-04-20 11:46:59,590 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-04-20 11:46:59,590 INFO success: scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
13:46:59 [Q] ERROR Can not connect to Redis server.
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 559, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 615, in _connect
    raise err
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 603, in _connect
    sock.connect(socket_address)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/django_q/cluster.py", line 168, in __init__
    self.start()
  File "/usr/local/lib/python3.9/site-packages/django_q/cluster.py", line 171, in start
    self.broker.ping()
  File "/usr/local/lib/python3.9/site-packages/django_q/brokers/redis_broker.py", line 39, in ping
    raise e
  File "/usr/local/lib/python3.9/site-packages/django_q/brokers/redis_broker.py", line 36, in ping
    return self.connection.ping()
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 1378, in ping
    return self.execute_command('PING')
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 113 connecting to paperless-redis:6379. No route to host.
[2022-04-20 13:46:59,762] [INFO] [paperless.management.consumer] Using inotify to watch directory for changes: /usr/src/paperless/src/../consume
```


### Screenshots

_No response_

### Paperless-ngx version

1.6.0

### Host OS

Ubuntu 20.04 kernel 5.4.0-104 and containerd 1.4.12

### Installation method

Other (please describe above)

### Browser

_No response_

### Configuration changes

paperless-ngx is running in Kubernetes

### Other

_No response_


Discussion:
Is this really a bug or more of a feature request?

btw in docker compose you can have one container depend on another, which solves this, not sure if that exists in k8s
> Is this really a bug or more of a feature request?

In my case yes, because the task processing did not work and showed no obvious errors until I looked into the container logs. But yeah, a retry mechanism is indeed more like a feature request.

> btw in docker compose you can have one container depend on another, which solves this, not sure if that exists in k8s

I know about the `depends_on` feature in compose, but there is no feature like that in k8s. You would need to build something on your own e.g. via init containers which check if the depending deployment is ready or not.
It looks decently easy to do with [`redis-cli`](https://redis.io/docs/manual/cli/) and the  [`ping`](https://redis.io/commands/ping/) command.  It's a little shocking Kubernetes doesn't support dependencies, but I think we can work around it and make the image more stable for non-k8s users too.

Something like this, in a loop:
```
redis-cli -u ${PAPERLESS_REDIS} PING | grep "PONG"
```

Resolving patch:
diff --git a/docker/wait-for-redis.py b/docker/wait-for-redis.py
new file mode 100755
--- /dev/null
+++ b/docker/wait-for-redis.py
@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+"""
+Simple script which attempts to ping the Redis broker as set in the environment for
+a certain number of times, waiting a little bit in between
+
+"""
+import os
+import sys
+import time
+from typing import Final
+
+from redis import Redis
+
+if __name__ == "__main__":
+
+    MAX_RETRY_COUNT: Final[int] = 5
+    RETRY_SLEEP_SECONDS: Final[int] = 5
+
+    REDIS_URL: Final[str] = os.getenv("PAPERLESS_REDIS", "redis://localhost:6379")
+
+    print(f"Waiting for Redis: {REDIS_URL}", flush=True)
+
+    attempt = 0
+    with Redis.from_url(url=REDIS_URL) as client:
+        while attempt < MAX_RETRY_COUNT:
+            try:
+                client.ping()
+                break
+            except Exception:
+                print(
+                    f"Redis ping #{attempt} failed, waiting {RETRY_SLEEP_SECONDS}s",
+                    flush=True,
+                )
+                time.sleep(RETRY_SLEEP_SECONDS)
+                attempt += 1
+
+    if attempt >= MAX_RETRY_COUNT:
+        print(f"Failed to connect to: {REDIS_URL}")
+        sys.exit(os.EX_UNAVAILABLE)
+    else:
+        print(f"Connected to Redis broker: {REDIS_URL}")
+        sys.exit(os.EX_OK)


Test patch: