Repository: goauthentik/authentik

Issue:
Celery CPU usage 100% on new install
**Describe the bug**

I'm using the attached docker-compose and env file (don't worry about the secrets in the env file).

[env.txt](https://github.com/goauthentik/authentik/files/10758594/env.txt)
[docker-compose.yml.txt](https://github.com/goauthentik/authentik/files/10758616/docker-compose.yml.txt)


**To Reproduce**

Run `docker-compose up` with the default configuration. Watch as celery uses 100% of a single CPU core. I've followed the setup guide from scratch twice to make sure I was getting as close as possible to a default install.

**Logs**
<details>
    <summary>Stacktrace from authentik</summary>

```
Traceback (most recent call last):
django.db.utils.InterfaceError: connection already closed
```
</details>


**Version and Deployment (please complete the following information):**
- authentik version: 2023.2.2
- Deployment: docker compose


        


Discussion:
Any trouble reproducing this? I can't not reproduce it so far.
`/tmp/celerybeat-schedule` is also not being created.
May be increase the healthcheck settings of postgresql and redis in docker compose file will resolve this problem?

I've faced this problem before, some core services take long time to start up, especially in low end hardware, and the healthcheck is too short before docker force restart these services, it make the whole system endlessly restarting while waiting each others go "live"

We should have a small note about this possible problem in the document
I tried it, it didn't fix it. I can hear the fans going out loud, and the cpu usage going through the roof.
One temporary fix is to limit the cpu usage of the server/worker, depending on how much it needs to still function. In my case, I'm able to fix it to 20%. Less and it starts hanging.
Also ran into this. I don't run a mission critical piece of infra, and downgrading to 2023.2 mitigated this issue for me for now, same as https://github.com/goauthentik/authentik/issues/3125#issuecomment-1467593856

My symptoms are identical to the original commenters in #3125: the python celery job is pinned at 100% CPU and the worker logs are filled with "Task Started" -> "Task Finished" for celery. 
Any news on this one? I'm seeing the same on the worker container. Very frequent cpu spikes with the celery process using 100% of cpu.
It seems that the very frequent cpu usage spike is caused by: https://github.com/goauthentik/authentik/blob/main/lifecycle/ak#L83
and used by: https://github.com/goauthentik/authentik/blob/main/Dockerfile#L116
If I add:
```
healthcheck:
      disable: true
```
to the compose file for the worker container, these frequent spikes are gone.
Now there are still other uses of celery which is scheduled like every 1 or 2 mins, but those seem more important like these ones:
```
scim_signal_direct
clean_temporary_users
save_reputation
clean_expired_models
```

@BeryJu Do you have any clues why the celery process uses so much cpu especially for the healthcheck? Maybe it could be replaced with something not that demanding?

Resolving patch:
diff --git a/authentik/root/celery.py b/authentik/root/celery.py
--- a/authentik/root/celery.py
+++ b/authentik/root/celery.py
@@ -2,9 +2,12 @@
 import os
 from contextvars import ContextVar
 from logging.config import dictConfig
+from pathlib import Path
+from tempfile import gettempdir
 from typing import Callable
 
-from celery import Celery
+from celery import Celery, bootsteps
+from celery.apps.worker import Worker
 from celery.signals import (
     after_task_publish,
     setup_logging,
@@ -28,6 +31,7 @@
 LOGGER = get_logger()
 CELERY_APP = Celery("authentik")
 CTX_TASK_ID = ContextVar(STRUCTLOG_KEY_PREFIX + "task_id", default=Ellipsis)
+HEARTBEAT_FILE = Path(gettempdir() + "/authentik-worker")
 
 
 @setup_logging.connect
@@ -99,6 +103,33 @@ def worker_ready_hook(*args, **kwargs):
     start_blueprint_watcher()
 
 
+class LivenessProbe(bootsteps.StartStopStep):
+    """Add a timed task to touch a temporary file for healthchecking reasons"""
+
+    requires = {"celery.worker.components:Timer"}
+
+    def __init__(self, parent, **kwargs):
+        super().__init__(parent, **kwargs)
+        self.requests = []
+        self.tref = None
+
+    def start(self, parent: Worker):
+        self.tref = parent.timer.call_repeatedly(
+            10.0,
+            self.update_heartbeat_file,
+            (parent,),
+            priority=10,
+        )
+        self.update_heartbeat_file(parent)
+
+    def stop(self, parent: Worker):
+        HEARTBEAT_FILE.unlink(missing_ok=True)
+
+    def update_heartbeat_file(self, worker: Worker):
+        """Touch heartbeat file"""
+        HEARTBEAT_FILE.touch()
+
+
 # Using a string here means the worker doesn't have to serialize
 # the configuration object to child processes.
 # - namespace='CELERY' means all celery-related configuration keys
@@ -107,3 +138,4 @@ def worker_ready_hook(*args, **kwargs):
 
 # Load task modules from all registered Django app configs.
 CELERY_APP.autodiscover_tasks()
+CELERY_APP.steps["worker"].add(LivenessProbe)


Test patch: