Repository: goauthentik/authentik

Issue:
404 error for kubernetes depoyment
**Describe the bug**
/if/flow/initial-setup/ endpoint in the browser gives a 404 not found error

**To Reproduce**
Steps to reproduce the behavior:
1. Go to 'https://goauthentik.io/docs/installation/kubernetes'
2. Create Values.yaml
3. run helm commands
4. all pods are up and running
5. go to browser http://<ing-url>/if/flow/initial-setup/

**Expected behavior**
Page should load letting me setup ak-admin password

**Screenshots**
404 not found

**Logs**
{"event": "/api/v3/outposts/instances/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 24, "remote": "127.0.0.1", "request_id": "454efe5b57f34713bf837681449b91a6", "runtime": 35, "scheme": "http", "status": 403, "timestamp": "2022-07-11T10:39:00.436171", "user": "", "user_agent": "goauthentik.io/outpost/2022.7.2"}
{"event": "Forbidden: /api/v3/outposts/instances/", "level": "warning", "logger": "django.request", "timestamp": 1657535940.437195}
{"error":"403 Forbidden","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller","timestamp":"2022-07-11T10:39:00Z"}
**Version and Deployment (please complete the following information):**
 - authentik version: authentik-2022.7.2 
 - Deployment: [kubectl 1.21, helm v3.1.0]

**Additional context**
Add any other context about the problem here.



Discussion:
I'm getting something similar on a docker-compose deployment. Just tried to set it up earlier today and I'm getting
<details>
<summary>docker logs -f authentik</summary>

```
{"event": "Forbidden: /api/v3/outposts/instances/", "level": "warning", "logger": "django.request", "timestamp": 1657631480.029771}
{"error":"403 Forbidden","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller","timestamp":"2022-07-12T13:11:20Z"}
{"event": "/api/v3/outposts/instances/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 39, "remote": "127.0.0.1", "request_id": "75b1db68f25c4a4aa0c114405dd952b7", "runtime": 28, "scheme": "http", "status": 403, "timestamp": "2022-07-12T13:11:23.063320", "user": "", "user_agent": "goauthentik.io/outpost/2022.7.2"}
{"event": "Forbidden: /api/v3/outposts/instances/", "level": "warning", "logger": "django.request", "timestamp": 1657631483.064359}
{"error":"403 Forbidden","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller","timestamp":"2022-07-12T13:11:23Z"}
{"event": "/api/v3/outposts/instances/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 38, "remote": "127.0.0.1", "request_id": "335acdceb95847edab5eb7a499df52b9", "runtime": 30, "scheme": "http", "status": 403, "timestamp": "2022-07-12T13:11:26.101239", "user": "", "user_agent": "goauthentik.io/outpost/2022.7.2"}
{"event": "Forbidden: /api/v3/outposts/instances/", "level": "warning", "logger": "django.request", "timestamp": 1657631486.1027539}
{"error":"403 Forbidden","event":"Failed to fetch outpost configuration, retrying in 3 seconds","level":"error","logger":"authentik.outpost.ak-api-controller","timestamp":"2022-07-12T13:11:26Z"}
```

</details>

I've tried both `image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2022.7.2}` and `image: beryju/authentik` but no luck so far.

<details>
<summary>The relevant part of my docker-compose.yml</summary>

```
    authentik-postgresql:
      image: postgres:12-alpine
      container_name: authentik-postgresql
      restart: unless-stopped
      healthcheck:
        test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
        start_period: 20s
        interval: 30s
        retries: 5
        timeout: 5s
      volumes:
        - ./authentik/db:/var/lib/postgresql/data
      environment:
        - POSTGRES_PASSWORD=${PG_PASS:?database password required}
        - POSTGRES_USER=${PG_USER:-authentik}
        - POSTGRES_DB=${PG_DB:-authentik}
      env_file:
        - .env
        
    authentik-redis:
      image: redis:alpine
      container_name: authentik-redis
      restart: unless-stopped
      healthcheck:
        test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
        start_period: 20s
        interval: 30s
        retries: 5
        timeout: 3s

    authentik:
      image: beryju/authentik
      container_name: authentik
      restart: unless-stopped
      command: server
      environment:
        AUTHENTIK_REDIS__HOST: authentik-redis
        AUTHENTIK_POSTGRESQL__HOST: authentik-postgresql
        AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
        AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
        AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
        # AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
        # WORKERS: 2
      volumes:
        - ./authentik/media:/media
        - ./authentik/custom-templates:/templates
      env_file:
        - .env
      ports:
        - 9000:9000
        - 9443:9443

    authentik-worker:
      image: beryju/authentik
      container_name: authentik-worker
      restart: unless-stopped
      command: worker
      environment:
        AUTHENTIK_REDIS__HOST: authentik-redis
        AUTHENTIK_POSTGRESQL__HOST: authentik-postgresql
        AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
        AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
        AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
        # AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
      volumes:
        - ./authentik/media:/media
        - ./authentik/certs:/certs
        - ./authentik/worker/custom-templates:/templates
      env_file:
        - .env
```

</details>

<details>
<summary>My .env file</summary>

```
PG_PASS=<REDACTED>
AUTHENTIK_SECRET_KEY=<REDACTED>

# SMTP Host Emails are sent to
AUTHENTIK_EMAIL__HOST=<REDACTED>
AUTHENTIK_EMAIL__PORT=25
# Optionally authenticate (don't add quotation marks to you password)
AUTHENTIK_EMAIL__USERNAME=noreply@<REDACTED>
AUTHENTIK_EMAIL__PASSWORD=<REDACTED>
# Use StartTLS
AUTHENTIK_EMAIL__USE_TLS=false
# Use SSL
AUTHENTIK_EMAIL__USE_SSL=true
AUTHENTIK_EMAIL__TIMEOUT=10
# Email address authentik will send from, should have a correct @domain
AUTHENTIK_EMAIL__FROM=noreply@<REDACTED>

AUTHENTIK_DISABLE_UPDATE_CHECK=true
AUTHENTIK_ERROR_REPORTING__ENABLED=false
AUTHENTIK_DISABLE_STARTUP_ANALYTICS=true
```

</details>

My docker-compose.yml is loosely based off of [this](https://goauthentik.io/docs/installation/docker-compose). When using 2022.6.3 the issue isn't present so it seems like this is related to #3224.
@gopaljayanthi for a temporary solution, I guess you could roll back to an earlier version while the devs work on the issue
the `{"event": "Forbidden: /api/v3/outposts/instances/", "level": "warning", "logger": "django.request", "timestamp": 1657535940.437195}` error is caused by some system tasks not having run on a fresh install, check if there's any errors in your worker container/give that a restart, and also try `kubectl exec -it deployment/authentik-worker -c authentik -- ak bootstrap_tasks `
@Venem are you using the `file://` prefix for any of the environment variables?
I don't think so. The .env and docker-compose.yml files I posted are pretty much the same as the ones I'm using on the server apart from the passwords/emails that are redacted
if you're getting the above permission error after an upgrade instead of a fresh install (at least that's what I'm understanding), please open a new issue as that should not happen at all
I'm on a fresh install. Sorry, I should've made it clearer.
while I am on unraid and running into other issues, I recently did a fresh install to try and solve those issues and this creeped up. 
I looked at the worker logs and noticed a TON of errors all of a sudden, so I did a restart on the worker, while the main app was running and the main apps log cleared up and solved the problem. I then restarted both of them just for validation, and they both came back up fine after a reboot.
@BeryJu {"event": "Failed to load GeoIP database", "exc": "FileNotFoundError(2, 'No such file or directory')", "level": "warning", "logger": "authentik.events.geo", "pid": 2286, "timestamp": "2022-07-13T04:51:50.930932"} is what I am seeing when i run kubectl exec -it deployment/authentik-worker -c authentik -- ak bootstrap_tasks @Venem which version is a wstable working one the images i am using now are       image: docker.io/bitnami/postgresql:11.14.0-debian-10-r28
      image: docker.io/bitnami/redis:6.2.6-debian-10-r97
      image: ghcr.io/goauthentik/server:2022.7.2
      image: ghcr.io/goauthentik/server:2022.7.2 also after restarting the pods I am not seeing the Failed to fetch outpost configuration error, now I am seeing a new error {"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2022-07-13T04:49:44Z"}
@gopaljayanthi 2022.6.3 seems to be the latest authentik version that works. If you're still running into issues on that version then try downgrading more or (if you haven't already) delete any files that the broken installation created.
Thanks @Venem now i am bale to get to the login screnn with 2022.6.2 but after putting in email and password twice , I am seeing this error  Request has been denied. Unknown error
Reverted back 2022.7.2 and changed ingress class to nginx, added tls part with same hostname and a secret, added certmanager issuer, now the ui is comingup.
I would reopen since this bug does still need to be fixed. Reverting to an older version is only a temporary fix and this needs to be sorted

Resolving patch:
diff --git a/authentik/managed/tasks.py b/authentik/managed/tasks.py
--- a/authentik/managed/tasks.py
+++ b/authentik/managed/tasks.py
@@ -11,7 +11,11 @@
 from authentik.managed.manager import ObjectManager
 
 
-@CELERY_APP.task(bind=True, base=MonitoredTask)
+@CELERY_APP.task(
+    bind=True,
+    base=MonitoredTask,
+    retry_backoff=True,
+)
 @prefill_task
 def managed_reconcile(self: MonitoredTask):
     """Run ObjectManager to ensure objects are up-to-date"""
@@ -22,3 +26,4 @@ def managed_reconcile(self: MonitoredTask):
         )
     except DatabaseError as exc:  # pragma: no cover
         self.set_status(TaskResult(TaskResultStatus.WARNING, [str(exc)]))
+        self.retry()
diff --git a/lifecycle/migrate.py b/lifecycle/migrate.py
--- a/lifecycle/migrate.py
+++ b/lifecycle/migrate.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 """System Migration handler"""
 import os
+import sys
 from importlib.util import module_from_spec, spec_from_file_location
 from inspect import getmembers, isclass
 from pathlib import Path
@@ -50,7 +51,16 @@ def release_lock():
     curr.execute("SELECT pg_advisory_unlock(%s)", (ADV_LOCK_UID,))
 
 
+def is_locked():
+    """Check if lock is currently active (used by worker to wait for migrations)"""
+    curr.executor("SELECT count(*) FROM pg_locks WHERE objid = %s", (ADV_LOCK_UID,))
+    return curr.rowcount
+
+
 if __name__ == "__main__":
+    if len(sys.argv) > 1:
+        if sys.argv[1] == "check_lock":
+            sys.exit(is_locked())
 
     conn = connect(
         dbname=CONFIG.y("postgresql.name"),


Test patch: