Repository: cubernetes/ft-transcendence
Pull request #153: Integrate ELK fully into vault
Author: login: cubernetes
Created: 2025-05-18T18:18:48Z | Merged: 2025-05-19T17:04:36Z

Finally integrates ELK secrets fully into vault. This was surprisingly fun, but quite a difficult challenge!
For example:
- Ridiculous shell hack (`eval exec env -- - "$env_params" '"${0#-}"' "$(...)"`) to pass environment variable that are not valid identifiers (e.g. `bootstrap.memory_lock`, the dot is not valid for variables, but indeed valid for environment variables) (which is common practice for elastic services..., but for example `dash` (aka `sh`) purges such environment variables upon startup, not cool)
- Healthcheck migration: Healthcheck must a child process of the initial `entrypoint.sh` process tree, otherwise it won't have access to the environment variables extracted from vault. This also required a crazy `curl` feature that even ChatGPT didn't immediately know about. See elasticsearch `startup.sh` for example.

Also does some small things along the way, like
- curl `--no-progress-meter` instead of `--silent` where appropriate
- enable `git commit --squash=/--fixup=` to work with commit-msg hook
- enable compose bake for faster build times
- disable auto-restarting of ELK and backend containers, because it doesn't make sense when they get their config from vault
- fix names of volumes in `Makefile.clean`, they were all wrong and even some were missing lol
- support for JSON path-like references in env.json

Closes #144 
Closes #143 
Closes #99 

Linked issue #144: Manage ELK creds with vault

Diff:
diff --git a/.githooks/commit-msg b/.githooks/commit-msg
index 3083847a..19d2f651 100755
--- a/.githooks/commit-msg
+++ b/.githooks/commit-msg
@@ -8,10 +8,11 @@ die () {
 	exit 1
 }
 
-regex_titlecase_verb='^[A-Z][a-z]+ '
-regex_imperative_verb='^[A-Z][a-z]+ed '
+optional_squash_regex='((squash|fixup)! )?'
+regex_titlecase_verb='[A-Z][a-z]+ '
+regex_imperative_verb='[A-Z][a-z]+ed '
 
-grep -qE -e "$regex_titlecase_verb" -- "$1" || \
+grep -qE -e "^$optional_squash_regex$regex_titlecase_verb" -- "$1" || \
 	die "Commit message format is incorrect (must start with titlecase verb). Example: 'Add feature' or 'Update file'"
-! grep -qE -e "$regex_imperative_verb" "$1" || \
+! grep -qE -e "^$optional_squash_regex$regex_imperative_verb" "$1" || \
 	die "Commit message format is incorrect (use imperative verb form). Example: 'Add feature' or 'Update file'"
diff --git a/Makefile b/Makefile
index a7ffc864..8e43c21e 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,5 @@
 # Change to docker-compose if you need
-DC := COMPOSE_BAKE=false docker compose
+DC := COMPOSE_BAKE=true docker compose
 # Change to podman if you need
 D := docker
 # Enables you do run `make` alone
@@ -11,7 +11,10 @@ D := docker
 # ./.secrets/${service}_vault_token
 # where ${service} is the same as the JSON root subkeys in ./env.json
 VAULT_TOKEN_EXCHANGE_FILES := \
-	./.secrets/backend_vault_token
+	./.secrets/backend_vault_token \
+	./.secrets/elasticsearch_vault_token \
+	./.secrets/logstash_vault_token \
+	./.secrets/kibana_vault_token
 
 # Must be in standard dotenv format
 -include .env
@@ -38,18 +41,16 @@ dev-old-compose: check-env
 	@$(call dev-env,watch --no-up)
 
 .PHONY: dev
-dev: check-env
-	$(MAKE) ensure-secret-files
-	@$(call dev-env,     \
-		up               \
-		--remove-orphans \
-		--build          \
-		--watch          \
+dev: check-env ensure-secret-files
+	@unset -v LOGSTASH_HOST && $(call dev-env, \
+		up                                     \
+		--remove-orphans                       \
+		--build                                \
+		--watch                                \
 		$(ARGS))
 
 .PHONY: dev-elk
-dev-elk: check-env
-	$(MAKE) ensure-secret-files
+dev-elk: check-env ensure-secret-files
 	@$(call dev-env,     \
 		--profile elk    \
 		up               \
@@ -71,13 +72,12 @@ actual-prod: ensure-secret-files
 
 # Temporary fix, so it deploys. No ELK, etc.
 .PHONY: prod
-prod: check-env
-	$(MAKE) ensure-secret-files
-	@$(call dev-env,     \
-		up               \
-		--remove-orphans \
-		--build          \
-		--detach         \
+prod: check-env ensure-secret-files
+	@unset -v LOGSTASH_HOST && $(call dev-env, \
+		up                                     \
+		--remove-orphans                       \
+		--build                                \
+		--detach                               \
 		$(ARGS))
 
 .PHONY: down
@@ -98,3 +98,12 @@ install:
 .PHONY: test
 test: install
 	npm --prefix=backend test
+
+# Usage: make get-env SERVICE=elasticsearch
+.PHONY: get-env
+get-env:
+	$(D) exec vault sh -c 'SERVICE=$(SERVICE); wget --header="X-Vault-Token: $$(cat /vault/secret/root_token)" --quiet --output-document=- "$$(cat /tmp/vault_addr)"/v1/secret/data/$$SERVICE | jq --raw-output --color-output .data.data'
+
+.PHONY: wait
+wait:
+	wait_for_service () { service=$$1; [ -n "$$service" ] || { echo "Expected 1 argument"; return 1; }; i=0; while [ ! "$$(2>/dev/null docker inspect -f "{{.State.Health.Status}}" "$$service")" = "healthy" ]; do printf "\015""Waiting_for_service_$${service}_to_start%$${i}s" "" | sed -u "s/ /./g;s/_/ /g"; i=$$((i + 1)); sleep 0.3; done; echo; } && start=$$(date +%s); wait_for_service vault && wait_for_service elasticsearch && wait_for_service logstash && wait_for_service kibana && wait_for_service backend && echo "Waiting 15 more seconds" && sleep 15 && echo "Frontend: http://localhost:8080" && echo "OpenAPI:  http://localhost:8080/api/docs" && echo "ELK       http://localhost:5601" ; now=$$(date +%s); min=$$(( (now - start) / 60)); sec=$$(( (now - start) % 60)); echo "Took $${min}m$${sec}s"
diff --git a/Makefile.clean b/Makefile.clean
index 52a05759..728f9ca0 100644
--- a/Makefile.clean
+++ b/Makefile.clean
@@ -1,25 +1,34 @@
+PROJECT_NAME := ft-transcendence
+
+define rm-volume
+	@{ echo $(D) volume rm "$(PROJECT_NAME)_$(1)" ; $(D) volume rm "$(PROJECT_NAME)_$(1)" 2>/dev/null; } || echo "[ WARN ] Volume couldn't be removed because it doesn't exist: $(PROJECT_NAME)_$(1)"
+endef
+
 ### clean specific volumes
 .PHONY: clean-frontend-volume
 clean-frontend-volume: down
-	$(D) volume rm --force ft-transcendence_frontend
+	$(call rm-volume,frontend)
 
 .PHONY: clean-vault
 clean-vault: down
-	$(D) volume rm --force ft-transcendence_vault
-	$(D) volume rm --force ft-transcendence_vault-logs
-	$(D) volume rm --force ft-transcendence_vault-secrets
+	$(call rm-volume,vault-logs)
+	$(call rm-volume,vault-secret)
+	$(call rm-volume,vault-storage)
 
 .PHONY: clean-db
 clean-db: down
-	$(D) volume rm --force ft-transcendence_drizzle
+	$(call rm-volume,drizzle)
 
 .PHONY: clean-elk
 clean-elk: down
-	$(D) volume rm --force ft-transcendence_elasticsearch-data
+	$(call rm-volume,elasticsearch-data)
+	$(call rm-volume,elasticsearch-certs)
+	$(call rm-volume,kibana-data)
+	$(call rm-volume,logstash-data)
 
 .PHONY: clean-caddy
 clean-caddy: down
-	$(D) volume rm --force ft-transcendence_elasticsearch-caddy-data
+	$(call rm-volume,caddy-data)
 
 ### clean all volumes, including caddy-db (user data) and caddy-data (TLS certificates)
 ### not suitable for production, obviously
diff --git a/backend/Dockerfile b/backend/Dockerfile
index 3f7a6479..07ab68f6 100644
--- a/backend/Dockerfile
+++ b/backend/Dockerfile
@@ -13,6 +13,6 @@ COPY . .
 ENV WATCH=0
 
 HEALTHCHECK --interval=10s --timeout=2s --start-period=75s --start-interval=1s --retries=3 \
-  CMD curl --fail http://localhost:${BACKEND_PORT}/healthcheck
+  CMD curl --fail --no-progress-meter http://localhost:${BACKEND_PORT}/healthcheck
 
 CMD ["sh", "-c", "if [ \"$WATCH\" = \"0\" ]; then npm run build && npm start; else npm run start:dev; fi"]
diff --git a/compose.yaml b/compose.yaml
index 2db9c160..0b00b4a2 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -74,7 +74,7 @@ services:
     backend:
         build: "./backend/"
         container_name: backend
-        restart: unless-stopped
+        restart: no # cannot restart because of expiring vault token
         networks:
             ft-transcendence-net:
                 ipv4_address: &backend-ip 10.42.42.2
@@ -84,7 +84,7 @@ services:
             DB_PATH: "${DB_PATH:-}"
             LOG_LEVEL: "${LOG_LEVEL:-debug}"
 
-            LOGSTASH_HOST: ${LOGSTASH_HOST:-}
+            LOGSTASH_HOST: "${LOGSTASH_HOST:-}"
             LOGSTASH_PORT: "${LOGSTASH_PORT:-5050}"
         volumes:
             - "drizzle:/backend/drizzle/"
@@ -113,130 +113,67 @@ services:
 
     elasticsearch:
         profiles: [elk]
-        image: docker.elastic.co/elasticsearch/elasticsearch:8.17.3
+        build: "./elk/elasticsearch/"
         container_name: elasticsearch
-        environment:
-            ES_JAVA_OPTS: "-Xms512m -Xmx512m"
-
-            discovery.type: "single-node"
-            bootstrap.memory_lock: "true"
-            cluster.name: "ft-transcendence-logs"
-            node.name: "single-node"
-
-            ELASTIC_PASSWORD: "${ELASTIC_PASSWORD}"
-            ELASTICSEARCH_PORT: "${ELASTICSEARCH_PORT:-9200}"
-            ELASTIC_KEYSTORE_PASS: "${ELASTIC_KEYSTORE_PASS}" # Password for the keystore
-
-            LOGSTASH_USER: "${LOGSTASH_USER}"
-            LOGSTASH_PASSWORD: "${LOGSTASH_PASSWORD}"
-
-            KIBANA_USER: "${KIBANA_USER}"
-            KIBANA_PASSWORD: "${KIBANA_PASSWORD}"
         volumes:
             - "elasticsearch-data:/usr/share/elasticsearch/data/"
             - "elasticsearch-certs:/usr/share/elasticsearch/config/certs/"
+            - "./.secrets/elasticsearch_vault_token:/run/secrets/elasticsearch_vault_token"
 
             - "./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro"
             - "./elk/elasticsearch/config/generate-certs.sh:/usr/share/elasticsearch/config/generate-certs.sh:ro"
             - "./elk/elasticsearch/config/startup.sh:/usr/share/elasticsearch/config/startup.sh:ro"
             - "./elk/elasticsearch/config/ilm-policy.json:/usr/share/elasticsearch/config/ilm-policy.json:ro"
             - "./elk/elasticsearch/setup-single-node.sh:/usr/share/elasticsearch/setup-single-node.sh:ro"
-        command: ["/usr/share/elasticsearch/config/startup.sh"]
         user: root # Run as root to ensure we can set permissions
         ports:
-            - "${ELASTICSEARCH_PORT:-9200}:${ELASTICSEARCH_PORT:-9200}"
+            - "9200:9200"
         networks:
             ft-transcendence-net:
                 ipv4_address: &elasticsearch-ip 10.42.42.3
-        restart: unless-stopped
-        healthcheck:
-            test:
-                [
-                    "CMD",
-                    "curl",
-                    "--insecure",
-                    "--user",
-                    "elastic:${ELASTIC_PASSWORD}",
-                    "--silent",
-                    "--fail",
-                    "https://localhost:${ELASTICSEARCH_PORT}/_cluster/health",
-                ]
-            interval: 30s
-            timeout: 10s
-            retries: 5
+        depends_on:
+            vault:
+                restart: true
+                condition: service_healthy
+        restart: no # cannot restart because of expiring vault token
 
     logstash:
         profiles: [elk]
-        image: docker.elastic.co/logstash/logstash:8.17.3
+        build: "./elk/logstash/"
         container_name: logstash
         volumes:
             - "logstash-data:/usr/share/logstash/data/"
             - "elasticsearch-certs:/etc/logstash/config/certs/"
+            - "./.secrets/logstash_vault_token:/run/secrets/logstash_vault_token"
+
             - "./elk/logstash/pipeline/:/usr/share/logstash/pipeline/:ro"
             - "./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro"
             - "./elk/logstash/scripts/import-logstash-ca.sh:/usr/local/bin/import-logstash-ca.sh:ro"
         ports:
-            - "${LOGSTASH_PORT:-5050}:${LOGSTASH_PORT:-5050}/tcp"
-            - "${LOGSTASH_PORT:-5050}:${LOGSTASH_PORT:-5050}/udp"
-            - "${LOGSTASH_API_PORT:-9600}:${LOGSTASH_API_PORT:-9600}"
-        environment:
-            LS_JAVA_OPTS: "-Xmx256m -Xms256m"
-            DEBUG: "${DEBUG:-false}"
-
-            ELASTIC_PASSWORD: "${ELASTIC_PASSWORD}"
-            ELASTICSEARCH_PORT: "${ELASTICSEARCH_PORT:-9200}"
-
-            LOGSTASH_PORT: "${LOGSTASH_PORT:-5050}"
-            LOGSTASH_USER: "${LOGSTASH_USER}"
-            LOGSTASH_PASSWORD: "${LOGSTASH_PASSWORD}"
-
-            KIBANA_USER: "${KIBANA_USER}"
-            KIBANA_PASSWORD: "${KIBANA_PASSWORD}"
+            - "5050:5050/tcp"
+            - "5050:5050/udp"
+            - "9600:9600"
         networks:
             ft-transcendence-net:
                 ipv4_address: &logstash-ip 10.42.42.4
         depends_on:
             elasticsearch:
                 condition: service_healthy
-        restart: unless-stopped
-        healthcheck:
-            test:
-                [
-                    "CMD",
-                    "curl",
-                    "--silent",
-                    "--fail",
-                    "http://localhost:${LOGSTASH_API_PORT:-9600}/_node/stats",
-                ]
-            interval: 10s
-            timeout: 5s
-            retries: 3
-            start_period: 30s
-        entrypoint:
-            ["sh", "-c", "/usr/local/bin/import-logstash-ca.sh && /usr/local/bin/docker-entrypoint"]
+            vault:
+                restart: true
+                condition: service_healthy
+        restart: no # cannot restart because of expiring vault token
 
     kibana:
         profiles: [elk]
-        image: docker.elastic.co/kibana/kibana:8.17.3
+        build: "./elk/kibana/"
         container_name: kibana
         ports:
-            - "${KIBANA_PORT:-5601}:${KIBANA_PORT:-5601}"
-        environment:
-            SERVER_NAME: "kibana"
-            SERVER_HOST: "0.0.0.0"
-            SERVER_PORT: ${KIBANA_PORT:-5601}
-            XPACK_SECURITY_ENABLED: "true"
-
-            ELASTICSEARCH_HOSTS: "https://elasticsearch:${ELASTICSEARCH_PORT:-9200}"
-            ELASTICSEARCH_PORT: "${ELASTICSEARCH_PORT:-9200}"
-            ELASTICSEARCH_USERNAME: "${KIBANA_USER}"
-            ELASTICSEARCH_PASSWORD: "${KIBANA_PASSWORD}"
-            ELASTICSEARCH_SSL_VERIFICATIONMODE: "none"
-
-            KIBANA_USER: "${KIBANA_USER}"
-            KIBANA_PASSWORD: "${KIBANA_PASSWORD}"
+            - "5601:5601"
         volumes:
             - "kibana-data:/usr/share/kibana/data"
+            - "./.secrets/kibana_vault_token:/run/secrets/kibana_vault_token"
+
             - "./elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro"
             - "./elk/kibana/dashboards:/usr/share/kibana/dashboards/:ro"
             - "./elk/kibana/setup-dashboards.sh:/usr/share/kibana/setup-dashboards.sh:ro"
@@ -246,29 +183,16 @@ services:
         depends_on:
             elasticsearch:
                 condition: service_healthy
-        restart: unless-stopped
-        healthcheck:
-            test: ["CMD", "curl", "--fail", "http://localhost:${KIBANA_PORT:-5601}/api/status"]
-            interval: 30s
-            timeout: 10s
-            retries: 5
-            start_period: 30s
-
-        command: >
-            sh -ec '
-              /usr/local/bin/kibana-docker &
-              KIBANA_PID=$!
-              echo "Kibana starting with PID $$KIBANA_PID"
-              sleep 120
-              /usr/share/kibana/setup-dashboards.sh &
-              wait $$KIBANA_PID
-            '
+            vault:
+                restart: true
+                condition: service_healthy
+        restart: no # cannot restart because of expiring vault token
 
     vault:
         build:
             context: "./vault/"
             args:
-                VAULT_API_PORT: ${VAULT_API_PORT:-8200}
+                VAULT_API_PORT: "${VAULT_API_PORT:-8200}"
         container_name: vault
         restart: unless-stopped
         networks:
@@ -286,13 +210,18 @@ services:
             - "vault-storage:/vault/file/"
             - "vault-secret:/vault/secret/"
             - "vault-logs:/vault/logs/"
-            - "./env.json:/vault/config/env.json:ro"
+
             - "./.secrets/backend_vault_token:/run/secrets/backend_vault_token"
+            - "./.secrets/elasticsearch_vault_token:/run/secrets/elasticsearch_vault_token"
+            - "./.secrets/logstash_vault_token:/run/secrets/logstash_vault_token"
+            - "./.secrets/kibana_vault_token:/run/secrets/kibana_vault_token"
+
+            - "./env.json:/vault/config/env.json:ro"
         environment:
             <<: *env-watch
-            LOGLEVEL: "debug"
-            SAVE_UNSEAL_KEYS: "1"
-            SAVE_ROOT_TOKEN: "1"
+            LOG_LEVEL: "${VAULT_LOG_LEVEL:-info}"
+            SAVE_UNSEAL_KEYS: "${SAVE_UNSEAL_KEYS:-0}"
+            SAVE_ROOT_TOKEN: "${SAVE_ROOT_TOKEN:-0}"
         tty: true
         develop:
             watch:
@@ -313,16 +242,16 @@ services:
                   target: "/vault/policies/"
 
 volumes:
-    vault-storage:
-    vault-secret:
     vault-logs:
+    vault-secret:
+    vault-storage:
     caddy-data:
     drizzle:
     frontend:
     elasticsearch-data:
     elasticsearch-certs: # New volume for certificates
-    kibana-data:
     logstash-data:
+    kibana-data:
 
 networks:
     ft-transcendence-net:
diff --git a/config.env b/config.env
index 10a7a999..b20d84af 100644
--- a/config.env
+++ b/config.env
@@ -31,17 +31,18 @@ PROD_CADDY_EXTRA_SITE_DIRECTIVES=
 
 LIVE_RELOAD_PORT=35725
 
+#TODO can be handled in env.json if Caddy is also in vault
 BACKEND_PORT=3000
 DB_PATH=./drizzle/db.sqlite
 
+# Vault configuration
 VAULT_API_PORT=8200
 
-ELASTICSEARCH_PORT=9200
+#TODO turn off for eval/production
+SAVE_UNSEAL_KEYS=1
 
-#LOGSTASH_HOST=logstash
-LOGSTASH_PORT=5050
-LOGSTASH_API_PORT=9600
-LOGSTASH_USER=logstash_writer
+#TOOD turn off for eval/production
+SAVE_ROOT_TOKEN=1
 
-KIBANA_PORT=5601
-KIBANA_USER=kibana_transcendence
+#TODO turn off for eval/production
+VAULT_LOG_LEVEL=debug
diff --git a/elk/elasticsearch/Dockerfile b/elk/elasticsearch/Dockerfile
new file mode 100644
index 00000000..383342ad
--- /dev/null
+++ b/elk/elasticsearch/Dockerfile
@@ -0,0 +1,77 @@
+### Customization Point 1 ###
+FROM docker.elastic.co/elasticsearch/elasticsearch:8.17.3
+
+USER 0
+
+ADD --chmod=755 https://github.com/moparisthebest/static-curl/releases/download/v8.11.0/curl-amd64 /usr/bin/curl
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt update -y && apt install -y jq
+
+COPY --chmod=755 <<EOF /entrypoint.sh
+#!/bin/bash
+#Must be bash since sh (specifically dash) sanitizes all environment variables which are not
+#valid identifiers (e.g. the case for "bootstrap.memory_lock=true"). Bash passes them through, which
+#is in line with POSIX.
+
+set -e # exit on any error
+set -u # treat failed expansion as error
+#set -vx # for debugging
+
+### Customization Point 2 ###
+service=elasticsearch
+
+vault_token=\$(cat "/run/secrets/\${service}_vault_token")
+vault_addr=http://vault:8200
+
+get_all_secrets_as_env_params () {
+	curl --no-progress-meter --header "X-Vault-Token: \$vault_token" \\
+		"\$vault_addr/v1/secret/data/\$service" | jq --raw-output '
+			[
+				.data.data | to_entries[] |
+					"'\\''"
+					+ .key
+					+ "="
+					+ (
+						if .value | type == "string" then
+							(
+								.value |
+									gsub(
+										"'\\''";
+										"'\\''\\\\'\\'''\\''"
+									)
+							)
+						else
+							.value
+						end
+					  )
+					+ "'\\''"
+			] | join(" ")
+		'
+}
+
+env_params=\$(get_all_secrets_as_env_params)
+
+# Print all secrets for logging
+printf "Environment start"
+eval printf '"%s\\n"' "\$env_params"
+printf "Environment end"
+
+# Truncate file for good measure
+: > "/run/secrets/\${service}_vault_token"
+
+### Customization Point 3 ###
+eval exec env -- "\$env_params" /bin/tini -- '"\$@"'
+EOF
+
+CMD ["/usr/share/elasticsearch/config/startup.sh"]
+
+# No need to set WORKDIR, will be inherited from base image
+# No need to set CMD, will be inherited from base image
+# No need to set any ENV, will be inherited from base image
+# No need to set USER, will be inherited from base image
+
+# Healthcheck defined in ${REPO_ROOT}/elk/elasticsearch/config/startup.sh
+HEALTHCHECK --interval=30s --timeout=10s --retries=5 --start-period=60s --start-interval=1s CMD ["grep", "-Fxq", "0", "/tmp/healthcheck"]
+
+USER 1000
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/elk/elasticsearch/config/generate-certs.sh b/elk/elasticsearch/config/generate-certs.sh
index 18252bf3..18e25774 100755
--- a/elk/elasticsearch/config/generate-certs.sh
+++ b/elk/elasticsearch/config/generate-certs.sh
@@ -2,9 +2,10 @@
 
 # Exit on error
 set -e
+#set -vx
 
 CERTS_DIR="/usr/share/elasticsearch/config/certs"
-KEYSTORE_PASSWORD=${ELASTIC_KEYSTORE_PASS:-changeme}
+KEYSTORE_PASSWORD=${ELASTIC_KEYSTORE_PASS}
 
 # Create certs directory if it doesn't exist and set proper permissions
 chown -R elasticsearch:elasticsearch "$CERTS_DIR"
diff --git a/elk/elasticsearch/config/startup.sh b/elk/elasticsearch/config/startup.sh
index 2019f9ee..c2b79fca 100755
--- a/elk/elasticsearch/config/startup.sh
+++ b/elk/elasticsearch/config/startup.sh
@@ -1,6 +1,7 @@
 #!/bin/bash
 
 set -e
+#set -vx
 
 : "${ELASTIC_PASSWORD:?Missing ELASTIC_PASSWORD}"
 : "${LOGSTASH_USER:?Missing LOGSTASH_USER}"
@@ -9,6 +10,17 @@ set -e
 : "${KIBANA_PASSWORD:?Missing KIBANA_PASSWORD}"
 : "${ELASTIC_KEYSTORE_PASS:?Missing ELASTIC_KEYSTORE_PASS}"
 : "${ELASTICSEARCH_PORT:?Missing ELASTICSEARCH_PORT}"
+: "${ELASTIC_USER:?Missing ELASTIC_USER}" # MUST be "elastic"! https://discuss.elastic.co/t/how-to-change-the-username-by-own-instead-of-elastic/337552/2
+
+# Healthcheck background service (because only this process has access to the environment
+# and an external healthcheck would need access to those, which is not intended)
+TERM=linux setsid -f watch -xtcn5 curl \
+		--no-progress-meter \
+		--insecure \
+		--user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" \
+		--fail \
+		--write-out '%output{/tmp/healthcheck}%{exitcode}' \
+		"https://localhost:${ELASTICSEARCH_PORT}/_cluster/health" 1>/dev/null 2>&1
 
 # Set up permissions for certificates directory
 CERTS_DIR="/usr/share/elasticsearch/config/certs"
@@ -17,7 +29,7 @@ chown -R elasticsearch:elasticsearch "$CERTS_DIR"
 chmod -R 770 "$CERTS_DIR"
 
 # Checking if certificates exist
-su  --whitelist-environment='ELASTIC_PASSWORD,LOGSTASH_USER,LOGSTASH_PASSWORD,KIBANA_USER,KIBANA_PASSWORD,ELASTIC_KEYSTORE_PASS,ELASTICSEARCH_PORT' \
+su  --whitelist-environment='ELASTIC_PASSWORD,LOGSTASH_USER,LOGSTASH_PASSWORD,KIBANA_USER,KIBANA_PASSWORD,ELASTIC_KEYSTORE_PASS,ELASTICSEARCH_PORT,ELASTIC_USER' \
     --command="/usr/share/elasticsearch/config/generate-certs.sh" \
     --login \
     elasticsearch
@@ -33,7 +45,7 @@ if [ ! -f "/usr/share/elasticsearch/config/elasticsearch.keystore" ]; then
 fi
 
 # Run the setup script as elasticsearch user
-su  --whitelist-environment='ELASTIC_PASSWORD,LOGSTASH_USER,LOGSTASH_PASSWORD,KIBANA_USER,KIBANA_PASSWORD,ELASTIC_KEYSTORE_PASS,ELASTICSEARCH_PORT' \
+su  --whitelist-environment='ELASTIC_PASSWORD,LOGSTASH_USER,LOGSTASH_PASSWORD,KIBANA_USER,KIBANA_PASSWORD,ELASTIC_KEYSTORE_PASS,ELASTICSEARCH_PORT,ELASTIC_USER' \
     --command="/usr/share/elasticsearch/setup-single-node.sh" \
     --login \
     elasticsearch
diff --git a/elk/elasticsearch/setup-ilm.sh b/elk/elasticsearch/setup-ilm.sh
index 45d26302..f4915972 100755
--- a/elk/elasticsearch/setup-ilm.sh
+++ b/elk/elasticsearch/setup-ilm.sh
@@ -13,13 +13,13 @@ done
 
 # Apply ILM policy
 echo "Creating Index Lifecycle Management policy..."
-curl -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/_ilm/policy/ft-transcendence-logs-policy" \
+curl --no-progress-meter -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/_ilm/policy/ft-transcendence-logs-policy" \
   --header 'Content-Type: application/json' \
   --data "@/usr/share/elasticsearch/config/ilm-policy.json"
 
 # Create index template with ILM policy
 echo "Creating index template with ILM policy..."
-curl -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/_template/ft-transcendence-logs" \
+curl --no-progress-meter -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/_template/ft-transcendence-logs" \
   --header 'Content-Type: application/json' \
   --data '{
     "index_patterns": ["ft-transcendence-logs-*"],
@@ -46,7 +46,7 @@ curl -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/_template/ft-transcenden
 
 # Create initial index
 echo "Creating initial index..."
-curl -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/%3Cft-transcendence-logs-%7Bnow%2Fd%7D-000001%3E" \
+curl --no-progress-meter -X PUT "http://elasticsearch:${ELASTICSEARCH_PORT}/%3Cft-transcendence-logs-%7Bnow%2Fd%7D-000001%3E" \
   --header 'Content-Type: application/json' \
   --data '{
     "aliases": {
diff --git a/elk/elasticsearch/setup-single-node.sh b/elk/elasticsearch/setup-single-node.sh
index 29b08ac0..a6b3c8e9 100755
--- a/elk/elasticsearch/setup-single-node.sh
+++ b/elk/elasticsearch/setup-single-node.sh
@@ -1,6 +1,7 @@
 #!/bin/bash
 
 set -e
+#set -vx
 
 : "${ELASTIC_PASSWORD:?Missing ELASTIC_PASSWORD}"
 : "${ELASTICSEARCH_PORT:?Missing ELASTICSEARCH_PORT}"
@@ -8,36 +9,38 @@ set -e
 : "${KIBANA_PASSWORD:?Missing KIBANA_PASSWORD}"
 : "${LOGSTASH_USER:?Missing LOGSTASH_USER}"
 : "${LOGSTASH_PASSWORD:?Missing LOGSTASH_PASSWORD}"
+: "${ELASTIC_USER:?Missing ELASTIC_USER}" # MUST be "elastic"! https://discuss.elastic.co/t/how-to-change-the-username-by-own-instead-of-elastic/337552/2
 
 # Start Elasticsearch in the background
 /usr/local/bin/docker-entrypoint.sh eswrapper &
 ES_PID=$!
 
 # Wait for Elasticsearch to start
-echo "Waiting for Elasticsearch to start..."
-until curl --silent --insecure --user "elastic:${ELASTIC_PASSWORD}" --fail "https://localhost:${ELASTICSEARCH_PORT}/_cluster/health"; do
+i=0
+until echo "Waiting for Elasticsearch to start ($i seconds passed)..."; curl --no-progress-meter --insecure --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --fail "https://localhost:${ELASTICSEARCH_PORT}/_cluster/health"; do
     sleep 5
+	i=$((i + 5))
 done
 
 # Create users and set up security
 echo "Setting up users and security..."
 
 # Create logstash user
-curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${LOGSTASH_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
+curl --no-progress-meter --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${LOGSTASH_USER}" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data '{
   "password" : "'"${LOGSTASH_PASSWORD}"'",
   "roles" : [ "superuser" ],
   "full_name" : "Logstash User"
 }'
 
 # Create kibana user with both superuser and kibana_system roles
-curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${KIBANA_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
+curl --no-progress-meter --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${KIBANA_USER}" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data '{
   "password" : "'"${KIBANA_PASSWORD}"'",
   "roles" : [ "superuser", "kibana_system" ],
   "full_name" : "Kibana System User"
 }'
 
 # Set up ILM policy
-curl --insecure -X PUT "https://localhost:${ELASTICSEARCH_PORT}/_ilm/policy/logs" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data "@/usr/share/elasticsearch/config/ilm-policy.json"
+curl --no-progress-meter --insecure -X PUT "https://localhost:${ELASTICSEARCH_PORT}/_ilm/policy/logs" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data "@/usr/share/elasticsearch/config/ilm-policy.json"
 
 # Wait for the original Elasticsearch process
 wait $ES_PID
diff --git a/elk/kibana/Dockerfile b/elk/kibana/Dockerfile
new file mode 100644
index 00000000..cd8f56d4
--- /dev/null
+++ b/elk/kibana/Dockerfile
@@ -0,0 +1,74 @@
+### Customization Point 1 ###
+FROM docker.elastic.co/kibana/kibana:8.17.3
+
+USER 0
+
+ADD --chmod=755 https://github.com/moparisthebest/static-curl/releases/download/v8.11.0/curl-amd64 /usr/bin/curl
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt update -y && apt install -y curl jq
+
+COPY --chmod=755 <<EOF /entrypoint.sh
+#!/bin/sh
+
+set -e # exit on any error
+set -u # treat failed expansion as error
+#set -vx # for debugging
+
+### Customization Point 2 ###
+service=kibana
+
+vault_token=\$(cat "/run/secrets/\${service}_vault_token")
+vault_addr=http://vault:8200
+
+get_all_secrets_as_env_params () {
+	curl --no-progress-meter --header "X-Vault-Token: \$vault_token" \\
+		"\$vault_addr/v1/secret/data/\$service" | jq --raw-output '
+			[
+				.data.data | to_entries[] |
+					"'\\''"
+					+ .key
+					+ "="
+					+ (
+						if .value | type == "string" then
+							(
+								.value |
+									gsub(
+										"'\\''";
+										"'\\''\\\\'\\'''\\''"
+									)
+							)
+						else
+							.value
+						end
+					  )
+					+ "'\\''"
+			] | join(" ")
+		'
+}
+
+env_params=\$(get_all_secrets_as_env_params)
+
+# Print all secrets for logging
+printf "Environment start"
+eval printf '"%s\\n"' "\$env_params"
+printf "Environment end"
+
+# Truncate file for good measure
+: > "/run/secrets/\${service}_vault_token"
+
+### Customization Point 3 ###
+eval exec env -- "\$env_params" /bin/tini -- '"\$@"'
+EOF
+
+CMD ["sh", "-xec", ": Healthcheck ; TERM=linux setsid -f watch -xtcn5 curl --no-progress-meter --fail --write-out '%output{/tmp/healthcheck}%{exitcode}' \"http://localhost:${KIBANA_PORT}/api/status\" 1>/dev/null 2>&1 ; /usr/local/bin/kibana-docker & KIBANA_PID=$! ; echo \"Kibana starting with PID $KIBANA_PID\" ; sleep 120 ; /usr/share/kibana/setup-dashboards.sh & wait $KIBANA_PID"]
+
+# No need to set WORKDIR, will be inherited from base image
+# No need to set CMD, will be inherited from base image
+# No need to set any ENV, will be inherited from base image
+# No need to set USER, will be inherited from base image
+
+# Healthcheck defined inline in the Dockerfile CMD directive, 7 lines above /\
+HEALTHCHECK --interval=30s --timeout=10s --retries=5 --start-period=60s --start-interval=1s CMD ["grep", "-Fxq", "0", "/tmp/healthcheck"]
+
+USER 1000
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/elk/kibana/setup-dashboards.sh b/elk/kibana/setup-dashboards.sh
index bfbd5288..8b61a56d 100755
--- a/elk/kibana/setup-dashboards.sh
+++ b/elk/kibana/setup-dashboards.sh
@@ -2,7 +2,7 @@
 
 # Wait for Kibana to be fully operational
 echo "Waiting for Kibana to be fully operational..."
-while ! curl --silent "http://kibana:5601/api/status" | grep -q '"overall":{"level":"available"'; do
+while ! curl --no-progress-meter "http://kibana:5601/api/status" | grep -q '"overall":{"level":"available"'; do
   echo "Kibana is not yet fully available - waiting..."
   sleep 10
 done
diff --git a/elk/logstash/Dockerfile b/elk/logstash/Dockerfile
new file mode 100644
index 00000000..1c348164
--- /dev/null
+++ b/elk/logstash/Dockerfile
@@ -0,0 +1,75 @@
+### Customization Point 1 ###
+FROM docker.elastic.co/logstash/logstash:8.17.3
+
+USER 0
+
+ADD --chmod=755 https://github.com/moparisthebest/static-curl/releases/download/v8.11.0/curl-amd64 /usr/bin/curl
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt update -y && apt install -y jq
+
+COPY --chmod=755 <<EOF /entrypoint.sh
+#!/bin/bash
+#Must be bash since sh (specifically dash) sanitizes all environment variables which are not
+#valid identifiers (e.g. the case for "bootstrap.memory_lock=true"). Bash passes them through, which
+#is in line with POSIX.
+
+set -e # exit on any error
+set -u # treat failed expansion as error
+#set -vx # for debugging
+
+### Customization Point 2 ###
+service=logstash
+
+vault_token=\$(cat "/run/secrets/\${service}_vault_token")
+vault_addr=http://vault:8200
+
+get_all_secrets_as_env_params () {
+	curl --no-progress-meter --header "X-Vault-Token: \$vault_token" \\
+		"\$vault_addr/v1/secret/data/\$service" | jq --raw-output '
+			[
+				.data.data | to_entries[] |
+					"'\\''"
+					+ .key
+					+ "="
+					+ (
+						if .value | type == "string" then
+							(
+								.value |
+									gsub(
+										"'\\''";
+										"'\\''\\\\'\\'''\\''"
+									)
+							)
+						else
+							.value
+						end
+					  )
+					+ "'\\''"
+			] | join(" ")
+		'
+}
+
+env_params=\$(get_all_secrets_as_env_params)
+
+# Print all secrets for logging
+printf "Environment start"
+eval printf '"%s\\n"' "\$env_params"
+printf "Environment end"
+
+# Truncate file for good measure
+: > "/run/secrets/\${service}_vault_token"
+
+### Customization Point 3 ###
+eval exec env -- "\$env_params" bash -c "'/usr/local/bin/import-logstash-ca.sh && /usr/local/bin/docker-entrypoint'"
+EOF
+
+# No need to set WORKDIR, will be inherited from base image
+# No need to set CMD, will be inherited from base image
+# No need to set any ENV, will be inherited from base image
+# No need to set USER, will be inherited from base image
+
+# Healthcheck defined in ${REPO_ROOT}/elk/logstash/scripts/import-logstash-ca.sh
+HEALTHCHECK --interval=10s --timeout=5s --retries=3 --start-period=60s --start-interval=1s CMD ["grep", "-Fxq", "0", "/tmp/healthcheck"]
+
+USER 1000
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/elk/logstash/scripts/import-logstash-ca.sh b/elk/logstash/scripts/import-logstash-ca.sh
index 86da2b24..f640ac01 100755
--- a/elk/logstash/scripts/import-logstash-ca.sh
+++ b/elk/logstash/scripts/import-logstash-ca.sh
@@ -1,4 +1,14 @@
-#!/bin/sh
+#!/bin/bash
+
+#set -vx
+
+# Healthcheck background service (because only this process has access to the environment
+# and an external healthcheck would need access to those, which is not intended)
+TERM=linux setsid -f watch -xtcn5 curl \
+	--no-progress-meter \
+	--fail \
+	--write-out '%output{/tmp/healthcheck}%{exitcode}' \
+	"http://localhost:${LOGSTASH_API_PORT}/_node/stats" 1>/dev/null 2>&1
 
 # Path to the CA cert inside the container
 CA_CERT_PATH="/etc/logstash/config/certs/ca.crt"
@@ -9,7 +19,7 @@ TRUSTSTORE_PATH="/usr/share/logstash/jdk/lib/security/cacerts"
 
 KEYTOOL="/usr/share/logstash/jdk/bin/keytool"
 # Default truststore password
-STOREPASS="changeme"
+STOREPASS="generic_password"
 
 if [ ! -f "$CA_CERT_PATH" ]; then
   echo "CA certificate not found at $CA_CERT_PATH"
diff --git a/env.json b/env.json
index 2df7c50f..556e3823 100644
--- a/env.json
+++ b/env.json
@@ -1,5 +1,51 @@
 {
     "backend": {
         "JWT_SECRET": "{{alnum:64}}"
+    },
+    "elasticsearch": {
+        "ES_JAVA_OPTS": "-Xms512m -Xmx512m",
+
+        "discovery.type": "single-node",
+        "bootstrap.memory_lock": "true",
+        "cluster.name": "ft-transcendence-logs",
+        "node.name": "single-node",
+
+        "ELASTICSEARCH_PORT": "9200",
+        "ELASTIC_KEYSTORE_PASS": "{{alnum:32}}",
+        "ELASTIC_PASSWORD": "{{alnum:32}}",
+        "ELASTIC_USER": "elastic",
+        "KIBANA_PASSWORD": "{{alnum:32}}",
+        "KIBANA_USER": "kibana_transcendence",
+        "LOGSTASH_PASSWORD": "{{alnum:32}}",
+        "LOGSTASH_USER": "logstash_writer"
+    },
+    "logstash": {
+        "LS_JAVA_OPTS": "-Xmx256m -Xms256m",
+        "DEBUG": "true",
+
+        "ELASTICSEARCH_PORT": "{{ref:'elasticsearch','ELASTICSEARCH_PORT'}}",
+        "ELASTIC_PASSWORD": "{{ref:'elasticsearch','ELASTIC_PASSWORD'}}",
+        "KIBANA_PASSWORD": "{{ref:'elasticsearch','KIBANA_PASSWORD'}}",
+        "KIBANA_USER": "{{ref:'elasticsearch','KIBANA_USER'}}",
+        "LOGSTASH_API_PORT": "9600",
+        "LOGSTASH_HOST": "logstash",
+        "LOGSTASH_PASSWORD": "{{ref:'elasticsearch','LOGSTASH_PASSWORD'}}",
+        "LOGSTASH_PORT": "5050",
+        "LOGSTASH_USER": "{{ref:'elasticsearch','LOGSTASH_USER'}}"
+    },
+    "kibana": {
+        "XPACK_SECURITY_ENABLED": "true",
+        "SERVER_NAME": "kibana",
+        "SERVER_HOST": "0.0.0.0",
+        "SERVER_PORT": "5601",
+
+        "ELASTICSEARCH_HOSTS": "https://elasticsearch:{{ref:'elasticsearch','ELASTICSEARCH_PORT'}}",
+        "ELASTICSEARCH_PASSWORD": "{{ref:'elasticsearch','KIBANA_PASSWORD'}}",
+        "ELASTICSEARCH_PORT": "{{ref:'elasticsearch','ELASTICSEARCH_PORT'}}",
+        "ELASTICSEARCH_SSL_VERIFICATIONMODE": "none",
+        "ELASTICSEARCH_USERNAME": "{{ref:'elasticsearch','KIBANA_USER'}}",
+        "KIBANA_PASSWORD": "{{ref:'elasticsearch','KIBANA_PASSWORD'}}",
+        "KIBANA_PORT": "{{ref:'kibana','SERVER_PORT'}}",
+        "KIBANA_USER": "{{ref:'elasticsearch','KIBANA_USER'}}"
     }
 }
diff --git a/script/quote_exceptions b/script/quote_exceptions
index 66c9b060..a9955ad5 100644
--- a/script/quote_exceptions
+++ b/script/quote_exceptions
@@ -98,11 +98,8 @@ backend/src/utils/logger.ts://         score: `player1: ${state.score.player1},
 # Other exceptions
 elk/kibana/setup-dashboards.sh:while ! curl --silent "http://kibana:5601/api/status" | grep -q '"overall":{"level":"available"'; do
 elk/elasticsearch/setup-ilm.sh:until curl --silent "http://elasticsearch:${ELASTICSEARCH_PORT}/_cluster/health" | grep -q '"status":"green"\|"status":"yellow"'; do
-elk/elasticsearch/setup-single-node.sh:curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${LOGSTASH_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
-elk/elasticsearch/setup-single-node.sh:curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${KIBANA_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
-elk/elasticsearch/setup-single-node.sh:curl --insecure -X PUT "https://localhost:${ELASTICSEARCH_PORT}/_ilm/policy/logs" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data "@/usr/share/elasticsearch/config/ilm-policy.json"
-./elk/kibana/setup-dashboards.sh:while ! curl --silent "http://kibana:5601/api/status" | grep -q '"overall":{"level":"available"'; do
-./elk/elasticsearch/setup-single-node.sh:curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${LOGSTASH_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
-./elk/elasticsearch/setup-single-node.sh:curl --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${KIBANA_USER}" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data '{
-./elk/elasticsearch/setup-single-node.sh:curl --insecure -X PUT "https://localhost:${ELASTICSEARCH_PORT}/_ilm/policy/logs" --header 'Content-Type: application/json' --user "elastic:${ELASTIC_PASSWORD}" --data "@/usr/share/elasticsearch/config/ilm-policy.json"
-./elk/elasticsearch/setup-ilm.sh:until curl --silent "http://elasticsearch:${ELASTICSEARCH_PORT}/_cluster/health" | grep -q '"status":"green"\|"status":"yellow"'; do
+elk/elasticsearch/setup-single-node.sh:curl --no-progress-meter --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${LOGSTASH_USER}" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data '{
+elk/elasticsearch/setup-single-node.sh:curl --no-progress-meter --insecure "https://localhost:${ELASTICSEARCH_PORT}/_security/user/${KIBANA_USER}" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data '{
+elk/elasticsearch/setup-single-node.sh:curl --no-progress-meter --insecure -X PUT "https://localhost:${ELASTICSEARCH_PORT}/_ilm/policy/logs" --header 'Content-Type: application/json' --user "${ELASTIC_USER}:${ELASTIC_PASSWORD}" --data "@/usr/share/elasticsearch/config/ilm-policy.json"
+elk/kibana/setup-dashboards.sh:while ! curl --no-progress-meter "http://kibana:5601/api/status" | grep -q '"overall":{"level":"available"'; do
+env.json:        "ELASTICSEARCH_HOSTS": "https://elasticsearch:{{ref:'elasticsearch','ELASTICSEARCH_PORT'}}",
diff --git a/vault/README.md b/vault/README.md
index 8e0ac2fe..0f5e61c2 100644
--- a/vault/README.md
+++ b/vault/README.md
@@ -203,7 +203,7 @@ Let's say your service is called `foo`, then the steps would be the following:
 
     set -e # exit on any error
     set -u # treat failed expansion as error
-    # set -x # for debugging
+    # set -vx # for debugging
 
     ### Customization Point 2 ###
     service=foo
@@ -212,7 +212,7 @@ Let's say your service is called `foo`, then the steps would be the following:
     vault_addr=http://vault:8200
 
     get_all_secrets_as_export_lines () {
-        curl --header "X-Vault-Token: \$vault_token" \\
+        curl --no-progress-meter --fail --header "X-Vault-Token: \$vault_token" \\
             "\$vault_addr/v1/secret/data/\$service" |
             jq --raw-output '
              .data.data|to_entries[] |
diff --git a/vault/entrypoint.sh b/vault/entrypoint.sh
index d6621d44..a3cebc09 100755
--- a/vault/entrypoint.sh
+++ b/vault/entrypoint.sh
@@ -3,9 +3,9 @@
 set -e
 set -u
 set -m
-#set -x
+#set -vx
 
-: "${LOGLEVEL:=INFO}"
+: "${LOG_LEVEL:=INFO}"
 : "${SAVE_UNSEAL_KEYS:=0}"
 : "${SAVE_ROOT_TOKEN:=0}"
 : "${vault_config:=/vault/config/config.hcl}"
@@ -54,7 +54,7 @@ log () {
 			set "(Invalid loglevel for log function): " "$*"
 			;;
 	esac
-	[ "$(loglevel_to_num "$level")" -gt "$(loglevel_to_num "$LOGLEVEL")" ] && return 0
+	[ "$(loglevel_to_num "$level")" -gt "$(loglevel_to_num "$LOG_LEVEL")" ] && return 0
 
 	printf '[ %b%-5s%b ]: %s\n' "$ansi_color" "$level" "$ansi_reset" "$(printf "$@")"
 	IFS=$OLDIFS
@@ -324,13 +324,13 @@ populate_kv_secrets () {
 revoke_all_client_tokens () {
 	info "Revoking all client (non-root) tokens"
 	while read -r accessor; do
-		display_name=$(vault token lookup -format=json -accessor "$accessor" | jq '.data.display_name')
-		policies=$(vault token lookup -format=json -accessor "$accessor" | jq '.data.policies[]' | xargs | tr ' ' ',')
+		display_name=$(vault token lookup -format=json -accessor "$accessor" | jq --raw-output '.data.display_name')
+		policies=$(vault token lookup -format=json -accessor "$accessor" | jq --raw-output '.data.policies[]' | xargs | tr ' ' ',')
 		if ! vault token lookup -format=json -accessor "$accessor" | jq --exit-status '.data.policies | index("root")' 1>/dev/null; then
-			info "Token with name $display_name and policies '$policies' is not a root token, revoking it"
+			info "Token with name '$display_name' and policies '$policies' is not a root token, revoking it"
 			1>/dev/null vault token revoke -accessor "$accessor"
 		else
-			info "Token with name $display_name and policies '$policies' is a root token, not revoking it"
+			info "Token with name '$display_name' and policies '$policies' is a root token, not revoking it"
 		fi
 	done <<-EOF
 	$(vault list -format=json auth/token/accessors | jq --raw-output '.[]')
@@ -363,7 +363,7 @@ create_policies () {
 	EOF
 }
 
-create_token_role () {
+create_token_roles () {
 	ensure_env_json
 
 	if [ "$WATCH" = "1" ]; then
@@ -378,6 +378,7 @@ create_token_role () {
 		info "Determining IP for service \033\133;93m%s\033\133m" "$service"
 		allowed_ip=$({ timeout 0.3 ping -q4 -s0 -w1 -W1 -c1 "$service" | grep -o '\([[:digit:]]\+\.\)\{3\}[[:digit:]]\+' | head -n 1; } 2>/dev/null)
 		[ -n "$allowed_ip" ] || die "Failed to resolve domain \033\133;93m%s\033\133m to an IP address" "$service"
+		debug "Service name resolves to: \033\133;93m%s -> %s\033\133m" "$service" "$allowed_ip"
 
 		info "Creating new role for service \033\133;93m%s\033\133m" "$service"
 		1>/dev/null vault write auth/token/roles/"$service" \
@@ -401,6 +402,8 @@ create_and_share_client_tokens () {
 	while read -r service; do
 		info "Creating new client token for service \033\133;93m%s\033\133m with role \033\133;93m%s\033\133m and with policy \033\133;93m%s\033\133m" "$service" "$service" "$service"
 		vault token create -role="$service" -policy="$service" -format=json | jq --raw-output '.auth.client_token' > "$token_exchange_dir/${service}_vault_token"
+		test -s "$token_exchange_dir/${service}_vault_token" || die "File \033\133;93m%s\033\133m is empty!" "$token_exchange_dir/${service}_vault_token"
+		debug "%s token contents: \033\133;93m%s\033\133m" "$service" "$(cat "$token_exchange_dir/${service}_vault_token")"
 	done <<-EOF
 	$(<"$vault_env_json" jq --raw-output 'to_entries[].key')
 	EOF
@@ -438,7 +441,7 @@ main () {
 
 		revoke_all_client_tokens
 		create_policies
-		create_token_role
+		create_token_roles
 		create_and_share_client_tokens
 		#shutdown_server # For debugging
 	fi
diff --git a/vault/policies/elasticsearch.hcl b/vault/policies/elasticsearch.hcl
new file mode 100644
index 00000000..82e1a83f
--- /dev/null
+++ b/vault/policies/elasticsearch.hcl
@@ -0,0 +1,3 @@
+path "secret/data/elasticsearch" {
+	capabilities = ["read"]
+}
diff --git a/vault/policies/kibana.hcl b/vault/policies/kibana.hcl
new file mode 100644
index 00000000..214abf8b
--- /dev/null
+++ b/vault/policies/kibana.hcl
@@ -0,0 +1,3 @@
+path "secret/data/kibana" {
+	capabilities = ["read"]
+}
diff --git a/vault/policies/logstash.hcl b/vault/policies/logstash.hcl
new file mode 100644
index 00000000..8ef1fe65
--- /dev/null
+++ b/vault/policies/logstash.hcl
@@ -0,0 +1,3 @@
+path "secret/data/logstash" {
+	capabilities = ["read"]
+}
diff --git a/vault/replace_json_templates.py b/vault/replace_json_templates.py
index 6eb5fcca..31d03240 100755
--- a/vault/replace_json_templates.py
+++ b/vault/replace_json_templates.py
@@ -22,10 +22,13 @@
 import re
 import sys
 import json
+import shlex
 import random
 import string
 import contextlib
 
+# Maximum 5 levels of indirection
+PASSES = 5
 
 character_sets = {
     'alnum': string.ascii_letters + string.digits,
@@ -58,44 +61,106 @@ def parse_json(name: str, args: list[str]) -> dict:
             print(f"Error parsing JSON: {e}", file=sys.stderr, flush=True)
             return {}
 
-    if not data:
-        print(f"JSON object is empty", file=sys.stderr, flush=True)
     return data
 
-def replace_templates(s: str) -> str:
-    if (m := re.match(r'\{\{(\w+):(\d+)\}\}', s)):
-        template_type = m[1]
-        template_len = int(m[2])
+def resolve_json_path(json_object: dict | list | str | int | float | bool | None, json_path: list) -> dict | list | str | int | float | bool | None:
+    for key in json_path:
+        try:
+            if isinstance(json_object, dict) and isinstance(key, str):
+                json_object = json_object[key]
+            elif isinstance(json_object, list) and isinstance(key, int):
+                json_object = json_object[key]
+            elif isinstance(json_object, dict) or isinstance(json_object, list):
+                print(f'\033\133;91mWARNING\033\133m: Ref JSON Path: Trying to use a string index on a list or an int key on an object. Leaving template unmodified', file=sys.stderr, flush=True)
+                return "___error___"
+            else:
+                print(f'\033\133;91mWARNING\033\133m: Ref JSON Path: Trying to use a string index or an int key on something that is neither object nor list. Leaving template unmodified', file=sys.stderr, flush=True)
+                return "___error___"
+        except KeyError:
+            print(f'\033\133;91mWARNING\033\133m: Ref JSON Path: KeyError. Leaving template unmodified', file=sys.stderr, flush=True)
+            return "___error___"
+        except IndexError:
+            print(f'\033\133;91mWARNING\033\133m: Ref JSON Path: IndexError. Leaving template unmodified', file=sys.stderr, flush=True)
+            return "___error___"
+    return json_object
+
+def replacement_function(match: re.Match, *, references: bool) -> str:
+    global json_object
+    raw_match_string = match[0]
+    template_type = match[1]
+    template_argument = match[2]
+    if template_type in ['alnum', 'alpha', 'digit']:
         character_set = character_sets.get(template_type, '')
         if not character_set:
             print(f'\033\133;91mWARNING\033\133m: Unknown template type: \033\133;93m{template_type}\033\133m, supported types are \033\133;92m{", ".join(character_sets.keys())}\033\133m. Leaving it unmodified', file=sys.stderr, flush=True)
-            return s
-        return ''.join(random.choices(character_set, k=template_len))
-    elif re.match(r'\{\{.*\}\}', s):
-        print(f"\033\133;91mWARNING\033\133m: Found template string \033\133;93m{s}\033\133m, but it doesn't match the expected regex \033\133;94m\\{{\\{{(\\w+):(\\d+)\\}}\\}}\033\133m. Leaving it unmodified", file=sys.stderr, flush=True)
-        return s
+            return raw_match_string
+        return ''.join(random.choices(character_set, k=int(template_argument)))
+    elif template_type == 'ref':
+        if not references:
+            return raw_match_string
+        lexer = shlex.shlex(template_argument)
+        lexer.whitespace = ','
+        lexer.whitespace_split = True
+        lexer.quotes = "'"
+        try:
+            tokens = list(iter(lexer.get_token, ''))
+            json_path = []
+            for token in tokens:
+                if not token:
+                    print(f'\033\133;91mWARNING\033\133m: Ref JSON path contains empty string. Leaving template unmodified', file=sys.stderr, flush=True)
+                    return raw_match_string
+                elif token[0] == "'" == token[-1]:
+                    json_path.append(token[1:-1])
+                elif token.isdecimal():
+                    json_path.append(int(token))
+                else:
+                    print(f'\033\133;91mWARNING\033\133m: Ref JSON path contains invalid token: \033\133;93m{token}\033\133m. Leaving template unmodified', file=sys.stderr, flush=True)
+                    return raw_match_string
+            json_item = resolve_json_path(json_object, json_path)
+            if json_item == "___error___":
+                return raw_match_string
+            if not isinstance(json_item, str) or json_item is None:
+                    print(f'\033\133;91mWARNING\033\133m: Ref JSON path did not return a string. Leaving template unmodified', file=sys.stderr, flush=True)
+                    return raw_match_string
+            return json_item
+        except ValueError:
+            print(f'\033\133;91mWARNING\033\133m: Ref JSON path is wrongly quoted. Leaving template unmodified', file=sys.stderr, flush=True)
+            return raw_match_string
     else:
-        return s
-
-def transform_json(data: dict | list | str | int | float) -> dict:
-    new_data = {}
-    for key, value in data.items():
-        new_key = replace_templates(key)
-        if isinstance(value, str):
-            new_value = replace_templates(value)
-        else:
-            new_value = transform_json(value)
-        new_data[new_key] = new_value
-    return new_data
+        return raw_match_string
+
+def replace_templates(s: str, *, references: bool) -> str:
+    replaced = re.sub(r'\{\{(\w+):([^}]+)\}\}', lambda m: replacement_function(m, references=references), s)
+
+    return replaced
+
+def transform_json(data: dict | list | str | int | float | bool | None, *, references: bool) -> dict | list | str | int | float | bool | None:
+    if isinstance(data, dict):
+        new_data = {}
+        for key, value in data.items():
+            new_key = replace_templates(key, references=references)
+            new_value = transform_json(value, references=references)
+            new_data[new_key] = new_value
+        return new_data
+    elif isinstance(data, list):
+        new_data = []
+        for value in data:
+            new_data.append(transform_json(value, references=references))
+        return new_data
+    elif isinstance(data, str):
+        return replace_templates(data, references=references)
+    else:
+        return data
 
 def main(name: str, args: list[str]) -> int:
-    data = parse_json(name, args)
-    if not data:
-        return 1
+    global json_object
+    json_object = parse_json(name, args)
 
-    new_data = transform_json(data)
+    for _ in range(PASSES):
+        json_object = transform_json(json_object, references=False)
+        json_object = transform_json(json_object, references=True)
 
-    print(json.dumps(new_data, indent=4, ensure_ascii=False))
+    print(json.dumps(json_object, indent=4, ensure_ascii=False))
     return 0
 
 if __name__ == "__main__":


Reviews:
user: login: darrenkuro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T09:01:20Z
user: login: darrenkuro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T09:02:43Z
user: login: darrenkuro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T09:30:56Z
user: login: darrenkuro
body: Everything seems to be working fine as far as I could tell though. Approved since none of above was anything major. 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-19T09:41:54Z
user: login: cubernetes
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-19T12:05:01Z
user: login: cubernetes
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-19T12:21:04Z
user: login: cubernetes
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-19T12:22:23Z

Review comments:
diff_hunk: @@ -1,5 +1,5 @@
 # Change to docker-compose if you need
-DC := COMPOSE_BAKE=false docker compose
+DC := COMPOSE_BAKE=true docker compose
path: Makefile
user: login: darrenkuro
body: What is COMPSE_BAKE? Seems new, couldn't find a lot of info..
created_at: 2025-05-19T09:01:20Z
author_association: COLLABORATOR
diff_hunk: @@ -98,3 +101,12 @@ install:
 .PHONY: test
 test: install
 	npm --prefix=backend test
+
+# Usage: make get-env SERVICE=elasticsearch
+.PHONY: get-env
+get-env:
+	$(D) exec vault sh -c 'SERVICE=$(SERVICE); wget --header="X-Vault-Token: $$(cat /vault/secret/root_token)" --quiet --output-document=- "$$(cat /tmp/vault_addr)"/v1/secret/data/$$SERVICE | jq --raw-output --color-output .data.data'
+
+.PHONY: wait
+wait:
+	wait_for_service () { service=$$1; [ -n "$$service" ] || { echo "Expected 1 argument"; return 1; }; i=0; while [ ! "$$(2>/dev/null docker inspect -f "{{.State.Health.Status}}" "$$service")" = "healthy" ]; do printf "\015""Waiting_for_service_$${service}_to_start%$${i}s" "" | sed -u "s/ /./g;s/_/ /g"; i=$$((i + 1)); sleep 0.3; done; echo; } && start=$$(date +%s); wait_for_service vault && wait_for_service elasticsearch && wait_for_service logstash && wait_for_service kibana && wait_for_service backend && echo "Waiting 15 more seconds" && sleep 15 && echo "Frontend: http://localhost:8080" && echo "OpenAPI:  http://localhost:8080/api/docs" && echo "ELK       http://localhost:5601" ; now=$$(date +%s); min=$$(( (now - start) / 60)); sec=$$(( (now - start) % 60)); echo "Took $${min}m$${sec}s"
path: Makefile
user: login: darrenkuro
body: Since we already have a script dir, maybe we can make this a file instead of writing it in one line?

Also, this just let you know it's ready and how long it took, right? 
created_at: 2025-05-19T09:02:43Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,77 @@
+### Customization Point 1 ###
+FROM docker.elastic.co/elasticsearch/elasticsearch:8.17.3
+
+USER 0
+
+ADD --chmod=755 https://github.com/moparisthebest/static-curl/releases/download/v8.11.0/curl-amd64 /usr/bin/curl
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt update -y && apt install -y jq
+
+COPY --chmod=755 <<EOF /entrypoint.sh
+#!/bin/bash
+#Must be bash since sh (specifically dash) sanitizes all environment variables which are not
+#valid identifiers (e.g. the case for "bootstrap.memory_lock=true"). Bash passes them through, which
+#is in line with POSIX.
+
+set -e # exit on any error
+set -u # treat failed expansion as error
+#set -vx # for debugging
+
+### Customization Point 2 ###
+service=elasticsearch
+
+vault_token=\$(cat "/run/secrets/\${service}_vault_token")
+vault_addr=http://vault:8200
path: elk/elasticsearch/Dockerfile
user: login: darrenkuro
body: Hardcoded port that needs to be replaced? Same with the other two Dockerfiles
created_at: 2025-05-19T09:30:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,77 @@
+### Customization Point 1 ###
+FROM docker.elastic.co/elasticsearch/elasticsearch:8.17.3
+
+USER 0
+
+ADD --chmod=755 https://github.com/moparisthebest/static-curl/releases/download/v8.11.0/curl-amd64 /usr/bin/curl
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt update -y && apt install -y jq
+
+COPY --chmod=755 <<EOF /entrypoint.sh
+#!/bin/bash
+#Must be bash since sh (specifically dash) sanitizes all environment variables which are not
+#valid identifiers (e.g. the case for "bootstrap.memory_lock=true"). Bash passes them through, which
+#is in line with POSIX.
+
+set -e # exit on any error
+set -u # treat failed expansion as error
+#set -vx # for debugging
+
+### Customization Point 2 ###
+service=elasticsearch
+
+vault_token=\$(cat "/run/secrets/\${service}_vault_token")
+vault_addr=http://vault:8200
path: elk/elasticsearch/Dockerfile
user: login: cubernetes
body: Created an issue! 
created_at: 2025-05-19T12:05:01Z
author_association: OWNER
diff_hunk: @@ -1,5 +1,5 @@
 # Change to docker-compose if you need
-DC := COMPOSE_BAKE=false docker compose
+DC := COMPOSE_BAKE=true docker compose
path: Makefile
user: login: cubernetes
body: It's an opt in feature of compose that enables for parallelized builds (faster). I always had it explicitly disabled to get rid of the advertisement message docker compose tends to print, and also because it broke things for me when I was creating a CTF, but now I just enabled it and it seems to do what it promises to do.
created_at: 2025-05-19T12:21:04Z
author_association: OWNER
diff_hunk: @@ -98,3 +101,12 @@ install:
 .PHONY: test
 test: install
 	npm --prefix=backend test
+
+# Usage: make get-env SERVICE=elasticsearch
+.PHONY: get-env
+get-env:
+	$(D) exec vault sh -c 'SERVICE=$(SERVICE); wget --header="X-Vault-Token: $$(cat /vault/secret/root_token)" --quiet --output-document=- "$$(cat /tmp/vault_addr)"/v1/secret/data/$$SERVICE | jq --raw-output --color-output .data.data'
+
+.PHONY: wait
+wait:
+	wait_for_service () { service=$$1; [ -n "$$service" ] || { echo "Expected 1 argument"; return 1; }; i=0; while [ ! "$$(2>/dev/null docker inspect -f "{{.State.Health.Status}}" "$$service")" = "healthy" ]; do printf "\015""Waiting_for_service_$${service}_to_start%$${i}s" "" | sed -u "s/ /./g;s/_/ /g"; i=$$((i + 1)); sleep 0.3; done; echo; } && start=$$(date +%s); wait_for_service vault && wait_for_service elasticsearch && wait_for_service logstash && wait_for_service kibana && wait_for_service backend && echo "Waiting 15 more seconds" && sleep 15 && echo "Frontend: http://localhost:8080" && echo "OpenAPI:  http://localhost:8080/api/docs" && echo "ELK       http://localhost:5601" ; now=$$(date +%s); min=$$(( (now - start) / 60)); sec=$$(( (now - start) % 60)); echo "Took $${min}m$${sec}s"
path: Makefile
user: login: cubernetes
body: Good idea, I'll do that. And yes, it tells you how long it took, but more importantly you can edit the script to automatically open the browser for you, meanwhile you can be busy with other things.
created_at: 2025-05-19T12:22:23Z
author_association: OWNER

PR comments:
user: login: cubernetes
created_at: 2025-05-18T21:14:08Z
body: @darrenkuro can you confirm that this closes #99 as per your expectations?
author_association: OWNER
user: login: cubernetes
created_at: 2025-05-18T21:18:32Z
body: PRO TIP for review:
In one terminal run: `make dev-elk` (potentially `make clean` before)
In another terminal run: `make wait`

This will wait for all services to be healthy. You can also tweak the target to open the browser directly.

Another PRO TIP:
Run `make get-env SERVICE=elasticsearch` to get the password for the elasticsearch web UI (`ELASTIC_PASSWORD`)
author_association: OWNER
user: login: cubernetes
created_at: 2025-05-18T21:56:26Z
body: Ok, ready to review. Hope it works on your machines @darrenkuro @yas0nia. 
author_association: OWNER
user: login: darrenkuro
created_at: 2025-05-19T09:25:44Z
body: Could you maybe elaborate a little on the design of env.json vs config.env? I thought env.json was for generated stuff, but if it saves regular vars like port, why wouldn't we say move backend port in there (or pretty much everything)? Is it due to accessibility, like what container needs to access what variable and which way is easier? I'm not sure if I see a conceptual difference between the two and seems only to be about what's easier to parse in which context.

Just wondering, feel free to talk about it later if it's too much to type. 
author_association: COLLABORATOR
user: login: darrenkuro
created_at: 2025-05-19T09:39:20Z
body: > @darrenkuro can you confirm that this closes #99 as per your expectations?

Hmm, I am not sure. I was thinking that there is a centralized variable that you can change that will control the ports of ELK services similar to the one in vault, backend, live reload, etc. I don't think the ones in env.json is changeable? Since it's hardcoded in compose, as well as config/script files in ELK. 

It might be an overkill and these ports might be super standard so we could scrape that if it's too much work. (In which case maybe all other internal services just use a hardcoded port as well for consistency?) 

The reason why I wanted make it changeable originally was due to A) consistency with other containers, B) possible conflict that would be block the service for painless startup (manually changing ports in weeds of config files might be too difficult). 
author_association: COLLABORATOR
user: login: cubernetes
created_at: 2025-05-19T12:26:20Z
body: Yes! Good points that we'll def talk about 6pm. Thanks for reviewing!
author_association: OWNER