Repository: wri/project-zeno
Pull request #161: add workflow to build and push image for db container
Author: login: batpad
Created: 2025-05-06T05:40:55Z | Merged: 2025-05-07T06:57:01Z

Refs https://github.com/wri/project-zeno-deploy/issues/16

To deploy the latest changes, we need to publish the new `db` container to pull and run it in the cluster.

This PR adds a Github workflow to publish the db image to the same ECR we use for the main image.

I'm not super happy with this as this will mean two different commit hashes that we need to keep track of between `project-zeno` and `project-zeno-deploy`. I'll work with @sunu to outline a plan to do this a bit better, but I think this is the most straightforward way to deploy these changes right now and unblock / move ahead.

cc @sunu @yellowcap 



Diff:
diff --git a/.github/workflows/docker-build-db.yml b/.github/workflows/docker-build-db.yml
new file mode 100644
index 00000000..1306cd6d
--- /dev/null
+++ b/.github/workflows/docker-build-db.yml
@@ -0,0 +1,50 @@
+name: Build and Publish DB Docker Image
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'db/**'
+
+env:
+  REGISTRY: public.ecr.aws/b7u8b0a6
+  IMAGE_NAME: project-zeno/zeno-db
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+
+      - name: Create ECR repository if not exists
+        run: |
+          aws ecr-public create-repository \
+            --repository-name ${{ env.IMAGE_NAME }} \
+            --region us-east-1 || true
+
+      - name: Login to Amazon ECR Public
+        id: login-ecr-public
+        uses: aws-actions/amazon-ecr-login@v2
+        with:
+          registry-type: public
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: ./db
+          push: true
+          tags: |
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
diff --git a/db/Dockerfile b/db/Dockerfile
index 573e43e8..2d874273 100644
--- a/db/Dockerfile
+++ b/db/Dockerfile
@@ -2,14 +2,18 @@ FROM python:3.11-slim
 
 WORKDIR /app/db
 
+# Install PostgreSQL client and Alembic
+RUN apt-get update && \
+    apt-get install -y postgresql-client
+
 # Install Alembic
 RUN pip install --no-cache-dir alembic psycopg2-binary pydantic
 
 # Copy migration scripts and models
 COPY . /app/db
 
-RUN chmod +x /app/db/init-multiple-dbs.sh
-
+RUN chmod +x /app/db/init-multiple-dbs.sh && \
+    chmod +x /app/db/entrypoint.sh
 
 # Default command (can be overridden)
-CMD ["alembic", "upgrade", "head"]
+CMD ["./entrypoint.sh"]
diff --git a/db/entrypoint.sh b/db/entrypoint.sh
new file mode 100644
index 00000000..8dcb966e
--- /dev/null
+++ b/db/entrypoint.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+export DATABASE_URL="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${APP_DB}"
+./init-multiple-dbs.sh && \
+alembic upgrade head
diff --git a/db/init-multiple-dbs.sh b/db/init-multiple-dbs.sh
index e2ea5898..7825e49e 100755
--- a/db/init-multiple-dbs.sh
+++ b/db/init-multiple-dbs.sh
@@ -1,8 +1,19 @@
 #!/bin/bash
 set -e
 
+# Set password for psql
+export PGPASSWORD="${POSTGRES_PASSWORD}"
+
 # Create additional databases if they do not exist
-psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
-    CREATE DATABASE "${APP_DB:-zeno-data}";
-    CREATE DATABASE "${LANGFUSE_DB:-langfuse}";
+psql -h "${POSTGRES_HOST}" -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
+    DO \$\$
+    BEGIN
+        IF NOT EXISTS (SELECT FROM pg_database WHERE datname = '${APP_DB:-zeno-data}') THEN
+            CREATE DATABASE "${APP_DB:-zeno-data}";
+        END IF;
+        IF NOT EXISTS (SELECT FROM pg_database WHERE datname = '${LANGFUSE_DB:-langfuse}') THEN
+            CREATE DATABASE "${LANGFUSE_DB:-langfuse}";
+        END IF;
+    END
+    \$\$;
 EOSQL
diff --git a/docker-compose.yaml b/docker-compose.yaml
index ffe00947..846c8b78 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -77,6 +77,7 @@ services:
     environment:
       - POSTGRES_USER=postgres
       - POSTGRES_PASSWORD=postgres
+      - POSTGRES_HOST=db
       - APP_DB=zeno-data
       - LANGFUSE_DB=langfuse
     ports:
@@ -92,7 +93,10 @@ services:
       db:
         condition: service_healthy
     environment:
-      - DATABASE_URL=postgresql+psycopg2://postgres:postgres@db:5432/zeno-data
+      - POSTGRES_USER=postgres
+      - POSTGRES_PASSWORD=postgres
+      - POSTGRES_HOST=db
+      - APP_DB=zeno-data
 
 volumes:
   database_data:


Reviews:
user: login: sunu
body: Added some minor comments, but overall, this looks good to me!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-07T06:12:29Z
user: login: batpad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-07T06:21:01Z

Review comments:
diff_hunk: @@ -0,0 +1,4 @@
+#!/bin/bash
+export DATABASE_URL="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${APP_DB}"
+./init-multiple-dbs.sh && \
path: db/entrypoint.sh
user: login: sunu
body: Should we merge the init-multiple-dbs script into the entrypoint script?
created_at: 2025-05-07T06:07:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+name: Build and Publish DB Docker Image
+
+on:
+  push:
+    branches:
+      - main
+      - docker-build-publish
+      - publish-docker-db
path: .github/workflows/docker-build-db.yml
user: login: sunu
body: Just commenting as a reminder that we might want to remove the extra branches before merging
created_at: 2025-05-07T06:07:54Z
author_association: COLLABORATOR
path: .github/workflows/docker-build-db.yml
user: login: sunu
body: Maybe we could add this as a separate step in the current workflow, but I think it's fine for now since we'll be consolidating everything into a single image soon anyway.
created_at: 2025-05-07T06:11:44Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,4 @@
+#!/bin/bash
+export DATABASE_URL="postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${APP_DB}"
+./init-multiple-dbs.sh && \
path: db/entrypoint.sh
user: login: batpad
body: hmm am on the fence - i think it's fine as a separate script?
created_at: 2025-05-07T06:21:01Z
author_association: COLLABORATOR

PR comments:
user: login: batpad
created_at: 2025-05-06T05:48:04Z
body: Also wrapped around the `init-multiple-dbs.sh` script to ensure that's run before running migrations so the db gets created the first time this is run. cc @leothomas 
author_association: COLLABORATOR
user: login: batpad
created_at: 2025-05-06T07:01:31Z
body: @sunu I made some more changes to:
 - Create the ECR repository if it does not exist in the Action
 - Consolidating the env vars passed to the db container to not have to pass in DATABASE_URL and things like POSTGRES_HOST separately, so now it just needs POSTGRES_HOST, POSTGRES_USER, POSTGRES_PASSWORD as separate env vars and the DATABASE_URL is constructed within the script

I'll test with the image built from this branch.. If that works, let me know if you have any suggestions to improve or if I can merge this.
author_association: COLLABORATOR
user: login: batpad
created_at: 2025-05-07T06:23:10Z
body: @sunu I commented out the special-cased branches. Also commented back in the line to only build if changes have been made to the `db/` folder to avoid too many unnecessary builds - not sure if you feel that makes sense to have or just let it build with all pushes to main?
author_association: COLLABORATOR
user: login: sunu
created_at: 2025-05-07T06:54:48Z
body: Yeah, I think it makes sense to trigger the build only when there are changes in the `db/` directory.
author_association: COLLABORATOR