Repository: datahub-project/datahub
Pull request #13500: ci(workflow): postgres consolidation & release unit tests
Author: login: david-leifker
Labels: name: devops
description: PR or Issue related to DataHub backend & deployment
name: publish-docker
name: pending-submitter-merge
Created: 2025-05-12T20:57:44Z | Merged: 2025-05-13T18:24:05Z

* consolidate postgres-setup into unified workflow
* run all unit tests on release

<!--

Thank you for contributing to DataHub!

Before you submit your PR, please go through the checklist below:

- [ ] The PR conforms to DataHub's [Contributing Guideline](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) (particularly [Commit Message Format](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md#commit-message-format))
- [ ] Links to related issues (if applicable)
- [ ] Tests for the changes have been added/updated (if applicable)
- [ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
- [ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in [Updating DataHub](https://github.com/datahub-project/datahub/blob/master/docs/how/updating-datahub.md)

-->



Diff:
diff --git a/.github/workflows/docker-postgres-setup.yml b/.github/workflows/docker-postgres-setup.yml
deleted file mode 100644
index c028bfb55d48d5..00000000000000
--- a/.github/workflows/docker-postgres-setup.yml
+++ /dev/null
@@ -1,61 +0,0 @@
-name: postgres-setup docker
-on:
-  push:
-    branches:
-      - master
-    paths:
-      - "docker/postgres-setup/**"
-      - ".github/workflows/docker-postgres-setup.yml"
-  pull_request:
-    branches:
-      - "**"
-    paths:
-      - "docker/postgres-setup/**"
-      - ".github/workflows/docker-postgres-setup.yml"
-  release:
-    types: [published]
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  setup:
-    runs-on: ubuntu-latest
-    outputs:
-      tag: ${{ steps.tag.outputs.tag }}
-      publish: ${{ steps.publish.outputs.publish }}
-    steps:
-      - name: Checkout
-        uses: acryldata/sane-checkout-action@v3
-      - name: Compute Tag
-        id: tag
-        run: |
-          source .github/scripts/docker_helpers.sh
-          echo "tag=$(get_tag)" >> $GITHUB_OUTPUT
-      - name: Check whether publishing enabled
-        id: publish
-        env:
-          ENABLE_PUBLISH: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
-        run: |
-          echo "Enable publish: ${{ env.ENABLE_PUBLISH != '' }}"
-          echo "publish=${{ env.ENABLE_PUBLISH != '' }}" >> $GITHUB_OUTPUT
-  push_to_registries:
-    name: Build and Push Docker Image to Docker Hub
-    runs-on: ubuntu-latest
-    needs: setup
-    steps:
-      - name: Check out the repo
-        uses: acryldata/sane-checkout-action@v3
-      - name: Build and push
-        uses: ./.github/actions/docker-custom-build-and-push
-        with:
-          images: |
-            acryldata/datahub-postgres-setup
-          image_tag: ${{ needs.setup.outputs.tag }}
-          username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
-          password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
-          publish: ${{ needs.setup.outputs.publish == 'true' }}
-          context: .
-          file: ./docker/postgres-setup/Dockerfile
-          platforms: linux/amd64,linux/arm64
diff --git a/.github/workflows/docker-unified.yml b/.github/workflows/docker-unified.yml
index dcfd05b7f67e01..33f3660113fa92 100644
--- a/.github/workflows/docker-unified.yml
+++ b/.github/workflows/docker-unified.yml
@@ -48,11 +48,11 @@ jobs:
       python_release_version: ${{ steps.tag.outputs.python_release_version }}
       branch_name: ${{ steps.tag.outputs.branch_name }}
       repository_name: ${{ steps.tag.outputs.repository_name }}
-      frontend_change: ${{ steps.ci-optimize.outputs.frontend-change == 'true' }}
-      actions_change: ${{ steps.ci-optimize.outputs.actions-change == 'true' }}
-      ingestion_change: ${{ steps.ci-optimize.outputs.ingestion-change == 'true' }}
+      frontend_change: ${{ steps.ci-optimize.outputs.frontend-change == 'true' || github.event_name == 'release' }}
+      actions_change: ${{ steps.ci-optimize.outputs.actions-change == 'true' || github.event_name == 'release' }}
+      ingestion_change: ${{ steps.ci-optimize.outputs.ingestion-change == 'true' || github.event_name == 'release' }}
       ingestion_base_change: ${{ steps.ci-optimize.outputs.ingestion-base-change == 'true' }}
-      backend_change: ${{ steps.ci-optimize.outputs.backend-change == 'true' }}
+      backend_change: ${{ steps.ci-optimize.outputs.backend-change == 'true' || github.event_name == 'release' }}
       frontend_only: ${{ steps.ci-optimize.outputs.frontend-only == 'true' }}
       ingestion_only: ${{ steps.ci-optimize.outputs.ingestion-only == 'true' }}
       backend_only: ${{ steps.ci-optimize.outputs.backend-only == 'true' }}
diff --git a/.github/workflows/metadata-io.yml b/.github/workflows/metadata-io.yml
index 6514b9e7226b60..02947949238a20 100644
--- a/.github/workflows/metadata-io.yml
+++ b/.github/workflows/metadata-io.yml
@@ -30,9 +30,9 @@ jobs:
   setup:
     runs-on: ubuntu-latest
     outputs:
-      frontend_change: ${{ steps.ci-optimize.outputs.frontend-change == 'true' }}
-      ingestion_change: ${{ steps.ci-optimize.outputs.ingestion-change == 'true' }}
-      backend_change: ${{ steps.ci-optimize.outputs.backend-change == 'true' }}
+      frontend_change: ${{ steps.ci-optimize.outputs.frontend-change == 'true' || github.event_name == 'release' }}
+      ingestion_change: ${{ steps.ci-optimize.outputs.ingestion-change == 'true' || github.event_name == 'release' }}
+      backend_change: ${{ steps.ci-optimize.outputs.backend-change == 'true' || github.event_name == 'release' }}
       docker_change: ${{ steps.ci-optimize.outputs.docker-change == 'true' }}
       frontend_only: ${{ steps.ci-optimize.outputs.frontend-only == 'true' }}
       ingestion_only: ${{ steps.ci-optimize.outputs.ingestion-only == 'true' }}
diff --git a/docker/build.gradle b/docker/build.gradle
index 2683d487c0fb47..ca2c437ac98d4c 100644
--- a/docker/build.gradle
+++ b/docker/build.gradle
@@ -102,7 +102,8 @@ ext {
                         ':metadata-jobs:mce-consumer-job',
                         ':metadata-jobs:mae-consumer-job',
                         ':datahub-actions',
-                        ':docker:datahub-ingestion'
+                        ':docker:datahub-ingestion',
+                        ':docker:postgres-setup'
                     ]
             ]
     ]


Reviews:
user: login: github-actions[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T20:58:47Z
user: login: github-actions[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T20:58:48Z
user: login: github-actions[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T20:58:49Z
user: login: github-actions[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T20:58:50Z
user: login: github-actions[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T20:58:50Z
user: login: RyanHolstien
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-12T21:10:44Z

Review comments:
path: .github/workflows/docker-unified.yml
user: login: github-actions[bot]
body: 🚫 **[actionlint]** <sub>reported by [reviewdog](https://github.com/reviewdog/reviewdog) :dog:</sub><br>property "run-publish-images" is not defined in object type {ci-optimize: {conclusion: string; outcome: string; outputs: {actions-change: string; backend-change: string; backend-only: string; docker-change: string; elasticsearch-setup-change: string; frontend-change: string; frontend-only: string; ingestion-base-change: string; ingestion-change: string; ingestion-only: string; kafka-setup-change: string; mysql-setup-change: string; postgres-setup-change: string; smoke-test-change: string}}; docker-login: {conclusion: string; outcome: string; outputs: {string => string}}; pr-publish: {conclusion: string; outcome: string; outputs: {string => string}}; publish: {conclusion: string; outcome: string; outputs: {string => string}}; set-runner: {conclusion: string; outcome: string; outputs: {string => string}}; tag: {conclusion: string; outcome: string; outputs: {string => string}}} [expression]

https://github.com/datahub-project/datahub/blob/ac2b1e00d604ffadbf04d80d8800d71e03c7ecaf/.github/workflows/docker-unified.yml#L67
<!-- __reviewdog__:ChBlYmNiYWVkYWEzMjMwODAyEgphY3Rpb25saW50 -->

created_at: 2025-05-12T20:58:47Z
author_association: CONTRIBUTOR
path: .github/workflows/docker-unified.yml
user: login: github-actions[bot]
body: 🚫 **[actionlint]** <sub>reported by [reviewdog](https://github.com/reviewdog/reviewdog) :dog:</sub><br>shellcheck reported issue in this script: SC2129:style:2:3: Consider using { cmd1; cmd2; } >> file instead of individual redirects [shellcheck]

https://github.com/datahub-project/datahub/blob/ac2b1e00d604ffadbf04d80d8800d71e03c7ecaf/.github/workflows/docker-unified.yml#L134
<!-- __reviewdog__:ChAxYzViZTVlNTAzZmIxNTA1EgphY3Rpb25saW50 -->

created_at: 2025-05-12T20:58:48Z
author_association: CONTRIBUTOR
path: .github/workflows/docker-unified.yml
user: login: github-actions[bot]
body: 🚫 **[actionlint]** <sub>reported by [reviewdog](https://github.com/reviewdog/reviewdog) :dog:</sub><br>shellcheck reported issue in this script: SC2129:style:7:3: Consider using { cmd1; cmd2; } >> file instead of individual redirects [shellcheck]

https://github.com/datahub-project/datahub/blob/ac2b1e00d604ffadbf04d80d8800d71e03c7ecaf/.github/workflows/docker-unified.yml#L134
<!-- __reviewdog__:ChAxOTJiZmQwOTFjZTFlZGQzEgphY3Rpb25saW50 -->

created_at: 2025-05-12T20:58:49Z
author_association: CONTRIBUTOR
path: .github/workflows/docker-unified.yml
user: login: github-actions[bot]
body: 📝 **[actionlint]** <sub>reported by [reviewdog](https://github.com/reviewdog/reviewdog) :dog:</sub><br>shellcheck reported issue in this script: SC2086:info:5:115: Double quote to prevent globbing and word splitting [shellcheck]

https://github.com/datahub-project/datahub/blob/ac2b1e00d604ffadbf04d80d8800d71e03c7ecaf/.github/workflows/docker-unified.yml#L243
<!-- __reviewdog__:ChBhODNiMGFhYzg1N2FjMjE4EgphY3Rpb25saW50 -->

created_at: 2025-05-12T20:58:50Z
author_association: CONTRIBUTOR
path: .github/workflows/docker-unified.yml
user: login: github-actions[bot]
body: 📝 **[actionlint]** <sub>reported by [reviewdog](https://github.com/reviewdog/reviewdog) :dog:</sub><br>shellcheck reported issue in this script: SC2086:info:3:117: Double quote to prevent globbing and word splitting [shellcheck]

https://github.com/datahub-project/datahub/blob/ac2b1e00d604ffadbf04d80d8800d71e03c7ecaf/.github/workflows/docker-unified.yml#L282
<!-- __reviewdog__:ChAxMjQxNjA2ZWFhNjViZGVkEgphY3Rpb25saW50 -->

created_at: 2025-05-12T20:58:50Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-05-12T21:04:22Z
body: ## [Codecov](https://app.codecov.io/gh/datahub-project/datahub/pull/13500?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datahub-project) Report
All modified and coverable lines are covered by tests :white_check_mark:

:white_check_mark: All tests successful. No failed tests found.


:loudspeaker: Thoughts on this report? [Let us know!](https://about.codecov.io/pull-request-comment-report/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datahub-project)

author_association: NONE
user: login: github-advanced-security[bot]
created_at: 2025-05-13T17:36:20Z
body: This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on [this overview](/datahub-project/datahub/security/code-scanning?query=pr%3A13500+is%3Aopen). Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out [the documentation](https://docs.github.com/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning). 
author_association: NONE
user: login: david-leifker
created_at: 2025-05-13T18:24:01Z
body: Publish worked -> https://hub.docker.com/layers/acryldata/datahub-postgres-setup/pr13500/images/sha256-f3e6d9207d2818a7933566a2f96b2278fb3c066b7edfd16c41a874683d5bf72d
author_association: COLLABORATOR