Repository: PowerGridModel/power-grid-model
Pull request #965: Patch automatic publish PyPI
Author: login: mgovers
Labels: name: bug
description: Something isn't working
Created: 2025-04-23T09:44:13Z | Merged: 2025-04-28T08:36:09Z

Very unfortunate patch to #944 

This seems to be caused by the fact that GitHub explicitly excludes releases created by GitHub Actions workflows from triggering `on: release` events. See also https://github.com/orgs/community/discussions/25281

NOTE: this is probably not a sustainable solution.


Diff:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 8741f4ea71..485447f393 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
+    if: (github.event_name == 'workflow_dispatch') || github.event_name == 'push'
+
+    steps:
+      - name: Download assets from GitHub release
+        uses: robinraju/release-downloader@v1
+        with:
+          repository: ${{ github.repository }}
+          # download the latest release
+          latest: true
+          # don't download pre-releases
+          preRelease: false
+          fileName: "*"
+          # don't download GitHub-generated source tar and zip files
+          tarBall: false
+          zipBall: false
+          # create a directory to store the downloaded assets
+          out-file-path: assets-to-publish
+          # don't extract downloaded files
+          extract: false
+
+      - name: List downloaded assets
+        run: ls -la assets-to-publish
+      
+      - name: Upload assets to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          # To test, use the TestPyPI:
+          # repository-url: https://test.pypi.org/legacy/
+          # You must also create an account and project on TestPyPI,
+          # as well as set the trusted-publisher in the project settings:
+          # https://docs.pypi.org/trusted-publishers/adding-a-publisher/
+          # To publish to the official PyPI repository, just keep
+          # repository-url commented out.
+          packages-dir: assets-to-publish
+          skip-existing: true
+          print-hash: true
+          verbose: true
diff --git a/.github/workflows/publish-pypi.yml b/.github/workflows/publish-pypi.yml
deleted file mode 100644
index 125641ab94..0000000000
--- a/.github/workflows/publish-pypi.yml
+++ /dev/null
@@ -1,55 +0,0 @@
-# SPDX-FileCopyrightText: Contributors to the Power Grid Model project <powergridmodel@lfenergy.org>
-#
-# SPDX-License-Identifier: MPL-2.0
-
-name: Publish to PyPI
-
-on:
-  release:
-    types: [published]
-  workflow_dispatch:
-
-permissions:
-  contents: write
-  id-token: write  # Required for Trusted Publishing
-
-jobs:
-  publish:
-    name: Publish to PyPI
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Download assets from GitHub release
-        uses: robinraju/release-downloader@v1
-        with:
-          repository: ${{ github.repository }}
-          # download the latest release
-          latest: true
-          # don't donwload pre-releases
-          preRelease: false
-          fileName: "*"
-          # don't download GitHub-generated source tar and zip files
-          tarBall: false
-          zipBall: false
-          # create a directory to store the downloaded assets
-          out-file-path: assets-to-publish
-          # don't extract downloaded files
-          extract: false
-
-      - name: List downloaded assets
-        run: ls -la assets-to-publish
-      
-      - name: Upload assets to PyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        with:
-          # To test, use the TestPyPI:
-          # repository-url: https://test.pypi.org/legacy/
-          # You must also create an account and project on TestPyPI,
-          # as well as set the trusted-publisher in the project settings:
-          # https://docs.pypi.org/trusted-publishers/adding-a-publisher/
-          # To publish to the official PyPI repository, just keep
-          # repository-url commented out.
-          packages-dir: assets-to-publish
-          skip-existing: true
-          print-hash: true
-          verbose: true


Reviews:
user: login: figueroa1395
body: So basically, the original failed because `on-release` isn't actually `on-release` if called from an another workflow? Hence making it part of the "main" workflow fixes it? That unless we want to go with the official way to fix this (which we shouldn't) which is to [use personal tokens to access](https://github.com/orgs/community/discussions/25281#discussioncomment-3247262)? Doesn't make any sense to be honest, and worse that it isn't documented beyond threads were people complain... Besides the small suggestion, I am on board with this.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T11:06:21Z
user: login: mgovers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T11:08:59Z
user: login: figueroa1395
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T11:10:35Z
user: login: figueroa1395
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T07:36:03Z
user: login: figueroa1395
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T07:38:58Z
user: login: mgovers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T07:45:17Z
user: login: TonyXiang8787
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T08:13:44Z
user: login: mgovers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T08:17:37Z
user: login: TonyXiang8787
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-28T08:19:27Z

Review comments:
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
path: .github/workflows/ci.yml
user: login: figueroa1395
body: ```suggestion
    needs: ci-passed
```
created_at: 2025-04-23T11:03:50Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
path: .github/workflows/ci.yml
user: login: mgovers
body: i had this officially, but then i realized that the original version depended on ONLY the `Create GitHub Release` job, which is part of the `build-test-release` workflow. Fine with changing, but it would change behavior
created_at: 2025-04-23T11:08:58Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
path: .github/workflows/ci.yml
user: login: figueroa1395
body: I'm on board with this chance since we are already in this workflow.
created_at: 2025-04-23T11:10:35Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
path: .github/workflows/ci.yml
user: login: figueroa1395
body: @mgovers since we don't want to break anything, let's just keep it as it was and scratch this suggestion. Resolving.
created_at: 2025-04-28T07:36:03Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
+    if: (github.event_name == 'workflow_dispatch' && inputs.create_release) || github.event_name == 'push'
path: .github/workflows/ci.yml
user: login: figueroa1395
body: ```suggestion
    if: (github.event_name == 'workflow_dispatch') || github.event_name == 'push'
```
Since we now have `preRelease: false`, this would not publish anything via workflow dispatch. Or on the contrary, it would, but not the pre-release but the actual latest release. So we may as well remove it from the conditional.
created_at: 2025-04-28T07:38:58Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
+    if: (github.event_name == 'workflow_dispatch') || github.event_name == 'push'
+
+    steps:
+      - name: Download assets from GitHub release
+        uses: robinraju/release-downloader@v1
+        with:
+          repository: ${{ github.repository }}
+          # download the latest release
+          latest: true
+          # don't donwload pre-releases
path: .github/workflows/ci.yml
user: login: mgovers
body: ```suggestion
          # don't download pre-releases
```
created_at: 2025-04-28T07:44:58Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
+    if: (github.event_name == 'workflow_dispatch') || github.event_name == 'push'
+
+    steps:
+      - name: Download assets from GitHub release
+        uses: robinraju/release-downloader@v1
+        with:
+          repository: ${{ github.repository }}
+          # download the latest release
+          latest: true
+          # don't download pre-releases
+          preRelease: false
path: .github/workflows/ci.yml
user: login: TonyXiang8787
body: We decided to be able to upload a alpha version right? So this has to be changed back to true?
created_at: 2025-04-28T08:13:44Z
author_association: MEMBER
diff_hunk: @@ -61,3 +61,45 @@ jobs:
 
     steps:
       - run: echo "ci passed"
+
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    needs: build-test-release
+    if: (github.event_name == 'workflow_dispatch') || github.event_name == 'push'
+
+    steps:
+      - name: Download assets from GitHub release
+        uses: robinraju/release-downloader@v1
+        with:
+          repository: ${{ github.repository }}
+          # download the latest release
+          latest: true
+          # don't download pre-releases
+          preRelease: false
path: .github/workflows/ci.yml
user: login: mgovers
body: i'd consider that out of scope of this PR. currently we are unable to upload to PyPI at all (not even manually from `main`). Let's first fix that and then we can do follow-ups
created_at: 2025-04-28T08:17:37Z
author_association: MEMBER

PR comments:
user: login: mgovers
created_at: 2025-04-23T11:13:46Z
body: Added do-not-merge label as input is desired
author_association: MEMBER
user: login: TonyXiang8787
created_at: 2025-04-23T11:51:36Z
body: @mgovers is this the only way? Can we still put the upload-pypi in a separate workflow. And in the CI-passed job of the main workflow, we trigger the other workflow by some means.
author_association: MEMBER
user: login: mgovers
created_at: 2025-04-23T11:55:31Z
body: > @mgovers is this the only way? Can we still put the upload-pypi in a separate workflow. And in the CI-passed job of the main workflow, we trigger the other workflow by some means.

That would indeed be the alternative i mentioned in the offline thread. The way to do that would be to add a `repository_dispatch` trigger in the `publish-pypi.yml` combined with adding something like this to the `build-test-release.yml`

```yml
      - name: Trigger PyPI publish event
        if: (inputs.create_release)
        run: |
          gh api \
            --method POST \
            -H "Accept: application/vnd.github+json" \
            -H "X-GitHub-Api-Version: 2022-11-28" \
            /repos/PowerGridModel/power-grid-model/dispatches \
            -f "event_type=pypi-publish" -F "client_payload[unit]=false" -F "client_payload[integration]=true"
```

However, again, we cannot test this unless we do a 2-stage merge so that the `repository_dispatch` trigger is in `main`
author_association: MEMBER
user: login: TonyXiang8787
created_at: 2025-04-23T11:58:41Z
body: > > @mgovers is this the only way? Can we still put the upload-pypi in a separate workflow. And in the CI-passed job of the main workflow, we trigger the other workflow by some means.
> 
> That would indeed be the alternative i mentioned in the offline thread. The way to do that would be to add a `repository_dispatch` trigger in the `publish-pypi.yml` combined with adding something like this to the `build-test-release.yml`
> 
> ```yaml
>       - name: Trigger PyPI publish event
>         if: (inputs.create_release)
>         run: |
>           gh api \
>             --method POST \
>             -H "Accept: application/vnd.github+json" \
>             -H "X-GitHub-Api-Version: 2022-11-28" \
>             /repos/PowerGridModel/power-grid-model/dispatches \
>             -f "event_type=pypi-publish" -F "client_payload[unit]=false" -F "client_payload[integration]=true"
> ```
> 
> However, again, we cannot test this unless we do a 2-stage merge so that the `repository_dispatch` trigger is in `main`

Do we have an existing action to do that in the market place?
author_association: MEMBER
user: login: figueroa1395
created_at: 2025-04-23T12:01:13Z
body: > @mgovers is this the only way? Can we still put the upload-pypi in a separate workflow. And in the CI-passed job of the main workflow, we trigger the other workflow by some means.

The problem is that the publishing action to PyPI can't be triggered from another workflow. So if we want this publishing workflow to be standalone, we would have to be able to trigger it within the workflow itself with something different than `on: release`. However, maybe we can do something like:
```
name: Publish to PyPI
on:
  workflow_run:
    workflows: ["Wathever"]  # name of the CI finished workflow for instance (we need to add a name to it)
    types:
      - completed
  workflow_dispatch:
  

jobs:
  publish:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    ...
```

would something like this work instead?
author_association: MEMBER
user: login: mgovers
created_at: 2025-04-23T12:02:08Z
body: > > > @mgovers is this the only way? Can we still put the upload-pypi in a separate workflow. And in the CI-passed job of the main workflow, we trigger the other workflow by some means.
> > 
> > 
> > That would indeed be the alternative i mentioned in the offline thread. The way to do that would be to add a `repository_dispatch` trigger in the `publish-pypi.yml` combined with adding something like this to the `build-test-release.yml`
> > ```yaml
> >       - name: Trigger PyPI publish event
> >         if: (inputs.create_release)
> >         run: |
> >           gh api \
> >             --method POST \
> >             -H "Accept: application/vnd.github+json" \
> >             -H "X-GitHub-Api-Version: 2022-11-28" \
> >             /repos/PowerGridModel/power-grid-model/dispatches \
> >             -f "event_type=pypi-publish" -F "client_payload[unit]=false" -F "client_payload[integration]=true"
> > ```
> >   
> > However, again, we cannot test this unless we do a 2-stage merge so that the `repository_dispatch` trigger is in `main`
> 
> Do we have an existing action to do that in the market place?

maybe, i haven't been able to investigate this yet due to other priorities, but i didn't find anything other than this (github official way) in my quick 1min search.

author_association: MEMBER
user: login: TonyXiang8787
created_at: 2025-04-23T12:12:15Z
body: @mgovers @figueroa1395 

I think the solution would be, as similar to other organizations such as `conda-forge`, to create service bot account. We can then create personal access token for the service account and use in the release step of main CI. In that way the release is not created by `GITHUB_TOKEN` but by our own token. The publish PyPI can then be triggerd. This is also suggested in official GitHub Actions documentation:

https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow#triggering-a-workflow-from-a-workflow

It also mentioned to use GitHub App to generate a token.
author_association: MEMBER
user: login: figueroa1395
created_at: 2025-04-23T14:08:45Z
body: > @mgovers @figueroa1395
> 
> I think the solution would be, as similar to other organizations such as `conda-forge`, to create service bot account. We can then create personal access token for the service account and use in the release step of main CI. In that way the release is not created by `GITHUB_TOKEN` but by our own token. The publish PyPI can then be triggerd. This is also suggested in official GitHub Actions documentation:
> 
> https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow#triggering-a-workflow-from-a-workflow
> 
> It also mentioned to use GitHub App to generate a token.

In summary, create an account to be used as a bot, make it a maintainer of the pgm projects, and then use its personal GitHub access token to run the action instead of the workflow token? It doesn't look like it is too hard, but probably needs a bit of investigation and some trial an error.
author_association: MEMBER
user: login: mgovers
created_at: 2025-04-28T07:47:22Z
body: > Added do-not-merge label as input is desired

do-not-merge label removed cfr. offline discussions: skip publishing the last 2 patch releases (no important changes to them) and proceed with the lastest
author_association: MEMBER
user: login: sonarqubecloud[bot]
created_at: 2025-04-28T07:54:35Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=PowerGridModel_power-grid-model&pullRequest=965) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=PowerGridModel_power-grid-model&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=PowerGridModel_power-grid-model&pullRequest=965&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=PowerGridModel_power-grid-model&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=PowerGridModel_power-grid-model&pullRequest=965&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=PowerGridModel_power-grid-model&pullRequest=965&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=PowerGridModel_power-grid-model&pullRequest=965)


author_association: NONE