Repository: mrc-ide/packit-deploy
Pull request #13: Dev docs
Author: login: richfitz
Created: 2025-05-13T10:28:39Z | Merged: 2025-05-14T11:03:23Z

This PR adds some docs to describe running things up locally for development and in doing so fixes a few things:

* Drop ancient python versions that are not present on github runners, breaking the build
* Removes the need for the `basicauth` configuration to touch the vault, which makes it nicer to work with locally
* Updates the proxy configuration to look for the api at `/api`; this is consistent with the *relative* path of the api to the packit root elsewhere
* Fixes the ruff invocations and configuration
* Prints out the username and password that will allow login

To test locally, change `tag: main` to `tag: dev-docs` in the `proxy` section of the configuration.  That tests pass without this shows a missing e2e test that would require interaction with playright or similar


Diff:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
new file mode 100644
index 0000000..9c68521
--- /dev/null
+++ b/.github/workflows/release.yml
@@ -0,0 +1,53 @@
+on:
+  push:
+    branches: [ master ]
+    tags: [ v* ]
+  pull_request:
+    branches: [ master ]
+
+name: Publish package
+jobs:
+  build:
+    name: Build distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.x"
+      - name: Install pypa/build
+        run: |
+          python -m pip install build
+      - name: Build a binary wheel and a source tarball
+        run: |
+          python -m build
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          if-no-files-found: error
+          name: python-artifacts
+          path: dist
+
+  # This assumes a PyPI Trusted Publisher has been configured for the `hipercow` package.
+  # See https://docs.pypi.org/trusted-publishers/ for more details.
+  publish-to-pypi:
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
+    name: Publish Python distribution to PyPI
+    needs:
+      - build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/hipercow
+    permissions:
+      # This permission is needed for the workflow to authenticate against PyPI
+      id-token: write
+    steps:
+    - name: Download the artifacts
+      uses: actions/download-artifact@v4
+      with:
+        name: python-artifacts
+        path: dist/
+    - name: Publish distribution to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 9b73057..1937449 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -18,7 +18,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
 
     steps:
       - name: Set up QEMU
diff --git a/README.md b/README.md
index e851683..a40208b 100644
--- a/README.md
+++ b/README.md
@@ -59,45 +59,27 @@ On CI, use `hatch run cov-ci` to generate an xml report.
 hatch build
 ```
 
-## Install from local sources
+## Publishing to PyPI
 
-1. `hatch build`
-2. `pip install dist/packit_deploy-{version}.tar.gz`
+Automatically publish to [PyPI](https://pypi.org/project/packit-deploy).  Assuming a version number `0.1.2`:
 
-## Publish to PyPi
+* Create a [release on github](https://github.com/reside-ic/packit-deploy/releases/new)
+* Choose a tag -> Create a new tag: `v0.1.2`
+* Use this version as the description
+* Optionally describe the release
+* Click "Publish release"
+* This triggers the release workflow and the package will be available on PyPI in a few minutes
 
-Ensure you have built a new version of the package:
-1. `hatch clean`
-2. `hatch build`
+Settings are configured [here on PyPI](https://pypi.org/manage/project/packit-deploy/settings/publishing)
 
-Then publish to the test server:
-
-```console
-hatch publish -r test
-```
-
-You will be prompted to enter your [test.pypi.org](https://test.pypi.org/legacy/) username and password.
-To test the installation, first run Python in a container:
-
-```
-docker run --rm -it --entrypoint bash python
-```
-
-Then:
-
-```
-pip install --index-url https://test.pypi.org/simple packit-deploy --extra-index-url https://pypi.org/simple
-```
-
-Now you should be able to run `packit` from the command line and see the usage instructions.
+## Install from local sources
 
-If it is working, you can publish to the real PyPi:
+You should not need to do this very often, but if you really want to:
 
-```console
-hatch publish
-```
+1. `hatch build`
+2. `pip install dist/packit_deploy-{version}.tar.gz`
 
-## Config
+## Example configurations
 
 The following example configurations are included under `/config`:
 
@@ -108,10 +90,30 @@ The following example configurations are included under `/config`:
 - `nodemo`: does not include the demo data
 - `noproxy`: does not include proxy container
 
-These configurations should all be runnable for local testing, except for `complete`, which includes non-existent vault secrets.
-You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
+## Running locally
+
+You can bring up most of the configurations above for local testing (except for `complete` which includes non-existant vault secrets and `noproxy` which will not actually expose anything to interact with).  You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+For example, to bring up the `basicauth` configuration, you can run:
+
+```console
+hatch env run -- packit start --pull config/basicauth
+./scripts/create-super-user
+```
+
+The `--` allows `--pull` to make it through to the deploy (and not be swallowed by `hatch`).  Alternatively you can run `packit start --pull config/basicauth` after running `hatch shell`.  The `create-super-user` script sets up a user that you can log in with and prints the login details to stdout.  After this, packit will be running at `https://localhost` though with a self-signed certificate so you will need to tell your browser that it's ok to connect.
+
+To bring things down, run
+
+```console
+hatch env run -- packit stop --kill config/basicauth
+```
+
+If you need to see what lurks in the database, connect with
+
+```console
+docker exec -it packit-packit-db psql -U packituser -d packit
+```
 
-If developing in **basic auth** mode, a super admin user can be created by running `./scripts/create-super-user.sh` after the app is running.
+If you have anything else running on port 80 or 443, nothing will work as expected; either stop that service or change the proxy port in the configuration that you are using.
diff --git a/config/basicauth/packit.yml b/config/basicauth/packit.yml
index 8b6b21d..060b1c7 100644
--- a/config/basicauth/packit.yml
+++ b/config/basicauth/packit.yml
@@ -38,7 +38,6 @@ outpack:
   migrate:
     name: outpack.orderly
     tag: main
-# TODO: change to main basic auth stuff is merged
 packit:
   api:
     name: packit-api
@@ -56,9 +55,7 @@ packit:
     auth_method: basic
     expiry_days: 1
     jwt:
-      # Secret used to generate JWT tokens - this can be any string, the secret at this key in the vault is a random
-      # 32 char string, and is probably fine to re-use
-      secret: VAULT:secret/packit/githubauth/auth/jwt:secret
+      secret: "0b4g4f8z4mdsrhoxfde2mam8f00vmt0f"
 
 ## If running a proxy directly, fill this section in.  Otherwise you
 ## are responsible for proxying the application out of the docker
@@ -71,19 +68,3 @@ proxy:
   image:
     name: packit-proxy
     tag: main
-
-vault:
-  ## Address of the vault server.  This should be a string if it is
-  ## present.
-  addr: https://vault.dide.ic.ac.uk:8200
-  auth:
-    ## Authentication type - must be either "token" or the name of a
-    ## supported authentication method.  These seem to be poorly
-    ## documented in the hvac, but include "github" for github
-    ## authentication.
-    ##
-    ## On a vault client object, see auth.implemented_class_names for
-    ## a list, which is currently
-    ##
-    ##     azure, github, gcp, kubernetes, ldap, mfa, okta
-    method: github
diff --git a/proxy/nginx.conf b/proxy/nginx.conf
index 5752717..718e780 100644
--- a/proxy/nginx.conf
+++ b/proxy/nginx.conf
@@ -68,7 +68,7 @@ http {
 
         root /usr/share/nginx/html;
 
-        location /packit/api/ {
+        location /api/ {
             proxy_pass http://${PACKIT_API}/;
         }
 
diff --git a/pyproject.toml b/pyproject.toml
index c46e7f1..97bf91f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -7,7 +7,7 @@ name = "packit-deploy"
 dynamic = ["version"]
 description = ''
 readme = "README.md"
-requires-python = ">=3.7"
+requires-python = ">=3.9"
 license = "MIT"
 keywords = []
 authors = [
@@ -16,11 +16,11 @@ authors = [
 classifiers = [
     "Development Status :: 4 - Beta",
     "Programming Language :: Python",
-    "Programming Language :: Python :: 3.7",
-    "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: Implementation :: CPython",
     "Programming Language :: Python :: Implementation :: PyPy",
 ]
@@ -71,7 +71,7 @@ cov-ci = [
 ]
 
 [[tool.hatch.envs.all.matrix]]
-python = ["3.7", "3.8", "3.9", "3.10", "3.11"]
+python = ["3.9", "3.10", "3.11", "3.12", "3.13"]
 
 [tool.hatch.envs.lint]
 detached = true
@@ -83,12 +83,14 @@ dependencies = [
 [tool.hatch.envs.lint.scripts]
 typing = "mypy --install-types --non-interactive {args:src/packit_deploy tests}"
 style = [
-    "ruff {args:.}",
+    "ruff check {args:.}",
+    "ruff format --check {args:.}",
     "black --check --diff {args:.}",
 ]
 fmt = [
     "black {args:.}",
-    "ruff --fix {args:.}",
+    "ruff check --fix {args:.}",
+    "ruff format {args:.}",
     "style",
 ]
 all = [
@@ -97,13 +99,13 @@ all = [
 ]
 
 [tool.black]
-target-version = ["py37"]
 line-length = 120
 skip-string-normalization = true
 
 [tool.ruff]
-target-version = "py37"
 line-length = 120
+
+[tool.ruff.lint]
 select = [
     "A",
     "ARG",
@@ -148,13 +150,13 @@ unfixable = [
     "F401",
 ]
 
-[tool.ruff.isort]
+[tool.ruff.lint.isort]
 known-first-party = ["packit_deploy"]
 
-[tool.ruff.flake8-tidy-imports]
+[tool.ruff.lint.flake8-tidy-imports]
 ban-relative-imports = "all"
 
-[tool.ruff.per-file-ignores]
+[tool.ruff.lint.per-file-ignores]
 # Tests can use magic values, assertions, and relative imports
 "tests/**/*" = ["PLR2004", "S101", "TID252"]
 
diff --git a/scripts/create-super-user b/scripts/create-super-user
index a83a72b..a2ab30e 100755
--- a/scripts/create-super-user
+++ b/scripts/create-super-user
@@ -9,3 +9,7 @@ ADMIN_PASSWORD_ENCODED='$2y$10$snpZ8bgdkh2hy8lDtyHF7ejD5.K1vsMqaFteCkmBhdBQj3JTl
 ADMIN_UUID='2754daf8-fea0-4bf4-af60-810764f24d71'
 
 docker exec packit-packit-db create-super-user --email "$ADMIN_EMAIL" --password "$ADMIN_PASSWORD_ENCODED" --uuid "$ADMIN_UUID"
+
+echo "Log in with":
+echo "  username: $ADMIN_EMAIL"
+echo "  password: password"


Reviews:
user: login: EmmaLRussell
body: Works for me!

state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T15:26:44Z
user: login: richfitz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T16:18:19Z
user: login: david-mears-2
body: After doing `packit start config/basicauth`, I still saw the mysterious 405 status code coming back from the packit/api/ endpoint -- until I did `packit start config/basicauth --pull` instead.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T16:39:51Z
user: login: david-mears-2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T16:41:28Z
user: login: richfitz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T16:51:13Z
user: login: richfitz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T16:51:53Z
user: login: david-mears-2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T10:50:28Z
user: login: david-mears-2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T10:59:15Z
user: login: david-mears-2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T11:01:41Z
user: login: david-mears-2
body: :tada: 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T11:03:01Z

Review comments:
diff_hunk: @@ -9,3 +9,7 @@ ADMIN_PASSWORD_ENCODED='$2y$10$snpZ8bgdkh2hy8lDtyHF7ejD5.K1vsMqaFteCkmBhdBQj3JTl
 ADMIN_UUID='2754daf8-fea0-4bf4-af60-810764f24d71'
 
 docker exec packit-packit-db create-super-user --email "$ADMIN_EMAIL" --password "$ADMIN_PASSWORD_ENCODED" --uuid "$ADMIN_UUID"
+
+echo "Log in with":
+echo "  username: resideUser@resideAdmin.ic.ac.uk"
path: scripts/create-super-user
user: login: EmmaLRussell
body: Could use $ADMIN_EMAIL for this I guess!
created_at: 2025-05-13T15:15:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -18,7 +18,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
path: .github/workflows/test.yml
user: login: david-mears-2
body: ```suggestion
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
```
created_at: 2025-05-13T16:16:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -9,3 +9,7 @@ ADMIN_PASSWORD_ENCODED='$2y$10$snpZ8bgdkh2hy8lDtyHF7ejD5.K1vsMqaFteCkmBhdBQj3JTl
 ADMIN_UUID='2754daf8-fea0-4bf4-af60-810764f24d71'
 
 docker exec packit-packit-db create-super-user --email "$ADMIN_EMAIL" --password "$ADMIN_PASSWORD_ENCODED" --uuid "$ADMIN_UUID"
+
+echo "Log in with":
+echo "  username: resideUser@resideAdmin.ic.ac.uk"
path: scripts/create-super-user
user: login: richfitz
body: Good idea. Though looking at this I am surprised that encoded password works properly - should it not use the new secret that I added in the config?
created_at: 2025-05-13T16:18:19Z
author_association: MEMBER
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
path: README.md
user: login: david-mears-2
body: I think you should retain the third `#` if you want "For example" to relate to the previous paragraph.

You might want to add a callback like "First, install from local sources as above" linking to 'Install from local sources'. Alternatively, put this section as a sub-section of that.
created_at: 2025-05-13T16:20:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -71,7 +71,7 @@ cov-ci = [
 ]
path: pyproject.toml
user: login: david-mears-2
body: Line 10 in this file should go from 
requires-python = ">=3.7"
to
requires-python = ">=3.9"
created_at: 2025-05-13T16:24:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -71,7 +71,7 @@ cov-ci = [
 ]
path: pyproject.toml
user: login: david-mears-2
body: Also see 'classifiers' line 16, which lists the old pythons and misses out python 3.12
created_at: 2025-05-13T16:24:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
 
-If developing in **basic auth** mode, a super admin user can be created by running `./scripts/create-super-user.sh` after the app is running.
+For example, to bring up the `basicauth` configuration, you can run:
+
+```console
+hatch env run -- packit start --pull config/basicauth
+./scripts/create-super-user
+```
path: README.md
user: login: david-mears-2
body: It happens on the other branches too, but I reliably get competition for use of my proxy ports 80 and 443 (no idea what's using them), so I always have to update those in the config to be some unused ports.

If that's a common problem it could be worth mentioning in the README.
created_at: 2025-05-13T16:34:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -18,7 +18,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
path: .github/workflows/test.yml
user: login: david-mears-2
body: Also - 3.8 is end-of-life, so I removed it on my branch
created_at: 2025-05-13T16:41:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
path: README.md
user: login: richfitz
body: oh I should probably just remove the section on local installation tbh
created_at: 2025-05-13T16:51:13Z
author_association: MEMBER
diff_hunk: @@ -71,7 +71,7 @@ cov-ci = [
 ]
path: pyproject.toml
user: login: richfitz
body: this whole thing is just so awkward - I have no idea why the package skeletons don't have a single source of truth that does not require listing out individual versions rather than ranges
created_at: 2025-05-13T16:51:53Z
author_association: MEMBER
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
path: README.md
user: login: david-mears-2
body: I disagree - I don't know how to run this in development without installing from local sources, and I don't know how to install from local sources without this part of the README. I see you've kept it in though.
created_at: 2025-05-14T10:50:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
path: README.md
user: login: david-mears-2
body: OK, I've now learnt that installing from local sources is unnecessary. So I want to update the README to make the best dev workflow obvious.
created_at: 2025-05-14T10:59:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,25 @@ These configurations should all be runnable for local testing, except for `compl
 You will need access to the vault to run the `githubauth` configuration, which requires secrets for the github oauth2 client app
 details.
 
-### Notes
+## Running locally
path: README.md
user: login: david-mears-2
body: Well, actually, I think it's fine now, it was just such a sparse README before that I invented my own way of doing things, but now there is correct guidance for the dev workflow.
created_at: 2025-05-14T11:01:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-05-13T14:34:22Z
body: ## [Codecov](https://app.codecov.io/gh/mrc-ide/packit-deploy/pull/13?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 91.93%. Comparing base [(`31c6bde`)](https://app.codecov.io/gh/mrc-ide/packit-deploy/commit/31c6bdec8cdc094378ddc4d617cb9cb4398f2095?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide) to head [(`5782103`)](https://app.codecov.io/gh/mrc-ide/packit-deploy/commit/5782103bc1063330dd85b0370e108f691c2863e0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide).
> Report is 3 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main      #13   +/-   ##
=======================================
  Coverage   91.93%   91.93%           
=======================================
  Files           4        4           
  Lines         248      248           
  Branches       31       29    -2     
=======================================
  Hits          228      228           
  Misses         12       12           
  Partials        8        8           
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/mrc-ide/packit-deploy/pull/13?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: EmmaLRussell
created_at: 2025-05-13T15:11:09Z
body: > That tests pass without this shows a missing e2e test that would require interaction with playright or similar

Added a ticket for this: https://mrc-ide.myjetbrains.com/youtrack/issue/mrc-6471/e2e-test-in-packit-deploy
author_association: CONTRIBUTOR
user: login: richfitz
created_at: 2025-05-14T09:21:58Z
body: > until I did packit start config/basicauth --pull instead

This is expected; we have merged a fix in packit which requires you to pull (https://github.com/mrc-ide/packit/pull/204)
author_association: MEMBER