Repository: canonical/operator (language: python)

Issue:
Run 'tox -e smoke' in CI
We currently don't run the smoke tests (which at the moment is just packing and deploying on supported bases) in CI, which means that these tests don't get exercised regularly.

We should add a workflow that runs `tox -e smoke` - note that this requires a real Juju, so the environment will need to install Juju, charmcraft, and a local cloud of some sort (e.g. microk8s/k8s or LXD), and bootstrap a controller.

We need to figure out:
* Do we want to just pick a Juju and charmcraft version and use that, or do we want to put those in a matrix (perhaps Juju 2.9, 3.5 (until 3.6 is released), and 4.0, with the latest charmcraft 2.x and 3.x)?
* What should trigger this? A schedule (perhaps just before we expect to release, like the charm dep checks), or only on a merge to main, or on each PR? The tests will be slow, but wouldn't have to be required.


Discussion / hints:
Poor recollection of stand-up discussion:

- probably a good idea to run the smoke tests sometimes
- can be optional in a PR (is run, not required to merge)
- can be on main branch only (we'd notice before a release)

Fix patch:
diff --git a/.github/workflows/smoke.yaml b/.github/workflows/smoke.yaml
new file mode 100644
index 000000000..2a36d6f62
--- /dev/null
+++ b/.github/workflows/smoke.yaml
@@ -0,0 +1,68 @@
+name: ops Smoke Tests
+
+on:
+  workflow_dispatch:
+    schedule:
+      - cron:  '0 7 25 * *'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        # pylibjuju does not currently support Juju 4.x
+        # The smoke tests do not yet work on Juju 2.9.
+        juju-version: ['3.5']
+        charmcraft-version: ['2.x', '3.x']
+        cloud: ['lxd', 'microk8s']
+
+    env:
+      JUJU_VERSION: "${{ matrix.juju-version }}"
+
+    steps:
+      # LXD is required for charmcraft to pack, even if it's not used as the
+      # Juju cloud.
+      - name: Set up LXD
+        uses: canonical/setup-lxd@8fb85546a934dfb994becf81341dd387ffe6aabb
+        with:
+          channel: 5.0/stable
+
+      - name: Set up Microk8s
+        if: matrix.cloud == 'microk8s'
+        uses: balchua/microk8s-actions@v0.3.2
+        with:
+          channel: '1.26-strict/stable'
+          devMode: 'true'
+          addons: '["dns", "hostpath-storage"]'
+
+      - name: Set up Juju (classic)
+        if: matrix.juju-version == '2.9'
+        run: sudo snap install juju --classic --channel=${{ matrix.juju-version }}
+
+      - name: Set up Juju
+        if: matrix.juju-version != '2.9'
+        run: sudo snap install juju --channel=${{ matrix.juju-version }}
+
+      - name: Bootstrap Juju controller (k8s)
+        if: matrix.cloud == 'microk8s'
+        run: sg snap_microk8s -c 'juju bootstrap microk8s'
+
+      - name: Bootstrap Juju controller (lxd)
+        if: matrix.cloud == 'lxd'
+        run: juju bootstrap localhost
+
+      - name: Install charmcraft
+        run: sudo snap install charmcraft --channel=${{ matrix.charmcraft-version }} --classic
+
+      - name: Checkout the repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3
+        uses: actions/setup-python@v5
+
+      - name: Install tox
+        run: pip install tox~=4.2
+
+      - name: Run smoke tests
+        run: tox -e smoke
diff --git a/tox.ini b/tox.ini
index cf097a12f..0e462abfb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -136,6 +136,7 @@ description = Run a smoke test against a Juju controller.
 allowlist_externals = juju
                       charmcraft
                       bash
+passenv = JUJU_VERSION
 deps =
     build
     coverage[toml]~=7.0
@@ -147,6 +148,8 @@ commands =
     python -m build --sdist --outdir={toxinidir}/test/charms/test_smoke/
     # Inject the tarball into the smoke test charm's requirements.
     bash -c 'echo "./$(ls -1 ./test/charms/test_smoke/ | grep tar.gz)" > ./test/charms/test_smoke/requirements.txt'
+    # If a specific Juju version is set, then make sure we are using that version of pylibjuju.
+    bash -c 'if [ -n "$JUJU_VERSION" ]; then pip install "juju ~= $JUJU_VERSION"; fi'
 
     # Run our smoke tests (this will build the charm, then run the tests).
     pytest -v --tb native --log-cli-level=INFO -s {posargs} {toxinidir}/test/smoke/


Test patch:
diff --git a/test/smoke/test_smoke.py b/test/smoke/test_smoke.py
index acf049722..f08936358 100644
--- a/test/smoke/test_smoke.py
+++ b/test/smoke/test_smoke.py
@@ -14,7 +14,11 @@
 #
 # Learn more about testing at: https://juju.is/docs/sdk/testing
 
+import grp
 import logging
+import os
+import pathlib
+import subprocess
 
 import pytest
 from pytest_operator.plugin import OpsTest
@@ -45,6 +49,33 @@
 """
 
 
+def pack(charm_dir: pathlib.Path):
+    """Pack the charm.
+
+    The pytest-operator plugin has a pack method, but it doesn't work out of the
+    box in GitHub actions, and there isn't really any reason that it should be
+    part of the plugin, so we just have a simple subprocess here.
+    """
+    cmd = ['charmcraft', 'pack', '--verbose']
+    # We need to use `sudo` in the GitHub actions environment, just as in
+    # the pack test. `sg lxd -c` should work, but does not - perhaps because of
+    # the way we are installing LXD?
+    if 'lxd' not in {grp.getgrgid(g).gr_name for g in os.getgroups()}:
+        cmd.insert(0, 'sudo')
+
+    logger.info('Building charm with %r', cmd)
+    subprocess.run(cmd, cwd=charm_dir, check=True)
+    logger.info('Built charm')
+
+    # Move the packed charm to the charm directory.
+    dest_name = None
+    for charm in charm_dir.glob('*.charm'):
+        dest_name = charm_dir / charm.name
+        charm.rename(dest_name)
+    # With the way we use charmcraft, we know that there will only be one.
+    return dest_name.absolute()
+
+
 @pytest.mark.parametrize(
     'base,charmcraft_version,name',
     (
@@ -55,13 +86,23 @@
 )
 async def test_smoke(ops_test: OpsTest, base: str, charmcraft_version: int, name: str):
     """Verify that we can build and deploy charms from supported bases."""
+    available_charmcraft_version = (
+        subprocess.run(['charmcraft', 'version'], check=True, capture_output=True)  # noqa: S607
+        .stdout.decode()
+        .strip()
+        .rsplit()[-1]
+        .split('.')
+    )
+    if int(available_charmcraft_version[0]) < charmcraft_version:
+        pytest.skip(f'charmcraft version {available_charmcraft_version} is too old for this test')
+        return
     charmcraft_yaml = {
         2: CHARMCRAFT2_YAML,
         3: CHARMCRAFT3_YAML,
     }[charmcraft_version].format(base=base)
     with open('./test/charms/test_smoke/charmcraft.yaml', 'w') as outf:
         outf.write(charmcraft_yaml)
-    charm = await ops_test.build_charm('./test/charms/test_smoke/')
+    charm = pack(pathlib.Path('./test/charms/test_smoke/'))
 
     app = await ops_test.model.deploy(
         charm, base=f'ubuntu@{base}', application_name=f'{name}-smoke'