Repository: posit-dev/py-shiny (language: python)

Issue:
Modify GitHub Actions Workflow to leverage draft mode status and remove merge queue
Modify GitHub Actions Workflow to accommodate the following - 
- Remove merge queue from all the jobs
- Split the playwright jobs into multiple jobs per browser - e.g.  Ubuntu 3.12 - Firefox, Ubuntu 3.12 - Chrome
    - All jobs should be done within ~ 4 mins
    - When merged we would need to adjust the required jobs in repo settings
- If the PR is in draft mode, only run Ubuntu checks for all job types
- If PR is not in draft mode, run all jobs
- when switching from Draft PR -> ready for review, all jobs should be automatically run without a commit
```yaml
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
```

[Reference doc for draft conditions](https://stackoverflow.com/questions/68349031/only-run-actions-on-non-draft-pull-request)
[Reference for PR](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request)


Discussion / hints:
For extra context, here are the places where I've encountered friction with the merge queue:

1. The tests in the merge queue are still sometimes flaky and can fail. AFAICT, you can't rerun an individual job like you can in a PR branch, so you have to add the PR back to the merge queue. If a flaky test fails on a different runner, you end up repeating the process.

   With PR checks, I can re-run individual checks until I get an overall green check before merging.

2. The merge queue tends to obscure the workflow. I find it harder to see the checks in general, but they're impossible to review on GitHub's mobile app. I often look through failing checks or trigger a re-run from my phone for PR checks, but can't do this for merge queue checks.

Fix patch:
diff --git a/.github/py-shiny/setup/action.yaml b/.github/py-shiny/setup/action.yaml
index 0b507a22f..dfa398f9f 100644
--- a/.github/py-shiny/setup/action.yaml
+++ b/.github/py-shiny/setup/action.yaml
@@ -12,38 +12,40 @@ runs:
         uses: actions/setup-python@v5
         with:
           python-version: ${{ inputs.python-version }}
-          # # Caching with pip only saves ~15 seconds. Not work risks of confusion.
-          # cache: 'pip'
-          # cache-dependency-path: |
-          #   setup.cfg
 
-      - name: Upgrade pip
+      - name: Upgrade `pip`
         shell: bash
-        run: python -m pip install --upgrade pip
+        run: |
+          python -m pip install --upgrade pip
 
-      - name: Pip list
+      - name: Install `uv`
+        shell: bash
+        run: |
+          pip install uv
+
+      # https://github.com/astral-sh/uv/blob/main/docs/guides/integration/github.md#using-uv-pip
+      - name: Allow uv to use the system Python by default
         shell: bash
         run: |
-          pip list
+          echo "UV_SYSTEM_PYTHON=1" >> $GITHUB_ENV
 
       - name: Install dependencies
         shell: bash
         run: |
-          pip install https://github.com/rstudio/py-htmltools/tarball/main
-          make install-deps
+          make ci-install-deps
 
       - name: Install
         shell: bash
         run: |
-          make install
+          make ci-install-wheel
 
       - name: Install backports.tarfile
         if: ${{ startsWith(inputs.python-version, '3.8') }}
         shell: bash
         run: |
-          pip install backports.tarfile
+          uv pip install backports.tarfile
 
       - name: Pip list
         shell: bash
         run: |
-          pip list
+          uv pip list
diff --git a/.github/workflows/build-docs.yaml b/.github/workflows/build-docs.yaml
index f0b3c670d..7b98723c4 100644
--- a/.github/workflows/build-docs.yaml
+++ b/.github/workflows/build-docs.yaml
@@ -1,11 +1,14 @@
 name: Build API docs and Shinylive for GitHub Pages
 
+# Allow for `main` branch to build full website and deploy
+# Allow branches that start with `docs-` to build full website, but not deploy
+# Allow for PRs to build quartodoc only. (No shinylive, no site build, no deploy)
+
 on:
   workflow_dispatch:
   push:
-    branches: ["main"]
+    branches: ["main", "docs-**"]
   pull_request:
-  merge_group:
 
 jobs:
   build-docs:
@@ -18,14 +21,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v5
+      - name: Setup py-shiny
+        uses: ./.github/py-shiny/setup
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Upgrade pip
-        run: python -m pip install --upgrade pip
-
       # =====================================================
       # API docs
       # =====================================================
@@ -36,20 +36,17 @@ jobs:
 
       - name: Install dependencies
         run: |
-          cd docs
-          make ../venv
-          make deps
+          make ci-install-docs
 
       - name: Run quartodoc
         run: |
-          cd docs
-          make quartodoc
+          make docs-quartodoc
 
       # =====================================================
       # Shinylive
       # =====================================================
       - name: Check out shinylive
-        if: github.ref == 'refs/heads/main'
+        if: github.event_name != 'pull_request'
         uses: actions/checkout@v4
         with:
           repository: rstudio/shinylive
@@ -57,7 +54,7 @@ jobs:
           path: shinylive-repo
 
       - name: Update shinylive's copy of shiny and htmltools
-        if: github.ref == 'refs/heads/main'
+        if: github.event_name != 'pull_request'
         run: |
           cd shinylive-repo
           make submodules
@@ -65,24 +62,22 @@ jobs:
           make submodules-pull-htmltools
 
       - name: Build shinylive
-        if: github.ref == 'refs/heads/main'
+        if: github.event_name != 'pull_request'
         run: |
           cd shinylive-repo
           make all
 
       - name: Use local build of shinylive for building docs
-        if: github.ref == 'refs/heads/main'
+        if: github.event_name != 'pull_request'
         run: |
-          . venv/bin/activate
-          cd shinylive-repo
-          shinylive assets install-from-local ./build
+          cd shinylive-repo && shinylive assets install-from-local ./build
 
       # =====================================================
       # Build site
       # =====================================================
 
       - name: Build site
-        if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'merge_group' || startsWith(github.head_ref, 'docs')  }}
+        if: github.event_name != 'pull_request'
         run: |
           cd docs
           make site
diff --git a/.github/workflows/verify-js-built.yaml b/.github/workflows/verify-js-built.yaml
index d38cd1a7d..02127f3b0 100644
--- a/.github/workflows/verify-js-built.yaml
+++ b/.github/workflows/verify-js-built.yaml
@@ -4,7 +4,6 @@ on:
   push:
     branches: ["main", "rc-*"]
   pull_request:
-  merge_group:
 
 jobs:
   verify_js_built:
diff --git a/Makefile b/Makefile
index 0dfc9f9c0..ecabe5f9f 100644
--- a/Makefile
+++ b/Makefile
@@ -98,7 +98,7 @@ check-pyright: pyright-typings
 	pyright
 check-pytest: FORCE
 	@echo "-------- Running tests with pytest ----------"
-	python3 tests/pytest/asyncio_prevent.py
+	python tests/pytest/asyncio_prevent.py
 	pytest
 
 # Check types with pyright
@@ -172,7 +172,7 @@ playwright-shiny: FORCE
 	$(MAKE) playwright TEST_FILE="tests/playwright/shiny/$(SUB_FILE)"
 
 # end-to-end tests on deployed apps with playwright; (SUB_FILE="" within tests/playwright/deploys/)
-playwright-deploys: install-rsconnect
+playwright-deploys: FORCE
 	$(MAKE) playwright TEST_FILE="tests/playwright/deploys/$(SUB_FILE)" PYTEST_BROWSERS="$(PYTEST_DEPLOYS_BROWSERS)"
 
 # end-to-end tests on all py-shiny examples with playwright; (SUB_FILE="" within tests/playwright/examples/)
@@ -189,20 +189,41 @@ release: dist ## package and upload a release
 
 dist: clean ## builds source and wheel package
 	pip install setuptools
-	python3 setup.py sdist
-	python3 setup.py bdist_wheel
+	python setup.py sdist
+	python setup.py bdist_wheel
 	ls -l dist
 
+
 ## install the package to the active Python's site-packages
 # Note that instead of --force-reinstall, we uninstall and then install, because
 # --force-reinstall also reinstalls all deps. And if we also used --no-deps, then the
 # deps wouldn't be installed the first time.
 install: dist
 	pip uninstall -y shiny
-	python3 -m pip install dist/shiny*.whl
+	python -m pip install dist/shiny*.whl
+ci-install-wheel: dist FORCE
+	# `uv` version of `make install`
+	uv pip uninstall shiny
+	uv pip install dist/shiny*.whl
 
 install-deps: FORCE ## install dependencies
 	pip install -e ".[dev,test]" --upgrade
+ci-install-deps: FORCE
+	uv pip install "htmltools @ git+https://github.com/posit-dev/py-htmltools.git"
+	uv pip install -e ".[dev,test]"
+
+install-docs: FORCE
+	pip install -e ".[dev,test,doc]"
+	pip install https://github.com/posit-dev/py-htmltools/tarball/main
+	pip install https://github.com/posit-dev/py-shinylive/tarball/main
+ci-install-docs: FORCE
+	uv pip install -e ".[dev,test,doc]"
+	uv pip install "htmltools @ git+https://github.com/posit-dev/py-htmltools.git" \
+			"shinylive @ git+https://github.com/posit-dev/py-shinylive.git"
+
+ci-install-rsconnect: FORCE
+	uv pip install "rsconnect-python @ git+https://github.com/rstudio/rsconnect-python.git"
+
 
 # ## If caching is ever used, we could run:
 # install-deps: FORCE ## install latest dependencies
diff --git a/docs/Makefile b/docs/Makefile
index 84a47d432..c863dccf0 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -1,4 +1,7 @@
-.PHONY: help Makefile
+# Using `FORCE` as prerequisite to _force_ the target to always run; https://www.gnu.org/software/make/manual/make.html#index-FORCE
+FORCE: ;
+
+.PHONY: Makefile
 .DEFAULT_GOAL := help
 
 define BROWSER_PYSCRIPT
@@ -23,84 +26,57 @@ export PRINT_HELP_PYSCRIPT
 
 BROWSER := python -c "$$BROWSER_PYSCRIPT"
 
-# Use venv from parent
-VENV = ../venv
-PYBIN = $(VENV)/bin
-
-# Any targets that depend on $(VENV) or $(PYBIN) will cause the venv to be
-# created. To use the venv, python scripts should run with the prefix $(PYBIN),
-# as in `$(PYBIN)/pip`.
-$(VENV):
-	python3 -m venv $(VENV)
-
-$(PYBIN): $(VENV)
-
-
-help:
+help: FORCE
 	@python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
 
-dev-htmltools: $(PYBIN)  ## Install development version of htmltools
-	$(PYBIN)/pip install https://github.com/posit-dev/py-htmltools/tarball/main
-
-dev-shinylive: $(PYBIN)  ## Install development version of shinylive
-	$(PYBIN)/pip install https://github.com/posit-dev/py-shinylive/tarball/main
-
-deps: $(PYBIN) dev-htmltools dev-shinylive ## Install build dependencies
-	$(PYBIN)/pip install pip --upgrade
-	$(PYBIN)/pip install ..[doc]
+deps: FORCE ## Install build dependencies
+	cd .. && $(MAKE) install-docs
 
-quartodoc: quartodoc_build_core quartodoc_build_express quartodoc_build_test quartodoc_post ## Build quartodocs for express and core
+quartodoc: quartodoc_build_core quartodoc_build_express quartodoc_build_test quartodoc_post ## Build quartodocs for express, core, and testing
 
 ## Build interlinks for API docs
-quartodoc_interlinks: $(PYBIN)
-	. $(PYBIN)/activate \
-	  && quartodoc interlinks
+quartodoc_interlinks: FORCE
+	quartodoc interlinks
 
 ## Build core API docs
-quartodoc_build_core: $(PYBIN) quartodoc_interlinks
+quartodoc_build_core: quartodoc_interlinks FORCE
 	$(eval export SHINY_ADD_EXAMPLES=true)
 	$(eval export IN_QUARTODOC=true)
 	$(eval export SHINY_MODE=core)
-	. $(PYBIN)/activate \
-	  && echo "::group::quartodoc build core docs" \
-	  && quartodoc build --config _quartodoc-core.yml --verbose \
-	  && mv objects.json _objects_core.json \
-	  && echo "::endgroup::"
+	@echo "::group::quartodoc build core docs"
+	quartodoc build --config _quartodoc-core.yml --verbose \
+	  && mv objects.json _objects_core.json
+	@echo "::endgroup::"
 
 ## Build express API docs
-quartodoc_build_express: $(PYBIN) quartodoc_interlinks
+quartodoc_build_express: quartodoc_interlinks FORCE
 	$(eval export SHINY_ADD_EXAMPLES=true)
 	$(eval export IN_QUARTODOC=true)
 	$(eval export SHINY_MODE=express)
-	. $(PYBIN)/activate \
-	  && echo "::group::quartodoc build express docs" \
-	  && quartodoc build --config _quartodoc-express.yml --verbose \
-	  && mv objects.json _objects_express.json \
-	  && echo "::endgroup::"
+	@echo "::group::quartodoc build express docs"
+	quartodoc build --config _quartodoc-express.yml --verbose \
+	  && mv objects.json _objects_express.json
+	@echo "::endgroup::"
 
 ## Build test API docs
-quartodoc_build_test: $(PYBIN) quartodoc_interlinks
+quartodoc_build_test: quartodoc_interlinks FORCE
 	$(eval export SHINY_ADD_EXAMPLES=true)
 	$(eval export IN_QUARTODOC=true)
 	$(eval export SHINY_MODE=express)
-	. $(PYBIN)/activate \
-	  && echo "::group::quartodoc build testing docs" \
-	  && quartodoc build --config _quartodoc-testing.yml --verbose \
-	  && mv objects.json _objects_test.json \
-	  && echo "::endgroup::"
+	@echo "::group::quartodoc build testing docs"
+	quartodoc build --config _quartodoc-testing.yml --verbose \
+	  && mv objects.json _objects_test.json
+	@echo "::endgroup::"
 
 ## Clean up after quartodoc build
-quartodoc_post: $(PYBIN)
-	. $(PYBIN)/activate \
-	  && python _combine_objects_json.py
+quartodoc_post: FORCE
+	python _combine_objects_json.py
 
-site:  ## Build website
-	. $(PYBIN)/activate \
-	  && quarto render
+site: FORCE  ## Build website (quarto render)
+	quarto render
 
-serve:  ## Build website and serve
-	. $(PYBIN)/activate \
-	  && quarto preview --port 8080
+serve: FORCE  ## Build website and serve (quarto preview)
+	quarto preview --port 8080
 
-clean:  ## Clean build artifacts
+clean: FORCE  ## Clean build artifacts
 	rm -rf _inv api _site .quarto


Test patch:
diff --git a/.github/py-shiny/pytest-browsers/action.yaml b/.github/py-shiny/pytest-browsers/action.yaml
index b9f470277..67a3843cf 100644
--- a/.github/py-shiny/pytest-browsers/action.yaml
+++ b/.github/py-shiny/pytest-browsers/action.yaml
@@ -1,14 +1,28 @@
-name: 'Custom merge queue browsers'
-description: 'Trim down pytest browsers for any github event other than merge_group.'
+name: 'Trim down pytest browsers'
+description: 'Trim down pytest browsers so the browser tabs are not shut down between tests, speeding up testing.'
 inputs:
+  browser:
+    description: 'Browser to use for testing. Currently supports `chromium`, `firefox`, and `webkit`.'
+    required: false
+    default: ''
   all-browsers:
     description: 'Force all pytest browsers to used when testing'
     required: false
     default: 'false'
+  disable-playwright-diagnostics:
+    description: 'Disable playwright diagnostics: tracing, video, screenshot'
+    required: false
+    default: 'true'
 outputs:
   browsers:
     description: 'pytest browsers to use'
     value: ${{ steps.browsers.outputs.browsers }}
+  has-playwright-diagnostics:
+    description: 'Whether playwright diagnostics have been enabled'
+    value: ${{ steps.browsers.outputs.has-playwright-diagnostics }}
+  playwright-diagnostic-args:
+    description: 'Args to supply to `make playwright` like commands.'
+    value: ${{ steps.browsers.outputs.playwright-diagnostic-args }}
 runs:
   using: "composite"
   steps:
@@ -16,6 +30,30 @@ runs:
         shell: bash
         id: browsers
         run: |
+          # Determine which browsers to use
+
+          if [ "${{ inputs.disable-playwright-diagnostics }}" == "true" ]; then
+            echo "Disabling playwright diagnostics!"
+            echo 'has-playwright-diagnostics=false' >> "$GITHUB_OUTPUT"
+            echo 'playwright-diagnostic-args=--tracing off --video off --screenshot off' >> "$GITHUB_OUTPUT"
+          else
+            echo "Using playwright diagnostics!"
+            echo 'has-playwright-diagnostics=true' >> "$GITHUB_OUTPUT"
+            echo 'playwright-diagnostic-args=--tracing=retain-on-failure --video=retain-on-failure --screenshot=only-on-failure --full-page-screenshot --output=test-results' >> "$GITHUB_OUTPUT"
+          fi
+
+          if [ "${{ inputs.browser }}" != "" ]; then
+            BROWSER="${{ inputs.browser }}"
+
+            if [ "$BROWSER" == "chromium" ] || [ "$BROWSER" == "firefox" ] || [ "$BROWSER" == "webkit" ]; then
+              echo "Using custom browser $BROWSER !"
+              echo "browsers=PYTEST_BROWSERS=\"--browser $BROWSER\"" >> "$GITHUB_OUTPUT"
+              exit 0
+            fi
+            echo "Unknown browser: $BROWSER"
+            exit 1
+          fi
+
           if [ "${{ inputs.all-browsers }}" == "true" ]; then
             echo "Using all browsers!"
             exit 0
diff --git a/.github/workflows/deploy-tests.yaml b/.github/workflows/deploy-tests.yaml
index 5a68555a8..d0e17f727 100644
--- a/.github/workflows/deploy-tests.yaml
+++ b/.github/workflows/deploy-tests.yaml
@@ -25,6 +25,10 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
+      - name: Install rsconnect
+        run: |
+          make ci-install-rsconnect
+
       - name: Test that deployable example apps work
         timeout-minutes: 5 # ~10s locally
         env:
diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml
index 50e80b837..43e5338d0 100644
--- a/.github/workflows/pytest.yaml
+++ b/.github/workflows/pytest.yaml
@@ -5,7 +5,7 @@ on:
   push:
     branches: ["main", "rc-*"]
   pull_request:
-  merge_group:
+    types: [opened, synchronize, reopened, ready_for_review]
   release:
     types: [published]
   schedule:
@@ -13,12 +13,19 @@ on:
 
 jobs:
   check:
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
         # "3.10" must be a string; otherwise it is interpreted as 3.1.
         python-version: ["3.12", "3.11", "3.10", "3.9", "3.8"]
         os: [ubuntu-latest, windows-latest, macOS-latest]
+        exclude:
+          - python-version: ${{ github.event.pull_request.draft && '3.11' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.10' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.9' }}
+          - os: ${{ github.event.pull_request.draft && 'windows-latest' }}
+          - os: ${{ github.event.pull_request.draft && 'macOS-latest' }}
+
       fail-fast: false
 
     steps:
@@ -49,13 +56,63 @@ jobs:
         run: |
           make check-format
 
+  pypi:
+    name: "Deploy to PyPI"
+    runs-on: ubuntu-latest
+    if: github.event_name == 'release'
+    needs: [check]
+    steps:
+      - uses: actions/checkout@v4
+      - name: "Set up Python 3.10"
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install https://github.com/rstudio/py-htmltools/tarball/main
+          make install-deps
+          make install
+      - name: "Build Package"
+        run: |
+          make dist
+
+      # test deploy ----
+      - name: "Test Deploy to PyPI"
+        uses: pypa/gh-action-pypi-publish@release/v1
+        if: startsWith(github.event.release.name, 'TEST')
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_TEST_API_TOKEN }}
+          repository-url: https://test.pypi.org/legacy/
+
+      ## prod deploy ----
+      - name: "Deploy to PyPI"
+        uses: pypa/gh-action-pypi-publish@release/v1
+        if: startsWith(github.event.release.name, 'shiny')
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_API_TOKEN }}
+
   playwright-shiny:
-    runs-on: ${{ matrix.os }}
     if: github.event_name != 'release'
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version: ["3.12", "3.11", "3.10", "3.9", "3.8"]
-        os: [ubuntu-latest]
+        browser: ["chromium", "firefox", "webkit"]
+        exclude:
+          - python-version: ${{ github.event.pull_request.draft && '3.11' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.10' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.9' }}
+          - browser: ${{ github.event.pull_request.draft && 'firefox' }}
+          - browser: ${{ github.event.pull_request.draft && 'webkit' }}
+          # There are many unexplained tests that fail on webkit w/ python 3.8, 3.9
+          # Given the more recent versions of python work, we will exclude this combination
+          - browser: "webkit"
+            python-version: "3.8"
+          - browser: "webkit"
+            python-version: "3.9"
       fail-fast: false
 
     steps:
@@ -69,28 +126,35 @@ jobs:
         uses: ./.github/py-shiny/pytest-browsers
         id: browsers
         with:
-          all-browsers: ${{ startsWith(github.head_ref, 'playwright') }}
-      - name: Display browser
-        shell: bash
-        run: echo '${{ steps.browsers.outputs.browsers }}'
+          browser: ${{ matrix.browser }}
+          # If anything other than `true`, it will heavily reduce webkit performance
+          # Related: https://github.com/microsoft/playwright/issues/18119
+          disable-playwright-diagnostics: ${{ matrix.browser == 'webkit' || matrix.browser == 'firefox' }}
+
       - name: Run End-to-End tests
-        timeout-minutes: 20
+        timeout-minutes: 60
         run: |
-          make playwright-shiny SUB_FILE=". -vv" ${{ steps.browsers.outputs.browsers }}
+          make playwright-shiny SUB_FILE=". --numprocesses 3 ${{ steps.browsers.outputs.playwright-diagnostic-args }}" ${{ steps.browsers.outputs.browsers }}
       - uses: actions/upload-artifact@v4
-        if: failure()
+        if: failure() && steps.browsers.outputs.has-playwright-diagnostics
         with:
-          name: "playright-shiny-${{ matrix.os }}-${{ matrix.python-version }}-results"
+          name: "playright-shiny-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.browser }}-results"
           path: test-results/
           retention-days: 5
 
   playwright-examples:
-    runs-on: ${{ matrix.os }}
     if: github.event_name != 'release'
+    runs-on: ubuntu-20.04
     strategy:
       matrix:
         python-version: ["3.12", "3.11", "3.10", "3.9", "3.8"]
-        os: [ubuntu-latest]
+        browser: ["chromium", "firefox", "webkit"]
+        exclude:
+          - python-version: ${{ github.event.pull_request.draft && '3.11' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.10' }}
+          - python-version: ${{ github.event.pull_request.draft && '3.9' }}
+          - browser: ${{ github.event.pull_request.draft && 'firefox' }}
+          - browser: ${{ github.event.pull_request.draft && 'webkit' }}
       fail-fast: false
 
     steps:
@@ -99,6 +163,14 @@ jobs:
         uses: ./.github/py-shiny/setup
         with:
           python-version: ${{ matrix.python-version }}
+      - name: Determine browsers for testing
+        uses: ./.github/py-shiny/pytest-browsers
+        id: browsers
+        with:
+          browser: ${{ matrix.browser }}
+          # If anything other than `true`, it will heavily reduce webkit performance
+          # Related: https://github.com/microsoft/playwright/issues/18119
+          disable-playwright-diagnostics: ${{ matrix.browser == 'webkit' || matrix.browser == 'firefox' }}
 
       - name: Install node.js
         uses: actions/setup-node@v4
@@ -111,30 +183,24 @@ jobs:
         run: |
           npm ci
 
-      - name: Determine browsers for testing
-        uses: ./.github/py-shiny/pytest-browsers
-        id: browsers
-        with:
-          all-browsers: ${{ startsWith(github.head_ref, 'playwright') }}
       - name: Run example app tests
-        timeout-minutes: 20
+        timeout-minutes: 60
         run: |
-          make playwright-examples SUB_FILE=". -vv" ${{ steps.browsers.outputs.browsers }}
+          make playwright-examples SUB_FILE=". --numprocesses 3 ${{ steps.browsers.outputs.playwright-diagnostic-args }}" ${{ steps.browsers.outputs.browsers }}
       - uses: actions/upload-artifact@v4
-        if: failure()
+        if: failure() && steps.browsers.outputs.has-playwright-diagnostics
         with:
-          name: "playright-examples-${{ matrix.os }}-${{ matrix.python-version }}-results"
+          name: "playright-examples-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.browser }}-results"
           path: test-results/
           retention-days: 5
 
   playwright-deploys-precheck:
     if: github.event_name != 'release'
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         # Matches deploy server python version
         python-version: ["3.10"]
-        os: [ubuntu-latest]
       fail-fast: false
 
     steps:
@@ -143,55 +209,22 @@ jobs:
         uses: ./.github/py-shiny/setup
         with:
           python-version: ${{ matrix.python-version }}
+      - name: Determine browsers for testing
+        uses: ./.github/py-shiny/pytest-browsers
+        id: browsers
+        with:
+          all-browsers: ${{ ! github.event.pull_request.draft }}
 
       - name: Test that deployable example apps work
         timeout-minutes: 5 # ~10s locally
         env:
           DEPLOY_APPS: "false"
         run: |
-          make playwright-deploys SUB_FILE=". -vv"
+          make playwright-deploys
 
       - uses: actions/upload-artifact@v4
         if: failure()
         with:
-          name: "playright-examples-${{ matrix.os }}-${{ matrix.python-version }}-results"
+          name: "playright-examples-${{ runner.os }}-${{ matrix.python-version }}-results"
           path: test-results/
           retention-days: 5
-
-  pypi:
-    name: "Deploy to PyPI"
-    runs-on: ubuntu-latest
-    if: github.event_name == 'release'
-    needs: [check]
-    steps:
-      - uses: actions/checkout@v4
-      - name: "Set up Python 3.10"
-        uses: actions/setup-python@v4
-        with:
-          python-version: "3.10"
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install https://github.com/rstudio/py-htmltools/tarball/main
-          make install-deps
-          make install
-      - name: "Build Package"
-        run: |
-          make dist
-
-      # test deploy ----
-      - name: "Test Deploy to PyPI"
-        uses: pypa/gh-action-pypi-publish@release/v1
-        if: startsWith(github.event.release.name, 'TEST')
-        with:
-          user: __token__
-          password: ${{ secrets.PYPI_TEST_API_TOKEN }}
-          repository_url: https://test.pypi.org/legacy/
-
-      ## prod deploy ----
-      - name: "Deploy to PyPI"
-        uses: pypa/gh-action-pypi-publish@release/v1
-        if: startsWith(github.event.release.name, 'shiny')
-        with:
-          user: __token__
-          password: ${{ secrets.PYPI_API_TOKEN }}
diff --git a/pytest.ini b/pytest.ini
index 1c02d79f4..267fc8c6c 100644
--- a/pytest.ini
+++ b/pytest.ini
@@ -2,4 +2,5 @@
 asyncio_mode=strict
 testpaths=tests/pytest/
 ; Note: Browsers are set within `./Makefile`
-addopts = --strict-markers --durations=6 --durations-min=5.0 --numprocesses auto --tracing=retain-on-failure --video=retain-on-failure
+addopts = --strict-markers --durations=6 --durations-min=5.0 --numprocesses auto
+verbosity_test_cases=2
diff --git a/tests/playwright/conftest.py b/tests/playwright/conftest.py
index efe5b5516..886c69dca 100644
--- a/tests/playwright/conftest.py
+++ b/tests/playwright/conftest.py
@@ -1,13 +1,9 @@
 # This file is necessary for pytest to find relative module files
 # such as examples/example_apps.py
-
 from __future__ import annotations
 
 from pathlib import PurePath
 
-import pytest
-from playwright.sync_api import BrowserContext, Page
-
 from shiny.pytest import ScopeName as ScopeName
 from shiny.pytest import create_app_fixture
 
@@ -23,44 +19,6 @@
 here_root = here.parent.parent
 
 
-# Make a single page fixture that can be used by all tests
-@pytest.fixture(scope="session")
-# By using a single page, the browser is only launched once and all tests run in the same tab / page.
-def session_page(browser: BrowserContext) -> Page:
-    """
-    Create a new page within the given browser context.
-
-    Parameters:
-        browser (BrowserContext): The browser context in which to create the new page.
-
-    Returns:
-        Page: The newly created page.
-
-    """
-    return browser.new_page()
-
-
-@pytest.fixture(scope="function")
-# By going to `about:blank`, we _reset_ the page to a known state before each test.
-# It is not perfect, but it is faster than making a new page for each test.
-# This must be done before each test
-def page(session_page: Page) -> Page:
-    """
-    Reset the given page to a known state before each test.
-
-    The page is built on the session_page, which is maintained over the full session.
-    The page will visit "about:blank" to reset between apps.
-    The default viewport size is set to 1920 x 1080 (1080p) for each test function.
-
-    Parameters:
-        session_page (Page): The page to reset.
-    """
-    session_page.goto("about:blank")
-    # Reset screen size to 1080p
-    session_page.set_viewport_size({"width": 1920, "height": 1080})
-    return session_page
-
-
 def create_example_fixture(
     example_name: str,
     example_file: str = "app.py",
diff --git a/tests/playwright/shiny/components/chat/append_user_msg/test_chat_append_user_msg.py b/tests/playwright/shiny/components/chat/append_user_msg/test_chat_append_user_msg.py
index 1d4e614e6..c42c5a214 100644
--- a/tests/playwright/shiny/components/chat/append_user_msg/test_chat_append_user_msg.py
+++ b/tests/playwright/shiny/components/chat/append_user_msg/test_chat_append_user_msg.py
@@ -10,8 +10,8 @@ def test_validate_chat_append_user_message(page: Page, local_app: ShinyAppProc)
     chat = controller.Chat(page, "chat")
 
     # Verify starting state
-    expect(chat.loc).to_be_visible()
-    chat.expect_latest_message("A user message")
+    expect(chat.loc).to_be_visible(timeout=30 * 1000)
+    chat.expect_latest_message("A user message", timeout=30 * 1000)
 
     # Verify that the message state is as expected
     message_state = controller.OutputCode(page, "message_state")
diff --git a/tests/playwright/shiny/components/chat/basic/test_chat_basic.py b/tests/playwright/shiny/components/chat/basic/test_chat_basic.py
index 2d1c3b147..82e1bcf42 100644
--- a/tests/playwright/shiny/components/chat/basic/test_chat_basic.py
+++ b/tests/playwright/shiny/components/chat/basic/test_chat_basic.py
@@ -10,9 +10,9 @@ def test_validate_chat_basic(page: Page, local_app: ShinyAppProc) -> None:
     chat = controller.Chat(page, "chat")
 
     # Verify starting state
-    expect(chat.loc).to_be_visible()
+    expect(chat.loc).to_be_visible(timeout=30 * 1000)
     initial_message = "Hello! How can I help you today?"
-    chat.expect_latest_message(initial_message)
+    chat.expect_latest_message(initial_message, timeout=30 * 1000)
 
     # Verify user input is empty and input button / enter is disabled
     chat.expect_user_input("")
diff --git a/tests/playwright/shiny/components/chat/errors/app.py b/tests/playwright/shiny/components/chat/errors/app.py
index bc699ba6c..6a9920fb8 100644
--- a/tests/playwright/shiny/components/chat/errors/app.py
+++ b/tests/playwright/shiny/components/chat/errors/app.py
@@ -1,4 +1,4 @@
-from shiny.express import ui
+from shiny.express import render, ui
 
 # Set some Shiny page options
 ui.page_opts(title="Hello Chat")
@@ -11,3 +11,11 @@
 @chat.on_user_submit
 async def _():
     raise Exception("boom!")
+
+
+"Message state:"
+
+
+@render.code
+def message_state():
+    return str(chat.messages())
diff --git a/tests/playwright/shiny/components/chat/errors/test_chat_errors.py b/tests/playwright/shiny/components/chat/errors/test_chat_errors.py
index c0401f88d..c38bbd5c4 100644
--- a/tests/playwright/shiny/components/chat/errors/test_chat_errors.py
+++ b/tests/playwright/shiny/components/chat/errors/test_chat_errors.py
@@ -4,15 +4,20 @@
 from shiny.run import ShinyAppProc
 
 
-def test_validate_chat_basic(page: Page, local_app: ShinyAppProc) -> None:
+def test_validate_chat_basic_error(page: Page, local_app: ShinyAppProc) -> None:
     page.goto(local_app.url)
 
     chat = controller.Chat(page, "chat")
-    expect(chat.loc).to_be_visible()
 
+    controller.OutputCode(page, "message_state").expect.not_to_have_text(
+        "",
+        timeout=30 * 1000,
+    )
+
+    expect(chat.loc).to_be_visible(timeout=30 * 1000)
     chat.set_user_input("Hello!")
     chat.send_user_input()
-    chat.expect_latest_message("Hello!")
+    chat.expect_latest_message("Hello!", timeout=30 * 1000)
 
     error_loc = page.locator(".shiny-notification-error")
     expect(error_loc).to_be_visible()
diff --git a/tests/playwright/shiny/components/chat/stream/test_chat_stream.py b/tests/playwright/shiny/components/chat/stream/test_chat_stream.py
index e40868948..04889b7ce 100644
--- a/tests/playwright/shiny/components/chat/stream/test_chat_stream.py
+++ b/tests/playwright/shiny/components/chat/stream/test_chat_stream.py
@@ -12,6 +12,9 @@ def test_validate_chat(page: Page, local_app: ShinyAppProc) -> None:
     chat = controller.Chat(page, "chat")
     message_state = controller.OutputCode(page, "message_state")
 
+    # Wait for app to load
+    message_state.expect.not_to_have_text("", timeout=30 * 1000)
+
     expect(chat.loc).to_be_visible()
     expect(chat.loc_input_button).to_be_disabled()
 
diff --git a/tests/playwright/shiny/components/chat/transform/test_chat_transform.py b/tests/playwright/shiny/components/chat/transform/test_chat_transform.py
index bce40b89a..4b06be4e9 100644
--- a/tests/playwright/shiny/components/chat/transform/test_chat_transform.py
+++ b/tests/playwright/shiny/components/chat/transform/test_chat_transform.py
@@ -8,14 +8,22 @@ def test_validate_chat_transform(page: Page, local_app: ShinyAppProc) -> None:
     page.goto(local_app.url)
 
     chat = controller.Chat(page, "chat")
+    message_state = controller.OutputCode(page, "message_state")
+    message_state2 = controller.OutputCode(page, "message_state2")
+
+    # Wait for app to load
+    message_state.expect_value("()", timeout=30 * 1000)
 
-    expect(chat.loc).to_be_visible()
+    expect(chat.loc).to_be_visible(timeout=30 * 1000)
     expect(chat.loc_input_button).to_be_disabled()
 
     user_msg = "hello"
     chat.set_user_input(user_msg)
     chat.send_user_input()
-    chat.expect_latest_message(f"Transformed input: {user_msg.upper()}")
+    chat.expect_latest_message(
+        f"Transformed input: {user_msg.upper()}",
+        timeout=30 * 1000,
+    )
 
     user_msg2 = "return None"
     chat.set_user_input(user_msg2)
@@ -27,7 +35,6 @@ def test_validate_chat_transform(page: Page, local_app: ShinyAppProc) -> None:
     chat.send_user_input()
     chat.expect_latest_message("Custom message")
 
-    message_state = controller.OutputCode(page, "message_state")
     message_state_expected = tuple(
         [
             {"content": user_msg.upper(), "role": "user"},
@@ -39,7 +46,6 @@ def test_validate_chat_transform(page: Page, local_app: ShinyAppProc) -> None:
     )
     message_state.expect_value(str(message_state_expected))
 
-    message_state2 = controller.OutputCode(page, "message_state2")
     message_state_expected2 = tuple(
         [
             {"content": user_msg, "role": "user"},
diff --git a/tests/playwright/shiny/components/chat/transform_assistant/test_chat_transform_assistant.py b/tests/playwright/shiny/components/chat/transform_assistant/test_chat_transform_assistant.py
index 0f16784af..7246a08e3 100644
--- a/tests/playwright/shiny/components/chat/transform_assistant/test_chat_transform_assistant.py
+++ b/tests/playwright/shiny/components/chat/transform_assistant/test_chat_transform_assistant.py
@@ -9,15 +9,20 @@ def test_validate_chat_transform_assistant(page: Page, local_app: ShinyAppProc)
     page.goto(local_app.url)
 
     chat = controller.Chat(page, "chat")
+    message_state = controller.OutputCode(page, "message_state")
+    message_state2 = controller.OutputCode(page, "message_state2")
+
+    # Wait for app to load
+    message_state.expect_value("()", timeout=30 * 1000)
 
-    expect(chat.loc).to_be_visible()
+    expect(chat.loc).to_be_visible(timeout=30 * 1000)
     expect(chat.loc_input_button).to_be_disabled()
 
     user_msg = "hello"
     chat.set_user_input(user_msg)
     chat.send_user_input()
     code = chat.loc_latest_message.locator("code")
-    expect(code).to_have_text("hello")
+    expect(code).to_have_text("hello", timeout=30 * 1000)
 
     user_msg2 = "return HTML"
     chat.set_user_input(user_msg2)
@@ -25,7 +30,6 @@ def test_validate_chat_transform_assistant(page: Page, local_app: ShinyAppProc)
     bold = chat.loc_latest_message.locator("b")
     expect(bold).to_have_text("Transformed response")
 
-    message_state = controller.OutputCode(page, "message_state")
     message_state_expected = tuple(
         [
             {"content": "hello", "role": "user"},
@@ -36,7 +40,6 @@ def test_validate_chat_transform_assistant(page: Page, local_app: ShinyAppProc)
     )
     message_state.expect_value(str(message_state_expected))
 
-    message_state2 = controller.OutputCode(page, "message_state2")
     message_state_expected2 = tuple(
         [
             {"content": "hello", "role": "user"},
diff --git a/tests/pytest/_utils.py b/tests/pytest/_utils.py
new file mode 100644
index 000000000..a36b04069
--- /dev/null
+++ b/tests/pytest/_utils.py
@@ -0,0 +1,17 @@
+from __future__ import annotations
+
+import sys
+from typing import Any, Callable, TypeVar
+
+import pytest
+
+CallableT = TypeVar("CallableT", bound=Callable[..., Any])
+
+
+def skip_on_windows(fn: CallableT) -> CallableT:
+    fn = pytest.mark.skipif(
+        sys.platform.startswith("win"),
+        reason="Does not run on windows",
+    )(fn)
+
+    return fn
diff --git a/tests/pytest/test_express_ui.py b/tests/pytest/test_express_ui.py
index 90b36c360..b53477cc9 100644
--- a/tests/pytest/test_express_ui.py
+++ b/tests/pytest/test_express_ui.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import sys
 import tempfile
 from pathlib import Path
@@ -122,9 +124,9 @@ def test_recall_context_manager():
         )
     )
 
-    with tempfile.NamedTemporaryFile(mode="w+t") as temp_file:
-        temp_file.write(card_app_express_text)
-        temp_file.flush()
-        res = run_express(Path(temp_file.name)).tagify()
+    with tempfile.TemporaryDirectory() as temp_dir:
+        temp_file = Path(temp_dir, "temp.file")
+        temp_file.write_text(card_app_express_text)
+        res = run_express(temp_file).tagify()
 
     assert str(res) == str(card_app_core)
diff --git a/tests/pytest/test_named_temporary_file.py b/tests/pytest/test_named_temporary_file.py
new file mode 100644
index 000000000..a96d2432c
--- /dev/null
+++ b/tests/pytest/test_named_temporary_file.py
@@ -0,0 +1,99 @@
+from __future__ import annotations
+
+import glob
+from pathlib import Path
+from typing import Dict, Set
+
+from tests.pytest._utils import skip_on_windows
+
+# File names that contain `tempfile.NamedTemporaryFile` but are known to be correct.
+# The key is the file name, the value is a set of lines stripped that contain `NamedTemporaryFile(`  that are known to be correct.
+# This file (test_named_temporary_file.py) is automatically excluded.
+known_entries: Dict[str, Set[str]] = {
+    "tests/pytest/test_poll.py": {
+        "tmpfile = tempfile.NamedTemporaryFile(delete=False)",
+        "tmpfile1 = tempfile.NamedTemporaryFile(delete=False)",
+    }
+}
+
+# Trim all line values of `known_entries`
+for k, v in known_entries.items():
+    known_entries[k] = {x.strip() for x in v}
+
+
+@skip_on_windows
+def test_named_temporary_file_is_not_used():
+    """
+    Windows does not work well with tempfile.NamedTemporaryFile.
+
+    * https://github.com/python/cpython/issues/58451
+    * https://github.com/appveyor/ci/issues/2547
+    * https://github.com/marickmanrho/pip-audit/commit/086ea4d684b41b795d9505b51ce7d079c990dca6
+    * https://github.com/IdentityPython/pysaml2/pull/665/files
+    * https://github.com/aws-cloudformation/cloudformation-cli/pull/924/files
+
+    Fair fix:
+    * Use a temp dir and create the file in the temp dir. On exit, the dir will be deleted.
+        * https://stackoverflow.com/a/77536782/591574
+    Possible future fix:
+    * Related Issue: https://github.com/pypa/pip-audit/issues/646
+    * Their fix: https://github.com/marickmanrho/pip-audit/commit/086ea4d684b41b795d9505b51ce7d079c990dca6#diff-a182a096790cc91a1771db39e19b337dec83c579775e46a45956a463b903b616
+    """
+
+    root_here = Path(__file__).parent.parent.parent
+    shiny_files = glob.glob(str(root_here / "shiny" / "**" / "*.py"), recursive=True)
+    tests_files = glob.glob(str(root_here / "tests" / "**" / "*.py"), recursive=True)
+
+    assert len(shiny_files) > 0
+    assert len(tests_files) > 0
+
+    all_files = [*shiny_files, *tests_files]
+
+    search_string = "NamedTemporaryFile("
+
+    bad_entries: list[tuple[Path, int, str]] = []
+
+    # For every python file...
+    for path in all_files:
+        path = Path(path)
+        # Skip if dir
+        if path.is_dir():
+            continue
+
+        # Skip this file
+        if path.name in {"test_named_temporary_file.py"}:
+            continue
+
+        with open(path, "r") as f:
+            # Read file contents
+            txt = f.read()
+
+            # Skip if search string is not in file
+            if search_string not in txt:
+                continue
+
+            # Split file contents by line
+            lines = txt.split("\n")
+            rel_path = path.relative_to(root_here)
+            known_lines = known_entries.get(str(rel_path), set())
+            seen_lines: set[str] = set()
+
+            # If the search string is in the line
+            # and the line is not in the known lines,
+            # add it to the bad entries
+            for i, line in enumerate(lines):
+                line = line.strip()
+                if search_string in line:
+                    seen_lines.add(line)
+                    if line not in known_lines:
+                        bad_entries.append((rel_path, i + 1, line))
+
+            if (len(known_lines) > 0) and (len(seen_lines) != len(known_lines)):
+                raise ValueError(
+                    f"Lines not found in {rel_path}: {known_lines - seen_lines}"
+                    "\nPlease remove them from the known_entries dictionary."
+                )
+
+    assert (
+        len(bad_entries) == 0
+    ), f"Unexpected files containing `TemporaryDirectory`: {str(bad_entries)}"
diff --git a/tests/pytest/test_theme.py b/tests/pytest/test_theme.py
index 826e1280e..1bfc42b87 100644
--- a/tests/pytest/test_theme.py
+++ b/tests/pytest/test_theme.py
@@ -21,6 +21,8 @@
     shiny_theme_presets_bundled,
 )
 
+from ._utils import skip_on_windows
+
 
 def test_theme_stores_values_correctly():
     theme = (
@@ -78,6 +80,7 @@ def test_theme_preset_must_be_valid():
         Theme("not_a_valid_preset")  # type: ignore
 
 
+@skip_on_windows
 @pytest.mark.parametrize("preset", shiny_theme_presets)
 def test_theme_css_compiles_and_is_cached(preset: ShinyThemePreset):
     theme = Theme(preset)
@@ -156,6 +159,7 @@ def _page_sidebar(*args, **kwargs) -> Tag:  # type: ignore
     return page_sidebar(sidebar("Sidebar"), *args, **kwargs)  # type: ignore
 
 
+@skip_on_windows
 @pytest.mark.parametrize(
     "page_fn",
     [