Repository: newfold-labs/workflows
Pull request #26: Test changes to plugin testing workflow
Author: login: desrosj
Created: 2025-04-09T13:35:04Z | Merged: 2025-05-09T17:01:55Z

## Proposed changes

This introduces a new workflow tests the `module-plugin-test.yml` one when changes are made.

This will help provide some confidence in workflow logic changes without having to wait for a module to run tests to confirm.

<!-- Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. -->

## Type of Change

<!-- What types of changes does your code introduce? -->
<!-- _Put an `x` in the boxes that apply_ -->

#### Production

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Dependency update
- [ ] Refactoring / housekeeping (changes to files not directly related to functionality)

<!-- Bugfixes should explain how to reproduce the bug -->

#### Development

- [ ] Tests
- [ ] Dependency update
- [ ] Environment update / refactoring
- [ ] Documentation Update

<!-- All PRs should endeavor to include tests -->
<!-- PRs with new tools or dependencies should explain how to use them -->

## Video

<!-- Add a short video demonstrating where the change takes effect and how to can be reproduced by reviewers -->
<!-- On macOS press cmd+shift+5 to open the screen recording tool. It will save the video to the desktop  -->

<!-- Drag and drop the video file into the GitHub editor to attach it -->

## Checklist

<!-- _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ -->

- [ ] I have read the [CONTRIBUTING](https://github.com/bluehost/.github/blob/master/.github/contributing.md) doc
- [ ] I have viewed my change in a web-browser
- [ ] Linting and tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added necessary documentation (if appropriate)

<!-- 
## Pre-deployment Checklist

- [ ] E.g. environmental variables which need to be set before deployment will work.
- [ ] E.g. other tickets which need to be complete before an API this change relies on will be ready
-->

<!--
## Post-deployment Checklist

How can the change be verified?

- [ ] E.g. temporarily enable logging and observe specific logs.
- [ ] E.g. observe new data in a specific view or table.
-->

## Further comments

<!-- If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc... -->
<!-- Now is a good time to create additional tickets for any necessary follow-up work. -->


Diff:
diff --git a/.github/workflows/module-plugin-test.yml b/.github/workflows/module-plugin-test.yml
index c944fc8..d9f7adf 100644
--- a/.github/workflows/module-plugin-test.yml
+++ b/.github/workflows/module-plugin-test.yml
@@ -51,6 +51,11 @@ on:
         type: boolean
         default: false
         required: false
+      artifact-name:
+        description: The name of the plugin artifact to upload.
+        type: string
+        default: ''
+        required: false
   workflow_dispatch:
     inputs:
       php-version:
@@ -227,7 +232,7 @@ jobs:
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ${{ steps.workflow.outputs.DIST }}
-          name: ${{ steps.workflow.outputs.PACKAGE }}
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}
 
       - name: Configure WordPress
         run: echo '{"plugins":["${{ steps.workflow.outputs.DIST }}/${{ steps.workflow.outputs.PACKAGE }}"]}' > .wp-env.override.json
@@ -255,7 +260,7 @@ jobs:
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ./tests/cypress/screenshots
-          name: ${{ steps.workflow.outputs.PACKAGE }}-screenshots
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}-screenshots
 
       - name: Output debug.log file contents
         if: always()
diff --git a/.github/workflows/verify-module-plugin-test.yml b/.github/workflows/verify-module-plugin-test.yml
new file mode 100644
index 0000000..e38d77d
--- /dev/null
+++ b/.github/workflows/verify-module-plugin-test.yml
@@ -0,0 +1,149 @@
+# Verifies changes to the brand plugin testing workflow.
+name: Verify Module Plugin Testing
+
+on:
+  push:
+    branches:
+      - main
+      - develop
+    paths:
+      - '**module-plugin-test.yml'
+  pull_request:
+    types: [ opened, edited, synchronize, reopened, ready_for_review ]
+    paths:
+      - '**module-plugin-test.yml'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.head_ref || github.sha }}
+  cancel-in-progress: true
+
+# Disable permissions for all available scopes by default.
+# Any needed permissions should be configured individually for each job.
+permissions: {}
+
+jobs:
+  # The following methodology is applied when building out the matrix below:
+  #
+  # - One job per brand plugin runs all tests.
+  # - One module per brand plugin with missing Cypress tests is tested. The rest are excluded.
+  # - All modules containing Cypress tests bundled with all three brand plugins are tested.
+  # - Each retired brand plugin is tested once in combination with a module containing Cypress tests.
+  verify-module-plugin-test:
+    name: ${{ matrix.name }} (${{ format( '{0}{1}{2}', matrix.module, ( matrix.secondary-module-repo && format( '{0}{1}', ' with ', matrix.secondary-module-repo ) || '' ), ( ! matrix.only-module-tests && ' - all tests' || '' ) ) }})
+    strategy:
+      fail-fast: false
+      matrix:
+        # The brand plugins to test against.
+        brand: [ 'bluehost', 'hostgator', 'web.com' ]
+        # Modules with Cypress tests that are included in all active brand plugins.
+        module: [ 'coming-soon', 'deactivation', 'features', 'marketplace', 'notifications', 'performance' ]
+        only-module-tests: [ true ]
+
+        include:
+          # Define the plugin-repo value for each brand plugin.
+          - brand: 'bluehost'
+            name: 'Bluehost'
+            plugin-repo: 'newfold-labs/wp-plugin-bluehost'
+          - brand: 'hostgator'
+            name: 'HostGator'
+            plugin-repo: 'newfold-labs/wp-plugin-hostgator'
+          - brand: 'web.com'
+            name: 'Web.com'
+            plugin-repo: 'newfold-labs/wp-plugin-web'
+
+          # Include one job for each brand plugin that runs all tests.
+          - brand: 'bluehost'
+            name: 'Bluehost'
+            module: 'staging'
+            plugin-repo: 'newfold-labs/wp-plugin-bluehost'
+            only-module-tests: false
+          - brand: 'hostgator'
+            name: 'HostGator'
+            plugin-repo: 'newfold-labs/wp-plugin-hostgator'
+            module: 'global-ctb'
+            only-module-tests: false
+          - brand: 'web.com'
+            name: 'Web.com'
+            plugin-repo: 'newfold-labs/wp-plugin-web'
+            module: 'notifications'
+            only-module-tests: false
+
+          # Include one job for each brand plugin that runs with a secondary module.
+          - brand: 'bluehost'
+            name: 'Bluehost'
+            plugin-repo: 'newfold-labs/wp-plugin-bluehost'
+            module: 'hosting'
+            secondary-module-repo: 'installer'
+            only-module-tests: true
+          - brand: 'hostgator'
+            name: 'HostGator'
+            plugin-repo: 'newfold-labs/wp-plugin-hostgator'
+            module: 'help-center'
+            secondary-module-repo: 'migration'
+            only-module-tests: true
+          - brand: 'web.com'
+            name: 'Web.com'
+            plugin-repo: 'newfold-labs/wp-plugin-web'
+            module: 'notifications'
+            secondary-module-repo: 'performance'
+            only-module-tests: true
+
+          # Include one job for each brand plugin with a module lacking Cypress tests.
+          #
+          # Current modules included in all brand plugins lacking Cypress tests:
+          # - ai (Bluehost only)
+          # - activation
+          # - atomic
+          # - context
+          # - data
+          # - facebook (Bluehost & HostGator only)
+          # - loader
+          # - onboarding-data (Bluehost only)
+          # - pls (Bluehost only)
+          # - runtime
+          # - secure-passwords
+          # - sso
+          - brand: 'bluehost'
+            name: 'Bluehost'
+            plugin-repo: 'newfold-labs/wp-plugin-bluehost'
+            module: 'ai'
+            only-module-tests: false
+          - brand: 'hostgator'
+            name: 'HostGator'
+            plugin-repo: 'newfold-labs/wp-plugin-hostgator'
+            module: 'facebook'
+            only-module-tests: false
+          - brand: 'web.com'
+            name: 'Web.com'
+            plugin-repo: 'newfold-labs/wp-plugin-web'
+            module: 'secure-passwords'
+            only-module-tests: false
+
+          # Include one job for each retired brand plugin.
+          - brand: 'crazydomains'
+            name: 'Crazy Domains'
+            plugin-repo: 'newfold-labs/wp-plugin-crazy-domains'
+            module: 'performance'
+            only-module-tests: true
+          - brand: 'mojo'
+            name: 'Mojo'
+            plugin-repo: 'newfold-labs/wp-plugin-mojo'
+            module: 'coming-soon'
+            only-module-tests: true
+
+        exclude:
+          # The Web.com plugin currently excludes tests for the coming-soon module in its test configuration.
+          # This causes a failure because Cypress tests are present, but none are found.
+          # See https://github.com/newfold-labs/wp-plugin-web/blob/main/cypress.config.js#L87.
+          - brand: 'web.com'
+            module: 'coming-soon'
+
+    uses: ./.github/workflows/module-plugin-test.yml
+    with:
+      plugin-repo: ${{ matrix.plugin-repo }}
+      module-repo: newfold-labs/wp-module-${{ matrix.module }}
+      module-branch: 'main'
+      node-version: '20'
+      only-module-tests: ${{ matrix.only-module-tests && true || false }}
+      artifact-name: ${{ format( 'wp-plugin-{0}-{1}-{2}', matrix.brand, matrix.module, strategy.job-index ) }}
+    secrets: inherit


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces a new workflow to test changes made to the module plugin test YAML files, ensuring workflow logic is validated early.  
- Adds a new workflow (verify-module-plugin-test.yml) to test module plugin configurations on push and pull_request events.  
- Extends the module-plugin-test.yml workflow to support an optional artifact-name input for more customizable artifact naming.

### Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

| File | Description |
| ---- | ----------- |
| .github/workflows/verify-module-plugin-test.yml | New workflow setup for verifying module plugin tests. |
| .github/workflows/module-plugin-test.yml | Added artifact-name input and updated artifact upload names based on this input. |




state: COMMENTED
author_association: NONE
submitted_at: 2025-05-06T14:21:42Z
user: login: desrosj
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T13:37:08Z
user: login: desrosj
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T13:37:13Z
user: login: circlecube
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-09T15:29:49Z

Review comments:
diff_hunk: @@ -227,7 +232,7 @@ jobs:
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ${{ steps.workflow.outputs.DIST }}
-          name: ${{ steps.workflow.outputs.PACKAGE }}
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}
path: .github/workflows/module-plugin-test.yml
user: login: Copilot
body: [nitpick] The conditional expression for selecting an artifact name is hard to read; consider using a more explicit ternary-like syntax (if supported) or refactoring for clarity.
created_at: 2025-05-06T14:21:42Z
author_association: NONE
diff_hunk: @@ -255,7 +260,7 @@ jobs:
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ./tests/cypress/screenshots
-          name: ${{ steps.workflow.outputs.PACKAGE }}-screenshots
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}-screenshots
path: .github/workflows/module-plugin-test.yml
user: login: Copilot
body: [nitpick] The expression for determining the artifact name for screenshots is a bit convoluted; consider refactoring for improved readability and future maintainability.
created_at: 2025-05-06T14:21:42Z
author_association: NONE
diff_hunk: @@ -227,7 +232,7 @@ jobs:
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ${{ steps.workflow.outputs.DIST }}
-          name: ${{ steps.workflow.outputs.PACKAGE }}
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}
path: .github/workflows/module-plugin-test.yml
user: login: desrosj
body: Unfortunately, this is the syntax required for valid GitHub Actions expressions.
created_at: 2025-05-09T13:37:08Z
author_association: MEMBER
diff_hunk: @@ -255,7 +260,7 @@ jobs:
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
           path: ./tests/cypress/screenshots
-          name: ${{ steps.workflow.outputs.PACKAGE }}-screenshots
+          name: ${{ inputs.artifact-name != '' && inputs.artifact-name || steps.workflow.outputs.PACKAGE }}-screenshots
path: .github/workflows/module-plugin-test.yml
user: login: desrosj
body: Unfortunately, this is the syntax required for valid GitHub Actions expressions.
created_at: 2025-05-09T13:37:13Z
author_association: MEMBER

PR comments:
user: login: circlecube
created_at: 2025-05-07T19:29:13Z
body: Looks good but as discussed, we can skip the web plugin for now at least, since it is pretty outdated. Once it is updated we can add it back.
author_association: MEMBER
user: login: desrosj
created_at: 2025-05-09T15:15:53Z
body: @circlecube I fixed a few issues and excluded the one job that would never pass due to tests being excluded in the module.

To clarify, I was suggesting we only skip the `coming-soon` module for the Web plugin. The other modules test successfully, so I think it's good to continue confirming they work as expected.
author_association: MEMBER