Repository: dotnet/aspire
Pull request #8015: [CI] Simplify gathering test projects to run for non-templates on PRs
Author: login: radical
Labels: name: area-engineering-systems
description: infrastructure helix infra engineering repo stuff
Created: 2025-03-11T20:25:55Z | Merged: 2025-03-12T22:20:50Z

Two separate jobs (for Linux, and Windows) are currently used to get the list of non-template projects to run. But only the `Playground` is skipped on Windows. So, this can be simplified to use a single job.


Diff:
diff --git a/.github/actions/enumerate-tests/action.yml b/.github/actions/enumerate-tests/action.yml
index 92d8015fec2..2378d1b311b 100644
--- a/.github/actions/enumerate-tests/action.yml
+++ b/.github/actions/enumerate-tests/action.yml
@@ -35,6 +35,15 @@ runs:
         $lines = Get-Content $filePath
         $jsonObject = @{
             "shortname" = $lines | Sort-Object
+            "os" = @(
+              @{
+                  "name"="ubuntu-latest"
+                  "title"="Linux"
+              }
+              @{
+                  "name"="windows-latest"
+                  "title"="Windows"
+              })
         }
         $jsonString = ConvertTo-Json $jsonObject -Compress
         "tests_matrix=$jsonString"
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
index 3d4b8fcaa2d..fef19e72846 100644
--- a/.github/workflows/run-tests.yml
+++ b/.github/workflows/run-tests.yml
@@ -33,7 +33,8 @@ jobs:
   test:
     runs-on: ${{ inputs.os }}
     timeout-minutes: 60
-    name: ${{ inputs.os }}
+    name: ${{ inputs.testShortName }}
+    if: inputs.os != 'windows-latest' || inputs.testShortName != 'Playground'
     steps:
       - name: Validate arguments
         shell: pwsh
diff --git a/.github/workflows/template-tests.yml b/.github/workflows/template-tests.yml
index d411fcef42b..2c0e242d3ae 100644
--- a/.github/workflows/template-tests.yml
+++ b/.github/workflows/template-tests.yml
@@ -67,7 +67,7 @@ jobs:
   results: # This job is used for branch protection. It ensures all the above tests passed
     if: ${{ always() }}
     runs-on: ubuntu-latest
-    name: Final Results
+    name: Templates Results
     needs: [test_linux, test_windows]
     steps:
       # get all the test-job-result* artifacts into a single directory
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index fc47fa6d6ef..74944102e34 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -14,8 +14,8 @@ jobs:
   # Duplicated jobs so their dependencies are not blocked on both the
   # setup jobs
 
-  setup_for_tests_lin:
-    name: Setup for tests (Linux)
+  setup_for_tests:
+    name: Setup for tests
     runs-on: ubuntu-latest
     outputs:
       tests_matrix: ${{ steps.generate_test_matrix.outputs.tests_matrix }}
@@ -27,48 +27,22 @@ jobs:
       - uses: ./.github/actions/enumerate-tests
         id: generate_test_matrix
 
-  setup_for_tests_win:
-    name: Setup for tests (Windows)
-    runs-on: windows-latest
-    outputs:
-      tests_matrix: ${{ steps.generate_test_matrix.outputs.tests_matrix }}
-    steps:
-      - uses: actions/checkout@v4
-
-      - id: generate_test_matrix
-        uses: ./.github/actions/enumerate-tests
-
-  test_linux:
-    uses: ./.github/workflows/run-tests.yml
-    name: Linux
-    needs: setup_for_tests_lin
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.setup_for_tests_lin.outputs.tests_matrix) }}
-    with:
-      testShortName: ${{ matrix.shortname }}
-      os: ubuntu-latest
-      testSessionTimeoutMs: ${{ matrix.testSessionTimeoutMs }}
-      extraTestArgs: ${{ matrix.extraTestArgs }}
-
-  test_windows:
+  test:
     uses: ./.github/workflows/run-tests.yml
-    name: Windows
-    needs: setup_for_tests_win
+    name: ${{ matrix.os.title }}
+    needs: setup_for_tests
     strategy:
       fail-fast: false
-      matrix: ${{ fromJson(needs.setup_for_tests_win.outputs.tests_matrix) }}
+      matrix: ${{ fromJson(needs.setup_for_tests.outputs.tests_matrix) }}
     with:
       testShortName: ${{ matrix.shortname }}
-      os: windows-latest
-      testSessionTimeoutMs: ${{ matrix.testSessionTimeoutMs }}
-      extraTestArgs: ${{ matrix.extraTestArgs }}
+      os: ${{ matrix.os.name }}
 
   results: # This job is used for branch protection. It ensures all the above tests passed
     if: ${{ always() }}
     runs-on: ubuntu-latest
-    name: Final Results
-    needs: [test_linux, test_windows]
+    name: Integration Results
+    needs: [test]
     steps:
       # get all the test-job-result* artifacts into a single directory
       - uses: actions/download-artifact@v4
diff --git a/tests/Aspire.Playground.Tests/AppHostTests.cs b/tests/Aspire.Playground.Tests/AppHostTests.cs
index 71e77b6c187..288a118bfc9 100644
--- a/tests/Aspire.Playground.Tests/AppHostTests.cs
+++ b/tests/Aspire.Playground.Tests/AppHostTests.cs
@@ -4,6 +4,7 @@
 using System.Net;
 using System.Text;
 using System.Text.RegularExpressions;
+using Aspire.Components.Common.Tests;
 using Aspire.Hosting;
 using Aspire.Hosting.ApplicationModel;
 using Aspire.Hosting.Tests.Utils;
@@ -18,6 +19,7 @@
 
 namespace Aspire.Playground.Tests;
 
+[RequiresDocker]
 public class AppHostTests
 {
     private readonly ITestOutputHelper _testOutput;
diff --git a/tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj b/tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj
index f0c34119019..78d21910a4e 100644
--- a/tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj
+++ b/tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj
@@ -8,8 +8,6 @@
 
     <!-- no docker support on helix/windows yet -->
     <RunTestsOnHelix Condition="'$(OS)' != 'Windows_NT'">true</RunTestsOnHelix>
-    <!-- no docker support on github-actions/windows yet -->
-    <RunTestsOnGithubActions Condition="'$(OS)' == 'Windows_NT'">false</RunTestsOnGithubActions>
     <SkipTests Condition="'$(OS)' == 'Windows_NT'">true</SkipTests>
 
     <DeployOutsideOfRepoSupportFilesRelativeDir>staging-archive\</DeployOutsideOfRepoSupportFilesRelativeDir>
diff --git a/tests/Aspire.Playground.Tests/ProjectSpecificTests.cs b/tests/Aspire.Playground.Tests/ProjectSpecificTests.cs
index cacb13f3ee5..57b7634cf57 100644
--- a/tests/Aspire.Playground.Tests/ProjectSpecificTests.cs
+++ b/tests/Aspire.Playground.Tests/ProjectSpecificTests.cs
@@ -11,6 +11,7 @@
 
 namespace Aspire.Playground.Tests;
 
+[RequiresDocker]
 public class ProjectSpecificTests(ITestOutputHelper _testOutput)
 {
     [Fact]


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR simplifies the test project enumeration process by consolidating the previously separate Linux and Windows jobs into a single job that uses a matrix strategy. Key changes include:
- Updating the action.yml to include an "os" list with details for Linux and Windows.
- Modifying run-tests.yml to use a new job name from the test matrix and an updated if-condition to skip the Playground tests on Windows.
- Consolidating the setup and test jobs in tests.yml into a single flow using the unified test matrix.

### Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

| File | Description |
| ---- | ----------- |
| .github/actions/enumerate-tests/action.yml | Added an "os" field listing Linux and Windows configurations |
| .github/workflows/run-tests.yml | Updated job naming and added a condition to skip running Playground tests on Windows |
| .github/workflows/tests.yml | Merged Linux and Windows setup and test jobs into one unified test job |


<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**.github/workflows/run-tests.yml:37**
* Wrap the conditional expression in ${{ ... }} to ensure it is correctly evaluated by GitHub Actions. For example, use: if: ${{ inputs.os != 'windows-latest' || inputs.testShortName != 'Playground' }}.
```
if: inputs.os != 'windows-latest' || inputs.testShortName != 'Playground'
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T19:18:56Z
user: login: danmoseley
body: if it works
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-12T19:21:25Z
user: login: eerhardt
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T19:26:31Z
user: login: eerhardt
body: LGTM. Thanks!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-12T19:27:18Z
user: login: radical
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T19:31:55Z

Review comments:
diff_hunk: @@ -33,7 +33,8 @@ jobs:
   test:
     runs-on: ${{ inputs.os }}
     timeout-minutes: 60
-    name: ${{ inputs.os }}
+    name: ${{ inputs.testShortName }}
+    if: inputs.os != 'windows-latest' || inputs.testShortName != 'Playground'
path: .github/workflows/run-tests.yml
user: login: eerhardt
body: Where were the "playground" tests filtered out previously? Can that be removed now that we are filtering them out here?

Another option could be to update the playground tests to have `[RequiresDocker]` on them, and just always run them, even if they no-op for now.
created_at: 2025-03-12T19:26:31Z
author_association: MEMBER
diff_hunk: @@ -33,7 +33,8 @@ jobs:
   test:
     runs-on: ${{ inputs.os }}
     timeout-minutes: 60
-    name: ${{ inputs.os }}
+    name: ${{ inputs.testShortName }}
+    if: inputs.os != 'windows-latest' || inputs.testShortName != 'Playground'
path: .github/workflows/run-tests.yml
user: login: radical
body: They are controlled from the project file:
https://github.com/dotnet/aspire/blob/0574cfcd61a28a538acca941bec71880e3b7f95f/tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj#L9-L13

I'll move to using the attribute instead of a condition here.
created_at: 2025-03-12T19:31:54Z
author_association: MEMBER

PR comments:
user: login: radical
created_at: 2025-03-11T20:26:08Z
body: First attempt.
author_association: MEMBER
user: login: radical
created_at: 2025-03-12T19:18:15Z
body: As a result of this the way the jobs are displayed on github changes from:
<img width="337" alt="Screenshot 2025-03-12 at 12 12 15" src="https://github.com/user-attachments/assets/73bcb4b8-f8b3-48d7-9231-c4ff77d9d5d8" />
... to ...
<img width="370" alt="Screenshot 2025-03-12 at 12 10 43" src="https://github.com/user-attachments/assets/e282963e-659b-4d89-8d07-0a09b79b29f2" />

author_association: MEMBER
user: login: radical
created_at: 2025-03-12T20:25:38Z
body: `main` pipeline validation run - https://dev.azure.com/dnceng-public/public/_build/results?buildId=977721&view=results
author_association: MEMBER