Repository: dhsc-govuk/FingertipsNext
Pull request #203: DHSCFT-287 add chart component screenshot testing
Author: login: williamwillis11
Created: 2025-02-28T16:49:02Z | Merged: 2025-03-06T09:12:41Z

# Description

Jira ticket: [DHSCFT-287](https://bjss-enterprise.atlassian.net/browse/DHSCFT-287)

Added visual screenshot comparison testing of the chart components for E2E tests when they are executed in CI against docker. Note that this testing is not performed when running the e2e tests locally or in CD mode against azure, and that the screenshots are stored in cache, not in the repo.

Currently the assertion is in a try catch as agreed in the demo.

## Changes

- added screenshot comparison of chart components current against expected, stored in github cache - currently warning only
- added manual workflow job to allow updating of the base screenshots against a branch
- this workflow also creates the base snapshots for a branch when the branch is opened
- removed double cache of the node dependencies in the ui and e2e yml templates
- changes to playwright config to support screenshot testing
- README updates to describe screenshot testing
- fixed a bug where we were not checking the line chart table for visibility

## Validation

E2E tests pass locally and in CI
UI tests pass locally and in CI



Diff:
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index 3292b71b3..4ae26ee99 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -14,3 +14,5 @@ Include a summary of what this pull request is changing and any relevant backgro
 ## Validation
 
 Details of any validation you have performed to show that your changes are behaving as expected. For example, screenshots of how the UI looks with your changes, or output of any tests you did manually to demonstrate things were working as expected.
+
+If you had to update the chart component screenshots then please highlight that here by linking to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job.
diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml
index 646319aa6..9e47d9ffb 100644
--- a/.github/workflows/e2e-tests.yml
+++ b/.github/workflows/e2e-tests.yml
@@ -12,6 +12,11 @@ on:
         description: 'container image to run the e2e tests in - for CD test executions'
         required: false
         type: string
+      update-screenshots:
+        description: 'Update screenshot snapshots?'
+        type: string
+        default: 'false'
+        required: false
     secrets:
       slack-webhook-url:
         description: 'URL of the slack webhook'
@@ -43,12 +48,15 @@ jobs:
 
       - name: Cache dependencies
         uses: actions/cache@v3
+        id: cache
         with:
           path: |
-            ${{ env.frontend-directory }}/node_modules
             ~/.cache/ms-playwright
-          key: ${{ runner.os }}-deps-${{ hashFiles('./frontend/fingertips-frontend/package-lock.json') }}
-          restore-keys: ${{ runner.os }}-deps-
+            ${{ env.frontend-directory }}/.test/**
+          key: cache/${{github.repository}}/${{github.ref}}
+          restore-keys: |
+            cache/${{github.repository}}/refs/pull/${{github.event.pull_request.number}}/merge
+            cache/${{github.repository}}/refs/heads/main
 
       - name: Install dependencies
         run: npm ci
@@ -64,6 +72,13 @@ jobs:
         if: inputs.e2e-test-container-image == null
         run: npx playwright install --with-deps chromium webkit
 
+      - name: Create base snapshots if none in cache, or during manual snapshot update or when merging to main
+        if: >
+          steps.cache.outputs.cache-hit != 'true' ||
+          inputs.update-screenshots == 'true' ||
+          github.ref == 'refs/heads/main'
+        run: npm run test-e2e-ci-update-snapshots
+
       - name: Run Playwright tests in CI
         if: inputs.e2e-test-container-image == null
         run: npm run test-e2e-ci-docker
@@ -79,15 +94,15 @@ jobs:
         env:
           FINGERTIPS_FRONTEND_URL: ${{ vars.FINGERTIPS_FRONTEND_URL }}
 
-      - name: Upload playwright report and screenshot if e2e tests fail - always on failure
+      - name: Upload playwright artefacts
         uses: actions/upload-artifact@v4
         if: failure()
         with:
-          name: playwright-failure-artefacts
+          name: playwright-artefacts
           path: |
             ${{ env.frontend-directory }}/playwright-report/
             ${{ env.frontend-directory }}/test-results/*/*.png
-          retention-days: 10
+          retention-days: 5
 
       - name: Send slack message if e2e tests fail - for CD test execution
         if: failure() && inputs.e2e-test-container-image != null
diff --git a/.github/workflows/fingertips-workflow.yml b/.github/workflows/fingertips-workflow.yml
index 200e800ad..0d5c541a6 100644
--- a/.github/workflows/fingertips-workflow.yml
+++ b/.github/workflows/fingertips-workflow.yml
@@ -2,6 +2,12 @@ name: Fingertips Workflow
 
 on:
   workflow_dispatch:
+    inputs:
+      update-screenshots:
+        description: 'Update screenshot snapshots?'
+        type: boolean
+        default: false
+        required: false
   push:
     branches:
       - '**'
@@ -223,6 +229,8 @@ jobs:
         )
       )
     uses: ./.github/workflows/e2e-tests.yml
+    with:
+      update-screenshots: ${{ inputs.update-screenshots }}
     secrets:
       fingertips_e2e_db_password: ${{ secrets.FINGERTIPS_E2E_DB_PASSWORD }}
 
diff --git a/.github/workflows/ui-tests.yml b/.github/workflows/ui-tests.yml
index 39a5ef8ce..6af0a117b 100644
--- a/.github/workflows/ui-tests.yml
+++ b/.github/workflows/ui-tests.yml
@@ -35,14 +35,6 @@ jobs:
           cache: 'npm'
           cache-dependency-path: ${{ env.frontend-directory }}/package-lock.json
 
-      - name: Cache dependencies
-        uses: actions/cache@v3
-        with:
-          path: |
-            ${{ env.frontend-directory }}/node_modules
-          key: ${{ runner.os }}-deps-${{ hashFiles('./frontend/fingertips-frontend/package-lock.json') }}
-          restore-keys: ${{ runner.os }}-deps-
-
       - name: Install dependencies
         run: npm ci
 
@@ -57,4 +49,4 @@ jobs:
           path: |
             ${{ env.frontend-directory }}/playwright-report/
             ${{ env.frontend-directory }}/test-results/*/*.png
-          retention-days: 10
+          retention-days: 5
diff --git a/frontend/fingertips-frontend/.gitignore b/frontend/fingertips-frontend/.gitignore
index 77a45b06a..54c3945a2 100644
--- a/frontend/fingertips-frontend/.gitignore
+++ b/frontend/fingertips-frontend/.gitignore
@@ -43,3 +43,4 @@ next-env.d.ts
 /ctrf/
 /blob-report/
 /playwright/.cache/
+.test/
diff --git a/frontend/fingertips-frontend/README.md b/frontend/fingertips-frontend/README.md
index cf98be98f..c70a2b2cf 100644
--- a/frontend/fingertips-frontend/README.md
+++ b/frontend/fingertips-frontend/README.md
@@ -81,17 +81,17 @@ You can then open [http://localhost:3000](http://localhost:3000) with your brows
 
 This project uses Jest + React Testing Library for unit testing and Playwright for ui and e2e testing.
 
-Isolated ui testing occurs in both CI/CD pipelines on push and pull requests, as well as when code merges to main.
+Isolated ui testing, covering accessibility, page navigation and validations occurs in both CI on push and pull requests, as well in CD as when code merges to main.
 
-Note that while all tests are executed in CI/CD pipelines, for e2e tests there is a difference between where and how the tests are executed:
+For e2e tests there is a difference between where and how the tests are executed:
 
- 1. In CI, which occurs on push and in pull requests, we execute the e2e tests against a dockerised container instance of fingertips running in the github runner agent.
+ 1. In CI, which occurs on push and in pull requests, we execute the e2e tests against a dockerised container instance of fingertips running in the github runner agent. Note that we also perform visual screenshot snapshot testing at this point.
 
- 2. In CD, which occurs when code merges into main, we execute the e2e tests against the deployed azure instance of fingertips.
+ 2. In CD, which occurs when code merges into main, we execute the e2e tests against the deployed azure instance of fingertips. Note we do not perform visual screenshot snapshot testing at this point.
 
 For local development we also have the option to run the tests locally against mocks or against a locally dockerised container instance of fingertips.
 
-### Running the Unit tests
+### Running the Jest Unit tests
 
 ```bash
 npm run test
@@ -124,9 +124,9 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
@@ -134,6 +134,24 @@ To check there are 0 accessibility violations call expectNoAccessibilityViolatio
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see `maxDiffPixelRatio` in the playwright config file), then the e2e tests will fail and you will need to update the base screenshots. To do this:
+
+1. Download `playwright-artefacts` from the github workflow summary page, and open the `index.html` file in the `playwright-report` folder, then in the Playwright report open the failed test and you will be presented with a 'Diff' page that shows the before and after.
+2. Review and compare the expected (base) screenshots and actual (current) screenshots in the playwright report with a BA to confirm the new screenshots are correct.
+3. Once the changes have been confirmed as correct go to `https://github.com/dhsc-govuk/FingertipsNext/actions/workflows/fingertips-workflow.yml` and click `Run workflow` then *pick your branch* and tick the `Update screenshot snapshots?` checkbox. This will run a new workflow in which the base screenshots will be updated in the cache against your branch reference. The e2e tests run in this workflow as well, using these new screenshots, and they should now pass.
+4. Now that the screenshots are updated in the cache *for your current branch* all subsequent workflow executions that trigger the e2e tests in CI, for that branch will pass and when you merge to main the screenshots will be automatically updated in the cache for main.
+5. Ensure that when you put your PR up for review you explicitly mention that your changes caused the cached screenshots to need to be updated, and link to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job. This allows the PR reviewer to check the before and after screenshots.
+
+Note that if you merge main into your branch, and if the changes that come into your branch changed the way any of the tested chart components look, then your next push will fail on the screenshot comparisons and you will need to execute step 3 (in isolation) from above.
+
 ## Code structure
 
 The `app` folder contains the pages that are rendered server side. The pages are in folders that will correspond to the route.
diff --git a/frontend/fingertips-frontend/package.json b/frontend/fingertips-frontend/package.json
index 93b90787b..7f895aaf0 100644
--- a/frontend/fingertips-frontend/package.json
+++ b/frontend/fingertips-frontend/package.json
@@ -18,10 +18,11 @@
     "typecheck": "tsc",
     "test": "jest",
     "test-ci": "jest --ci --silent",
-    "test-ui-local-mocks": "playwright test isolated_ui_tests",
-    "test-e2e-local-docker": "MOCK_SERVER=false npm run start:all && playwright test fully_integrated_e2e_tests || true && npm run stop:all",
+    "test-ui-local-mocks": "playwright test isolated_ui_tests --ignore-snapshots",
+    "test-e2e-local-docker": "MOCK_SERVER=false npm run start:all && playwright test fully_integrated_e2e_tests --ignore-snapshots || true && npm run stop:all",
     "test-e2e-ci-docker": "MOCK_SERVER=false npm run start:all && playwright test fully_integrated_e2e_tests",
-    "test-e2e-ci-azure": "MOCK_SERVER=false playwright test fully_integrated_e2e_tests",
+    "test-e2e-ci-update-snapshots": "MOCK_SERVER=false npm run start:all && playwright test fully_integrated_e2e_tests --update-snapshots",
+    "test-e2e-ci-azure": "MOCK_SERVER=false playwright test fully_integrated_e2e_tests --ignore-snapshots",
     "prettier": "prettier --write .",
     "prettier-ci": "prettier . --check"
   },
diff --git a/frontend/fingertips-frontend/playwright.config.ts b/frontend/fingertips-frontend/playwright.config.ts
index 147465c3f..bb24e575e 100644
--- a/frontend/fingertips-frontend/playwright.config.ts
+++ b/frontend/fingertips-frontend/playwright.config.ts
@@ -1,6 +1,8 @@
 import { defineConfig, devices, PlaywrightTestConfig } from '@playwright/test';
 
+const isCI = !!process.env.CI;
 const url = process.env.FINGERTIPS_FRONTEND_URL || 'http://localhost:3000';
+console.log(`The target URL for this test execution is ${url}`); // allows to see where tests are executed
 const jobUrl = process.env.JOB_URL;
 const runCommand =
   process.env.MOCK_SERVER === 'false' ? 'npm run dev-no-mocks' : 'npm run dev';
@@ -9,11 +11,18 @@ const runCommand =
 const config: PlaywrightTestConfig = {
   testDir: './playwright/tests',
   fullyParallel: true,
-  forbidOnly: !!process.env.CI, // fails the build on CI if you accidentally left test.only in the source code.
-  retries: process.env.CI ? 1 : 0,
-  workers: process.env.CI ? 2 : undefined,
-  expect: { timeout: 10_000 },
-  reporter: process.env.CI
+  forbidOnly: isCI, // fails the build on CI if you accidentally left test.only in the source code
+  retries: isCI ? 1 : 0,
+  workers: isCI ? 2 : undefined,
+  expect: {
+    timeout: 10_000,
+    toHaveScreenshot: {
+      maxDiffPixelRatio: 0.01,
+      pathTemplate: '.test/spec/snaps/{projectName}/{testFilePath}/{arg}{ext}',
+    },
+  },
+
+  reporter: isCI
     ? [
         ['list'],
         ['@estruyf/github-actions-reporter'],
diff --git a/frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts b/frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
index 7efbcab2a..218a3dbc5 100644
--- a/frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
+++ b/frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
@@ -35,27 +35,45 @@ export default class ChartPage extends BasePage {
     );
     console.log(
       `for indicator mode: ${indicatorMode} + area mode: ${areaMode} - checking that chart components: ${visibleComponents} are displayed and that`,
-      `chart components: ${hiddenComponents} are not displayed.`
+      `chart components: ${hiddenComponents} are not displayed. Also checking the visible components via screenshot snapshot testing.`
     );
     // Check that components expected to be visible are displayed
-    for (const component of visibleComponents) {
-      if (component !== 'lineChartTable-component') {
-        await expect(this.page.getByTestId(component)).toBeVisible({
+    for (const visibleComponent of visibleComponents) {
+      if (visibleComponent !== 'lineChartTable-component') {
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
           visible: true,
         });
-        // click into the tab view
-        if (component === 'lineChartTable-component') {
-          await this.page.getByTestId('tabTitle-table').click();
-          await expect(this.page.getByTestId(component)).toBeVisible({
-            visible: true,
-          });
-        }
+      }
+      // click into the tab view if checking lineChartTable
+      if (visibleComponent === 'lineChartTable-component') {
+        await this.page.getByTestId('tabTitle-table').click();
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
+          visible: true,
+        });
+      }
+
+      // screenshot snapshot comparisons are skipped when running e2e test locally or against deployed azure environments
+      console.log(
+        `checking component:${visibleComponent} for unexpected visual changes - see directory README.md for details.`
+      );
+      await this.page.waitForTimeout(500);
+
+      // for now just warn if visual comparisons do not match
+      try {
+        await expect(
+          this.page.getByTestId(visibleComponent)
+        ).toHaveScreenshot();
+      } catch (error) {
+        const typedError = error as Error;
+        console.warn(
+          `⚠️ Screenshot comparison warning for ${visibleComponent}: ${typedError.message}`
+        );
       }
     }
 
     // Check that components expected not to be visible are not displayed
-    for (const component of hiddenComponents) {
-      await expect(this.page.getByTestId(component)).toBeVisible({
+    for (const hiddenComponent of hiddenComponents) {
+      await expect(this.page.getByTestId(hiddenComponent)).toBeVisible({
         visible: false,
       });
     }
diff --git a/frontend/fingertips-frontend/playwright/page-objects/pages/resultsPage.ts b/frontend/fingertips-frontend/playwright/page-objects/pages/resultsPage.ts
index 4a475ec76..6c6af0c8e 100644
--- a/frontend/fingertips-frontend/playwright/page-objects/pages/resultsPage.ts
+++ b/frontend/fingertips-frontend/playwright/page-objects/pages/resultsPage.ts
@@ -156,7 +156,7 @@ export default class ResultsPage extends BasePage {
    * @param searchTerm - search term to be used in the URL check
    */
   async selectAreasFiltersAndCheckURL(areaMode: AreaMode, searchTerm: string) {
-    // For area type filter currently defaulting to using regions (except for England area mode) - this will be refactored in the future
+    // For area type filter currently defaulting to using regions (except for England area mode) - this will be refactored in DHSCFT-416
     const defaultAreaTypeFilter = 'regions';
     const defaultGroupType = 'England';
 
@@ -167,7 +167,7 @@ export default class ResultsPage extends BasePage {
       .selectOption(defaultAreaTypeFilter);
     await this.waitForURLToContain(defaultAreaTypeFilter);
 
-    // For group type filter currently defaults to using England due to picking regions for area type above - this will be refactored in the future
+    // For group type filter currently defaults to using England due to picking regions for area type above - this will be refactored in DHSCFT-416
 
     // Select appropriate number of checkboxes based on area mode
     const areaCheckboxList = this.page
diff --git a/frontend/fingertips-frontend/playwright/testHelpers.ts b/frontend/fingertips-frontend/playwright/testHelpers.ts
index fe0de853e..217f5e0bb 100644
--- a/frontend/fingertips-frontend/playwright/testHelpers.ts
+++ b/frontend/fingertips-frontend/playwright/testHelpers.ts
@@ -24,7 +24,7 @@ export function getScenarioConfig(
   indicatorMode: IndicatorMode,
   areaMode: AreaMode
 ): ScenarioConfig {
-  // Temporarily disabled until the pop pyramid is implemented fully under DHSCFT-148.
+  // Temporarily disabled until the pop pyramid is implemented fully under DHSCFT-148
   // const defaultVisible = [this.populationPyramidComponent];
   const defaultVisible: never[] = [];
 
@@ -42,7 +42,7 @@ export function getScenarioConfig(
       ChartPage.lineChartComponent,
       ChartPage.lineChartTableComponent,
       ChartPage.barChartComponent,
-      // Temporarily disabled until the pop pyramid is implemented fully under DHSCFT-148.
+      // Temporarily disabled until the pop pyramid is implemented fully under DHSCFT-148
       // ChartPage.populationPyramidComponent,
       ChartPage.inequalitiesComponent,
     ],
diff --git a/frontend/fingertips-frontend/playwright/tests/fully_integrated_e2e_tests/core_journeys.spec.ts b/frontend/fingertips-frontend/playwright/tests/fully_integrated_e2e_tests/core_journeys.spec.ts
index dd9dfa6ab..b29db1e83 100644
--- a/frontend/fingertips-frontend/playwright/tests/fully_integrated_e2e_tests/core_journeys.spec.ts
+++ b/frontend/fingertips-frontend/playwright/tests/fully_integrated_e2e_tests/core_journeys.spec.ts
@@ -10,6 +10,7 @@ import { IndicatorDocument } from '@/lib/search/searchTypes';
 const indicatorData = indicators as IndicatorDocument[];
 const searchTerm = 'resident';
 let allIndicatorIDs: string[];
+
 interface TestParams {
   indicatorMode: IndicatorMode;
   areaMode: AreaMode;


Reviews:
user: login: dlzhry2
body: Couple of minor comments added. Happy to approve once addressed - nice one!
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T15:59:58Z
user: login: dlzhry2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T16:01:11Z
user: login: MonkeyDate
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T16:02:28Z
user: login: williamwillis11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T17:14:16Z
user: login: williamwillis11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T17:17:21Z
user: login: dlzhry2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T08:56:52Z
user: login: dlzhry2
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T08:58:58Z
user: login: williamwillis11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T09:08:14Z
user: login: dlzhry2
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T09:10:07Z

Review comments:
diff_hunk: @@ -81,17 +81,17 @@ You can then open [http://localhost:3000](http://localhost:3000) with your brows
 
 This project uses Jest + React Testing Library for unit testing and Playwright for ui and e2e testing.
 
-Isolated ui testing occurs in both CI/CD pipelines on push and pull requests, as well as when code merges to main.
+Isolated ui testing, covering accessibility, page navigation and validations occurs in both CI on push and pull requests, as well in CD as when code merges to main.
 
-Note that while all tests are executed in CI/CD pipelines, for e2e tests there is a difference between where and how the tests are executed:
+For e2e tests there is a difference between where and how the tests are executed:
 
- 1. In CI, which occurs on push and in pull requests, we execute the e2e tests against a dockerised container instance of fingertips running in the github runner agent.
+ 1. In CI, which occurs on push and in pull requests, we execute the e2e tests against a dockerised container instance of fingertips running in the github runner agent. Note that we also perform visual screenshot snapshot testing at this point.
 
- 2. In CD, which occurs when code merges into main, we execute the e2e tests against the deployed azure instance of fingertips.
+ 2. In CD, which occurs when code merges into main, we execute the e2e tests against the deployed azure instance of fingertips. Note we do not perform visual screenshot snapshot testing at this point.
 
-For local development we also have the option to run the tests locally against mocks or against a locally dockerised container instance of fingertips.
+For local development we also have the option to run the tests locally against mocks or against a locally dockerised container instance of fingertips. 
path: frontend/fingertips-frontend/README.md
user: login: dlzhry2
body: Very minor but for consistency.

```suggestion
For local development we also have the option to run the tests locally against mocks or against a locally dockerised container instance of fingertips.
```
created_at: 2025-03-05T15:45:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,16 +124,34 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
 To check there are 0 accessibility violations call expectNoAccessibilityViolations();
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see maxDiffPixelRatio in the playwright config file), then the e2e tests will fail and you you will need to update the base screenshots. To do this:
path: frontend/fingertips-frontend/README.md
user: login: dlzhry2
body: ```suggestion
If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see `maxDiffPixelRatio` in the playwright config file), then the e2e tests will fail and you will need to update the base screenshots. To do this:
```
created_at: 2025-03-05T15:46:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,16 +124,34 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
 To check there are 0 accessibility violations call expectNoAccessibilityViolations();
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see maxDiffPixelRatio in the playwright config file), then the e2e tests will fail and you you will need to update the base screenshots. To do this:
+
+1. Download `playwright-artefacts` from the github workflow summary page, and open the `index.html` file in the `playwright-report` folder, then in the Playwright report open the failed test and you will be presented with a 'Diff' page that shows the before and after.
+2. Review and compare the expected (base) screenshots and actual (current) screenshots in the playwright report with a BA to confirm the new screenshots are correct.
+3. Once the changes have been confirmed as correct go to `https://github.com/dhsc-govuk/FingertipsNext/actions/workflows/fingertips-workflow.yml` and click `Run workflow` then *pick your branch* and tick the `Update screenshot snapshots?` checkbox. This will run a new workflow in which the base screenshots will be updated in the cache against your branch reference. The e2e tests run in this workflow as well, using these new screenshots, and they should now pass.
+4. Now that the screenshots are updated in the cache *for your current branch* all subsequent workflow executions that trigger the e2e tests in CI, for that branch will pass and when you merge to main the screenshots will be automatically updated in the cache for main.
+5. Ensure that when you put your PR up for review you explicitly mention that your changes caused the cached screenshots to need to be updated, and link to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job. This allows the PR reviewer to check the before and after screenshots.
path: frontend/fingertips-frontend/README.md
user: login: dlzhry2
body: Should we put a note in the pull request template to ensure this isn't forgotten? The `Validation` section of the template could be a sensible place to drop this note.
created_at: 2025-03-05T15:49:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,27 +35,45 @@ export default class ChartPage extends BasePage {
     );
     console.log(
       `for indicator mode: ${indicatorMode} + area mode: ${areaMode} - checking that chart components: ${visibleComponents} are displayed and that`,
-      `chart components: ${hiddenComponents} are not displayed.`
+      `chart components: ${hiddenComponents} are not displayed. Also checking the visible components via screenshot snapshot testing.`
     );
     // Check that components expected to be visible are displayed
-    for (const component of visibleComponents) {
-      if (component !== 'lineChartTable-component') {
-        await expect(this.page.getByTestId(component)).toBeVisible({
+    for (const visibleComponent of visibleComponents) {
+      if (visibleComponent !== 'lineChartTable-component') {
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
           visible: true,
         });
-        // click into the tab view
-        if (component === 'lineChartTable-component') {
-          await this.page.getByTestId('tabTitle-table').click();
-          await expect(this.page.getByTestId(component)).toBeVisible({
-            visible: true,
-          });
-        }
+      }
+      // click into the tab view if checking lineChartTable
+      if (visibleComponent === 'lineChartTable-component') {
+        await this.page.getByTestId('tabTitle-table').click();
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
+          visible: true,
+        });
+      }
+
+      // screenshot snapshot comparisons are skipped when running e2e test locally or against deployed azure environments
+      console.log(
+        `checking component:${visibleComponent} for unexpected visual changes - see directory README.md for details.`
+      );
+      await this.page.waitForTimeout(500);
path: frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
user: login: dlzhry2
body: Did we say we would try to use a different waiting strategy on the call? E.g. `waitForSelector` https://www.browserstack.com/guide/playwright-wait-types
created_at: 2025-03-05T16:01:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,16 +124,34 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
 To check there are 0 accessibility violations call expectNoAccessibilityViolations();
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see maxDiffPixelRatio in the playwright config file), then the e2e tests will fail and you you will need to update the base screenshots. To do this:
+
+1. Download `playwright-artefacts` from the github workflow summary page, and open the `index.html` file in the `playwright-report` folder, then in the Playwright report open the failed test and you will be presented with a 'Diff' page that shows the before and after.
+2. Review and compare the expected (base) screenshots and actual (current) screenshots in the playwright report with a BA to confirm the new screenshots are correct.
+3. Once the changes have been confirmed as correct go to `https://github.com/dhsc-govuk/FingertipsNext/actions/workflows/fingertips-workflow.yml` and click `Run workflow` then *pick your branch* and tick the `Update screenshot snapshots?` checkbox. This will run a new workflow in which the base screenshots will be updated in the cache against your branch reference. The e2e tests run in this workflow as well, using these new screenshots, and they should now pass.
+4. Now that the screenshots are updated in the cache *for your current branch* all subsequent workflow executions that trigger the e2e tests in CI, for that branch will pass and when you merge to main the screenshots will be automatically updated in the cache for main.
+5. Ensure that when you put your PR up for review you explicitly mention that your changes caused the cached screenshots to need to be updated, and link to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job. This allows the PR reviewer to check the before and after screenshots.
path: frontend/fingertips-frontend/README.md
user: login: williamwillis11
body: yes good idea, but i dont know how we do that. hmm
created_at: 2025-03-05T17:14:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,27 +35,45 @@ export default class ChartPage extends BasePage {
     );
     console.log(
       `for indicator mode: ${indicatorMode} + area mode: ${areaMode} - checking that chart components: ${visibleComponents} are displayed and that`,
-      `chart components: ${hiddenComponents} are not displayed.`
+      `chart components: ${hiddenComponents} are not displayed. Also checking the visible components via screenshot snapshot testing.`
     );
     // Check that components expected to be visible are displayed
-    for (const component of visibleComponents) {
-      if (component !== 'lineChartTable-component') {
-        await expect(this.page.getByTestId(component)).toBeVisible({
+    for (const visibleComponent of visibleComponents) {
+      if (visibleComponent !== 'lineChartTable-component') {
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
           visible: true,
         });
-        // click into the tab view
-        if (component === 'lineChartTable-component') {
-          await this.page.getByTestId('tabTitle-table').click();
-          await expect(this.page.getByTestId(component)).toBeVisible({
-            visible: true,
-          });
-        }
+      }
+      // click into the tab view if checking lineChartTable
+      if (visibleComponent === 'lineChartTable-component') {
+        await this.page.getByTestId('tabTitle-table').click();
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
+          visible: true,
+        });
+      }
+
+      // screenshot snapshot comparisons are skipped when running e2e test locally or against deployed azure environments
+      console.log(
+        `checking component:${visibleComponent} for unexpected visual changes - see directory README.md for details.`
+      );
+      await this.page.waitForTimeout(500);
path: frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
user: login: williamwillis11
body: yes i am going to address this but do you mind if i do it in a subsequent ticket/branch/PR? there isnt a readily available selector just yet to work with to know 100% the data has rendered in the chart component
created_at: 2025-03-05T17:17:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,27 +35,45 @@ export default class ChartPage extends BasePage {
     );
     console.log(
       `for indicator mode: ${indicatorMode} + area mode: ${areaMode} - checking that chart components: ${visibleComponents} are displayed and that`,
-      `chart components: ${hiddenComponents} are not displayed.`
+      `chart components: ${hiddenComponents} are not displayed. Also checking the visible components via screenshot snapshot testing.`
     );
     // Check that components expected to be visible are displayed
-    for (const component of visibleComponents) {
-      if (component !== 'lineChartTable-component') {
-        await expect(this.page.getByTestId(component)).toBeVisible({
+    for (const visibleComponent of visibleComponents) {
+      if (visibleComponent !== 'lineChartTable-component') {
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
           visible: true,
         });
-        // click into the tab view
-        if (component === 'lineChartTable-component') {
-          await this.page.getByTestId('tabTitle-table').click();
-          await expect(this.page.getByTestId(component)).toBeVisible({
-            visible: true,
-          });
-        }
+      }
+      // click into the tab view if checking lineChartTable
+      if (visibleComponent === 'lineChartTable-component') {
+        await this.page.getByTestId('tabTitle-table').click();
+        await expect(this.page.getByTestId(visibleComponent)).toBeVisible({
+          visible: true,
+        });
+      }
+
+      // screenshot snapshot comparisons are skipped when running e2e test locally or against deployed azure environments
+      console.log(
+        `checking component:${visibleComponent} for unexpected visual changes - see directory README.md for details.`
+      );
+      await this.page.waitForTimeout(500);
path: frontend/fingertips-frontend/playwright/page-objects/pages/chartPage.ts
user: login: dlzhry2
body: Yes, that sounds fine to me. Resolving.
created_at: 2025-03-06T08:56:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,16 +124,34 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
 To check there are 0 accessibility violations call expectNoAccessibilityViolations();
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see maxDiffPixelRatio in the playwright config file), then the e2e tests will fail and you you will need to update the base screenshots. To do this:
+
+1. Download `playwright-artefacts` from the github workflow summary page, and open the `index.html` file in the `playwright-report` folder, then in the Playwright report open the failed test and you will be presented with a 'Diff' page that shows the before and after.
+2. Review and compare the expected (base) screenshots and actual (current) screenshots in the playwright report with a BA to confirm the new screenshots are correct.
+3. Once the changes have been confirmed as correct go to `https://github.com/dhsc-govuk/FingertipsNext/actions/workflows/fingertips-workflow.yml` and click `Run workflow` then *pick your branch* and tick the `Update screenshot snapshots?` checkbox. This will run a new workflow in which the base screenshots will be updated in the cache against your branch reference. The e2e tests run in this workflow as well, using these new screenshots, and they should now pass.
+4. Now that the screenshots are updated in the cache *for your current branch* all subsequent workflow executions that trigger the e2e tests in CI, for that branch will pass and when you merge to main the screenshots will be automatically updated in the cache for main.
+5. Ensure that when you put your PR up for review you explicitly mention that your changes caused the cached screenshots to need to be updated, and link to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job. This allows the PR reviewer to check the before and after screenshots.
path: frontend/fingertips-frontend/README.md
user: login: dlzhry2
body: You can add a note in here and it'll appear in the pull request template: `.github/pull_request_template.md`

Could be as simple as a reminder note to add a link to the before/after screenshots for your reviewers.
created_at: 2025-03-06T08:58:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,16 +124,34 @@ Each test will be executed in parallel using Chromium and Webkit as defined in p
 
 Note we use the full chromium headless mode offered by recent playwright versions see https://playwright.dev/docs/release-notes#try-new-chromium-headless for details, we do to this make our ui and e2e testing as close to real world as possible.
 
-### Accessibility Testing:
+### Accessibility Testing
 
-Performed at the ui stage. Libraries used: @axe-core/playwright and axe-playwright. 
+Performed in the ui tests. Libraries used: @axe-core/playwright and axe-playwright. 
 
 Configured to the WCAG2.2 AA standard in the following file playwright/page-objects/pageFactory.ts.
 
 To check there are 0 accessibility violations call expectNoAccessibilityViolations();
 
 Any violations of this standard cause a test failure unless the rule violated has been accepted in pageFactory.ts.
 
+### Visual Screenshot Snapshot Testing
+
+Only performed in the e2e tests and only when they run in CI. Therefore they are not performed when the e2e tests are run locally, and they are not performed in CD when we merge into main. Running them locally is not recommended, following best practice as defined by the playwright docs - https://playwright.dev/docs/test-snapshots to avoid flake and complexity as the screenshot snapshots will be different for different platforms.
+
+All screenshot snapshots are stored in github cache, not directly in the repository.
+
+If you are in a new branch, there wont be any screenshots for that branch yet, so the cache dependencies job will check the main branch for base screenshots using the fallback restore-keys.
+
+If you have made changes in your branch that have correctly resulted in the screenshots generated not matching the cached base screenshots, within the tolerance ratio (see maxDiffPixelRatio in the playwright config file), then the e2e tests will fail and you you will need to update the base screenshots. To do this:
+
+1. Download `playwright-artefacts` from the github workflow summary page, and open the `index.html` file in the `playwright-report` folder, then in the Playwright report open the failed test and you will be presented with a 'Diff' page that shows the before and after.
+2. Review and compare the expected (base) screenshots and actual (current) screenshots in the playwright report with a BA to confirm the new screenshots are correct.
+3. Once the changes have been confirmed as correct go to `https://github.com/dhsc-govuk/FingertipsNext/actions/workflows/fingertips-workflow.yml` and click `Run workflow` then *pick your branch* and tick the `Update screenshot snapshots?` checkbox. This will run a new workflow in which the base screenshots will be updated in the cache against your branch reference. The e2e tests run in this workflow as well, using these new screenshots, and they should now pass.
+4. Now that the screenshots are updated in the cache *for your current branch* all subsequent workflow executions that trigger the e2e tests in CI, for that branch will pass and when you merge to main the screenshots will be automatically updated in the cache for main.
+5. Ensure that when you put your PR up for review you explicitly mention that your changes caused the cached screenshots to need to be updated, and link to the `Artifact download URL:` from the `Upload playwright artefacts` step of the `e2e-tests-local-docker` failure job. This allows the PR reviewer to check the before and after screenshots.
path: frontend/fingertips-frontend/README.md
user: login: williamwillis11
body: done and pushed - thanks for showing me this
created_at: 2025-03-06T09:08:14Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-06T09:08:36Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=dhsc-govuk_FingertipsNext&pullRequest=203) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=dhsc-govuk_FingertipsNext&pullRequest=203&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=dhsc-govuk_FingertipsNext&pullRequest=203&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=dhsc-govuk_FingertipsNext&pullRequest=203&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=dhsc-govuk_FingertipsNext&pullRequest=203&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=dhsc-govuk_FingertipsNext&pullRequest=203&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=dhsc-govuk_FingertipsNext&pullRequest=203)


author_association: NONE