Issue 1:
System info

Playwright Version: [v1.35.1] (latest docker container)

Operating System: [Ubuntu Gitlab Runner]

Browser: [Chromium]

Other info: Running on Gitlab Enterprise CI

Source code

Config file

import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
// Look for test files in the "tests" directory, relative to this configuration file.
testDir: "e2e",
testIgnore: "e2e/todo/**",

timeout: 60000,

// Run all tests in parallel.
fullyParallel: true,

// // Fail the build on CI if you accidentally left test.only in the source code.
// forbidOnly: !!process.env.CI,

// // Retry on CI only.
retries: process.env.CI ? 2 : 0,

// // Opt out of parallel tests on CI.
workers: process.env.CI ? 1 : undefined,

// Reporter to use
reporter: "html",

outputDir: "test-results",

use: {
// Collect trace when retrying the failed test.
trace: "on",
screenshot: "only-on-failure",
},
// Configure projects for major browsers.
projects: [
// Setup project
{ name: "setup", testMatch: /.*\.setup\.ts/ },
{
name: "chromium",
use: {
...devices["Desktop Chrome"],
storageState: "playwright/.auth/keycloak.json",
},
dependencies: ["setup"],
},
],
// Run your local dev server before starting the tests.
webServer: {
command: "npm run start",
url: "http://localhost:5173",
reuseExistingServer: true,
},
});

Test file (self-contained)

import { test, expect } from "@playwright/test";
import { test_param_id } from "./params";

const sc_options = {
fullPage: true,
maxDiffPixels: 100,
};

test.describe("Screenshot Tests", () => {
test.beforeEach(async ({ page }) => {
await page.goto(`http://localhost:5173/projects/load?ip=${test_param_id}`);
await page.waitForURL("http://localhost:5173/steps/1");
await page.setViewportSize({ width: 2560, height: 1800 });
await expect(page.locator(".right-step-1")).toBeVisible();
});

test("step 1", async ({ page }) => {
await expect(page).toHaveScreenshot(sc_options);
});
test("step 2", async ({ page }) => {
await page.click(".right-step-1");
await page.waitForURL("http://localhost:5173/steps/2");
await expect(page).toHaveScreenshot(sc_options);
});
test("step 3", async ({ page }) => {
await page.click(".right-step-1");
await page.waitForURL("http://localhost:5173/steps/2");
await page.click(".right-step-2");
await page.waitForURL("http://localhost:5173/steps/3");
await expect(page).toHaveScreenshot(sc_options);
});
test("step 4", async ({ page }) => {
await page.click(".right-step-1");
await page.waitForURL("http://localhost:5173/steps/2");
await page.click(".right-step-2");
await page.waitForURL("http://localhost:5173/steps/3");
await page.click(".right-step-3");
await page.waitForURL("http://localhost:5173/steps/4");
await expect(page).toHaveScreenshot(sc_options);
});
test("step 5", async ({ page }) => {
await page.click(".right-step-1");
await page.waitForURL("http://localhost:5173/steps/2");
await page.click(".right-step-2");
await page.waitForURL("http://localhost:5173/steps/3");
await page.click(".right-step-3");
await page.waitForURL("http://localhost:5173/steps/4");
await page.click(".right-step-4");
await page.waitForURL("http://localhost:5173/steps/5");
await expect(page).toHaveScreenshot(sc_options);
});
test("step 6", async ({ page }) => {
test.setTimeout(180000);
await page.click(".right-step-1");
await page.waitForURL("http://localhost:5173/steps/2");
await page.click(".right-step-2");
await page.waitForURL("http://localhost:5173/steps/3");
await page.click(".right-step-3");
await page.waitForURL("http://localhost:5173/steps/4");
await page.click(".right-step-4");
await page.waitForURL("http://localhost:5173/steps/5");
await page.click(".right-step-5");
await page.waitForURL("http://localhost:5173/steps/6");
// Wait for text "Berechnungs­szenarium" to appear
await page.waitForSelector("text=Sturm", { timeout: 120000 });
await expect(page).toHaveScreenshot({
...sc_options,
maxDiffPixels: 12000,
});
});
});

Gitlab CI

# test with screenshots
playwright-dev:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
allow_failure: true
stage: test
image: mcr.microsoft.com/playwright
script:
- npm install
- npx playwright install --with-deps chromium
- npx playwright test
artifacts:
when: always
paths:
- playwright-report/
- test-results/
expire_in: 30 days
untracked: true

Steps

See Gitlab CI file

Expected

Visual Comparison between last commit (or pipeline run) and current commit to see visual changes between commits.

Screenshot Snapshots are supposed to be accessible in cache between multiple pipeline (job) runs.

Actual

Snapshots don't exist and are only created within the scope of the current run. Because they cannot carry over from the previous run (where snapshots are created on first execution of npx playwright text), the visual comparison tests always fail.

Error: A snapshot doesn't exist at /builds/.../frontends/frontend-new/e2e/screenshots.spec.ts-snapshots/Screenshot-Tests-step-1-1-chromium-linux.png.

Issue 2:
Hi,

I read that if one of the tests in a suite fails - the beforeAll hook is running again (As the worker disposed)

I have a case when my beforeAll hook used for a data preparation using API calls.

In addition, my application has a limitation that I can't create a duplication of an entity.

As mentioned, if one of the tests in a suite failed - beforeAll hook is running again and due to the duplication limitation, the beforeAll hook failed and all tests failed accordingly as well.

I was wonder what could be the best practice in such case?

Currently I implemented it using a fixture which is called only from the first test in the suite, but I feel it's not the best practice.