Issue 1:
System info

Playwright Version: [v1.XX]

Operating System: [macOS 13.2, etc.]

Browser: Chromium]

Other info:

Source code

I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

[https://github.com/your_profile/playwright_issue_title]

or

Config file

// playwright.java
Playwright playwright = Playwright.create();
Browser browser = playwright.chromium().launch(
new BrowserType.LaunchOptions().setHeadless(false)
.setChannel("chrome")
.setDevtools(true)
.setTimeout(300000)
);
BrowserContext browserContext = browser.newContext();
Page page = browserContext.newPage();
page.setDefaultTimeout(300000);
page.setDefaultNavigationTimeout(300000);
page.navigate(url, new Page.NavigateOptions().setTimeout(300000));
page.close();
browserContext.close();
browser.close();
playwright.close();

Steps

[Run the test]

[...]Run the above code in parallel using executor service (10 threads)

Expected

[Successful navigation]

Actual

[WARNING: Timed out while waiting for driver process to exit
com.microsoft.playwright.PlaywrightException: Error {
message='net::ERR_ABORTED at url
=========================== logs ===========================
navigating to "url", waiting until "load"
============================================================
name='Error
stack='Error: net::ERR_ABORTED at url
=========================== logs ===========================
navigating to "url", waiting until "load"
============================================================
at FrameSession._navigate (/private/var/folders/mg/1pyjgqbs4sj89gyr8wy3zkkc0000gq/T/playwright-java-1669077734166046551/package/lib/server/chromium/crPage.js:509:35)
]

Issue 2:
System info

Playwright Version: [v1.34.3]

Operating System: [Windows 10, Ubuntu 22.04]

Browser: [Chromium]

Source code

I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

edumserrano/playwright-bug-vrt-diff

Bug report

The issue I have is that some scenarios with screenshot comparison don't seem to work as expected, or at least as I'd expect. Perhaps some of these issues aren't qualified to be bugs.

I've found issues with unexpected differences in screenshot tests:

running in headless vs not headless mode.

running on Chromium Windows vs Chromium Ubuntu.

running outside debug vs in debug mode.

Please see the details below.

Issues trying to get consistent VRT

Note

All these scenarios were executed from a Windows OS as the base OS.

Scenario 1: Hardware acceleration and headless mode

Go to the root of the repo.

Run the tests via npm run test.

Run the app via npm run start.

Compare the screenshot generated via npm run test at tests/example.spec.ts-snapshots/basic-test-1-chromium-win32.png with how the app running via npm run start looks like in regards to the blur effect.

Screenshot from npm run test

Screenshot from npm run start

Note

My PC has a higher resolution than 1920x1080, I've used the browser tools to try and get close for the manual screenshot I took. Anyways, what's important in this screenshot is looking at how the blur looks correct and how different it is from when running the test via npm run test.

When running the tests with npm run test it seems the chromium browser running in headless mode does not have hardware acceleration enabled, therefore the image produced is not as expected.

You can compare the image produced from the test with how the app looks like when running via npm run start. I believe this is related with hardware acceleration because when running the app via npm run start if you enable/disable hardware accelearation in the browser settings, the blur applied looks different.

Another way to view this is related with hardware acceleration is that if you add:

launchOptions: {
args: ["--disable-gpu"],
},

to the chromium project settings in the playwright.config.ts file, then the blur of screenshot from debug will look equal to when running without debug.

Note

The way the blur is intended to look like is how it looks when hardware acceleration is enabled

The blur effect is being created with:

background-color: transparent;
-webkit-backdrop-filter: blur(50px);
backdrop-filter: blur(50px);

Scenario 2: Chromium screenshot in Windows vs Chromium screenshot in Ubuntu

Go to the root of the repo.

Uncomment the line // ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file.

Run the tests via npm run test and generate the screenshot in Windows.

Duplicate the screenshot basic-test-1-chromium-win32.png at /tests/example.spec.ts-snapshots and rename it to basic-test-1-chromium-linux.png.

Run the tests in Ubuntu via docker by executing docker run -it --rm --ipc=host --workdir=/app -v ${PWD}:/app mcr.microsoft.com/playwright:v1.34.3-jammy npx playwright test.

Open the html report via npx playwright show-report tests/test-results/reporters/html and look at the differences in the screenshots.

The screenshot generated from Ubuntu will have some slight differences regarding the font (looks bolder in windows?). Furthermore, the page scrollbar on Windows is a bit wider than the one in Ubuntu.

Diff

Screenshot from Windows

Screenshot from Ubuntu

Note

The font I'm using on the app is Lato Regular 400 from google fonts.

See the below at index.html:

<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Lato&display=swap" rel="stylesheet">

And CSS at styles.less:

@font-family-lato: 'Lato', sans-serif;

body {
font-family: @font-family-lato;
}

Scenario 3: Differences when running the test in/out of debug mode

This one might not be a bug but it still feels like a bit of an issue.

Install the playwright extension for VSCode.

Go to the root of the repo.

Uncomment the line // ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file.

Generate the test screenshot by running the tests with npm run test.

Go to VS Code and debug the test using the playwright extension for VS Code.

Open the html report via npx playwright show-report tests/test-results/reporters/html and look at the differences in the screenshots.

The debug will fail because there are differences in the screenshots:

the screenshot from debug mode will have will have the appropriate blur. Perhaps it runs with hardware acceleration enabled and therefore shows as expected (equal to what is visible when running the app with npm run start).

the screenshot from debug mode will have slight differences in the font. Or perhaps this is incorrect, perhaps the issue is not with the font but with the fact that the scrollbar width is slightly different and the effect that the difference in blur also has on the fonts.

the screenshot from debug mode seems to have "weaker" coloured red line (the red line below the label Where do you study or work?). This again might just be related with the blur difference.

the screenshot from debug mode has a scrollbar with smaller width.

Diff

Screenshot from running via "npm run test"

Screenshot from running in debug via VS Code playwright extension

Note

To make the comparison easier in debug vs not debug I've added ignoreDefaultArgs: ["--hide-scrollbars"] in the playwright.config.ts file. This makes sure scrollbar is always visible whether running the test in debug mode or not.

However I'd say that this was also unexpected to me even if it's default behavior. Couldn't find any mention to the fact that the scrollbar is disabled in headless mode when searching the playwright docs. Perhaps a chance for improving the docs?

Related

Perhaps related with the effort on #8161