Issue 1:
System info

Playwright Version: 1.32.3 but also reproduced on 1.30.2

Operating System:Windows

Browser: Chrome

Other info: Node: 18.12.1

Source code

await this.page.waitForLoadState("load", { timeout: 10 * 1000 });
await expect(this.projectCard).toHaveScreenshot({
mask: [
this.projectNameInput,
this.footerRegisterLabel,
this.projectTabNumberInput,
],
maxDiffPixelRatio: 0.01,
timeout: 5 * 1000,

Config file

const config: PlaywrightTestConfig = {
testDir: "./tests",
/* Maximum time one test can run for. */
timeout: 5 * 60 * 1000,
expect: {
/**
* Maximum time expect() should wait for the condition to be met.
* For example in `await expect(locator).toHaveText();`
*/
timeout: 5000,
},
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
// retries: process.env.CI ? 2 : 0,
// /* Opt out of parallel tests on CI. */
// workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: [
["list"],
["html", { outputFolder: "./outputs", open: "never" }],
["junit", { outputFile: "./outputs/results.xml" }],
],
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
/* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */
actionTimeout: 10 * 1000,
headless: true,
screenshot: "only-on-failure",
video: "retain-on-failure",
viewport: { width: 1920, height: 1080 },
/* Base URL to use in actions like `await page.goto('/')`. */
// baseURL: 'http://localhost:3000',

/* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
trace: "retain-on-failure",
},

Expected

I expect to "expect(someLocator).toHaveScreenshot" to retry assertion for fail/success until it timeout is reached.

Actual

Both global timeout (set in config), but also direct set as toHaveScreenshot(timeout: //will be ignored) are ignored as assertion fails < 1000ms

Plase checkout that picture. It is clear that 5000 as timeout has been passed, but also visible that assertion has failed after only 900ms

Issue 2:
System info

Playwright Version: "1.29.2"

Operating System: Linux/AMD64

Browser: Chromium

java.version: '11.0.17'

Selenium: Selenium grid 4 with image version:4.8.1-20230221

Code snippet:

const options: LaunchOptions = {
args: ['--remote-allow-origins=*'],
headless: false,
slowMo: 100,
};

BeforeAll(async () => {
console.log('starting browser');
switch (process.env.BROWSER) {
case 'firefox':
global.browser = await firefox.launch(options);
break;
case 'webkit':
global.browser = await webkit.launch(options);
break;
default:
global.browser = await chromium.launch(options);
}
});

AfterAll(async () => {
await global.browser.close();
console.log('starting browser');
});

Before(async () => {
global.context = await global.browser.newContext({ ignoreHTTPSErrors: true });
global.page = await global.context.newPage();

const git = simpleGit();
const branchName = (await git.branch()).current.trim();
if (branchName !== 'main') {
console.log(`[E2E] Running regression on feature branch "${branchName}"`);
await global.page.goto(global.BASE_URL + '/search/form.uri?display=basic#basic');
await global.page.evaluate('window.microUiOverrides?.clear();');

const microUis = [

"List of microUIS here"
];

await Promise.all(
microUis.map((microUi) => {
global.page.evaluate(`window.microUiOverrides?.useBranch('${microUi}','${branchName}')`);
}),
);
}
});

After(async function (scenario) {
if (scenario.result!.status === Status.FAILED) {
const buffer = await global.page.screenshot({
path: `../../dist/apps/e2e/${scenario.gherkinDocument.uri}-${scenario.pickle.name}.png`,
fullPage: true,
});
await this.attach(buffer, 'image/png');
}
await global.page.close();
await global.context.close();
});

Steps

We are running the playwright testes on selenium grid 4 on EKS and when we run the tests via jenkins we get the below error:

Jenkins error:

......2023-03-22T13:30:54.390Z pw:browser <ws disconnected> wss://<selenium-url>/session/2c5f661ddb68a0a4e3b9dc4afd4ce612/se/cdp code=1005 reason=
F---2023-03-22T13:30:54.407Z pw:browser <selenium> disconnecting from sessionId=2c5f661ddb68a0a4e3b9dc4afd4ce612
----FF----------FF---2023-03-22T13:30:54.515Z pw:browser <error disconnecting from selenium>: Error: fetch failed: server returned code 404. URL: https://<selenium-url>/wd/hub/session/2c5f661ddb68a0a4e3b9dc4afd4ce612
2023-03-22T13:30:54.529Z pw:browser <selenium> disconnected from sessionId=2c5f661ddb68a0a4e3b9dc4afd4ce612
FF---FF---FF---FF---FF---FF--FF--FF------FF------FF---------------FF---------------Fstarting browser
.......2023-03-22T13:30:58.645Z pw:browser <ws disconnecting> wss://<selenium-url>/session/a07f6e717605940d092e0975b64de72c/se/cdp
2023-03-22T13:30:58.683Z pw:browser <ws disconnected> wss://<selenium-url>/session/a07f6e717605940d092e0975b64de72c/se/cdp code=1005 reason=
2023-03-22T13:30:58.691Z pw:browser <selenium> disconnecting from sessionId=a07f6e717605940d092e0975b64de72c
2023-03-22T13:30:58.743Z pw:browser <error disconnecting from selenium>: Error: fetch failed: server returned code 404. URL: https://<selenium-url>/wd/hub/session/a07f6e717605940d092e0975b64de72c
2023-03-22T13:30:58.743Z pw:browser <selenium> disconnected from sessionId=a07f6e717605940d092e0975b64de72c

Before # ../../shared-e2e/support/hooks.ts:26
browser.newContext: Target page, context or browser has been closed
at async World.<anonymous> (/home/jenkins/agent/workspace/e2e-selenium-test-sample/shared-e2e/support/hooks.ts:27:22)

After # ../../shared-e2e/support/hooks.ts:59
page.screenshot: Target page, context or browser has been closed
at World.<anonymous> (/home/jenkins/agent/workspace/e2e-selenium-test-sample/shared-e2e/support/hooks.ts:61:42)

Selenium hub error:

at org.openqa.selenium.remote.http.jdk.JdkHttpClient$5.lambda$send$2(JdkHttpClient.java:222)
at org.openqa.selenium.remote.http.jdk.JdkHttpClient$5.send(JdkHttpClient.java:229)
... 9 more
13:27:06.679 INFO [GridModel.release] - Releasing slot for session id e4fc90b85375366c41f665e96db248fe
13:27:06.679 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: e4fc90b85375366c41f665e96db248fe
13:27:07.258 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 499120264588998c705a4b8d05e1df5f
13:27:07.497 WARN [DefaultChannelPipeline.onUnhandledInboundException] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
org.openqa.selenium.WebDriverException: java.lang.IllegalArgumentException: statusCode
Build info: version: '4.8.1', revision: '8ebccac989'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.228-132.418.amzn2.x86_64', java.version: '11.0.17'
Driver info: driver.version: unknown
at org.openqa.selenium.remote.http.jdk.JdkHttpClient$5.send(JdkHttpClient.java:239)
at org.openqa.selenium.netty.server.WebSocketMessageHandler.lambda$channelRead0$0(WebSocketMessageHandler.java:47)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: statusCode
at java.net.http/jdk.internal.net.http.websocket.WebSocketImpl.sendClose(WebSocketImpl.java:301)
at org.openqa.selenium.remote.http.jdk.JdkHttpClient$5.lambda$send$2(JdkHttpClient.java:222)
at org.openqa.selenium.remote.http.jdk.JdkHttpClient$5.send(JdkHttpClient.java:229)
... 9 more

Expected

All tests should run without above error.

Actual

Failing with above mentioned error

NOTE

If we run the exact same suite on selenium grid running on EC2 instance, it works fine.

Solutions tried

We have already implemented suggestions from here SeleniumHQ/selenium#11408 (comment)

Tried by increasing session timeout value for sessionqueue and chrome node.

Tried with selenium hub: 4.5.0 as well.