Issue 1:
System info

Playwright Version: [v1.36.1]

Operating System: [Linux 6.4.2-3-MANJARO]

Browser: [All]

Other info:

Source code

I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

[https://github.com/deepanchal/playwright-auth-setup-tests-bug]

Steps

Clone the repo

cd playwright-auth-setup-tests-bug

npm install

npx playwright install

npm run test:e2e - Runs playwright test - Everything passes in headless mode

npm run test:e2e:headed - Runs playwright test --headed - Browser opens and then crashes when running auth setup tests and skips all other tests

Logs

Output from DEBUG='pw:api,pw:browser*' pnpm test:e2e:headed

Running 4 tests using 2 workers
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:api => selectors.setTestIdAttribute started +0ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:api => selectors.setTestIdAttribute started +0ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:api => browserType.launch started +4ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:api => browserType.launch started +4ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:api <= selectors.setTestIdAttribute succeeded +6ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:api <= selectors.setTestIdAttribute succeeded +6ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser <launching> /home/deep/.cache/ms-playwright/chromium-1071/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-5I2MQ2 --remote-debugging-pipe --no-startup-window +0ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser <launching> /home/deep/.cache/ms-playwright/chromium-1071/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-G9J9QP --remote-debugging-pipe --no-startup-window +0ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser <launched> pid=794121 +5ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser <launched> pid=794122 +3ms
pw:browser [pid=794122][err] Gtk-Message: 14:09:41.482: Failed to load module "appmenu-gtk-module" +96ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] Gtk-Message: 14:09:41.482: Failed to load module "appmenu-gtk-module" +97ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] [794122:794122:0718/140941.552875:ERROR:chrome_browser_cloud_management_controller.cc(162)] Cloud management controller initialization aborted as CBCM is not enabled. +70ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] [794121:794121:0718/140941.579003:ERROR:chrome_browser_cloud_management_controller.cc(162)] Cloud management controller initialization aborted as CBCM is not enabled. +96ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:api <= browserType.launch succeeded +272ms
pw:api => browser.newContext started +4ms
pw:api <= browser.newContext succeeded +11ms
pw:api => browserContext.newPage started +6ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:api <= browserType.launch succeeded +295ms
pw:api => browser.newContext started +4ms
pw:api <= browser.newContext succeeded +8ms
pw:api => browserContext.newPage started +5ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] [0718/140941.651295:ERROR:elf_dynamic_array_reader.h(64)] tag not found +98ms
pw:browser [pid=794122][err] [0718/140941.651647:ERROR:elf_dynamic_array_reader.h(64)] tag not found +0ms
pw:browser [pid=794122][err] [0718/140941.651706:ERROR:elf_dynamic_array_reader.h(64)] tag not found +0ms
pw:browser [pid=794122][err] [0718/140941.653636:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2) +2ms
pw:browser [pid=794122][err] [0718/140941.653652:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2) +0ms
pw:browser [pid=794122][err] Received signal 11 SEGV_MAPERR 000000000148 +5ms
pw:browser [pid=794122][err] #0 0x55d34825d302 base::debug::CollectStackTrace() +5ms
pw:browser [pid=794122][err] #1 0x55d34824a753 base::debug::StackTrace::StackTrace() +4ms
pw:browser [pid=794122][err] #2 0x55d34825cd91 base::debug::(anonymous namespace)::StackDumpSignalHandler() +3ms
pw:browser [pid=794122][err] #3 0x7f3c5b4bfab0 (/usr/lib/libc.so.6+0x39aaf) +1ms
pw:browser [pid=794122][err] #4 0x55d343fc9670 base::ObserverList<>::AddObserver() +0ms
pw:browser [pid=794122][err] #5 0x55d34da99375 AvatarMenu::AvatarMenu() +14ms
pw:browser [pid=794122][err] #6 0x55d34da95dff DbusAppmenu::Initialize() +12ms
pw:browser [pid=794122][err] #7 0x55d34da98296 DbusAppmenuRegistrar::InitializeMenu() +10ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] [0718/140941.708612:ERROR:elf_dynamic_array_reader.h(64)] tag not found +129ms
pw:browser [pid=794121][err] [0718/140941.709143:ERROR:elf_dynamic_array_reader.h(64)] tag not found +1ms
pw:browser [pid=794121][err] [0718/140941.709227:ERROR:elf_dynamic_array_reader.h(64)] tag not found +0ms
pw:browser [pid=794121][err] [0718/140941.711820:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2) +2ms
pw:browser [pid=794121][err] [0718/140941.711844:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2) +1ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #8 0x55d34da98aa3 DbusAppmenuRegistrar::OnNameOwnerChanged() +9ms
pw:browser [pid=794122][err] #9 0x55d343fbc4f1 base::internal::Invoker<>::Run() +0ms
pw:browser [pid=794122][err] #10 0x55d3447b0911 base::internal::Invoker<>::RunOnce() +1ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] Received signal 11 SEGV_MAPERR 000000000148 +5ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #11 0x55d3481ed1c4 base::TaskAnnotator::RunTaskImpl() +4ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #0 0x55b529df1302 base::debug::CollectStackTrace() +6ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #12 0x55d34820752f base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() +4ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #1 0x55b529dde753 base::debug::StackTrace::StackTrace() +4ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #13 0x55d348206fa9 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() +4ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #2 0x55b529df0d91 base::debug::(anonymous namespace)::StackDumpSignalHandler() +4ms
pw:browser [pid=794121][err] #3 0x7fe52acf1ab0 (/usr/lib/libc.so.6+0x39aaf) +0ms
pw:browser [pid=794121][err] #4 0x55b525b5d670 base::ObserverList<>::AddObserver() +0ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #14 0x55d348207a05 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() +4ms
pw:browser [pid=794122][err] #15 0x55d348270e14 base::MessagePumpGlib::Run() +4ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #5 0x55b52f62d375 AvatarMenu::AvatarMenu() +10ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #16 0x55d348207d79 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() +6ms
pw:browser [pid=794122][err] #17 0x55d3481cde66 base::RunLoop::Run() +5ms
pw:browser [pid=794122][err] #18 0x55d3461b0fa3 content::BrowserMainLoop::RunMainMessageLoop() +2ms
pw:browser [pid=794122][err] #19 0x55d3461b2aa2 content::BrowserMainRunnerImpl::Run() +2ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #6 0x55b52f629dff DbusAppmenu::Initialize() +12ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #20 0x55d3461ae616 content::BrowserMain() +2ms
pw:browser [pid=794122][err] #21 0x55d347653088 content::RunBrowserProcessMain() +3ms
pw:browser [pid=794122][err] #22 0x55d3476549e4 content::ContentMainRunnerImpl::RunBrowser() +3ms
pw:browser [pid=794122][err] #23 0x55d3476547d6 content::ContentMainRunnerImpl::Run() +3ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #7 0x55b52f62c296 DbusAppmenuRegistrar::InitializeMenu() +11ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #24 0x55d347651d90 content::RunContentProcess() +4ms
pw:browser [pid=794122][err] #25 0x55d347651fbd content::ContentMain() +3ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #8 0x55b52f62caa3 DbusAppmenuRegistrar::OnNameOwnerChanged() +13ms
pw:browser [pid=794121][err] #9 0x55b525b504f1 base::internal::Invoker<>::Run() +0ms
pw:browser [pid=794121][err] #10 0x55b526344911 base::internal::Invoker<>::RunOnce() +0ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #26 0x55d343e1a309 ChromeMain +11ms
pw:browser [pid=794122][err] #27 0x7f3c5b4a9850 (/usr/lib/libc.so.6+0x2384f) +1ms
pw:browser [pid=794122][err] #28 0x7f3c5b4a990a __libc_start_main +0ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #11 0x55b529d811c4 base::TaskAnnotator::RunTaskImpl() +6ms
pw:browser [pid=794121][err] #12 0x55b529d9b52f base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl() +5ms
pw:browser [pid=794121][err] #13 0x55b529d9afa9 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() +4ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] #29 0x55d343e1a02a _start +11ms
pw:browser [pid=794122][err] r8: 0000000000000002 r9: 0000000000000000 r10: 000055d34ec43b80 r11: 0000000000000040 +0ms
pw:browser [pid=794122][err] r12: 000055d34eb8a000 r13: 00003a4800f32d40 r14: 00003a4800f32d00 r15: 00003a4800f32d18 +0ms
pw:browser [pid=794122][err] di: 0000000000000148 si: 00003a4800f32d00 bp: 00007ffe29b90130 bx: 0000000000000148 +0ms
pw:browser [pid=794122][err] dx: 000055d341f38b20 ax: 0000000000000000 cx: 000000004e593401 sp: 00007ffe29b900e0 +0ms
pw:browser [pid=794122][err] ip: 000055d343fc9670 efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000004 +0ms
pw:browser [pid=794122][err] trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000148 +0ms
pw:browser [pid=794122][err] [end of stack trace] +0ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121][err] #14 0x55b529d9ba05 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() +5ms
pw:browser [pid=794121][err] #15 0x55b529e04e14 base::MessagePumpGlib::Run() +4ms
pw:browser [pid=794121][err] #16 0x55b529d9bd79 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run() +4ms
pw:browser [pid=794121][err] #17 0x55b529d61e66 base::RunLoop::Run() +4ms
pw:browser [pid=794121][err] #18 0x55b527d44fa3 content::BrowserMainLoop::RunMainMessageLoop() +2ms
pw:browser [pid=794121][err] #19 0x55b527d46aa2 content::BrowserMainRunnerImpl::Run() +2ms
pw:browser [pid=794121][err] #20 0x55b527d42616 content::BrowserMain() +2ms
pw:browser [pid=794121][err] #21 0x55b5291e7088 content::RunBrowserProcessMain() +3ms
pw:browser [pid=794121][err] #22 0x55b5291e89e4 content::ContentMainRunnerImpl::RunBrowser() +3ms
pw:browser [pid=794121][err] #23 0x55b5291e87d6 content::ContentMainRunnerImpl::Run() +3ms
pw:browser [pid=794121][err] #24 0x55b5291e5d90 content::RunContentProcess() +4ms
pw:browser [pid=794121][err] #25 0x55b5291e5fbd content::ContentMain() +3ms
pw:browser [pid=794121][err] #26 0x55b5259ae309 ChromeMain +10ms
pw:browser [pid=794121][err] #27 0x7fe52acdb850 (/usr/lib/libc.so.6+0x2384f) +0ms
pw:browser [pid=794121][err] #28 0x7fe52acdb90a __libc_start_main +0ms
pw:browser [pid=794121][err] #29 0x55b5259ae02a _start +10ms
pw:browser [pid=794121][err] r8: 0000000000000002 r9: 0000000000000000 r10: 000055b5307d7b80 r11: 00007fe529b6e988 +0ms
pw:browser [pid=794121][err] r12: 000055b53071e000 r13: 000007fc00f4e200 r14: 000007fc00f4e1c0 r15: 000007fc00f4e1d8 +0ms
pw:browser [pid=794121][err] di: 0000000000000148 si: 000007fc00f4e1c0 bp: 00007ffc13df7700 bx: 0000000000000148 +0ms
pw:browser [pid=794121][err] dx: 000055b523accb20 ax: 0000000000000000 cx: 0000000030127401 sp: 00007ffc13df76b0 +0ms
pw:browser [pid=794121][err] ip: 000055b525b5d670 efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000004 +0ms
pw:browser [pid=794121][err] trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000148 +1ms
pw:browser [pid=794121][err] [end of stack trace] +0ms
pw:browser [pid=794121][err] libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) +41ms
[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122][err] libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null) +103ms
[setup] › auth.setup.ts:10:6 › authenticate as admin
pw:browser [pid=794121] <process did exit: exitCode=null, signal=SIGSEGV> +465ms
pw:browser [pid=794121] starting temporary directories cleanup +0ms
pw:api <= browserContext.newPage failed +737ms
1) [setup] › auth.setup.ts:10:6 › authenticate as admin ──────────────────

Error: Page closed

[setup] › auth.setup.ts:3:6 › authenticate as user
pw:browser [pid=794122] <process did exit: exitCode=null, signal=SIGSEGV> +489ms
pw:browser [pid=794122] starting temporary directories cleanup +0ms
pw:api <= browserContext.newPage failed +781ms
2) [setup] › auth.setup.ts:3:6 › authenticate as user ────────────────────────────────────────────

Error: Page closed

2 failed
[setup] › auth.setup.ts:3:6 › authenticate as user ─────────────────────────────────────────────
[setup] › auth.setup.ts:10:6 › authenticate as admin ───────────────────────────────────────────
2 skipped

To open last HTML report run:

npx playwright show-report

ELIFECYCLE  Command failed with exit code 1.

Expected

Auth setup tests should run in headed mode and not skip all other tests.

Actual

Auth setup tests fails and skips other tests in headed mode. Currently, it also prevents tester from running tests in debug mode as it tries to open browser for auth setup tests.

Issue 2:
Note: this is ultimately a problem between chair and keyboard but I'm filing an issue as it might be worth considering to make Playwright more robust against the problem described here. If not, maybe it'll at least help someone else who runs into this in the future.

Context:

Playwright Version: 1.15.0

Operating System: Windows

Node.js version: 16.10.0 (was also able to repro in 14.17.6 LTS)

Browser: Chromium / All

Code Snippet

Repo: https://github.com/fkleuver/playwright-test-repro

(although it should be reproducible with any setup)

Simplest steps to repro:

cd C:\projects
git clone git@github.com:fkleuver/playwright-test-repro.git
cd playwright-test-repro
npm ci
npm run --prefix ../Playwright-test-repro e2e
# ^ note the capital P

The above repro steps fail with the following error:

Error: test\w3.spec.js:3:1: test() can only be called in a test file
at errorWithLocation (C:\projects\Playwright-test-repro\node_modules\@playwright\test\lib\test\util.js:195:10)
at TestTypeImpl._createTest (C:\projects\Playwright-test-repro\node_modules\@playwright\test\lib\test\testType.js:78:51)
at C:\projects\Playwright-test-repro\node_modules\@playwright\test\lib\test\transform.js:133:12
at file:///C:/projects/Playwright-test-repro/test/w3.spec.js:3:1
at ModuleJob.run (node:internal/modules/esm/module_job:183:25)
at Loader.import (node:internal/modules/esm/loader:178:24)
at importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
at Loader._requireOrImport (C:\projects\playwright-test-repro\node_modules\@playwright\test\lib\test\loader.js:214:52)
at Loader.loadTestFile (C:\projects\playwright-test-repro\node_modules\@playwright\test\lib\test\loader.js:130:7)
at Runner._run (C:\projects\playwright-test-repro\node_modules\@playwright\test\lib\test\runner.js:219:40)

The repro steps may seem a bit contrived but it's simply the easiest way to reproduce if you folks want to debug this.

Describe the bug

The "culprit" is module-scoped variables in Playwright which are used as a cache.

File: src/test/globals.ts (fns setCurrentlyLoadingFileSuite / currentlyLoadingFileSuite)

Apparently when the entry point is invoked with the wrong casing, two of those modules end up being loaded. setCurrentlyLoadingFileSuite is called on one instance of the module whereas currentlyLoadingFileSuite is called on another instance (where the backing currentFileSuite variable is always undefined)

How I ran into this: I had a shortcut to open cmd.exe in C:\Projects\my-project (the actual folder is C:\projects\my-project). Typo, my bad, but it worked fine until I switched from mocha to the playwright test runner.

Since the error itself doesn't really help (nor does googling for it), it took a while to find out the root cause.

Imagine the fun:

run tests: fails

try a bunch of things, up to eventually cd .. && rmdir my-project /s /q followed by a fresh git clone & install

run tests: succeeds

open another command prompt to the same folder, run tests: fails (?!)

I don't know the inner workings and constraints of Playwright well enough to confidently suggest any particular solution, but at least detecting duplicate module instantiation to throw a more helpful error in this scenario would seem like good form.

Something like this at the top of globals.ts would accomplish just that:

const s = Symbol.for('__playwright');
if (globalThis[s]) {
throw new Error(`more than one instance of globals.js was loaded`);
} else {
Reflect.defineProperty(globalThis, s, { value: true, configurable: true });
}

Then the error will directly point to this specific issue if it's caused by duplicate module instantiation, rather than potentially send the user on a trail to nowhere :-)