Issue 1:
While my tests are executing in the correct sequence the HTML reporter is reporting them in the wrong order.

After a little digging I figured out why:

the HTML reporter is sorting by line numbers (highlighted below) but without accounting for the presence of multiple files.

Notice in the illustration that I have circled all the line numbers. All these tests, and a dozen or so more omitted for brevity, are in strictly monotonically increasing order. That is WRONG.

If they were all in the same file, sorting by line number would always be correct. But because they are in different files, the test order is essentially randomized by sorting only by line number.

The correct order for the steps shown is:

k8s > create cluster > kind create cluster
k8s > 1. initialize system > add new system
k8s > 1. initialize system > install system
k8s > 2. verify mutating policies > mutating json patch: Add
k8s > 3. verify validating and mutating policies > validating policy
k8s > 4. verify context aware policies > validating policy

System info

Playwright Version: v1.32

Operating System: macOS 12.6

Browser: Chromium

Other info: NA

Source code

I provided exact source code that allows reproducing the issue locally.

Config file

// playwright.config.ts
export default defineConfig({
testDir: 'tests',
projects: [
{
name: 'chromium',
use: {
...devices['Desktop Chrome'],
}
}
]
})

Test files (self-contained)

Test file: main.spec.js

import firstTest from '.first.js'
import secondTest from '.second.js'

import {test} from '#test'

test.describe('main', () => {
test.describe('should run first', firstTest)
test.describe('should run second', secondTest)
})

Test file: first.js

import {expect, test} from '#test'

// comments simply to push the definition of the test to start at a higher line number
// comment
// comment
// comment
// comment
export default function () {
test('test in first.js', async () => {
expect(5).toBeTruthy()
})
}

Test file: second.js

import {expect, test} from '#test'

export default function () {
test('test in second.js', async () => {
expect(5).toBeTruthy()
})
}

Steps

npx playwright test

npx playwright show-report

Expected

✓ 1 [chromium] › k8s/first.js:9:3 › main › should run first › test in first.js (5ms)
✓ 2 [chromium] › k8s/second.js:4:3 › main › should run second › test in second.js (1ms)

Then in the report view:

Actual

✓ 1 [chromium] › k8s/first.js:9:3 › main › should run first › test in first.js (5ms)
✓ 2 [chromium] › k8s/second.js:4:3 › main › should run second › test in second.js (1ms)

Then in the report view:

Issue 2:
System info

Playwright Version: 1.33.0

Operating System: Mac OS Ventura 13.1

Browser: All

Other info: Updated from 1.29.x Playwright version to 1.33.0

Source code

I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

[https://github.com/your_profile/playwright_issue_title]

or

Config file

import type { PlaywrightTestConfig } from '@playwright/test';
// import { devices } from '@playwright/test';

/**
* Read environment variables from file.
* https://github.com/motdotla/dotenv
*/
// eslint-disable-next-line @typescript-eslint/no-var-requires
require('dotenv').config({ path: './.env.local' });

console.log('BASE_URL', process.env.BASE_URL);
console.log('CI', process.env.CI);
console.log('process.env.BUILD_ENVIRONMENT', process.env.BUILD_ENVIRONMENT);

const config: PlaywrightTestConfig = {
testDir: './tests/functional',
/* ignore SIT (kitchen sink) tests if running against prod env */
testIgnore: process.env.BUILD_ENVIRONMENT === 'prod' ? 'SIT/*' : '',
/* Maximum time one test can run for. */
timeout: 30 * 1000,
expect: {
timeout: 5000
},
/* Run tests in files in parallel */
fullyParallel: true,
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
/* Retry on CI only */
retries: process.env.CI ? 2 : 0,
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: [['html', { open: 'never' }]],
/* Run your local dev server before starting the tests */
webServer: {
command: 'yarn dev',
port: 3000,
reuseExistingServer: true
},
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
/* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */
actionTimeout: 0,
/* Base URL to use in actions like `await page.goto('/')`. */
baseURL: 'http://localhost:3000',
/* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
trace: 'on-first-retry',
screenshot: 'only-on-failure',
httpCredentials: {
username: process.env.BASIC_AUTH_USER as string,
password: process.env.BASIC_AUTH_PASSWORD as string
}
},
/* Configure projects for major browsers */
// TODO: determine appropriate strategy for envs
projects: [
// {
// name: 'chromium',
// use: {
// ...devices['Desktop Chrome']
// }
// },
// {
// name: 'firefox',
// use: {
// ...devices['Desktop Firefox']
// }
// },
// {
// name: 'webkit',
// use: {
// ...devices['Desktop Safari']
// }
// },
// /* Test against mobile viewports. */
// {
// name: 'Mobile Chrome',
// use: {
// ...devices['Pixel 5']
// }
// },
// {
// name: 'Mobile Safari',
// use: {
// ...devices['iPhone 13']
// }
// },
// {
// name: 'Mobile Safari Pro Max',
// use: {
// ...devices['iPhone 13 Pro Max']
// }
// },
// {
// name: 'Mobile Safari ipad portrait',
// use: {
// ...devices['iPad (gen 7)']
// }
// },
// /* Test against branded browsers. */
// {
// name: 'Microsoft Edge',
// use: {
// channel: 'msedge'
// }
// },
{
name: 'Google Chrome',
use: {
channel: 'chrome'
}
}
],
/* Folder for test artifacts such as screenshots, videos, traces, etc. */
outputDir: 'test-results/'
};

export default config;

Test file (self-contained)

import { Locator, Page } from '@playwright/test';

export class NotFoundPage {
readonly page: Page;
readonly pageGridDiv: Locator;
readonly pageHeading: Locator;
readonly pageParagraph: Locator;
readonly pageButton: Locator;

constructor(page: Page) {
this.page = page;
this.pageGridDiv = page.locator('[data-testid="404PageGrid"]');
this.pageHeading = page.getByTestId('404heading');
this.pageParagraph = page.getByTestId('404paragraph');
this.pageButton = page.getByTestId('404returnHome');
}
async goto() {
await this.page.goto('http://localhost:3000/null/');
}
}

//seperate file test file:

import { test, expect } from '@playwright/test';
import { NotFoundPage } from './model/NoFoundPage';

test('Verify 404 page components are visible', async ({ page }) => {
const notFoundPage = new NotFoundPage(page);
await notFoundPage.goto();
await expect(notFoundPage.pageGridDiv).toBeVisible();
await expect(notFoundPage.pageHeading).toBeVisible();
await expect(notFoundPage.pageParagraph).toBeVisible();
});

test('Verify Go back home button return"', async ({ page }) => {
const notFoundPage = new NotFoundPage(page);
await notFoundPage.goto();
const returnHomePromise = page.waitForEvent('requestfinished');
await notFoundPage.pageButton.click();
await expect(page).toHaveURL('http://localhost:3000/');
await returnHomePromise;
});

Steps

[Updated playwright from 1.29.x. to 1.33.0]

[npx playwright test --ui]

Expected

[Expecting UI mode to open and show my tests]

Actual

[UI Mode opens fine, but I can't see any of my tests -> displays "no tests". However if I run the tests using Visual studio Test Explorer (using the play button), they run fine & all expected tests are identified.

Also when I run our command yarn test:functional all tests are run fine as well.

Error in terminal:

➜ website git:(dev) npx playwright test --ui

BASE_URL https://sit.org.au/

CI undefined

process.env.BUILD_ENVIRONMENT sit

BASE_URL https://sit.org.au/

CI undefined

process.env.BUILD_ENVIRONMENT sit

Error: TypeError: Cannot read properties of undefined (reading 'source')

at http://localhost:54477/trace/uiMode.79da54b1.js:1:7954

at Array.map ()

at Y (http://localhost:54477/trace/uiMode.79da54b1.js:1:7924)

at Pe._parseProject (http://localhost:54477/trace/uiMode.79da54b1.js:1:5251)

at Pe._onBegin (http://localhost:54477/trace/uiMode.79da54b1.js:1:2836)

at Pe.dispatch (http://localhost:54477/trace/uiMode.79da54b1.js:1:2003)

at window.dispatch (http://localhost:54477/trace/uiMode.79da54b1.js:3:354)

at eval (eval at evaluate (:197:30), :2:10)

at UtilityScript.evaluate (:199:17)

at UtilityScript. (:1:44)Error: TypeError: Cannot read properties of undefined (reading 'configFile')

at te (http://localhost:54477/trace/uiMode.79da54b1.js:1:10800)

at re (http://localhost:54477/trace/uiMode.79da54b1.js:2:4912)

at X (http://localhost:54477/trace/uiMode.79da54b1.js:2:5004)

at Object.onEnd (http://localhost:54477/trace/uiMode.79da54b1.js:2:5226)

at Pe._onEnd (http://localhost:54477/trace/uiMode.79da54b1.js:1:4856)

at Pe.dispatch (http://localhost:54477/trace/uiMode.79da54b1.js:1:2472)

at window.dispatch (http://localhost:54477/trace/uiMode.79da54b1.js:3:354)

at eval (eval at evaluate (:197:30), :2:10)

at UtilityScript.evaluate (:199:17)

at UtilityScript. (:1:44)]