npm package: @midleman/playwright-reporter
Description: Playwright reporter that writes test results to S3 as Parquet files
Latest version: 0.47.15

# @midleman/playwright-reporter

A Playwright reporter that sends test results to the E2E Test Insights API, plus a CLI for predictive test sharding.

## Installation

```bash
npm install @midleman/playwright-reporter
```

## Quick Start

```typescript
// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
    reporter: [
        ['html'],
        ['@midleman/playwright-reporter', { mode: 'prod' }],
    ],
});
```

Set the `CONNECT_API_KEY` environment variable (required for `mode: 'prod'`).

## Reporter Configuration

### Mode

| Mode       | API Endpoint                                     | Use Case          |
| ---------- | ------------------------------------------------ | ----------------- |
| `dev`      | `http://localhost:8000`                          | Local development |
| `prod`     | `https://connect.posit.it/e2e-test-insights-api` | CI/Production     |
| `disabled` | None                                             | Skip reporting    |

> **Note:** `mode: 'prod'` only works in CI (`GITHUB_ACTIONS=true` or `CI=true`). Running locally with `mode: 'prod'` disables reporting to avoid polluting production data.

### Options

```typescript
['@midleman/playwright-reporter', {
    mode: 'prod',           // Required: 'dev', 'prod', or 'disabled'
    repoName: 'my-repo',    // Optional: defaults to GITHUB_REPOSITORY
    verbose: true,          // Optional: enable debug logging
}]
```

### Environment Variables

| Variable             | Description                              | Required |
| -------------------- | ---------------------------------------- | -------- |
| `CONNECT_API_KEY`    | Posit Connect API key                    | For prod |
| `REPORTER_REPO_NAME` | Repository name override                 | No       |
| `REPORT_URL`         | Base URL for Playwright HTML report links| No       |

---

## Predictive Sharding CLI

Distributes test specs across CI shards using historical duration data for optimal load balancing.

### How It Works

```
┌────────────────────────────────────────────────────────────────────────────┐
│                              CI WORKFLOW                                   │
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  npx e2e-insights run-shard --shard-index=2 --shard-total=6 ...      │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  1. playwright test --list --reporter=json                           │  │
│  │     → Discovers all spec files in project                            │  │
│  │     → ["auth.spec.ts", "editor.spec.ts", "console.spec.ts", ...]     │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  2. POST /shards/compute                                             │  │
│  │     {                                                                │  │
│  │       "specs": ["auth.spec.ts", "editor.spec.ts", ...],              │  │
│  │       "shard_index": 1,                                              │  │
│  │       "num_shards": 6,                                               │  │
│  │       "repo": "posit-dev/positron"                                   │  │
│  │     }                                                                │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │                         E2E INSIGHTS API                            │   │
│  │                                                                     │   │
│  │   ┌─────────────────┐    ┌─────────────────┐    ┌───────────────┐   │   │
│  │   │ Load test data  │ →  │ Filter & group  │ →  │ LPT algorithm │   │   │
│  │   │ (14-day window) │    │ by spec/os/etc  │    │ bin-packing   │   │   │
│  │   └─────────────────┘    └─────────────────┘    └───────────────┘   │   │
│  │                                                          │          │   │
│  │   Response:                                              ▼          │   │
│  │   {                                                                 │   │
│  │     "specs": ["editor.spec.ts", "console.spec.ts"],  ← shard 2      │   │
│  │     "estimated_duration_ms": 180000                                 │   │
│  │   }                                                                 │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  3. playwright test editor.spec.ts console.spec.ts                   │  │
│  │     → Runs ONLY the specs assigned to this shard                     │  │
│  │     → Reporter sends results back to API                             │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘
```

### GitHub Actions Example

```yaml
jobs:
  e2e-tests:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - name: Run E2E Tests
        env:
          CONNECT_API_KEY: ${{ secrets.CONNECT_API_KEY }}
        run: |
          npx e2e-insights run-shard \
            --shard-index=${{ matrix.shard }} \
            --shard-total=4 \
            --repo=${{ github.repository }} \
            --project=e2e-electron
```

### CLI Options

| Option          | Required | Description                                              |
| --------------- | -------- | -------------------------------------------------------- |
| `--shard-index` | Yes      | Shard index (1-based)                                    |
| `--shard-total` | Yes      | Total number of shards                                   |
| `--repo`        | Yes      | Repository identifier (e.g., `posit-dev/positron`)       |
| `--browser`     | No       | Override browser (auto-inferred from `--project`)        |
| `--fallback`    | No       | `native` (default) or `fail` if API unavailable          |

Additional arguments are passed through to Playwright:

```bash
npx e2e-insights run-shard \
  --shard-index=1 --shard-total=4 --repo=org/repo \
  --workers=4 --retries=2 --project=e2e-electron
```

### Auto-Detection

**OS:** Detected from runner platform (`mac`, `win`, `ubuntu`, `linux`).

**Browser:**
- **CLI:** Parses the `--project` flag for keywords (e.g., `e2e-electron` → `electron`). Cannot auto-detect since Playwright hasn't started yet. If no match, `GITHUB_JOB` is used for shard naming.
- **Reporter:** Auto-detects at runtime from Playwright's `project.use.browserName` config, or infers from project name, or defaults to `electron`.

### Shard Naming & Unique Run IDs

When running sharded tests, each shard needs a unique identifier so results don't overwrite each other in S3. The CLI handles this automatically by setting a `SHARD_NAME` environment variable before running Playwright.

**Format:** `{browser}-{os}-{shardIndex}`

**Example:** A 4-shard matrix on Ubuntu with `--project=e2e-electron` produces:
```
shard 1 → SHARD_NAME=electron-ubuntu-1 → runId: 12345-1-electron-ubuntu-1
shard 2 → SHARD_NAME=electron-ubuntu-2 → runId: 12345-1-electron-ubuntu-2
shard 3 → SHARD_NAME=electron-ubuntu-3 → runId: 12345-1-electron-ubuntu-3
shard 4 → SHARD_NAME=electron-ubuntu-4 → runId: 12345-1-electron-ubuntu-4
```

Each shard writes to its own S3 path, preventing collisions.

### Fallback Behavior

If the API is unavailable or has no historical data, the CLI falls back to Playwright's native sharding:

```bash
npx playwright test --shard=1/4
```

### Shard Coordination

When multiple shards run in parallel, they may query the API at slightly different times. To ensure all shards get **consistent test assignments** (no duplicates, no gaps), the API uses the GitHub workflow run ID to coordinate:

```
Shard 1 queries at 11:59  ──┐
Shard 2 queries at 12:00  ──┼──→  Same workflow_run_id  ──→  Same historical data  ──→  Same assignments
Shard 3 queries at 12:01  ──┘
```

**How it works:**

1. Each shard passes its `workflow_run_id` (from `GITHUB_RUN_ID`) to the API
2. The API uses this ID to derive a **deterministic cutoff time** for the historical data query
3. Since all shards have the same workflow_run_id, they all see the same historical data
4. Same data → same LPT algorithm result → consistent, non-overlapping assignments

**Why this matters:**

Without coordination, shards querying at different times might see slightly different historical data (e.g., a new test run completed between queries). This could cause the LPT algorithm to make different decisions for each shard, resulting in some tests being assigned to multiple shards or skipped entirely.

### Comparison with Playwright Native

| Aspect              | Playwright Native   | E2E Insights             |
| ------------------- | ------------------- | ------------------------ |
| Algorithm           | Round-robin by test | LPT by spec file         |
| Balance             | Even test count     | Even duration            |
| Adaptation          | Static              | Uses historical data     |
| Platform awareness  | No                  | Yes (OS/browser filter)  |

---

## Publishing

```bash
npm test
npm run build
npm version patch  # or minor/major
npm publish --access public
git push && git push --tags
```