npm package: @magentic-platform/postman-playwright
Description: Migrate Postman collections to a fully typed, runnable Playwright API test monorepo
Latest version: 0.4.0

# PostmanPlaywright

[![npm](https://img.shields.io/npm/v/@magentic-platform/postman-playwright)](https://www.npmjs.com/package/@magentic-platform/postman-playwright)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)

> Point at a folder of Postman collections — get back a fully typed, runnable Playwright API test monorepo. Run again for the next module — it detects existing structure and slots in cleanly.

**CLI-driven · TypeScript-strict · pnpm workspaces · Foundation model agnostic · Open source · MIT**

---

## Quick Start

```bash
npx @magentic-platform/postman-playwright
```

No flags needed on first run. The CLI walks you through setup interactively — choose your foundation model provider, enter credentials, point it at your collections folder:

```
  PostmanPlaywright — Postman → Playwright Migration Orchestrator
  ─────────────────────────────────────────────────────────────────

  No config found. Let's get you set up.

  ? Foundation model provider:
  ❯ Anthropic Claude     (API key)
    OpenAI               (API key — GPT-4o, o1, …)
    Azure OpenAI         (API key + endpoint)
    AWS Bedrock          (IAM role / access key — Claude, Llama, Titan, …)
    Google Vertex AI     (service account — Gemini Pro, …)
    Mistral AI           (API key — mistral-large, codestral, …)
    Ollama (local)       (no key — Llama 3, Mistral, Phi-3, Gemma 2, …)

  ? Postman collections folder: ./postman-collections
  ? Output folder for generated tests: ./playwright-api-tests

  ✓ Config saved to ~/.postman-playwright.json
  ✓ API key saved to ~/.postman-playwright.env
```

Subsequent runs load config automatically — just pick the next module:

```bash
npx @magentic-platform/postman-playwright migrate --module payments
```

---

## End-to-End Workflow

```bash
# ── Step 1: One-time setup ──────────────────────────────────────────────────
npx @magentic-platform/postman-playwright setup
# Choose provider, enter API key, set input/output folders
# Saves: .postman-playwright.json + .env

# ── Step 2: Migrate a module ────────────────────────────────────────────────
npx @magentic-platform/postman-playwright migrate --module transforms
# [1/6] Scaffold monorepo (first run only)
# [2/6] Parse Postman collections + environment files → CollectionIR
#        Auto-detects baseUrl from Postman env → populates .env
# [3/6] Generate TypeScript types (models/)
# [4/6] Generate fixtures + typed API client
# [5/6] LLM → Playwright spec files (tests/transforms/)
#        Negative test detection · Postman status enforcement · example body values
# [6/6] tsc check → auto-fix assertion aliases → LLM repair method mismatches → ESLint

# ── Step 3: Run the generated tests ─────────────────────────────────────────
cd playwright-api-tests
# .env is auto-populated with baseUrl from Postman environment if available
# Only edit .env manually if no Postman environment file was provided
pnpm install
npx playwright install --with-deps chromium
pnpm test:smoke                 # fast feedback — happy path only
pnpm test                       # full suite

# ── Step 4: Validate parity (optional but recommended) ──────────────────────
npx @magentic-platform/postman-playwright validate \
  --module transforms \
  --env    staging \
  --delta  0.05
# Runs Newman against the original collections + Playwright against staging
# Reports pass-rate delta and per-endpoint mismatches

# ── Step 5: Add the next module — additive, never overwrites ─────────────────
npx @magentic-platform/postman-playwright migrate --module auth
npx @magentic-platform/postman-playwright migrate --module payments
```

---

## What Gets Generated

Given a folder of Postman collection JSON files, the tool outputs a **complete, runnable Playwright API test monorepo**:

```
playwright-api-tests/
├── package.json                    ← scripts: test, test:smoke, typecheck, lint
├── playwright.config.ts            ← projects: smoke · regression · contract · security · staging
├── tsconfig.json                   ← strict: true, ESNext + bundler module resolution
├── pnpm-workspace.yaml
├── .env.example                    ← required env vars documented
├── .gitignore
├── global-setup.ts                 ← health check + env var validation on startup
│
├── api/
│   ├── base.api.ts                 ← abstract BaseApi: retry, rate-limit, auth headers
│   └── {module}.api.ts             ← typed methods per request, path params resolved
│
├── config/
│   └── env.ts                      ← lazy env var getters (no crash at import time)
│
├── fixtures/
│   ├── auth.fixture.ts             ← JWT / OAuth2 / API key / Basic / None strategies
│   ├── base.fixture.ts             ← root fixture composing auth + all API clients
│   └── {module}.fixture.ts         ← per-module API client fixture
│
├── utils/
│   └── assertions.ts               ← assertStatus, assertArray, assertHasProperty,
│                                      assertHeaderExists, assertInclusion, assertEnumIncludes
│
├── models/                         ← TypeScript interfaces inferred from response shapes
│
├── tests/
│   └── {module}/
│       ├── list-*.spec.ts          ← @smoke · @contract · @security tags per spec
│       ├── create-*.spec.ts
│       └── poll-*.spec.ts
│
└── .github/
    └── workflows/
        └── ci.yml                  ← smoke suite on every PR, artifact upload
```

---

## CLI Commands

```bash
# Interactive first run — setup + migrate
npx @magentic-platform/postman-playwright

# Migrate a specific module (flags override saved config)
npx @magentic-platform/postman-playwright migrate \
  --input  ./postman-collections \
  --output ./playwright-api-tests \
  --module auth

# Subsequent modules — additive, never overwrites existing
npx @magentic-platform/postman-playwright migrate --module payments
npx @magentic-platform/postman-playwright migrate --module exports --force  # overwrite if exists

# Migrate ALL modules at once (each subdirectory of --input = one module)
npx @magentic-platform/postman-playwright migrate --all --input ./postman-collections

# Migrate + immediately validate parity in one command
npx @magentic-platform/postman-playwright migrate --module auth --validate

# Reconfigure provider (e.g. switch from Anthropic → Azure OpenAI)
npx @magentic-platform/postman-playwright setup

# Show migration status across all modules
npx @magentic-platform/postman-playwright status --output ./playwright-api-tests

# Validate parity — Newman vs Playwright (standalone, run any time after migration)
npx @magentic-platform/postman-playwright validate \
  --module auth \
  --env    staging \
  --delta  0.02     # acceptable pass-rate delta (default: 0.02 = 2%)
```

### Validate output example

```
  Validating auth → staging  (Δ threshold: 2%)

  Newman
    Requests : 8/10 passed
    Failed   : 2

  Playwright
    Tests    : 14/16 passed
    Failed   : 2
    Skipped  : 0

  Parity delta
    3.1%  (threshold: 2%)

  Endpoint mismatches
    GET /auth/me                             ✓ Newman  ✗ Playwright
    POST /auth/refresh                       ✗ Newman  ✓ Playwright

  ✗ Parity check failed — fix failing specs then re-run validate.
```

---

## Supported Foundation Models

All 7 providers are wired and selectable via the setup wizard. API keys are stored in `.env` — never in the config JSON.

| Provider | Setup | Model examples |
|---|---|---|
| **Anthropic Claude** | `ANTHROPIC_API_KEY` | claude-opus-4-6, claude-sonnet-4-6 — optional `baseUrl` for proxy |
| **OpenAI** | `OPENAI_API_KEY` | gpt-4o, o1, o3-mini |
| **OpenAI-compatible endpoint** | `OPENAI_API_KEY` + `baseUrl` | Any hub (Azure AI Foundry, etc.) |
| **Azure OpenAI** | `AZURE_OPENAI_API_KEY` + endpoint + deployment | Any GPT-4 deployment |
| **AWS Bedrock** | IAM credentials + `AWS_REGION` | Claude, Llama 3, Titan |
| **Google Vertex AI** | GCP service account + project | Gemini 1.5 Pro/Flash |
| **Mistral AI** | `MISTRAL_API_KEY` | mistral-large, codestral |
| **Ollama (local)** | No key — `http://localhost:11434/v1` | Llama 3.1, Phi-3, Gemma 2, CodeLlama |

---

## Idempotent Run Model

The tool inspects the output folder before every run and behaves accordingly:

| State | What happens |
|---|---|
| **`empty`** | Scaffolds full monorepo (config, fixtures, utils, CI) then migrates the module |
| **`initialised`** | Reads existing structure, adds only the new module, merges `base.fixture.ts` |
| **`has-module`** | Dry-run by default — prompts to skip or overwrite. `--force` skips the prompt |

This means you can safely run it on every CI push — it will never clobber a module already migrated.

---

## Migration Pipeline

```
Postman JSON files (./postman-collections/<module>/)
  │
  ├─ [1/6] Scaffold         pnpm-workspace · playwright.config · base.api · utils · CI · eslint.config.mjs
  ├─ [2/6] Parse            Postman JSON + *.environment.json → CollectionIR
  │                          (method, path, headers, body, auth strategy,
  │                          pre-request scripts, pm.test() assertions,
  │                          setNextRequest chains, env var usage)
  │                         Reads Postman environment files — auto-sets API_URL in .env
  │                          from baseUrl variable if present (no manual editing needed)
  ├─ [3/6] Type generation  response body shapes → models/*.ts TypeScript interfaces
  ├─ [4/6] Fixture + API    auth strategy → auth.fixture.ts
  │   client generation      CollectionIR requests → api/{module}.api.ts typed methods
  │                          (path params: {{variable}} and :param both resolved)
  │                          GraphQL requests → dedicated query()/mutate() wrappers with embedded operations
  ├─ [5/6] Test generation  Foundation Model → tests/{module}/*.spec.ts
  │                          Positive requests → happy path @smoke + validation @validation + auth @security
  │                          Negative requests (4xx) → detected by name/assertions, tested at expected error status
  │                          Postman status codes extracted and enforced — LLM cannot guess/change them
  │                          Postman example body values injected into prompt (not just types)
  │                          Sibling endpoint schemas available for correct setup/teardown data
  └─ [6/6] Quality checks   tsc --noEmit → structured violations per file + line
                             Auto-fix pass 1: near-miss assertion names → aliases in utils/assertions.ts
                             Auto-fix pass 2: LLM repair — TS2339 method mismatches in specs
                               (reads real api/{module}.api.ts, rewrites broken calls via same adapter)
                             Auto-fix pass 3: tsc re-run to verify repairs
                             ESLint pass (warns, never blocks migration)
                             Per-module parity matrix generated in HTML + JSON reports
```

---

## Running Generated Tests

```bash
cd playwright-api-tests

# 1. Install dependencies
pnpm install
npx playwright install --with-deps chromium

# 2. Configure environment
# .env is created automatically. If your Postman environment file contains a
# baseUrl variable, API_URL is populated automatically — no manual editing needed.
# Falls back to localhost:8080 if no Postman baseUrl is found.
# API_USER / API_PASS are only needed if your API has an auth layer

# 3. Run
pnpm test:smoke        # happy path only — fast feedback
pnpm test              # full suite
pnpm test:contract     # schema/contract tests
pnpm test:security     # auth/401 tests
pnpm test:staging      # against staging URL
pnpm typecheck         # tsc --noEmit — catch spec issues
```

---

## Configuration File

`.postman-playwright.json` is saved in your home directory (`~/.postman-playwright.json`) so it works from any folder. Sensitive values use `${ENV_VAR}` references — the actual keys live in `~/.postman-playwright.env` only.

```json
{
  "provider": "openai",
  "inputDir": "./postman-collections",
  "outputDir": "./playwright-api-tests",
  "version": "0.1.0",
  "credentials": {
    "type": "openai",
    "apiKey": "${OPENAI_API_KEY}",
    "model": "gpt-4o",
    "baseUrl": "https://your-hub.example.com/v1"
  }
}
```

Run `npx @magentic-platform/postman-playwright setup` at any time to switch provider or update credentials.

---

## Known Limitations

| Limitation | Notes |
|---|---|
| **`npx playwright install` not auto-run** | You need to install Playwright browsers manually after migration |
| **Type inference is heuristic-only** | Complex nested response shapes may need manual type refinement |
| **Chained request specs** | `setNextRequest` chains are detected but not yet woven into sequential test specs |

See the full [roadmap](https://github.com/magentic-platform/postman-playright-agent#known-gaps--roadmap) for details.

---

## Contributing

See the [contributing guide](https://github.com/magentic-platform/postman-playright-agent/blob/main/CONTRIBUTING.md) for local development setup, architecture, and versioning.

## License

MIT © [Magentic Platform](https://github.com/magentic-platform)