npm package: @sam2insane/validate-qa-runner
Description: validate.qa local test runner — execute Playwright tests on your machine, connected to validate.qa cloud.
Latest version: 0.1.9

# @worldofz/validate.qa

Local test runner for the **[validate.qa](https://validate.qa)** AI testing platform.

This package enables you to execute Playwright and Appium tests generated from your voice narrations directly on your local machine, securely connecting to your validate.qa cloud project.

**Features:**
- 🎭 **Web Testing** — Execute Playwright tests in headless Chromium
- 📱 **Mobile Testing** — Execute Appium tests on iOS Simulator, Android Emulator, AND real devices via USB
- 🔧 **Doctor Command** — Check system prerequisites for web + mobile testing
- 🔐 **Secure Authentication** — Project-scoped runner tokens
- 🌐 **Tunnel Support** — Connect HTTPS dashboards to local devices with token-based auth

---

## Quick Start

### 1. Install (Global or NPX)

You don't need to install it globally if you use `npx`, but it's completely supported:

```bash
npm install -g @worldofz/validate.qa
```

### 2. Login

Generate a runner token from your validate.qa **Project Settings → Local Runner**. Your credentials will be saved securely to `~/.validate.qa/credentials.json`.

```bash
npx @worldofz/validate.qa login --token urt_xxxxxxxxxxxx --server https://your-server.com
```

### 3. Start the Runner

Starts the runner daemon, polling for pending test runs from the cloud and executing them locally:

```bash
npx @worldofz/validate.qa start
```

---

## Commands

| Command | Description |
|---------|-------------|
| `login` | Saves credentials for future test executions |
| `start` | Starts the local runner (uses saved credentials or CLI flags) |
| `status` | Pings the server to verify your runner token and connection |
| `logout` | Clears all saved runner credentials from your machine |
| `doctor` | Checks system prerequisites for web and mobile testing |

### Command Options

#### `login`
```bash
npx @worldofz/validate.qa login \
  --token <token> \        # Runner token (urt_...) from dashboard
  --server <url> \         # validate.qa server URL
  --project <id>           # Optional: Project ID (embedded in urt_ tokens)
```

#### `start`
```bash
npx @worldofz/validate.qa start \
  --token <token> \        # Override saved token
  --server <url> \         # Override saved server URL
  --project <id> \         # Override saved project ID
  --mobile \               # Enable mobile testing (Appium + bridge)
  --tunnel \               # Enable ngrok tunnel for HTTPS dashboard access (with token auth)
  --mode <mode> \          # Execution mode: playwright-native (default)
  --poll-interval <ms>     # Poll interval: 2000 (default)
```

#### `status`
```bash
npx @worldofz/validate.qa status
# Shows: Server URL, Project ID, Token (masked), Connection status
```

#### `doctor`
```bash
npx @worldofz/validate.qa doctor
# Checks: Node.js, Playwright, Xcode, Android SDK, Appium
```

---

## 📱 Mobile Testing

### Overview

The runner supports native mobile app testing on:
- **iOS:** Simulators (macOS) + Real devices via USB
- **Android:** Emulators + Real devices via USB

Tests are executed using **Appium 2** with XCUITest (iOS) and UiAutomator2 (Android) drivers.

### How It Works

```
┌─────────────────────────────────────────────────────────────┐
│ 1. START RUNNER WITH MOBILE                                 │
│    npx @worldofz/validate.qa start --mobile                 │
│    → Starts Appium server (port 4723)                       │
│    → Starts mobile bridge (port 9515)                       │
│    → Discovers available devices                            │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 2. RECORD FROM DASHBOARD                                    │
│    - Open validate.qa dashboard                             │
│    - Start Recording → Select iOS/Android Native            │
│    - Interact with mirrored device screen                   │
│    - Actions captured with screenshots + UI trees           │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 3. GENERATE TESTS                                           │
│    - Stop recording                                         │
│    - Click "Generate from Recording"                        │
│    - AI creates MobileTestStep[] array                      │
│    - Test saved with framework: 'APPIUM'                    │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 4. EXECUTE TESTS                                            │
│    - Click "Run" in dashboard (or queue from API)           │
│    - Runner claims mobile test run                          │
│    → Boots simulator/emulator                               │
│    → Launches app via Appium                                │
│    → Executes each step (tap, type, swipe, assert)          │
│    → Captures screenshots                                   │
│    → Reports results back to server                         │
└─────────────────────────────────────────────────────────────┘
```

### Prerequisites

#### macOS (Required for iOS)
- **Xcode** — Install from Mac App Store
- **Xcode Command Line Tools**: `xcode-select --install`
- **iOS Simulator** — Included with Xcode

#### Android (Optional)
- **Android Studio** — https://developer.android.com/studio
- **Android SDK** — Installed via Android Studio
- **Environment variables**: `ANDROID_HOME`, `ANDROID_SDK_ROOT`
- **Emulator** — Create via Android Studio → Device Manager

#### Node.js
- **Version**: Node 18 or higher
- **npm**: Latest stable version

### Setup Steps

#### 1. Check Prerequisites
```bash
npx @worldofz/validate.qa doctor
```

Expected output:
```
✓  Node.js v20.x.x
✓  Playwright installed
✓  Xcode 15.x (iOS SDK)
✓  Android SDK (API 34)
✓  Appium 2.x available
```

#### 2. Boot a Simulator/Emulator

**iOS:**
```bash
# List available simulators
xcrun simctl list devices

# Boot a specific simulator
xcrun simctl boot "iPhone 15"

# Or open Simulator.app and boot from menu
open -a Simulator
```

**Android:**
```bash
# List available emulators
emulator -list-avds

# Start an emulator
emulator -avd Pixel_7_API_34

# Or use Android Studio → Device Manager → Start
```

#### 3. Install Your App

**iOS:**
```bash
# Install .app or .ipa on simulator
xcrun simctl install booted /path/to/YourApp.app
```

**Android:**
```bash
# Install APK on emulator
adb install /path/to/YourApp.apk

# Or drag-and-drop APK onto emulator window
```

#### 4. Start Runner with Mobile

**For local dashboard (http://localhost):**
```bash
npx @worldofz/validate.qa start --mobile
```

**For production dashboard (HTTPS):**
```bash
npx @worldofz/validate.qa start --mobile --tunnel
```

Expected output (with tunnel):
```
🎙️  validate.qa Local Runner
─────────────────────────────────────
ℹ  Server:   https://your-server.com
ℹ  Project:  your-project-id
ℹ  Mode:     🎭 Playwright Native + 📱 Mobile 🌐 Tunnel
ℹ  Interval: 2000ms
✓  Browser:  Chrome with user profile
ℹ  📱 Mobile: iOS ✓ | Android ✓ | 2 device(s)
✓  📱 Appium server started
🌐 Public tunnel started: https://abc123.ngrok.io
🔐 Tunnel token: a7f3b2c8d9e1f4a5b6c7d8e9f0a1b2c3d4e5f6a7
   Add this token to your dashboard Project Settings

✓  Runner started — waiting for test runs... (web + mobile)
```

> **Note:** The tunnel token is automatically sent to your dashboard via the server API. You don't need to manually configure it.

#### 5. Record Your First Mobile Test

1. **Open Dashboard** → Your Project
2. **Click "Start Recording"** → Select **"iOS Native"** or **"Android Native"**
3. **Enter App ID**:
   - iOS: Bundle ID (e.g., `com.example.myapp`)
   - Android: Package name (e.g., `com.example.myapp`)
4. **Select Device** from the dropdown (detected automatically)
5. **Click "Start"** — Device screen mirrors in your browser
6. **Interact**:
   - Click on screen → Taps recorded
   - Use swipe buttons → Swipes recorded
   - Click "Type Text" → Keyboard input
   - Back/Home buttons → Navigation
7. **Click "Stop Recording"** when done
8. **Generate Tests** → Click "Generate from Recording"

#### 6. Run Mobile Tests

**Note:** The "Run Test" button in the UI is currently disabled for mobile tests (shows "Run Pending"). Tests execute automatically when:

1. You queue a run from the dashboard
2. The runner is online with `--mobile` flag
3. A matching device is available

The runner will:
- Claim the mobile test run from the queue
- Launch the app on the target device
- Execute each step via Appium
- Capture screenshots after each step
- Report pass/fail results

---

## Runner Modes

### Web Testing (Default)
```bash
npx @worldofz/validate.qa start
# or explicitly:
npx @worldofz/validate.qa start --mode playwright-native
```

Executes Playwright tests in headless Chromium with:
- Persistent browser profiles (cookies/logins persist)
- Test variable injection
- AI healing support (optional)

### Mobile Testing
```bash
npx @worldofz/validate.qa start --mobile
```

Enables:
- Appium 2 server (port 4723)
- Mobile bridge server (port 9515)
- Device discovery and capabilities reporting
- iOS Simulator + Android Emulator support

### Execution Flow Comparison

| Feature | Web Tests | Mobile Tests |
|---------|-----------|--------------|
| Framework | Playwright | Appium 2 |
| Browser/Device | Headless Chromium | iOS Simulator / Android Emulator |
| Locator Strategy | CSS selectors, XPath | Accessibility ID, XPath, Resource ID |
| Actions | click, fill, navigate | tap, type, swipe, back, home |
| Screenshots | On failure | After each step |
| AI Healing | ✅ Supported | ❌ Not supported |
| Cloud Runner | ✅ Supported | ❌ Local only |

---

## Configuration

### Saved Credentials

Credentials are stored in `~/.validate.qa/credentials.json`:

```json
{
  "token": "urt_xxxxxxxxxxxx",
  "serverUrl": "https://your-server.com",
  "projectId": "your-project-id"
}
```

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `SERVER_URL` | validate.qa server URL | — |
| `AUTH_TOKEN` | Runner token | — |
| `PROJECT_ID` | Project ID | — |
| `POLL_INTERVAL_MS` | Poll interval | `2000` |
| `RUNS_DIR` | Directory for test run artifacts | `~/.validate.qa/runs` |
| `CHROME_USER_DATA_DIR` | Chrome profile path | Auto-detected |

### Example: Custom Configuration

```bash
# Run with environment variables
SERVER_URL=https://your-server.com \
AUTH_TOKEN=urt_xxxx \
POLL_INTERVAL_MS=5000 \
npx @worldofz/validate.qa start --mobile
```

---

## Troubleshooting

### "No devices found"
```bash
# iOS: Boot a simulator
xcrun simctl boot "iPhone 15"

# Android: Start an emulator
emulator -avd Pixel_7_API_34

# Then restart runner with --mobile
```

### "App not installed"
```bash
# iOS
xcrun simctl install booted /path/to/App.app

# Android
adb install /path/to/App.apk
```

### "Bridge not responding"
```bash
# Check if bridge is running
curl http://127.0.0.1:9515/bridge/health

# Restart runner with --mobile flag
```

### "Appium failed to start"
```bash
# Check Appium installation
appium --version

# Reinstall Appium drivers
appium driver install xcuitest
appium driver install uiautomator2

# Run doctor to diagnose
npx @worldofz/validate.qa doctor
```

### "Tests stay PENDING"
```bash
# 1. Verify runner is online
npx @worldofz/validate.qa status

# 2. Check runner is started with --mobile
npx @worldofz/validate.qa start --mobile

# 3. Verify device is booted
xcrun simctl list devices  # iOS
adb devices                # Android
```

---

## Security

### Runner Token Security

`urt_` runner tokens are:
- **Project-scoped:** They cannot access sessions, user accounts, or cross-project data.
- **Revocable:** Instantly cancel tokens from the dashboard if compromised.
- **Client-only:** Your local browser/device instances run locally and post test results back.
- **No code upload:** Tests execute on your machine — source code never leaves your infrastructure.

### Mobile Tunnel Security

When using `--tunnel` for mobile testing from an HTTPS dashboard:

**Security Features:**
- **Token-based authentication:** Each tunnel session generates a random 64-character hex token
- **Per-session tokens:** Tokens are regenerated every time you restart the runner
- **Header validation:** All bridge requests require `X-Bridge-Token` header or `tunnelToken` query param
- **Health endpoint public:** Only `/bridge/health` is accessible without auth (for monitoring)
- **All other endpoints protected:** Device control, screenshots, and actions require valid token

**How It Works:**
```
1. Runner starts with --tunnel
   → Generates random token (e.g., a7f3b2c8d9e1f4a5...)
   → Starts ngrok tunnel to your local bridge
   → Displays token in console

2. Dashboard fetches tunnel URL + token from server API
   → Server receives tunnel info via runner heartbeat
   → Dashboard stores token in memory (never persisted)

3. All bridge requests include token:
   → HTTP header: X-Bridge-Token: a7f3b2c8d9e1f4a5...
   → Or query param: ?tunnelToken=a7f3b2c8d9e1f4a5...

4. Bridge validates token on every request:
   → Invalid/missing token → 401 Unauthorized
   → Valid token → Request processed
```

**Best Practices:**
- ✅ **Safe for production use:** Token is required for all device control operations
- ✅ **Token never exposed to client:** Dashboard fetches from server API, not console
- ✅ **Session-limited:** Token changes each time runner restarts
- ⚠️ **Don't share console output:** The full token is shown in runner console
- ⚠️ **Use HTTPS dashboard:** Prevents man-in-the-middle attacks on token transmission

**For Enterprise Deployments:**
Consider using a paid ngrok account for:
- Fixed domain URLs (no changing URLs)
- IP whitelisting
- Request inspection
- Higher bandwidth limits

---

## Architecture

```
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Dashboard     │────▶│   Server API     │◀────│   Local Runner  │
│  (validate.qa)  │     │  (your-infra)    │     │  (your-machine) │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                │                        │
                                │                        ├──→ Playwright (web)
                                │                        └──→ Appium (mobile)
                                │                               │
                                │                               ├──→ iOS Simulator
                                │                               └──→ Android Emulator
                                ▼
                        ┌──────────────────┐
                        │   PostgreSQL     │
                        │   (test metadata)│
                        └──────────────────┘
```

---

## Version

**Current:** 0.1.7

**Changelog:**
- **0.1.7** — Fixed ngrok tunnel support for mobile testing (uses system ngrok binary)
- **0.1.6** — Mobile testing support (Appium 2, device discovery, bridge server)
- **0.1.5** — Initial public release

---

## Support

- **Documentation:** https://docs.validate.qa
- **npm Package:** https://www.npmjs.com/package/@worldofz/validate.qa
- **GitHub:** https://github.com/your-org/validate.qa
- **Discord:** [Join community](https://discord.gg/your-invite)

---

*Generated by validate.qa*