npm package: @sdet-agents/test-health-cli
Description: Analyze Playwright test suite health - SDET Agent Suite #1
Author: SDET Agent Suite
Latest version: 0.4.4

# Test Health CLI

**SDET Agent Suite - Agent #1**

Analyze Playwright test suite health and get actionable recommendations to improve test reliability, reduce flakiness, and optimize test maintenance.

## Features

### Two-Pillar Architecture

The Test Health CLI analyzes your test suite across **two pillars**:

1. **📐 Code Quality** - General coding best practices (6 analyzers)
2. **🧪 Test Quality** - Test automation best practices (8 analyzers)

### Code Quality Pillar (6 Analyzers)

All analyzers work without an API key, but AI enhancement improves recommendations when available.

- ✅ **Configuration Analyzer** - Playwright config, setup, timeouts (80% pattern, 20% AI)
  - Detects missing config files, invalid timeout settings, missing baseURL
  - AI provides context-aware configuration recommendations
  
- ✅ **Error Handling Analyzer** - Missing error handling, uncaught exceptions (60% pattern, 40% AI)
  - Finds uncaught async operations, generic error messages, missing error assertions
  - AI suggests specific error handling patterns for your codebase
  
- ✅ **Organization Analyzer** - File structure, naming, nesting depth (60% pattern, 40% AI)
  - Identifies very large files, deep nesting, mixed concerns
  - AI provides refactoring plans and organization strategies
  
- ✅ **Reusability Analyzer** - Code duplication, DRY violations (70% pattern, 30% AI)
  - Detects duplicate code blocks across files, repeated patterns
  - AI suggests extraction strategies and refactoring approaches
  
- ✅ **Documentation Analyzer** - JSDoc quality, comment clarity (40% pattern, 60% AI)
  - Checks JSDoc presence, function documentation coverage, commented-out code
  - AI evaluates documentation clarity and provides improvement suggestions
  
- ✅ **Security Analyzer** - Hard-coded secrets, vulnerabilities (70% pattern, 30% AI)
  - Detects hard-coded API keys, passwords, tokens, and security anti-patterns
  - AI identifies potential security vulnerabilities in test code

### Test Quality Pillar (8 Analyzers)

Most analyzers work without an API key, but AI enhancement improves recommendations. Test Design and Maintainability analyzers are AI-heavy and provide deeper insights with an API key.

#### Pattern-Based with AI Enhancement (No API Key Required)
- ✅ **Locator Analyzer** - Brittle selectors, robust alternatives (60% pattern, 40% AI)
  - Detects CSS class selectors, nth-child, XPath, and recommends data-testid/getByRole
  - AI suggests better selector strategies based on your codebase patterns
  
- ✅ **Sync Analyzer** - Hard-coded waits, smart wait strategies (70% pattern, 30% AI)
  - Finds waitForTimeout, missing waits after actions, suggests smart wait patterns
  - AI recommends optimal wait strategies for specific scenarios
  
- ✅ **Test Data Analyzer** - Shared data conflicts, isolation issues (70% pattern, 30% AI)
  - Detects hard-coded test data, shared data conflicts, missing cleanup hooks
  - AI suggests test data factory patterns and isolation strategies
  
- ✅ **Assertion Analyzer** - Weak/missing assertions, assertion quality (50% pattern, 50% AI)
  - Identifies weak assertions (toBeTruthy), missing assertions, assertion quality
  - AI evaluates assertion strength and suggests more specific assertions
  
- ✅ **Isolation Analyzer** - Test dependencies, shared state (80% pattern, 20% AI)
  - Detects test dependencies, shared state, order-dependent tests, serial suites
  - AI identifies cross-file dependencies and suggests isolation improvements
  
- ✅ **Performance Analyzer** - Slow tests, optimization opportunities (80% pattern, 20% AI)
  - Finds redundant waits, sequential operations, inefficient selectors, unnecessary page loads
  - AI suggests parallelization opportunities and performance optimizations

#### AI-Powered (Requires API Key - BYOK) ⭐
- ✅ **Test Design Analyzer** - Coverage gaps, missing scenarios (20% pattern, 80% AI)
  - AI analyzes test coverage, identifies missing scenarios, redundant tests
  - Provides deep insights into test design quality and edge case coverage
  
- ✅ **Maintainability Analyzer** - Test fragility, refactoring needs (30% pattern, 70% AI)
  - AI evaluates test fragility, tight coupling, complex logic, missing abstractions
  - Suggests refactoring strategies and maintainability improvements

### Output Formats
- ✅ **HTML Dashboard** (default) - Interactive web-based report with two-pillar breakdown
- ✅ **Markdown Report** - Human-readable report
- ✅ **JSON Report** - Machine-readable data for CI/CD integration

### Roadmap
- [ ] Parallel execution issues
- [ ] Security vulnerability detection
- [ ] Auto-fix mode
- [ ] CI/CD integration

## Installation

```bash
npm install -g @sdet-agents/test-health-cli
```

Or use without installation:

```bash
npx @sdet-agents/test-health-cli analyze
```

## Usage

### Basic Analysis (No API Key Required)

The tool works without an API key! All 14 analyzers run automatically, with 12 analyzers providing pattern-based analysis. The 2 AI-heavy analyzers (Test Design and Maintainability) provide basic pattern-based analysis without an API key, but deliver much deeper insights with AI enhancement.

```bash
cd your-playwright-project
test-health analyze  # Generates test-health-report.html by default
```

When run in an interactive terminal, you will be prompted to select which analyzers to run (or "Run all"). In CI or when stdin is not a TTY, all analyzers run by default so the command does not hang.

Or explicitly skip AI features:

```bash
test-health analyze --skip-ai
```

### Enable AI Features (BYOK - Bring Your Own Key)

To enable AI-powered analysis and get enhanced recommendations from all analyzers (especially Test Design and Maintainability), provide your API key (Anthropic or OpenAI):

```bash
# Using command line option (auto-detects provider from key format)
test-health analyze --api-key sk-ant-xxxxx  # Anthropic
test-health analyze --api-key sk-xxxxx      # OpenAI

# Or set environment variable
export ANTHROPIC_API_KEY=sk-ant-xxxxx
# Or
export OPENAI_API_KEY=sk-xxxxx
test-health analyze
```

### With Options

```bash
# Specify test directory
test-health analyze --path ./e2e-tests

# Enable AI features with API key
test-health analyze --api-key sk-ant-xxxxx

# Skip AI features (pattern-based analyzers only)
test-health analyze --skip-ai

# Choose output format (default: html)
test-health analyze                    # Generates HTML report by default
test-health analyze --output md        # Generate Markdown report
test-health analyze --output json      # Generate JSON report
```

### Configuration File

Create `.test-health.json` in your project root:

```json
{
  "testDir": "./tests",
  "framework": "playwright",
  "ignore": ["**/node_modules/**", "**/*.backup.ts"],
  "severity_thresholds": {
    "critical": 0,
    "high": 5,
    "medium": 20
  }
}
```

## API Key Setup (BYOK - Bring Your Own Key)

The API key is **optional**. Without it, all 14 analyzers work fine with pattern-based analysis. With it, you get:
- **Enhanced recommendations** from all analyzers (AI provides context-aware suggestions)
- **Deep AI analysis** from Test Design and Maintainability analyzers (coverage gaps, test fragility, refactoring strategies)
- **Better issue prioritization** and more actionable recommendations

### Quick Setup (Recommended)

Create a `.env` file in your project root:

```bash
echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env
```

The CLI automatically loads this file. **Remember to add `.env` to `.gitignore`!**

### Other Methods

**Command line:**
```bash
test-health analyze --api-key sk-ant-xxxxx  # Anthropic (auto-detected)
test-health analyze --api-key sk-xxxxx      # OpenAI (auto-detected)
```

**Environment variable:**
```bash
export ANTHROPIC_API_KEY=sk-ant-xxxxx  # Anthropic
# Or
export OPENAI_API_KEY=sk-xxxxx         # OpenAI
test-health analyze
```

**Priority order:** Command line → Environment variable → `.env` file → Not provided

**Provider auto-detection:** Keys starting with `sk-ant-` use Anthropic, others use OpenAI. Use `--ai-provider` to override.

📖 **For detailed setup instructions, see [API_KEY_SETUP.md](./API_KEY_SETUP.md)**

Get API keys from:
- Anthropic: https://console.anthropic.com/
- OpenAI: https://platform.openai.com/api-keys

## Output

The tool generates:

1. **test-health-report.html** (default) - Interactive web-based dashboard with two-pillar breakdown, visual scores, and detailed recommendations
2. **test-health-report.md** - Human-readable Markdown report with recommendations
3. **test-health-report.json** - Machine-readable data for CI/CD integration
4. Terminal summary with health score and top issues

## Example Report

The HTML report (default) provides an interactive dashboard with:
- **Two-Pillar Score Breakdown** - Separate Code Quality and Test Quality scores
- **Visual Health Indicators** - Color-coded scores and progress bars
- **Issue Categorization** - Issues grouped by analyzer and severity
- **Actionable Recommendations** - Context-aware suggestions with code examples
- **Quick Wins** - High-impact, low-effort improvements

The report includes detailed analysis from all 14 analyzers across both pillars, with AI-enhanced recommendations when an API key is provided.

## Development

```bash
# Clone repo
git clone https://github.com/your-org/test-health-cli
cd test-health-cli

# Install dependencies
npm install

# Run in development mode
npm run dev

# Build
npm run build

# Test
npm test
```

## Architecture

### Two-Pillar Structure

```
src/
├── cli.ts                    # CLI entry point
├── core/
│   ├── analyzer.ts           # Main analysis orchestrator (two-pillar)
│   ├── scorer.ts             # Two-pillar scorer (weighted scores)
│   ├── ai-coordinator.ts     # AI batching and coordination
│   └── config.ts             # Configuration loader
├── analyzers/
│   ├── base-analyzer.ts      # Abstract base class
│   ├── code-quality/         # Code Quality Pillar (6 analyzers)
│   │   ├── configuration-analyzer.ts
│   │   ├── error-handling-analyzer.ts
│   │   ├── organization-analyzer.ts
│   │   ├── reusability-analyzer.ts
│   │   ├── documentation-analyzer.ts
│   │   └── security-analyzer.ts
│   └── test-quality/         # Test Quality Pillar (8 analyzers)
│       ├── locator-analyzer.ts
│       ├── sync-analyzer.ts
│       ├── test-data-analyzer.ts
│       ├── assertion-analyzer.ts
│       ├── isolation-analyzer.ts
│       ├── test-design-analyzer.ts (AI-heavy)
│       ├── maintainability-analyzer.ts (AI-heavy)
│       └── performance-analyzer.ts
├── ai/
│   ├── context-builder.ts    # Project context for AI
│   └── prompts/              # AI prompt templates
├── reporters/
│   ├── markdown-reporter.ts  # MD output
│   ├── json-reporter.ts      # JSON output
│   └── html-reporter.ts      # HTML dashboard
└── utils/
    ├── ai-client.ts          # AI client abstraction
    └── file-scanner.ts       # File discovery
```

### Score Calculation

- **Overall Score** = Code Quality (35%) + Test Quality (65%)
- Each pillar score is a weighted average of its analyzers
- Individual analyzer scores range from 0-10

## Contributing

This is Agent #1 in the SDET Agent Suite. Future agents will include:
- Test Generator Agent
- Flake Fixer Agent
- Visual Regression Agent
- Performance Test Agent

## License

MIT

## Support

For issues and questions:
- GitHub Issues: https://github.com/your-org/test-health-cli/issues
- Documentation: https://docs.sdet-agents.dev

---

**Built with ❤️ by the SDET Agent Suite team**