npm package: @raxtzu/ai-test
Description: 🤖 AI-powered visual testing assistant that automatically analyzes UI failures with multiple LLM providers and provides actionable fixes for web applications
Author: Raveen Abeywickrama
Latest version: 2.0.0

# 🤖 AI Visual Testing Assistant

[![npm version](https://badge.fury.io/js/%40raxtzu%2Fai-test.svg)](https://badge.fury.io/js/%40raxtzu%2Fai-test)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](http://www.typescriptlang.org/)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](https://github.com/RaxTzu/ai-test)

> **🎯 The smartest AI-powered visual testing CLI that automatically analyzes UI failures and provides actionable fixes for web applications.**

**✅ Production Ready - Comprehensive framework support with intelligent automation**

Transform your visual testing workflow with AI-powered analysis. Get specific, implementable solutions for UI/UX issues through intelligent detection, smart recommendations, and multi-provider AI analysis.

## 🚨 Current Implementation Status

### ✅ **What Actually Works**
- **Core Analysis Engine** - Full `ai-test run` functionality with comprehensive orchestration
- **Playwright Integration** - Complete test execution and screenshot capture
- **Gemini Provider** - Production-ready with real API integration  
- **Template System** - Sophisticated prompt engineering with customization
- **TypeScript Architecture** - Robust SOLID principles implementation
- **CLI Framework** - Complete command structure with proper error handling

### 🚧 **In Development (Partial/Mock)**
- **Setup & Init Commands** - Basic stubs, not fully functional
- **Configuration Management** - No persistence, hardcoded values
- **Multiple AI Providers** - Claude, OpenAI, Azure return mock responses only
- **Framework Detection** - Returns hardcoded Playwright results

### ❌ **Not Yet Implemented** 
- **Ollama (Local LLM)** - Placeholder only, non-functional
- **Interactive Setup Wizard** - Planned but not implemented
- **Cypress/Selenium/Puppeteer** - Only Playwright works
- **Custom Reporter** - Planned feature

> **Bottom Line:** This is a sophisticated **Playwright + Gemini** visual testing tool with extensive architecture for future expansion. Core analysis functionality is production-ready.

## ✨ Key Features

### 🎯 **Multi-LLM Support**
- **Google Gemini** ✅ - Fully implemented with real API integration
- **Claude 3.5 Sonnet** 🚧 - Basic implementation (mock responses)
- **OpenAI GPT-4o** 🚧 - Basic implementation (mock responses)  
- **Azure OpenAI** 🚧 - Basic implementation (mock responses)
- **Local Ollama** ❌ - Placeholder only (not functional)

### 🔍 **LLM-Powered Analysis**
- **Visual Regression Detection** - LLM analysis of layout and styling issues via screenshots
- **Accessibility Guidance** - AI-generated WCAG 2.1 AA compliance suggestions
- **Responsive Design Review** - LLM analysis of cross-device compatibility
- **UI/UX Best Practices** - AI-powered design standards evaluation
- **Performance Indicators** - LLM assessment of loading states and user experience

> **Note:** Analysis is performed through sophisticated prompt engineering with LLMs, not computer vision algorithms.

### 🛠️ **Framework Integration**
- **Playwright** ✅ - Fully implemented with advanced screenshot and trace support
- **Cypress** 🚧 - Planned (not yet implemented)
- **Selenium** 🚧 - Planned (not yet implemented)
- **Puppeteer** 🚧 - Planned (not yet implemented)

### 📊 **Actionable Outputs**
- **Specific CSS Fixes** - Exact selectors and property changes
- **JavaScript Solutions** - Framework-specific code improvements
- **Test Enhancements** - Better selectors and strategies
- **Implementation Guides** - Step-by-step fix instructions

## 🚀 Quick Start

### Installation

```bash
# Install globally for CLI usage
npm install -g @raxtzu/ai-test

# Or install locally in your project
npm install --save-dev @raxtzu/ai-test
```

### Setup (Basic)

```bash
ai-test setup
```

**⚠️ Currently a basic implementation that shows:**
- Manual API key configuration instructions
- Basic provider information
- Next steps for getting started

> **Note:** Interactive wizard is planned but not yet implemented.

### Initialize Your Project

```bash
# Basic project initialization (stub implementation)
ai-test init my-project

# Options available but not fully functional yet
ai-test init --type=vue --provider=claude
```

**⚠️ Currently a stub implementation that shows success message only.**
> **Note:** Actual project initialization logic is planned but not yet implemented.

### Run Analysis ✅

```bash
# Analyze failed test screenshots (fully implemented)
ai-test run

# Continuous improvement mode (works)
ai-test run --continuous --max-iterations=5

# Target specific test patterns (works)
ai-test run --pattern="responsive" --output-format=html

# Additional options available
ai-test run --dry-run --provider=gemini --debug
```

**✅ This is the core implemented feature with full functionality.**

## 📋 Real-World Use Cases

### 1. **Professional Trading Platform Quality Assurance**

```javascript
// Playwright test that fails due to visual issues
test('trading dashboard professional appearance', async ({ page }) => {
  await page.goto('/dashboard');
  
  // Test will fail if layout is broken
  await expect(page.locator('.price-chart')).toBeVisible();
  await expect(page.locator('.trading-controls')).toBeVisible();
  
  // Screenshot automatically captured on failure
});
```

**AI Analysis Output:**
```
🔍 Visual Issues Found:
• Price display uses #666 text on #f8f9fa background (contrast ratio 2.1:1)
• Trading buttons lack visual hierarchy for quick decision-making
• Chart legend overlaps with price data on mobile viewports

💡 Specific Fixes:
.price-display {
  color: #1a1a1a !important;  /* WCAG AA compliant */
  font-weight: 600;
  font-size: 1.25rem;
}

.trading-button.buy {
  background: linear-gradient(135deg, #02c076, #059669);
  box-shadow: 0 2px 4px rgba(2, 192, 118, 0.3);
}

📱 Mobile Fix:
@media (max-width: 768px) {
  .chart-legend {
    position: absolute;
    top: 10px;
    right: 10px;
    background: rgba(255, 255, 255, 0.95);
  }
}
```

### 2. **E-commerce Conversion Optimization**

```javascript
test('product page conversion elements', async ({ page }) => {
  await page.goto('/product/premium-plan');
  
  // Test pricing visibility and CTA prominence
  await expect(page.locator('[data-test="price"]')).toBeVisible();
  await expect(page.locator('[data-test="cta-button"]')).toBeVisible();
});
```

**AI Analysis Output:**
```
🎯 Conversion Issues Detected:
• CTA button "Start Free Trial" has insufficient visual weight
• Price display lacks urgency indicators
• Trust signals positioned below the fold

🚀 Conversion Optimizations:
.cta-button {
  background: #ff6b35;           /* High-contrast orange */
  font-size: 1.1rem;
  padding: 1rem 2rem;
  box-shadow: 0 4px 12px rgba(255, 107, 53, 0.4);
  animation: pulse 2s infinite;  /* Subtle attention-grabber */
}

.price-container::after {
  content: "Limited Time Offer";
  background: #ff4444;
  color: white;
  padding: 0.25rem 0.5rem;
  border-radius: 4px;
  font-size: 0.75rem;
}
```

## 🎛️ Configuration Options

### Provider Selection

```bash
# Basic configuration (limited implementation)
ai-test config --list
ai-test config --get provider

# Setup command (basic implementation)
ai-test setup
```

### Analysis Customization

```json
{
  "analysisFeatures": [
    "visual-regression",    // Layout and styling issues
    "accessibility",        // WCAG compliance
    "responsive",          // Mobile/tablet compatibility
    "ux-practices",        // Professional design standards
    "performance",         // Loading states and UX
    "branding"            // Brand consistency
  ],
  "analysisDepth": "comprehensive",  // quick | balanced | comprehensive
  "maxIterations": 5,
  "outputFormat": "markdown"         // markdown | html | json | all
}
```

### Quality Standards

```json
{
  "qualityStandards": {
    "contrastRatio": 4.5,        // WCAG AA compliance
    "minTouchTarget": 44,        // Mobile accessibility
    "maxLoadTime": 2000,         // Performance expectations
    "brandColors": ["#0066cc", "#ff6b35"]  // Brand palette validation
  }
}
```

## 🏗️ Architecture & Design Principles

### SOLID Principles Implementation

- **Single Responsibility**: Each class has one clear purpose
- **Open/Closed**: Extensible provider system without modifying core
- **Liskov Substitution**: All providers implement consistent interfaces
- **Interface Segregation**: Focused interfaces for specific capabilities
- **Dependency Inversion**: High-level modules depend on abstractions

### Modular Structure

```
src/
├── core/                    # Core interfaces and abstractions
│   ├── interfaces.ts        # Contract definitions
│   └── container.ts         # Dependency injection
├── providers/               # AI provider implementations
│   ├── base-provider.ts     # Abstract base class
│   ├── claude-provider.ts   # Anthropic Claude
│   ├── gemini-provider.ts   # Google Gemini
│   └── provider-factory.ts  # Factory pattern
├── services/                # Business logic services
│   ├── logger.ts           # Logging abstraction
│   ├── rate-limiter.ts     # API rate management
│   └── config-manager.ts   # Configuration handling
├── cli/                     # Command-line interface
│   ├── index.ts            # Main CLI application
│   └── commands/           # Command implementations
└── types/                   # TypeScript definitions
    └── index.ts            # Type definitions
```

## 🧪 Integration Examples

### GitHub Actions CI/CD

```yaml
name: AI Visual Quality Assurance
on: [push, pull_request]

jobs:
  visual-testing:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Install AI Visual Testing
        run: npm install -g @raxtzu/ai-test
        
      - name: Run Playwright tests
        run: npx playwright test
        continue-on-error: true
        
      - name: AI Visual Analysis
        run: ai-test run --max-iterations=3
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          
      - name: Upload Analysis Reports
        uses: actions/upload-artifact@v4
        with:
          name: visual-analysis-reports
          path: .ai-testing/output/reports/
```

### Package.json Scripts

```json
{
  "scripts": {
    "test:visual": "ai-test run --pattern='visual'",
    "test:accessibility": "ai-test run --provider=gemini",
    "test:mobile": "ai-test run --pattern='responsive'",
    "setup:ai": "ai-test setup",
    "qa:full": "npm run test && ai-test run"
  }
}
```

## 📊 Cost Optimization

### Provider Cost Comparison

| Provider | Implementation Status | Cost (per 1M tokens) | Working Status |
|----------|---------------------|---------------------|----------------|
| **Gemini 2.0 Flash** | ✅ Fully Implemented | $1.25 / $5.00 | ✅ Production Ready |
| **Claude 3.5 Sonnet** | 🚧 Mock Implementation | $3.00 / $15.00 | ❌ Returns Mock Data |
| **GPT-4o** | 🚧 Mock Implementation | Variable by image | ❌ Returns Mock Data |
| **Azure OpenAI** | 🚧 Mock Implementation | Enterprise pricing | ❌ Returns Mock Data |
| **Ollama (Local)** | ❌ Placeholder Only | Infrastructure only | ❌ Non-functional |

### Smart Cost Management

```bash
# Basic configuration commands (stub implementations)
ai-test config --set provider=gemini
ai-test config --get provider

# Core analysis command (fully implemented)
ai-test run --provider=gemini --max-iterations=3
```

## 🔧 Advanced Usage

### Custom Analysis Templates

```javascript
// .ai-testing/templates/trading-platform.md
You are analyzing a professional trading platform interface.

Focus on:
- Real-time data visibility and clarity
- Quick action button accessibility
- Professional color schemes (Bloomberg Terminal standards)
- Information density vs. readability balance
- Mobile trading workflow optimization

Provide fixes that meet financial industry UI standards.
```

### Programmatic API

```javascript
import { AnalysisEngine, ProviderFactory } from '@raxtzu/ai-test';

const engine = new AnalysisEngine();
const provider = ProviderFactory.create('claude', { apiKey: 'your-key' });

const result = await engine.analyze(screenshots, {
  provider,
  features: ['accessibility', 'responsive'],
  depth: 'comprehensive'
});

console.log('Issues found:', result.analysis.visualIssues.length);
console.log('CSS fixes:', result.fixes.css);
```

### Playwright Integration

```javascript
// playwright.config.js
export default defineConfig({
  use: {
    screenshot: 'only-on-failure',    // AI needs failure screenshots
    video: 'retain-on-failure',      // Additional context
    trace: 'retain-on-failure',      // Detailed execution trace
  },
  
  // Custom reporter for AI integration (not yet implemented)
  reporter: [
    ['html'],
    ['@raxtzu/ai-test/reporter']  // Planned feature
  ]
});
```

## 🐛 Troubleshooting

### Common Issues

**No screenshots found:**
```bash
ai-test doctor  # Basic health check
ai-test status  # Check service status
```

**API issues:**
```bash
ai-test status  # Check provider status
# Note: Only Gemini provider is fully functional
```

**Basic analysis commands:**
```bash
ai-test run --debug  # Debug mode
ai-test run --dry-run  # Preview mode
```

**Permission errors:**
```bash
ai-test doctor  # Basic diagnostics
```

## 📚 Learning Resources

### Example Projects
- [Trading Platform Demo](./examples/trading-platform/) - Professional financial interface
- [E-commerce Store](./examples/ecommerce/) - Conversion optimization focus
- [SaaS Dashboard](./examples/saas-dashboard/) - Complex data visualization
- [Design System](./examples/design-system/) - Component library validation

### Best Practices
- **Screenshot Strategy**: Focus on failure points, not success states
- **Prompt Engineering**: Use specific, actionable language
- **Iterative Improvement**: Run analysis after each fix cycle
- **Cost Management**: Use appropriate models for analysis depth needed

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.

### Development Setup

```bash
git clone https://github.com/RaxTzu/ai-test.git
cd ai-test
npm install
npm run dev
```

### Running Tests

```bash
npm test              # Unit tests
npm run test:integration  # Integration tests
npm run lint          # Code quality
```

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

## 🙋‍♂️ Support & Community

- **Issues**: [GitHub Issues](https://github.com/RaxTzu/ai-test/issues)
- **Discussions**: [GitHub Discussions](https://github.com/RaxTzu/ai-test/discussions)
- **Documentation**: [Full API Reference](./docs/api-reference.md)

---

**Made with ❤️ by [Raveen Abeywickrama](https://github.com/RaxTzu)**

*Empowering developers to build visually perfect web applications with AI assistance.*