Skill: adityak6798/website-usability-test-nova-act (version 1.0.3)

--- name: nova-act-usability version: 1.0.2 description: AI-orchestrated usability testing using Amazon Nova Act. The agent generates personas, runs tests to collect raw data, interprets responses to determine goal achievement, and generates HTML reports. Tests real user workflows (booking, checkout, posting) with safety guardrails. Use when asked to "test website usability", "run usability test", "generate usability report", "evaluate user experience", "test checkout flow", "test booking process", or "analyze website UX". metadata: openclaw: requires: config: - ~/.openclaw/config/nova-act.json bins: - python3 --- # Nova Act Usability Testing v1.0.2 **AI-orchestrated** usability testing with digital twin personas powered by Amazon Nova Act. ## ⚠️ Prerequisites & Credentials **This skill requires an Amazon Nova Act API key.** | Requirement | Details | |-------------|---------| | **API Key** | Nova Act API key from [AWS Console](https://console.aws.amazon.com/) | | **Config Location** | `~/.openclaw/config/nova-act.json` | | **Format** | `{"apiKey": "your-nova-act-api-key-here"}` | | **Dependencies** | `pip3 install nova-act pydantic playwright` | | **Browser** | `playwright install chromium` (~300MB download) | ## 🔒 Data & Privacy Notice **What this skill accesses:** - **Reads:** `~/.openclaw/config/nova-act.json` (your API key) - **Writes:** `./nova_act_logs/` (trace files with screenshots), `./test_results_adaptive.json`, `./nova_act_usability_report.html` **What trace files contain:** - Screenshots of every page visited - Full page content (HTML, text) - Browser actions and AI decisions **Recommendations:** - Run tests only on **non-production** or **test environments** - Be aware traces may capture **PII or sensitive data** visible on tested pages - Review/delete trace files after use if they contain sensitive content - Consider running in a **sandboxed environment** (container/VM) for untrusted sites --- ## Features **Agent-Driven Interpretation**: The script no longer interprets responses. YOU (the agent) must: 1. Run the test script → collect raw data 2. Read JSON → interpret each `raw_response` 3. Set `goal_achieved` and `overall_success` 4. Generate the report No hardcoded regex. No extra API calls. The agent doing the work is already running. ## Quick Start (For AI Agents) **When a user asks to test a website, YOU (the AI agent) must complete ALL 4 phases:** | Phase | What Happens | Who Does It | |-------|--------------|-------------| | 1. Setup | Generate personas, run test script | Agent + Script | | 2. Collect | Script captures raw Nova Act responses | Script | | 3. Interpret | Read JSON, determine goal_achieved for each step | **Agent** | | 4. Report | Generate HTML report with interpreted results | Agent | **⚠️ The script does NOT interpret responses or generate the final report. You must do phases 3-4.** ### 🎯 Recommended: AI Agent Generates Personas **You're already an AI (Claude) - use your intelligence to generate contextual personas!** ```python import subprocess import os import sys import json import tempfile # Step 1: Check dependencies try: import nova_act print("✅ Dependencies ready") except ImportError: print("📦 Dependencies not installed. Please run:") print(" pip3 install nova-act pydantic playwright") print(" playwright install chromium") sys.exit(1) # Step 2: Verify Nova Act API key config_file = os.path.expanduser("~/.openclaw/config/nova-act.json") with open(config_file, 'r') as f: config = json.load(f) if config.get('apiKey') == 'your-nova-act-api-key-here': print(f"⚠️ Please add your Nova Act API key to {config_file}") sys.exit(1) # Step 3: YOU (the AI agent) generate personas # Example for https://www.pgatour.com/ (golf tournament site) website_url = "https://www.pgatour.com/" personas = [ { "name": "Marcus Chen", "archetype": "tournament_follower", "age": 42, "tech_proficiency": "high", "description": "Avid golf fan who follows multiple tours and tracks player stats", "goals": [ "Check current tournament leaderboard", "View recent tournament results", "Track favorite player performance" ] }, { "name": "Dorothy Williams", "archetype": "casual_viewer", "age": 68, "tech_proficiency": "low", "description": "Occasional golf viewer who watches major tournaments", "goals": [ "Find when the next tournament is", "See who won recently", "Understand how to watch online" ] } ] # Step 4: Save personas and run test with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: json.dump(personas, f, indent=2) personas_file = f.name skill_dir = os.path.expanduser("~/.openclaw/skills/nova-act-usability") test_script = os.path.join(skill_dir, "scripts", "run_adaptive_test.py") # Run with AI-generated personas subprocess.run([sys.executable, test_script, website_url, personas_file]) # Clean up temp file os.unlink(personas_file) ``` **Persona Template:** ```json { "name": "FirstName LastName", "archetype": "descriptive_identifier", "age": 30, "tech_proficiency": "low|medium|high", "description": "One sentence about who they are", "goals": [ "First goal relevant to this website", "Second goal relevant to this website", "Third goal relevant to this website" ] } ``` ### 📝 Alternative: Simple Custom Persona If user specifies a persona description, pass it as a string: ```python # User: "Test PGA Tour site as a golf enthusiast" website_url = "https://www.pgatour.com/" user_persona = "golf enthusiast who follows tournaments closely" subprocess.run([sys.executable, test_script, website_url, user_persona]) # Script will parse this and create personas automatically ``` ### ⚠️ Fallback: Auto-Generation (Not Recommended) Let the script guess personas based on basic category keywords: ```python # Generic, less contextual personas subprocess.run([sys.executable, test_script, website_url]) ``` ### Why YOU Should Generate Personas **✅ Advantages:** - **Better context:** You have full conversation history and domain knowledge - **Smarter inference:** You can analyze the URL, industry, and user intent - **No duplicate API calls:** You're already Claude - don't call yourself again! - **User preferences:** You can adapt based on stated preferences - **Clarifying questions:** You can ask the user about target demographics **❌ What to avoid:** - Don't let Python script make its own Claude API call (wasteful) - Don't rely on generic fallback personas (less accurate) - Don't skip persona generation (hurts test quality) ### 💡 Tips for Persona Generation **Analyze the website:** - **URL domain:** `.gov` → citizens, `.edu` → students/faculty - **Keywords:** "shop" → shoppers, "book" → travelers, "play" → gamers - **Industry:** Golf → fans/players, Banking → customers/businesses **Create diverse personas:** - Mix experience levels (beginner, intermediate, expert) - Mix tech proficiency (low, medium, high) - Mix age ranges (young, middle-aged, senior) - Mix motivations (casual, professional, enthusiastic) **Generate realistic goals:** - Specific to the website's purpose - Actionable and measurable - Match the persona's characteristics **Examples by industry:** - **E-commerce:** bargain_hunter, comparison_shopper, impulse_buyer - **News:** daily_reader, topic_follower, casual_browser - **Sports:** die_hard_fan, casual_viewer, stats_tracker - **Travel:** business_traveler, vacation_planner, deal_seeker - **SaaS:** power_user, evaluator, beginner ## User Invocation Users can trigger this skill by saying: - "Test the usability of [website URL]" - "Run a usability test on [website URL]" - "Generate a usability report for [website URL]" - "Evaluate the UX of [website URL]" - "Analyze [website URL] for usability issues" - **NEW:** "Test the booking flow on [website]" - **NEW:** "Test the checkout process on [e-commerce site]" - **NEW:** "Test posting workflow on [social media site]" **The AI will automatically:** 1. Load the Nova Act cookbook for guidance 2. Analyze the page to understand it 3. Detect if it's a workflow-based site (booking, e-commerce, social, etc.) 4. **Generate contextual personas:** - If custom persona specified → Create persona matching that description - If no custom persona → Use Claude AI to infer the 3 most plausible real-world user types - Fallback to category-based personas if AI unavailable 5. Create realistic test cases (including full workflows when appropriate) 6. Run adaptive, iterative tests with Nova Act 7. **NEW:** Apply safety stops before material impact actions (payment, posting, account creation) 8. Generate comprehensive HTML report with trace links 9. Provide viewing instructions ## Workflow Testing **NEW in this version:** The skill now tests complete user journeys, not just information-finding! ### Supported Workflows **E-Commerce:** - Product search → Add to cart → Checkout → **STOP before payment** **Flight/Hotel Booking:** - Search → Select → Fill details → **STOP before booking** **Social Media:** - Create post → Add content → **STOP before publishing** **Account Signup:** - Fill registration → **STOP before final submission** **Form Submission:** - Fill form → **STOP before submit** ### Safety Guarantees The skill will **NEVER:** - Complete actual purchases - Create real accounts - Post publicly - Send emails/messages - Subscribe to newsletters - Make any action with monetary/legal/reputational impact The skill will **ALWAYS:** - Test up to (but not including) the final action - Verify the final button exists and is accessible - Document the safety stop in observations ## 🧠 Agent Analysis (CRITICAL) **You (the AI agent) must analyze test results!** The script collects raw responses but does NOT interpret them. ### Why Agent Analysis? The script returns raw Nova Act responses like: - `"No"` - Is there a pricing link? - `"I don't see any documentation"` - Is there docs? - `"Amazon Nova Act"` - What is the headline? **You must determine if each response means the goal was achieved:** | Response | Goal Achieved? | |----------|---------------| | `"No"` | ❌ NOT achieved | | `"I don't see..."` | ❌ NOT achieved | | `"Not found"` | ❌ NOT achieved | | `"Yes, I found..."` | ✅ Achieved | | `"Amazon Nova Act"` (content) | ✅ Achieved | | `"The pricing is $29/mo"` | ✅ Achieved | ### Result Data Structure After the test script runs, read the JSON results. Each step contains: ```json { "step_name": "check_nav_for_pricing", "prompt": "Is there a pricing link in the navigation?", "expected_outcome": "Find pricing in navigation", "raw_response": "No", "api_success": true, "needs_agent_analysis": true, "attempts": [ { "prompt": "Is there a pricing link in the navigation?", "response": "No", "approach": "original" } ] } ``` **Key fields you analyze:** - `raw_response`: The actual Nova Act response - YOU determine what it means - `api_success`: Did the API call work? (script handles this) - `needs_agent_analysis`: Always `true` - your cue to interpret - `attempts`: All attempts made (script tries up to 3 alternative approaches) ### How to Analyze **For each step, determine:** 1. `goal_achieved`: Did the response indicate success or failure? 2. `friction_level`: How hard was it? (attempts.length > 1 = friction) 3. `observations`: UX insights from the response **Analysis example:** ``` Step 1: "Is there a pricing link?" → Response: "No" (1 attempt) → Goal achieved: NO (explicit negative) → Friction: HIGH (not discoverable) Step 2: "What is the headline?" → Response: "Amazon Nova Act" (1 attempt) → Goal achieved: YES (actual content) → Friction: LOW (immediately visible) Step 3: "Find documentation" → Response: "I found a docs link in the footer" (3 attempts) → Goal achieved: YES (found eventually) → Friction: MEDIUM (required multiple approaches) ``` ### Helper Functions (For Script Integration) The `response_interpreter.py` provides helpers if you want structured prompts: ```python from scripts.response_interpreter import ( format_for_agent_analysis, create_agent_prompt_for_interpretation, create_agent_prompt_for_alternative ) # Format all results for analysis formatted = format_for_agent_analysis(results) # Get interpretation prompt for one step prompt = create_agent_prompt_for_interpretation(step_result) # Get retry prompt when goal not achieved retry_prompt = create_agent_prompt_for_alternative( original_prompt="Is there a pricing link?", failed_response="No", attempt_number=2 ) ``` ### Complete Analysis Workflow (MANDATORY) **The script does NOT generate the final report automatically.** You (the agent) must: 1. **Run the test script** → outputs `test_results_adaptive.json` with raw data 2. **Read the JSON** into your context 3. **Interpret each step** → set `goal_achieved: true/false` based on `raw_response` 4. **Set overall success** → set `overall_success: true/false` on each test 5. **Save updated JSON** 6. **Call report generator** with interpreted results **Step-by-step code for the agent to execute:** ```python import json import os import sys # Add skill scripts to path sys.path.insert(0, os.path.expanduser("~/.openclaw/skills/nova-act-usability/scripts")) from enhanced_report_generator import generate_enhanced_report # 1. Read raw results with open('test_results_adaptive.json', 'r') as f: results = json.load(f) # 2. YOU (the agent) interpret each step for test in results: goals_achieved = 0 for step in test.get('steps', []): raw = step.get('raw_response', '') # AGENT INTERPRETS: Does this response indicate goal was achieved? # You decide based on the response content and expected outcome # Example interpretations: # "No" → goal_achieved = False # "Leaderboard, News, Schedule, Players" → goal_achieved = True (content found) # "Yes" → goal_achieved = True # "I don't see any..." → goal_achieved = False step['goal_achieved'] = ??? # YOU set this based on your interpretation if step['goal_achieved']: goals_achieved += 1 # 3. Set overall success (e.g., >= 50% goals achieved) total = len(test.get('steps', [])) test['goals_achieved'] = goals_achieved test['overall_success'] = (goals_achieved / total >= 0.5) if total > 0 else False # 4. Save interpreted results with open('test_results_adaptive.json', 'w') as f: json.dump(results, f, indent=2) # 5. Generate report with interpreted data page_analysis = { 'title': '...', # From your earlier analysis 'purpose': '...', 'navigation': [...] } all_traces = [] for r in results: all_traces.extend(r.get('trace_files', [])) report_path = generate_enhanced_report(page_analysis, results, all_traces) print(f"Report: {report_path}") ``` **Why the agent must interpret:** - No hardcoded regex or pattern matching - You understand context (what "Yes" means for this specific question) - You can reason about partial success, edge cases - No duplicate Claude API calls - you're already running! ## ⚠️ Critical: Keep Nova Act Prompts Simple **Nova Act is a browser automation tool, NOT a reasoning engine.** The Claude agent (you) does all reasoning about: - What to test based on the persona - Whether results are good or bad - What the UX implications are Nova Act just: - Clicks, types, scrolls - Reports what it sees ### ❌ WRONG: Asking Nova Act to reason ```python # DON'T ask Nova Act to think about personas nova.act("As a beginner user, can you easily find the documentation?") nova.act("Would a business professional find the pricing clear?") nova.act("Is this task accomplishable for someone with low technical skills?") ``` ### ✅ RIGHT: Simple, direct browser commands ```python # Simple browser actions nova.act("Click the Documentation link in the navigation") nova.act("Find and click a link containing 'Pricing'") nova.act_get("What text is displayed in the main heading?") nova.act_get("List the navigation menu items visible on this page") ``` ### The Correct Workflow 1. **Agent** (you) decides what to test based on persona: "Dorothy is 68 with low tech skills - she wants to know how to watch golf online" 2. **Agent** generates simple Nova Act prompts: "Click 'Watch & Listen' in the navigation" 3. **Nova Act** executes browser task and returns raw results: "Clicked Watch & Listen, now on video page" 4. **Agent** interprets results: "Dorothy would find this confusing because the options are unclear..." ## How This Works **You (the AI) are the orchestrator.** This skill provides: 1. **Nova Act cookbook** (`references/nova-act-cookbook.md`) - Best practices, workflow patterns, and safety guidelines (automatically loaded at test start) 2. **Adaptive test orchestrator** (`run_adaptive_test.py`) - Main execution script with workflow detection 3. **Dynamic strategy generator** (`scripts/dynamic_exploration.py`) - Generates workflow-appropriate test strategies 4. **Session management** (`scripts/nova_session.py`) - Nova Act wrapper 5. **Report generator** (`enhanced_report_generator.py`) - Auto-generated HTML reports **Execution Flow:** ### CRITICAL: Check Dependencies First **Before running ANY test, check if dependencies are installed:** ```bash # Check if nova-act is installed python3 -c "import nova_act" 2>/dev/null if [ $? -ne 0 ]; then echo "Dependencies not installed. Please run:" echo " pip3 install nova-act pydantic playwright" echo " playwright install chromium" exit 1 fi # Check API key if ! grep -q '"apiKey":.*[^"]' ~/.openclaw/config/nova-act.json; then echo "⚠️ Please add your Nova Act API key to ~/.openclaw/config/nova-act.json" exit 1 fi ``` **Or use Python to check:** ```python import sys # Check if nova-act is installed try: import nova_act print("✅ Dependencies already installed") except ImportError: print("📦 Dependencies not installed. Please run:") print(" pip3 install nova-act pydantic playwright") print(" playwright install chromium") sys.exit(1) ``` ### Running Tests (After Dependencies Confirmed) When a user asks for usability testing: ```bash # Find the skill directory SKILL_DIR=~/.openclaw/skills/nova-act-usability # Run the adaptive test script python3 "$SKILL_DIR/scripts/run_adaptive_test.py" "https://example.com" # This will: # - Create nova_act_logs/ in current directory # - Create test_results_adaptive.json in current directory # - Create nova_act_usability_report.html in current directory # - Provide 60-second status updates during test ``` ### ⏱️ Timeout Guidance **Recommended timeout: 30 minutes (1800 seconds)** Full usability tests with 3 personas × 3 goals = 9 tests can take 10-20+ minutes depending on: - Website load times (slow sites like media-heavy sports sites take longer) - Nova Act API response times (each act() call takes 5-60 seconds) - Network conditions **Graceful shutdown:** If the test is interrupted (timeout, SIGTERM, SIGINT), it will: 1. Save all completed test results to `test_results_adaptive.json` 2. Generate a **partial report** clearly marked as incomplete 3. Show how many tests completed vs planned **For shorter tests:** Use fewer personas or goals: ```python # Quick test with 1 persona personas = [{"name": "Test User", "archetype": "casual", ...}] ``` ### What You (the AI) Need to Do: 1. **Check dependencies** (run the check above) 2. **If missing**: Tell user to run `pip3 install nova-act pydantic playwright && playwright install chromium` 3. **If present**: Extract the website URL from user's request 4. **Run the test** with the URL as argument 5. **Monitor progress** (status updates every 60 seconds) 6. **Share the report** viewing instructions with user ## Quick Start **When user requests usability testing:** ```python import subprocess import os # Get skill directory skill_dir = os.path.expanduser("~/.openclaw/skills/nova-act-usability") if not os.path.exists(skill_dir): # Try workspace location skill_dir = os.path.join(os.getcwd(), "nova-act-usability") script_path = os.path.join(skill_dir, "scripts", "run_adaptive_test.py") # Run test result = subprocess.run( ["python3", script_path, "https://example.com"], env={**os.environ, [REDACTED_SECRET]: "1"}, capture_output=True, text=True ) print(result.stdout) ``` ## Detailed Workflow (Internal) The adaptive test script (`run_adaptive_test.py`) handles: ### Step 1: Page Analysis - Loads the page with Nova Act - Extracts title, navigation, purpose - Identifies key elements (docs, demo, pricing) ### Step 2: Contextual Persona Generation - Creates personas based on what the page offers - Developer persona if API/code focused - Business persona for evaluation - Beginner persona if demo available ### Step 3: Realistic Test Case Generation - Top 3 use cases per persona - Based on actual page content - Matched to persona goals ### Step 4: Iterative Test Execution For each persona + task combination: ```python from scripts.nova_session import nova_session from nova_act import BOOL_SCHEMA import time observations = [] with nova_session(website_url) as nova: start_time = time.time() # Initial navigation observations.append({ "step": "navigate", "action": f"Loaded {website_url}", "success": True, "notes": "Initial page load" }) # Execute task step-by-step (AI-orchestrated) # Break into small act() calls based on cookbook guidance # Example: "Find pricing information" task # Step 1: Look for pricing link nova.act("Look for a link or button for pricing, plans, or subscription") found = nova.act_get( "Is there a visible pricing or plans link?", schema=BOOL_SCHEMA ) observations.append({ "step": "find_pricing_link", "action": "Search for pricing navigation", "success": found.parsed_response, "notes": "Easy to find" if found.parsed_response else "Not immediately visible - UX friction" }) if found.parsed_response: # Step 2: Navigate to pricing nova.act("Click on the pricing or plans link") # Step 3: Analyze pricing page is_clear = nova.act_get( "Is the pricing information clearly displayed with prices and features?", schema=BOOL_SCHEMA ) observations.append({ "step": "view_pricing", "action": "Accessed pricing page", "success": is_clear.parsed_response, "notes": "Clear pricing display" if is_clear.parsed_response else "Pricing unclear or confusing" }) else: # Alternative path - try search nova.act("Look for a search function") # ... continue orchestrating duration = time.time() - start_time # Document overall task result task_success = all(obs["success"] for obs in observations if obs["success"] is not None) results.append({ "persona": persona_name, "task": task_description, "success": task_success, "duration": duration, "observations": observations, "friction_points": [obs for obs in observations if not obs.get("success")] }) ``` ### Step 5: Pool and Analyze Results After all tests: 1. Identify common friction points across personas 2. Note accessibility issues for low-tech personas 3. Flag efficiency problems (too many steps) 4. Document task failures (major UX issues) ### Step 6: Generate Report ```python import json from scripts.enhanced_report_generator import generate_enhanced_report # Save results with open("test_results_adaptive.json", "w") as f: json.dump(results, f, indent=2) # Generate HTML report report_path = generate_enhanced_report( page_analysis=page_analysis, results=test_results ) print(f"Report: {report_path}") ``` ## Key Principles ### Dynamic Task Decomposition The AI should decide how to break down each task based on: - Website complexity - Persona's technical level - Task nature (navigation vs data entry vs search) **Low-tech persona example:** ```python # More explicit, step-by-step nova.act("Look for a button labeled 'Contact' or 'Contact Us'") nova.act("Click on the Contact button") result = nova.act_get("Is there a phone number or email address visible?") ``` **High-tech persona example:** ```python # Test efficiency features nova.act("Look for keyboard shortcuts or quick access features") nova.act("Try to use search (Ctrl+K or Cmd+K)") ``` ### Real-Time Observation After EVERY `act()` call, analyze: - Did it succeed? - Was the UI element easy to find? - Was the label clear? - How many attempts needed? - Any error messages? Document friction immediately in observations. ### Persona-Aware Prompting Adapt `act()` prompts to persona characteristics: - **Elderly/low-tech:** Look for obvious, labeled buttons; read everything - **Power users:** Try keyboard shortcuts, advanced features - **Mobile users:** Test mobile responsiveness, touch targets - **Screen reader users:** Test keyboard navigation, ARIA labels ## Resources ### `references/nova-act-cookbook.md` **MUST READ before starting any test.** Contains best practices for: - Effective act() prompting - Task decomposition strategies - Data extraction patterns - Error handling - Persona adaptation ### `references/persona-examples.md` Template personas with detailed profiles: - Tech-savvy millennial - Elderly first-timer - Busy professional - Student/budget-conscious - Accessibility-focused - International/non-native speaker ### `scripts/nova_session.py` Thin wrapper providing Nova Act session primitive: ```python with nova_session(url, headless=True, logs_dir="./logs") as nova: nova.act("action") result = nova.act_get("query", schema=Schema) ``` ### `scripts/enhanced_report_generator.py` Compiles observations into HTML usability report with trace file links. ### `assets/report-template.html` Professional HTML template for usability reports. ## ⚠️ IMPORTANT: First-Time Setup Required **This skill requires dependencies that must be installed before use.** ### For AI Agents: Dependency Check **ALWAYS check if dependencies are installed before running tests:** ```python # Quick dependency check try: import nova_act print("✅ Dependencies installed") except ImportError: print("📦 Dependencies not installed. Please run:") print(" pip3 install nova-act pydantic playwright") print(" playwright install chromium") print("") print("This will take 2-3 minutes to download browsers (~300MB)") ``` ### For Users: One-Time Setup **Step 1: Install Python packages** ```bash pip3 install nova-act pydantic playwright ``` **Step 2: Install Playwright browser** ```bash playwright install chromium ``` **Step 3: Configure API key** 1. Get your Nova Act API key from [AWS Console](https://console.aws.amazon.com/) 2. Create config file: ```bash mkdir -p ~/.openclaw/config echo '{"apiKey": "your-key-here"}' > ~/.openclaw/config/nova-act.json ``` 3. Replace `your-key-here` with your actual Nova Act API key ## Example: AI-Orchestrated Test **User request:** "Test example.com for elderly users" **AI orchestration:** 1. Read `references/nova-act-cookbook.md` 2. Read `references/persona-examples.md` 3. Generate elderly persona (Dorothy, 72, low tech proficiency) 4. Generate tasks: - "Find contact information" - "Read about services" - "Navigate to FAQ" 5. For each task, dynamically orchestrate Nova Act: - Start session - Execute small act() steps - Observe and analyze each result - Take notes on friction (small text, unclear labels, etc.) - Continue or adapt based on observations 6. Pool observations 7. Generate HTML report with findings and recommendations **The AI decides every step.** The skill just provides tools and guidance. ## File Structure ``` nova-act-usability/ ├── SKILL.md # This file ├── README.md # User documentation ├── skill.json # Skill manifest ├── scripts/ │ ├── run_adaptive_test.py # Main orchestrator (accepts URL arg) │ ├── nova_session.py # Session wrapper │ ├── enhanced_report_generator.py # HTML report generator │ └── trace_finder.py # Extract trace file paths ├── references/ │ ├── nova-act-cookbook.md # Best practices │ └── persona-examples.md # Template personas └── assets/ └── report-template.html # HTML template ``` ## Output Files (Created in Working Directory) When you run a test, these files are created in your current working directory: ``` ./ ├── nova_act_logs/ # Nova Act trace files │ ├── act_<id>_output.html # Session recordings │ └── ... ├── test_results_adaptive.json # Raw test results └── nova_act_usability_report.html # Final report ``` All paths are relative - works from any installation location!

path: skill.json
content: {
  "name": "nova-act-usability",
  "version": "1.0.2",
  "description": "AI-orchestrated usability testing with workflow testing (booking, checkout, posting) and safety guardrails using Amazon Nova Act",
  "author": "Adi",
  "license": "MIT",
  "tags": [
    "testing",
    "browser-automation",
    "usability",
    "nova-act",
    "playwright",
    "web-testing"
  ],
  "requirements": {
    "python": ">=3.8",
    "packages": [
      "nova-act",
      "playwright"
    ]
  },
  "configuration": {
    "required": ["apiKey"],
    "configPath": "~/.openclaw/config/nova-act.json",
    "example": {
      "apiKey": "your-nova-act-api-key-here"
    }
  },
  "entrypoint": "SKILL.md",
  "category": "Testing & Automation"
}


path: README.md
content: # Nova Act Usability Testing Skill v1.0.2

AI-orchestrated usability testing for websites using Amazon Nova Act browser automation.

## ⚠️ Prerequisites & Credentials

**This skill requires an Amazon Nova Act API key.**

| Requirement | Details |
|-------------|---------|
| **API Key** | Nova Act API key from [AWS Console](https://console.aws.amazon.com/) |
| **Config Location** | `~/.openclaw/config/nova-act.json` |
| **Format** | `{"apiKey": "your-nova-act-api-key-here"}` |

## 🔒 Data & Privacy Notice

**What this skill accesses:**
- **Reads:** `~/.openclaw/config/nova-act.json` (your API key)
- **Writes:** `./nova_act_logs/`, `./test_results_adaptive.json`, `./nova_act_usability_report.html`

**Trace files contain screenshots and full page content.** Run tests only on non-production environments. Traces may capture PII visible on tested pages.

---

## Features

🎯 **Agent-Driven Interpretation**: The script collects raw data, the AI agent (you) interprets responses and generates reports. No hardcoded regex, no extra API calls.

📊 **Three-Phase Flow**:
1. **Collect** - Script runs Nova Act, captures raw responses
2. **Interpret** - Agent reads JSON, determines goal achievement
3. **Report** - Agent generates HTML with accurate pass/fail status

## What It Does

- **Workflow Testing**: Tests complete user journeys (booking flights, checkout, posting) with safety guardrails
- **Adaptive Testing**: AI-driven browser automation that explores websites like a real user
- **Safety First**: Automatically stops before material impact (payment, posting, account creation)
- **Contextual Personas**: Analyzes your site and generates relevant user personas automatically
- **Realistic Test Cases**: Creates targeted test scenarios based on what your page actually offers (including full workflows)
- **Cookbook-Guided**: Loads best practices and safety guidelines automatically at test start
- **Comprehensive Reports**: Auto-generates HTML reports with detailed findings and session trace links

## Features

✅ **Workflow Testing** - Tests complete user journeys end-to-end (booking, checkout, posting, signup)  
✅ **Safety Guardrails** - Automatically stops before payment, posting, or account creation  
✅ **Real Browser Automation** - Actual Playwright browser control via Nova Act  
✅ **Cookbook Integration** - Loads best practices and workflow patterns automatically  
✅ **Fully Dynamic Testing** - Exploration strategies generated per website/persona (no hardcoded logic!)  
✅ **Smart Persona Generation** - Analyzes page content to create relevant user types  
✅ **Adaptive Testing** - AI tries multiple variations when element text doesn't match exactly  
✅ **Robust Error Handling** - Handles scroll loops, timeouts, and Nova Act failures gracefully  
✅ **Detailed Reporting** - Professional HTML reports with step-by-step observations  
✅ **Trace File Integration** - Links to Nova Act's HTML session recordings for replay

### Supported Workflows

**E-Commerce:**
- Product search → Add to cart → Checkout → **STOP before payment**

**Booking (Flights/Hotels):**
- Search → Select → Fill details → **STOP before booking**

**Social Media:**
- Create post → Add content → **STOP before publishing**

**Account Signup:**
- Fill registration form → **STOP before final submission**

**Form Submission:**
- Fill form fields → **STOP before submit**

### Safety Guarantees

The skill will **NEVER:**
- ❌ Complete actual purchases
- ❌ Create real accounts
- ❌ Post publicly
- ❌ Send emails/messages
- ❌ Subscribe to newsletters

The skill will **ALWAYS:**
- ✅ Test up to (but not including) final action
- ✅ Verify final button exists and is accessible
- ✅ Document safety stop in observations  

## Installation

### Step 1: Install Python packages

```bash
pip3 install nova-act pydantic playwright
```

### Step 2: Install Playwright browser

```bash
playwright install chromium
```

> **Note:** On Linux, you may also need system dependencies:
> `sudo playwright install-deps chromium`

### Step 3: Configure API key

```bash
# Create config directory and file
mkdir -p ~/.openclaw/config
cat > ~/.openclaw/config/nova-act.json << EOF
{
  "apiKey": "your-nova-act-api-key-here"
}
EOF
```

Then:
1. Get your Nova Act API key from [AWS Console](https://console.aws.amazon.com/)
2. Edit `~/.openclaw/config/nova-act.json`
3. Replace `"your-nova-act-api-key-here"` with your actual key

### Requirements

- **Python 3.8+** (Python 3.12+ recommended)
- **~300MB disk space** for Playwright browser
- **Nova Act API key** from [AWS Console](https://console.aws.amazon.com/)

## Usage

Ask your OpenClaw agent:

```
Test https://example.com for usability
Run a usability test on example.com
```

The agent will:
1. Analyze the page structure
2. Generate contextual personas
3. Create realistic test cases
4. Run adaptive browser tests
5. Auto-generate an HTML report

## Example Output

**Test Results:**
- **Tech-savvy developer**: 3/3 tasks ✅ - Found docs, playground, value proposition
- **Business decision-maker**: 1/3 tasks - Found value prop, ❌ no pricing page, ❌ getting started not implemented
- **Beginner user**: 1/3 tasks - Found value prop, ❌ no tutorials, ❌ no help/support

**Report includes:**
- Executive summary with success rates
- Detailed step-by-step observations
- Links to Nova Act HTML trace files for session replay
- Recommendations for improvements

## How It Works

### Phase 1: Data Collection (Script)
1. **Page Analysis**: Captures title, navigation, key elements
2. **Persona Generation**: Creates contextual user types based on page content
3. **Test Case Creation**: Generates realistic tasks per persona
4. **Browser Automation**: Nova Act executes simple browser commands
5. **Raw Data Capture**: Saves responses with `needs_agent_analysis: true`

### Phase 2: Agent Interpretation (You)
6. **Read JSON Results**: Load `test_results_adaptive.json`
7. **Interpret Responses**: For each step, determine if `raw_response` indicates success
8. **Set Goal Achievement**: Mark `goal_achieved: true/false` on each step
9. **Calculate Success**: Set `overall_success` based on goals achieved

### Phase 3: Report Generation (Agent)
10. **Call Report Generator**: Pass interpreted results to `generate_enhanced_report()`
11. **View Report**: HTML shows ✅ PASSED / ❌ FAILED / ⏳ PENDING

### Report Status Indicators

| Status | Meaning |
|--------|---------|
| ✅ PASSED | Agent interpreted, goals achieved |
| ❌ FAILED | Agent interpreted, goals not achieved |
| ⏳ PENDING | Awaiting agent interpretation |

### Why "Dynamic"?

Unlike traditional testing tools with hardcoded scenarios, this skill **generates the test strategy at runtime**:

- **Hardcoded approach** (old way): `if test_case == "find docs": ask "Do you see Documentation?"`
- **Dynamic approach** (this skill): AI analyzes the test case + persona + page context → generates contextual exploration steps with fallback strategies

This means the skill adapts to ANY website without requiring updates to hardcoded logic!

## Nova Act Quirks

Nova Act uses **exact text matching** - if the page says "Docs" but you ask for "Documentation", it returns FALSE. This skill handles that by:
- Trying multiple variations per test ("Documentation" → "Docs" → "API" → "Developer")
- Iterative exploration rather than rigid scripts
- Logging all attempts for debugging

## Files

- `SKILL.md` - Main skill instructions for OpenClaw agents
- `scripts/run_adaptive_test.py` - Adaptive testing engine
- `scripts/enhanced_report_generator.py` - HTML report generator with trace links
- `scripts/trace_finder.py` - Extracts trace file paths from Nova Act output
- `references/nova-act-cookbook.md` - Nova Act usage patterns and quirks
- `references/persona-examples.md` - Sample personas for different site types
- `assets/report-template.html` - HTML report template

## Contributing

Found a bug? Have an improvement? Submit a PR or open an issue on GitHub.

## License

MIT

## Credits

Built for OpenClaw by Adi using Amazon Nova Act SDK.


path: .gitignore
content: # Python
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
*.so
*.egg
*.egg-info/
dist/
build/

# Virtual environments
venv/
env/
ENV/

# IDE
.vscode/
.idea/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db

# Test outputs
test_results*.json
*_report.html
act_*_output.html


path: CHANGELOG.md
content: # Changelog

All notable changes to the Nova Act Usability Testing skill will be documented in this file.

## [2.0.0] - 2026-02-06

### 🎯 Major: Agent-Driven Interpretation (Breaking Change)

**Problem:** The script was setting `overall_success: false` always and attempting hardcoded regex-based interpretation of responses. This was wrong because:
1. Hardcoded patterns can't understand context
2. Extra Claude API calls from Python are wasteful (agent is already running)
3. Reports showed incorrect pass/fail status

**Solution:** Complete separation of concerns:
- **Script** collects raw data only → outputs JSON with `needs_agent_analysis: true`
- **Agent** (Claude) interprets responses → sets `goal_achieved` and `overall_success`
- **Agent** calls report generator → produces accurate HTML report

### Architecture: Three-Phase Flow

```
┌─────────────────────────────────────────────────────────────────┐
│ Phase 1: DATA COLLECTION (run_adaptive_test.py)                 │
│ - Runs Nova Act browser automation                              │
│ - Captures raw_response from each step                          │
│ - Sets api_success (did API call work?)                         │
│ - Sets needs_agent_analysis: true                               │
│ - Outputs: test_results_adaptive.json                           │
│ - Does NOT interpret success/failure                            │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 2: AGENT INTERPRETATION (orchestrating AI agent)          │
│ - Reads JSON results                                            │
│ - For each step: interprets raw_response contextually           │
│ - Sets goal_achieved: true/false based on meaning               │
│ - Sets overall_success based on goals achieved                  │
│ - Saves updated JSON                                            │
│ - NO regex, NO hardcoded patterns, NO extra API calls           │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 3: REPORT GENERATION (enhanced_report_generator.py)       │
│ - Reads interpreted JSON                                        │
│ - Shows ✅ PASSED / ❌ FAILED based on goal_achieved            │
│ - Shows ⏳ PENDING if agent hasn't interpreted yet              │
│ - Global recording numbering (1→N across all tests)             │
└─────────────────────────────────────────────────────────────────┘
```

### Changed

#### run_adaptive_test.py
- **Removed** `interpret_step_success()` function (hardcoded interpretation)
- **Removed** automatic `overall_success` calculation
- Script now outputs raw data with `needs_agent_analysis: true`
- Completion status based on API success only, not goal achievement
- Agent must interpret and set final values

#### enhanced_report_generator.py
- **Added** `goal_achieved` field support (agent-set interpretation)
- **Added** "⏳ PENDING" status for un-interpreted tests
- **Added** "Awaiting agent interpretation" warnings on steps
- **Fixed** recording numbering: now globally sequential (1, 2, 3... not restarting per test)
- **Added** CSS for `.pending` status styling
- Smart status detection: error states show FAILED, interpreted show PASSED/FAILED, uninterpreted show PENDING

#### SKILL.md
- **Added** complete agent workflow documentation with code examples
- **Updated** "Complete Analysis Workflow" section with mandatory interpretation steps
- **Added** interpretation code template for agents to use
- Clarified that agent MUST interpret before generating final report

### Key Data Fields

| Field | Set By | Meaning |
|-------|--------|---------|
| `raw_response` | Script | Actual text Nova Act returned |
| `api_success` | Script | Did the API call complete? |
| `needs_agent_analysis` | Script | Always `true` - agent must interpret |
| `goal_achieved` | Agent | Did the response indicate success? |
| `goals_achieved` | Agent | Count of steps where goal was achieved |
| `overall_success` | Agent | Test passed (≥50% goals achieved) |

### Why This Matters

1. **No hardcoded interpretation** - Agent understands "Leaderboard, News, Stats" is useful content
2. **No wasted API calls** - Agent doing the work is already running
3. **Correct reports** - Tests that found content show ✅, tests with "No" show ❌
4. **Transparency** - PENDING status shows when interpretation is needed

---

## [2.0.0-beta] - 2026-02-05

### 🎯 Simplified Nova Act Prompts

**Problem:** Prompts were asking Nova Act to reason about personas and usability, e.g.:
- "As a tournament_follower with high technical skills, can you easily accomplish this task?"

This is wrong - Nova Act is a browser automation tool, not a reasoning engine.

**Solution:** Nova Act now gets simple, direct browser commands. The Claude agent handles all reasoning.

### Changed
- **Cookbook updated**: Added "Nova Act is NOT a Reasoning Engine" as Principle #0
  - ❌ "As a beginner user, can you easily find the docs?"
  - ✅ "Click the Documentation link in the navigation"
- **dynamic_exploration.py**: Generic fallback now generates direct browser commands
  - Removed persona-specific prompt generation
  - `adapt_prompt_for_persona()` now returns prompts unchanged
- **response_interpreter.py**: `generate_alternative_approach()` now strips persona prefixes
  - Alternative prompts are simple: "Check the navigation menu for: {task}"
- **SKILL.md**: Added "Keep Nova Act Prompts Simple" section
  - Clear examples of wrong vs right prompts
  - Explains correct workflow: Agent reasons → Nova Act executes → Agent interprets

### Key Principle
- **Agent decides** what to test (based on persona goals)
- **Nova Act executes** simple browser tasks
- **Agent interprets** results in persona context

---

## [1.3.1] - 2026-02-05

### 🛡️ Graceful Shutdown & Partial Reports

**Problem:** Tests could be killed (timeout/signal) mid-run with no report generated.

**Solution:** Added signal handlers and atexit hooks to generate partial reports on interruption.

### Added
- **Signal handling**: Catches SIGTERM/SIGINT and generates report before exit
- **Partial report indicator**: HTML report clearly shows when incomplete
  - Yellow warning banner at top
  - Shows "X of Y planned tests completed"
- **Progress tracking**: Global state tracks completed vs planned tests
- **Intermediate saves**: Results saved after each test completion

### Changed
- Recommended timeout updated to **30 minutes** (was 15 min)
- SKILL.md now includes timeout guidance section
- Report generator accepts `_partial_report` flag in page_analysis

### Fixed
- **Simplified page analysis**: Nova Act now just reports what's visible
  - Extracts: title, navigation, purpose, visible sections
  - Removed hardcoded key_elements checks (pricing/docs/demo)
  - Orchestrating AI agent interprets the data and decides what matters
  - Nova Act does what it's good at (browser interaction), agent does reasoning

---

## [1.3.0] - 2026-02-05

### 🎯 Major: Semantic Response Interpretation

**Critical Fix:** Nova Act returning "No" was being treated as success because the API call worked.
Now we interpret responses semantically to determine if the **goal was actually achieved**.

### Added
- **Agent-Analyzed Results**: The orchestrating AI agent (OpenClaw/Claude) now interprets test results
  - Script returns raw responses with `needs_agent_analysis: true`
  - Agent determines `goal_achieved` based on response content
  - No external API calls needed - uses the agent that's already running!
  
- **Response Interpreter** (`response_interpreter.py`): Structures data for agent analysis
  - Captures raw Nova Act responses
  - Detects obvious negatives for automatic retries ("No", "not found")
  - Provides prompts/templates for agent analysis
  - Returns `api_success` (API worked) vs `goal_achieved` (agent determines)

- **Adaptive Exploration**: Up to 3 different approaches per test step
  - If first approach fails, tries alternative strategies automatically
  - Strategy 1: Look in navigation vs content (or vice versa)
  - Strategy 2: Scroll and look again
  - Strategy 3: Broaden the search terms
  - `generate_alternative_approach()` creates context-aware retries

- **New Result Fields**:
  - `goal_achieved`: Boolean - did we find what we were looking for?
  - `goals_achieved`: Count of steps where goal was achieved
  - `api_successes`: Count of steps where API call succeeded
  - `attempts`: List of all attempts per step with prompts and responses

### Changed
- `execute_exploration_step()` → `execute_exploration_step_adaptive()`
  - Uses `safe_act_get` for queries (captures actual response text)
  - Interprets responses before marking success
  - Retries with alternative approaches when goal not achieved
  
- **Success Calculation**: Now based on `goal_achieved`, not just API success
  - "No" response → `success=True, goal_achieved=False` → Test step FAILED
  - "Yes, I found X" → `success=True, goal_achieved=True` → Test step PASSED

- **Test Output**: Shows "goals achieved" instead of "steps successful"

### Fixed
- **CRITICAL**: Tests no longer pass when Nova Act returns negative answers
- Steps that return "No" or "not found" are now correctly marked as failed
- Overall test success now reflects whether the user's goal was accomplished

---

## [1.2.0] - 2026-02-05

### 🎯 Major Features

#### AI Agent-Orchestrated Persona Generation
- **AI agent (Claude) generates personas** and passes them as JSON to the test script
- Removes duplicate API calls (agent is already Claude)
- Better context (agent has conversation history, domain knowledge)
- Script accepts 3 argument types:
  1. **JSON file**: `personas.json` (recommended)
  2. **JSON string**: `'[{"name": "...", ...}]'` (recommended)
  3. **Simple description**: `"golf enthusiast"` (fallback)

#### Workflow Testing with Safety Guardrails
- Complete user journey testing: booking, checkout, posting, signup, form submission
- **Automatic safety stops** before material impact (payment, publishing, account creation)
- 6 workflow types detected and tested appropriately
- Observes but doesn't execute final actions with material impact

#### Enhanced Report Generation
- Per-test trace file links (not just global)
- WSL-compatible file paths (`file://wsl$/Ubuntu/...`)
- Dynamic page analysis based on site type
- Session recordings section with clickable trace links

### 🔧 Bug Fixes

#### Critical Fixes
- **safe_act returns observations**: Added `ActResult`/`QueryResult` dataclasses with observation tracking
- **Trace files filtered by run**: Only includes traces from current test run (not old sessions)
- **Executive Summary rendering**: Fixed f-string evaluation in report templates
- **Error message handling**: Fixed type mismatch in safe_act error handling

#### Code Quality Improvements
- **Consistent error handling**: Standardized on Result dataclass types
- **Safe JSON extraction**: Replaced regex with multi-strategy `extract_json_safely()`
- **Cookbook integration**: Prompts now use cookbook guidance for better Nova Act interactions
- **Configurable defaults**: Timeouts and thresholds now centralized as constants
- **URL parameter passing**: Removed global variable mutation

### 📦 Installation & Setup
- Manual installation via `pip3 install nova-act pydantic playwright`
- Playwright browser: `playwright install chromium`
- Config file at `~/.openclaw/config/nova-act.json`
- 60-second status updates during test execution

### 📝 Documentation
- Complete SKILL.md rewrite with AI agent guidance
- Persona generation examples by industry
- Workflow testing patterns and safety guidelines
- Technical overview document

### Technical Details

#### New Files/Functions
- `extract_json_safely()`: Multi-strategy JSON extraction
- `parse_cookbook_hints()`: Extract guidance from cookbook
- `apply_cookbook_guidance()`: Apply best practices to prompts
- `ActResult`/`QueryResult`: Dataclasses for consistent returns
- Legacy tuple compatibility functions for backwards compat

#### Key Parameters
- `DEFAULT_TIMEOUT = 20` seconds
- `DEFAULT_MAX_RETRIES = 1`
- `SLOW_OPERATION_THRESHOLD = 15` seconds
- Test start time filtering for traces

---

## [1.1.0] - 2026-02-04

### Added
- Initial release with adaptive usability testing
- Dynamic persona generation
- Contextual test case generation
- Fully dynamic exploration strategies
- Robust error handling
- HTML report generation with trace links

## [1.0.0] - 2026-02-03

### Added
- Basic Nova Act usability testing framework
- Manual persona configuration
- Hardcoded test strategies
- Basic reporting


path: assets/report-template.html
content: <!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Usability Testing Report</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        
        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
            line-height: 1.6;
            color: #333;
            background: #f5f5f5;
            padding: 20px;
        }
        
        .container {
            max-width: 1200px;
            margin: 0 auto;
            background: white;
            padding: 40px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
            border-radius: 8px;
        }
        
        h1 {
            color: #2c3e50;
            border-bottom: 3px solid #3498db;
            padding-bottom: 15px;
            margin-bottom: 30px;
            font-size: 2.5em;
        }
        
        h2 {
            color: #34495e;
            margin-top: 40px;
            margin-bottom: 20px;
            font-size: 1.8em;
            border-bottom: 2px solid #ecf0f1;
            padding-bottom: 10px;
        }
        
        h3 {
            color: #2c3e50;
            margin-top: 25px;
            margin-bottom: 15px;
            font-size: 1.3em;
        }
        
        .executive-summary {
            background: #ecf0f1;
            padding: 25px;
            border-radius: 6px;
            margin: 20px 0;
            border-left: 4px solid #3498db;
        }
        
        .executive-summary p {
            margin: 10px 0;
            font-size: 1.1em;
        }
        
        .executive-summary strong {
            color: #2c3e50;
        }
        
        .persona-section {
            background: #fff;
            border: 1px solid #ddd;
            border-radius: 6px;
            padding: 20px;
            margin: 20px 0;
            box-shadow: 0 1px 3px rgba(0,0,0,0.05);
        }
        
        .persona-section h3 {
            color: #3498db;
            margin-top: 0;
        }
        
        ul {
            margin: 15px 0;
            padding-left: 25px;
        }
        
        li {
            margin: 8px 0;
            line-height: 1.6;
        }
        
        .success-rate {
            display: inline-block;
            background: #2ecc71;
            color: white;
            padding: 4px 12px;
            border-radius: 4px;
            font-weight: bold;
            margin-left: 10px;
        }
        
        .failure-rate {
            display: inline-block;
            background: #e74c3c;
            color: white;
            padding: 4px 12px;
            border-radius: 4px;
            font-weight: bold;
            margin-left: 10px;
        }
        
        .recommendations {
            background: #fff3cd;
            border-left: 4px solid #ffc107;
            padding: 20px;
            margin: 20px 0;
            border-radius: 4px;
        }
        
        .friction-points {
            background: #f8d7da;
            border-left: 4px solid #e74c3c;
            padding: 20px;
            margin: 20px 0;
            border-radius: 4px;
        }
        
        .footer {
            margin-top: 50px;
            padding-top: 20px;
            border-top: 1px solid #ddd;
            text-align: center;
            color: #7f8c8d;
            font-size: 0.9em;
        }
        
        @media print {
            body {
                background: white;
                padding: 0;
            }
            
            .container {
                box-shadow: none;
                padding: 20px;
            }
        }
    </style>
</head>
<body>
    <div class="container">
        <h1>🦅 Usability Testing Report</h1>
        
        {{EXECUTIVE_SUMMARY}}
        
        {{PERSONA_FINDINGS}}
        
        <div class="friction-points">
            {{FRICTION_POINTS}}
        </div>
        
        <div class="recommendations">
            {{RECOMMENDATIONS}}
        </div>
        
        <div class="footer">
            <p>Generated by Nova Act Usability Testing Suite | Powered by OpenClaw</p>
        </div>
    </div>
</body>
</html>


path: references/persona-examples.md
content: # Persona Examples

Sample digital twin personas for usability testing. Use these as templates or inspiration when generating custom personas.

## Tech-Savvy Millennial

**Name:** Alex Chen  
**Age:** 28  
**Tech Proficiency:** High  

**Goals:**
- Complete tasks efficiently with minimal clicks
- Discover and use advanced features
- Keyboard shortcuts and power-user features

**Behaviors:**
- Scans interfaces quickly, doesn't read everything
- Expects instant feedback and fast load times
- Comfortable experimenting with new features
- Multi-tasks frequently

**Frustrations:**
- Slow performance or loading
- Unnecessary confirmation dialogs
- Hidden or hard-to-find advanced features
- Over-simplified interfaces that lack depth

**Accessibility Needs:** None typically

---

## Elderly First-Time User

**Name:** Dorothy Williams  
**Age:** 72  
**Tech Proficiency:** Low  

**Goals:**
- Understand how to use the website
- Complete simple tasks confidently
- Avoid making mistakes

**Behaviors:**
- Reads instructions carefully
- Hesitates before clicking unfamiliar buttons
- Prefers familiar UI patterns
- May use mouse exclusively (no keyboard shortcuts)

**Frustrations:**
- Small text or low contrast
- Unclear button labels
- Too many options presented at once
- Technical jargon or unfamiliar terms
- Unexpected behavior or popups

**Accessibility Needs:**
- Large, readable text (16px minimum)
- High contrast colors
- Clear, descriptive labels
- Predictable navigation

---

## Busy Professional

**Name:** Marcus Johnson  
**Age:** 42  
**Tech Proficiency:** Medium  

**Goals:**
- Find information quickly
- Complete tasks on mobile while commuting
- Minimize time spent on administrative tasks

**Behaviors:**
- Scans rather than reads thoroughly
- Often uses mobile devices
- Impatient with multi-step processes
- Expects good search functionality

**Frustrations:**
- Complex forms with many fields
- Poor mobile experience
- Missing or ineffective search
- Buried information requiring many clicks

**Accessibility Needs:**
- Mobile-responsive design
- Clear information hierarchy
- Efficient workflows

---

## Student/Budget-Conscious User

**Name:** Priya Patel  
**Age:** 21  
**Tech Proficiency:** Medium-High  

**Goals:**
- Find free or discounted options
- Understand pricing clearly
- Avoid unwanted commitments or subscriptions

**Behaviors:**
- Compares options carefully
- Reads fine print about pricing
- Sensitive to misleading or hidden fees
- May abandon cart if confused

**Frustrations:**
- Unclear pricing structure
- Hidden fees revealed late
- Aggressive upselling
- Required payment info for free trials

**Accessibility Needs:** Standard

---

## Accessibility-Focused User

**Name:** James Martinez  
**Age:** 35  
**Tech Proficiency:** High  

**Goals:**
- Navigate website using screen reader
- Complete tasks with keyboard only
- Access all functionality without mouse

**Behaviors:**
- Uses keyboard navigation extensively (Tab, Enter, Space)
- Relies on proper ARIA labels and semantic HTML
- Needs clear focus indicators
- Expects logical tab order

**Frustrations:**
- Missing alt text on images
- Keyboard traps or inaccessible widgets
- Poor focus management
- Visual-only information (graphs without data tables)

**Accessibility Needs:**
- Full keyboard navigation
- Screen reader compatibility
- Proper semantic HTML and ARIA labels
- Clear focus indicators
- Text alternatives for visual content

---

## International/Non-Native Speaker

**Name:** Yuki Tanaka  
**Age:** 29  
**Tech Proficiency:** Medium  

**Goals:**
- Understand content in clear, simple language
- Find information despite language barriers
- Complete tasks with minimal text input

**Behaviors:**
- Relies heavily on visual cues and icons
- May use translation tools
- Prefers simple, common words over jargon
- Benefits from clear visual hierarchy

**Frustrations:**
- Complex or idiomatic language
- Country/region-specific assumptions
- Forms requiring specific formats (phone, address)
- Unclear icons without labels

**Accessibility Needs:**
- Clear, simple language
- Visual icons with text labels
- Flexible input formats
- Internationalization support


path: references/nova-act-cookbook.md
content: # Nova Act Cookbook

Best practices for using Amazon Nova Act effectively in usability testing.

## Core Principles

### 0. Nova Act is a Browser Automation Tool, NOT a Reasoning Engine

**CRITICAL:** Nova Act executes browser actions. It should NOT:
- Reason about user personas
- Judge if a task is "easy" or "hard"
- Evaluate usability or user experience
- Decide what's "important" for a user type

**The Claude agent does the reasoning.** Nova Act just clicks, types, and reports what it sees.

❌ **WRONG:** Asking Nova Act to reason
```python
# Don't ask Nova Act to think about personas or UX
nova.act("As a beginner user, can you easily find the documentation?")
nova.act("Would a business professional find the pricing clear?")
nova.act("Is this task accomplishable for someone with low technical skills?")
```

✅ **RIGHT:** Give Nova Act direct browser tasks
```python
# Simple, direct browser commands
nova.act("Click the Documentation link in the navigation")
nova.act("Find and click a link containing 'Pricing'")
nova.act_get("What text is displayed in the main heading?")
nova.act_get("List the navigation menu items visible on this page")
```

**The workflow:**
1. **Agent** decides what to test based on persona (e.g., "Can Dorothy find how to watch golf?")
2. **Agent** generates simple prompts for Nova Act ("Click 'Watch & Listen' in the navigation")
3. **Nova Act** executes the browser task and returns raw results
4. **Agent** interprets results in context of persona ("Dorothy would struggle here because...")

### 1. Nova Act Matching Behavior

**CRITICAL:** Nova Act has two matching modes:

**Exact Matching (with quotes):**
- `"Documentation"` → Only matches the exact word "Documentation"
- `"API Documentation"` → Only matches that exact phrase
- Use for precise element identification

**Loose Matching (without quotes):**
- `Documentation` → Can match "Documentation", "Docs", "API Documentation", etc.
- More flexible, fuzzy matching
- Use for broader searches

❌ **WRONG:** Using quotes when you want flexible matching
```python
# This is too strict - will miss "API Docs", "Developer Docs", etc.
nova.act_get('Is there a link labeled "Documentation"?')
```

✅ **RIGHT:** Use quotes strategically
```python
# Loose matching - finds variations
nova.act_get('Is there a link with Documentation in it?')

# Exact matching - when you know the precise text
nova.act_get('Click the link that says "Sign Up"')
```

**Best Practice:** When searching for elements:
1. Start broad (no quotes): "What links do you see in the navigation?"
2. Use loose matching first: "Is there a link with Documentation?"
3. If you need precision: Use quotes for exact text matching
4. Try variations if first attempt fails

### 1. Break Tasks into Small Steps

Nova Act works most reliably when tasks can be accomplished in **fewer than 30 steps**.

❌ DON'T: Single large act() call
```python
nova.act("book me a hotel that costs less than $100 with highest rating then find car rental and book lunch")
```

✅ DO: Multiple small act() calls
```python
hotel = nova.act_get("book a hotel for $100 or less, return the address")
nova.act(f"book restaurant near {hotel.response} at 12:30pm")
nova.act(f"rent a car near {hotel.response}")
```

### 2. Be Direct and Specific

Make prompts clear about exactly what should happen.

❌ DON'T: Vague instructions
```python
nova.act("Let's see what routes are available")
```

✅ DO: Direct instructions  
```python
nova.act("Navigate to the routes tab")
```

### 3. Extract Information with Schemas

Use `act_get()` with Pydantic schemas for structured data extraction.

```python
from pydantic import BaseModel

class PricingInfo(BaseModel):
    price: float
    currency: str
    features: list[str]

result = nova.act_get(
    "Find the pricing information on this page",
    schema=PricingInfo.model_json_schema()
)

pricing = PricingInfo.model_validate(result.parsed_response)
```

### 4. Observe and Analyze Each Step

After each `act()` call, analyze the result before deciding the next action.

```python
# Navigate to page
nova.act("Go to the pricing page")

# Check if successful
is_found = nova.act_get(
    "Is there a pricing table visible on this page?",
    schema=BOOL_SCHEMA
)

if is_found.parsed_response:
    # Extract pricing
    pricing = nova.act_get("Extract all pricing tiers", schema=PricingSchema)
else:
    # Try alternative path
    nova.act("Look for a 'Plans' or 'Subscribe' link")
```

### 5. Use Playwright for Fine Control

For sensitive data or precise actions, use Playwright APIs directly:

```python
# Focus on field with act(), then type with Playwright
nova.act("click on the password field")
nova.page.keyboard.type(password)  # Doesn't send over network
nova.act("click sign in")
```

## Common Patterns

### Navigation Testing
```python
# Check if navigation is clear
nova.act("Navigate to the main menu")
is_clear = nova.act_get(
    "Are the menu items clearly labeled and easy to understand?",
    schema=BOOL_SCHEMA
)
```

### Form Filling
```python
# Fill forms step by step
nova.act("Click on the signup form")
nova.act("Enter email address [REDACTED_SECRET]")
nova.act("Enter name 'John Doe'")
result = nova.act_get("Is there a clear submit button visible?")
```

### Search Testing
```python
# Test search functionality
nova.act("search for 'pricing'. press enter to initiate search")
results = nova.act_get(
    "Are search results relevant to 'pricing'?",
    schema=BOOL_SCHEMA
)
```

### Information Architecture
```python
# Test if information is findable
time_to_find = measure_time()
nova.act("Find the contact information")
duration = time_to_find()

# Note: If it took >30 steps or multiple attempts, that's a UX issue
```

## Persona Adaptation

Adjust act() prompts based on persona tech proficiency:

**Low proficiency (elderly user):**
```python
# More explicit, step-by-step
nova.act("Look for a button that says 'Contact' or 'Contact Us'")
nova.act("Click on the Contact button")
```

**High proficiency (power user):**
```python
# Test efficiency paths
nova.act("Find keyboard shortcut for search")
nova.act("Use keyboard shortcut to open search")
```

## Error Handling

```python
try:
    result = nova.act("Complete checkout")
except ActAgentError as e:
    # Agent couldn't complete - this is a UX issue!
    note_friction_point(
        "Checkout flow failed - agent couldn't figure it out",
        error=str(e)
    )
```

## Iterative Exploration Pattern

For usability testing, use an **explore-adapt-verify** approach:

**Step 1: Broad Discovery**
```python
# Don't assume - ask what's there
nav_items = nova.act_get("What navigation links do you see at the top?", schema=StringArraySchema)
```

**Step 2: Adapt Based on Findings**
```python
# If you found "API Documentation", now search for it specifically
if "API" in nav_items:
    nova.act("Click on the link containing 'API'")
```

**Step 3: Verify Hypothesis Multiple Ways**
```python
# Hypothesis: "Users can't find pricing"
# Approach 1: Check nav
has_nav_pricing = nova.act_get("Is 'Pricing' in the navigation?", schema=BOOL_SCHEMA)

# Approach 2: Check page body
if not has_nav_pricing:
    nova.act("Scroll down")
    has_body_pricing = nova.act_get("Do you see pricing or cost information?", schema=BOOL_SCHEMA)

# Approach 3: Check variations
if not has_body_pricing:
    has_plans = nova.act_get("Do you see 'Plans' or 'Subscribe'?", schema=BOOL_SCHEMA)
```

## Nova Act Trace Files

**Nova Act automatically generates detailed HTML trace files** for every session!

These trace files contain:
- Screenshots at each step
- AI reasoning and decisions
- Actions taken and their results
- Full timeline of the session

**Where to find them:**
- Set `logs_directory` when creating NovaAct instance
- Files are named: `act_<uuid>_output.html`
- Each test captures these files
- **Linked automatically in the HTML report**

```python
with NovaAct(
    starting_page=url,
    logs_directory="/path/to/logs"  # Set custom directory
) as nova:
    nova.act("Navigate somewhere")
    # Trace file automatically created
```

## Workflow Testing (Not Just Information Finding)

**Modern usability testing goes beyond "can users find the docs?"** — it tests whether users can **complete real tasks end-to-end**.

### Real User Journeys to Test

**Flight Booking Sites:**
- Search for flights (origin, destination, dates)
- Filter results (price, time, airline)
- Select a flight
- Fill passenger information
- **STOP before payment**

**E-Commerce Sites:**
- Search for product
- Add to cart
- Modify quantity
- Proceed to checkout
- Fill shipping info
- **STOP before payment/order placement**

**Social Media Platforms:**
- Create new post
- Add text content
- Attach media (if applicable)
- Preview post
- **STOP before publishing**

**SaaS/Software Sites:**
- Sign up flow (fill registration form)
- **STOP before final submission** (unless explicitly testing signup)
- Demo/trial access workflow
- Feature exploration
- Settings configuration

**Newsletter/Form Submissions:**
- Fill out form fields
- Verify validation works
- **STOP before final submit button** (unless explicitly testing)

### Safety Guardrails - STOP Before Material Impact ⚠️

**ALWAYS stop testing before actions that cause:**
- 💳 **Monetary impact**: Charges, purchases, subscriptions, donations
- 📧 **External communication**: Sending emails, posting publicly, messaging real users
- 🔐 **Account creation**: Creating real accounts (use "test" flows if available)
- 🗑️ **Data modification**: Deleting, editing, or corrupting existing data
- 📝 **Legal commitment**: Agreeing to terms, signing contracts, submitting official forms
- 📬 **Spam/annoyance**: Newsletter signups, notification opt-ins

**How to Test Safely:**

1. **Navigate TO the final step** (checkout page, publish screen, submit button)
2. **Verify the final action is accessible** (button exists, is enabled, is clear)
3. **Use `act_get()` to observe without acting**:
   ```python
   # Good: Observe the checkout button
   checkout_ready = nova.act_get(
       "Is there a 'Complete Purchase' or 'Pay Now' button visible and enabled?",
       schema=BOOL_SCHEMA
   )
   ```
4. **Document readiness but DO NOT CLICK**
5. **In observations, explicitly note the safety stop**:
   ```python
   observations.append({
       "step": "verify_checkout_accessible",
       "action": "Confirmed payment button is reachable",
       "success": checkout_ready.parsed_response,
       "notes": "✅ Workflow complete up to payment. STOPPED per safety guidelines - no actual purchase made."
   })
   ```

**Example: Safe Flight Booking Test**

```python
def test_flight_booking_workflow(nova, persona):
    """Test flight booking WITHOUT actually booking."""
    
    observations = []
    
    # Step 1: Search
    nova.act("Find the flight search form")
    nova.act("Enter departure city: New York")
    nova.act("Enter destination city: Los Angeles")
    nova.act("Select departure date 2 weeks from today")
    nova.act("Select return date 3 weeks from today")
    nova.act("Click the search button")
    
    search_success = nova.act_get(
        "Are flight results displayed with prices and times?",
        schema=BOOL_SCHEMA
    )
    
    observations.append({
        "step": "search_flights",
        "action": "Searched for NYC to LAX flights",
        "success": search_success.parsed_response,
        "notes": "Flight search returned results" if search_success.parsed_response else "Search failed or no results"
    })
    
    if not search_success.parsed_response:
        return observations  # Can't continue if search failed
    
    # Step 2: Select flight
    nova.act("Click on the first available flight option")
    
    flight_selected = nova.act_get(
        "Is the selected flight now highlighted or showing details?",
        schema=BOOL_SCHEMA
    )
    
    observations.append({
        "step": "select_flight",
        "action": "Selected first available flight",
        "success": flight_selected.parsed_response,
        "notes": "Flight selection worked" if flight_selected.parsed_response else "Could not select flight"
    })
    
    # Step 3: Fill passenger info
    nova.act("Click continue or proceed to passenger information")
    nova.act("Enter passenger name: John Doe")
    nova.act("Enter email: [REDACTED_SECRET]")
    nova.act("Enter phone: 555-0123")
    
    form_filled = nova.act_get(
        "Are all passenger information fields filled out?",
        schema=BOOL_SCHEMA
    )
    
    observations.append({
        "step": "fill_passenger_info",
        "action": "Filled passenger details",
        "success": form_filled.parsed_response,
        "notes": "Form filled successfully" if form_filled.parsed_response else "Form filling incomplete"
    })
    
    # Step 4: Verify checkout is reachable (BUT DON'T CLICK)
    checkout_accessible = nova.act_get(
        "Is there a 'Continue to Payment', 'Proceed to Checkout', or 'Complete Booking' button visible?",
        schema=BOOL_SCHEMA
    )
    
    observations.append({
        "step": "verify_payment_reachable",
        "action": "Verified checkout button exists",
        "success": checkout_accessible.parsed_response,
        "notes": "⚠️ SAFETY STOP: Checkout accessible but NOT clicked. Booking workflow verified up to payment step. No actual booking made."
    })
    
    # Overall success: Could we reach checkout?
    workflow_success = checkout_accessible.parsed_response
    
    return observations, workflow_success
```

**Example: Safe E-Commerce Test**

```python
def test_ecommerce_purchase(nova, persona):
    """Test product purchase workflow WITHOUT completing transaction."""
    
    # Search for product
    nova.act("Search for 'laptop'")
    nova.act("Click on the first product in search results")
    
    # Add to cart
    nova.act("Click the 'Add to Cart' button")
    
    cart_success = nova.act_get(
        "Is there confirmation the item was added to cart?",
        schema=BOOL_SCHEMA
    )
    
    # Proceed to checkout
    nova.act("Click on the cart icon or 'View Cart' button")
    nova.act("Click 'Proceed to Checkout' or 'Checkout'")
    
    # Fill shipping (use fake data)
    nova.act("Fill shipping name: Test User")
    nova.act("Fill address: 123 Test St")
    nova.act("Fill city: Test City")
    nova.act("Fill zip: 12345")
    
    # Verify we reached payment step
    payment_page = nova.act_get(
        "Is there a payment method section or credit card form visible?",
        schema=BOOL_SCHEMA
    )
    
    # STOP HERE - do not enter payment info or submit
    return {
        "success": payment_page.parsed_response,
        "notes": "⚠️ SAFETY STOP: Reached payment page. Cart and checkout flow functional. NO PURCHASE MADE."
    }
```

### Detecting Material Impact Actions

When generating test strategies, identify if the test case involves:

```python
MATERIAL_IMPACT_KEYWORDS = [
    # Monetary
    "buy", "purchase", "checkout", "pay", "subscribe", "donate",
    # Communication
    "post", "publish", "share", "send", "email", "message",
    # Account creation
    "sign up", "register", "create account",
    # Submissions
    "submit", "apply", "enroll",
    # Newsletter/notifications
    "subscribe", "sign up for newsletter", "get updates"
]

def requires_safety_stop(test_case: str) -> bool:
    """Check if test case involves material impact."""
    test_lower = test_case.lower()
    return any(keyword in test_lower for keyword in MATERIAL_IMPACT_KEYWORDS)
```

If detected, **modify the test strategy** to:
1. Include all steps UP TO the final action
2. Replace final action with verification
3. Document the safety stop in observations

## Usability Observations

Document friction points as you observe them:

- **Task failed after multiple approaches** = Major UX issue
- **Found after 2+ attempts** = Moderate UX issue (discoverability)
- **Found but unclear label** = Minor UX issue
- **Small text, poor contrast** = Accessibility issue
- **>20 steps for simple task** = Efficiency issue
- **Workflow reachable but confusing** = Navigation/flow issue
- **Form validation unclear or missing** = Usability issue
- **Mobile responsiveness problems** = Accessibility issue


path: scripts/nova_session.py
content: #!/usr/bin/env python3
"""
Thin wrapper for Nova Act session management.
Provides primitives for AI orchestration.
"""

import json
import os
from pathlib import Path
from typing import Optional, Dict, Any
from contextlib import contextmanager
import glob


def load_config() -> Dict[str, Any]:
    """Load Nova Act configuration."""
    config_path = Path.home() / ".openclaw" / "config" / "nova-act.json"
    if not config_path.exists():
        raise FileNotFoundError(
            "Nova Act config not found at ~/.openclaw/config/nova-act.json"
        )
    
    with open(config_path, 'r') as f:
        return json.load(f)


@contextmanager
def nova_session(starting_page: str, headless: bool = True, logs_dir: Optional[str] = None):
    """
    Create a Nova Act session context.
    
    Args:
        starting_page: URL to start from
        headless: Run in headless mode (default True)
        logs_dir: Directory to store Nova Act trace files (optional)
    
    Yields:
        NovaAct instance for orchestration
    
    Example:
        with nova_session("https://example.com", logs_dir="/path/to/logs") as nova:
            nova.act("Navigate to pricing")
            result = nova.act_get("Extract pricing", schema=PricingSchema)
    """
    try:
        from nova_act import NovaAct
    except ImportError:
        raise ImportError("Nova Act SDK not installed. Run: pip install nova-act")
    
    # Load API key and set as environment variable
    config = load_config()
    os.environ['NOVA_ACT_API_KEY'] = config['apiKey']
    
    # Set up logs directory (default to current working directory)
    if logs_dir is None:
        logs_dir = os.path.join(os.getcwd(), "nova_act_logs")
    
    # Create logs directory if it doesn't exist
    os.makedirs(logs_dir, exist_ok=True)
    
    # Use NovaAct directly with API key (no Workflow needed)
    with NovaAct(
        starting_page=starting_page,
        tty=not headless,
        logs_directory=logs_dir
    ) as nova:
        yield nova


if __name__ == "__main__":
    # Example usage
    import sys
    
    if len(sys.argv) < 2:
        print("Usage: nova_session.py <url>")
        sys.exit(1)
    
    url = sys.argv[1]
    
    with nova_session(url, headless=False) as nova:
        print(f"Session started at {url}")
        print("Nova Act instance ready for commands")
        print("Use nova.act() for actions and nova.act_get() for extraction")


path: scripts/run_adaptive_test.py
content: #!/usr/bin/env python3
"""
Adaptive AI-orchestrated usability testing - FULLY DYNAMIC VERSION.
Exploration strategy, prompts, and questions are all generated per test case.
"""

import sys
import os
import time
import json
import glob
import re
import signal
import atexit
from datetime import datetime
from pydantic import BaseModel
from typing import List, Dict, Optional

# Global state for graceful shutdown
_shutdown_state = {
    'all_results': [],
    'page_analysis': None,
    'test_start_time': None,
    'total_planned_tests': 0,
    'completed_tests': 0,
    'interrupted': False
}

def _generate_partial_report():
    """Generate report from whatever results we have on shutdown."""
    if _shutdown_state['interrupted']:
        return  # Already handled
    
    _shutdown_state['interrupted'] = True
    
    results = _shutdown_state['all_results']
    page_analysis = _shutdown_state['page_analysis']
    test_start_time = _shutdown_state['test_start_time']
    
    if not results or not page_analysis:
        print("\n⚠️ No results to save on shutdown")
        return
    
    print("\n" + "="*60)
    print("⚠️ INTERRUPTED - Generating partial report...")
    print("="*60)
    
    try:
        # Save whatever results we have
        with open(RESULTS_FILE, 'w') as f:
            json.dump(results, f, indent=2)
        print(f"✅ Saved {len(results)} test results to {RESULTS_FILE}")
        
        # Get trace files from this run
        if test_start_time:
            trace_pattern = os.path.join(LOGS_DIR, "**", "*.html")
            all_traces = glob.glob(trace_pattern, recursive=True)
            traces = sorted(
                [t for t in all_traces if os.path.getmtime(t) >= test_start_time],
                key=os.path.getmtime
            )
        else:
            traces = []
        
        # Mark page analysis as partial
        page_analysis['_partial_report'] = True
        page_analysis['_completed_tests'] = _shutdown_state['completed_tests']
        page_analysis['_total_planned_tests'] = _shutdown_state['total_planned_tests']
        
        # Generate report
        from enhanced_report_generator import generate_enhanced_report
        report_path = generate_enhanced_report(page_analysis, results, traces)
        
        print(f"✅ Generated PARTIAL report: {report_path}")
        print(f"   ({_shutdown_state['completed_tests']}/{_shutdown_state['total_planned_tests']} tests completed)")
        
    except Exception as e:
        print(f"❌ Failed to generate partial report: {e}")

def _signal_handler(signum, frame):
    """Handle SIGTERM/SIGINT gracefully."""
    sig_name = signal.Signals(signum).name
    print(f"\n\n🛑 Received {sig_name} - shutting down gracefully...")
    _generate_partial_report()
    sys.exit(128 + signum)

# Register signal handlers
signal.signal(signal.SIGTERM, _signal_handler)
signal.signal(signal.SIGINT, _signal_handler)

# Also generate report on normal exit if we have partial results
atexit.register(_generate_partial_report)

# Dynamic path resolution
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
SKILL_DIR = os.path.dirname(SCRIPT_DIR)
WORKSPACE_DIR = os.getcwd()

sys.path.insert(0, SCRIPT_DIR)

from nova_session import nova_session
from enhanced_report_generator import generate_enhanced_report
from trace_finder import get_session_traces
from safe_nova_wrapper import safe_act_tuple as safe_act, safe_act_get_tuple as safe_act_get, safe_scroll, is_session_healthy, ActResult
from response_interpreter import interpret_response, generate_alternative_approach, format_for_agent_analysis
from dynamic_exploration import generate_exploration_strategy, adapt_prompt_for_persona
from status_reporter import start_status_reporter, stop_status_reporter, update_status, mark_complete, emit_final

WEBSITE_URL = "https://nova.amazon.com/act"  # Default
RESULTS_FILE = os.path.join(WORKSPACE_DIR, "test_results_adaptive.json")
LOGS_DIR = os.path.join(WORKSPACE_DIR, "nova_act_logs")

def load_cookbook() -> str:
    """Load the Nova Act cookbook for guidance on testing strategies and safety."""
    cookbook_path = os.path.join(SKILL_DIR, "references", "nova-act-cookbook.md")
    try:
        with open(cookbook_path, 'r') as f:
            content = f.read()
            print(f"✅ Loaded Nova Act cookbook ({len(content)} chars)")
            return content
    except Exception as e:
        print(f"⚠️ Could not load cookbook: {e}")
        return ""

class NavigationLinks(BaseModel):
    links: List[str]

class PageAnalysis(BaseModel):
    main_purpose: str
    key_features: List[str]
    target_audience: str

def analyze_page(url: str) -> Dict:
    """
    Step 1: Analyze the page with graceful error handling.
    """
    print(f"\n🔍 ANALYZING PAGE: {url}")
    print("="*60)
    
    analysis = {
        'title': 'Unknown',
        'navigation': [],
        'purpose': 'Unknown purpose',
        'visible_sections': ''
    }
    
    try:
        with nova_session(url, headless=True, logs_dir=LOGS_DIR) as nova:
            print("→ Reading page title and main heading...")
            ok, response, error = safe_act_get(
                nova,
                "What is the main title or headline on this page?",
                schema={"type": "object", "properties": {"title": {"type": "string"}}, "required": ["title"]},
                timeout=20
            )
            if ok and response:
                analysis['title'] = response.get('title', 'Unknown')
                print(f"  Title: {analysis['title']}")
            else:
                print(f"  ⚠️ Could not extract title: {error}")
            
            if is_session_healthy(nova):
                print("→ Analyzing navigation...")
                ok, response, error = safe_act_get(
                    nova,
                    "List all the navigation links you see at the top of the page (just the text of each link, separated by commas)",
                    schema={"type": "object", "properties": {"links": {"type": "string"}}, "required": ["links"]},
                    timeout=20
                )
                if ok and response:
                    nav_text = response.get('links', '')
                    analysis['navigation'] = [link.strip() for link in nav_text.split(',') if link.strip()]
                    print(f"  Navigation: {analysis['navigation']}")
                else:
                    print(f"  ⚠️ Could not extract navigation: {error}")
            
            if is_session_healthy(nova):
                print("→ Understanding page purpose...")
                ok, response, error = safe_act_get(
                    nova,
                    "In one sentence, what does this page help users do?",
                    schema={"type": "object", "properties": {"purpose": {"type": "string"}}, "required": ["purpose"]},
                    timeout=20
                )
                if ok and response:
                    analysis['purpose'] = response.get('purpose', 'Unknown purpose')
                    print(f"  Purpose: {analysis['purpose']}")
                else:
                    print(f"  ⚠️ Could not extract purpose: {error}")
            
            # Get visible sections/areas on the page for agent analysis
            if is_session_healthy(nova):
                print("→ Identifying main content sections...")
                ok, response, error = safe_act_get(
                    nova,
                    "List the main sections or content areas visible on this page (e.g., 'hero banner, news feed, leaderboard sidebar, footer links'). Just describe what you see.",
                    schema={"type": "object", "properties": {"sections": {"type": "string"}}, "required": ["sections"]},
                    timeout=20
                )
                if ok and response:
                    analysis['visible_sections'] = response.get('sections', '')
                    print(f"  Sections: {analysis['visible_sections']}")
                else:
                    print(f"  ⚠️ Could not identify sections: {error}")
            
            # Note: The orchestrating AI agent will analyze this data
            # and determine what's important based on title, navigation, 
            # purpose, and visible sections. No need for hardcoded checks.
    
    except Exception as e:
        print(f"⚠️ Page analysis encountered error: {str(e)}")
        print("   Continuing with default analysis...")
    
    return analysis

def extract_json_safely(text: str) -> Optional[List]:
    """
    Safely extract JSON array from text (Bug #13: Better than regex).
    Tries multiple strategies:
    1. Direct parse (if response is pure JSON)
    2. Find array boundaries and parse
    3. Line-by-line cleanup
    """
    # Strategy 1: Try direct parse
    try:
        result = json.loads(text.strip())
        if isinstance(result, list):
            return result
    except json.JSONDecodeError:
        pass
    
    # Strategy 2: Find array boundaries
    start_idx = text.find('[')
    if start_idx == -1:
        return None
    
    # Find matching closing bracket
    depth = 0
    end_idx = -1
    for i, char in enumerate(text[start_idx:], start_idx):
        if char == '[':
            depth += 1
        elif char == ']':
            depth -= 1
            if depth == 0:
                end_idx = i
                break
    
    if end_idx == -1:
        return None
    
    try:
        result = json.loads(text[start_idx:end_idx + 1])
        if isinstance(result, list):
            return result
    except json.JSONDecodeError:
        pass
    
    # Strategy 3: Clean up common issues
    json_str = text[start_idx:end_idx + 1]
    # Remove trailing commas before ] or }
    cleaned = re.sub(r',\s*([}\]])', r'\1', json_str)
    try:
        result = json.loads(cleaned)
        if isinstance(result, list):
            return result
    except json.JSONDecodeError:
        pass
    
    return None


def infer_plausible_user_types(page_analysis: Dict) -> List[Dict]:
    """
    DEPRECATED (v1.4.0): Use AI agent-generated personas instead.
    
    This fallback uses Claude API to infer user types.
    Prefer passing AI-generated personas via the persona_arg parameter.
    
    Returns list of user type definitions with characteristics and goals.
    """
    print("→ [FALLBACK] Using AI to infer user types (prefer AI agent personas)...")
    
    purpose = page_analysis.get('purpose', 'Unknown')
    title = page_analysis.get('title', 'Unknown')
    navigation = ', '.join(page_analysis.get('navigation', []))
    
    prompt = f"""Based on this website analysis, identify the 3 most plausible user types who would visit this site in real life:

Website Title: {title}
Purpose: {purpose}
Navigation: {navigation}

For each of the 3 user types, provide:
1. archetype: A descriptive identifier (e.g., "frequent_business_traveler", "casual_sports_fan", "power_user")
2. name: A realistic first and last name for this persona
3. description: One sentence describing who they are (e.g., "Business professional who travels frequently for work")
4. age: Typical age (as a number)
5. tech_proficiency: Either "low", "medium", or "high"
6. goals: Array of exactly 3 specific goals this user would have on THIS website

Think about WHO actually uses this type of website. Be specific to this site's content and purpose.

Return EXACTLY 3 user types as a JSON array with this structure:
[
  {{"archetype": "...", "name": "...", "description": "...", "age": 35, "tech_proficiency": "medium", "goals": ["...", "...", "..."]}},
  ...
]"""
    
    try:
        import anthropic
        [REDACTED_SECRET]'ANTHROPIC_API_KEY')
        if not [REDACTED_SECRET]"  ⚠️ ANTHROPIC_API_KEY not set, using fallback personas")
            return []
        
        client = anthropic.Anthropic([REDACTED_SECRET])
        
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=2000,
            messages=[{
                "role": "user",
                "content": prompt
            }]
        )
        
        # Extract JSON from response (Bug #13: Use safer extraction)
        content = response.content[0].text
        
        user_types = extract_json_safely(content)
        if user_types and len(user_types) >= 2:
            print(f"  ✅ AI identified {len(user_types)} plausible user types:")
            for ut in user_types:
                print(f"     • {ut.get('name', '?')} ({ut.get('archetype', '?')}): {ut.get('description', '?')}")
            return user_types[:3]  # Return top 3
        elif user_types:
            print(f"  ⚠️ AI returned only {len(user_types)} user types, need at least 2")
            return []
        else:
            print("  ⚠️ Could not parse JSON from AI response")
            return []
            
    except Exception as e:
        print(f"  ⚠️ AI inference failed: {e}")
        return []

def generate_personas_from_fallback_categories(page_analysis: Dict) -> tuple:
    """
    Fallback: Use hardcoded category detection when AI inference fails.
    Returns (personas_list, category_name).
    """
    purpose = page_analysis.get('purpose', '').lower()
    title = page_analysis.get('title', '').lower()
    navigation = ' '.join(page_analysis.get('navigation', [])).lower()
    
    personas = []
    site_category = None
    
    # Sports/Tournament sites
    if any(word in purpose + title + navigation for word in ['sport', 'tournament', 'game', 'score', 'player', 'team', 'league', 'match', 'golf', 'football', 'basketball', 'baseball']):
        site_category = 'sports'
        personas.append({
            "name": "Jordan Martinez",
            "archetype": "sports_enthusiast",
            "age": 34,
            "tech_proficiency": "medium",
            "goals": ["Find current standings/scores", "Check upcoming games/events", "View player/team statistics", "Access highlights or live coverage"],
            "description": "Sports fan who follows teams and tournaments regularly"
        })
        personas.append({
            "name": "Pat Wilson",
            "archetype": "casual_fan",
            "age": 52,
            "tech_proficiency": "low",
            "goals": ["Find basic information about favorite team", "Check game times", "Understand what's happening"],
            "description": "Casual sports viewer who checks in occasionally"
        })
    
    # E-commerce/Shopping sites
    elif any(word in purpose + title + navigation for word in ['shop', 'store', 'buy', 'product', 'cart', 'checkout', 'price', 'purchase', 'sale']):
        site_category = 'ecommerce'
        personas.append({
            "name": "Sam Taylor",
            "archetype": "online_shopper",
            "age": 29,
            "tech_proficiency": "high",
            "goals": ["Find specific products quickly", "Compare options and prices", "Read reviews", "Complete purchase efficiently"],
            "description": "Frequent online shopper who values efficiency and good deals"
        })
        personas.append({
            "name": "Maria Rodriguez",
            "archetype": "careful_buyer",
            "age": 45,
            "tech_proficiency": "medium",
            "goals": ["Research products thoroughly", "Ensure secure checkout", "Understand return policy", "Find customer service if needed"],
            "description": "Cautious shopper who researches before buying"
        })
    
    # News/Media sites
    elif any(word in purpose + title + navigation for word in ['news', 'article', 'story', 'blog', 'media', 'journalism', 'reporter']):
        site_category = 'news'
        personas.append({
            "name": "Alex Chen",
            "archetype": "news_reader",
            "age": 36,
            "tech_proficiency": "high",
            "goals": ["Find latest breaking news", "Read in-depth analysis", "Follow specific topics", "Share articles"],
            "description": "Regular news consumer who stays informed on current events"
        })
        personas.append({
            "name": "Dorothy Williams",
            "archetype": "occasional_reader",
            "age": 67,
            "tech_proficiency": "low",
            "goals": ["Find news on topics of interest", "Read without confusion", "Understand how to navigate"],
            "description": "Reads news occasionally, prefers simple, clear presentation"
        })
    
    # Booking/Travel sites
    elif any(word in purpose + title + navigation for word in ['book', 'reserve', 'hotel', 'flight', 'travel', 'vacation', 'rental', 'car rental']):
        site_category = 'booking'
        personas.append({
            "name": "Marcus Johnson",
            "archetype": "frequent_traveler",
            "age": 42,
            "tech_proficiency": "high",
            "goals": ["Book quickly and efficiently", "Find best rates", "Manage reservations", "Access loyalty benefits"],
            "description": "Business traveler who books frequently and values speed"
        })
        personas.append({
            "name": "Emma Davis",
            "archetype": "vacation_planner",
            "age": 38,
            "tech_proficiency": "medium",
            "goals": ["Research options thoroughly", "Compare prices and amenities", "Read reviews", "Understand cancellation policies"],
            "description": "Plans family vacations and wants to make informed decisions"
        })
    
    # Entertainment/Streaming sites
    elif any(word in purpose + title + navigation for word in ['watch', 'video', 'stream', 'show', 'movie', 'series', 'entertainment']):
        site_category = 'entertainment'
        personas.append({
            "name": "Taylor Kim",
            "archetype": "content_consumer",
            "age": 26,
            "tech_proficiency": "high",
            "goals": ["Find content to watch", "Browse recommendations", "Create watchlist", "Resume watching"],
            "description": "Regular viewer who enjoys discovering new content"
        })
        personas.append({
            "name": "Sarah Williams",
            "archetype": "casual_viewer",
            "age": 55,
            "tech_proficiency": "low",
            "goals": ["Find specific shows or movies", "Navigate without confusion", "Understand how to play content"],
            "description": "Watches occasionally, prefers simple interface"
        })
    
    # Developer/API/Technical sites
    elif any(word in purpose + title + navigation for word in ['developer', 'api', 'code', 'documentation', 'sdk', 'technical']):
        site_category = 'developer'
        personas.append({
            "name": "Alex Chen",
            "archetype": "developer",
            "age": 28,
            "tech_proficiency": "high",
            "goals": ["Integrate API", "Find technical docs", "See code examples", "Understand authentication"],
            "description": "Software developer looking to integrate this tool"
        })
        personas.append({
            "name": "Jordan Martinez",
            "archetype": "technical_lead",
            "age": 35,
            "tech_proficiency": "high",
            "goals": ["Evaluate technical capabilities", "Understand pricing/limits", "Assess security", "Check scalability"],
            "description": "Tech lead evaluating solutions for their team"
        })
    
    # SaaS/Business tools (default for unclear sites)
    else:
        site_category = 'saas'
        personas.append({
            "name": "Alex Chen",
            "archetype": "tech_savvy_user",
            "age": 28,
            "tech_proficiency": "high",
            "goals": ["Explore advanced features", "Try the tool", "Understand capabilities"],
            "description": "Tech-savvy early adopter"
        })
        personas.append({
            "name": "Marcus Johnson",
            "archetype": "business_professional",
            "age": 42,
            "tech_proficiency": "medium",
            "goals": ["Understand ROI", "Check pricing", "Evaluate ease of use"],
            "description": "Business professional evaluating tools"
        })
        if page_analysis.get('key_elements', {}).get('demo'):
            personas.append({
                "name": "Sarah Williams",
                "archetype": "beginner",
                "age": 35,
                "tech_proficiency": "low",
                "goals": ["Understand what it does", "Try simple example", "Get help if stuck"],
                "description": "First-time user with basic tech skills"
            })
    
    return personas, site_category

def generate_personas(page_analysis: Dict, user_persona_request: Optional[str] = None) -> List[Dict]:
    """
    Step 2: Generate contextual personas.
    
    Args:
        page_analysis: Analysis of the website
        user_persona_request: Optional user-specified persona description (e.g., "golf enthusiast tracking tournaments")
    
    Returns:
        List of persona dictionaries
    """
    print(f"\n👥 GENERATING PERSONAS")
    print("="*60)
    
    personas = []
    
    # If user specified a custom persona, create it
    if user_persona_request:
        print(f"→ Creating custom persona from user request: '{user_persona_request}'")
        
        # Parse the persona description to extract key attributes
        persona_desc_lower = user_persona_request.lower()
        
        # Determine tech proficiency based on description
        if any(word in persona_desc_lower for word in ['tech', 'developer', 'engineer', 'advanced', 'power']):
            tech_level = "high"
        elif any(word in persona_desc_lower for word in ['beginner', 'first-time', 'new', 'novice', 'elderly']):
            tech_level = "low"
        else:
            tech_level = "medium"
        
        # Determine archetype based on description
        if 'enthusiast' in persona_desc_lower or 'fan' in persona_desc_lower:
            archetype = "enthusiast"
            name = "Jordan Martinez"
        elif 'professional' in persona_desc_lower or 'business' in persona_desc_lower:
            archetype = "professional"
            name = "Alex Chen"
        elif 'casual' in persona_desc_lower or 'occasional' in persona_desc_lower:
            archetype = "casual_user"
            name = "Sam Williams"
        elif 'beginner' in persona_desc_lower or 'first-time' in persona_desc_lower:
            archetype = "beginner"
            name = "Sarah Johnson"
        else:
            archetype = "engaged_user"
            name = "Taylor Davis"
        
        # Extract goals based on context (website purpose + persona desc)
        purpose = page_analysis.get('purpose', '').lower()
        goals = []
        
        # Sport/content site goals
        if any(word in purpose for word in ['tournament', 'sport', 'score', 'game', 'match']):
            if 'following' in persona_desc_lower or 'tracking' in persona_desc_lower:
                goals = [
                    "Quickly find current tournament standings",
                    "Check latest scores and updates",
                    "View player/team statistics",
                    "Access live coverage or highlights"
                ]
        # E-commerce goals
        elif any(word in purpose for word in ['shop', 'buy', 'product', 'store']):
            if 'enthusiast' in persona_desc_lower:
                goals = [
                    "Find specific products in their interest area",
                    "Compare options and read reviews",
                    "Complete purchase quickly",
                    "Track orders and manage account"
                ]
        # Content/news goals
        elif any(word in purpose for word in ['news', 'article', 'content', 'blog']):
            goals = [
                "Find latest updates in area of interest",
                "Read in-depth articles",
                "Follow specific topics or tags",
                "Share content with others"
            ]
        # Default goals
        else:
            goals = [
                "Accomplish primary task efficiently",
                "Find relevant information easily",
                "Navigate without confusion"
            ]
        
        custom_persona = {
            "name": name,
            "archetype": archetype,
            "age": 32,  # Default reasonable age
            "tech_proficiency": tech_level,
            "goals": goals,
            "description": user_persona_request,
            "custom": True
        }
        
        personas.append(custom_persona)
        print(f"  ✅ Created: {name} ({archetype}, {tech_level} proficiency)")
        print(f"     Goals: {', '.join(goals[:2])}...")
        
        # Add one complementary persona for comparison
        if tech_level != "low":
            # Add a beginner persona for contrast
            personas.append({
                "name": "Pat Wilson",
                "archetype": "beginner",
                "age": 58,
                "tech_proficiency": "low",
                "goals": ["Understand the basics", "Navigate without getting lost", "Find help if needed"],
                "description": "First-time visitor with basic tech skills"
            })
            print(f"  ✅ Added contrast persona: Pat Wilson (beginner)")
    else:
        # No custom persona - use AI to infer plausible user types
        print(f"→ Auto-generating personas based on website exploration...")
        
        # Try AI-powered inference first
        ai_user_types = infer_plausible_user_types(page_analysis)
        
        if ai_user_types and len(ai_user_types) >= 2:
            # Success! Use AI-generated personas
            personas = ai_user_types
            print(f"  ✅ Using AI-generated personas")
        else:
            # Fallback to category-based detection
            print(f"  → Falling back to category-based persona generation...")
            personas, site_category = generate_personas_from_fallback_categories(page_analysis)
            print(f"  Detected site category: {site_category}")
        
        for p in personas:
            print(f"  → {p['name']} ({p['archetype']}): {p['description']}")
    
    return personas

def generate_test_cases(persona: Dict, page_analysis: Dict) -> List[str]:
    """Step 3: Generate realistic test cases (including workflow tests when appropriate)."""
    archetype = persona['archetype']
    key_elements = page_analysis.get('key_elements', {})
    has_pricing = key_elements.get('pricing', False)
    has_docs = key_elements.get('documentation', False)
    has_demo = key_elements.get('demo', False)
    purpose = page_analysis.get('purpose', '').lower()
    
    test_cases = []
    
    # Detect if this is a transactional/workflow-oriented site
    is_ecommerce = any(kw in purpose for kw in ['shop', 'store', 'buy', 'product', 'purchase'])
    is_booking = any(kw in purpose for kw in ['book', 'reserve', 'schedule', 'hotel', 'flight', 'appointment'])
    is_social = any(kw in purpose for kw in ['post', 'share', 'social', 'community', 'publish'])
    is_saas = any(kw in purpose for kw in ['signup', 'register', 'subscribe', 'trial'])
    
    # Generate workflow tests for transactional sites
    if is_ecommerce and archetype in ["tech_savvy_user", "business_professional"]:
        test_cases.append("Search for a product and add it to cart")
        test_cases.append("Complete checkout flow up to payment")
    elif is_booking:
        test_cases.append("Complete a booking workflow from search to checkout")
    elif is_social:
        test_cases.append("Create and prepare a post for publishing")
    elif is_saas:
        test_cases.append("Complete signup flow up to final submission")
    
    # Original information-finding tests
    if archetype == "developer" or archetype == "tech_savvy_user":
        if has_docs:
            test_cases.append("Find and access technical documentation")
        if has_demo:
            test_cases.append("Try the interactive demo or playground")
        if not is_ecommerce and not is_booking:  # Don't duplicate
            test_cases.append("Understand the core capabilities and features")
        
    elif archetype == "business_professional":
        if has_pricing:
            test_cases.append("Find pricing information and compare plans")
        test_cases.append("Evaluate if this meets business needs")
        
    elif archetype == "beginner":
        test_cases.append("Understand what this website does")
        if has_docs:
            test_cases.append("Find help or getting started guide")
    
    # If we still don't have test cases, use persona goals
    if not test_cases and 'goals' in persona:
        test_cases = [goal for goal in persona['goals'][:3]]
    
    # Fallback
    if not test_cases:
        test_cases = ["Explore the main features of the website"]
    
    return test_cases

def execute_exploration_step_adaptive(nova, step: Dict, persona: Dict, step_index: int = 0, max_attempts: int = 3) -> Dict:
    """
    Execute a single exploration step and capture RAW responses.
    
    IMPORTANT: This function does NOT interpret responses.
    The orchestrating AI agent (OpenClaw/Claude) should analyze the raw_response
    to determine if the goal was achieved.
    
    Returns structured data with:
    - raw_response: The actual text Nova Act returned
    - api_success: Whether the API call worked
    - needs_agent_analysis: Flag indicating agent should interpret
    """
    step_num = step_index
    action = step.get('prompt', step.get('action', ''))
    rationale = step.get('expected_outcome', step.get('rationale', ''))
    step_name = step.get('step_name', f'Step {step_num}')
    is_safety_stop = step.get('is_safety_stop', False)
    action_type = step.get('action_type', 'query')
    
    print(f"\n   Step {step_num}: {step_name}")
    print(f"   Action: {action[:80]}...")
    print(f"   Expected: {rationale}")
    if is_safety_stop:
        print(f"   ⚠️  SAFETY STOP: Will query only, no execution")
    
    result = {
        'step_number': step_num,
        'step_name': step_name,
        'prompt': action,
        'expected_outcome': rationale,
        'is_safety_stop': is_safety_stop,
        'raw_response': None,
        'api_success': False,
        'attempts': [],
        'error': None,
        'needs_agent_analysis': True  # Agent must interpret this!
    }
    
    # Adapt the action prompt for this persona
    adapted_action = adapt_prompt_for_persona(action, persona)
    current_prompt = adapted_action
    
    for attempt in range(1, max_attempts + 1):
        attempt_result = {
            'attempt': attempt,
            'prompt': current_prompt,
            'raw_response': None,
            'error': None
        }
        
        try:
            print(f"   → Attempt {attempt}/{max_attempts}: {current_prompt[:60]}...")
            
            if is_safety_stop:
                query_prompt = f"What would happen if I: {current_prompt}? Describe what you see but DO NOT execute the action."
                ok, response, error = safe_act_get(
                    nova, query_prompt,
                    schema={"type": "object", "properties": {"observation": {"type": "string"}}, "required": ["observation"]},
                    timeout=30
                )
                response_text = response.get('observation', '') if ok and response else None
                if not ok:
                    attempt_result['error'] = error
            
            elif action_type == 'navigate':
                ok, error_or_obs = safe_act(nova, current_prompt, timeout=30)
                if ok:
                    ok2, verification, _ = safe_act_get(
                        nova, "Briefly describe what you now see on the page",
                        schema={"type": "object", "properties": {"description": {"type": "string"}}, "required": ["description"]},
                        timeout=20
                    )
                    response_text = verification.get('description', 'Navigation completed') if ok2 and verification else 'Navigation completed'
                else:
                    response_text = None
                    attempt_result['error'] = error_or_obs
            
            else:
                ok, response, error = safe_act_get(
                    nova, current_prompt,
                    schema={"type": "object", "properties": {"answer": {"type": "string"}}, "required": ["answer"]},
                    timeout=30
                )
                response_text = response.get('answer', str(response)) if ok and response else None
                if not ok:
                    attempt_result['error'] = error
            
            attempt_result['raw_response'] = response_text
            
            if response_text:
                result['api_success'] = True
                result['raw_response'] = response_text
                print(f"   📝 Response: {response_text[:80]}...")
                print(f"   ⏳ (Agent will analyze if goal achieved)")
                result['attempts'].append(attempt_result)
                
                # Check for obvious negatives to try alternatives
                response_lower = response_text.lower().strip()
                obvious_negative = response_lower in ['no', 'false'] or \
                                   response_lower.startswith('no ') or \
                                   'not found' in response_lower or \
                                   'i don\'t see' in response_lower or \
                                   'i do not see' in response_lower
                
                if obvious_negative and attempt < max_attempts:
                    alt_prompt = generate_alternative_approach(adapted_action, response_text, attempt)
                    if alt_prompt:
                        print(f"   🔄 Response appears negative, trying alternative...")
                        current_prompt = alt_prompt
                        continue
                
                # Got a response - return it for agent analysis
                return result
            else:
                print(f"   ❌ No response: {attempt_result.get('error', 'Unknown error')}")
                result['error'] = attempt_result.get('error')
                
                if attempt < max_attempts:
                    alt_prompt = generate_alternative_approach(adapted_action, "No response", attempt)
                    if alt_prompt:
                        current_prompt = alt_prompt
                        print(f"   🔄 Retrying with different approach...")
        
        except Exception as e:
            print(f"   ❌ Exception: {str(e)}")
            attempt_result['error'] = str(e)
            result['error'] = str(e)
        
        result['attempts'].append(attempt_result)
    
    print(f"   ❌ All {max_attempts} attempts exhausted")
    return result


# Keep old function name for backwards compatibility
def execute_exploration_step(nova, step: Dict, persona: Dict, step_index: int = 0) -> Dict:
    """Wrapper for backwards compatibility."""
    return execute_exploration_step_adaptive(nova, step, persona, step_index, max_attempts=3)

def iterative_test_dynamic(persona: Dict, test_case: str, page_analysis: Dict, cookbook: str = "", website_url: str = None) -> Dict:
    """
    Execute one test case iteratively with dynamic strategy generation.
    """
    print(f"\n{'='*80}")
    print(f"🎭 TESTING: {persona['name']} ({persona['archetype']})")
    print(f"📋 TEST CASE: {test_case}")
    print(f"{'='*80}")
    
    result = {
        'persona': persona,
        'test_case': test_case,
        'steps': [],
        'overall_success': False,
        'completion_status': 'incomplete',
        'error': None,
        'trace_files': []
    }
    
    try:
        # Generate exploration strategy for this specific test case and persona
        steps = generate_exploration_strategy(test_case, persona, page_analysis, cookbook)
        
        if not steps:
            print("⚠️ Failed to generate exploration strategy")
            result['error'] = "Failed to generate exploration strategy"
            return result
        
        print(f"\n📝 Generated {len(steps)} exploration steps")
        
        # Capture existing trace files BEFORE this test
        trace_pattern = os.path.join(LOGS_DIR, "**", "*.html")
        existing_traces = set(glob.glob(trace_pattern, recursive=True))
        
        # Execute each step (Bug #11: use parameter instead of global)
        target_url = website_url or WEBSITE_URL
        with nova_session(target_url, headless=True, logs_dir=LOGS_DIR) as nova:
            for idx, step in enumerate(steps):
                step_result = execute_exploration_step(nova, step, persona, idx)
                result['steps'].append(step_result)
                
                # If a step fails completely (API error), stop
                if not step_result.get('api_success') and not step_result.get('is_safety_stop'):
                    print(f"\n⚠️ Step {idx} failed (API error), stopping test")
                    break
                
                # If goal not achieved after all retries, continue but note it
                if step_result.get('api_success') and not step_result.get('goal_achieved'):
                    print(f"   📝 Step {idx}: API worked but goal not achieved")
                
                # Small delay between steps
                time.sleep(0.5)
        
        # Capture NEW trace files created during this test
        all_traces = set(glob.glob(trace_pattern, recursive=True))
        new_traces = sorted(all_traces - existing_traces, key=os.path.getmtime)
        result['trace_files'] = new_traces
        if new_traces:
            print(f"\n🎬 Captured {len(new_traces)} trace file(s) for this test")
        
        # Count API successes - goal achievement will be determined by the orchestrating agent
        api_successes = sum(1 for s in result['steps'] if s.get('api_success', False))
        total_steps = len(result['steps'])
        
        # Raw data summary - agent will interpret and set actual success values
        result['api_successes'] = api_successes
        result['total_steps'] = total_steps
        result['needs_agent_analysis'] = True  # Agent MUST interpret raw_response values
        
        # Preliminary completion status based on API success only
        # Agent will update overall_success after interpreting responses
        if api_successes == total_steps:
            result['completion_status'] = 'complete'
        elif api_successes >= total_steps * 0.5:
            result['completion_status'] = 'partial'
        else:
            result['completion_status'] = 'incomplete'
        
        print(f"\n{'='*80}")
        print(f"📊 Raw data collected: {api_successes}/{total_steps} API calls succeeded")
        print(f"⏳ AWAITING AGENT ANALYSIS")
        print(f"   Agent must interpret raw_response in each step to determine goal achievement")
        print(f"{'='*80}")
    
    except Exception as e:
        print(f"\n❌ Test failed with exception: {str(e)}")
        result['error'] = str(e)
        result['completion_status'] = 'error'
    
    return result

def main():
    global WEBSITE_URL
    
    # Parse command-line arguments
    if len(sys.argv) < 2:
        print("Usage: python3 run_adaptive_test.py <website_url> [persona_arg]")
        print("\nExamples:")
        print('  # Auto-generate personas (fallback)')
        print('  python3 run_adaptive_test.py "https://www.hertz.com/"')
        print()
        print('  # Custom persona description')
        print('  python3 run_adaptive_test.py "https://www.pgatour.com/" "golf enthusiast interested in the latest tournament"')
        print()
        print('  # AI-generated personas (JSON file)')
        print('  python3 run_adaptive_test.py "https://www.pgatour.com/" "personas.json"')
        print()
        print('  # AI-generated personas (JSON string)')
        print("  python3 run_adaptive_test.py \"https://www.pgatour.com/\" '[{\"name\":\"Jordan\",...}]'")
        sys.exit(1)
    
    WEBSITE_URL = sys.argv[1]
    persona_arg = sys.argv[2] if len(sys.argv) >= 3 else None
    
    # Determine what type of persona argument we received
    ai_generated_personas = None
    user_persona_request = None
    
    if persona_arg:
        # Check if it's a JSON file path
        if persona_arg.endswith('.json') and os.path.exists(persona_arg):
            print(f"\n🎯 Testing {WEBSITE_URL}")
            print(f"👥 Loading AI-generated personas from: {persona_arg}\n")
            try:
                with open(persona_arg, 'r') as f:
                    ai_generated_personas = json.load(f)
                    print(f"✅ Loaded {len(ai_generated_personas)} personas from file")
            except Exception as e:
                print(f"❌ Failed to load personas from {persona_arg}: {e}")
                print("   Falling back to auto-generation")
                ai_generated_personas = None
        
        # Check if it's a JSON string (starts with [ or {)
        elif persona_arg.strip().startswith(('[', '{')):
            print(f"\n🎯 Testing {WEBSITE_URL}")
            print(f"👥 Using AI-generated personas from argument\n")
            try:
                ai_generated_personas = json.loads(persona_arg)
                print(f"✅ Parsed {len(ai_generated_personas)} personas from JSON")
            except Exception as e:
                print(f"❌ Failed to parse JSON personas: {e}")
                print("   Treating as custom persona description instead")
                user_persona_request = persona_arg
        
        # Otherwise, treat as custom persona description
        else:
            print(f"\n🎯 Testing {WEBSITE_URL}")
            print(f"👤 Custom persona: {persona_arg}\n")
            user_persona_request = persona_arg
    else:
        print(f"\n🎯 Testing {WEBSITE_URL}")
        print(f"👤 Auto-generating personas (fallback mode)\n")
    
    # Start status reporter (60-second updates)
    reporter_process = start_status_reporter()
    
    # Record test start time for filtering trace files (Bug #10)
    test_start_time = time.time()
    _shutdown_state['test_start_time'] = test_start_time
    
    try:
        update_status("Loading cookbook...")
        cookbook = load_cookbook()
        
        update_status("Analyzing website...")
        page_analysis = analyze_page(WEBSITE_URL)
        _shutdown_state['page_analysis'] = page_analysis
        
        # Generate or use provided personas
        if ai_generated_personas:
            print(f"\n👥 USING AI-GENERATED PERSONAS")
            print("="*60)
            personas = ai_generated_personas
            for p in personas:
                print(f"  → {p.get('name', '?')} ({p.get('archetype', '?')}): {p.get('description', '?')}")
        else:
            update_status("Generating test personas...")
            personas = generate_personas(page_analysis, user_persona_request)
        
        if not personas:
            print("❌ Failed to generate personas")
            mark_complete(success=False)
            return
        
        # Calculate total planned tests for progress tracking
        total_planned = sum(len(generate_test_cases(p, page_analysis)) for p in personas)
        _shutdown_state['total_planned_tests'] = total_planned
        
        all_results = []
        _shutdown_state['all_results'] = all_results  # Share reference
        
        for i, persona in enumerate(personas, 1):
            update_status(f"Testing persona {i}/{len(personas)}: {persona['name']}...")
            
            test_cases = generate_test_cases(persona, page_analysis)
            print(f"\n📋 Generated {len(test_cases)} test cases for {persona['name']}")
            
            for j, test_case in enumerate(test_cases, 1):
                update_status(f"Running test {j}/{len(test_cases)} for {persona['name']}...")
                
                result = iterative_test_dynamic(persona, test_case, page_analysis, cookbook, website_url=WEBSITE_URL)
                all_results.append(result)
                _shutdown_state['completed_tests'] = len(all_results)
                
                # Save intermediate results
                with open(RESULTS_FILE, 'w') as f:
                    json.dump(all_results, f, indent=2)
        
        update_status("Generating final report...")
        
        # Generate HTML report with trace files from THIS test run only (Bug #10)
        trace_pattern = os.path.join(LOGS_DIR, "**", "*.html")
        all_traces = glob.glob(trace_pattern, recursive=True)
        # Filter to only traces created during this run
        traces = sorted(
            [t for t in all_traces if os.path.getmtime(t) >= test_start_time],
            key=os.path.getmtime
        )
        report_path = generate_enhanced_report(page_analysis, all_results, traces)
        
        print(f"\n{'='*80}")
        print(f"✅ ALL TESTS COMPLETE")
        print(f"{'='*80}")
        print(f"📊 Report: {report_path}")
        print(f"📁 Results: {RESULTS_FILE}")
        print(f"🎬 Traces: {LOGS_DIR}")
        
        # Calculate success rate
        successful_tests = sum(1 for r in all_results if r['overall_success'])
        total_tests = len(all_results)
        success_rate = (successful_tests / total_tests * 100) if total_tests > 0 else 0
        
        print(f"\n✅ Complete: {successful_tests}/{total_tests} tests passed ({success_rate:.0f}%)")
        mark_complete(success=True)
        emit_final()
    
    except Exception as e:
        print(f"\n❌ FATAL ERROR: {str(e)}")
        import traceback
        traceback.print_exc()
        mark_complete(success=False)
    finally:
        stop_status_reporter()

if __name__ == "__main__":
    main()


path: scripts/enhanced_report_generator.py
content: #!/usr/bin/env python3
"""
Enhanced HTML report generator with detailed explanations and trace links.
"""

import os
from datetime import datetime
from typing import List, Dict

def is_wsl() -> bool:
    """Check if running on Windows Subsystem for Linux."""
    try:
        with open('/proc/version', 'r') as f:
            return 'microsoft' in f.read().lower() or 'wsl' in f.read().lower()
    except:
        return False

def convert_to_wsl_path(path: str) -> str:
    """Convert Linux path to WSL path accessible from Windows browser."""
    if is_wsl() and path.startswith('/'):
        # Convert /home/user/... to file://wsl$/Ubuntu/home/user/...
        return f"file://wsl$/Ubuntu{path}"
    elif path.startswith('/'):
        # Linux path to file:// URL
        return f"file://{path}"
    return path

def generate_enhanced_report(page_analysis: Dict, results: List[Dict], traces: List[str] = None) -> str:
    """
    Generate comprehensive HTML report with:
    - Links to Nova Act trace files (WSL-compatible)
    - Detailed explanations of each test
    - Easy dive-in points
    - Workflow testing support
    
    Args:
        page_analysis: Dict with website analysis (title, navigation, purpose, key_elements)
        results: List of test result dictionaries
        traces: Optional list of trace file paths from Nova Act sessions
    """
    traces = traces or []
    
    # Calculate summary stats
    total_tests = len(results)
    successful = sum(1 for r in results if r.get('overall_success', False))
    failed = total_tests - successful
    success_rate = (successful / total_tests * 100) if total_tests > 0 else 0
    
    # Detect test type (workflow vs information-finding)
    test_types = set()
    for result in results:
        test_case = result.get('test_case', '').lower()
        if any(kw in test_case for kw in ['book', 'purchase', 'checkout', 'post', 'signup', 'submit']):
            test_types.add('workflow')
        else:
            test_types.add('information_finding')
    
    is_workflow_test = 'workflow' in test_types
    
    # Group by persona (using persona name as key since dict isn't hashable)
    persona_results = {}
    for result in results:
        persona = result['persona']
        persona_name = persona.get('name', 'Unknown')
        if persona_name not in persona_results:
            persona_results[persona_name] = {'persona': persona, 'results': []}
        persona_results[persona_name]['results'].append(result)
    
    # Build HTML
    html = f"""<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Nova Act Usability Test Report</title>
    <style>
        * {{
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }}
        
        body {{
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            line-height: 1.6;
            color: #333;
            background: #f5f5f5;
            padding: 20px;
        }}
        
        .container {{
            max-width: 1400px;
            margin: 0 auto;
            background: white;
            padding: 40px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
            border-radius: 8px;
        }}
        
        h1 {{
            color: #2c3e50;
            border-bottom: 3px solid #3498db;
            padding-bottom: 15px;
            margin-bottom: 30px;
            font-size: 2.5em;
        }}
        
        h2 {{
            color: #34495e;
            margin-top: 40px;
            margin-bottom: 20px;
            font-size: 1.8em;
            border-bottom: 2px solid #ecf0f1;
            padding-bottom: 10px;
        }}
        
        h3 {{
            color: #2c3e50;
            margin-top: 25px;
            margin-bottom: 15px;
            font-size: 1.3em;
        }}
        
        .page-analysis {{
            background: #e8f4f8;
            padding: 25px;
            border-radius: 6px;
            margin: 20px 0;
            border-left: 4px solid #3498db;
        }}
        
        .page-analysis h3 {{
            margin-top: 0;
            color: #2980b9;
        }}
        
        .page-analysis ul {{
            margin: 10px 0;
            padding-left: 25px;
        }}
        
        .executive-summary {{
            background: #ecf0f1;
            padding: 25px;
            border-radius: 6px;
            margin: 20px 0;
            border-left: 4px solid #3498db;
        }}
        
        .executive-summary p {{
            margin: 10px 0;
            font-size: 1.1em;
        }}
        
        .executive-summary strong {{
            color: #2c3e50;
        }}
        
        .persona-section {{
            background: #fff;
            border: 2px solid #ddd;
            border-radius: 8px;
            padding: 25px;
            margin: 25px 0;
            box-shadow: 0 2px 5px rgba(0,0,0,0.08);
        }}
        
        .persona-section h3 {{
            color: #3498db;
            margin-top: 0;
            display: flex;
            align-items: center;
            gap: 10px;
        }}
        
        .test-case {{
            background: #f8f9fa;
            border-left: 4px solid #95a5a6;
            padding: 20px;
            margin: 20px 0;
            border-radius: 4px;
        }}
        
        .test-case.success {{
            border-left-color: #2ecc71;
            background: #eafaf1;
        }}
        
        .test-case.failure {{
            border-left-color: #e74c3c;
            background: #fadbd8;
        }}
        
        .test-header {{
            display: flex;
            justify-content: space-between;
            align-items: center;
            margin-bottom: 15px;
        }}
        
        .test-title {{
            font-size: 1.1em;
            font-weight: 600;
            color: #2c3e50;
        }}
        
        .test-status {{
            display: inline-block;
            padding: 6px 12px;
            border-radius: 4px;
            font-weight: bold;
            font-size: 0.9em;
        }}
        
        .test-status.success {{
            background: #2ecc71;
            color: white;
        }}
        
        .test-status.failure {{
            background: #e74c3c;
            color: white;
        }}
        
        .test-status.pending {{
            background: #f39c12;
            color: white;
        }}
        
        .test-case.pending {{
            border-left-color: #f39c12;
            background: #fef9e7;
        }}
        
        .observation {{
            background: white;
            border: 1px solid #ddd;
            border-radius: 4px;
            padding: 15px;
            margin: 10px 0;
        }}
        
        .observation-header {{
            display: flex;
            justify-content: space-between;
            align-items: center;
            margin-bottom: 8px;
        }}
        
        .step-name {{
            font-weight: 600;
            color: #34495e;
        }}
        
        .step-result {{
            padding: 4px 8px;
            border-radius: 3px;
            font-size: 0.85em;
            font-weight: 600;
        }}
        
        .step-result.success {{
            background: #d5f4e6;
            color: #27ae60;
        }}
        
        .step-result.failure {{
            background: #fadbd8;
            color: #c0392b;
        }}
        
        .observation-action {{
            color: #555;
            font-style: italic;
            margin: 5px 0;
        }}
        
        .observation-notes {{
            background: #f8f9fa;
            padding: 10px;
            border-radius: 3px;
            margin-top: 8px;
            color: #555;
        }}
        
        .observation-notes.issue {{
            background: #fff3cd;
            border-left: 3px solid #ffc107;
        }}
        
        .trace-link {{
            display: inline-block;
            margin-top: 10px;
            padding: 8px 12px;
            background: #3498db;
            color: white;
            text-decoration: none;
            border-radius: 4px;
            font-size: 0.9em;
        }}
        
        .trace-link:hover {{
            background: #2980b9;
        }}
        
        .insights {{
            background: #fff3cd;
            border-left: 4px solid #ffc107;
            padding: 20px;
            margin: 20px 0;
            border-radius: 4px;
        }}
        
        .insights h3 {{
            margin-top: 0;
            color: #f39c12;
        }}
        
        .insight-item {{
            margin: 10px 0;
            padding-left: 20px;
            position: relative;
        }}
        
        .insight-item:before {{
            content: "💡";
            position: absolute;
            left: 0;
        }}
        
        .footer {{
            margin-top: 50px;
            padding-top: 20px;
            border-top: 1px solid #ddd;
            text-align: center;
            color: #7f8c8d;
            font-size: 0.9em;
        }}
        
        .metric {{
            display: inline-block;
            margin: 0 15px;
            font-size: 1.1em;
        }}
        
        .metric-value {{
            font-weight: bold;
            color: #3498db;
            font-size: 1.3em;
        }}
        
        details {{
            margin: 15px 0;
        }}
        
        summary {{
            cursor: pointer;
            font-weight: 600;
            padding: 10px;
            background: #f8f9fa;
            border-radius: 4px;
            user-select: none;
        }}
        
        summary:hover {{
            background: #e9ecef;
        }}
        .partial-warning {{
            background: #fff3cd;
            border: 2px solid #ffc107;
            border-radius: 8px;
            padding: 20px;
            margin: 20px 0;
            text-align: center;
        }}
        
        .partial-warning h2 {{
            color: #856404;
            margin: 0 0 10px 0;
            border: none;
            font-size: 1.5em;
        }}
        
        .partial-warning p {{
            color: #856404;
            margin: 5px 0;
        }}
    </style>
</head>
<body>
    <div class="container">
        <h1>🦅 Nova Act Usability Test Report</h1>
"""
    
    # Check if this is a partial report (interrupted)
    if page_analysis.get('_partial_report'):
        completed = page_analysis.get('_completed_tests', 0)
        total = page_analysis.get('_total_planned_tests', '?')
        html += f"""
        <div class="partial-warning">
            <h2>⚠️ PARTIAL REPORT - Test Interrupted</h2>
            <p>This report was generated after the test was interrupted (timeout or signal).</p>
            <p><strong>{completed} of {total} planned tests completed</strong></p>
            <p>Results below reflect only the tests that finished before interruption.</p>
        </div>
"""
    
    html += f"""
        <div class="page-analysis">
            <h3>📄 Page Analysis: {page_analysis.get('title', 'Unknown')}</h3>
            <p><strong>Purpose:</strong> {page_analysis.get('purpose', 'Not analyzed')}</p>
            <p><strong>Navigation:</strong> {', '.join(page_analysis.get('navigation', ['None found']))}</p>
"""
    
    # Dynamic key elements based on test type AND site category
    purpose_lower = page_analysis.get('purpose', '').lower()
    title_lower = page_analysis.get('title', '').lower()
    navigation = ' '.join(page_analysis.get('navigation', [])).lower()
    
    # Detect site category (same logic as persona generation)
    site_category = 'unknown'
    if any(word in purpose_lower + title_lower + navigation for word in ['sport', 'tournament', 'game', 'score', 'player', 'team', 'league', 'match']):
        site_category = 'sports'
    elif any(word in purpose_lower + title_lower + navigation for word in ['shop', 'store', 'buy', 'product', 'cart', 'checkout']):
        site_category = 'ecommerce'
    elif any(word in purpose_lower + title_lower + navigation for word in ['news', 'article', 'story', 'blog', 'media']):
        site_category = 'news'
    elif any(word in purpose_lower + title_lower + navigation for word in ['book', 'reserve', 'hotel', 'flight', 'travel', 'rental']):
        site_category = 'booking'
    elif any(word in purpose_lower + title_lower + navigation for word in ['watch', 'video', 'stream', 'show', 'movie']):
        site_category = 'entertainment'
    elif any(word in purpose_lower + title_lower + navigation for word in ['developer', 'api', 'code', 'documentation', 'sdk']):
        site_category = 'developer'
    else:
        site_category = 'saas'
    
    # Category-specific elements
    if site_category == 'sports':
        html += f"""
            <p><strong>Site Category:</strong> Sports/Tournament Content</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Leaderboard/Standings: {'✅ Found in navigation' if 'leaderboard' in navigation or 'standing' in navigation else '⚠️ Not easily accessible'}</li>
                <li>Schedule/Calendar: {'✅ Found in navigation' if 'schedule' in navigation or 'calendar' in navigation else '⚠️ Not easily accessible'}</li>
                <li>Player/Team Stats: {'✅ Found in navigation' if 'player' in navigation or 'stats' in navigation or 'team' in navigation else '⚠️ Not easily accessible'}</li>
                <li>Live Scores/Updates: {'✅ Content suggests live coverage' if 'live' in purpose_lower or 'watch' in navigation else '⚠️ Not evident'}</li>
            </ul>
"""
    elif site_category == 'ecommerce':
        html += f"""
            <p><strong>Site Category:</strong> E-Commerce / Shopping</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Product Search: {'✅ Available' if page_analysis.get('has_homepage_search') is True else '⚠️ Not immediately visible'}</li>
                <li>Shopping Cart: {'✅ Found in navigation' if 'cart' in navigation else '⚠️ Not easily accessible'}</li>
                <li>Checkout: {'✅ E-commerce functionality detected' if 'checkout' in navigation or 'cart' in navigation else '⚠️ Unclear'}</li>
                <li>Pricing: {'✅ Product pricing visible' if page_analysis.get('key_elements', {}).get('pricing') else '⚠️ Pricing not immediately clear'}</li>
            </ul>
"""
    elif site_category == 'news':
        html += f"""
            <p><strong>Site Category:</strong> News / Content / Media</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Article Access: {'✅ Content-focused site' if 'news' in purpose_lower or 'article' in purpose_lower else '⚠️ Purpose unclear'}</li>
                <li>Navigation: {'✅ Clear menu structure' if len(page_analysis.get('navigation', [])) > 3 else '⚠️ Limited navigation'}</li>
                <li>Search: {'✅ Available' if page_analysis.get('has_homepage_search') is True else '⚠️ Not immediately visible'}</li>
                <li>Content Organization: {'✅ Categories/sections visible' if len(page_analysis.get('navigation', [])) > 5 else '⚠️ May be limited'}</li>
            </ul>
"""
    elif site_category == 'booking':
        html += f"""
            <p><strong>Site Category:</strong> Booking / Reservation / Travel</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Search Widget: {'✅ Available' if page_analysis.get('has_homepage_search') is True else '❌ Not found - critical for booking sites'}</li>
                <li>Loyalty Program: {'✅ Visible' if page_analysis.get('has_loyalty_program') is True else '⚠️ Not prominently displayed'}</li>
                <li>Pricing Transparency: {'✅ Pricing info accessible' if page_analysis.get('key_elements', {}).get('pricing') else '⚠️ Pricing not upfront'}</li>
                <li>Booking Flow: {'✅ Clear path to reservation' if page_analysis.get('has_homepage_search') else '⚠️ May require exploration'}</li>
            </ul>
"""
    elif site_category == 'entertainment':
        html += f"""
            <p><strong>Site Category:</strong> Entertainment / Streaming / Video</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Content Discovery: {'✅ Browse/categories available' if len(page_analysis.get('navigation', [])) > 3 else '⚠️ Limited browsing'}</li>
                <li>Search: {'✅ Available' if page_analysis.get('has_homepage_search') is True else '⚠️ Not immediately visible'}</li>
                <li>Watch/Play Access: {'✅ Video functionality detected' if 'watch' in navigation or 'video' in navigation else '⚠️ Unclear'}</li>
                <li>User Features: {'✅ Account/profile features' if 'sign' in navigation or 'account' in navigation else '⚠️ Not evident'}</li>
            </ul>
"""
    elif site_category == 'developer':
        html += f"""
            <p><strong>Site Category:</strong> Developer / API / Technical Documentation</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Documentation: {'✅ Available' if page_analysis.get('key_elements', {}).get('documentation') else '❌ Not found - critical for developers'}</li>
                <li>API Reference: {'✅ Detected' if 'api' in navigation or 'reference' in navigation else '⚠️ Not easily accessible'}</li>
                <li>Code Examples: {'✅ Demo/playground available' if page_analysis.get('key_elements', {}).get('demo') else '⚠️ May be limited'}</li>
                <li>Getting Started: {'✅ Onboarding present' if 'start' in navigation or 'guide' in navigation else '⚠️ May require search'}</li>
            </ul>
"""
    else:  # saas or unknown
        html += f"""
            <p><strong>Site Category:</strong> SaaS / Business Tool</p>
            <p><strong>Key Features:</strong></p>
            <ul>
                <li>Documentation: {'✅ Available' if page_analysis.get('key_elements', {}).get('documentation') else '⚠️ Not found'}</li>
                <li>Interactive Demo: {'✅ Available' if page_analysis.get('key_elements', {}).get('demo') else '⚠️ Not found'}</li>
                <li>Pricing: {'✅ Available' if page_analysis.get('key_elements', {}).get('pricing') else '⚠️ Not found'}</li>
                <li>Getting Started: {'✅ Clear onboarding' if 'start' in navigation or 'docs' in navigation else '⚠️ May require exploration'}</li>
            </ul>
"""
    
    html += f"""
        </div>
        
        <div class="executive-summary">
            <h2>Executive Summary</h2>
            <p><strong>Test Date:</strong> {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</p>
            <p><strong>Tests Conducted:</strong> {total_tests}</p>
            <p>
                <span class="metric">
                    <span class="metric-value">{successful}</span> Passed
                </span>
                <span class="metric">
                    <span class="metric-value">{failed}</span> Failed
                </span>
                <span class="metric">
                    <span class="metric-value">{success_rate:.1f}%</span> Success Rate
                </span>
            </p>
            <p><strong>Personas Tested:</strong> {len(persona_results)}</p>
        </div>
"""
    
    # Per-persona detailed results
    html += "<h2>Detailed Test Results</h2>\n"
    
    # Global recording counter for consistent numbering across all tests
    global_recording_index = 1
    
    for persona_name, persona_data in persona_results.items():
        persona_tests = persona_data['results']
        persona_obj = persona_data['persona']
        
        persona_success = sum(1 for t in persona_tests if t.get('overall_success', False))
        persona_total = len(persona_tests)
        persona_rate = (persona_success / persona_total * 100) if persona_total > 0 else 0
        
        archetype = persona_obj.get('archetype', 'unknown')
        tech_level = persona_obj.get('tech_proficiency', 'medium')
        
        html += f"""
        <div class="persona-section">
            <h3>
                <span>{persona_name}</span>
                <span style="font-size: 0.8em; color: #7f8c8d;">({archetype} - {tech_level} proficiency)</span>
            </h3>
            <p><strong>Success Rate:</strong> {persona_success}/{persona_total} ({persona_rate:.1f}%)</p>
        """
        
        for test in persona_tests:
            steps = test.get('steps', [])
            overall_success = test.get('overall_success', False)
            has_error = test.get('error') or test.get('completion_status') == 'error'
            
            # Check if agent has interpreted this test
            test_interpreted = any('goal_achieved' in s for s in steps)
            
            # Determine display status
            if has_error and not steps:
                # Error before any steps ran - show as failed
                test_class = "failure"
                status_text = "❌ FAILED"
            elif test_interpreted or 'goals_achieved' in test:
                # Agent has interpreted - show actual result
                test_class = "success" if overall_success else "failure"
                status_text = "✅ PASSED" if overall_success else "❌ FAILED"
            elif test.get('needs_agent_analysis') and steps:
                # Has steps but not yet interpreted - show pending
                test_class = "pending"
                status_text = "⏳ PENDING"
            else:
                # Fallback
                test_class = "success" if overall_success else "failure"
                status_text = "✅ PASSED" if overall_success else "❌ FAILED"
            
            html += f"""
            <div class="test-case {test_class}">
                <div class="test-header">
                    <div class="test-title">{test['test_case']}</div>
                    <div class="test-status {test_class}">{status_text}</div>
                </div>
                <p><strong>Completion:</strong> {test.get('completion_status', 'unknown')}</p>
                
                <details open>
                    <summary>Step-by-Step Observations ({len(steps)} steps)</summary>
            """
            
            # Detailed observations
            for step in steps:
                step_num = step.get('step_number', 0)
                
                # Check if agent has interpreted this step
                # goal_achieved = agent's interpretation of whether the goal was met
                # If not set, agent analysis is still pending
                if 'goal_achieved' in step:
                    obs_success = step.get('goal_achieved', False)
                    needs_interpretation = False
                else:
                    # Fallback: show API status but mark as needing interpretation
                    obs_success = step.get('api_success', False)
                    needs_interpretation = step.get('needs_agent_analysis', True)
                
                if obs_success is True:
                    result_class = "success"
                    result_text = "✓"
                elif obs_success is False:
                    result_class = "failure"
                    result_text = "✗"
                else:
                    result_class = ""
                    result_text = "•"
                
                # Combine observations into notes - support both old and new formats
                observations_list = step.get('observations', [])
                raw_response = step.get('raw_response', '')
                error_msg = step.get('error', '')
                
                if observations_list:
                    notes = '; '.join(str(o) for o in observations_list)
                elif raw_response:
                    # Use raw_response as the observation
                    notes = f"Response: {raw_response}"
                elif error_msg:
                    notes = f"Error: {error_msg}"
                else:
                    notes = "No observations recorded"
                
                is_issue = any(word in notes.upper() for word in ['ERROR', 'FAILED', 'ISSUE', 'PROBLEM', 'CRITICAL'])
                notes_class = "issue" if is_issue else ""
                
                # Support both old format (action) and new format (prompt)
                action = step.get('action') or step.get('prompt', '') or 'No action'
                rationale = step.get('rationale') or step.get('expected_outcome', '')
                
                # Truncate action intelligently at word boundary
                if len(action) > 60:
                    action_display = action[:57].rsplit(' ', 1)[0] + '...'
                else:
                    action_display = action
                
                # Show warning if agent hasn't interpreted this step yet
                interpretation_warning = ""
                if needs_interpretation and 'goal_achieved' not in step:
                    interpretation_warning = '<div style="background: #fff3cd; padding: 5px 10px; border-radius: 3px; margin-top: 5px; font-size: 0.85em;">⏳ <strong>Awaiting agent interpretation</strong> - run analysis workflow to determine goal achievement</div>'
                
                html += f"""
                <div class="observation">
                    <div class="observation-header">
                        <span class="step-name">Step {step_num + 1}: {action_display}</span>
                        {f'<span class="step-result {result_class}">{result_text}</span>' if result_class else ''}
                    </div>
                    {f'<div style="color: #7f8c8d; font-size: 0.9em; margin: 5px 0;">Expected: {rationale}</div>' if rationale else ''}
                    <div class="observation-notes {notes_class}">
                        <strong>{"⚠️ " if is_issue else ""}Observation:</strong> {notes}
                    </div>
                    {interpretation_warning}
                </div>
                """
            
            html += """
                </details>
                
            """
            
            # Add Nova Act trace file links with global numbering
            trace_files = test.get('trace_files', [])
            if trace_files:
                # Calculate starting index for this test's recordings
                test_start_index = global_recording_index
                
                html += f"""
                <div style="margin-top: 15px; padding: 15px; background: #e3f2fd; border-radius: 4px;">
                    <strong>🔍 Nova Act Session Recordings ({len(trace_files)}):</strong>
                    <div style="margin-top: 10px;">
                """
                for trace_file in trace_files:
                    # Convert to WSL-compatible path if needed
                    browser_path = convert_to_wsl_path(trace_file)
                    display_name = os.path.basename(trace_file)
                    
                    html += f"""
                        <div style="margin: 5px 0;">
                            <a href="{browser_path}" class="trace-link" target="_blank">
                                📹 Recording {global_recording_index}: {display_name}
                            </a>
                            <span style="font-size: 0.85em; color: #666; margin-left: 10px;">
                                ({browser_path})
                            </span>
                        </div>
                    """
                    global_recording_index += 1
                    
                html += """
                    </div>
                    <p style="margin-top: 10px; font-size: 0.9em; color: #555;">
                        <em>Click to view detailed Nova Act trace showing every action, screenshot, and AI decision</em>
                    </p>
                </div>
                """
            
            html += "</div>\n"
        
        html += "</div>\n"
    
    # Key insights
    html += """
        <div class="insights">
            <h3>🔍 Key Insights</h3>
    """
    
    # Generate insights
    all_notes = []
    for test in results:
        for obs in test.get('observations', []):
            notes = obs.get('notes', '')
            if any(word in notes.upper() for word in ['ISSUE', 'FRICTION', 'CRITICAL', 'MAJOR', 'PROBLEM']):
                all_notes.append(notes)
    
    if all_notes:
        html += "<h4>UX Issues Discovered:</h4>\n"
        for note in set(all_notes):  # Unique issues
            html += f'<div class="insight-item">{note}</div>\n'
    
    # Success patterns
    successes = [t for t in results if t.get('overall_success', False)]
    if successes:
        html += "<h4>What Worked Well:</h4>\n"
        for test in successes[:3]:  # Top 3
            steps_count = len(test.get('steps', []))
            status = test.get('completion_status', 'unknown')
            html += f'<div class="insight-item">{test["test_case"]} - {status.title()} ({steps_count} steps)</div>\n'
    
    html += "</div>\n"
    
    # Session Recordings section (if traces provided)
    if traces:
        html += """
        <div class="insights" style="background: #e3f2fd; border-left-color: #2196f3;">
            <h3>🎬 Session Recordings</h3>
            <p style="margin-bottom: 15px;">Nova Act recorded detailed traces for each test session. Click to view step-by-step actions, screenshots, and AI decisions.</p>
        """
        for i, trace_file in enumerate(traces, 1):
            browser_path = convert_to_wsl_path(trace_file)
            display_name = os.path.basename(trace_file)
            # Extract session info from path if possible
            parent_dir = os.path.basename(os.path.dirname(trace_file))
            
            html += f"""
            <div style="margin: 8px 0; padding: 10px; background: white; border-radius: 4px;">
                <a href="{browser_path}" class="trace-link" target="_blank" style="text-decoration: none;">
                    📹 {display_name}
                </a>
                <div style="font-size: 0.85em; color: #666; margin-top: 5px;">
                    Session: {parent_dir}
                </div>
            </div>
            """
        html += "</div>\n"
    
    # Footer
    html += f"""
        <div class="footer">
            <p>Generated by Nova Act Usability Testing Suite | Powered by OpenClaw</p>
            <p>Report generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</p>
        </div>
    </div>
</body>
</html>
"""
    
    # Write report to current working directory
    report_path = os.path.join(os.getcwd(), "nova_act_usability_report.html")
    with open(report_path, 'w') as f:
        f.write(html)
    
    return report_path


path: scripts/dynamic_exploration.py
content: #!/usr/bin/env python3
"""
Dynamic exploration strategy generator.
Uses cookbook guidance to generate contextual Nova Act prompts and workflows.
"""

from typing import Dict, List, Tuple
import json
import re

# Material impact keywords that require safety stops
MATERIAL_IMPACT_KEYWORDS = [
    # Monetary
    "buy", "purchase", "checkout", "pay", "subscribe", "donate", "order",
    # Communication
    "post", "publish", "share", "send", "email", "message", "tweet",
    # Account creation
    "sign up", "register", "create account", "join",
    # Submissions
    "submit", "apply", "enroll", "book", "reserve",
    # Newsletter/notifications
    "newsletter", "get updates", "notify"
]

def requires_safety_stop(test_case: str) -> bool:
    """Check if test case involves material impact and needs safety stop."""
    test_lower = test_case.lower()
    return any(keyword in test_lower for keyword in MATERIAL_IMPACT_KEYWORDS)

def detect_workflow_type(test_case: str) -> Tuple[bool, str]:
    """
    Determine if this is a workflow test (multi-step journey) vs information-finding.
    
    Returns: (is_workflow, workflow_type)
    """
    workflows = {
        "booking": ["book", "reserve", "schedule", "appointment"],
        "purchasing": ["buy", "purchase", "order", "checkout", "add to cart"],
        "posting": ["post", "publish", "share", "upload", "tweet", "comment"],
        "signup": ["sign up", "register", "create account", "join"],
        "submission": ["submit", "send", "contact", "apply", "enroll"],
        "search": ["search for", "find product", "look for item"]
    }
    
    test_lower = test_case.lower()
    for workflow_type, keywords in workflows.items():
        if any(kw in test_lower for kw in keywords):
            return True, workflow_type
    
    return False, "information_finding"

def parse_cookbook_hints(cookbook: str) -> Dict:
    """
    Extract actionable hints from cookbook content.
    Returns guidance for prompt construction.
    """
    hints = {
        "use_loose_matching": True,  # Default: avoid quotes for flexibility
        "small_steps": True,  # Break into small steps
        "observe_after_action": True,  # Check results after each action
        "use_schemas": True,  # Use structured extraction
        "max_steps": 30,  # Keep under 30 steps
    }
    
    if not cookbook:
        return hints
    
    cookbook_lower = cookbook.lower()
    
    # Check for matching guidance
    if "exact matching" in cookbook_lower and "quotes" in cookbook_lower:
        hints["matching_guidance"] = "Use quotes only for exact text; prefer loose matching for flexibility"
    
    # Check for step size guidance
    if "fewer than 30 steps" in cookbook_lower or "small steps" in cookbook_lower:
        hints["step_guidance"] = "Keep tasks small and atomic"
    
    # Check for observation guidance
    if "observe and analyze" in cookbook_lower:
        hints["observation_guidance"] = "Verify each action succeeded before proceeding"
    
    return hints


def generate_exploration_strategy(
    test_case: str,
    persona: Dict,
    page_analysis: Dict,
    cookbook: str = ""
) -> List[Dict]:
    """
    Generate a dynamic exploration strategy for a given test case.
    Uses cookbook guidance for workflow testing and safety stops.
    
    Args:
        test_case: What the user wants to accomplish
        persona: User persona with archetype and tech_proficiency
        page_analysis: Analysis of the target page
        cookbook: Nova Act cookbook content for prompt guidance (Bug #14: NOW USED!)
    
    Returns a list of exploration steps, each containing:
    - step_name: Human-readable step name
    - action_type: "query" or "navigate" or "scroll"
    - prompt: The actual Nova Act prompt to use
    - expected_outcome: What we're looking for
    - fallback_prompts: Alternative prompts if first fails
    - is_safety_stop: True if this step should observe but not execute
    
    This replaces hardcoded if/elif logic with contextual, cookbook-informed prompts.
    """
    
    # Parse cookbook for actionable guidance (Bug #14: Use the cookbook!)
    cookbook_hints = parse_cookbook_hints(cookbook)
    
    # Extract context
    archetype = persona.get('archetype', 'user')
    tech_level = persona.get('tech_proficiency', 'medium')
    page_title = page_analysis.get('title', 'this page')
    navigation = page_analysis.get('navigation', [])
    key_elements = page_analysis.get('key_elements', {})
    
    # Detect if this is a workflow test
    is_workflow, workflow_type = detect_workflow_type(test_case)
    needs_safety_stop = requires_safety_stop(test_case)
    
    if is_workflow:
        print(f"  🔄 Detected workflow type: {workflow_type}")
    if needs_safety_stop:
        print(f"  ⚠️  Safety stop required - will not complete final action")
    if cookbook:
        print(f"  📖 Using cookbook guidance for prompts")
    
    strategy = []
    
    # Apply cookbook guidance to prompts (Bug #14: Actually use cookbook!)
    def apply_cookbook_guidance(prompt: str) -> str:
        """Apply cookbook best practices to a prompt."""
        modified = prompt
        
        # Cookbook says: use loose matching (no quotes) for flexibility
        if cookbook_hints.get("use_loose_matching"):
            # Remove unnecessary quotes around search terms
            # e.g., 'link labeled "Documentation"' -> 'link with Documentation'
            modified = re.sub(r'"([^"]+)"', r'\1', modified)
        
        # Cookbook says: be direct and specific
        if cookbook_hints.get("step_guidance"):
            # Remove vague language
            modified = modified.replace("Let's see", "Check")
            modified = modified.replace("try to find", "find")
            modified = modified.replace("maybe look for", "look for")
        
        return modified
    
    # ===== WORKFLOW TESTS (Multi-step journeys) =====
    if is_workflow:
        if workflow_type == "booking":
            # Flight/hotel/appointment booking workflow
            strategy.extend([
                {
                    "step_name": "find_booking_form",
                    "action_type": "query",
                    "prompt": "Is there a search form or booking interface visible on this page?",
                    "expected_outcome": "Locate booking form",
                    "fallback_prompts": [
                        "Do you see input fields for dates, location, or search?",
                        "Is there a 'Book Now' or 'Search' button visible?"
                    ]
                },
                {
                    "step_name": "fill_search_criteria",
                    "action_type": "navigate",
                    "prompt": "Fill in the search form with test data: departure 'New York', destination 'Los Angeles', dates 2 weeks from now",
                    "expected_outcome": "Search form filled",
                    "fallback_prompts": []
                },
                {
                    "step_name": "initiate_search",
                    "action_type": "navigate",
                    "prompt": "Click the search or submit button to see results",
                    "expected_outcome": "Results displayed",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_results",
                    "action_type": "query",
                    "prompt": "Are booking options or results now displayed with relevant information?",
                    "expected_outcome": "Results visible",
                    "fallback_prompts": ["Did the page change to show available options?"]
                },
                {
                    "step_name": "select_option",
                    "action_type": "navigate",
                    "prompt": "Select the first available option",
                    "expected_outcome": "Option selected",
                    "fallback_prompts": []
                },
                {
                    "step_name": "proceed_to_details",
                    "action_type": "navigate",
                    "prompt": "Click continue, next, or proceed to enter details",
                    "expected_outcome": "Details form shown",
                    "fallback_prompts": []
                },
                {
                    "step_name": "fill_user_details",
                    "action_type": "navigate",
                    "prompt": "Fill in user details: Name 'Test User', Email '[REDACTED_SECRET]', Phone '5550123'",
                    "expected_outcome": "Details filled",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_checkout_accessible",
                    "action_type": "query",
                    "prompt": "Is there a 'Continue to Payment', 'Checkout', or 'Complete Booking' button visible?",
                    "expected_outcome": "⚠️ SAFETY STOP: Checkout accessible but NOT clicked",
                    "fallback_prompts": [],
                    "is_safety_stop": True
                }
            ])
            
        elif workflow_type == "purchasing":
            # E-commerce purchase workflow
            strategy.extend([
                {
                    "step_name": "search_product",
                    "action_type": "navigate",
                    "prompt": "Find and use the search function to search for 'laptop' or similar product",
                    "expected_outcome": "Product search executed",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_search_results",
                    "action_type": "query",
                    "prompt": "Are product search results displayed with images and prices?",
                    "expected_outcome": "Results shown",
                    "fallback_prompts": []
                },
                {
                    "step_name": "select_product",
                    "action_type": "navigate",
                    "prompt": "Click on the first product in the results",
                    "expected_outcome": "Product page loaded",
                    "fallback_prompts": []
                },
                {
                    "step_name": "add_to_cart",
                    "action_type": "navigate",
                    "prompt": "Click the 'Add to Cart' or 'Buy' button",
                    "expected_outcome": "Item added to cart",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_cart_updated",
                    "action_type": "query",
                    "prompt": "Is there confirmation the item was added (cart icon updated, notification shown, etc.)?",
                    "expected_outcome": "Cart confirmation",
                    "fallback_prompts": []
                },
                {
                    "step_name": "navigate_to_cart",
                    "action_type": "navigate",
                    "prompt": "Click on the cart icon or 'View Cart' button",
                    "expected_outcome": "Cart page loaded",
                    "fallback_prompts": []
                },
                {
                    "step_name": "proceed_to_checkout",
                    "action_type": "navigate",
                    "prompt": "Click 'Proceed to Checkout' or 'Checkout' button",
                    "expected_outcome": "Checkout initiated",
                    "fallback_prompts": []
                },
                {
                    "step_name": "fill_shipping",
                    "action_type": "navigate",
                    "prompt": "Fill shipping information: Name 'Test User', Address '123 Test St', City 'Test City', ZIP '12345'",
                    "expected_outcome": "Shipping filled",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_payment_page",
                    "action_type": "query",
                    "prompt": "Is there a payment method section, credit card form, or 'Complete Purchase' button visible?",
                    "expected_outcome": "⚠️ SAFETY STOP: Payment page accessible but NO PURCHASE MADE",
                    "fallback_prompts": [],
                    "is_safety_stop": True
                }
            ])
            
        elif workflow_type == "posting":
            # Social media posting workflow
            strategy.extend([
                {
                    "step_name": "find_create_button",
                    "action_type": "query",
                    "prompt": "Is there a 'Create Post', 'New Post', 'Tweet', or similar button visible?",
                    "expected_outcome": "Post creation button found",
                    "fallback_prompts": [
                        "Do you see a text box where you can write something?",
                        "Is there a '+' or 'Compose' button?"
                    ]
                },
                {
                    "step_name": "click_create",
                    "action_type": "navigate",
                    "prompt": "Click the post creation button or click in the text area",
                    "expected_outcome": "Compose interface opened",
                    "fallback_prompts": []
                },
                {
                    "step_name": "enter_content",
                    "action_type": "navigate",
                    "prompt": "Type test content: 'This is a usability test post - please ignore'",
                    "expected_outcome": "Content entered",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_post_button",
                    "action_type": "query",
                    "prompt": "Is there a 'Post', 'Publish', 'Share', or 'Tweet' button visible to complete posting?",
                    "expected_outcome": "⚠️ SAFETY STOP: Post button accessible but NOT clicked",
                    "fallback_prompts": [],
                    "is_safety_stop": True
                }
            ])
            
        elif workflow_type == "signup":
            # Account creation workflow
            strategy.extend([
                {
                    "step_name": "find_signup",
                    "action_type": "query",
                    "prompt": "Is there a 'Sign Up', 'Register', 'Create Account', or 'Join' button or link visible?",
                    "expected_outcome": "Signup button found",
                    "fallback_prompts": []
                },
                {
                    "step_name": "click_signup",
                    "action_type": "navigate",
                    "prompt": "Click the signup or register button",
                    "expected_outcome": "Registration form loaded",
                    "fallback_prompts": []
                },
                {
                    "step_name": "fill_registration",
                    "action_type": "navigate",
                    "prompt": "Fill registration form: Email '[REDACTED_SECRET]', Username 'testuser123', Password 'TestPass123!'",
                    "expected_outcome": "Form filled",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_submit_button",
                    "action_type": "query",
                    "prompt": "Is there a 'Create Account', 'Sign Up', or 'Register' button to submit the form?",
                    "expected_outcome": "⚠️ SAFETY STOP: Submit button accessible but NOT clicked",
                    "fallback_prompts": [],
                    "is_safety_stop": True
                }
            ])
            
        elif workflow_type == "submission":
            # Form submission workflow (contact, newsletter, etc.)
            strategy.extend([
                {
                    "step_name": "find_form",
                    "action_type": "query",
                    "prompt": "Is there a contact form, submission form, or input fields visible?",
                    "expected_outcome": "Form located",
                    "fallback_prompts": [
                        "Do you see a 'Contact Us' form?",
                        "Is there an email or message input field?"
                    ]
                },
                {
                    "step_name": "fill_form_fields",
                    "action_type": "navigate",
                    "prompt": "Fill the form with test data: Name 'Test User', Email '[REDACTED_SECRET]', Message 'Usability test message'",
                    "expected_outcome": "Fields filled",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_submit_accessible",
                    "action_type": "query",
                    "prompt": "Is there a 'Submit', 'Send', or 'Contact Us' button visible to complete submission?",
                    "expected_outcome": "⚠️ SAFETY STOP: Submit button accessible but NOT clicked",
                    "fallback_prompts": [],
                    "is_safety_stop": True
                }
            ])
            
        elif workflow_type == "search":
            # Product/content search workflow
            strategy.extend([
                {
                    "step_name": "find_search",
                    "action_type": "query",
                    "prompt": "Is there a search bar or search button visible?",
                    "expected_outcome": "Search function located",
                    "fallback_prompts": ["Do you see a magnifying glass icon or search input?"]
                },
                {
                    "step_name": "perform_search",
                    "action_type": "navigate",
                    "prompt": "Enter 'test query' into the search and press enter or click search",
                    "expected_outcome": "Search executed",
                    "fallback_prompts": []
                },
                {
                    "step_name": "verify_results",
                    "action_type": "query",
                    "prompt": "Are search results displayed that are relevant to the query?",
                    "expected_outcome": "Results shown",
                    "fallback_prompts": ["Did the page show any matching items or content?"]
                },
                {
                    "step_name": "assess_result_quality",
                    "action_type": "query",
                    "prompt": "Are the results clearly presented with relevant information (titles, descriptions, images)?",
                    "expected_outcome": "Results quality assessment",
                    "fallback_prompts": []
                }
            ])
        
        # Apply cookbook guidance to all prompts before returning
        for step in strategy:
            step['prompt'] = apply_cookbook_guidance(step['prompt'])
            step['fallback_prompts'] = [apply_cookbook_guidance(p) for p in step.get('fallback_prompts', [])]
        
        # Return workflow strategy early (don't mix with info-finding)
        return strategy
    
    # ===== INFORMATION-FINDING TESTS (Original logic) =====
    # Analyze the test case and generate appropriate exploration steps
    test_lower = test_case.lower()
    
    # ===== DOCUMENTATION / TECHNICAL RESOURCES =====
    if any(keyword in test_lower for keyword in ['documentation', 'docs', 'api', 'technical', 'developer']):
        # Step 1: Look in navigation first
        strategy.append({
            "step_name": "check_navigation_for_docs",
            "action_type": "query",
            "prompt": f"Is there a navigation link related to documentation, API, developer resources, or technical guides?",
            "expected_outcome": "Find docs link in nav",
            "fallback_prompts": [
                "Do you see any link with 'Docs', 'API', 'Developer', 'Guide', or 'Reference'?",
                "Is there a 'Resources' or 'Learn' section in the navigation?"
            ]
        })
        
        # Step 2: Check page content if nav fails
        strategy.append({
            "step_name": "check_page_for_docs",
            "action_type": "query",
            "prompt": f"Looking at the main content area, is there any mention of documentation, getting started guides, or technical resources?",
            "expected_outcome": "Find docs reference in content",
            "fallback_prompts": [
                "Are there any code snippets, API examples, or technical documentation visible?",
                f"Does {page_title} explain how developers can use or integrate this?"
            ]
        })
        
        # Tech-savvy users might look for specific things
        if tech_level == "high":
            strategy.append({
                "step_name": "check_advanced_docs",
                "action_type": "query",
                "prompt": "Are there links to GitHub, SDK downloads, or API reference documentation?",
                "expected_outcome": "Find advanced technical resources",
                "fallback_prompts": []
            })
    
    # ===== DEMO / INTERACTIVE FEATURES =====
    elif any(keyword in test_lower for keyword in ['demo', 'playground', 'try', 'interactive', 'example']):
        # Step 1: Look for interactive elements
        strategy.append({
            "step_name": "find_interactive_element",
            "action_type": "query",
            "prompt": "Is there an input field, text box, or interactive area where you can try something out?",
            "expected_outcome": "Find interactive element on page",
            "fallback_prompts": [
                "Do you see a 'Try it', 'Demo', or 'Playground' button or section?",
                "Is there anywhere you can type or interact with a live example?"
            ]
        })
        
        # Step 2: Check navigation
        strategy.append({
            "step_name": "check_nav_for_demo",
            "action_type": "query",
            "prompt": "Is there a navigation link for 'Demo', 'Playground', 'Try', or 'Examples'?",
            "expected_outcome": "Find demo link in navigation",
            "fallback_prompts": [
                "Do you see 'Get Started', 'Live Demo', or 'Interactive' in the menu?"
            ]
        })
        
        # Beginners need obvious calls-to-action
        if tech_level == "low":
            strategy.append({
                "step_name": "check_prominent_cta",
                "action_type": "query",
                "prompt": "Is there a large, obvious button near the top that invites you to try or start using the tool?",
                "expected_outcome": "Find beginner-friendly CTA",
                "fallback_prompts": []
            })
    
    # ===== PRICING / COST INFORMATION =====
    elif any(keyword in test_lower for keyword in ['pricing', 'price', 'cost', 'plan', 'subscription', 'fee']):
        # Step 1: Navigation check
        strategy.append({
            "step_name": "check_nav_for_pricing",
            "action_type": "query",
            "prompt": "Is there a 'Pricing', 'Plans', or 'Cost' link in the navigation menu?",
            "expected_outcome": "Find pricing in navigation",
            "fallback_prompts": [
                "Do you see 'Subscribe', 'Buy', or 'Get Started' with pricing info?",
                "Is there a 'Free' or 'Pro' tier mentioned in the navigation?"
            ]
        })
        
        # Step 2: Visible pricing on current page
        strategy.append({
            "step_name": "check_visible_pricing",
            "action_type": "query",
            "prompt": "Is there any pricing information, cost, or subscription tiers visible on the current page?",
            "expected_outcome": "Find pricing in content",
            "fallback_prompts": [
                "Do you see dollar amounts, price tags, or cost comparisons?",
                "Is there mention of 'free', 'premium', or different plan levels?"
            ]
        })
        
        # Step 3: Scroll for pricing
        strategy.append({
            "step_name": "scroll_for_pricing",
            "action_type": "scroll",
            "direction": "down",
            "prompt": "After scrolling, do you now see any pricing, cost, or subscription information?",
            "expected_outcome": "Find pricing after scroll",
            "fallback_prompts": []
        })
        
        # Business users care about transparency
        if archetype == "business_professional":
            strategy.append({
                "step_name": "check_transparent_pricing",
                "action_type": "query",
                "prompt": "If pricing is visible, are the actual dollar amounts clearly displayed, or is it 'contact us' / 'request quote'?",
                "expected_outcome": "Assess pricing transparency",
                "fallback_prompts": []
            })
    
    # ===== VALUE PROPOSITION / UNDERSTANDING =====
    elif any(keyword in test_lower for keyword in ['understand', 'value', 'what', 'purpose', 'does', 'benefit']):
        # Step 1: Check hero section
        strategy.append({
            "step_name": "check_hero_tagline",
            "action_type": "query",
            "prompt": "Near the top of the page, is there a clear headline or tagline that explains what this tool/product does or what problem it solves?",
            "expected_outcome": "Find clear value proposition",
            "fallback_prompts": [
                f"Does {page_title} have a sentence that tells you immediately what it's for?",
                "Is the main benefit or use case explained prominently?"
            ]
        })
        
        # Step 2: Supporting copy
        strategy.append({
            "step_name": "check_supporting_copy",
            "action_type": "query",
            "prompt": "Below the main headline, is there supporting text that elaborates on features, benefits, or how it works?",
            "expected_outcome": "Find detailed explanation",
            "fallback_prompts": [
                "Are there bullet points, feature lists, or use cases described?",
                "Can you quickly understand what makes this different or useful?"
            ]
        })
        
        # Beginners need simple language
        if tech_level == "low":
            strategy.append({
                "step_name": "check_simple_language",
                "action_type": "query",
                "prompt": "Is the description written in simple, non-technical language that anyone could understand?",
                "expected_outcome": "Assess language clarity",
                "fallback_prompts": [
                    "Are there lots of jargon, acronyms, or technical terms that might confuse a beginner?"
                ]
            })
    
    # ===== HELP / SUPPORT =====
    elif any(keyword in test_lower for keyword in ['help', 'support', 'contact', 'assistance', 'faq']):
        # Step 1: Header check
        strategy.append({
            "step_name": "check_header_for_help",
            "action_type": "query",
            "prompt": "In the top navigation or header area, is there a 'Help', 'Support', 'Contact', or 'FAQ' link?",
            "expected_outcome": "Find help in header",
            "fallback_prompts": [
                "Do you see 'Get Help', 'Contact Us', 'Support Center', or similar?",
                "Is there a question mark icon or help button visible?"
            ]
        })
        
        # Step 2: Footer check (scroll required)
        strategy.append({
            "step_name": "scroll_to_footer",
            "action_type": "scroll",
            "direction": "down",
            "prompt": "In the footer area, is there a 'Help', 'Support', 'Contact', or 'FAQ' link?",
            "expected_outcome": "Find help in footer",
            "fallback_prompts": [
                "Do you see contact email, phone number, or support links at the bottom?",
                "Is there a 'Help Center' or 'Resources' section in the footer?"
            ]
        })
        
        # Beginners might need more obvious help options
        if tech_level == "low":
            strategy.append({
                "step_name": "check_chat_widget",
                "action_type": "query",
                "prompt": "Is there a chat widget, chatbot, or live support button visible anywhere on the page?",
                "expected_outcome": "Find live help option",
                "fallback_prompts": []
            })
    
    # ===== GETTING STARTED / ONBOARDING =====
    elif any(keyword in test_lower for keyword in ['getting started', 'get started', 'onboard', 'begin', 'setup']):
        strategy.append({
            "step_name": "check_cta",
            "action_type": "query",
            "prompt": "Is there a prominent 'Get Started', 'Sign Up', 'Try Now', or 'Start Free' button or call-to-action?",
            "expected_outcome": "Find getting started CTA",
            "fallback_prompts": [
                "What's the main action button on this page?",
                "How would a new user begin using this product?"
            ]
        })
        
        strategy.append({
            "step_name": "check_onboarding_guide",
            "action_type": "query",
            "prompt": "Is there a 'Getting Started Guide', 'Quick Start', or step-by-step tutorial linked or visible?",
            "expected_outcome": "Find onboarding resources",
            "fallback_prompts": [
                "Are there numbered steps or a 'how to get started' section?",
                "Does the page explain the first steps clearly?"
            ]
        })
    
    # ===== GENERIC FALLBACK =====
    else:
        # If we don't recognize the test case, generate direct browser tasks
        # NOTE: Nova Act should NOT reason about personas - just execute browser actions
        # The agent interprets results in persona context afterward
        
        strategy.append({
            "step_name": "check_navigation",
            "action_type": "query",
            "prompt": f"List the main navigation menu items visible on this page",
            "expected_outcome": "Identify available navigation options",
            "fallback_prompts": [
                "What links are visible in the header or top navigation?",
                "What menu options can you see?"
            ]
        })
        
        strategy.append({
            "step_name": "find_relevant_link",
            "action_type": "query",
            "prompt": f"Is there a link or button related to: {test_case}?",
            "expected_outcome": "Find relevant element",
            "fallback_prompts": [
                f"What elements on this page might help with: {test_case}?",
                f"Do you see any text mentioning: {test_case}?"
            ]
        })
    
    # Apply cookbook guidance to all prompts before returning (Bug #14)
    for step in strategy:
        step['prompt'] = apply_cookbook_guidance(step['prompt'])
        step['fallback_prompts'] = [apply_cookbook_guidance(p) for p in step.get('fallback_prompts', [])]
    
    return strategy


def adapt_prompt_for_persona(base_prompt: str, persona: Dict) -> str:
    """
    NOTE: This function is intentionally minimal.
    
    Nova Act should NOT reason about personas - it just executes browser tasks.
    The Claude agent interprets results in the context of the persona afterward.
    
    We keep prompts simple and direct. The agent decides:
    - WHAT to test (based on persona goals)
    - HOW to interpret results (based on persona characteristics)
    
    Nova Act just:
    - Clicks, types, scrolls
    - Reports what it sees
    """
    # Return prompt unchanged - no persona-specific modifications
    # Persona context is used by the agent when interpreting results, not by Nova Act
    return base_prompt


def generate_fallback_questions(failed_step: Dict, context: Dict) -> List[str]:
    """
    If a step fails, generate contextual follow-up questions to try.
    
    This allows the test to adapt when the first approach doesn't work.
    """
    step_name = failed_step.get('step_name', '')
    
    fallbacks = []
    
    # If navigation check failed, try content area
    if 'navigation' in step_name or 'nav' in step_name:
        fallbacks.append("Ignoring the navigation menu, is the information visible anywhere in the main content area?")
        fallbacks.append("Is there a search box where you could search for this?")
    
    # If exact match failed, try fuzzy matching
    if 'exact' in step_name or 'specific' in step_name:
        fallbacks.append("Using different wording, is there anything similar or related visible?")
        fallbacks.append("What IS visible that might be related?")
    
    # If scroll didn't help, try other strategies
    if 'scroll' in step_name:
        fallbacks.append("Back at the top of the page, did we miss anything obvious?")
        fallbacks.append("Is there a sitemap, footer navigation, or breadcrumbs that might help?")
    
    return fallbacks


path: scripts/safe_nova_wrapper.py
content: #!/usr/bin/env python3
"""
Robust wrapper for Nova Act calls with error handling and timeout protection.
"""

import time
import functools
from typing import Any, Callable, Optional, Tuple
from dataclasses import dataclass

# ============================================================================
# Configuration (Bug #15: Centralized defaults)
# ============================================================================

DEFAULT_TIMEOUT = 20  # seconds
DEFAULT_MAX_RETRIES = 1
SLOW_OPERATION_THRESHOLD = 15  # seconds
HEALTH_CHECK_TIMEOUT = 10  # seconds


# ============================================================================
# Result Types (Bug #12: Consistent error handling)
# ============================================================================

@dataclass
class ActResult:
    """Result of a Nova Act action."""
    success: bool
    observation: Optional[str] = None
    error: Optional[str] = None
    duration: float = 0.0


@dataclass
class QueryResult:
    """Result of a Nova Act query."""
    success: bool
    data: Any = None
    error: Optional[str] = None
    duration: float = 0.0


# ============================================================================
# Exceptions
# ============================================================================

class NovaActTimeout(Exception):
    """Raised when a Nova Act operation times out."""
    pass


class NovaActError(Exception):
    """Raised when a Nova Act operation fails."""
    pass


# ============================================================================
# Timeout Decorator
# ============================================================================

def with_timeout(timeout_seconds: int = DEFAULT_TIMEOUT):
    """
    Decorator to add timeout protection to Nova Act calls.
    Uses threading for cross-platform support (Bug #7).
    """
    def decorator(func: Callable) -> Callable:
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            import signal
            
            def timeout_handler(signum, frame):
                raise NovaActTimeout(f"Operation timed out after {timeout_seconds}s")
            
            # Set up timeout (Unix only - Windows falls back to no timeout)
            try:
                old_handler = signal.signal(signal.SIGALRM, timeout_handler)
                signal.alarm(timeout_seconds)
                try:
                    result = func(*args, **kwargs)
                finally:
                    signal.alarm(0)
                    signal.signal(signal.SIGALRM, old_handler)
                return result
            except AttributeError:
                # Windows doesn't have SIGALRM, fall back to no timeout
                return func(*args, **kwargs)
        
        return wrapper
    return decorator


# ============================================================================
# Safe Wrappers (Bug #8: Return observations)
# ============================================================================

def safe_act(nova, action: str, timeout: int = DEFAULT_TIMEOUT, 
             max_retries: int = DEFAULT_MAX_RETRIES) -> ActResult:
    """
    Safely execute a Nova Act action with error handling.
    
    Returns:
        ActResult with success, observation, error, and duration
    """
    for attempt in range(max_retries):
        try:
            start = time.time()
            result = nova.act(action)
            duration = time.time() - start
            
            # Extract observation from result if available
            observation = None
            if result is not None:
                if hasattr(result, 'response') and result.response:
                    observation = str(result.response)
                elif hasattr(result, 'matches_response') and result.matches_response:
                    observation = str(result.matches_response)
                elif isinstance(result, str):
                    observation = result
            
            # Detect if it took suspiciously long (possible loop)
            if duration > SLOW_OPERATION_THRESHOLD:
                return ActResult(
                    success=False,
                    observation=observation,
                    error=f"Action took {duration:.1f}s (possible scroll loop or hang)",
                    duration=duration
                )
            
            return ActResult(
                success=True,
                observation=observation or f"Action completed: {action[:50]}...",
                duration=duration
            )
            
        except Exception as e:
            error_msg = str(e)
            duration = time.time() - start if 'start' in dir() else 0
            
            # Check for common Nova Act errors
            if "timeout" in error_msg.lower():
                return ActResult(success=False, error="Nova Act timeout - page may be unresponsive", duration=duration)
            elif "element not found" in error_msg.lower():
                return ActResult(success=False, error="Element not found on page", duration=duration)
            elif "scroll" in error_msg.lower() and "loop" in error_msg.lower():
                return ActResult(success=False, error="Scroll loop detected", duration=duration)
            elif attempt < max_retries - 1:
                time.sleep(2)  # Brief pause before retry
                continue
            else:
                return ActResult(success=False, error=f"Nova Act error: {error_msg}", duration=duration)
    
    return ActResult(success=False, error="Max retries exceeded")


def safe_act_get(nova, query: str, schema: Any, timeout: int = DEFAULT_TIMEOUT,
                 max_retries: int = DEFAULT_MAX_RETRIES) -> QueryResult:
    """
    Safely execute a Nova Act query with error handling.
    
    Returns:
        QueryResult with success, data, error, and duration
    """
    for attempt in range(max_retries):
        try:
            start = time.time()
            result = nova.act_get(query, schema=schema)
            duration = time.time() - start
            
            # Detect if it took suspiciously long
            if duration > SLOW_OPERATION_THRESHOLD:
                return QueryResult(
                    success=False,
                    data=result.parsed_response if result else None,
                    error=f"Query took {duration:.1f}s (possible hang)",
                    duration=duration
                )
            
            return QueryResult(
                success=True,
                data=result.parsed_response,
                duration=duration
            )
            
        except Exception as e:
            error_msg = str(e)
            duration = time.time() - start if 'start' in dir() else 0
            
            if "timeout" in error_msg.lower():
                return QueryResult(success=False, error="Nova Act timeout", duration=duration)
            elif attempt < max_retries - 1:
                time.sleep(2)
                continue
            else:
                return QueryResult(success=False, error=f"Nova Act error: {error_msg}", duration=duration)
    
    return QueryResult(success=False, error="Max retries exceeded")


def safe_scroll(nova, direction: str = "down", max_attempts: int = 3) -> ActResult:
    """
    Safely scroll with loop detection.
    
    Args:
        nova: Nova Act session
        direction: "down" or "up"
        max_attempts: Max scroll attempts before declaring loop
    
    Returns:
        ActResult with success, observation, and error
    """
    for attempt in range(max_attempts):
        try:
            start = time.time()
            
            if direction == "down":
                nova.act("Scroll down to see more content")
            else:
                nova.act("Scroll up")
            
            duration = time.time() - start
            
            # If scroll takes too long, probably stuck
            if duration > 10:
                return ActResult(
                    success=False,
                    error="Scroll operation taking too long (possible loop)",
                    duration=duration
                )
            
            # Small delay to let page settle
            time.sleep(1)
            
            return ActResult(
                success=True,
                observation=f"Scrolled {direction}",
                duration=duration
            )
            
        except Exception as e:
            error_msg = str(e)
            if "scroll" in error_msg.lower() and "loop" in error_msg.lower():
                return ActResult(success=False, error="Scroll loop detected by Nova Act")
            elif attempt < max_attempts - 1:
                time.sleep(2)
                continue
            else:
                return ActResult(success=False, error=f"Scroll error: {error_msg}")
    
    return ActResult(success=False, error="Scroll stuck (max attempts)")


def is_session_healthy(nova) -> bool:
    """
    Quick health check to see if Nova Act session is still responsive.
    
    Returns:
        True if session is healthy, False otherwise
    """
    try:
        start = time.time()
        # Simple query that should always work
        result = nova.act_get(
            "Is there any text visible on the page?",
            schema={"type": "boolean"}
        )
        duration = time.time() - start
        
        # If it takes more than threshold for a simple check, session is degraded
        return duration < HEALTH_CHECK_TIMEOUT and result.parsed_response is not None
        
    except:
        return False


# ============================================================================
# Legacy Compatibility (tuple returns)
# ============================================================================

def safe_act_tuple(nova, action: str, timeout: int = DEFAULT_TIMEOUT,
                   max_retries: int = DEFAULT_MAX_RETRIES) -> Tuple[bool, Optional[str]]:
    """Legacy compatibility: returns (success, error_or_observation)."""
    result = safe_act(nova, action, timeout, max_retries)
    if result.success:
        return True, result.observation
    return False, result.error


def safe_act_get_tuple(nova, query: str, schema: Any, timeout: int = DEFAULT_TIMEOUT,
                       max_retries: int = DEFAULT_MAX_RETRIES) -> Tuple[bool, Any, Optional[str]]:
    """Legacy compatibility: returns (success, data, error)."""
    result = safe_act_get(nova, query, schema, timeout, max_retries)
    return result.success, result.data, result.error


def safe_scroll_tuple(nova, direction: str = "down", max_attempts: int = 3) -> Tuple[bool, Optional[str]]:
    """Legacy compatibility: returns (success, error_or_observation)."""
    result = safe_scroll(nova, direction, max_attempts)
    if result.success:
        return True, result.observation
    return False, result.error


path: scripts/status_reporter.py
content: #!/usr/bin/env python3
"""
Status reporter for providing 60-second progress updates during long-running tests.
"""

import time
import threading
from typing import Optional, Callable
from datetime import datetime

class StatusReporter:
    """
    Reports test progress every 60 seconds in a background thread.
    """
    
    def __init__(self, update_interval: int = 60, output_func: Optional[Callable] = None):
        """
        Initialize status reporter.
        
        Args:
            update_interval: Seconds between status updates (default 60)
            output_func: Function to call with status messages (default: print)
        """
        self.update_interval = update_interval
        self.output = output_func or print
        self.running = False
        self.thread = None
        self.start_time = None
        
        # Status tracking
        self.current_phase = "Initializing"
        self.current_persona = None
        self.current_test = None
        self.tests_completed = 0
        self.tests_total = 0
        self.tests_passed = 0
        self.tests_failed = 0
    
    def start(self):
        """Start background status reporting."""
        if self.running:
            return
        
        self.running = True
        self.start_time = time.time()
        self.thread = threading.Thread(target=self._report_loop, daemon=True)
        self.thread.start()
    
    def stop(self):
        """Stop background status reporting."""
        self.running = False
        if self.thread:
            self.thread.join(timeout=2)
    
    def update_phase(self, phase: str):
        """Update current test phase."""
        self.current_phase = phase
    
    def update_persona(self, persona: str):
        """Update current persona being tested."""
        self.current_persona = persona
    
    def update_test(self, test_name: str, total: Optional[int] = None):
        """Update current test being executed."""
        self.current_test = test_name
        if total is not None:
            self.tests_total = total
    
    def mark_test_complete(self, success: bool):
        """Mark current test as complete."""
        self.tests_completed += 1
        if success:
            self.tests_passed += 1
        else:
            self.tests_failed += 1
        self.current_test = None
    
    def _report_loop(self):
        """Background loop that reports status every interval."""
        last_report = time.time()
        
        while self.running:
            time.sleep(1)  # Check every second
            
            if time.time() - last_report >= self.update_interval:
                self._emit_status()
                last_report = time.time()
    
    def _emit_status(self):
        """Emit current status report."""
        elapsed = time.time() - self.start_time if self.start_time else 0
        elapsed_min = int(elapsed // 60)
        elapsed_sec = int(elapsed % 60)
        
        # Build status message
        status_lines = [
            "",
            "="*60,
            f"📊 STATUS UPDATE ({datetime.now().strftime('%H:%M:%S')})",
            "="*60,
            f"⏱️  Elapsed Time: {elapsed_min}m {elapsed_sec}s",
            f"🔄 Current Phase: {self.current_phase}",
        ]
        
        if self.current_persona:
            status_lines.append(f"👤 Testing Persona: {self.current_persona}")
        
        if self.current_test:
            status_lines.append(f"🧪 Current Test: {self.current_test}")
        
        if self.tests_total > 0:
            progress_pct = (self.tests_completed / self.tests_total * 100)
            status_lines.append(f"📈 Progress: {self.tests_completed}/{self.tests_total} tests ({progress_pct:.0f}%)")
            status_lines.append(f"✅ Passed: {self.tests_passed}  |  ❌ Failed: {self.tests_failed}")
        
        status_lines.append("="*60)
        status_lines.append("")
        
        self.output("\n".join(status_lines))
    
    def emit_final_report(self):
        """Emit final status when test completes."""
        elapsed = time.time() - self.start_time if self.start_time else 0
        elapsed_min = int(elapsed // 60)
        elapsed_sec = int(elapsed % 60)
        
        success_rate = (self.tests_passed / self.tests_total * 100) if self.tests_total > 0 else 0
        
        report_lines = [
            "",
            "="*60,
            "🎉 TEST COMPLETE",
            "="*60,
            f"⏱️  Total Time: {elapsed_min}m {elapsed_sec}s",
            f"📊 Tests Run: {self.tests_completed}/{self.tests_total}",
            f"✅ Passed: {self.tests_passed}",
            f"❌ Failed: {self.tests_failed}",
            f"📈 Success Rate: {success_rate:.1f}%",
            "="*60,
            ""
        ]
        
        self.output("\n".join(report_lines))


# Convenience functions for easy integration
_global_reporter: Optional[StatusReporter] = None

def start_status_reporter(update_interval: int = 60):
    """Start global status reporter (call at test start)."""
    global _global_reporter
    if _global_reporter is None:
        _global_reporter = StatusReporter(update_interval=update_interval)
    _global_reporter.start()
    return _global_reporter

def stop_status_reporter():
    """Stop global status reporter (call at test end)."""
    global _global_reporter
    if _global_reporter:
        _global_reporter.stop()

def get_reporter() -> Optional[StatusReporter]:
    """Get the global status reporter instance."""
    return _global_reporter

def update_status(phase: Optional[str] = None, 
                  persona: Optional[str] = None, 
                  test: Optional[str] = None,
                  total: Optional[int] = None):
    """Update current test status (convenience function)."""
    reporter = get_reporter()
    if reporter:
        if phase:
            reporter.update_phase(phase)
        if persona:
            reporter.update_persona(persona)
        if test:
            reporter.update_test(test, total)

def mark_complete(success: bool):
    """Mark current test as complete (convenience function)."""
    reporter = get_reporter()
    if reporter:
        reporter.mark_test_complete(success)

def emit_final():
    """Emit final status report (convenience function)."""
    reporter = get_reporter()
    if reporter:
        reporter.emit_final_report()


path: scripts/trace_finder.py
content: #!/usr/bin/env python3
"""
Utility to find Nova Act trace files for a session.
"""

import os
import glob
from typing import List


def find_trace_files(logs_dir: str, session_start_time: float) -> List[str]:
    """
    Find Nova Act HTML trace files generated during a test session.
    
    Args:
        logs_dir: Directory where Nova Act stores logs
        session_start_time: Unix timestamp when session started
    
    Returns:
        List of paths to HTML trace files
    """
    if not os.path.exists(logs_dir):
        return []
    
    # Find all HTML files in the logs directory and subdirectories
    pattern = os.path.join(logs_dir, "**", "*.html")
    html_files = glob.glob(pattern, recursive=True)
    
    # Filter to files created after session start
    trace_files = []
    for filepath in html_files:
        file_mtime = os.path.getmtime(filepath)
        if file_mtime >= session_start_time:
            trace_files.append(filepath)
    
    return sorted(trace_files, key=os.path.getmtime)


def get_latest_session_dir(logs_dir: str) -> str:
    """
    Get the most recently created session directory.
    Nova Act creates directories like: <logs_dir>/<session_id>/
    """
    if not os.path.exists(logs_dir):
        return ""
    
    # Find all subdirectories
    subdirs = [d for d in glob.glob(os.path.join(logs_dir, "*")) if os.path.isdir(d)]
    
    if not subdirs:
        return ""
    
    # Return most recently created
    return max(subdirs, key=os.path.getmtime)


def get_session_traces(logs_dir: str) -> List[str]:
    """
    Get all trace files from the most recent session directory.
    
    Args:
        logs_dir: Base logs directory
    
    Returns:
        List of HTML trace file paths
    """
    session_dir = get_latest_session_dir(logs_dir)
    
    if not session_dir:
        return []
    
    # Find all HTML files in this session directory
    pattern = os.path.join(session_dir, "*.html")
    trace_files = glob.glob(pattern)
    
    return sorted(trace_files, key=os.path.getmtime)


path: scripts/response_interpreter.py
content: #!/usr/bin/env python3
"""
Response Interpreter - Structures responses for AI agent analysis.

This module does NOT interpret responses itself. Instead, it:
1. Captures raw Nova Act responses
2. Structures them for the orchestrating AI agent (OpenClaw/Claude)
3. The AGENT does all the intelligent interpretation

The orchestrating agent should:
- Analyze if the goal was achieved based on the response
- Decide if retries are needed
- Generate alternative approaches
"""

from typing import Dict, Optional, List
from dataclasses import dataclass, asdict
import json


@dataclass
class StepResult:
    """Structured result for AI agent to analyze."""
    step_name: str
    prompt: str
    expected_outcome: str
    raw_response: str
    api_success: bool  # Did the API call work?
    error: Optional[str] = None
    attempts: int = 1
    
    def to_dict(self) -> Dict:
        return asdict(self)
    
    def to_json(self) -> str:
        return json.dumps(self.to_dict(), indent=2)


def format_for_agent_analysis(results: List[Dict]) -> str:
    """
    Format test results for AI agent to analyze.
    
    The agent should look at each result and determine:
    1. Was the goal achieved? (based on raw_response content)
    2. Should we retry with a different approach?
    3. What's the overall test outcome?
    """
    output = []
    output.append("=" * 60)
    output.append("RESULTS FOR AI AGENT ANALYSIS")
    output.append("=" * 60)
    output.append("")
    output.append("For each step, analyze the raw_response to determine:")
    output.append("- goal_achieved: Did we find/accomplish what we were looking for?")
    output.append("- A 'No' or 'not found' response means goal NOT achieved")
    output.append("- Actual content/information means goal WAS achieved")
    output.append("")
    
    for i, result in enumerate(results, 1):
        output.append(f"--- Step {i}: {result.get('step_name', 'Unknown')} ---")
        output.append(f"Prompt: {result.get('prompt', 'N/A')}")
        output.append(f"Expected: {result.get('expected_outcome', 'N/A')}")
        output.append(f"API Success: {result.get('api_success', False)}")
        output.append(f"Raw Response: {result.get('raw_response', 'No response')}")
        if result.get('error'):
            output.append(f"Error: {result.get('error')}")
        output.append("")
    
    output.append("=" * 60)
    output.append("AGENT: Please analyze each step and provide:")
    output.append("1. goal_achieved (true/false) for each step")
    output.append("2. Overall test success/failure")
    output.append("3. Key findings and recommendations")
    output.append("=" * 60)
    
    return "\n".join(output)


def create_agent_prompt_for_interpretation(step_result: Dict) -> str:
    """
    Create a prompt that the orchestrating agent should use
    to interpret a single step result.
    """
    return f"""Analyze this usability test step result:

**Step:** {step_result.get('step_name', 'Unknown')}
**Question Asked:** {step_result.get('prompt', 'N/A')}
**Expected Outcome:** {step_result.get('expected_outcome', 'N/A')}
**Nova Act Response:** {step_result.get('raw_response', 'No response')}

Based on the response, determine:
1. **Goal Achieved?** Did we find/accomplish what we were looking for?
   - "No", "not found", "I don't see" = Goal NOT achieved
   - Actual content, "Yes", information found = Goal achieved
   
2. **Should Retry?** If goal not achieved, should we try a different approach?

3. **Next Action:** What should we do next?"""


def create_agent_prompt_for_alternative(
    original_prompt: str,
    failed_response: str,
    attempt_number: int
) -> str:
    """
    Create a prompt for the agent to generate an alternative approach.
    """
    return f"""The previous usability test step did not achieve its goal.

**Original Question:** {original_prompt}
**Response (goal not achieved):** {failed_response}
**Attempt:** {attempt_number} of 3

Please suggest an alternative approach to accomplish the same goal.
Consider:
- Looking in a different location (navigation vs content, header vs footer)
- Scrolling to see more of the page
- Rephrasing the question
- Broadening the search

Provide just the new question/prompt to try."""


# For backwards compatibility - but now just returns raw data
def interpret_response(prompt: str, response: str, expected_outcome: str = "") -> Dict:
    """
    Returns structured data for agent analysis.
    Does NOT interpret - that's the agent's job.
    """
    return {
        'prompt': prompt,
        'raw_response': str(response) if response else "",
        'expected_outcome': expected_outcome,
        'api_success': response is not None and response != "",
        'needs_agent_analysis': True
    }


def generate_alternative_approach(
    original_prompt: str,
    failed_response: str, 
    attempt_number: int
) -> Optional[str]:
    """
    Generate alternative browser commands when first attempt fails.
    
    NOTE: Keep prompts SIMPLE and DIRECT - no persona reasoning.
    Nova Act just executes browser actions; the agent interprets results.
    """
    if attempt_number > 3:
        return None
    
    prompt_lower = original_prompt.lower()
    
    # Extract the core task from the prompt (remove any persona language)
    # Common patterns to strip out
    core_prompt = original_prompt
    persona_prefixes = [
        r"as a \w+ with \w+ technical skills,?\s*",
        r"as a \w+,?\s*",
        r"for a \w+ user,?\s*",
        r"can you easily\s*",
    ]
    import re
    for pattern in persona_prefixes:
        core_prompt = re.sub(pattern, "", core_prompt, flags=re.IGNORECASE)
    core_prompt = core_prompt.strip()
    
    if attempt_number == 1:
        # Try looking in navigation
        if "navigation" in prompt_lower or "menu" in prompt_lower:
            return f"Look in the main content area instead: {core_prompt}"
        else:
            return f"Check the navigation menu for: {core_prompt}"
    
    if attempt_number == 2:
        # Try scrolling
        return f"Scroll down the page and look for: {core_prompt}"
    
    if attempt_number == 3:
        # Broaden search
        return f"List any elements on this page related to: {core_prompt}"
    
    return None


path: scripts/generate_report.py
content: #!/usr/bin/env python3
"""
Generate HTML usability report from test results.
"""

import json
import sys
from pathlib import Path
from typing import List, Dict
from datetime import datetime


def analyze_results(results: List[Dict]) -> Dict:
    """
    Analyze test results and extract insights.
    
    Args:
        results: List of test result dicts
    
    Returns:
        Analysis dict with summary statistics and insights
    """
    total_tests = len(results)
    successful_tests = sum(1 for r in results if r.get("success"))
    failed_tests = total_tests - successful_tests
    
    # Group by persona
    persona_stats = {}
    for result in results:
        persona = result.get("persona", "Unknown")
        if persona not in persona_stats:
            persona_stats[persona] = {"total": 0, "success": 0, "tasks": []}
        
        persona_stats[persona]["total"] += 1
        if result.get("success"):
            persona_stats[persona]["success"] += 1
        persona_stats[persona]["tasks"].append({
            "task": result.get("task"),
            "success": result.get("success"),
            "duration": result.get("duration_seconds"),
            "observations": result.get("observations", [])
        })
    
    # Find common friction points
    friction_points = []
    for result in results:
        for obs in result.get("observations", []):
            if not obs.get("success") or "friction" in obs.get("notes", "").lower():
                friction_points.append({
                    "persona": result.get("persona"),
                    "task": result.get("task"),
                    "issue": obs.get("notes")
                })
    
    return {
        "total_tests": total_tests,
        "successful_tests": successful_tests,
        "failed_tests": failed_tests,
        "success_rate": (successful_tests / total_tests * 100) if total_tests > 0 else 0,
        "persona_stats": persona_stats,
        "friction_points": friction_points
    }


def generate_html_report(results: List[Dict], analysis: Dict, template_path: str, output_path: str):
    """
    Generate HTML report using template.
    
    Args:
        results: Raw test results
        analysis: Analyzed insights
        template_path: Path to HTML template
        output_path: Where to save report
    """
    # Load template
    with open(template_path, 'r') as f:
        template = f.read()
    
    # Generate report sections
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    
    executive_summary = f"""
    <div class="executive-summary">
        <h2>Executive Summary</h2>
        <p><strong>Test Date:</strong> {timestamp}</p>
        <p><strong>Tests Conducted:</strong> {analysis['total_tests']}</p>
        <p><strong>Success Rate:</strong> {analysis['success_rate']:.1f}%</p>
        <p><strong>Personas Tested:</strong> {len(analysis['persona_stats'])}</p>
    </div>
    """
    
    # Per-persona findings
    persona_findings = "<h2>Persona Findings</h2>"
    for persona_name, stats in analysis['persona_stats'].items():
        success_rate = (stats['success'] / stats['total'] * 100) if stats['total'] > 0 else 0
        persona_findings += f"""
        <div class="persona-section">
            <h3>{persona_name}</h3>
            <p><strong>Tasks Completed:</strong> {stats['success']}/{stats['total']} ({success_rate:.1f}%)</p>
            <ul>
        """
        for task in stats['tasks']:
            status = "✓" if task['success'] else "✗"
            persona_findings += f"<li>{status} {task['task']} ({task['duration']:.1f}s)</li>"
        persona_findings += "</ul></div>"
    
    # Friction points
    friction_section = "<h2>Common Friction Points</h2><ul>"
    for fp in analysis['friction_points'][:10]:  # Top 10
        friction_section += f"<li><strong>{fp['persona']}</strong> - {fp['task']}: {fp['issue']}</li>"
    friction_section += "</ul>"
    
    # Recommendations
    recommendations = """
    <h2>Recommendations</h2>
    <ul>
        <li>Address high-friction areas identified in testing</li>
        <li>Improve accessibility for low tech proficiency users</li>
        <li>Optimize task completion paths</li>
        <li>Consider persona-specific UI adaptations</li>
    </ul>
    """
    
    # Replace template placeholders
    html = template.replace("{{EXECUTIVE_SUMMARY}}", executive_summary)
    html = html.replace("{{PERSONA_FINDINGS}}", persona_findings)
    html = html.replace("{{FRICTION_POINTS}}", friction_section)
    html = html.replace("{{RECOMMENDATIONS}}", recommendations)
    
    # Write output
    with open(output_path, 'w') as f:
        f.write(html)
    
    print(f"Report generated: {output_path}")


def main():
    if len(sys.argv) < 2:
        print("Usage: generate_report.py <results_json_file> [output_html]")
        sys.exit(1)
    
    results_file = sys.argv[1]
    output_file = sys.argv[2] if len(sys.argv) > 2 else "usability_report.html"
    
    # Load results
    with open(results_file, 'r') as f:
        results = json.load(f)
    
    # Analyze
    analysis = analyze_results(results)
    
    # Find template
    script_dir = Path(__file__).parent
    template_path = script_dir.parent / "assets" / "report-template.html"
    
    # Generate report
    generate_html_report(results, analysis, str(template_path), output_file)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)