Skill: tltby12341/progressive-validator (version 1.0.1)

--- name: progressive-validator description: Multi-stage backtest validation framework — fail fast with short windows (smoke/stress/medium/full) before committing to expensive full-period backtests, saving hours of compute time. version: 1.0.0 metadata: openclaw: requires: bins: - python3 emoji: "\U0001F3AF" --- # Progressive Validator Stop wasting 3 hours on a backtest that was doomed from the start. This skill implements a multi-stage validation pipeline that eliminates bad strategies in 15 minutes instead of 3 hours. ## When to use - "Validate this strategy" - "Run the progressive test pipeline" - "Is this strategy worth a full backtest?" - When planning the validation sequence for a new strategy variant ## The Pipeline ``` 15 min 30 min 1 hour 3 hours +-----------+ +-----------+ +-----------+ +-----------+ | SMOKE | pass | STRESS | pass | MEDIUM | pass | FULL | | 3 months |------>| 5 months |------>| 18 months |------>| 3 years | | DD < 50% | | DD < 45% | | DD < 42% | | DD < 40% | +-----------+ +-----------+ +-----------+ +-----------+ | fail | fail | fail | fail v v v v REJECT REJECT REJECT REJECT (15 min lost) (45 min lost) (1.5h lost) (3h+ lost) ``` **Without progressive validation**: Every failed strategy costs 3 hours. **With progressive validation**: Most failures caught in 15-45 minutes. ## Validation Stages ### Stage 1: Smoke Test - **Period**: 2024-01-01 to 2024-03-31 (3 months) - **Time**: ~15-20 minutes - **Threshold**: Drawdown < 50% - **Purpose**: Catch compilation errors, logic bugs, and catastrophic structural flaws - **What it covers**: Q1 2024 (includes major tech rallies) ### Stage 2: Stress Test - **Period**: 2024-02-01 to 2024-06-30 (5 months) - **Time**: ~25-30 minutes - **Threshold**: Drawdown < 45% - **Purpose**: Test survival during the hardest market conditions - **What it covers**: 2024 H1 — historically the worst "meat grinder" period for options strategies ### Stage 3: Medium - **Period**: 2024-01-01 to 2025-06-30 (18 months) - **Time**: ~45-60 minutes - **Threshold**: Drawdown < 42% - **Purpose**: Validate across bull/bear transitions and seasonal effects - **What it covers**: Full 2024 volatility + 2025 early recovery ### Stage 4: Full Period - **Period**: 2023-01-01 to 2026-01-31 (3 years) - **Time**: ~2-3 hours - **Threshold**: Drawdown < 40%, Sharpe >= 2.0, Profit >= 300% - **Purpose**: Final acceptance test — benchmark against proven strategies - **What it covers**: Complete market cycle including 2023 AI rally, 2024 correction, 2025 recovery ## Usage ### Configure windows Define your validation windows in config: ```python BACKTEST_WINDOWS = { "smoke_test": { "start": "2024-01-01", "end": "2024-03-31", "max_dd": 0.50, "expected_time": "15-20 min", "purpose": "Eliminate garbage fast", }, "stress_test": { "start": "2024-02-01", "end": "2024-06-30", "max_dd": 0.45, "expected_time": "25-30 min", "purpose": "Survive worst conditions", }, "medium": { "start": "2024-01-01", "end": "2025-06-30", "max_dd": 0.42, "expected_time": "45-60 min", "purpose": "Bull/bear transition stability", }, "full": { "start": "2023-01-01", "end": "2026-01-31", "max_dd": 0.40, "expected_time": "2-3 hours", "purpose": "Final benchmark acceptance", }, } ``` ### Run each stage > **Prerequisite**: This skill coordinates validation stages. Actual backtest submission > is handled by the **backtest-poller** skill (`cli.py`). Ensure that skill is installed > and available on your path before running these commands. ```bash # Stage 1: Smoke # (using backtest-poller skill's cli.py) python3 ../backtest-poller/cli.py submit \ --main-file strategy.py --name "M31_smoke" # Check what to run next: python3 validator.py next M31 strategy.py # Record the result after smoke completes: python3 validator.py record M31 smoke_test --status passed --drawdown 0.32 --sharpe 2.1 # Stage 2: Stress (only if smoke passed) python3 ../backtest-poller/cli.py submit \ --main-file strategy.py --name "M31_stress" python3 validator.py record M31 stress_test --status passed --drawdown 0.38 --sharpe 2.0 # Stage 3: Medium (only if stress passed) python3 ../backtest-poller/cli.py submit \ --main-file strategy.py --name "M31_medium" python3 validator.py record M31 medium --status passed --drawdown 0.35 --sharpe 2.3 # Stage 4: Full (only if medium passed) python3 ../backtest-poller/cli.py submit \ --main-file strategy.py --name "M31_full" python3 validator.py record M31 full --status passed --drawdown 0.30 --sharpe 2.5 --profit 3.2 ``` ## Skip Rules Not every change needs to start from Smoke: | Change Type | Start From | |-------------|------------| | Entry logic changed | Smoke (Stage 1) | | Structural change (position sizing, survival) | Smoke (Stage 1) | | Profit management only | Medium (Stage 3) | | Date/parameter tweak | Same stage as before | ## Early-Stop Integration This skill works alongside the **backtest-poller** skill (a separate package). The backtest-poller's early-stop feature monitors drawdown in real time and deletes the backtest run if the threshold is exceeded after 20% progress — no need to wait for full completion of a doomed run. This validator tracks which stages passed or failed locally, so you always know where to resume. **Dependency**: Install the `backtest-poller` skill to enable submit/early-stop functionality. This validator does not submit backtests itself. ## Time Savings Example Testing 5 strategy variants, 3 of which are bad: | Approach | Time | |----------|------| | Full backtest only | 5 x 3h = **15 hours** | | Progressive validation | 3 x 15min + 1 x 45min + 1 x 3h = **~4.5 hours** | **Savings: ~70%** of compute time. ## Rules - **Never skip stages without justification.** The skip rules table above defines the only valid exceptions. If entry logic or survival structure changed, you must start from Smoke. - **A strategy must pass a stage before advancing.** Do not promote a strategy to the next stage if the current stage resulted in early-stop or failure. - **Do not modify stage thresholds mid-validation.** Changing `max_dd` between stages invalidates the progressive guarantee. Decide thresholds before starting. - **One strategy variant per validation run.** Do not change the strategy code between stages — the point is to validate the same code across increasingly demanding windows. - **Record every result**, even failures. Use `python3 validator.py record <strategy> <stage> --status passed|failed` to persist outcomes. Unrecorded results break the `next` and `status` commands.

path: config.example.json
content: {
    "stages": {
        "smoke_test": {
            "start": "2024-01-01",
            "end": "2024-03-31",
            "max_dd": 0.50,
            "expected_time": "15-20 min",
            "purpose": "Fast elimination",
            "order": 1
        },
        "stress_test": {
            "start": "2024-02-01",
            "end": "2024-06-30",
            "max_dd": 0.45,
            "expected_time": "25-30 min",
            "purpose": "Worst-case survival",
            "order": 2
        },
        "medium": {
            "start": "2024-01-01",
            "end": "2025-06-30",
            "max_dd": 0.42,
            "expected_time": "45-60 min",
            "purpose": "Seasonal validation",
            "order": 3
        },
        "full": {
            "start": "2023-01-01",
            "end": "2026-01-31",
            "max_dd": 0.40,
            "expected_time": "2-3 hours",
            "purpose": "Final benchmark",
            "order": 4
        }
    },
    "acceptance_criteria": {
        "min_sharpe": 2.0,
        "max_drawdown": 0.40,
        "min_profit": 3.0
    }
}


path: requirements.txt
content: # No external dependencies required


path: validator.py
content: """
Progressive Validator — Multi-Stage Backtest Validation Framework
=================================================================
Fail fast with short backtest windows before committing to expensive
full-period runs. Each stage has increasing duration and stricter thresholds.

Usage:
    validator = ProgressiveValidator(config_file="config.json")
    validator.run_stage("smoke_test", "path/to/strategy.py", "MyStrategy_v1")
    # If passed, run next stage:
    validator.run_stage("stress_test", "path/to/strategy.py", "MyStrategy_v1")
"""
import argparse
import json
import os
import sys
import time
from datetime import datetime
from typing import Dict, List, Optional


DEFAULT_STAGES = {
    "smoke_test": {
        "start": "2024-01-01",
        "end": "2024-03-31",
        "max_dd": 0.50,
        "expected_time": "15-20 min",
        "purpose": "Eliminate obvious bugs and catastrophic flaws",
        "order": 1,
    },
    "stress_test": {
        "start": "2024-02-01",
        "end": "2024-06-30",
        "max_dd": 0.45,
        "expected_time": "25-30 min",
        "purpose": "Test survival in worst market conditions",
        "order": 2,
    },
    "medium": {
        "start": "2024-01-01",
        "end": "2025-06-30",
        "max_dd": 0.42,
        "expected_time": "45-60 min",
        "purpose": "Validate across bull/bear transitions",
        "order": 3,
    },
    "full": {
        "start": "2023-01-01",
        "end": "2026-01-31",
        "max_dd": 0.40,
        "expected_time": "2-3 hours",
        "purpose": "Final acceptance benchmark",
        "order": 4,
    },
}

SKIP_RULES = """
=== Progressive Validation Skip Rules ===

1. NEVER skip the smoke_test stage.
   - It catches fatal bugs in minutes, not hours.

2. You MAY skip stress_test only if:
   - The strategy has already been validated on the same date range
     in a prior iteration with zero code changes.

3. You MAY skip medium only if:
   - Both smoke_test AND stress_test passed with Sharpe >= 2.5
     and drawdown < 30%.

4. The full stage can NEVER be skipped.
   - It is the final acceptance gate.

5. If any code change is made (even a single parameter tweak),
   restart from smoke_test.

6. Results older than 30 days should be considered stale.
   Re-run from the earliest stale stage.
"""


class ProgressiveValidator:
    """Multi-stage backtest validation pipeline."""

    def __init__(
        self,
        config: Optional[dict] = None,
        config_file: Optional[str] = None,
        results_file: str = "validation_results.json",
    ):
        if config_file and os.path.exists(config_file):
            with open(config_file) as f:
                loaded = json.load(f)
            self.stages = loaded.get("stages", DEFAULT_STAGES)
            self.acceptance_criteria = loaded.get(
                "acceptance_criteria",
                {"min_sharpe": 2.0, "max_drawdown": 0.40, "min_profit": 3.0},
            )
        elif config:
            self.stages = config.get("stages", DEFAULT_STAGES)
            self.acceptance_criteria = config.get("acceptance_criteria", {})
        else:
            self.stages = dict(DEFAULT_STAGES)
            self.acceptance_criteria = {
                "min_sharpe": 2.0,
                "max_drawdown": 0.40,
                "min_profit": 3.0,
            }

        self.results_file = results_file
        self.results: Dict = self._load_results()

    # ------------------------------------------------------------------
    # Persistence
    # ------------------------------------------------------------------

    def _load_results(self) -> dict:
        """Load previous validation results from disk."""
        if os.path.exists(self.results_file):
            with open(self.results_file) as f:
                return json.load(f)
        return {}

    def _save_results(self):
        """Persist validation results to disk."""
        with open(self.results_file, "w") as f:
            json.dump(self.results, f, indent=2, default=str)

    # ------------------------------------------------------------------
    # Stage ordering helpers
    # ------------------------------------------------------------------

    def get_stage_order(self) -> List[str]:
        """Return stage names sorted by their configured order."""
        return sorted(
            self.stages.keys(), key=lambda s: self.stages[s].get("order", 99)
        )

    def get_next_stage(self, strategy_name: str) -> Optional[str]:
        """Determine the next stage a strategy should run."""
        history = self.results.get(strategy_name, {})
        for stage in self.get_stage_order():
            if stage not in history or history[stage].get("status") != "passed":
                return stage
        return None  # All stages passed

    # ------------------------------------------------------------------
    # Display helpers
    # ------------------------------------------------------------------

    def show_pipeline(self):
        """Print the full validation pipeline to stdout."""
        ordered = self.get_stage_order()
        print("\n=== Progressive Validation Pipeline ===\n")
        for i, stage_name in enumerate(ordered, 1):
            stage = self.stages[stage_name]
            print(f"  Stage {i}: {stage_name}")
            print(f"    Period:  {stage['start']} ~ {stage['end']}")
            print(f"    Time:    ~{stage['expected_time']}")
            print(f"    Max DD:  {stage['max_dd'] * 100:.0f}%")
            print(f"    Purpose: {stage['purpose']}")
            if i < len(ordered):
                print("         |")
                print("         v  (pass to advance)")

        ac = self.acceptance_criteria
        print(
            f"\n  Final acceptance: "
            f"Sharpe >= {ac.get('min_sharpe', 'N/A')}, "
            f"DD <= {ac.get('max_drawdown', 'N/A')}, "
            f"Profit >= {ac.get('min_profit', 'N/A')}"
        )
        print()

    def show_status(self, strategy_name: str):
        """Print the current validation status for a given strategy."""
        history = self.results.get(strategy_name, {})
        ordered = self.get_stage_order()

        print(f"\n=== Validation Status: {strategy_name} ===\n")
        for stage_name in ordered:
            result = history.get(stage_name, {})
            status = result.get("status", "pending")
            icon = {
                "passed": "[PASS]",
                "failed": "[FAIL]",
                "pending": "[----]",
            }.get(status, "[????]")

            detail = ""
            if status == "passed":
                dd = result.get("drawdown", "?")
                sh = result.get("sharpe", "?")
                detail = f" (DD: {dd}, Sharpe: {sh})"
            elif status == "failed":
                detail = f" (Reason: {result.get('reason', '?')})"

            print(f"  {icon} {stage_name}{detail}")

        next_stage = self.get_next_stage(strategy_name)
        if next_stage:
            print(f"\n  Next: {next_stage}")
        else:
            print("\n  All stages PASSED!")
        print()

    # ------------------------------------------------------------------
    # Recording
    # ------------------------------------------------------------------

    def record_result(
        self,
        strategy_name: str,
        stage_name: str,
        status: str,
        sharpe: Optional[float] = None,
        drawdown: Optional[float] = None,
        profit: Optional[float] = None,
        reason: Optional[str] = None,
    ):
        """Record the outcome of a validation stage."""
        if strategy_name not in self.results:
            self.results[strategy_name] = {}

        self.results[strategy_name][stage_name] = {
            "status": status,
            "sharpe": sharpe,
            "drawdown": drawdown,
            "profit": profit,
            "reason": reason,
            "timestamp": datetime.now().isoformat(),
        }
        self._save_results()
        print(f"Recorded: {strategy_name} / {stage_name} -> {status}")

    # ------------------------------------------------------------------
    # Command suggestion
    # ------------------------------------------------------------------

    def suggest_command(
        self,
        strategy_name: str,
        strategy_file: str,
        cli_path: str = "../backtest-poller/cli.py",
    ) -> Optional[str]:
        """Generate the backtest submission command for the next validation stage.

        Submission is handled by the backtest-poller skill. ``cli_path`` should
        point to that skill's cli.py (default assumes skills are sibling directories).
        """
        next_stage = self.get_next_stage(strategy_name)
        if not next_stage:
            return None

        stage = self.stages[next_stage]
        return (
            f"python3 {cli_path} submit "
            f"--main-file {strategy_file} "
            f"--name {strategy_name}_{next_stage}"
        )


# ======================================================================
# CLI
# ======================================================================


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="validator",
        description="Progressive Validator — multi-stage backtest validation",
    )
    parser.add_argument(
        "--config",
        default=None,
        help="Path to JSON config file (default: use built-in stages)",
    )
    parser.add_argument(
        "--results",
        default="validation_results.json",
        help="Path to results JSON file (default: validation_results.json)",
    )

    sub = parser.add_subparsers(dest="command", help="Available commands")

    # pipeline
    sub.add_parser("pipeline", help="Display the full validation pipeline")

    # status
    p_status = sub.add_parser("status", help="Show validation status for a strategy")
    p_status.add_argument("strategy_name", help="Name of the strategy")

    # next
    p_next = sub.add_parser("next", help="Show the next command to run")
    p_next.add_argument("strategy_name", help="Name of the strategy")
    p_next.add_argument("strategy_file", help="Path to the strategy source file")

    # record
    p_record = sub.add_parser("record", help="Record a stage result")
    p_record.add_argument("strategy_name", help="Name of the strategy")
    p_record.add_argument("stage", help="Stage name (e.g. smoke_test)")
    p_record.add_argument(
        "--status",
        required=True,
        choices=["passed", "failed"],
        help="Result status",
    )
    p_record.add_argument("--sharpe", type=float, default=None, help="Sharpe ratio")
    p_record.add_argument("--drawdown", type=float, default=None, help="Max drawdown (0-1)")
    p_record.add_argument("--profit", type=float, default=None, help="Profit factor")
    p_record.add_argument("--reason", default=None, help="Failure reason (if failed)")

    # skip-rules
    sub.add_parser("skip-rules", help="Display the skip rules")

    return parser


def main():
    parser = build_parser()
    args = parser.parse_args()

    if not args.command:
        parser.print_help()
        sys.exit(1)

    validator = ProgressiveValidator(
        config_file=args.config, results_file=args.results
    )

    if args.command == "pipeline":
        validator.show_pipeline()

    elif args.command == "status":
        validator.show_status(args.strategy_name)

    elif args.command == "next":
        cmd = validator.suggest_command(args.strategy_name, args.strategy_file)
        if cmd:
            next_stage = validator.get_next_stage(args.strategy_name)
            stage = validator.stages[next_stage]
            print(f"\nNext stage: {next_stage}")
            print(f"  Period:  {stage['start']} ~ {stage['end']}")
            print(f"  Time:    ~{stage['expected_time']}")
            print(f"  Purpose: {stage['purpose']}")
            print(f"\nSuggested command:\n  {cmd}\n")
        else:
            print(f"\n{args.strategy_name}: all stages already passed.\n")

    elif args.command == "record":
        validator.record_result(
            strategy_name=args.strategy_name,
            stage_name=args.stage,
            status=args.status,
            sharpe=args.sharpe,
            drawdown=args.drawdown,
            profit=args.profit,
            reason=args.reason,
        )

    elif args.command == "skip-rules":
        print(SKIP_RULES)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)