Skill: ndtchan/strategy-backtester (version 1.0.0)

--- name: strategy-backtester description: Validates historical behavior of stock ranking, factor, and portfolio-selection strategies using reproducible backtests, benchmark comparison, turnover, drawdown, and bias warnings. compatibility: Requires historical signal/ranking CSVs and price history CSVs; uses local scripts and does not require network access. --- # Strategy Backtester ## Purpose Use this skill to test whether a ranking, factor mix, or portfolio-selection rule had useful historical behavior before treating it as an investment signal. ## Scope - Equity ranking and selection strategies. - Periodic rebalance backtests from local CSV inputs. - Benchmark comparison when benchmark data is available. - Bias and robustness review. ## Non-goals - Do not claim that historical performance predicts future returns. - Do not optimize parameters until a preferred result appears. - Do not issue absolute buy/sell instructions. - Do not fetch live market data. ## Input contract Required inputs: - `SIGNAL_CSV`: rows with `date`, `ticker`, and `score`. - `PRICE_CSV`: rows with `date`, `ticker`, and `close`. - `REBALANCE_FREQUENCY`: `monthly`, `quarterly`, or `yearly`. - `TOP_N`: number of selected names per rebalance. Optional inputs: - `BENCHMARK_CSV`: rows with `date` and `close` or `return`. - `FEE_BPS`: round-trip fee assumption in basis points. - `SLIPPAGE_BPS`: slippage assumption in basis points. - `UNIVERSE_HISTORY`: point-in-time membership if available. ## Execution workflow 1. Validate input files and required columns. 2. Estimate whether the test window and symbol coverage are sufficient. 3. Run `scripts/backtest_strategy.py` with explicit rebalance, fee, slippage, and top-N assumptions. 4. Review performance metrics and benchmark comparison. 5. Identify bias risks and robustness gaps. 6. Return the required output sections. ## Required output format 1. `Backtest Setup` - Strategy name, test window, rebalance frequency, top-N, fees, slippage, benchmark. 2. `Performance Summary` - Total return, CAGR, volatility, max drawdown, Sharpe, Sortino, turnover, hit rate when available. 3. `Benchmark Comparison` - Relative return, relative drawdown, and tracking observations when benchmark data exists. 4. `Robustness and Bias Warnings` - Survivorship bias, lookahead bias, data-snooping risk, liquidity assumptions, fee/slippage sensitivity. 5. `Confidence and Data Gaps` - Confidence level and missing inputs that could change the conclusion. 6. `Handoff Bundle` - Include `strategy_name`, `test_window`, `rebalance_frequency`, `fee_assumption`, `slippage_assumption`, `benchmark`, `metrics`, `bias_warnings`, `confidence`, and `data_gaps`. ## Shared confidence rubric - `High`: point-in-time signals, adequate price coverage, benchmark available, fees/slippage included, and test window covers multiple market regimes. - `Medium`: usable history and price coverage, but one major robustness input is missing. - `Low`: short history, missing benchmark, sparse price coverage, likely survivorship/lookahead risk, or no fee/slippage assumptions. ## Guardrails - Separate observed backtest results from assumptions and inference. - Always state that backtests are historical simulations, not forecasts. - Downgrade confidence if the test appears overfit or data is not point-in-time. - Treat backtest output as one input to `stock-picker-orchestrator`, not as a trading command. ## Trigger examples - "Backtest this VN30 value-quality ranking." - "Check whether this stock ranking strategy beat VNINDEX historically." - "Validate this screening rule before using it for shortlist selection."

path: agents/openai.yaml
content: interface:
  display_name: "Strategy Backtester"
  short_description: "Backtest ranking strategies with bias warnings."
  default_prompt: |
    Use $strategy-backtester to validate the supplied ranking or signal strategy.
    Require explicit signal, price, rebalance, fee, and slippage assumptions.
    Report performance, benchmark comparison, confidence, and bias warnings.

policy:
  allow_implicit_invocation: true

dependencies:
  tools: []


path: scripts/backtest_strategy.py
content: from __future__ import annotations

import argparse
import csv
import json
import sys
from pathlib import Path
from statistics import mean

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from performance_metrics import (  # noqa: E402
    annualized_return,
    annualized_volatility,
    average_turnover,
    max_drawdown,
    sharpe_ratio,
    sortino_ratio,
    total_return,
)


def read_signals(path: Path) -> dict[str, list[tuple[str, float]]]:
    signals: dict[str, list[tuple[str, float]]] = {}
    with path.open(newline="", encoding="utf-8") as handle:
        reader = csv.DictReader(handle)
        require_columns(reader.fieldnames, {"date", "ticker", "score"}, path)
        for row in reader:
            date = required_value(row, "date", path)
            ticker = required_value(row, "ticker", path)
            score = float(required_value(row, "score", path))
            signals.setdefault(date, []).append((ticker, score))
    return signals


def read_prices(path: Path) -> tuple[dict[str, dict[str, float]], list[str]]:
    prices: dict[str, dict[str, float]] = {}
    dates: set[str] = set()
    with path.open(newline="", encoding="utf-8") as handle:
        reader = csv.DictReader(handle)
        require_columns(reader.fieldnames, {"date", "ticker", "close"}, path)
        for row in reader:
            date = required_value(row, "date", path)
            ticker = required_value(row, "ticker", path)
            close = float(required_value(row, "close", path))
            prices.setdefault(ticker, {})[date] = close
            dates.add(date)
    return prices, sorted(dates)


def require_columns(fieldnames: list[str] | None, required: set[str], path: Path) -> None:
    columns = set(fieldnames or [])
    missing = sorted(required - columns)
    if missing:
        raise ValueError(f"{path} is missing required columns: {', '.join(missing)}")


def required_value(row: dict[str, str], field: str, path: Path) -> str:
    value = row.get(field, "").strip()
    if value == "":
        raise ValueError(f"{path} has an empty {field} value")
    return value


def entry_price_date(signal_date: str, price_dates: list[str]) -> str | None:
    for price_date in price_dates:
        if price_date >= signal_date:
            return price_date
    return None


def next_price_date(entry_date: str, price_dates: list[str]) -> str | None:
    for price_date in price_dates:
        if price_date > entry_date:
            return price_date
    return None


def select_top(scored_tickers: list[tuple[str, float]], top_n: int) -> list[str]:
    ranked = sorted(scored_tickers, key=lambda item: (-item[1], item[0]))
    return [ticker for ticker, _score in ranked[:top_n]]


def equal_weight_return(
    selected: list[str],
    start_date: str,
    end_date: str,
    prices: dict[str, dict[str, float]],
) -> float:
    returns = []
    for ticker in selected:
        ticker_prices = prices.get(ticker, {})
        if start_date not in ticker_prices or end_date not in ticker_prices:
            raise ValueError(
                f"missing price for ticker {ticker} between {start_date} and {end_date}"
            )
        start_close = ticker_prices[start_date]
        end_close = ticker_prices[end_date]
        if start_close <= 0:
            raise ValueError(f"non-positive entry price for ticker {ticker} on {start_date}")
        returns.append(end_close / start_close - 1.0)
    return mean(returns)


def turnover(previous: set[str] | None, current: set[str]) -> float:
    if not current:
        return 0.0
    if previous is None:
        return 1.0
    return len(current - previous) / len(current)


def run_backtest(
    signals: dict[str, list[tuple[str, float]]],
    prices: dict[str, dict[str, float]],
    price_dates: list[str],
    top_n: int,
    fee_bps: float,
    slippage_bps: float,
    periods_per_year: int,
) -> tuple[dict[str, object], list[dict[str, object]]]:
    if top_n <= 0:
        raise ValueError("--top-n must be greater than zero")
    if periods_per_year <= 0:
        raise ValueError("--periods-per-year must be greater than zero")
    if fee_bps < 0:
        raise ValueError("--fee-bps must be non-negative")
    if slippage_bps < 0:
        raise ValueError("--slippage-bps must be non-negative")

    cost_rate = (fee_bps + slippage_bps) / 10000.0
    periods: list[dict[str, object]] = []
    selected_history: list[set[str]] = []
    previous_selection: set[str] | None = None

    for signal_date in sorted(signals):
        candidates = signals[signal_date]
        if top_n > len(candidates):
            raise ValueError(
                f"--top-n {top_n} exceeds {len(candidates)} signal candidates on {signal_date}"
            )
        entry_date = entry_price_date(signal_date, price_dates)
        if entry_date is None:
            raise ValueError(f"no entry price date found on or after signal date {signal_date}")
        exit_date = next_price_date(entry_date, price_dates)
        if exit_date is None:
            raise ValueError(f"no exit price date found after entry date {entry_date}")

        selected = select_top(candidates, top_n)
        current_selection = set(selected)
        gross_return = equal_weight_return(selected, entry_date, exit_date, prices)
        period_turnover = turnover(previous_selection, current_selection)
        cost = period_turnover * cost_rate
        net_return = gross_return - cost
        if net_return <= -1.0:
            raise ValueError(
                f"cost assumptions make net return <= -100% for signal date {signal_date}"
            )

        periods.append(
            {
                "signal_date": signal_date,
                "entry_date": entry_date,
                "exit_date": exit_date,
                "selected_tickers": selected,
                "gross_return": gross_return,
                "turnover": period_turnover,
                "cost": cost,
                "net_return": net_return,
            }
        )
        selected_history.append(current_selection)
        previous_selection = current_selection

    returns = [float(period["net_return"]) for period in periods]
    summary: dict[str, object] = {
        "number_of_periods": len(periods),
        "top_n": top_n,
        "fee_bps": fee_bps,
        "slippage_bps": slippage_bps,
        "periods_per_year": periods_per_year,
        "total_return": total_return(returns),
        "annualized_return": annualized_return(returns, periods_per_year),
        "annualized_volatility": annualized_volatility(returns, periods_per_year),
        "max_drawdown": max_drawdown(returns),
        "sharpe_ratio": sharpe_ratio(returns, periods_per_year),
        "sortino_ratio": sortino_ratio(returns, periods_per_year),
        "average_turnover": average_turnover(selected_history),
        "hit_rate": mean([1.0 if item > 0 else 0.0 for item in returns]) if returns else None,
    }
    if periods:
        summary["start_signal_date"] = periods[0]["signal_date"]
        summary["end_signal_date"] = periods[-1]["signal_date"]
    return summary, periods


def write_outputs(outdir: Path, summary: dict[str, object], periods: list[dict[str, object]]) -> None:
    outdir.mkdir(parents=True, exist_ok=True)
    summary_path = outdir / "backtest_summary.json"
    periods_path = outdir / "backtest_periods.csv"

    with summary_path.open("w", encoding="utf-8") as handle:
        json.dump(summary, handle, indent=2, sort_keys=True)
        handle.write("\n")

    with periods_path.open("w", newline="", encoding="utf-8") as handle:
        fieldnames = [
            "signal_date",
            "entry_date",
            "exit_date",
            "selected_tickers",
            "gross_return",
            "turnover",
            "cost",
            "net_return",
        ]
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        writer.writeheader()
        for period in periods:
            row = dict(period)
            row["selected_tickers"] = ";".join(period["selected_tickers"])
            writer.writerow(row)


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Backtest top-N equal-weight ranking strategies from CSV inputs.",
    )
    parser.add_argument("--signals-file", required=True, type=Path)
    parser.add_argument("--prices-file", required=True, type=Path)
    parser.add_argument("--outdir", required=True, type=Path)
    parser.add_argument("--top-n", required=True, type=int)
    parser.add_argument("--fee-bps", default=0.0, type=float)
    parser.add_argument("--slippage-bps", default=0.0, type=float)
    parser.add_argument("--periods-per-year", default=12, type=int)
    return parser


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    return build_parser().parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    parser = build_parser()
    args = parser.parse_args(argv)
    try:
        signals = read_signals(args.signals_file)
        prices, price_dates = read_prices(args.prices_file)
        summary, periods = run_backtest(
            signals=signals,
            prices=prices,
            price_dates=price_dates,
            top_n=args.top_n,
            fee_bps=args.fee_bps,
            slippage_bps=args.slippage_bps,
            periods_per_year=args.periods_per_year,
        )
        write_outputs(args.outdir, summary, periods)
        print(json.dumps(summary, indent=2, sort_keys=True))
    except ValueError as exc:
        parser.error(str(exc))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/performance_metrics.py
content: from __future__ import annotations

import math
from statistics import mean, pstdev


def total_return(returns: list[float]) -> float:
    value = 1.0
    for item in returns:
        value *= 1.0 + item
    return value - 1.0


def annualized_return(returns: list[float], periods_per_year: int) -> float | None:
    if not returns:
        return None
    compounded = 1.0 + total_return(returns)
    years = len(returns) / periods_per_year
    if years <= 0 or compounded <= 0:
        return None
    return compounded ** (1.0 / years) - 1.0


def annualized_volatility(returns: list[float], periods_per_year: int) -> float | None:
    if len(returns) < 2:
        return None
    return pstdev(returns) * math.sqrt(periods_per_year)


def max_drawdown(returns: list[float]) -> float:
    value = 1.0
    peak = 1.0
    worst = 0.0
    for item in returns:
        value *= 1.0 + item
        peak = max(peak, value)
        worst = min(worst, value / peak - 1.0)
    return worst


def sharpe_ratio(
    returns: list[float],
    periods_per_year: int,
    risk_free_rate: float = 0.0,
) -> float | None:
    if len(returns) < 2:
        return None
    period_rf = risk_free_rate / periods_per_year
    excess = [item - period_rf for item in returns]
    volatility = pstdev(excess)
    if volatility == 0:
        return None
    return mean(excess) / volatility * math.sqrt(periods_per_year)


def sortino_ratio(
    returns: list[float],
    periods_per_year: int,
    risk_free_rate: float = 0.0,
) -> float | None:
    if len(returns) < 2:
        return None
    period_rf = risk_free_rate / periods_per_year
    downside = [min(0.0, item - period_rf) for item in returns]
    downside_dev = math.sqrt(mean([item * item for item in downside]))
    if downside_dev == 0:
        return None
    excess_mean = mean([item - period_rf for item in returns])
    return excess_mean / downside_dev * math.sqrt(periods_per_year)


def average_turnover(selected_history: list[set[str]]) -> float | None:
    if len(selected_history) < 2:
        return None
    turnovers = []
    for previous, current in zip(selected_history, selected_history[1:]):
        if not current:
            continue
        changed = len(current - previous)
        turnovers.append(changed / len(current))
    return mean(turnovers) if turnovers else None


path: tests/fixtures/prices.csv
content: date,ticker,close
2024-01-31,AAA,100
2024-02-29,AAA,105
2024-03-31,AAA,103
2024-04-30,AAA,110
2024-01-31,BBB,100
2024-02-29,BBB,98
2024-03-31,BBB,101
2024-04-30,BBB,99
2024-01-31,CCC,100
2024-02-29,CCC,102
2024-03-31,CCC,106
2024-04-30,CCC,108


path: tests/fixtures/signals.csv
content: date,ticker,score
2024-01-31,AAA,0.90
2024-01-31,BBB,0.30
2024-01-31,CCC,0.70
2024-02-29,AAA,0.40
2024-02-29,BBB,0.60
2024-02-29,CCC,0.80
2024-03-31,AAA,0.75
2024-03-31,BBB,0.20
2024-03-31,CCC,0.65


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)