chentang1999/PPACER

PACER AgentRoute

Reproducible experimental platform for single-agent, step-level model routing. The router selects one LLM per agent step using transition progress, reliability-calibrated capability evidence, and within-task posterior updates.

The source research documents are:

• PACER_ Progress-Aware Capability Estimation for Step-Level Agentic Model Routing方案设计.pdf
• agent-route-experiment-plan-and-codex-prompt.md

Current status

Stages 0-3 are complete within offline scope, and Stage 4 validated the shared-history, task-split, reliability-calibration, compressed posterior, and K=2 continuation-gold pipelines on provider-free synthetic data. Stage 5 has completed offline preparation: the serving extra imports, five exact Qwen3 revisions and weights are verified, three benchmark source commits are frozen, and readiness/tool contracts are tested. Model loading, benchmark adapters, GPU execution, live calls, agent smoke, pilots, paid runs, and K=4/8/16 real gold remain NOT_RUN. See STATUS.md for the authoritative state.

The comprehensive Chinese project and experiment summary is docs/agent_route_experiment_comprehensive_summary_zh.md.

Quick start

source .venv/bin/activate
make bootstrap
make check
agentroute preflight
uv sync --frozen --extra serving --extra benchmark-tau2
uv run --extra serving --extra benchmark-tau2 agentroute stage5-readiness

All live model calls are disabled by default in configs/approval.yaml.

The reviewed Stage 1 artifact is under results/processed/stage1_mock_final/. To create another uniquely named offline run:

agentroute mock-run \
  --run-id stage1-mock-example \
  --output-dir results/processed/stage1_mock_example

To reproduce the complete Stage 2 synthetic smoke under new paths:

agentroute baseline-smoke \
  --raw-root results/raw/stage2-baseline-smoke-example \
  --output-dir results/processed/stage2_baseline_smoke_example

The reviewed outputs are results/raw/stage2-baseline-smoke-final/ and results/processed/stage2_baseline_smoke_final/. They are synthetic interface checks, not real performance results.

The Stage 3 method contract and validation report are docs/stage3_method_contract.md and results/reports/stage3_pacer_report.md.

The Stage 4 data contract and offline report are docs/stage4_data_contract.md and results/reports/stage4_offline_data_report.md.

The Stage 5 preparation, benchmark adapter audit, and machine-readable gate report are docs/stage5_preparation_report.md, docs/stage5_benchmark_adapter_audit.md, and results/reports/stage5_readiness.json. They are readiness evidence, not real experiment results.