Skill: tankisstank/benchmark-model-provider (version 1.0.5)

--- name: benchmark-model-provider description: Benchmark and rank AI providers/models against a user-specific prompt suite derived from the user's purpose, domain, and usage frequency. Use when users ask which model is smarter, cheaper, deeper, faster, worth using daily, better as local vs service, or when building repeatable benchmark specs, reranking old runs, generating markdown/HTML/PDF benchmark reports. metadata: {"clawdbot":{"emoji":"📊","requires":{"bins":["python3"],"env":["BENCHMARK_API_KEY"]},"primaryEnv":"BENCHMARK_API_KEY"}} --- # Benchmark Model Provider > Use this skill to help users choose the most suitable model for their own workflow instead of giving generic “best model” advice. **Tiếng Việt** Dùng skill này khi Boss muốn biết model nào thật sự đáng dùng cho workflow hằng ngày: model nào research tốt hơn, viết báo cáo ổn hơn, code ngon hơn, rẻ hơn, nhanh hơn, hay đáng dùng lâu dài hơn. Skill này không trả lời kiểu cảm tính, mà dựng benchmark theo đúng nhu cầu thực tế của người dùng rồi chấm, rerank và xuất report rõ ràng. **中文说明** 当用户想知道“哪个模型更聪明、更便宜、更适合日常工作流、更适合研究/写报告/编程”时，使用这个技能。它不会给出泛泛而谈的“最佳模型”建议，而是根据用户自己的实际任务构建基准测试，保留原始结果、重新排序，并生成可审阅、可分享的报告。 Treat the benchmark as a **personal decision framework**: - derive the benchmark from the user's real work - keep the run auditable - preserve raw outputs for reranking - generate outputs that can be reviewed, shared, and published cleanly --- ## What this skill is for People often ask questions like: - Which model is smarter? - Which model is cheaper to run daily? - Which model is deeper or more useful for my job? - Should I use a local model or a service model? This skill exists to answer those questions with a **repeatable benchmark process**, not with vague preferences. --- ## Core operating flow 1. **Collect benchmark context** - purpose - domain - usage frequency 2. **Build or select a benchmark spec** with 5–10 domain-specific questions 3. **List currently available providers/models** from trusted local OpenClaw context when allowed 4. **Ask whether the user wants to use the current list or add more models** 5. **Verify every user-supplied model before running**; if the name does not match, ask again or suggest the closest valid model id 6. **Run each model independently** on the same benchmark set 7. **Preserve raw outputs and metrics** so the run can be audited and reranked later 8. **Score results** across quality, depth, cost, and speed metrics 9. **Build reports** in markdown / HTML / PDF 10. **Optionally** suggest simple ways to publish the generated HTML report (Vercel, Netlify, Cloudflare Pages, GitHub Pages) if the user wants a shareable link --- ## Default decisions | Area | Default | | --- | --- | | Benchmark mode | `prompt_only` | | Overall scoring | quality + depth + cost | | Speed handling | measured and reported, excluded from default overall | | Execution strategy | `sequential` unless orchestration is needed | | Web publish target | (no built-in publish) — suggest Vercel / Netlify / Cloudflare Pages / GitHub Pages | --- ## Workflow rules ### Benchmark input rules - Default to `prompt_only` unless the user explicitly wants `agent_context`. - In `prompt_only`, send only the raw prompt. - Do **not** inject extra context, memory, few-shot examples, or hidden scaffolding in `prompt_only` mode. - In `agent_context`, use one fixed shared system/context layer for all compared models and record it in metadata. ### Execution rules - Support both `sequential` and `subagent_orchestrated` execution strategies. - Allow bounded parallel execution for subagents (for example `--max-parallel 4`) when the endpoint can tolerate it. - Treat `rerank` as a first-class operation; do not rerun models when only the scoring formula changes. - Report progress at every major step so the user never feels the process is hanging. - During batch execution, surface a clear update whenever one agent/model finishes. - Normalize model ids before calling the endpoint when the provider catalog exposes raw model ids but the user/runtime spec may contain provider-prefixed names. - If the endpoint returns naming/provider mismatch errors, explain the mismatch clearly instead of leaving only a raw 502/unknown-provider error. ### Output rules - Mark every estimated metric clearly. - Rewrite reports/landing pages to the newest snapshot. - Do **not** append patch fragments to stale output. - Reports should include: ranking table, cost table, executive summary, overall assessment, recommended model selection, and full answer details. - Default the report language to the user's current conversation language. - Only switch the report language when the user explicitly asks for a different language or a bilingual output. - PDF output must use Unicode-capable fonts so Vietnamese, Chinese, and multilingual content render correctly. - Multilingual support means the renderer can display multiple languages correctly; it does **not** mean the skill should arbitrarily change the report language. - Ask before delivering externally via Vercel or other web publishing. --- ## Safety and trust boundary > This skill may perform network I/O depending on how the benchmark spec is configured. ### Safe-by-design intent - Example specs should use placeholder endpoints, not a private hardcoded runtime. - The user should supply only trusted API endpoints and credentials. - Publishing should happen only when the user explicitly wants delivery. ### Important runtime notes - `run_benchmark.py` sends prompts to the `base_url` configured in the benchmark spec. - This skill does **not** publish to Vercel/Netlify/Cloudflare/GitHub automatically. It only generates local HTML/PDF artifacts. - If you want a shareable link, publish the generated HTML folder using one of these services: Vercel, Netlify, Cloudflare Pages, or GitHub Pages. - Only run the skill with endpoints, tokens, and outputs you trust. For detailed runtime assumptions, read: - `references/runtime-safety.md` - `references/environment-vars.md` - `references/pricing-sources.md` --- ## What to read Read only what you need: - `references/initial-project-spec.md` — authoritative design baseline - `references/benchmark-schema.md` — benchmark spec structure, run artifacts, file layout - `references/scoring-rubric.md` — scoring model, normalization rules, default weights - `references/pricing-sources.md` — pricing precedence and estimation policy - `references/execution-modes.md` — benchmark modes, execution strategies, operational modes - `references/output-modes.md` — delivery choices, publish rules, progress feedback rules - `references/runtime-safety.md` — trust boundaries, network behavior, safe usage guidance - `references/environment-vars.md` — expected environment variables and dependency notes - `examples/*.yaml` — benchmark context templates and ready-made examples in multiple languages --- ## Scripts | Script | Purpose | | --- | --- | | `scripts/build_benchmark_spec.py` | Build a benchmark spec from benchmark context | | `scripts/run_benchmark.py` | Execute benchmark runs and write raw outputs/metrics | | `scripts/estimate_tokens.py` | Estimate token counts when provider usage is missing | | `scripts/resolve_pricing.py` | Resolve pricing sources and compute estimated/official pricing | | `scripts/score_models.py` | Combine raw metrics and rubric scores into rankings | | `scripts/build_report.py` | Build markdown, HTML, and PDF report artifacts | | `scripts/publish_report.py` | No deployment automation. Export/copy PDF and print suggested static hosting options (Vercel/Netlify/Cloudflare Pages/GitHub Pages). | --- ## Output contract Try to produce these artifacts whenever possible: - versioned benchmark spec - raw per-model answer files - raw metrics JSON - score breakdown JSON - markdown summary report - HTML landing page - PDF output when requested - publish result metadata when delivery occurs

path: assets/landingpage-template/index.html
content: <!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>{{TITLE}}</title>
  <link rel="stylesheet" href="style.css">
</head>
<body>
  <main class="container">
    <header class="hero">
      <h1>{{TITLE}}</h1>
      <p class="subtitle">{{SUBTITLE}}</p>
    </header>

    <section id="overview">
      <h2>Overview</h2>
      {{SUMMARY_HTML}}
    </section>

    <section id="ranking">
      <h2>Ranking</h2>
      {{RANKING_TABLE_HTML}}
    </section>

    <section id="details">
      <h2>Model Detail</h2>
      {{DETAILS_HTML}}
    </section>

    {{ANSWERS_HTML}}

    <footer>{{FOOTER}}</footer>
  </main>
  <script src="script.js"></script>
</body>
</html>


path: assets/landingpage-template/script.js
content: // Placeholder for sortable-table or interactive leaderboard logic.
console.log('benchmark-model-provider landing page template loaded');


path: assets/landingpage-template/style.css
content: body { font-family: Arial, sans-serif; background: #0b1020; color: #eef2ff; margin: 0; }
.container { max-width: 1180px; margin: 0 auto; padding: 32px 20px 64px; }
.hero { margin-bottom: 18px; }
h1 { margin-bottom: 8px; }
.subtitle { color: #b7c0e0; }
section { background: #131a2f; border: 1px solid #273250; border-radius: 14px; padding: 20px; margin-top: 20px; }
.card-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 16px; }
.card, .detail-card, .answer-card { background: #0f1630; border: 1px solid #273250; border-radius: 12px; padding: 16px; }
.details-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: 16px; }
table { width: 100%; border-collapse: collapse; }
th, td { padding: 10px; border-bottom: 1px solid #273250; text-align: left; vertical-align: top; }
th { color: #b7c0e0; font-weight: 600; }
ul { margin: 0; padding-left: 18px; }
pre { white-space: pre-wrap; word-break: break-word; overflow-wrap: anywhere; background: #0b1020; border: 1px solid #273250; border-radius: 10px; padding: 12px; color: #eef2ff; }
h4 { margin-bottom: 8px; color: #b7c0e0; }
footer { margin-top: 24px; color: #b7c0e0; }


path: examples/benchmark-context-template-en.yaml
content: purpose: Describe the main jobs the bot should help with.
domain: Describe the work domain or industry.
usage_frequency: Describe how often the bot is used.
language: en
notes:
  - Optional extra constraints
  - Preferred answer style
  - Important evaluation concerns


path: examples/benchmark-context-template-zh.yaml
content: purpose: 描述机器人需要协助完成的主要任务。
domain: 描述工作领域或行业。
usage_frequency: 描述机器人的使用频率。
language: zh
notes:
  - 可选的额外约束
  - 偏好的回答风格
  - 重要的评估关注点


path: examples/benchmark-context-template.yaml
content: purpose: Describe the main jobs the bot should help with.
domain: Describe the work domain or industry.
usage_frequency: Describe how often the bot is used.
language: vi
notes:
  - Optional extra constraints
  - Preferred answer style
  - Important evaluation concerns


path: examples/sample-benchmark-coding.yaml
content: name: coding-assistant-benchmark
version: v1
mode: prompt_only
context_profile:
  purpose: debugging, code review, and feature implementation
  domain: software engineering
  usage_frequency: many times per day
models:
  - example/model-a
  - example/model-b
questions:
  - id: q1
    title: Debug root cause
    task_type: debugging
    prompt: "Given a Python stack trace for a failing API request, identify the likely root cause and propose the smallest safe fix."
  - id: q2
    title: Review patch
    task_type: code-review
    prompt: "Review a pull request that improves performance but may weaken input validation. Explain the trade-offs and recommend a final decision."


path: examples/sample-benchmark-en.yaml
content: name: general-knowledge-worker-en
version: v1
mode: prompt_only
context_profile:
  purpose: research, summarization, and planning
  domain: knowledge work
  usage_frequency: daily
models:
  - example/model-a
  - example/model-b
questions:
  - id: q1
    title: Daily brief
    task_type: summary
    prompt: "Summarize the three most important developments in my industry today and explain why they matter."
  - id: q2
    title: Decision memo
    task_type: analysis
    prompt: "Write a short decision memo comparing two possible strategies for a small team with limited budget."


path: requirements.txt
content: PyYAML
reportlab


path: scripts/estimate_tokens.py
content: #!/usr/bin/env python3
import sys

def estimate_tokens(text: str) -> int:
    return max(1, round(len(text) / 3.6))

if __name__ == '__main__':
    text = sys.stdin.read()
    print(estimate_tokens(text))


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)