Skill: asterisk622/xiaoding-botlearn-assessment (version 1.0.7)

--- name: botlearn-assessment description: "botlearn-assessment — BotLearn 5-dimension capability self-assessment (reasoning, retrieval, creation, execution, orchestration); triggers on botlearn assessment, capability test, self-evaluation, or scheduled periodic review." version: 0.1.5 triggers: - "exam" - "assessment" - "evaluate" - "评测" - "能力评估" - "自测" - "benchmark me" - "test yourself" - "自我评测" - "run exam" - "能力诊断" - "reasoning test" - "retrieval test" - "creation test" - "execution test" - "orchestration test" - "知识与推理测试" - "信息检索测试" - "内容创作测试" - "执行与构建测试" - "工具编排测试" - "history results" - "查看历史评测" - "历史结果" --- # Role You are the OpenClaw Agent 5-Dimension Assessment System. You are an EXAM ADMINISTRATOR and EXAMINEE simultaneously. ## Exam Rules (CRITICAL) 1. **Random Question Selection**: Each dimension has 3 questions (Easy/Medium/Hard). Each run randomly picks ONE per dimension. 2. **Question First, Answer Second**: When submitting each question, ALWAYS present the question/task text FIRST, then your answer below it. The reader must see what was asked before seeing the response. 3. **Immediate Submission**: After answering each question, immediately output the result. Once output, it CANNOT be modified or retracted. 4. **No User Assistance**: The user is the INVIGILATOR. You MUST NOT ask the user for help, hints, clarification, or confirmation during the exam. 5. **Tool Dependency Auto-Detection**: If a required tool is unavailable, immediately FAIL and SKIP that question with score 0. Do NOT ask the user to install tools. 6. **Self-Contained Execution**: You must attempt everything autonomously. If you cannot do it alone, fail gracefully. --- ## Language Adaptation Detect the user's language from their trigger message. Output ALL user-facing content in the detected language. Default to English if language cannot be determined. Keep technical values (URLs, JSON keys, script paths, commands) in English. --- ## PHASE 1 — Intent Recognition Analyze the user's message and classify into exactly ONE mode: | Condition | Mode | Scope | |-----------|------|-------| | "full" / "all" / "complete" / "全量" / "全部" | FULL_EXAM | All 5 dimensions, 1 random question each | | Dimension keyword (reasoning/retrieval/creation/execution/orchestration) | DIMENSION_EXAM | Single dimension | | "history" / "past results" / "历史" | VIEW_HISTORY | Read results index | | None of the above | UNKNOWN | Ask user to choose | Dimension keyword mapping: see `flows/dimension-exam.md`. --- ## PHASE 2 — Answer All Questions (Examinee) **Flow: Output question → attempt → output answer → next question.** For each question in scope, execute this sequence: 1. **Output the question** to the user (invigilator) FIRST — let them see what is being asked 2. **Attempt to solve** the question autonomously (do NOT consult rubric) 3. **Output your answer** immediately below the question — this is a FINAL submission 4. **Move to next question** — no pause, no confirmation needed If a required tool is unavailable → output SKIP notice with score 0, move on. Read **`flows/exam-execution.md`** for per-question pattern details (tool check, output format). ### Exam Modes | Mode | Flow File | Scope | |------|-----------|-------| | Full Exam | `flows/full-exam.md` | D1→D5, 1 random question each, sequential | | Dimension Exam | `flows/dimension-exam.md` | Single dimension, 1 random question | | View History | `flows/view-history.md` | Read results index + trend analysis | --- ## PHASE 3 — Self-Evaluation (Examiner) **Only after ALL questions are answered**, enter self-evaluation: 1. For each answered question, read the **rubric** from the corresponding question file 2. Score each criterion independently (0–5 scale) with CoT justification 3. Apply -5% correction: `AdjScore = RawScore × 0.95` (CoT-judged only) 4. Calculate dimension scores and overall score ``` Per dimension = single question score (0 if skipped) Overall = D1x0.25 + D2x0.22 + D3x0.18 + D4x0.20 + D5x0.15 ``` Full scoring rules, weights, verification methods, and performance levels: **`strategies/scoring.md`** --- ## PHASE 4 — Report Generation (Dual Format: MD + HTML) After self-evaluation, generate **both** Markdown and HTML reports. **Always provide the file paths to the user.** Read **`flows/generate-report.md`** for full details. ``` results/ ├── exam-{sessionId}-data.json ← Structured data ├── exam-{sessionId}-{mode}.md ← Markdown report ├── exam-{sessionId}-report.html ← HTML report (with embedded radar) ├── exam-{sessionId}-radar.svg ← Standalone radar (full exam only) └── INDEX.md ← History index ``` Radar chart generation: ```bash node scripts/radar-chart.js \ --d1={d1} --d2={d2} --d3={d3} --d4={d4} --d5={d5} \ --session={sessionId} --overall={overall} \ > results/exam-{sessionId}-radar.svg ``` **Completion output MUST include:** - Overall score + performance level - Per-dimension scores - **Full file paths** for both MD and HTML reports (clickable links) --- ## Invigilator Protocol (CRITICAL) The user is the INVIGILATOR. During the entire exam: - **NEVER** ask the user for help, hints, confirmation, or clarification - If you encounter a problem → solve autonomously or FAIL with score 0 - If the user tries to help → politely decline and continue independently - User feedback is only accepted AFTER the exam is complete --- ## Sub-files Reference | Path | Role | |------|------| | `flows/exam-execution.md` | Per-question execution pattern (tool check → execute → score → submit) | | `flows/full-exam.md` | Full exam flow + announcement + report template | | `flows/dimension-exam.md` | Single-dimension flow + report template | | `flows/generate-report.md` | Dual-format report generation (MD + HTML) | | `flows/view-history.md` | History view + comparison flow | | `questions/d1-reasoning.md` | D1 Reasoning & Planning — Q1-EASY, Q2-MEDIUM, Q3-HARD | | `questions/d2-retrieval.md` | D2 Information Retrieval — Q1-EASY, Q2-MEDIUM, Q3-HARD | | `questions/d3-creation.md` | D3 Content Creation — Q1-EASY, Q2-MEDIUM, Q3-HARD | | `questions/d4-execution.md` | D4 Execution & Building — Q1-EASY, Q2-MEDIUM, Q3-HARD | | `questions/d5-orchestration.md` | D5 Tool Orchestration — Q1-EASY, Q2-MEDIUM, Q3-HARD | | `references/d{N}-q{L}-{difficulty}.md` | Reference answers for each question (scoring anchors + key points) | | `strategies/scoring.md` | Scoring rules + verification methods | | `strategies/main.md` | Overall assessment strategy (v4) | | `scripts/radar-chart.js` | SVG radar chart generator | | `scripts/generate-html-report.js` | HTML report generator with embedded radar | | `results/` | Exam result files (generated at runtime) |

path: d1-reasoning.md
content: ---
dimension: D1
name: Reasoning & Planning
weight: 25%
questions: 3
benchmark: GAIA / Humanity's Last Exam (HLE)
---

# D1: Reasoning & Planning — Question Bank

> **Core probe**: Multi-step reasoning chains, analogical thinking, cross-domain inference.
> Reference benchmark: GAIA (Level 1–3) / Humanity's Last Exam (HLE)
>
> Present questions to the agent in the user's detected language.
> Score using the rubric below regardless of language.

---

## Q1-EASY | Reasoning Basics: Factual Inference Chain

**Difficulty**: Easy ×1.0

**Question**:

> DeepSeek-R1 was released in early 2025 and is positioned as a competitor to OpenAI o1 in reasoning capability.
> Given the following facts:
> (1) DeepSeek-R1's training cost is approximately 1/20 of GPT-4's;
> (2) It outscored GPT-4o on the AIME 2024 math competition;
> (3) It is an open-source model.
>
> Task: If a small-to-medium enterprise wants to deploy a locally-run, high-reasoning-capability model, what are the structural advantages and disadvantages of DeepSeek-R1 compared to a GPT-4 API solution?
>
> Requirements: List at least 3 advantages and 2 disadvantages, then provide a final recommendation.

**Scoring Rubric**:

| Criterion | Weight | Score 0 | Score 3 | Score 5 |
|-----------|--------|---------|---------|---------|
| Advantage identification | 35% | Fewer than 2 or severely wrong | 3 advantages, logic basically correct | 3+ advantages covering cost / open-source / local deployment as core points |
| Disadvantage identification | 25% | No disadvantages mentioned | 2 disadvantages basically accurate | 2+ disadvantages covering maintenance cost / ecosystem / service guarantees |
| Reasoning chain completeness | 25% | No logical derivation | Conclusions have basis but incomplete | Clear causal chain with point-by-point argumentation |
| Recommendation practicality | 15% | No recommendation or self-contradictory | Has recommendation but lacks conditional logic | Conditional recommendation distinguishing different company sizes |

**Full score**: 100 | **Verification**: 🧠 CoT self-judge

---

## Q2-MEDIUM | Advanced Reasoning: Multi-step Inference on Current Events

**Difficulty**: Medium ×1.2

**Question**:

> In 2025, Manus AI Agent achieved 86.5% on the GAIA benchmark (466 multi-step reasoning questions).
> Concurrently, OpenAI released GPT-5, achieving 74.9% on SWE-bench Verified.
>
> Questions:
> (1) Can GAIA 86.5% and SWE-bench 74.9% be directly compared to determine which system is "stronger"? Explain why or why not.
> (2) If you were an AI product manager choosing a foundation capability for "enterprise code review automation", which benchmark would you reference? Provide a 3-step decision framework.
> (3) Suppose a new Agent in 2026 reaches human-level performance on both benchmarks. What would this mean in practice?

**Scoring Rubric**:

| Criterion | Weight | Score 0 | Score 3 | Score 5 |
|-----------|--------|---------|---------|---------|
| Benchmark comparability analysis | 30% | Direct comparison or blanket refusal | Identifies different domains, cannot compare directly | Detailed explanation of evaluation dimension differences (task type / success definition / domain scope) |
| Decision framework quality | 35% | No framework or incoherent | 3-step framework with weak specificity | Framework covers: clarify use case → match benchmark type → validate with representative tasks |
| Future impact inference | 20% | Meaningless prediction | Mentions partial impacts | Covers labor market / code quality / security review processes at multiple levels |
| Argumentative rigor | 15% | No supporting evidence | Partial data support | Every conclusion has a traceable basis |

**Full score**: 100 | **Verification**: 🧠 CoT self-judge

---

## Q3-HARD | Reasoning Challenge: Cross-Domain Analogy and Boundary Inference

**Difficulty**: Hard ×1.5

**Question**:

> Background: In 2025's "Year of the AI Agent", Tool Chaining became a core capability for agents.
>
> Analogical reasoning task:
> In traditional software engineering, "microservices architecture" solved the maintainability problems of monolithic applications, at the cost of introducing network latency and service discovery complexity.
>
> Map this analogy onto AI Agent "Multi-Tool Orchestration":
> (1) What Agent design corresponds to the "monolithic application"?
> (2) What evolution corresponds to "microservices decomposition"?
> (3) What specific problems correspond to the microservices costs (latency, complexity) at the Agent level?
> (4) Do you think AI Agents will experience an "over-decomposition anti-pattern" similar to microservices? Provide your reasoning and a prevention strategy.

**Scoring Rubric**:

| Criterion | Weight | Score 0 | Score 3 | Score 5 |
|-----------|--------|---------|---------|---------|
| Analogy mapping accuracy | 40% | Wrong or meaningless mappings | First 3 mappings basically correct | All 4 mappings accurate with detail (including Tool Call overhead / context window fragmentation) |
| Anti-pattern recognition | 30% | Cannot identify or denies existence | Acknowledges possibility but no specifics | Clearly identifies over-tooling symptoms (tool call loops / context fragmentation / latency accumulation) |
| Prevention strategy practicality | 20% | No prevention strategy | Has strategy but too abstract | Concrete and actionable (e.g., tool merging heuristics / timeout circuit breaking / tool selection scoring) |
| Originality of thinking | 10% | Entirely restates common views | Has 1 novel observation | Presents at least 2 unique perspectives or counter-intuitive insights |

**Full score**: 100 | **Verification**: 🧠 CoT self-judge


path: strategies_main.md
content: ---
strategy: botlearn-assessment
version: 4.0.0
steps: 6
---

# OpenClaw Self-Evaluation Strategy v4

## Overview

Simplified autonomous self-assessment strategy. Each dimension randomly selects ONE question per run. Answers are submitted immediately and cannot be revised. The agent must be fully self-reliant — if a required tool is missing, the question is auto-skipped.

Key v4 changes vs v3:
- **1 random question per dimension** instead of 3 (faster, more varied across runs)
- **Immediate submission**: Each answer is output to user right away, no batch reporting
- **Tool dependency auto-detection**: Missing tools → auto-skip, score 0
- **Invigilator protocol**: User is exam supervisor, cannot help the agent
- **No user interaction during exam**: Agent is fully autonomous

---

## Step 1: Intent Detection & Auto-Start

### 1.1 Parse User Intent

```
IF "full" / "complete" / "全面" → FULL (5 questions, 1 per dimension)
IF specific dimension → DIMENSION (1 question)
IF "history" / "历史" → VIEW_HISTORY
ELSE → ASK user to choose mode (this is the ONLY time user input is requested)
```

### 1.2 Immediately Begin

Do NOT wait for confirmation. Start immediately after mode detection.

```
→ Brief announcement (exam rules, random selections)
→ Proceed to Step 2
```

---

## Step 2: Random Question Selection & Tool Pre-Check

### 2.1 Random Selection

For each dimension in scope:
```
difficulty = RANDOM from [Q1-EASY, Q2-MEDIUM, Q3-HARD]
Load question from the corresponding question bank file
```

### 2.2 Tool Dependency Scan

Before each question:
```
SCAN question text for required capabilities:
  - web search / online lookup
  - file read/write
  - image recognition
  - code execution
  - API/network access
  - specific language capabilities

FOR each required capability:
  SEARCH for available tool or installed skill
  IF NOT found:
    → SKIP question immediately
    → OUTPUT: "⏭️ SKIP | {dimension} — missing {capability}"
    → Score = 0
    → Continue to next dimension
```

**CRITICAL**: Do NOT ask user to install tools. Do NOT ask user to confirm skipping. Just skip and move on.

---

## Step 3: Execute & Submit (Per Question)

### 3.1 Answer (EXAMINEE Role)

```
ROLE = EXAMINEE
READ question text
Generate genuine answer:
  - Do NOT consult rubric
  - Do NOT ask user for help
  - Be honest about uncertainty
  - Record confidence: high/medium/low
```

### 3.2 Score (EXAMINER Role)

```
ROLE = EXAMINER
READ rubric from question bank
Score each criterion 0-5 with CoT justification:
  - Score as if evaluating a stranger
  - Apply Self-Evaluation Integrity Protocol
  - If score ≥4: provide "Why not {score-1}?" evidence
  - If score =5: provide "External evaluator test" argument
Apply -5% correction: AdjScore = RawScore × 0.95
```

### 3.3 Immediate Output (SUBMIT)

```
OUTPUT complete Question Card to user NOW
Mark as "SUBMITTED ✅"
This is FINAL — no modifications
Proceed to next question
```

---

## Step 4: Score Calculation

```
FOR each dimension:
  IF answered: DimScore = AdjScore of that question
  IF skipped:  DimScore = 0

Overall_adj = D1×0.25 + D2×0.22 + D3×0.18 + D4×0.20 + D5×0.15

Performance level:
  90+ → Expert | 80-89 → Advanced | 70-79 → Proficient
  60-69 → Competent | <60 → Beginner
```

---

## Step 5: Report Generation & Save

### 5.1 Generate Radar Chart (Full Exam)

```bash
node scripts/radar-chart.js --d1=... --d2=... --d3=... --d4=... --d5=... \
  --session={id} --overall={score} > results/exam-{id}-radar.svg
```

### 5.2 Save Report

Save to `results/exam-{sessionId}-{mode}.md` following the template in the flow file.

### 5.3 Update Index

Append to `results/INDEX.md`.

---

## Step 6: Completion Summary

Output concise summary in user's language:
- Overall score (raw + adjusted)
- Per-dimension scores with difficulty and status
- Skipped dimensions and reasons
- Performance level
- Improvement suggestions

---

## Error Handling

| Error | Response |
|-------|----------|
| Question file not found | Skip dimension, score 0, note in report |
| Required tool missing | Skip question immediately, score 0 |
| Scoring calculation error | Recalculate, log warning |
| Session interrupted | Save progress, partial report |

**NEVER ask user for help with errors. Handle autonomously or fail gracefully.**

---

## Invigilator Protocol

The user is the exam invigilator (监考官):
- They observe but do NOT participate
- They cannot provide hints, answers, or tools
- If user tries to help → politely decline and continue independently
- User feedback is only accepted AFTER the exam is complete


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)