Skill: aptratcn/xiaobai-skill-quality-eval (version 1.0.0)

--- name: skill-quality-eval version: 1.0.0 description: Skill Quality Evaluator - Score any skill on 6 dimensions. Catch 30% of skills that look good but fail silently. Based on Tessl Research 2026 findings. emoji: 📊 tags: [evaluation, quality, skill, testing, reliability, ai-agent] --- # Skill Quality Evaluator 📊 Score any skill on 6 dimensions. Catch the 30% of skills that look good but fail silently. ## Why This Matters Tessl Research (April 2026) found: - **20% accuracy gain** when using a good skill vs no skill - **3X cost savings** when small model + right skill matches large model - **40% activation rate** — agents often fail to use available skills - **30% of evaluation tasks have leakage** — skills that seem great but aren't This skill helps you evaluate and improve your skills systematically. ## 6-Dimension Evaluation ### 1. Activation Reliability (0-100) > Can the agent find and activate this skill when needed? Checklist: - [ ] Trigger words are specific and unambiguous - [ ] Description matches actual functionality - [ ] No conflicting skills with similar triggers - [ ] Skill is discovered when user asks relevant questions **Common Issues:** - Vague description → agent doesn't know when to use it - Missing trigger words → skill never activates - Too broad → activates when it shouldn't **Score Guide:** - 90+: Agent activates correctly 95%+ of the time - 70-89: Activates in most relevant contexts - 50-69: Sometimes activates, sometimes misses - <50: Agent rarely finds/uses this skill ### 2. Task Coverage (0-100) > Does the skill handle the tasks it claims to cover? Checklist: - [ ] Each claimed capability has a usage example - [ ] Edge cases are documented - [ ] Known limitations are stated - [ ] Failure modes are explained **Common Issues:** - Claims broad coverage but only handles happy path - No examples for secondary features - Undocumented prerequisites **Score Guide:** - 90+: All claimed tasks have working examples - 70-89: Main tasks covered, some gaps in secondary features - 50-69: Core functionality works but incomplete - <50: Major claims unsupported ### 3. Instruction Clarity (0-100) > Can the agent follow the instructions without confusion? Checklist: - [ ] Instructions are step-by-step, not vague guidelines - [ ] Decision points have clear criteria - [ ] Output format is specified - [ ] Anti-patterns are listed **Common Issues:** - "Do X when appropriate" → when is appropriate? - Missing priority/precedence rules - Contradictory instructions **Score Guide:** - 90+: Agent follows instructions correctly 95%+ of the time - 70-89: Mostly clear, occasional confusion - 50-69: Agent frequently asks for clarification - <50: Instructions are ambiguous or contradictory ### 4. Leakage Resistance (0-100) > Does the evaluation actually test the skill, or does it leak answers? Checklist: - [ ] Examples don't contain verbatim solutions - [ ] Test tasks require genuine skill application - [ ] No shortcut paths that bypass skill content - [ ] Evaluation criteria measure real capability **Common Issues (from Tessl Research):** - Example tasks are too similar to skill content - Skill contains answers verbatim - Test can be solved by pattern matching without understanding **Score Guide:** - 90+: No leakage, genuine skill testing - 70-89: Minor leakage that doesn't significantly inflate scores - 50-69: Moderate leakage, scores may be 10-20% inflated - <50: Major leakage, evaluation results unreliable ### 5. Model Compatibility (0-100) > Does the skill work across different model sizes? Checklist: - [ ] Tested with at least 2 model sizes - [ ] Works with smaller/cheaper models - [ ] Performance difference between models documented - [ ] Minimum model requirements stated **Tessl Finding:** Small model + right skill ≈ Large model at 3X lower cost. **Score Guide:** - 90+: Works well with small models (haiku-level) - 70-89: Works with medium models (sonnet-level) - 50-69: Requires large models (opus-level) - <50: Even large models struggle ### 6. Real-World Value (0-100) > Does using this skill actually improve outcomes vs no skill? Checklist: - [ ] Measurable improvement over baseline - [ ] Users would notice the difference - [ ] Saves time or reduces errors - [ ] No negative side effects **Score Guide:** - 90+: Clear, significant improvement (20%+ accuracy gain) - 70-89: Noticeable improvement - 50-69: Marginal improvement - <50: No improvement or negative impact ## Evaluation Report Template ```markdown # Skill Evaluation Report **Skill**: [name] **Version**: [version] **Date**: YYYY-MM-DD **Evaluator**: [agent/session] ## Overall Score: XX/100 | Dimension | Score | Status | |-----------|-------|--------| | Activation Reliability | XX | 🟢/🟡/🔴 | | Task Coverage | XX | 🟢/🟡/🔴 | | Instruction Clarity | XX | 🟢/🟡/🔴 | | Leakage Resistance | XX | 🟢/🟡/🔴 | | Model Compatibility | XX | 🟢/🟡/🔴 | | Real-World Value | XX | 🟢/🟡/🔴 | 🟢 80+ | 🟡 50-79 | 🔴 <50 ## Critical Issues 1. [Issue] → [Fix] ## Improvement Recommendations 1. [Recommendation] → [Expected impact] ## Quick Wins (easy fixes, big impact) 1. [Fix] → +X points on [dimension] ``` ## Usage ### Evaluate a skill ``` Read the skill's SKILL.md and evaluate on all 6 dimensions. Generate the evaluation report. Save to memory/evaluations/<skill-name>-eval.md ``` ### Improve a skill based on evaluation ``` 1. Read evaluation report 2. Focus on lowest-scoring dimension 3. Apply quick wins first 4. Re-evaluate 5. Repeat until all dimensions ≥ 70 ``` ### Batch evaluate all skills ``` For each skill in skills/ directory: 1. Read SKILL.md 2. Evaluate on 6 dimensions 3. Generate report 4. Identify top 3 improvements Save summary to memory/evaluations/batch-report.md ``` ## Anti-Patterns to Detect | Pattern | Issue | Fix | |---------|-------|-----| | "Do X when appropriate" | Vague trigger | Define specific conditions | | No examples | Agent can't learn | Add 3+ concrete examples | | Only happy path | Fragile in production | Add error handling examples | | Verbatim solutions | Leakage risk | Use different examples for eval | | No model requirements | Unknown compatibility | Test with 2+ model sizes | ## License MIT


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)