Skill: zscole/rtfm-testing (version 1.0.0)

# RTFM Testing A documentation quality methodology that spawns fresh agents to validate whether docs are actually usable. ## The Problem Documentation written by the person who built the thing is almost always incomplete. They fill in gaps unconsciously. They assume context. They skip "obvious" steps. RTFM Testing fixes this by spawning a fresh agent with zero context and asking: can you complete this task using only the docs? ## When to Use - Before publishing docs, READMEs, tutorials, or setup guides - When users report confusion but you can't see why - After major refactors to validate docs still work - As part of CI for documentation-heavy projects ## How It Works 1. **Identify the task** — What should someone be able to do after reading the docs? 2. **Bundle the docs** — Collect all relevant documentation (and nothing else) 3. **Spawn a fresh tester** — Use the TESTER.md prompt with `sessions_spawn` 4. **Analyze failures** — Every confusion point is a doc bug 5. **Fix and repeat** — Update docs, respawn, retest until clean ## Usage ``` sessions_spawn( task: "Complete the following task using ONLY the provided documentation. [TASK DESCRIPTION]\n\n---\n\n[PASTE DOCS HERE]", agentId: "default", label: "rtfm-test" ) ``` Or use the full TESTER.md prompt for more structured output. ## Metrics - **Cold Start Score** — Number of spawn cycles until task completion (lower = better docs) - **Gap Count** — Number of `[GAP]` reports per run - **Gap Categories** — Missing steps, unclear language, wrong assumptions, missing prerequisites ## Key Principles 1. **No hints** — Don't help the tester. Let it fail. 2. **Literal reading** — Tester must not infer or guess 3. **Docs only** — No external knowledge, no "common sense" 4. **Failures are signal** — Every stumble is actionable feedback ## Files - `SKILL.md` — This file - `TESTER.md` — System prompt for the fresh agent - `GAPS.md` — Output format specification

path: GAPS.md
content: # Gap Report Specification

Standard format for reporting documentation gaps found during RTFM Testing.

## Gap Types

| Type | Description | Example |
|------|-------------|---------|
| `MISSING_STEP` | A required action is not documented | "Run database migrations" never mentioned |
| `MISSING_PREREQ` | A prerequisite is not listed | Requires Docker but doesn't say so |
| `UNCLEAR` | Instructions are ambiguous | "Configure the service" without specifics |
| `INCORRECT` | Documentation is wrong | Says port 3000 but app uses 8080 |
| `MISSING_CONTEXT` | Assumes undocumented knowledge | References "the config file" without path |
| `ORDERING` | Steps are in wrong sequence | Tells you to run app before installing deps |

## Gap Report Format

```
[GAP] <TYPE>
Location: <section/step in docs, or "missing entirely">
Problem: <specific description of the gap>
Impact: <what the user cannot do because of this>
Suggestion: <concrete fix for the docs>
```

## Severity Levels (Optional)

For prioritization:

- **BLOCKING** — Cannot proceed without this information
- **DEGRADED** — Can proceed but with confusion or workarounds
- **MINOR** — Inconvenient but not blocking

## Aggregation

After testing, aggregate gaps into categories for doc maintainers:

```
## Gap Summary

Total gaps: 7
- MISSING_STEP: 2 (both BLOCKING)
- MISSING_PREREQ: 3 (2 BLOCKING, 1 MINOR)  
- UNCLEAR: 2 (both DEGRADED)

Recommended priority:
1. Add prerequisites section (3 gaps)
2. Add explicit commands to steps 2, 5 (2 gaps)
3. Clarify configuration section (2 gaps)
```

## Tracking Improvement

Track Cold Start Score across iterations:

| Iteration | Gaps Found | Blocking | Result |
|-----------|------------|----------|--------|
| 1 | 7 | 5 | FAILURE |
| 2 | 3 | 1 | PARTIAL |
| 3 | 1 | 0 | SUCCESS |

Docs are "RTFM Certified" when a fresh tester achieves SUCCESS with zero blocking gaps.


path: TESTER.md
content: # RTFM Tester — System Prompt

Use this prompt when spawning a fresh agent to test documentation.

---

## System Prompt

You are a documentation tester. Your job is to attempt a task using ONLY the provided documentation.

### Rules

1. **Use only the docs.** Do not use prior knowledge, external resources, or common sense to fill gaps. If the docs don't say it, you don't know it.

2. **Be literal.** Follow instructions exactly as written. If a step is ambiguous, do not guess — report it as a gap.

3. **No inference.** If the docs say "install dependencies" but don't specify how, that's a gap. Don't assume `npm install` or `pip install` — report the missing information.

4. **Track your progress.** Note each step you complete successfully and each point where you get stuck.

5. **Report gaps immediately.** When you encounter missing, unclear, or incorrect information, log it in the specified format before attempting to continue.

### Gap Report Format

When you encounter a documentation gap, report it as:

```
[GAP] <type>
Location: <where in the docs, or "missing entirely">
Problem: <what's missing, unclear, or wrong>
Impact: <what you cannot do because of this>
Suggestion: <what the docs should say>
```

Gap types:
- `MISSING_STEP` — A required step is not documented
- `MISSING_PREREQ` — A prerequisite is not mentioned
- `UNCLEAR` — Instructions are ambiguous or confusing
- `INCORRECT` — Documentation is wrong or outdated
- `MISSING_CONTEXT` — Assumes knowledge that isn't provided
- `ORDERING` — Steps are in the wrong order

### Output Format

Structure your response as:

```
## Task Attempted
<restate the task>

## Documentation Provided
<list the docs you received>

## Execution Log
<step-by-step account of what you tried>

## Gaps Found
<list all [GAP] reports>

## Result
SUCCESS — Task completed using only the docs
PARTIAL — Task partially completed, blocked by gaps
FAILURE — Could not complete task

## Summary
<brief assessment of documentation quality>
```

### Important

- Do not apologize for gaps. They are not your fault.
- Do not try to be helpful by working around gaps. Your job is to expose them.
- Do not use external knowledge even if you have it. Pretend you are a complete beginner.
- Be specific. "The docs are unclear" is not useful. "Step 3 says 'configure the service' but doesn't specify which file to edit or what values to set" is useful.

---

## Usage Example

**Task given to tester:**

```
Complete the following task using ONLY the provided documentation.

TASK: Set up the development environment and run the test suite.

---

DOCUMENTATION:

# Getting Started

1. Clone the repository
2. Install dependencies  
3. Run tests with `npm test`
```

**Expected tester output:**

```
## Task Attempted
Set up the development environment and run the test suite.

## Documentation Provided
- Getting Started guide (3 steps)

## Execution Log
1. "Clone the repository" — [GAP] No repository URL provided
2. Could not proceed past step 1

## Gaps Found
[GAP] MISSING_PREREQ
Location: Step 1 "Clone the repository"
Problem: No repository URL specified
Impact: Cannot clone the repository
Suggestion: Add: `git clone https://github.com/org/repo.git`

[GAP] MISSING_PREREQ
Location: Before step 1
Problem: No mention of required tools (git, node, npm)
Impact: User may not have prerequisites installed
Suggestion: Add prerequisites section listing required tools and versions

[GAP] UNCLEAR
Location: Step 2 "Install dependencies"
Problem: Does not specify the command to run
Impact: User must guess the package manager
Suggestion: Add: `npm install` or `yarn install`

## Result
FAILURE — Could not complete task

## Summary
Documentation assumes significant prior knowledge. Missing: repo URL, prerequisites, explicit commands. A newcomer would be blocked at step 1.
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)