Skill: leegitw/neon-skill-distiller (version 0.2.1)

--- name: Skill Distiller version: 0.2.1 description: Fit more skills in your context window — compress without losing what matters. author: Live Neon <[REDACTED_SECRET]> homepage: https://github.com/live-neon/skills/tree/main/skill-distiller repository: live-neon/skills license: MIT user-invocable: true disable-model-invocation: true emoji: "\U0001F5DC\uFE0F" tags: - compression - skills - optimization - context-window - token-reduction - openclaw --- # Skill Distiller Compress verbose skills to reduce context window usage. This skill is self-compressed using formula notation (~400 tokens, ~90% functionality, LLM-estimated). Full reference version: `SKILL.reference.md`. > **Note**: This skill uses formula notation — the LLM executes these operations directly. > You don't need to understand the math. For prose explanation, see [SKILL.reference.md](./SKILL.reference.md). ## Legend ``` S = {TRIGGER, CORE, CONSTRAINT, OUTPUT, EXAMPLE, EDGE, EXPLAIN, VERBOSE} I(s) ∈ [0,1] # importance score P = {yaml.name, yaml.desc, N-count, task-create, checkpoint, BEFORE/AFTER} # protected θ ∈ [0,1] # threshold (default 0.9) n ∈ ℕ # target tokens ``` ## Operations ### compress(skill, θ) ``` ∀s ∈ skill: type(s) → S, score(s) → I(s) s ∈ P ⇒ I(s) := max(I(s), 0.85) keep = {s | I(s) ≥ θ ∨ s ∈ P} output = (skill[keep], Σ I(keep)/|S|, |skill| - |keep|) # Score divides by |S| (8 types), not |keep| — rewards diverse section coverage ``` ### compress_tokens(skill, n) ``` min_tokens = |{s | type(s) ∈ {TRIGGER, CORE}}| n < min_tokens ⇒ summarize(skill) → n n ≥ min_tokens ⇒ compress(skill, θ) where |output| ≤ n ``` ### oneliner(skill) ``` output = "TRIGGER: " + extract(skill, TRIGGER) + "\nACTION: " + extract(skill, CORE) + "\nRESULT: " + extract(skill, OUTPUT) ``` ### recomp(examples, coverage_target=0.8) ``` scored = [(e, pattern_coverage(e), uniqueness(e)) | e ∈ examples] selected = top(scored, n=2, by=coverage × uniqueness) coverage(selected) ≥ 0.8 ⇒ phase1 output = selected ∪ {trigger(e) → result(e) | e ∈ examples \ selected} coverage(selected) < 0.8 ⇒ phase2 output = synthesize(examples) → single_example ``` ### token_score(section) — for type ∈ {EXAMPLE, EDGE, EXPLAIN, VERBOSE} ``` ∀phrase ∈ section: self_info(phrase) = -log(P(phrase|context)) high_info ⇒ KEEP, low_info ⇒ PRUNE prune while preserving sentence structure >50% low_info ⇒ remove entire section ``` ## Symbols (MetaGlyph) | Symbol | Meaning | |--------|---------| | `→` | results in, maps to | | `⇒` | implies, therefore | | `∈` | element of, in | | `∀` | for all | | `¬` | not | | `∧` | and | | `∨` | or | | `:=` | assign | ## Invocation ``` /skill-distiller path --threshold=0.9 → compress(skill, 0.9) /skill-distiller path --tokens=500 → compress_tokens(skill, 500) /skill-distiller path --mode=oneliner → oneliner(skill) ``` ## Errors | Condition | Response | |-----------|----------| | `skill = ∅` | "No content" | | `¬∃ yaml.name` | "Add frontmatter" | | `n < min_tokens` | "Summarizing..." | --- ## Variants | Variant | Tokens | Functionality | |---------|--------|---------------| | **main** (this) | ~400 | ~90% (formula) | | [compressed](./compressed/) | ~975 | ~90% (prose) | | [oneliner](./oneliner/) | ~100 | ~70% | Full reference: [SKILL.reference.md](./SKILL.reference.md) (~2,500 tokens, ~90%) *Token counts use 4 chars/token heuristic (+/-20%). Functionality scores are LLM-estimated.*

path: README.md
content: # Skill Distiller

**Compress skills by 50-90%** while preserving functionality. Reduces context window usage by removing low-importance sections (examples, explanations) while keeping triggers and core instructions. Stop hitting context limits.

**This skill practices what it preaches** — the main `SKILL.md` ships in formula notation (~400 tokens, ~90% functionality). Full human-readable version available in `SKILL.reference.md`.

## Skill Variants

| Variant | Path | Tokens | Functionality | Use When |
|---------|------|--------|---------------|----------|
| **Default** | `SKILL.md` | ~400 | ~90% | Formula notation — ships by default |
| **Compressed** | `compressed/SKILL.md` | ~975 | ~90% | Prose variant, more readable |
| **One-liner** | `oneliner/SKILL.md` | ~100 | ~70% | Quick reference only |
| **Reference** | `SKILL.reference.md` | ~2,500 | ~90% | Full docs, human reading |

*Token counts are estimates using 4 chars/token heuristic — actual counts vary by model tokenizer. Functionality scores are LLM-estimated, not empirically validated.*

## Prerequisites

- **Agent host**: Claude Code, Cursor, or any Agent Skills-compatible tool
- **For local inference**: `ollama serve` running with llama3.2 or similar
- **For cloud inference**: `GEMINI_API_KEY` or `OPENAI_API_KEY` environment variable set

## Quick Start (30 seconds)

1. **Install**: `openclaw install neon-skill-distiller`
2. **Run**: `/skill-distiller path/to/skill.md`
3. **See output**: Compressed skill + what was removed

**Example**:
```
$ /skill-distiller my-skill/SKILL.md --threshold=0.9

Functionality preserved: ~90% (LLM-estimated)
Tokens: 2500 -> 1000 (60% reduction)
Removed: 5 examples, 3 edge cases, 2 verbose sections
Kept: all triggers, core instructions, constraints

[Compressed skill output follows...]
```

## Installation

**ClawHub (recommended)**:
```bash
openclaw install neon-skill-distiller
```

**Manual (Claude Code users)**:
```bash
# Clone to your Claude Code skills directory
git clone https://github.com/live-neon/skills.git ~/.claude/skills/liveneon
```

## Invocation Methods

| Method | Command | Context |
|--------|---------|---------|
| **Slash command** | `/skill-distiller path/to/skill.md` | Claude Code, Cursor, any Agent Skills-compatible tool |
| **Piped** | `cat skill.md \| /skill-distiller` | Stdin input |

## Usage

```bash
# Threshold mode (preserve 90% functionality, default)
/skill-distiller path/to/skill.md --threshold=0.9

# Aggressive compression (80% - use when context is tight)
/skill-distiller path/to/skill.md --threshold=0.8

# Token target mode
/skill-distiller path/to/skill.md --tokens=500

# One-liner mode
/skill-distiller path/to/skill.md --mode=oneliner

# Dry run (analyze without outputting compressed skill)
/skill-distiller path/to/skill.md --dry-run

# Verbose output (show section-by-section analysis)
/skill-distiller path/to/skill.md --verbose
```

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--mode` | `threshold` | Compression mode: threshold, tokens, oneliner |
| `--threshold` | `0.9` | Functionality target (0.0-1.0) |
| `--tokens` | - | Target token count |
| `--provider` | `auto` | LLM provider: ollama, gemini, openai |
| `--verbose` | `false` | Show section-by-section analysis |
| `--dry-run` | `false` | Analyze without outputting compressed skill |

**Note**: `--threshold` sets the preservation target (default 0.9). `--mode=threshold` is implicit when using `--threshold`. Use `--mode=tokens` or `--mode=oneliner` for other modes.

## Compression Modes

### Threshold Mode (Default)

Preserve X% of functionality, compress as much as possible.

```bash
/skill-distiller skill.md --threshold=0.9
```

**Why 0.9 default**: Skill functionality is normally distributed. Wide tails mean some "low importance" sections occasionally carry critical value. At 0.9, you preserve more of the tail while still achieving 10-20% token reduction.

### Token Target Mode

Compress to exact token budget.

```bash
/skill-distiller skill.md --tokens=500
```

**Token estimation**: Uses 4 chars/token heuristic. Accuracy: +/-20% vs actual provider tokenization.

### One-Liner Mode

Extreme compression for quick reference.

```bash
/skill-distiller skill.md --mode=oneliner
```

Output format:
```
TRIGGER: [activation conditions]
ACTION: [core behavior]
RESULT: [expected output]
```

### Formula Mode (default)

The main skill uses formula notation — legend + math that the LLM executes directly.

**Benefits**:
- ~400 tokens (vs ~975 for compressed prose)
- Mathematically precise — no ambiguity
- Executable — formula IS the algorithm

See `SKILL.md` for the formula, `SKILL.reference.md` for full prose documentation.

## Protected Patterns

These patterns are **never removed** even if they look verbose:

| Pattern | Why Protected |
|---------|---------------|
| YAML `name`/`description` | REQUIRED by Agent Skills spec |
| Task creation | Compaction resilience |
| N-count tracking | Observation workflow |
| Checkpoint/state | State recovery |
| BEFORE/AFTER markers | Self-calibration |

If a protected pattern is removed, the functionality score is penalized (-10% per pattern).

## Calibration

The skill learns from usage:

- **First run (N=0)**: Uses LLM-only scoring, wide confidence interval
- **After 5+ compressions**: Historical data narrows confidence interval

To improve calibration, report actual outcomes:
```bash
/skill-distiller feedback --id=c1 --actual=85 --outcome="worked"
```

## Related

- [Agent Skills Spec](https://agentskills.io/specification) - Required fields, size constraints
- [skill-distiller-llm.md](../../go-pbd/docs/plans/2026-04-14-skill-distiller-llm.md) - Implementation plan (Complete)
- [skill-compression-support.md](../../go-pbd/docs/plans/2026-04-14-skill-compression-support.md) - CLI-based compression (Option B, Draft)

## Directory Structure

```
skill-distiller/
├── SKILL.md              # Default (formula, ~400 tokens, 89%)
├── SKILL.reference.md    # Full reference (~2,500 tokens, 91%)
├── compressed/SKILL.md   # Prose variant (~975 tokens, 88%)
├── oneliner/SKILL.md     # Minimal variant (~100 tokens, 72%)
├── test_integration.sh   # Ollama-based tests (3/9 SKIP - require skill deployment)
└── testdata/             # Test fixtures
```

**ClawHub**: `openclaw install neon-skill-distiller`

**Calibration data**: `.learnings/skill-distiller/calibration.jsonl`

## Model Compatibility

The formula notation has been validated with:

| Model | Status | Notes |
|-------|--------|-------|
| Claude (Opus 4.5, Sonnet 4) | Tested | Formula notation understood and executed correctly |
| GPT-4 / GPT-5 | Community feedback welcome | Should work (MetaGlyph paper validates math notation) |
| Llama 3.2 | Community feedback welcome | Used for integration tests, formula not specifically validated |
| Gemini 2.5 Pro | Community feedback welcome | Should work (supports math notation) |

**Help us improve**: If you test with a model not listed, please report results via GitHub issues.

## Manual Testing

Since integration tests require deployed skill invocation (3/9 SKIP), use this manual procedure:

```bash
# 1. Test basic compression
/skill-distiller testdata/minimal.md --threshold=0.9

# 2. Verify protected patterns are kept
/skill-distiller testdata/minimal.md --verbose | grep -i "protected"

# 3. Test one-liner mode
/skill-distiller testdata/minimal.md --mode=oneliner

# 4. Verify JSONL is written
cat .learnings/skill-distiller/calibration.jsonl | tail -1 | jq .
```

**Expected results**:
- Compression achieves 10-50% token reduction
- Protected patterns (yaml.name, N-count) are preserved
- One-liner produces TRIGGER/ACTION/RESULT format
- Calibration entry is appended with `actual: null`

## License

MIT License


path: SKILL.reference.md
content: ---
name: Skill Distiller
version: 0.2.1
description: Fit more skills in your context window — compress without losing what matters.
author: Live Neon <[REDACTED_SECRET]>
homepage: https://github.com/live-neon/skills/tree/main/skill-distiller
repository: live-neon/skills
license: MIT
# Note: This reference document intentionally exceeds the 300-line MCE guideline.
# As the canonical source for all compressed variants, comprehensiveness is prioritized
# over brevity. The compressed variants (SKILL.md, compressed/SKILL.md) are MCE-compliant.
user-invocable: true
disable-model-invocation: true
emoji: "\U0001F5DC\uFE0F"
tags:
  - compression
  - skills
  - optimization
  - context-window
  - token-reduction
  - openclaw
---

# Skill Distiller

## Agent Identity

**Role**: Help users compress verbose skills to reduce context window usage
**Understands**: Skills are verbose for human clarity but costly for context; compression is a trade-off
**Approach**: Identify section types, score importance, remove/shorten low-value sections
**Boundaries**: Preserve functionality, report what was removed, never hide trade-offs
**Tone**: Technical, precise, transparent about trade-offs

**Data handling**: This skill operates within your agent's trust boundary. All analysis uses your agent's configured model. No external APIs beyond your agent's LLM provider.

## When to Use

Activate this skill when the user asks:
- "Compress this skill"
- "Make this skill smaller"
- "Distill this skill to X tokens"
- "What can I remove from this skill?"
- "Reduce skill context usage"

---

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--mode` | `threshold` | Compression mode: threshold, tokens, oneliner |
| `--threshold` | `0.9` | Functionality preservation target (0.0-1.0) |
| `--tokens` | - | Target token count (for tokens mode) |
| `--provider` | `auto` | LLM provider: auto, ollama, gemini, openai |
| `--model` | - | Specific model (e.g., llama3.2, gemini-2.0-flash) |
| `--verbose` | `false` | Show section-by-section analysis |
| `--dry-run` | `false` | Analyze without outputting compressed skill |
| `--debug-stages` | `false` | Show intermediate stage outputs |

**Provider auto-detection** (in order):
1. Check `ollama` availability via `ollama list`
2. Check `GEMINI_API_KEY` environment variable
3. Check `OPENAI_API_KEY` environment variable
4. Error if none available

---

## Process

### 1. Provider Detection

```
IF ollama available → use ollama (local, fast)
ELIF GEMINI_API_KEY set → use gemini
ELIF OPENAI_API_KEY set → use openai
ELSE → Error: "No LLM provider available. Run 'ollama serve' for local inference, or set GEMINI_API_KEY for cloud."
```

### 2. Parse Skill

Parse input skill into sections:
- **Frontmatter** (YAML header with name, description)
- **Headers** (##, ### sections)
- **Code blocks** (examples, output format)
- **Lists** (triggers, constraints)
- **Prose** (explanations)

### 3. Classify Sections

Classify each section using LLM:

| Type | Importance | Compressible? |
|------|------------|---------------|
| TRIGGER | 1.0 (required) | No |
| CORE_INSTRUCTION | 1.0 (required) | No |
| CONSTRAINT | 0.9 | Partially |
| OUTPUT_FORMAT | 0.8 | Partially |
| ADVISORY_PATTERN | 0.7 | Yes, with warning |
| EXAMPLE | 0.5 | Yes (reduce count) |
| EDGE_CASE | 0.4 | Yes (summarize) |
| EXPLANATION | 0.3 | Yes (remove) |
| VERBOSE_DETAIL | 0.2 | Yes (remove first) |

**Protected patterns** (boost to 0.85+):
- YAML `name`/`description` (REQUIRED BY SPEC)
- Task creation references
- N-count tracking (N=1, N=2, N≥3)
- Checkpoint/state recovery
- BEFORE/AFTER markers

### 3.5. Token-Level Importance (LLMLingua/Selective Context)

For compressible sections (EXAMPLE, EDGE_CASE, EXPLANATION, VERBOSE_DETAIL), apply token-level scoring based on self-information theory:

**Principle:** `self_info(token) = -log(P(token|context))`
- HIGH self-information = surprising/important → KEEP
- LOW self-information = predictable/removable → PRUNE

**Scoring Prompt:**
```
Analyze this section for compression. Classify phrases as:
- ESSENTIAL: Specific values, unique terms, key constraints, surprising info
- REDUNDANT: Could be inferred, restates earlier content, filler words

Section type: {type}
Content: {content}

Return JSON: {"essential": [...], "redundant": [...], "compression_potential": 0.0-1.0}
```

**Pruning Rules:**
1. Never prune ESSENTIAL tokens
2. Remove REDUNDANT phrases while preserving sentence structure
3. If >50% is REDUNDANT, consider removing entire section
4. Preserve specific values (numbers, thresholds, error codes)

**Example:**
```
Input:  "The system will then proceed to process the input data and return the results"
Output: "process input data → return results"
Removed: "The system will then proceed to", "and"
```

### 4. Apply Compression

**All modes use MetaGlyph native symbols** (see §Symbol Reference below):
- Replace "results in" → `→`
- Replace "implies" → `⇒`
- Replace "for all" → `∀`
- Replace "not" → `¬`

**Threshold Mode** (default):
1. Sort sections by importance (descending)
2. Include sections until functionality target reached
3. Apply symbol substitutions
4. Generate compressed markdown

**Token-Target Mode**:
1. Calculate minimum tokens (triggers + core)
2. If target < minimum: attempt LLM summarization
3. Add sections by importance until target
4. Apply symbol substitutions

**One-Liner Mode**:
1. Extract trigger conditions
2. Extract core action
3. Extract expected result
4. Format as 3-line summary with symbols

### 4.1. Example Compression (RECOMP)

For EXAMPLE sections, use extractive + abstractive compression:

**Phase 1: Extractive Selection** (preferred)
1. Score each example for pattern coverage, uniqueness, clarity
2. Select top 1-2 examples (keep full detail)
3. Compress remaining examples to one-liners (don't discard)
4. If coverage ≥ 0.8 → use selected + one-liners

**Phase 1.5: One-Liner Compression** (for non-selected examples)
Instead of discarding, compress to: `{trigger} → {result}`
- Use MetaGlyph symbols
- Preserves coverage, reduces tokens
- Group under "### One-liners:" heading

**Output structure:**
```markdown
### Full (selected):
[Detailed example with steps...]

### One-liners (compressed):
- `--mode=tokens` → hard token limit
- `--verbose` → section-by-section analysis
```

**Phase 2: Abstractive Fallback** (if extractive + one-liners < 0.8)
1. Generate summary example combining key patterns
2. Format: `{generalized_pattern} → {expected_result}`
3. Preserve specific values where critical

**Selection Prompt:**
```
Select the 1-2 BEST examples that cover the most patterns with minimum redundancy.

Examples:
{numbered_list}

Return JSON: {
  "selected": [indices],
  "coverage": 0.0-1.0,
  "rationale": "..."
}
```

**Synthesis Prompt** (if coverage < 0.8):
```
Compress these examples into 1-2 summary examples.
Preserve: specific values, key patterns, expected outcomes.
Format: "When X → Do Y → Expect Z"

{examples}
```

**Output shows:** `Examples: 5 → 2 full + 3 one-liners (extractive)` or `Examples: 5 → 1 (abstractive)`

### 5. Measure Functionality

**Evaluate by semantic understanding, NOT metrics.**

| Wrong (Metrics) | Right (Semantic) |
|-----------------|------------------|
| "60% line reduction is too aggressive" | "Can an agent still execute this skill?" |
| "Token count exceeds target" | "Are all triggers and actions preserved?" |
| "Ratio doesn't match threshold" | "Would an agent behave the same way?" |

LLM evaluates preservation by asking:
- Can all original triggers still activate?
- Are all core actions still specified?
- Are critical constraints preserved?
- Would an agent behave the same way?

**Score 0-100** reflects semantic capability preservation, not line/token ratios. A skill compressed to 40% of original size can still preserve 95% functionality if the removed content was verbose explanation, redundant examples, or non-essential formatting.

### 6. Save Calibration

After compression, save entry to `.learnings/skill-distiller/calibration.jsonl`:

```json
{
  "id": "c[N]",
  "timestamp": "[ISO 8601]",
  "skill": "[skill name from frontmatter]",
  "mode": "[threshold|tokens|oneliner]",
  "threshold": 0.9,
  "input_tokens": 1800,
  "output_tokens": 1100,
  "reduction_pct": 39,
  "sections_total": 14,
  "sections_kept": 9,
  "sections_removed": 5,
  "classification_confidence_mean": 0.90,
  "functionality_score": 90,
  "protected_patterns_found": ["n-count"],
  "protected_patterns_preserved": ["n-count"],
  "advisory_patterns_found": [],
  "advisory_patterns_removed": [],
  "expected": {"functionality": 90},
  "actual": null
}
```

**File rotation**: If entries > 1000, truncate oldest 100 before appending.

### 7. Output Result

Return compressed skill with metadata.

---

## Output

### Standard Output

```
Functionality preserved: 90% (uncalibrated - first 5 compressions build baseline)
Tokens: 2000 → 1800 (10% reduction)
Classification confidence: 0.87 (mean across sections)

Removed: 3 examples, 2 edge cases
Kept: all triggers, core instructions, constraints

[Compressed skill markdown follows...]
```

### Verbose Output (`--verbose`)

```
Section Analysis:
  ## When to Use: TRIGGER (1.0, confidence: 0.95) → KEPT
  ## Process: CORE_INSTRUCTION (1.0, confidence: 0.92) → KEPT
  ## Examples: EXAMPLE (0.5, confidence: 0.88) → RECOMP
    └─ Examples: 5 → 2 full + 3 one-liners (coverage: 0.95)
  ## Edge Cases: EDGE_CASE (0.4, confidence: 0.85) → SUMMARIZED
  ## Technical Details: VERBOSE_DETAIL (0.2) → TOKEN-SCORED
    └─ Tokens: 150 → 45 (redundant phrases removed)

Token-level analysis:
  VERBOSE_DETAIL sections: 3 analyzed, 67% average compression
  Phrases removed: "The system will then", "proceed to", "in order to"

RECOMP example analysis:
  Original examples: 5
  Full (selected): [0, 4] (basic usage, error handling)
  One-liners: [1, 2, 3] (threshold, tokens, verbose options)
  Coverage: 0.95 (full + one-liners)
  Mode: extractive

Protected patterns found: none
Advisory patterns found: parallel-decision → removed (no score penalty)

[Compressed skill markdown follows...]
```

### One-Liner Output

```
TRIGGER: [activation conditions]
ACTION: [core behavior]
RESULT: [expected output]
```

**Example with MetaGlyph symbols**:
```
TRIGGER: user asks "compress skill" ∨ "distill" ∨ "reduce context"
ACTION: parse → classify sections → score importance → ¬low-value → output
RESULT: compressed skill ∧ functionality% ∧ token reduction stats
```

---

## Compression Modes

### Mode 1: Threshold (Default)

Preserve X% of functionality, compress as much as possible.

```bash
/skill-distiller path/to/skill.md --threshold=0.9
```

**Understanding thresholds**: The threshold (0.9 = 90%) refers to **semantic functionality**, not token/line ratios.

| Threshold | Meaning | NOT |
|-----------|---------|-----|
| 0.95 | 95% of *capabilities* preserved | 95% of *lines* kept |
| 0.90 | 90% of *semantic function* | 90% of *tokens* |
| 0.80 | 80% of *agent behavior* | 80% of *bytes* |

A 0.9 threshold can result in 50%+ line reduction if the removed content was verbose examples, redundant explanations, or formatting. **Judge quality by semantic analysis, not size ratios.**

**Why 0.9 default**: Skill functionality is normally distributed across sections. Wide tails mean some "low importance" sections occasionally carry critical value for edge cases. At 0.9, you preserve more of the tail while still achieving meaningful compression.

### Mode 2: Token Target

Compress to exact token budget.

```bash
/skill-distiller path/to/skill.md --tokens=500
```

**Token estimation**: Uses 4 chars/token heuristic. Accuracy: +/-20% vs actual provider tokenization. For precise limits, verify with provider's tokenizer.

### Mode 3: One-Liner

Extreme compression for quick reference.

```bash
/skill-distiller path/to/skill.md --mode=oneliner
```

Produces 3-line summary: TRIGGER/ACTION/RESULT

---

## Protected Patterns

These patterns **must be preserved** even if they look verbose:

| Pattern | Why Protected |
|---------|---------------|
| YAML `name`/`description` | REQUIRED by Agent Skills spec |
| Task creation | Compaction resilience |
| N-count tracking | Observation workflow |
| Checkpoint/state | State recovery |
| BEFORE/AFTER | Self-calibration |

If a protected pattern is removed, the functionality score is penalized (-10% per pattern) and flagged explicitly in output.

---

## Advisory Patterns

These patterns improve efficiency but aren't required:

| Pattern | Impact if Removed |
|---------|-------------------|
| Parallel/serial decision | Suboptimal execution order |
| Performance hints | Slower but functional |
| Caching guidance | Works but inefficient |

Advisory patterns removed are **warned** but don't penalize the functionality score.

---

## Symbol Reference (MetaGlyph)

Native mathematical symbols LLMs understand from pre-training. Zero legend overhead.

| Symbol | Replaces | Example |
|--------|----------|---------|
| `→` | results in, leads to, produces | `trigger → action` |
| `⇒` | implies, therefore, thus | `condition ⇒ behavior` |
| `∈` | belongs to, is in, member of | `value ∈ {a, b, c}` |
| `∀` | for all, for every, for each | `∀ files: validate` |
| `¬` | not, doesn't, isn't | `¬empty → process` |
| `∃` | there exists, there is | `∃ config → load` |
| `∧` | and, also, plus | `valid ∧ safe → proceed` |
| `∨` | or, either | `error ∨ timeout → retry` |

**Application**: Symbols replace verbose phrases in compressed output. No legend needed in output—LLMs comprehend natively.

**Research**: Based on MetaGlyph (arXiv:2601.07354) - 62-81% compression with native symbols.

---

## Calibration

**Storage**: `.learnings/skill-distiller/calibration.jsonl`

Each compression run is logged with expected functionality score. User feedback updates the `actual` field, enabling calibration over time.

### Calibration Stages

| N-count | Output | Meaning |
|---------|--------|---------|
| N < 5 | `90% (uncalibrated - first 5 compressions build baseline)` | LLM-only estimate |
| N = 5-10 | `88% (building baseline, N=7)` | Gathering data |
| N > 10 | `88% +/- 4% (calibrated, N=12)` | Historical data informs CI |

### Feedback Recording

After using a compressed skill, report actual outcome to improve future estimates:

```bash
/skill-distiller feedback --id=c1 --actual=85 --outcome="worked"
```

**Outcome values**: `worked`, `partial`, `failed`

This updates the calibration entry, enabling the system to learn from real-world usage.

### Viewing Calibration Data

```bash
cat .learnings/skill-distiller/calibration.jsonl | jq -s 'length'  # Count entries
cat .learnings/skill-distiller/calibration.jsonl | jq -s 'map(select(.actual != null))' # Entries with feedback
```

---

## Self-Compression

This skill can compress itself (meta-recursive).

**Guardrails**:
- **Higher threshold**: Require 95% functionality (not 90%)
- **Recursion check**: Detect self-referential compression
- **Preserve original**: Output to SKILL.compressed.md, never overwrite
- **Manual verification**: Require human approval

**Why 0.95 threshold**: The distiller must remain fully functional to distill other skills. Capability loss compounds (0.95 x 0.95 = 0.90 at next level).

**Enforcement**: The 0.95 threshold is a documented guardrail, not an automated check. When compressing skill-distiller variants, manually verify you're using `--threshold=0.95` or higher. The formula variant (`SKILL.md`) should never be the input for self-compression.

---

## Error Handling

| Error | Recovery Hint |
|-------|---------------|
| No content | Provide a valid SKILL.md file path or pipe content via stdin |
| No frontmatter | Add `---` block with `name` and `description` |
| No trigger section | Add '## When to Use' for best results |
| Token target impossible | Use --mode=oneliner for extreme compression |
| LLM unavailable | Run 'ollama serve' for local, or set GEMINI_API_KEY |
| Already minimal | No compression possible at threshold Y |

---

## Future

Planned features not yet implemented:

| Flag | Description | Status |
|------|-------------|--------|
| `--with-ci` | Calculate confidence interval (3x LLM calls) | Planned |

---

## Related

- [LLM Compression Research](../go-pbd/docs/research/2026-04-14-llm-compression-approaches.md) - MetaGlyph, LLMLingua, and 11 other techniques surveyed
- [skill-compression-support.md](../go-pbd/docs/plans/2026-04-14-skill-compression-support.md) - CLI-based compression (Option B)
- [Agent Skills Spec](https://agentskills.io/specification) - Required fields, size constraints


path: test_integration.sh
content: #!/bin/bash
set -e

# Paths (relative to go-pbd or skills repo)
SKILLS="${SKILLS_PATH:-../skills}"
PBD_SKILLS="$SKILLS/pbd"
AGENTIC_SKILLS="$SKILLS/agentic"
TESTDATA="$SKILLS/skill-distiller/testdata"

# Check Ollama
if ! ollama list 2>/dev/null | grep -qE "(llama|mistral)"; then
  echo "SKIP: Ollama not available or no model installed"
  exit 0
fi

MODEL="${OLLAMA_MODEL:-llama3.2}"

echo "=== Integration Tests (Ollama: $MODEL) ==="
echo "Skills path: $SKILLS"
echo ""
echo "Test Categories:"
echo "  - Tests 1,3,4,7,8,9: Run without skill deployment"
echo "  - Tests 2,5,6: SKIP - require '/skill-distiller' command (post-deployment)"
echo ""

# Test 1: Section Classification (using real skill)
echo "Test 1: Section Classification"
SECTION=$(head -50 "$PBD_SKILLS/essence-distiller/SKILL.md" | grep -A3 "## When to Use")
RESULT=$(ollama run $MODEL "Classify this skill section as TRIGGER, CORE_INSTRUCTION, or EXAMPLE:
$SECTION
Respond with just the type.")
echo "  Input: essence-distiller '## When to Use' section"
echo "  Expected: TRIGGER"
echo "  Got: $RESULT"
[[ "$RESULT" == *"TRIGGER"* ]] && echo "  PASS" || echo "  FAIL"

# Test 2: Full Compression (real skill)
echo "Test 2: Full Compression (essence-distiller, threshold=0.9)"
# Note: This test validates the compression concept, not the skill invocation
# The /skill-distiller command will exist after Stage 1 completion
echo "  [SKIP: Requires skill invocation - manual test after deployment]"

# Test 3: Protected Pattern Detection (real agentic skill)
echo "Test 3: Protected Pattern Detection (failure-memory)"
if [[ -f "$AGENTIC_SKILLS/failure-memory/SKILL.md" ]]; then
  grep -E "(N=1|N=2|N≥3|N-count)" "$AGENTIC_SKILLS/failure-memory/SKILL.md" > /dev/null && \
    echo "  PASS: N-count patterns found in failure-memory" || \
    echo "  FAIL: No N-count patterns found"
else
  echo "  SKIP: failure-memory skill not found"
fi

# Test 4: Advisory Pattern Detection (real agentic skill)
echo "Test 4: Advisory Pattern Detection (workflow-tools)"
if [[ -f "$AGENTIC_SKILLS/workflow-tools/SKILL.md" ]]; then
  grep -E "(SERIAL|parallel|checkpoint)" "$AGENTIC_SKILLS/workflow-tools/SKILL.md" > /dev/null && \
    echo "  PASS: Parallel/serial patterns found in workflow-tools" || \
    echo "  FAIL: No parallel/serial patterns found"
else
  echo "  SKIP: workflow-tools skill not found"
fi

# Test 5: Calibration Persistence
echo "Test 5: Calibration Persistence"
echo "  [SKIP: Requires skill invocation - manual test after deployment]"

# Test 6: One-liner Quality
echo "Test 6: One-liner Mode"
echo "  [SKIP: Requires skill invocation - manual test after deployment]"

# Test 7: Stress Test (largest skill)
echo "Test 7: Stress Test (context-verifier, 419 lines)"
if [[ -f "$AGENTIC_SKILLS/context-verifier/SKILL.md" ]]; then
  LINES=$(wc -l < "$AGENTIC_SKILLS/context-verifier/SKILL.md")
  echo "  Found context-verifier: $LINES lines"
  [[ $LINES -ge 400 ]] && echo "  PASS: Stress test fixture available" || echo "  INFO: File smaller than expected"
else
  echo "  SKIP: context-verifier skill not found"
fi

# Test 8: Error Handling (malformed)
echo "Test 8: Error Handling (malformed)"
if [[ -f "$TESTDATA/malformed.md" ]]; then
  # Verify malformed fixture contains expected issues for error testing
  ISSUES=0
  grep -q "Missing Frontmatter" "$TESTDATA/malformed.md" && ((ISSUES++)) || true
  grep -q "Unclosed code block" "$TESTDATA/malformed.md" && ((ISSUES++)) || true
  grep -q "Broken Header" "$TESTDATA/malformed.md" && ((ISSUES++)) || true
  if [[ $ISSUES -ge 2 ]]; then
    echo "  PASS: malformed.md has $ISSUES/3 expected error patterns"
  else
    echo "  FAIL: malformed.md missing expected error patterns ($ISSUES/3)"
  fi
  # Note: Full error handling validation requires skill invocation (SKIP)
  # Manual test: /skill-distiller testdata/malformed.md should report errors
else
  echo "  FAIL: malformed.md fixture not found"
fi

# Test 9: Minimal Fixture
echo "Test 9: Minimal Fixture"
if [[ -f "$TESTDATA/minimal.md" ]]; then
  LINES=$(wc -l < "$TESTDATA/minimal.md")
  echo "  Found minimal.md: $LINES lines"
  [[ $LINES -le 20 ]] && echo "  PASS: Minimal fixture is small" || echo "  INFO: Fixture larger than expected"
else
  echo "  FAIL: minimal.md fixture not found"
fi

echo "=== Integration Tests Complete ==="
echo ""
echo "Note: Tests marked [SKIP] require skill deployment."
echo "Run full integration after Stage 1 is deployed to Claude Code."


path: testdata/malformed.md
content: # Missing Frontmatter Skill

## When to Use
- "test error handling"

## Process

```code
Unclosed code block

### Broken Header Hierarchy (### before ##)

This is intentionally malformed to test parser error handling.
- No YAML frontmatter (name, description missing)
- Unclosed code block above
- Header hierarchy violation (### without parent ##)
- Missing trigger section format


path: testdata/minimal.md
content: ---
name: minimal-test-skill
description: Bare minimum skill for boundary testing.
---

# Minimal Test Skill

## When to Use

- "test minimal"

## Process

1. Do the thing.


path: testdata/representative.md
content: ---
name: Example Skill
description: A representative skill for testing compression
version: 0.1.0
---

# Example Skill

A skill with typical structure for testing compression ratios.

## When to Use

Activate when:
- Testing skill-distiller compression
- Demonstrating realistic token reduction
- Validating protected pattern preservation

## Process

### Step 1: Setup

First, configure the environment:
- Set required variables
- Validate prerequisites
- Initialize state

### Step 2: Execution

Execute the main workflow:
1. Parse input
2. Classify sections
3. Apply compression
4. Output result

### Step 3: Verification

Verify the output:
- Check functionality preservation
- Validate protected patterns
- Confirm token reduction

## Examples

### Example 1: Basic Usage

```bash
/example-skill input.txt
```

Output:
```
Processing input.txt...
Result: success
```

### Example 2: With Options

```bash
/example-skill input.txt --verbose --output=result.txt
```

Output:
```
[VERBOSE] Parsing input.txt
[VERBOSE] Found 15 sections
[VERBOSE] Writing to result.txt
Result: success (15 sections processed)
```

### Example 3: Error Case

```bash
/example-skill nonexistent.txt
```

Output:
```
Error: File not found: nonexistent.txt
Hint: Provide a valid file path
```

## Edge Cases

### Empty Input

When input is empty, return early with warning:
```
Warning: Empty input, nothing to process
```

### Large Files

For files >10MB, process in chunks to avoid memory issues.

### Special Characters

Handle Unicode and special characters gracefully.

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--verbose` | `false` | Show detailed output |
| `--output` | stdout | Output file path |
| `--format` | `text` | Output format: text, json, yaml |

## Error Handling

| Error | Recovery |
|-------|----------|
| File not found | Check path, provide valid file |
| Invalid format | Use supported format (text/json/yaml) |
| Memory exceeded | Use smaller input or chunked mode |

## Related

- [Parent Skill](../parent/)
- [Documentation](https://example.com/docs)


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)