Skill: yuanyan/auto-spec (version 0.0.1)

--- name: auto-spec description: > Spec-driven development assistant: helps write precise behavioral specs before coding, or reverse-engineer specs from existing code for understanding and alignment. Use this skill when the user mentions "write a spec", "generate spec", "spec-driven", "behavioral contract", "reverse spec from code", "clarify this module's behavior", or "align on intent". Also applies when the user says "I want to add a feature, help me think it through first" or "what does this code actually do, walk me through it". --- # Auto-Spec: Spec-Driven Development Assistant ## Core Belief > **Code is the single source of truth.** Specs are **inputs** — they help humans align intent and assist AI coding — not deliverables. Specs don't need to stay in sync with code permanently; regenerate from code when needed. Three hard constraints follow: 1. **No bidirectional binding, no CI drift detection** — that's life support for dead docs, expensive and fragile. 2. **Specs are disposable** — once landed, mission complete; code and tests take over as truth. 3. **To see current state, reverse-engineer from code** — don't maintain a static doc that can't keep up. Therefore, specs generated by this skill **are presented in the conversation by default, not written to files**. Only persist to disk when the user explicitly asks, and treat it as a temporary snapshot, not a long-lived document. --- ## Two Modes ### Mode A: Forward Spec (Spec before code) The user has an idea or requirement but hasn't written code yet. Turn vague intent into precise behavioral contracts, then (if requested) implement according to the spec. **Typical triggers**: - "I want to add feature X, help me think it through first" - "Write me a spec" - "Align on intent before coding" ### Mode B: Reverse Spec (Spec from code) Code already exists. The user wants to understand what it does, or wants to establish a baseline before making changes. Extract behavioral contracts from the code. **Typical triggers**: - "What does this code actually do, walk me through it" - "Reverse-engineer a spec from the code" - "Help me understand this module's behavior" --- ## Spec Format A spec is not prose — it's a structured behavioral contract. Granularity is chosen automatically based on scope: ### Granularity Rules | User's scope | Granularity | Example | |---|---|---| | A single function/method | Function-level | Behavior of `CalculateQuota` | | A single file/class | Module-level | Responsibilities and interface of `quota_service.go` | | A feature/capability | Feature-level | Complete behavior of "quota management" | | A system/service | System-level | Overall behavior of "order system" | Don't ask the user to specify granularity — infer from context. When uncertain, go one level higher than what the user described, then drill down into key parts within the spec. ### Spec Template Each spec contains the following sections (trim as needed — don't pad with empty content just to fill the template): ```markdown # [Name] Spec ## Overview One or two sentences explaining what this is and why it exists. ## Glossary Only needed when domain-specific terms or easily confused concepts are present. - **TermA**: definition ## Behavioral Contracts ### Scenario 1: [Scenario Name] - **Precondition**: what state the system is in - **Input**: what triggers this behavior - **Expected behavior**: what the system should do - **Postcondition**: what state the system should be in afterward - **Error cases**: what happens if something goes wrong ### Scenario 2: ... ## Constraints & Boundaries - Performance constraints, security constraints, business rules, etc. - Explicitly stating "what we don't do" is equally important ## Dependencies - What external systems/modules/interfaces this depends on - Who depends on this ## Open Questions Items that need further confirmation (if any). ``` ### Writing Principles 1. **Precision over completeness**: one clear contract beats ten vague descriptions. 2. **Show examples**: abstract rule + concrete example = best understanding. Give input→output examples for key behaviors. 3. **Mark uncertainty**: use `[TBD]` for things you're not sure about — don't fabricate. 4. **Behavior, not implementation**: describe *what*, not *how*. Implementation details belong in code. 5. **Error paths matter as much as happy paths**: many bugs hide in scenarios that weren't thought through. --- ## Workflow ### Forward Spec Workflow 1. **Understand intent** - Read the user's requirement description - If the requirement is vague, ask targeted questions (no more than 3 at a time) - Focus on: edge cases, error scenarios, interactions with existing systems 2. **Research context** - Read relevant code to understand the existing system's structure and conventions - Find the "integration point" for the new feature — which existing modules will it interact with? - Note existing patterns and conventions; the spec should be consistent with them 3. **Generate spec** - Generate the spec using the template, present it directly in the conversation - Choose granularity automatically - Give concrete examples for key behaviors 4. **Iterate and confirm** - Ask the user to review the spec - Revise based on feedback until the user is satisfied - Mark all `[TBD]` items 5. **(Optional) Implement according to spec** - After the user confirms, implement if requested - Reference scenario numbers from the spec in code comments to maintain traceability ### Reverse Spec Workflow 1. **Locate the code** - Confirm which code the user wants to reverse-engineer - Read the relevant source files 2. **Extract behavior** - Extract actual behavior from code, not guessed intent - Distinguish "what the code actually does" from "what the code might have intended to do" - Flag suspicious behavior (things that look like bugs or legacy logic) 3. **Generate spec** - Generate the spec using the template, present it directly in the conversation - Use `[NOTE]` to flag anything that doesn't match expectations 4. **Deliver** - Present the spec, call out points worth attention - If the user plans to modify the code afterward, this spec serves as the pre-change baseline --- ## Important Notes - **Don't over-engineer specs**: a spec is a thinking tool, not a bureaucratic process. If a function's behavior can be explained in one sentence, use one sentence. - **Don't proactively save files**: unless the user explicitly asks, specs are presented in the conversation only. - **Don't repeat what the code already expresses clearly**: if the code itself is the best documentation (e.g., clear type signatures, well-named functions), the spec should focus on what code can't express — intent, constraints, edge cases. - **Stay honest**: say "I'm not sure" when you're not sure. Don't fabricate plausible-sounding but unverified contracts.

path: README.md
content: <img width="1738" height="905" alt="Auto-Spec" src="https://github.com/user-attachments/assets/f16b0339-feea-4d38-8ac0-a52774948429" />

# Auto Spec

An Agent skill for **Spec-Driven Development** with disposable specs.

Write specs before coding, or extract them from existing code when needed.

## 🎯 When to Use

Trigger this skill when you want to:
- **Design before coding**: *"I want to add feature X, help me think it through first"* or *"Write me a spec for..."*
- **Understand existing code**: *"What does this code actually do, walk me through it"* or *"Reverse-engineer a spec from this module"*
- **Align on intent**: Ensure you and the AI are on the same page before it generates complex logic.

## 💡 Philosophy

> **Code is the single source of truth.**

Specs are **inputs** — they help humans align intent and assist AI coding — not deliverables. Specs don't need to stay in sync with code permanently; regenerate from code when needed.

Three hard constraints follow:
1. **No bidirectional binding, no CI drift detection** — that's life support for dead docs, expensive and fragile.
2. **Specs are disposable** — once landed, mission complete; code and tests take over as truth.
3. **To see current state, reverse-engineer from code** — don't maintain a static doc that can't keep up.

*Note: By default, Auto Spec presents specs directly in the conversation rather than writing them to files, treating them as temporary conversational snapshots.*

## 🆚 Auto Spec vs. Spec-kit / OpenSpec

While traditional spec tools like **Spec-kit** or **OpenSpec** treat specs as formal, long-lived documentation, **Auto Spec** takes a fundamentally different, lightweight approach optimized for AI-assisted coding:

| Feature | Auto Spec | Spec-kit / OpenSpec |
| :--- | :--- | :--- |
| **Source of Truth** | **Code only.** The spec is just an input for alignment. | Treats the spec as a parallel or primary source of truth. |
| **Lifecycle** | **Disposable.** Read, align, code, then discard. | **Persistent.** Maintained alongside the codebase. |
| **Sync Mechanism** | **None.** Regenerate from code dynamically when needed. | **Bidirectional binding** or strict CI drift detection. |
| **Format & Output** | **Conversational.** Focused on human-AI alignment (Markdown). | **Strict schemas** (JSON/YAML/XML) or formal documentation. |
| **Maintenance Cost**| **Zero.** No "dead docs" to maintain. | **High.** Requires constant updates to prevent drift. |

## 🔄 Two Modes

### Mode A: Forward Spec (Spec before code)

User has an idea but no code yet. Turn vague intent into precise behavioral contracts, then (optionally) implement according to the spec.
- Understands intent and asks targeted questions for edge cases.
- Generates a structured spec based on existing code context.
- Iterates with you until alignment is reached, then implements the code if requested.

### Mode B: Reverse Spec (Spec from code)

Code already exists. Extract behavioral contracts from code to understand what it actually does, or to establish a baseline before modifications.
- Extracts factual behavior, not guessed intent.
- Flags suspicious patterns, potential bugs, or legacy logic.
- Serves as a pre-change baseline for safe refactoring.

## 📝 Spec Format & Granularity

Auto Spec automatically adjusts the granularity of the spec based on your scope (Function, Module, Feature, or System level). 

A generated spec focuses on **Behavioral Contracts**, typically including:
- **Scenarios**: Preconditions, Inputs, Expected behavior, Postconditions, and Error cases.
- **Constraints & Boundaries**: Performance, security, business rules, and what the system *doesn't* do.
- **Dependencies**: External systems, interfaces, and dependents.

**Key Writing Principles:**
- **Precision over completeness**: One clear contract beats ten vague descriptions.
- **Behavior, not implementation**: Describes *what*, not *how*.
- **Uncertainty marking**: Uses `[TBD]` or `[NOTE]` tags instead of false confidence.

## 📊 Eval Results

Evaluated against 3 test cases covering both modes and multiple granularity levels:

| Test Case | With Skill | Baseline | Delta |
|---|---|---|---|
| Forward Spec (feature-level) | **7/7 (100%)** | 4/7 (57%) | +43% |
| Reverse Spec (module-level) | **7/7 (100%)** | 5/7 (71%) | +29% |
| Forward Spec + Code (function-level) | **7/7 (100%)** | 6/7 (86%) | +14% |
| **Total** | **21/21 (100%)** | **15/21 (71%)** | **+29%** |

Key improvements with the skill:
- **Structured behavioral scenarios** (preconditions/inputs/expected behavior/postconditions/exceptions)
- **Uncertainty marking** with `[TBD]` / `[NOTE]` tags instead of false confidence
- **Behavior-oriented** (what, not how) — avoids leaking implementation details into specs
- **Insight discovery** in reverse specs — flags suspicious code patterns

Full eval outputs are in the `evals/` directory.

## 📄 License

MIT


path: evals/evals.json
content: {
  "skill_name": "auto-spec",
  "evals": [
    {
      "id": 1,
      "prompt": "I want to add a 'conversation history summarization' feature to the chat assistant: when the number of conversation turns exceeds 10, automatically compress earlier history into a summary to reduce token usage. Write me a spec first.",
      "expected_output": "A feature-level Forward Spec with behavioral contracts (happy path + error scenarios), constraints & boundaries, and dependencies — not written to any file",
      "files": []
    },
    {
      "id": 2,
      "prompt": "Reverse-engineer a spec for internal/orchestrator/supervisor.go — I want to understand what this supervisor actually does.",
      "expected_output": "A module-level Reverse Spec extracted from the code, with suspicious logic flagged using [NOTE] tags — not written to any file",
      "files": []
    },
    {
      "id": 3,
      "prompt": "I need a utility function that deduplicates a list of user-uploaded image URLs and validates their accessibility. Invalid URLs should be filtered out and logged. Write the spec first, then implement it.",
      "expected_output": "A function-level Forward Spec, followed by an implementation that follows the spec",
      "files": []
    }
  ]
}


path: evals/iteration-1/benchmark.json
content: {
  "skill_name": "auto-spec",
  "iteration": 1,
  "configurations": [
    {"name": "with_skill", "label": "Auto-Spec Skill"},
    {"name": "without_skill", "label": "Baseline (No Skill)"}
  ],
  "evals": [
    {
      "eval_name": "forward-spec-feature-level",
      "results": {
        "with_skill": {
          "pass_rate": 1.0,
          "passed": 7,
          "total": 7
        },
        "without_skill": {
          "pass_rate": 0.571,
          "passed": 4,
          "total": 7
        }
      }
    },
    {
      "eval_name": "reverse-spec-module-level",
      "results": {
        "with_skill": {
          "pass_rate": 1.0,
          "passed": 7,
          "total": 7
        },
        "without_skill": {
          "pass_rate": 0.714,
          "passed": 5,
          "total": 7
        }
      }
    },
    {
      "eval_name": "forward-spec-then-code",
      "results": {
        "with_skill": {
          "pass_rate": 1.0,
          "passed": 7,
          "total": 7
        },
        "without_skill": {
          "pass_rate": 0.857,
          "passed": 6,
          "total": 7
        }
      }
    }
  ],
  "summary": {
    "with_skill": {
      "overall_pass_rate": 1.0,
      "total_passed": 21,
      "total_assertions": 21
    },
    "without_skill": {
      "overall_pass_rate": 0.714,
      "total_passed": 15,
      "total_assertions": 21
    },
    "delta": "+28.6%"
  }
}


path: evals/iteration-1/forward-spec-feature-level/eval_metadata.json
content: {
  "eval_id": 1,
  "eval_name": "forward-spec-feature-level",
  "prompt": "I want to add a 'conversation history summarization' feature to the chat assistant: when the number of conversation turns exceeds 10, automatically compress earlier history into a summary to reduce token usage. Write me a spec first.",
  "assertions": [
    {"text": "Spec contains structured behavioral scenarios (precondition / input / expected behavior / postcondition / error cases)", "type": "qualitative"},
    {"text": "Spec covers error paths (e.g., fallback strategy when the summarization LLM call fails)", "type": "qualitative"},
    {"text": "Spec includes concrete input→output examples", "type": "qualitative"},
    {"text": "Spec researches existing code and references real module/function names", "type": "qualitative"},
    {"text": "Spec marks uncertain open questions with [TBD] tags", "type": "qualitative"},
    {"text": "Spec does not write any files to the project source directory", "type": "qualitative"},
    {"text": "Spec is behavior-oriented, not implementation-oriented (describes what, not how)", "type": "qualitative"}
  ]
}


path: evals/iteration-1/forward-spec-feature-level/with_skill/grading.json
content: {
  "eval_name": "forward-spec-feature-level",
  "configuration": "with_skill",
  "expectations": [
    {"text": "Spec contains structured behavioral scenarios (precondition / input / expected behavior / postcondition / error cases)", "passed": true, "evidence": "Contains 6 structured scenarios, each with precondition / input / expected behavior / postcondition / error cases"},
    {"text": "Spec covers error paths (e.g., fallback strategy when the summarization LLM call fails)", "passed": true, "evidence": "Scenario 2 explicitly describes fallback strategies for LLM call failure and timeout"},
    {"text": "Spec includes concrete input→output examples", "passed": true, "evidence": "Scenario 2 includes a full before/after comparison of the message sequence compression"},
    {"text": "Spec researches existing code and references real module/function names", "passed": true, "evidence": "References real function names such as StoreHistoryMessages and GetHistoryMessages"},
    {"text": "Spec marks uncertain open questions with [TBD] tags", "passed": true, "evidence": "Contains 5 open questions marked with [TBD]"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec is behavior-oriented, not implementation-oriented (describes what, not how)", "passed": true, "evidence": "Behavioral contracts focus on scenarios and expected behavior, with no pseudocode or implementation details"}
  ]
}


path: evals/iteration-1/forward-spec-feature-level/with_skill/outputs/spec_output.md
content: # Conversation History Summarization Spec

## Overview

When the chat assistant's session exceeds 10 conversation turns, automatically compress earlier history into a summary message, replacing the original message sequence. This reduces token usage for subsequent LLM calls while preserving enough context to prevent agent behavior degradation.

## Glossary

- **Conversation turn (Round)**: one user input + the agent's complete response (including all tool calls and their results) constitutes one turn.
- **History messages**: the message sequence persisted in the storage layer, read/written via `StoreHistoryMessages` / `GetHistoryMessages`.
- **Summary message**: a compressed text generated by an LLM, injected as a System-role message to replace the compressed original messages.
- **Retention window**: the most recent N turns of messages are excluded from compression and always kept in full.
- **Anchor messages**: system-level messages (e.g., user profile, environment context) that are never compressed and always retained.

## Behavioral Contracts

### Scenario 1: Turn count has not exceeded the threshold

- **Precondition**: the current session has ≤ 10 turns (default threshold).
- **Input**: user sends a new message, triggering the conversation flow.
- **Expected behavior**: no summarization is performed; history messages are passed to the agent unchanged, identical to current system behavior.
- **Postcondition**: message sequence is unchanged.

### Scenario 2: Turn count first exceeds the threshold — summarization triggered

- **Precondition**: the current session has > 10 turns, and no summary has been generated yet (or enough new turns have accumulated since the last summary).
- **Input**: user sends a new message; the message-building stage fetches history messages.
- **Expected behavior**:
  1. Count the number of conversation turns in history (by counting User-role messages).
  2. If turns > threshold, split the message sequence into two parts:
     - **Compression zone**: the first `(total turns − retention window)` turns.
     - **Retention zone**: the most recent `retention window` turns (default: 5), kept in full.
  3. Extract all non-anchor messages from the compression zone and call an LLM to generate a summary.
  4. Replace all non-anchor messages in the compression zone with a single System-role summary message.
  5. Final message sequence = anchor messages + summary message + retention zone messages + current user input.
- **Postcondition**:
  - The number of messages passed to the agent is significantly reduced.
  - The summarized message sequence is persisted back to the storage layer.
- **Error cases**:
  - Summarization LLM call fails: fall back to no compression, use full history, log the error, do not affect the main flow.
  - Summarization LLM call times out: set an independent timeout (recommended: 15s); fall back on timeout.

**Example**:

```
Original message sequence (12 turns, retention window = 5):
  [System: user profile]
  [System: environment context]
  [User: turn 1 input] [Assistant: turn 1 reply] [Assistant: tool_call] [Tool: tool result]
  [User: turn 2 input] [Assistant: turn 2 reply]
  ...
  [User: turn 7 input] [Assistant: turn 7 reply]  ← end of compression zone
  [User: turn 8 input] [Assistant: turn 8 reply]  ← start of retention zone
  ...
  [User: turn 12 input] [Assistant: turn 12 reply]

After compression:
  [System: user profile]                          ← anchor message, untouched
  [System: environment context]                   ← anchor message, untouched
  [System: Summary of earlier conversation: ...]  ← new summary message
  [User: turn 8 input] [Assistant: turn 8 reply]  ← retention zone, kept as-is
  ...
  [User: turn 12 input] [Assistant: turn 12 reply]
```

### Scenario 3: Conversation continues after an existing summary

- **Precondition**: history already contains a summary message; the user continues the conversation and the retention zone accumulates enough new turns to exceed the threshold again.
- **Input**: user sends a new message.
- **Expected behavior**:
  1. Identify the existing summary message.
  2. Feed the old summary + messages after the old summary but before the new retention window into the LLM to generate a merged summary.
  3. Replace the old summary + compressed messages with the new summary.
- **Postcondition**: there is always exactly one summary message in the message sequence.

### Scenario 4: Interaction with context cache

- **Precondition**: the current agent uses a session-level context cache.
- **Input**: summarization changes the message sequence.
- **Expected behavior**:
  1. After replacing messages, invalidate the context cache.
  2. The next LLM call rebuilds the cache from the updated message sequence.
- **Postcondition**: cache is consistent with the message sequence.

### Scenario 5: Interaction with middleware

- **Precondition**: summarization occurs during the message-building stage.
- **Input**: the pre-chat middleware needs to locate profile messages and sync tool messages.
- **Expected behavior**:
  1. Summarization does not compress anchor messages, so the pre-chat middleware's lookup logic is unaffected.
  2. Compressed tool messages no longer appear in the message sequence.
  3. The post-chat middleware persists the final message sequence (including the summary message) as usual.
- **Postcondition**: middleware behavior does not degrade.

### Scenario 6: Resource references in the summary

- **Precondition**: messages in the compression zone contain resource references (e.g., image variable names like `img_001.jpg`).
- **Input**: the summarization LLM processes messages containing resource references.
- **Expected behavior**:
  1. The summarization prompt instructs the LLM to preserve all resource variable names.
  2. The summary text records referenced resources in a list: "Resources referenced in earlier history: img_001.jpg, img_002.jpg, ..."
  3. The storage mapping from variable names to real URIs is unaffected and always available.
- **Postcondition**: the agent can still reference historical resources by variable name.
- **Error cases**: if the summarization LLM drops a resource variable name, the agent may be unable to reference that resource. This is an acceptable degradation — the user can re-upload.

## Constraints & Boundaries

### Performance
- The summarization LLM call should use a lightweight model to control cost.
- Summarization timeout cap: 15 seconds; fall back to no compression on timeout.
- Summarization must not block user-perceived response time — it should complete synchronously during the message-building stage.

### Business rules
- **Configurable thresholds**: the turn threshold (default: 10) and retention window (default: 5) should be delivered via a config center and adjustable at runtime.
- **Feature flag**: provide a global on/off switch via the config center.
- **No incremental summarization**: each trigger performs a full re-summarization of the compression zone — no incremental appending (keeps implementation simple and avoids quality degradation). [TBD: is incremental summarization needed to further reduce token usage?]

### What we don't do
- **No client protocol changes**: summarization is purely server-side; the client is unaware.
- **No impact on raw message storage**: the complete record of original user messages is unchanged; summarization only affects the history messages fed to the LLM.
- **No partial-turn summarization**: all tool calls and results within a single turn are treated as a unit — either fully retained or fully compressed.
- **No real-time token counting**: summarization is triggered by turn count, not exact token count, for simplicity. [TBD: is a token-count-based trigger strategy needed?]

## Dependencies

### Depends on
- **LLM service**: generates the summary text; requires a dedicated model endpoint.
- **Persistent storage**: reads/writes history messages; existing interface is sufficient.
- **Config center**: delivers threshold, retention window, and feature flag.

### Depended on by
- **Conversation main flow**: calls summarization logic during the message-building stage.
- **Agent middleware**: pre/post middleware must be compatible with the presence of a summary message.

## Open Questions

1. **[TBD] Summarization prompt design**: the prompt needs careful design to preserve key information (user preferences, completed actions, resource references).
2. **[TBD] Trigger strategy**: the current design triggers by turn count — should token count also be considered? Some turns with many tool calls may consume far more tokens than expected.
3. **[TBD] Summary quality evaluation**: how do we verify that summarization doesn't degrade agent behavior? Recommend A/B testing with long-conversation cases.
4. **[TBD] Incremental vs. full re-summarization**: the current design re-summarizes the full compression zone each time. For very long sessions (50+ turns), the summarization itself may consume significant tokens. Is an incremental strategy needed?
5. **[TBD] Integration point**: should summarization logic live in the message-building stage or in the pre-chat-model middleware? The former runs earlier; the latter has access to the full message state.


path: evals/iteration-1/forward-spec-feature-level/without_skill/grading.json
content: {
  "eval_name": "forward-spec-feature-level",
  "configuration": "without_skill",
  "expectations": [
    {"text": "Spec contains structured behavioral scenarios (precondition / input / expected behavior / postcondition / error cases)", "passed": false, "evidence": "Uses a traditional design doc format (overview / architecture analysis / detailed design / code change plan) — no structured behavioral scenarios"},
    {"text": "Spec covers error paths (e.g., fallback strategy when the summarization LLM call fails)", "passed": true, "evidence": "The risk & fallback section covers summarization failure and timeout scenarios"},
    {"text": "Spec includes concrete input→output examples", "passed": true, "evidence": "Includes a before/after example of message replacement"},
    {"text": "Spec researches existing code and references real module/function names", "passed": true, "evidence": "References real function names and file paths"},
    {"text": "Spec marks uncertain open questions with [TBD] tags", "passed": false, "evidence": "No [TBD] markers used — all content is presented with false certainty"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec is behavior-oriented, not implementation-oriented (describes what, not how)", "passed": false, "evidence": "Contains heavy implementation details: pseudocode, code change plan, config JSON, Go struct definitions"}
  ]
}


path: evals/iteration-1/forward-spec-feature-level/without_skill/outputs/spec_output.md
content: # Conversation History Summarization — Design Spec

## 1. Overview

### 1.1 Background

The chat assistant agent serializes the full history message sequence on every turn and loads it in full as LLM input context on the next turn. As the number of turns grows, token usage grows linearly, causing:

1. **Rising LLM inference cost**: each request carries more tokens.
2. **Increased latency**: longer context means longer time-to-first-token and total inference time.
3. **Context window overflow risk**: when total messages approach the model's context window limit, truncation may occur.
4. **Storage pressure**: the serialized message payload keeps growing.

### 1.2 Goal

When the number of conversation turns exceeds **10** (configurable), automatically compress earlier history into a text summary, replacing the original message sequence.

### 1.3 Non-goals

- No changes to frontend display logic or message delivery protocol
- No modification to raw message storage
- No impact on other message streams

---

## 2. Architecture Analysis

### 2.1 Message flow overview

```
User request
  │
  ▼
Conversation entry (service layer)
  │
  ├─ Parse user message
  ├─ Build agent input
  │    ├─ Load history messages from storage
  │    ├─ Inject system messages
  │    └─ Append current user message
  ├─ Build agent context
  │
  ▼
Agent inference
  │
  ├─ Pre-chat middleware: update profile, sync tool messages
  ├─ LLM inference
  └─ Post-chat middleware: moderation, persist messages
```

---

## 3. Detailed Design

### 3.1 Approach

Introduce summarization logic that checks turn count before LLM inference and triggers compression when the threshold is exceeded.

### 3.2 Turn counting

Count the number of messages where `role == User`.

### 3.3 Summarization strategy

- **Retain the most recent K turns** (default K=5) in full — no compression
- **Compress the first N-K turns** into a summary text
- System-level messages are always retained and never compressed

### 3.4 Summary generation

Use a lightweight LLM to generate the summary:

```go
type SummaryConfig struct {
    TriggerThreshold   int    `json:"trigger_threshold"`
    RetainRecentRounds int    `json:"retain_recent_rounds"`
    SummaryModel       string `json:"summary_model"`
    SummaryMaxTokens   int    `json:"summary_max_tokens"`
    Enabled            bool   `json:"enabled"`
}
```

### 3.5 Message replacement

```
Before:
  [System(profile)] [System(env)] [User1] [Assist1] [Tool1] [Assist1']
  [User2] [Assist2] ... [User10] [Assist10] [User11(current)]

After:
  [System(profile)] [System(env)] [System(summary: first 6 turns compressed)]
  [User7] [Assist7] [Tool7] [Assist7'] ... [User11(current)]
```

---

## 4. Code Change Plan

### 4.1 New files

| Path | Description |
|------|-------------|
| `internal/summarizer/summarizer.go` | Core summarization logic |
| `internal/summarizer/config.go` | Config struct definition |
| `internal/summarizer/prompt.go` | Summarization prompt template |
| `internal/summarizer/summarizer_test.go` | Unit tests |

### 4.2 Modified files

| Path | Change |
|------|--------|
| `internal/middleware/before_chat.go` | Call summarization logic |
| `internal/context/model.go` | Add `SummaryVersion` field |

---

## 5. Core pseudocode

```go
func SummarizeHistoryIfNeeded(ctx context.Context, state *AgentState) error {
    config := GetSummaryConfig(ctx)
    if !config.Enabled { return nil }

    rounds := countUserRounds(state.Messages)
    if rounds <= config.TriggerThreshold { return nil }

    splitIndex := findSplitIndex(state.Messages, config.RetainRecentRounds)
    msgsToSummarize := extractNonSystemMessages(state.Messages[:splitIndex])

    summaryText, err := generateSummary(ctx, config, msgsToSummarize)
    if err != nil {
        log.Warn(ctx, "generate summary failed, skip: %v", err)
        return nil
    }

    summaryMsg := &Message{Role: System, Content: "[SUMMARY]" + summaryText}
    state.Messages = append(systemMsgs, summaryMsg, state.Messages[splitIndex:]...)
    return nil
}
```

---

## 6. Risk & Fallback

| Risk | Mitigation |
|------|------------|
| Summary loses critical context | Retain the most recent 5 turns in full |
| Summarization LLM call fails | Skip summarization on failure, use full history |
| Cache conflict | Invalidate cache after summarization |

---

## 7. Token usage estimate

| Turns | Without summary | With summary | Savings |
|-------|----------------|--------------|---------|
| 10    | ~6,500         | ~6,500       | 0%      |
| 15    | ~9,750         | ~4,250       | ~56%    |
| 20    | ~13,000        | ~4,250       | ~67%    |
| 30    | ~19,500        | ~4,250       | ~78%    |


path: evals/iteration-1/forward-spec-then-code/eval_metadata.json
content: {
  "eval_id": 3,
  "eval_name": "forward-spec-then-code",
  "prompt": "I need a utility function that deduplicates a list of user-uploaded image URLs and validates their accessibility. Invalid URLs should be filtered out and logged. Write the spec first, then implement it.",
  "assertions": [
    {"text": "Spec is generated before the implementation (spec-driven development flow)", "type": "qualitative"},
    {"text": "Spec includes a function signature definition", "type": "qualitative"},
    {"text": "Spec covers edge cases (empty list, all duplicates, malformed URLs, network timeout)", "type": "qualitative"},
    {"text": "Implementation follows the behavior defined in the spec", "type": "qualitative"},
    {"text": "Implementation follows project conventions", "type": "qualitative"},
    {"text": "Spec does not write any files to the project source directory", "type": "qualitative"},
    {"text": "Spec stays at function granularity (does not inflate into a module-level design doc)", "type": "qualitative"}
  ]
}


path: evals/iteration-1/forward-spec-then-code/with_skill/grading.json
content: {
  "eval_name": "forward-spec-then-code",
  "configuration": "with_skill",
  "expectations": [
    {"text": "Spec is generated before the implementation (spec-driven development flow)", "passed": true, "evidence": "Separate spec_output.md and implementation.go produced, spec comes first"},
    {"text": "Spec includes a function signature definition", "passed": true, "evidence": "Spec ends with a complete Go function signature"},
    {"text": "Spec covers edge cases (empty list, all duplicates, malformed URLs, network timeout)", "passed": true, "evidence": "6 scenarios covering all edge cases"},
    {"text": "Implementation follows the behavior defined in the spec", "passed": true, "evidence": "Every branch in the implementation corresponds to a scenario in the spec"},
    {"text": "Implementation follows project conventions", "passed": true, "evidence": "package utils, uses logging library, ctx as first parameter"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec stays at function granularity (does not inflate into a module-level design doc)", "passed": true, "evidence": "Spec focuses on a single function's behavior, ~100 lines"}
  ]
}


path: evals/iteration-1/forward-spec-then-code/with_skill/outputs/implementation.go
content: package utils

import (
	"context"
	"log"
	"net/http"
	"net/url"
	"strings"
	"time"
)

func DeduplicateAndValidateImageURLs(ctx context.Context, urls []string) []string {
	if len(urls) == 0 {
		return []string{}
	}

	seen := make(map[string]struct{}, len(urls))
	deduped := make([]string, 0, len(urls))
	for _, rawURL := range urls {
		trimmed := strings.TrimSpace(rawURL)
		if trimmed == "" {
			continue
		}
		if _, exists := seen[trimmed]; exists {
			continue
		}
		seen[trimmed] = struct{}{}
		deduped = append(deduped, trimmed)
	}

	if len(deduped) == 0 {
		return []string{}
	}

	client := &http.Client{
		Timeout: 5 * time.Second,
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			return http.ErrUseLastResponse
		},
	}

	valid := make([]string, 0, len(deduped))
	for _, u := range deduped {
		parsed, err := url.Parse(u)
		if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") {
			log.Printf("[DeduplicateAndValidateImageURLs] invalid URL format, filtered: %s, err: %v", u, err)
			continue
		}

		resp, err := client.Head(u)
		if err != nil {
			log.Printf("[DeduplicateAndValidateImageURLs] HEAD request failed, filtered: %s, err: %s", u, err.Error())
			continue
		}
		resp.Body.Close()

		if resp.StatusCode < 200 || resp.StatusCode >= 300 {
			log.Printf("[DeduplicateAndValidateImageURLs] non-2xx status %d, filtered: %s", resp.StatusCode, u)
			continue
		}

		valid = append(valid, u)
	}

	return valid
}


path: evals/iteration-1/forward-spec-then-code/with_skill/outputs/spec_output.md
content: # DeduplicateAndValidateImageURLs Spec

## Overview

A utility function that accepts a list of user-uploaded image URLs, deduplicates them, validates their accessibility, and returns the deduplicated list of accessible URLs. Invalid URLs are filtered out and logged.

## Glossary

- **Deduplication**: exact-string deduplication of full URLs, preserving the order of first occurrence.
- **Accessibility validation**: sending an HTTP HEAD request to the URL and checking whether the response status code is 2xx.
- **Invalid URL**: a URL that is malformed (cannot be parsed) or fails accessibility validation (network error or non-2xx status code).

## Behavioral Contracts

### Scenario 1: Normal deduplication and validation

- **Precondition**: none
- **Input**: `ctx context.Context`, `urls []string` (list of user-uploaded image URLs)
- **Expected behavior**:
  1. Deduplicate `urls` in order of appearance (keep the first occurrence, discard subsequent duplicates)
  2. Validate the format of each deduplicated URL (`net/url.Parse` and scheme must be http/https)
  3. Send an HTTP HEAD request (5-second timeout) to each format-valid URL and check whether the status code is 2xx
  4. Return all URLs that pass validation (preserving original order)
- **Postcondition**: the returned list has no duplicates and every URL is accessible
- **Example**:
  ```
  Input:  ["https://a.com/1.jpg", "https://b.com/2.jpg", "https://a.com/1.jpg", "https://c.com/3.jpg"]
  Assume https://c.com/3.jpg returns 404
  Output: ["https://a.com/1.jpg", "https://b.com/2.jpg"]
  Log:    records that https://c.com/3.jpg failed validation
  ```

### Scenario 2: Empty list input

- **Precondition**: none
- **Input**: `urls` is `nil` or an empty slice
- **Expected behavior**: return an empty slice immediately without making any network requests
- **Postcondition**: returns `[]string{}` (non-nil empty slice)

### Scenario 3: All duplicates

- **Precondition**: none
- **Input**: all elements in `urls` are identical, e.g. `["https://a.com/1.jpg", "https://a.com/1.jpg"]`
- **Expected behavior**: after deduplication, only one URL remains; validate it
- **Postcondition**: returns a slice with 0 or 1 element

### Scenario 4: Malformed URL

- **Precondition**: none
- **Input**: `urls` contains malformed strings, e.g. `"not-a-url"`, `"ftp://x.com/a.jpg"`, `""`
- **Expected behavior**:
  1. Malformed URLs are filtered out immediately without making a network request
  2. Log the filtered URL and reason via `log.Warn`
- **Postcondition**: the returned list contains no malformed URLs

### Scenario 5: Network timeout or request failure

- **Precondition**: the target server is unreachable or the response times out
- **Input**: a list containing unreachable URLs
- **Expected behavior**:
  1. A HEAD request that times out after 5 seconds is treated as inaccessible
  2. Log the failed URL and error message via `log.Warn`
  3. Does not affect validation of other URLs (each URL is validated independently)
- **Postcondition**: only URLs that pass validation are returned

### Scenario 6: Whitespace in URLs

- **Precondition**: none
- **Input**: URLs may have leading or trailing whitespace, e.g. `" https://a.com/1.jpg "`
- **Expected behavior**: apply `strings.TrimSpace` to each URL before deduplication and validation
- **Postcondition**: returned URLs have no leading or trailing whitespace

## Constraints & Boundaries

- **Performance**: single HEAD request timeout is 5 seconds; the function does not validate concurrently (sequential) — callers can wrap it in goroutines if concurrency is needed.
- **What we don't do**:
  - Does not download image content — only validates accessibility (HEAD request)
  - Does not normalize URLs (e.g., case normalization, query parameter sorting) — deduplication is exact string matching after TrimSpace
  - Does not retry failed requests

## Dependencies

- `context`: standard library, for passing context
- `net/http`: standard library, for HEAD requests
- `net/url`: standard library, for URL format validation
- Project logging library: for recording invalid URLs

## Function Signature

```go
func DeduplicateAndValidateImageURLs(ctx context.Context, urls []string) []string
```

## Open Questions

None.


path: evals/iteration-1/forward-spec-then-code/without_skill/grading.json
content: {
  "eval_name": "forward-spec-then-code",
  "configuration": "without_skill",
  "expectations": [
    {"text": "Spec is generated before the implementation (spec-driven development flow)", "passed": true, "evidence": "Separate spec_output.md and implementation.go produced"},
    {"text": "Spec includes a function signature definition", "passed": true, "evidence": "Complete function signature included"},
    {"text": "Spec covers edge cases (empty list, all duplicates, malformed URLs, network timeout)", "passed": true, "evidence": "Edge case table covers these scenarios"},
    {"text": "Implementation follows the behavior defined in the spec", "passed": true, "evidence": "Implementation includes concurrent validation, format checking, and other behaviors defined in the spec"},
    {"text": "Implementation follows project conventions", "passed": true, "evidence": "package utils, uses logging library, ctx as first parameter"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec stays at function granularity (does not inflate into a module-level design doc)", "passed": false, "evidence": "Spec includes concurrency design, test points, and file placement suggestions — exceeds the scope of a pure behavioral contract"}
  ]
}


path: evals/iteration-1/forward-spec-then-code/without_skill/outputs/implementation.go
content: package utils

import (
	"context"
	"log"
	"net/http"
	"net/url"
	"strings"
	"sync"
	"time"
)

type urlCheckResult struct {
	index int
	url   string
	valid bool
}

func DeduplicateAndValidateImageURLs(
	ctx context.Context,
	urls []string,
	timeout time.Duration,
) (validURLs []string, invalidURLs []string) {
	if len(urls) == 0 {
		return nil, nil
	}

	seen := make(map[string]struct{}, len(urls))
	uniqueURLs := make([]string, 0, len(urls))

	for _, rawURL := range urls {
		trimmed := strings.TrimSpace(rawURL)
		if trimmed == "" {
			continue
		}
		if _, exists := seen[trimmed]; exists {
			log.Printf("[DeduplicateAndValidateImageURLs] duplicate url skipped: %s", trimmed)
			continue
		}
		seen[trimmed] = struct{}{}
		uniqueURLs = append(uniqueURLs, trimmed)
	}

	if len(uniqueURLs) == 0 {
		return nil, nil
	}

	const maxConcurrency = 10

	client := &http.Client{
		Timeout: timeout,
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			if len(via) >= 10 {
				return http.ErrUseLastResponse
			}
			return nil
		},
	}

	resultCh := make(chan urlCheckResult, len(uniqueURLs))
	semaphore := make(chan struct{}, maxConcurrency)
	var wg sync.WaitGroup

	for i, u := range uniqueURLs {
		wg.Add(1)
		go func(idx int, rawURL string) {
			defer wg.Done()
			semaphore <- struct{}{}
			defer func() { <-semaphore }()

			valid := checkImageURL(ctx, client, rawURL, timeout)
			resultCh <- urlCheckResult{index: idx, url: rawURL, valid: valid}
		}(i, u)
	}

	go func() {
		wg.Wait()
		close(resultCh)
	}()

	results := make([]urlCheckResult, len(uniqueURLs))
	for r := range resultCh {
		results[r.index] = r
	}

	validURLs = make([]string, 0, len(uniqueURLs))
	invalidURLs = make([]string, 0)

	for _, r := range results {
		if r.valid {
			validURLs = append(validURLs, r.url)
		} else {
			invalidURLs = append(invalidURLs, r.url)
		}
	}

	return validURLs, invalidURLs
}

func checkImageURL(ctx context.Context, client *http.Client, rawURL string, timeout time.Duration) bool {
	parsed, err := url.Parse(rawURL)
	if err != nil {
		log.Printf("[DeduplicateAndValidateImageURLs] invalid url format: %s, err: %v", rawURL, err)
		return false
	}
	if parsed.Scheme != "http" && parsed.Scheme != "https" {
		log.Printf("[DeduplicateAndValidateImageURLs] invalid url scheme: %s, url: %s", parsed.Scheme, rawURL)
		return false
	}
	if parsed.Host == "" {
		log.Printf("[DeduplicateAndValidateImageURLs] empty host in url: %s", rawURL)
		return false
	}

	reqCtx, cancel := context.WithTimeout(ctx, timeout)
	defer cancel()

	req, err := http.NewRequestWithContext(reqCtx, http.MethodHead, rawURL, nil)
	if err != nil {
		log.Printf("[DeduplicateAndValidateImageURLs] create request failed: %s, err: %v", rawURL, err)
		return false
	}

	resp, err := client.Do(req)
	if err != nil {
		log.Printf("[DeduplicateAndValidateImageURLs] url check failed: %s, err: %v", rawURL, err)
		return false
	}
	defer resp.Body.Close()

	if resp.StatusCode >= 200 && resp.StatusCode < 400 {
		return true
	}

	log.Printf("[DeduplicateAndValidateImageURLs] url not accessible: %s, status: %d", rawURL, resp.StatusCode)
	return false
}


path: evals/iteration-1/forward-spec-then-code/without_skill/outputs/spec_output.md
content: # Spec: Image URL Deduplication and Accessibility Validation Utility

## 1. Background & Goal

When users upload images, the client may send duplicate or invalid image URLs. A utility function is needed to:

1. **Deduplicate** — preserve the order of first occurrence, remove duplicates.
2. **Validate accessibility** — send an HTTP HEAD request to each URL and confirm a 2xx status code.
3. **Filter and log** — invalid URLs are filtered out and logged for debugging.

## 2. Function Signature

```go
package utils

func DeduplicateAndValidateImageURLs(
    ctx context.Context,
    urls []string,
    timeout time.Duration,
) (validURLs []string, invalidURLs []string)
```

## 3. Detailed Design

### 3.1 Deduplication strategy

- Use `map[string]struct{}` to track seen URLs.
- Iterate the input list, skip already-seen URLs, preserve first-occurrence order.
- Apply `strings.TrimSpace` to each URL before deduplication; skip empty strings.

### 3.2 URL format validation

- Parse with `net/url.Parse`.
- Scheme must be `http` or `https`.
- Host must be non-empty.
- Malformed URLs are logged as warn and added to `invalidURLs`.

### 3.3 Accessibility validation

- Create a HEAD request with `http.NewRequestWithContext` bound to a timeout context.
- Status codes in the 200–399 range are considered accessible.
- Network errors or non-2xx/3xx status codes are logged as warn and added to `invalidURLs`.

### 3.4 Concurrency control

- Use `sync.WaitGroup` + a buffered channel semaphore for concurrent validation, default concurrency: 10.
- Results are collected via a channel and reordered to match the original input order.

### 3.5 Logging conventions

- Duplicate URL: `log.Info("duplicate url skipped: %s", url)`
- Malformed URL: `log.Warn("invalid url format: %s, err: %v", url, err)`
- Inaccessible URL: `log.Warn("url not accessible: %s, status: %d", url, statusCode)`
- Request failure: `log.Warn("url check failed: %s, err: %v", url, err)`

### 3.6 Edge cases

| Scenario | Behavior |
|----------|----------|
| Input is nil or empty slice | Return two empty slices |
| All URLs are invalid | validURLs is empty; invalidURLs contains all invalid URLs |
| All URLs are valid and unique | invalidURLs is empty |
| URL has leading/trailing whitespace | TrimSpace before processing |
| URL is an empty string | Skip; not counted in any return value |
| Timeout | Treated as inaccessible; logged |

## 4. Dependencies

- Standard library: `context`, `net/http`, `net/url`, `strings`, `sync`, `time`
- Project logging library

## 5. Test Points

1. Empty input returns empty slices.
2. Duplicate URLs — only the first occurrence is kept.
3. Malformed URLs are filtered into `invalidURLs`.
4. Inaccessible URLs are filtered into `invalidURLs`.
5. Valid URLs preserve original order.
6. Concurrency safety (large URL input).
7. Timeout scenario.


path: evals/iteration-1/reverse-spec-module-level/eval_metadata.json
content: {
  "eval_id": 2,
  "eval_name": "reverse-spec-module-level",
  "prompt": "Reverse-engineer a spec for internal/orchestrator/supervisor.go — I want to understand what this supervisor actually does.",
  "assertions": [
    {"text": "Spec accurately describes the behavior of both core functions: GetSupervisor and Generate", "type": "qualitative"},
    {"text": "Spec covers the version-branching logic", "type": "qualitative"},
    {"text": "Spec flags suspicious behavior or noteworthy points using [NOTE] tags", "type": "qualitative"},
    {"text": "Spec distinguishes between what the code actually does and what it might have intended to do", "type": "qualitative"},
    {"text": "Spec includes dependency relationships (what it depends on / who depends on it)", "type": "qualitative"},
    {"text": "Spec does not write any files to the project source directory", "type": "qualitative"},
    {"text": "Spec describes the configuration priority chain", "type": "qualitative"}
  ]
}


path: evals/iteration-1/reverse-spec-module-level/with_skill/grading.json
content: {
  "eval_name": "reverse-spec-module-level",
  "configuration": "with_skill",
  "expectations": [
    {"text": "Spec accurately describes the behavior of both core functions: GetSupervisor and Generate", "passed": true, "evidence": "Scenario 1 details 6 sub-steps of GetSupervisor; Scenario 2 covers the full Generate flow"},
    {"text": "Spec covers the version-branching logic", "passed": true, "evidence": "Lists tool set differences across three version branches in detail"},
    {"text": "Spec flags suspicious behavior or noteworthy points using [NOTE] tags", "passed": true, "evidence": "Uses [NOTE] tags to flag 3 suspicious points: empty tool set on version parse failure, MaxIterations=100, stop channel passed as nil"},
    {"text": "Spec distinguishes between what the code actually does and what it might have intended to do", "passed": true, "evidence": "[NOTE] tags explicitly separate actual code behavior from possible intent"},
    {"text": "Spec includes dependency relationships (what it depends on / who depends on it)", "passed": true, "evidence": "Dependencies section has separate tables for external systems/modules and internal modules, with clear 'depended on by' entries"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec describes the configuration priority chain", "passed": true, "evidence": "Constraints & Boundaries section explicitly lists configuration priority for model params, System Prompt, and tool list"}
  ]
}


path: evals/iteration-1/reverse-spec-module-level/with_skill/outputs/spec_output.md
content: # Agent Orchestrator Spec

## Overview

`supervisor.go` is the core entry point of the AI assistant agent. It is responsible for **assembling and initializing an LLM agent instance** (Supervisor) that understands user intent and calls the appropriate tools to fulfill requests. The module exposes two functions: `GetSupervisor` (builds the agent) and `Generate` (runs the agent to handle user messages).

## Glossary

- **Supervisor**: the orchestrating agent that understands user intent and selects the right tool to call. It does not execute business logic directly — it orchestrates tool calls.
- **Tool**: a callable unit that encapsulates a specific capability.
- **Legacy tool set**: the set of tools used by older client versions.
- **Current tool set**: the set of tools used by newer client versions.
- **ConfigOverrides**: sparse override config delivered by the debug platform, allowing dynamic control over tool enable/disable, tool description overrides, System Prompt overrides, and model parameter overrides.
- **ConfigID**: a debug config identifier; when non-empty, the request follows the debug/experiment path.

## Behavioral Contracts

### Scenario 1: Building the agent instance (GetSupervisor)

- **Precondition**: `context` has the request context injected (including client version, user info, session info, etc.).
- **Input**: `context.Context`
- **Expected behavior**:

  #### 1.1 Pre-load ConfigOverrides
  - Extract `configID` from context.
  - If `configID` is non-empty, attempt to load ConfigOverrides from storage and store in context.
  - Load failure does not block the flow — silently skip.

  #### 1.2 Select tool set by client version
  - Extract the client version from context and parse it into a structured version number.
  - **Version < 2.0.0 (Legacy)**: use the legacy tool set (ToolA, ToolB, ToolC, etc.).
  - **Version ≥ 2.0.0 (Current)**: use the V2 tool set (ToolA_V2, ToolB_V2, ToolC_V2, etc.).
  - **Version ≥ 3.0.0**: additionally append the new-capability tool.
  - **Version parse failure**: no version-specific tools are added; only log the error. [NOTE] On parse failure, `useLegacyTool` stays `false` and the code takes the current-version branch but with an empty tool list — meaning **the agent will have no tools available**.

  #### 1.3 Apply tool overrides
  - If `configID` is non-empty, filter and override the tool list:
    - Tools marked as disabled are removed.
    - Tools with an override description have their description replaced.

  #### 1.4 Load middlewares
  - Retrieve the handler-level middleware list, which includes:
    - **Format-patch middleware**: fixes tool call format issues.
    - **Output-parse middleware**: intercepts specific tags in the stream and converts them to structured data for the client.
    - **Routing middleware** (optional): injects skill routing capability. Failure to create does not block — silently skip.

  #### 1.5 Assemble AgentConfig
  - **Model**: specified online model, Temperature=1.0, MaxTokens=6144, Timeout=1min.
  - **System Prompt**: hardcoded default in code; overridden at runtime by the version managed in the Prompt platform.
  - **Cache strategy**: session-level cache.
  - **Tool execution mode**: sequential.
  - **Max iterations**: 100. [NOTE] 100 is far larger than what normal conversations require (typically < 10). If reached, it likely means the agent is stuck in a loop.
  - **Agent-level middlewares**:
    - `BeforeChatModel`: before each LLM call, refresh profile info and sync persisted tool messages.
    - `AfterChatModel`: after each LLM response, wait for async moderation result and persist message history.
    - `WrapToolCall`: wraps tool calls for parameter transformation, user-stop checking, and metrics.

  #### 1.6 Create AgentModel
  - Internally resolves the final config by priority:
    1. If `configID` exists and ConfigOverrides are present: fetch the online default config, then apply sparse overrides.
    2. If `configID` exists but no ConfigOverrides: fall back to the full debug config.
    3. If no `configID`: use the online default config (code default → config center → A/B experiment → Prompt platform).

- **Postcondition**: returns an agent instance ready for inference.
- **Error cases**:
  - Middleware creation fails: return error immediately; agent is not created.
  - AgentModel creation fails: return error immediately.
  - ConfigOverrides load failure, config read failure, Prompt fetch failure: all non-blocking; fall back to code defaults.

### Scenario 2: Running the agent to handle user messages (Generate)

- **Precondition**: context has the full business context injected.
- **Input**: `context.Context` + list of user messages.
- **Expected behavior**:
  1. Call `GetSupervisor` to build the agent instance.
  2. Report initialization stage latency metrics.
  3. Call `agent.Run` to start agent inference with streaming enabled. [NOTE] The stop channel is passed as `nil`; user cancellation relies on reading a stop signal from context instead.
  4. Return an event stream iterator; the caller consumes agent-produced events via the iterator.
- **Postcondition**: returns an event stream iterator for streaming consumption.
- **Error cases**:
  - `GetSupervisor` fails: return error; inference is not started.

## Constraints & Boundaries

### Tool execution
- All tools execute **sequentially** — no parallel calls.
- Certain tools are marked `ReturnDirectly`: their output is returned directly to the user without a second LLM pass.

### Version compatibility
- Version checks are performed independently in two places: tool set selection and System Prompt key selection. [NOTE] These two checks are independent — it is theoretically possible for the tool set and System Prompt version to be mismatched (e.g., if ConfigOverrides overrides the System Prompt key but not the tool set).

### Configuration priority
- Model parameters: code default < config center < A/B experiment.
- System Prompt: hardcoded default < Prompt platform version < ConfigOverrides.
- Tool list: code version selection < ConfigOverrides.

### What we don't do
- The Supervisor itself **executes no business logic** — all capabilities are encapsulated in tools.
- The Supervisor **does not manage conversation history persistence** — that is handled by middleware.
- The Supervisor **does not handle streaming delivery directly** — that is handled by the output-parse middleware.

## Dependencies

### Depends on
| Dependency | Purpose |
|---|---|
| **LLM service** | Inference |
| **Prompt platform** | System Prompt hosting and versioning |
| **Config center** | Agent model parameter configuration |
| **A/B experiment platform** | Gradual rollout of agent config changes |
| **KV storage** | Session cache, message persistence, ConfigOverrides storage |
| **Agent framework** | Agent construction, tool orchestration, middleware, streaming events |

### Depended on by
- The conversation entry service layer, which calls `Generate` to start agent inference.

## Open Questions

1. [NOTE] **Empty tool set on version parse failure**: the agent will have no tools and can only respond with plain text. Is this the intended behavior? Should it fall back to a default tool set?

2. [NOTE] **MaxIterations = 100**: this is far larger than what normal conversations need (typically < 10). Should a timeout mechanism or a more reasonable upper bound be added?

3. [NOTE] **Stop channel passed as nil in Generate**: actual user cancellation is implemented by reading a stop signal from context, not via this parameter. Is this parameter effectively deprecated?


path: evals/iteration-1/reverse-spec-module-level/without_skill/grading.json
content: {
  "eval_name": "reverse-spec-module-level",
  "configuration": "without_skill",
  "expectations": [
    {"text": "Spec accurately describes the behavior of both core functions: GetSupervisor and Generate", "passed": true, "evidence": "Both functions are described in detail"},
    {"text": "Spec covers the version-branching logic", "passed": true, "evidence": "Version branches and corresponding tool sets are listed in a table"},
    {"text": "Spec flags suspicious behavior or noteworthy points using [NOTE] tags", "passed": false, "evidence": "No suspicious behavior flagged, no [NOTE] tags used — all content is purely descriptive"},
    {"text": "Spec distinguishes between what the code actually does and what it might have intended to do", "passed": false, "evidence": "Only describes what the code does, without distinguishing actual behavior from possible intent"},
    {"text": "Spec includes dependency relationships (what it depends on / who depends on it)", "passed": true, "evidence": "Dependency diagram lists all dependencies in detail"},
    {"text": "Spec does not write any files to the project source directory", "passed": true, "evidence": "Output saved only to the workspace outputs directory"},
    {"text": "Spec describes the configuration priority chain", "passed": true, "evidence": "Configuration override chain is explicitly described"}
  ]
}


path: evals/iteration-1/reverse-spec-module-level/without_skill/outputs/spec_output.md
content: # Supervisor Module Spec

## 1. Module Overview

**File**: `internal/orchestrator/supervisor.go`

**Core responsibility**: this file is the **main entry point** of the AI assistant agent. It builds and runs a ReAct LLM agent (Supervisor) that understands user intent and calls different tools to fulfill requests.

---

## 2. Exported Interface

- **`GetSupervisor`**: builds and returns a fully configured agent instance without running inference.
- **`Generate`**: the main external entry point, called by the service layer. Accepts a list of user messages, starts agent inference, and returns an event stream iterator.

## 3. Core Build Logic

### 3.1 ConfigOverrides pre-loading
If `configID` is present in context, pre-load ConfigOverrides and store in context. Used downstream to override model parameters, system prompt, tool enable/disable, and tool descriptions.

### 3.2 Client version branching — tool set selection

| Version range | Tool set |
|--------------|----------|
| `< 2.0.0` (Legacy) | ToolA, ToolB, ToolC, ToolD, ToolE, ToolF |
| `>= 2.0.0` (Current) | ToolA_V2, ToolB_V2, ToolC_V2, ToolD_V2, ToolE_V2, ToolF_V2 |
| `>= 3.0.0` (additional) | NewCapabilityTool |

### 3.3 Tool overrides
When ConfigOverrides are present, tools can be disabled by name or have their descriptions replaced.

### 3.4 Middleware loading
Three handler-level middlewares:
- **Format patch**: fixes tool call format compatibility issues
- **Output parse**: intercepts specific tags in the stream and converts them to structured data
- **Routing middleware**: injects skill routing capability

### 3.5 Agent model config
- **Cache**: session cache, supports multi-turn conversations
- **Max iterations**: 100-step ReAct loop
- **Sequential tool execution**: prevents concurrent delivery from causing client rendering issues
- **ReturnDirectly**: certain tools return their output directly without a second LLM pass

### 3.6 Multi-layer config override chain
`code default → config center → A/B experiment → ConfigOverrides`

## 4. Key Design Decisions
1. **Single agent + multiple tools, flat architecture** — the LLM autonomously decides which tool to call
2. **Version compatibility**: two tool sets maintained via version branching
3. **Config-driven + middleware decoupling**: agent behavior can be adjusted flexibly via config platforms
4. **Content moderation integration**: async moderation intercept after LLM output
5. **supervisor.go contains no business logic** — purely responsible for assembling and starting the agent


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)