Skill: lidekahdjdhdhsjjs-lang/hz-context-optimizer (version 1.0.0)

--- name: context-compression description: This skill should be used when the user asks to "compress context", "summarize conversation history", "implement compaction", "reduce token usage", or mentions context compression, structured summarization, tokens-per-task optimization, or long-running agent sessions exceeding context limits. --- # Context Compression Strategies When agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. The correct optimization target is tokens per task: total tokens consumed to complete a task, including re-fetching costs when compression loses critical information. ## When to Activate Activate this skill when: - Agent sessions exceed context window limits - Codebases exceed context windows (5M+ token systems) - Designing conversation summarization strategies - Debugging cases where agents "forget" what files they modified - Building evaluation frameworks for compression quality ## Core Concepts Context compression trades token savings against information loss. Three production-ready approaches exist: 1. **Anchored Iterative Summarization**: Maintain structured, persistent summaries with explicit sections for session intent, file modifications, decisions, and next steps. When compression triggers, summarize only the newly-truncated span and merge with the existing summary. Structure forces preservation by dedicating sections to specific information types. 2. **Opaque Compression**: Produce compressed representations optimized for reconstruction fidelity. Achieves highest compression ratios (99%+) but sacrifices interpretability. Cannot verify what was preserved. 3. **Regenerative Full Summary**: Generate detailed structured summaries on each compression. Produces readable output but may lose details across repeated compression cycles due to full regeneration rather than incremental merging. The critical insight: structure forces preservation. Dedicated sections act as checklists that the summarizer must populate, preventing silent information drift. ## Detailed Topics ### Why Tokens-Per-Task Matters Traditional compression metrics target tokens-per-request. This is the wrong optimization. When compression loses critical details like file paths or error messages, the agent must re-fetch information, re-explore approaches, and waste tokens recovering context. The right metric is tokens-per-task: total tokens consumed from task start to completion. A compression strategy saving 0.5% more tokens but causing 20% more re-fetching costs more overall. ### The Artifact Trail Problem Artifact trail integrity is the weakest dimension across all compression methods, scoring 2.2-2.5 out of 5.0 in evaluations. Even structured summarization with explicit file sections struggles to maintain complete file tracking across long sessions. Coding agents need to know: - Which files were created - Which files were modified and what changed - Which files were read but not changed - Function names, variable names, error messages This problem likely requires specialized handling beyond general summarization: a separate artifact index or explicit file-state tracking in agent scaffolding. ### Structured Summary Sections Effective structured summaries include explicit sections: ```markdown ## Session Intent [What the user is trying to accomplish] ## Files Modified - auth.controller.ts: Fixed JWT token generation - config/redis.ts: Updated connection pooling - tests/auth.test.ts: Added mock setup for new config ## Decisions Made - Using Redis connection pool instead of per-request connections - Retry logic with exponential backoff for transient failures ## Current State - 14 tests passing, 2 failing - Remaining: mock setup for session service tests ## Next Steps 1. Fix remaining test failures 2. Run full test suite 3. Update documentation ``` This structure prevents silent loss of file paths or decisions because each section must be explicitly addressed. ### Compression Trigger Strategies When to trigger compression matters as much as how to compress: | Strategy | Trigger Point | Trade-off | |----------|---------------|-----------| | Fixed threshold | 70-80% context utilization | Simple but may compress too early | | Sliding window | Keep last N turns + summary | Predictable context size | | Importance-based | Compress low-relevance sections first | Complex but preserves signal | | Task-boundary | Compress at logical task completions | Clean summaries but unpredictable timing | The sliding window approach with structured summaries provides the best balance of predictability and quality for most coding agent use cases. ### Probe-Based Evaluation Traditional metrics like ROUGE or embedding similarity fail to capture functional compression quality. A summary may score high on lexical overlap while missing the one file path the agent needs. Probe-based evaluation directly measures functional quality by asking questions after compression: | Probe Type | What It Tests | Example Question | |------------|---------------|------------------| | Recall | Factual retention | "What was the original error message?" | | Artifact | File tracking | "Which files have we modified?" | | Continuation | Task planning | "What should we do next?" | | Decision | Reasoning chain | "What did we decide about the Redis issue?" | If compression preserved the right information, the agent answers correctly. If not, it guesses or hallucinates. ### Evaluation Dimensions Six dimensions capture compression quality for coding agents: 1. **Accuracy**: Are technical details correct? File paths, function names, error codes. 2. **Context Awareness**: Does the response reflect current conversation state? 3. **Artifact Trail**: Does the agent know which files were read or modified? 4. **Completeness**: Does the response address all parts of the question? 5. **Continuity**: Can work continue without re-fetching information? 6. **Instruction Following**: Does the response respect stated constraints? Accuracy shows the largest variation between compression methods (0.6 point gap). Artifact trail is universally weak (2.2-2.5 range). ## Practical Guidance ### Three-Phase Compression Workflow For large codebases or agent systems exceeding context windows, apply compression through three phases: 1. **Research Phase**: Produce a research document from architecture diagrams, documentation, and key interfaces. Compress exploration into a structured analysis of components and dependencies. Output: single research document. 2. **Planning Phase**: Convert research into implementation specification with function signatures, type definitions, and data flow. A 5M token codebase compresses to approximately 2,000 words of specification. 3. **Implementation Phase**: Execute against the specification. Context remains focused on the spec rather than raw codebase exploration. ### Using Example Artifacts as Seeds When provided with a manual migration example or reference PR, use it as a template to understand the target pattern. The example reveals constraints that static analysis cannot surface: which invariants must hold, which services break on changes, and what a clean migration looks like. This is particularly important when the agent cannot distinguish essential complexity (business requirements) from accidental complexity (legacy workarounds). The example artifact encodes that distinction. ### Implementing Anchored Iterative Summarization 1. Define explicit summary sections matching your agent's needs 2. On first compression trigger, summarize truncated history into sections 3. On subsequent compressions, summarize only new truncated content 4. Merge new summary into existing sections rather than regenerating 5. Track which information came from which compression cycle for debugging ### When to Use Each Approach **Use anchored iterative summarization when:** - Sessions are long-running (100+ messages) - File tracking matters (coding, debugging) - You need to verify what was preserved **Use opaque compression when:** - Maximum token savings required - Sessions are relatively short - Re-fetching costs are low **Use regenerative summaries when:** - Summary interpretability is critical - Sessions have clear phase boundaries - Full context review is acceptable on each compression ### Compression Ratio Considerations | Method | Compression Ratio | Quality Score | Trade-off | |--------|-------------------|---------------|-----------| | Anchored Iterative | 98.6% | 3.70 | Best quality, slightly less compression | | Regenerative | 98.7% | 3.44 | Good quality, moderate compression | | Opaque | 99.3% | 3.35 | Best compression, quality loss | The 0.7% additional tokens retained by structured summarization buys 0.35 quality points. For any task where re-fetching costs matter, this trade-off favors structured approaches. ## Examples **Example 1: Debugging Session Compression** Original context (89,000 tokens, 178 messages): - 401 error on /api/auth/login endpoint - Traced through auth controller, middleware, session store - Found stale Redis connection - Fixed connection pooling, added retry logic - 14 tests passing, 2 failing Structured summary after compression: ```markdown ## Session Intent Debug 401 Unauthorized error on /api/auth/login despite valid credentials. ## Root Cause Stale Redis connection in session store. JWT generated correctly but session could not be persisted. ## Files Modified - auth.controller.ts: No changes (read only) - middleware/cors.ts: No changes (examined) - config/redis.ts: Fixed connection pooling configuration - services/session.service.ts: Added retry logic for transient failures - tests/auth.test.ts: Updated mock setup ## Test Status 14 passing, 2 failing (mock setup issues) ## Next Steps 1. Fix remaining test failures (mock session service) 2. Run full test suite 3. Deploy to staging ``` **Example 2: Probe Response Quality** After compression, asking "What was the original error?": Good response (structured summarization): > "The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users received this error with valid credentials. Root cause was stale Redis connection in session store." Poor response (aggressive compression): > "We were debugging an authentication issue. The login was failing. We fixed some configuration problems." The structured response preserves endpoint, error code, and root cause. The aggressive response loses all technical detail. ## Guidelines 1. Optimize for tokens-per-task, not tokens-per-request 2. Use structured summaries with explicit sections for file tracking 3. Trigger compression at 70-80% context utilization 4. Implement incremental merging rather than full regeneration 5. Test compression quality with probe-based evaluation 6. Track artifact trail separately if file tracking is critical 7. Accept slightly lower compression ratios for better quality retention 8. Monitor re-fetching frequency as a compression quality signal ## Integration This skill connects to several others in the collection: - context-degradation - Compression is a mitigation strategy for degradation - context-optimization - Compression is one optimization technique among many - evaluation - Probe-based evaluation applies to compression testing - memory-systems - Compression relates to scratchpad and summary memory patterns ## References Internal reference: - [Evaluation Framework Reference](./references/evaluation-framework.md) - Detailed probe types and scoring rubrics Related skills in this collection: - context-degradation - Understanding what compression prevents - context-optimization - Broader optimization strategies - evaluation - Building evaluation frameworks External resources: - Factory Research: Evaluating Context Compression for AI Agents (December 2025) - Research on LLM-as-judge evaluation methodology (Zheng et al., 2023) - Netflix Engineering: "The Infinite Software Crisis" - Three-phase workflow and context compression at scale (AI Summit 2025) --- ## Skill Metadata **Created**: 2025-12-22 **Last Updated**: 2025-12-26 **Author**: Agent Skills for Context Engineering Contributors **Version**: 1.1.0

path: _meta.json
content: {
  "name": "hz-context-optimizer",
  "displayName": "Huizai Context Optimizer",
  "description": "OpenClaw上下文优化技能 - 智能压缩、记忆分层、token节省",
  "author": {
    "name": "lidekahdjdhdhsjjs-lang",
    "url": "https://github.com/lidekahdjdhdhsjjs-lang"
  },
  "repository": "https://github.com/lidekahdjdhdhsjjs-lang/Huizai-openclaw",
  "tags": ["context", "token", "optimization", "openclaw", "huizai"],
  "homepage": "https://github.com/lidekahdjdhdhsjjs-lang/Huizai-openclaw"
}


path: references/evaluation-framework.md
content: # Context Compression Evaluation Framework

This document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration.

## Probe Types

### Recall Probes

Test factual retention of specific details from conversation history.

**Structure:**
```
Question: [Ask for specific fact from truncated history]
Expected: [Exact detail that should be preserved]
Scoring: Match accuracy of technical details
```

**Examples:**
- "What was the original error message that started this debugging session?"
- "What version of the dependency did we decide to use?"
- "What was the exact command that failed?"

### Artifact Probes

Test file tracking and modification awareness.

**Structure:**
```
Question: [Ask about files created, modified, or examined]
Expected: [Complete list with change descriptions]
Scoring: Completeness of file list and accuracy of change descriptions
```

**Examples:**
- "Which files have we modified? Describe what changed in each."
- "What new files did we create in this session?"
- "Which configuration files did we examine but not change?"

### Continuation Probes

Test ability to continue work without re-fetching context.

**Structure:**
```
Question: [Ask about next steps or current state]
Expected: [Actionable next steps based on session history]
Scoring: Ability to continue without requesting re-read of files
```

**Examples:**
- "What should we do next?"
- "What tests are still failing and why?"
- "What was left incomplete from our last step?"

### Decision Probes

Test retention of reasoning chains and decision rationale.

**Structure:**
```
Question: [Ask about why a decision was made]
Expected: [Reasoning that led to the decision]
Scoring: Preservation of decision context and alternatives considered
```

**Examples:**
- "We discussed options for the Redis issue. What did we decide and why?"
- "Why did we choose connection pooling over per-request connections?"
- "What alternatives did we consider for the authentication fix?"

## Scoring Rubrics

### Accuracy Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| accuracy_factual | Are facts, file paths, and technical details correct? | Completely incorrect or fabricated | Mostly accurate with minor errors | Perfectly accurate |
| accuracy_technical | Are code references and technical concepts correct? | Major technical errors | Generally correct with minor issues | Technically precise |

### Context Awareness Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| context_conversation_state | Does the response reflect current conversation state? | No awareness of prior context | General awareness with gaps | Full awareness of conversation history |
| context_artifact_state | Does the response reflect which files/artifacts were accessed? | No awareness of artifacts | Partial artifact awareness | Complete artifact state awareness |

### Artifact Trail Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| artifact_files_created | Does the agent know which files were created? | No knowledge | Knows most files | Perfect knowledge |
| artifact_files_modified | Does the agent know which files were modified and what changed? | No knowledge | Good knowledge of most modifications | Perfect knowledge of all modifications |
| artifact_key_details | Does the agent remember function names, variable names, error messages? | No recall | Recalls most key details | Perfect recall |

### Completeness Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| completeness_coverage | Does the response address all parts of the question? | Ignores most parts | Addresses most parts | Addresses all parts thoroughly |
| completeness_depth | Is sufficient detail provided? | Superficial or missing detail | Adequate detail | Comprehensive detail |

### Continuity Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| continuity_work_state | Can the agent continue without re-fetching previously accessed information? | Cannot continue without re-fetching all context | Can continue with minimal re-fetching | Can continue seamlessly |
| continuity_todo_state | Does the agent maintain awareness of pending tasks? | Lost track of all TODOs | Good awareness with some gaps | Perfect task awareness |
| continuity_reasoning | Does the agent retain rationale behind previous decisions? | No memory of reasoning | Generally remembers reasoning | Excellent retention |

### Instruction Following Dimension

| Criterion | Question | Score 0 | Score 3 | Score 5 |
|-----------|----------|---------|---------|---------|
| instruction_format | Does the response follow the requested format? | Ignores format | Generally follows format | Perfectly follows format |
| instruction_constraints | Does the response respect stated constraints? | Ignores constraints | Mostly respects constraints | Fully respects all constraints |

## LLM Judge Configuration

### System Prompt

```
You are an expert evaluator assessing AI assistant responses in software development conversations.

Your task is to grade responses against specific rubric criteria. For each criterion:
1. Read the criterion question carefully
2. Examine the response for evidence
3. Assign a score from 0-5 based on the scoring guide
4. Provide brief reasoning for your score

Be objective and consistent. Focus on what is present in the response, not what could have been included.
```

### Judge Input Format

```json
{
  "probe_question": "What was the original error message?",
  "model_response": "[Response to evaluate]",
  "compacted_context": "[The compressed context that was provided]",
  "ground_truth": "[Optional: known correct answer]",
  "rubric_criteria": ["accuracy_factual", "accuracy_technical", "context_conversation_state"]
}
```

### Judge Output Format

```json
{
  "criterionResults": [
    {
      "criterionId": "accuracy_factual",
      "score": 5,
      "reasoning": "Response correctly identifies the 401 error, specific endpoint, and root cause."
    }
  ],
  "aggregateScore": 4.8,
  "dimensionScores": {
    "accuracy": 4.9,
    "context_awareness": 4.5,
    "artifact_trail": 3.2,
    "completeness": 5.0,
    "continuity": 4.8,
    "instruction_following": 5.0
  }
}
```

## Benchmark Results Reference

Performance across compression methods (based on 36,000+ messages):

| Method | Overall | Accuracy | Context | Artifact | Complete | Continuity | Instruction |
|--------|---------|----------|---------|----------|----------|------------|-------------|
| Anchored Iterative | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 |
| Regenerative | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 |
| Opaque | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 |

**Key Findings:**

1. **Accuracy gap**: 0.61 points between best and worst methods
2. **Context awareness gap**: 0.45 points, favoring anchored iterative
3. **Artifact trail**: Universally weak (2.19-2.45), needs specialized handling
4. **Completeness and instruction following**: Minimal differentiation

## Statistical Considerations

- Differences of 0.26-0.35 points are consistent across task types and session lengths
- Pattern holds for both short and long sessions
- Pattern holds across debugging, feature implementation, and code review tasks
- Sample size: 36,611 messages across hundreds of compression points

## Implementation Notes

### Probe Generation

Generate probes at each compression point based on truncated history:
1. Extract factual claims for recall probes
2. Extract file operations for artifact probes
3. Extract incomplete tasks for continuation probes
4. Extract decision points for decision probes

### Grading Process

1. Feed probe question + model response + compressed context to judge
2. Evaluate against each criterion in rubric
3. Output structured JSON with scores and reasoning
4. Compute dimension scores as weighted averages
5. Compute overall score as unweighted average of dimensions

### Blinding

The judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods.



path: scripts/compression_evaluator.py
content: """
Context Compression Evaluation

This module provides utilities for evaluating context compression quality
using probe-based assessment.

PRODUCTION NOTES:
- The LLM judge calls are stubbed for demonstration. Production systems
  should implement actual API calls to GPT-5.2 or equivalent.
- Token estimation uses simplified heuristics. Production systems should
  use model-specific tokenizers.
- Ground truth extraction uses pattern matching. Production systems may
  benefit from more sophisticated fact extraction.
"""

from dataclasses import dataclass, field
from typing import List, Dict, Optional
from enum import Enum
import json
import re


class ProbeType(Enum):
    RECALL = "recall"
    ARTIFACT = "artifact"
    CONTINUATION = "continuation"
    DECISION = "decision"


@dataclass
class Probe:
    """A probe question for evaluating compression quality."""
    probe_type: ProbeType
    question: str
    ground_truth: Optional[str] = None
    context_reference: Optional[str] = None


@dataclass
class CriterionResult:
    """Result for a single evaluation criterion."""
    criterion_id: str
    score: float
    reasoning: str


@dataclass
class EvaluationResult:
    """Complete evaluation result for a probe response."""
    probe: Probe
    response: str
    criterion_results: List[CriterionResult]
    aggregate_score: float
    dimension_scores: Dict[str, float] = field(default_factory=dict)


# Evaluation Rubrics

RUBRIC_CRITERIA = {
    "accuracy": [
        {
            "id": "accuracy_factual",
            "question": "Are facts, file paths, and technical details correct?",
            "weight": 0.6
        },
        {
            "id": "accuracy_technical", 
            "question": "Are code references and technical concepts correct?",
            "weight": 0.4
        }
    ],
    "context_awareness": [
        {
            "id": "context_conversation_state",
            "question": "Does the response reflect current conversation state?",
            "weight": 0.5
        },
        {
            "id": "context_artifact_state",
            "question": "Does the response reflect which files/artifacts were accessed?",
            "weight": 0.5
        }
    ],
    "artifact_trail": [
        {
            "id": "artifact_files_created",
            "question": "Does the agent know which files were created?",
            "weight": 0.3
        },
        {
            "id": "artifact_files_modified",
            "question": "Does the agent know which files were modified?",
            "weight": 0.4
        },
        {
            "id": "artifact_key_details",
            "question": "Does the agent remember function names, variable names, error messages?",
            "weight": 0.3
        }
    ],
    "completeness": [
        {
            "id": "completeness_coverage",
            "question": "Does the response address all parts of the question?",
            "weight": 0.6
        },
        {
            "id": "completeness_depth",
            "question": "Is sufficient detail provided?",
            "weight": 0.4
        }
    ],
    "continuity": [
        {
            "id": "continuity_work_state",
            "question": "Can the agent continue without re-fetching information?",
            "weight": 0.4
        },
        {
            "id": "continuity_todo_state",
            "question": "Does the agent maintain awareness of pending tasks?",
            "weight": 0.3
        },
        {
            "id": "continuity_reasoning",
            "question": "Does the agent retain rationale behind previous decisions?",
            "weight": 0.3
        }
    ],
    "instruction_following": [
        {
            "id": "instruction_format",
            "question": "Does the response follow the requested format?",
            "weight": 0.5
        },
        {
            "id": "instruction_constraints",
            "question": "Does the response respect stated constraints?",
            "weight": 0.5
        }
    ]
}


class ProbeGenerator:
    """Generate probes from conversation history."""
    
    def __init__(self, conversation_history: str):
        self.history = conversation_history
        self.extracted_facts = self._extract_facts()
        self.extracted_files = self._extract_files()
        self.extracted_decisions = self._extract_decisions()
    
    def generate_probes(self) -> List[Probe]:
        """Generate all probe types for evaluation."""
        probes = []
        
        # Recall probes
        if self.extracted_facts:
            probes.append(Probe(
                probe_type=ProbeType.RECALL,
                question="What was the original error or issue that started this session?",
                ground_truth=self.extracted_facts.get("original_error"),
                context_reference="session_start"
            ))
        
        # Artifact probes
        if self.extracted_files:
            probes.append(Probe(
                probe_type=ProbeType.ARTIFACT,
                question="Which files have we modified? Describe what changed in each.",
                ground_truth=json.dumps(self.extracted_files),
                context_reference="file_operations"
            ))
        
        # Continuation probes
        probes.append(Probe(
            probe_type=ProbeType.CONTINUATION,
            question="What should we do next?",
            ground_truth=self.extracted_facts.get("next_steps"),
            context_reference="task_state"
        ))
        
        # Decision probes
        if self.extracted_decisions:
            probes.append(Probe(
                probe_type=ProbeType.DECISION,
                question="What key decisions did we make and why?",
                ground_truth=json.dumps(self.extracted_decisions),
                context_reference="decision_points"
            ))
        
        return probes
    
    def _extract_facts(self) -> Dict[str, str]:
        """Extract factual claims from history."""
        facts = {}
        
        # Extract error patterns
        error_patterns = [
            r"error[:\s]+(.+?)(?:\n|$)",
            r"(\d{3})\s+(Unauthorized|Not Found|Internal Server Error)",
            r"exception[:\s]+(.+?)(?:\n|$)"
        ]
        
        for pattern in error_patterns:
            match = re.search(pattern, self.history, re.IGNORECASE)
            if match:
                facts["original_error"] = match.group(0).strip()
                break
        
        # Extract next steps
        next_step_patterns = [
            r"next[:\s]+(.+?)(?:\n|$)",
            r"TODO[:\s]+(.+?)(?:\n|$)",
            r"remaining[:\s]+(.+?)(?:\n|$)"
        ]
        
        for pattern in next_step_patterns:
            match = re.search(pattern, self.history, re.IGNORECASE)
            if match:
                facts["next_steps"] = match.group(0).strip()
                break
        
        return facts
    
    def _extract_files(self) -> List[Dict[str, str]]:
        """Extract file operations from history."""
        files = []
        
        # Common file patterns
        file_patterns = [
            r"(?:modified|changed|updated|edited)\s+([^\s]+\.[a-z]+)",
            r"(?:created|added)\s+([^\s]+\.[a-z]+)",
            r"(?:read|examined|opened)\s+([^\s]+\.[a-z]+)"
        ]
        
        for pattern in file_patterns:
            matches = re.findall(pattern, self.history, re.IGNORECASE)
            for match in matches:
                if match not in [f["path"] for f in files]:
                    files.append({
                        "path": match,
                        "operation": "modified" if "modif" in pattern else "created" if "creat" in pattern else "read"
                    })
        
        return files
    
    def _extract_decisions(self) -> List[Dict[str, str]]:
        """Extract decision points from history."""
        decisions = []
        
        decision_patterns = [
            r"decided to\s+(.+?)(?:\n|$)",
            r"chose\s+(.+?)(?:\n|$)",
            r"going with\s+(.+?)(?:\n|$)",
            r"will use\s+(.+?)(?:\n|$)"
        ]
        
        for pattern in decision_patterns:
            matches = re.findall(pattern, self.history, re.IGNORECASE)
            for match in matches:
                decisions.append({
                    "decision": match.strip(),
                    "context": pattern.split("\\s+")[0]
                })
        
        return decisions[:5]  # Limit to 5 decisions


class CompressionEvaluator:
    """Evaluate compression quality using probes and LLM judge."""
    
    def __init__(self, model: str = "gpt-5.2"):
        self.model = model
        self.results: List[EvaluationResult] = []
    
    def evaluate(self, 
                 probe: Probe, 
                 response: str, 
                 compressed_context: str) -> EvaluationResult:
        """
        Evaluate a single probe response.
        
        Args:
            probe: The probe question
            response: The model's response to evaluate
            compressed_context: The compressed context that was provided
            
        Returns:
            EvaluationResult with scores and reasoning
        """
        # Get relevant criteria based on probe type
        criteria = self._get_criteria_for_probe(probe.probe_type)
        
        # Evaluate each criterion
        criterion_results = []
        for criterion in criteria:
            result = self._evaluate_criterion(
                criterion, 
                probe, 
                response, 
                compressed_context
            )
            criterion_results.append(result)
        
        # Calculate dimension scores
        dimension_scores = self._calculate_dimension_scores(criterion_results)
        
        # Calculate aggregate score
        aggregate_score = sum(dimension_scores.values()) / len(dimension_scores)
        
        result = EvaluationResult(
            probe=probe,
            response=response,
            criterion_results=criterion_results,
            aggregate_score=aggregate_score,
            dimension_scores=dimension_scores
        )
        
        self.results.append(result)
        return result
    
    def _get_criteria_for_probe(self, probe_type: ProbeType) -> List[Dict]:
        """Get relevant criteria for probe type."""
        criteria = []
        
        # All probes get accuracy and completeness
        criteria.extend(RUBRIC_CRITERIA["accuracy"])
        criteria.extend(RUBRIC_CRITERIA["completeness"])
        
        # Add type-specific criteria
        if probe_type == ProbeType.ARTIFACT:
            criteria.extend(RUBRIC_CRITERIA["artifact_trail"])
        elif probe_type == ProbeType.CONTINUATION:
            criteria.extend(RUBRIC_CRITERIA["continuity"])
        elif probe_type == ProbeType.RECALL:
            criteria.extend(RUBRIC_CRITERIA["context_awareness"])
        elif probe_type == ProbeType.DECISION:
            criteria.extend(RUBRIC_CRITERIA["context_awareness"])
            criteria.extend(RUBRIC_CRITERIA["continuity"])
        
        criteria.extend(RUBRIC_CRITERIA["instruction_following"])
        
        return criteria
    
    def _evaluate_criterion(self,
                           criterion: Dict,
                           probe: Probe,
                           response: str,
                           context: str) -> CriterionResult:
        """
        Evaluate a single criterion using LLM judge.
        
        PRODUCTION NOTE: This is a stub implementation.
        Production systems should call the actual LLM API:
        
        ```python
        result = openai.chat.completions.create(
            model="gpt-5.2",
            messages=[
                {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
                {"role": "user", "content": self._format_judge_input(criterion, probe, response, context)}
            ]
        )
        return self._parse_judge_output(result)
        ```
        """
        # Stub implementation - in production, call LLM judge
        score = self._heuristic_score(criterion, response, probe.ground_truth)
        reasoning = f"Evaluated {criterion['id']} based on response content."
        
        return CriterionResult(
            criterion_id=criterion["id"],
            score=score,
            reasoning=reasoning
        )
    
    def _heuristic_score(self, 
                         criterion: Dict, 
                         response: str, 
                         ground_truth: Optional[str]) -> float:
        """
        Heuristic scoring for demonstration.
        
        Production systems should use LLM judge instead.
        """
        score = 3.0  # Base score
        
        # Adjust based on response length and content
        if len(response) < 50:
            score -= 1.0  # Too short
        elif len(response) > 500:
            score += 0.5  # Detailed
        
        # Check for technical content
        if any(ext in response for ext in [".ts", ".py", ".js", ".md"]):
            score += 0.5  # Contains file references
        
        if ground_truth and ground_truth in response:
            score += 1.0  # Contains ground truth
        
        return min(5.0, max(0.0, score))
    
    def _calculate_dimension_scores(self, 
                                    criterion_results: List[CriterionResult]) -> Dict[str, float]:
        """Calculate dimension scores from criterion results."""
        dimension_scores = {}
        
        for dimension, criteria in RUBRIC_CRITERIA.items():
            criterion_ids = [c["id"] for c in criteria]
            relevant_results = [
                r for r in criterion_results 
                if r.criterion_id in criterion_ids
            ]
            
            if relevant_results:
                # Weighted average
                total_weight = sum(
                    c["weight"] for c in criteria 
                    if c["id"] in [r.criterion_id for r in relevant_results]
                )
                weighted_sum = sum(
                    r.score * next(c["weight"] for c in criteria if c["id"] == r.criterion_id)
                    for r in relevant_results
                )
                dimension_scores[dimension] = weighted_sum / total_weight if total_weight > 0 else 0.0
        
        return dimension_scores
    
    def get_summary(self) -> Dict:
        """Get summary of all evaluation results."""
        if not self.results:
            return {"error": "No evaluations performed"}
        
        avg_score = sum(r.aggregate_score for r in self.results) / len(self.results)
        
        # Average dimension scores
        dimension_totals = {}
        dimension_counts = {}
        
        for result in self.results:
            for dim, score in result.dimension_scores.items():
                dimension_totals[dim] = dimension_totals.get(dim, 0) + score
                dimension_counts[dim] = dimension_counts.get(dim, 0) + 1
        
        avg_dimensions = {
            dim: dimension_totals[dim] / dimension_counts[dim]
            for dim in dimension_totals
        }
        
        return {
            "total_evaluations": len(self.results),
            "average_score": avg_score,
            "dimension_averages": avg_dimensions,
            "weakest_dimension": min(avg_dimensions, key=avg_dimensions.get),
            "strongest_dimension": max(avg_dimensions, key=avg_dimensions.get)
        }


class StructuredSummarizer:
    """Generate structured summaries with explicit sections."""
    
    TEMPLATE = """## Session Intent
{intent}

## Files Modified
{files_modified}

## Files Read (Not Modified)
{files_read}

## Decisions Made
{decisions}

## Current State
{current_state}

## Next Steps
{next_steps}
"""
    
    def __init__(self):
        self.sections = {
            "intent": "",
            "files_modified": [],
            "files_read": [],
            "decisions": [],
            "current_state": "",
            "next_steps": []
        }
    
    def update_from_span(self, new_content: str) -> str:
        """
        Update summary from newly truncated content span.
        
        This implements anchored iterative summarization:
        - Extract information from new span
        - Merge with existing sections
        - Return updated summary
        """
        # Extract information from new content
        new_info = self._extract_from_content(new_content)
        
        # Merge with existing sections
        self._merge_sections(new_info)
        
        # Generate formatted summary
        return self._format_summary()
    
    def _extract_from_content(self, content: str) -> Dict:
        """Extract structured information from content."""
        extracted = {
            "intent": "",
            "files_modified": [],
            "files_read": [],
            "decisions": [],
            "current_state": "",
            "next_steps": []
        }
        
        # Extract file modifications
        mod_pattern = r"(?:modified|changed|updated|fixed)\s+([^\s]+\.[a-z]+)[:\s]*(.+?)(?:\n|$)"
        for match in re.finditer(mod_pattern, content, re.IGNORECASE):
            extracted["files_modified"].append({
                "path": match.group(1),
                "change": match.group(2).strip()[:100]
            })
        
        # Extract file reads
        read_pattern = r"(?:read|examined|opened|checked)\s+([^\s]+\.[a-z]+)"
        for match in re.finditer(read_pattern, content, re.IGNORECASE):
            file_path = match.group(1)
            if file_path not in [f["path"] for f in extracted["files_modified"]]:
                extracted["files_read"].append(file_path)
        
        # Extract decisions
        decision_pattern = r"(?:decided|chose|going with|will use)\s+(.+?)(?:\n|$)"
        for match in re.finditer(decision_pattern, content, re.IGNORECASE):
            extracted["decisions"].append(match.group(1).strip()[:150])
        
        return extracted
    
    def _merge_sections(self, new_info: Dict):
        """Merge new information with existing sections."""
        # Update intent if empty
        if new_info["intent"] and not self.sections["intent"]:
            self.sections["intent"] = new_info["intent"]
        
        # Merge file lists (deduplicate by path)
        existing_mod_paths = [f["path"] for f in self.sections["files_modified"]]
        for file_info in new_info["files_modified"]:
            if file_info["path"] not in existing_mod_paths:
                self.sections["files_modified"].append(file_info)
        
        # Merge read files
        for file_path in new_info["files_read"]:
            if file_path not in self.sections["files_read"]:
                self.sections["files_read"].append(file_path)
        
        # Append decisions
        self.sections["decisions"].extend(new_info["decisions"])
        
        # Update current state (latest wins)
        if new_info["current_state"]:
            self.sections["current_state"] = new_info["current_state"]
        
        # Merge next steps
        self.sections["next_steps"].extend(new_info["next_steps"])
    
    def _format_summary(self) -> str:
        """Format sections into summary string."""
        files_modified_str = "\n".join(
            f"- {f['path']}: {f['change']}" 
            for f in self.sections["files_modified"]
        ) or "None"
        
        files_read_str = "\n".join(
            f"- {f}" for f in self.sections["files_read"]
        ) or "None"
        
        decisions_str = "\n".join(
            f"- {d}" for d in self.sections["decisions"][-5:]  # Keep last 5
        ) or "None"
        
        next_steps_str = "\n".join(
            f"{i+1}. {s}" for i, s in enumerate(self.sections["next_steps"][-5:])
        ) or "None"
        
        return self.TEMPLATE.format(
            intent=self.sections["intent"] or "Not specified",
            files_modified=files_modified_str,
            files_read=files_read_str,
            decisions=decisions_str,
            current_state=self.sections["current_state"] or "In progress",
            next_steps=next_steps_str
        )


# Usage Example

def evaluate_compression_quality(
    original_history: str,
    compressed_context: str,
    model_response_fn
) -> Dict:
    """
    Evaluate compression quality for a conversation.
    
    Args:
        original_history: The full conversation before compression
        compressed_context: The compressed version
        model_response_fn: Function to get model responses given compressed context
        
    Returns:
        Evaluation summary with scores and recommendations
    """
    # Generate probes
    generator = ProbeGenerator(original_history)
    probes = generator.generate_probes()
    
    # Evaluate each probe
    evaluator = CompressionEvaluator()
    
    for probe in probes:
        # Get model response using compressed context
        response = model_response_fn(compressed_context, probe.question)
        
        # Evaluate response
        evaluator.evaluate(probe, response, compressed_context)
    
    # Get summary
    summary = evaluator.get_summary()
    
    # Add recommendations
    summary["recommendations"] = []
    
    if summary.get("weakest_dimension") == "artifact_trail":
        summary["recommendations"].append(
            "Consider implementing separate artifact tracking outside compression"
        )
    
    if summary["average_score"] < 3.5:
        summary["recommendations"].append(
            "Compression quality is below threshold - consider less aggressive compression"
        )
    
    return summary












Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)