Skill: karmaent/advanced-evaluation (version 1.0.0)

--- name: advanced-evaluation description: This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment. --- # Advanced Evaluation This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems. **Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops. ## When to Activate Activate this skill when: - Building automated evaluation pipelines for LLM outputs - Comparing multiple model responses to select the best one - Establishing consistent quality standards across evaluation teams - Debugging evaluation systems that show inconsistent results - Designing A/B tests for prompt or model changes - Creating rubrics for human or automated evaluation - Analyzing correlation between automated and human judgments ## Core Concepts ### The Evaluation Taxonomy Evaluation approaches fall into two primary categories with distinct reliability profiles: **Direct Scoring**: A single LLM rates one response on a defined scale. - Best for: Objective criteria (factual accuracy, instruction following, toxicity) - Reliability: Moderate to high for well-defined criteria - Failure mode: Score calibration drift, inconsistent scale interpretation **Pairwise Comparison**: An LLM compares two responses and selects the better one. - Best for: Subjective preferences (tone, style, persuasiveness) - Reliability: Higher than direct scoring for preferences - Failure mode: Position bias, length bias Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth. ### The Bias Landscape LLM judges exhibit systematic biases that must be actively mitigated: **Position Bias**: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check. **Length Bias**: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring. **Self-Enhancement Bias**: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation. **Verbosity Bias**: Detailed explanations receive higher scores even when unnecessary. Mitigation: Criteria-specific rubrics that penalize irrelevant detail. **Authority Bias**: Confident, authoritative tone rated higher regardless of accuracy. Mitigation: Require evidence citation, fact-checking layer. ### Metric Selection Framework Choose metrics based on the evaluation task structure: | Task Type | Primary Metrics | Secondary Metrics | |-----------|-----------------|-------------------| | Binary classification (pass/fail) | Recall, Precision, F1 | Cohen's κ | | Ordinal scale (1-5 rating) | Spearman's ρ, Kendall's τ | Cohen's κ (weighted) | | Pairwise preference | Agreement rate, Position consistency | Confidence calibration | | Multi-label | Macro-F1, Micro-F1 | Per-label precision/recall | The critical insight: High absolute agreement matters less than systematic disagreement patterns. A judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise. ## Evaluation Approaches ### Direct Scoring Implementation Direct scoring requires three components: clear criteria, a calibrated scale, and structured output format. **Criteria Definition Pattern**: ``` Criterion: [Name] Description: [What this criterion measures] Weight: [Relative importance, 0-1] ``` **Scale Calibration**: - 1-3 scales: Binary with neutral option, lowest cognitive load - 1-5 scales: Standard Likert, good balance of granularity and reliability - 1-10 scales: High granularity but harder to calibrate, use only with detailed rubrics **Prompt Structure for Direct Scoring**: ``` You are an expert evaluator assessing response quality. ## Task Evaluate the following response against each criterion. ## Original Prompt {prompt} ## Response to Evaluate {response} ## Criteria {for each criterion: name, description, weight} ## Instructions For each criterion: 1. Find specific evidence in the response 2. Score according to the rubric (1-{max} scale) 3. Justify your score with evidence 4. Suggest one specific improvement ## Output Format Respond with structured JSON containing scores, justifications, and summary. ``` **Chain-of-Thought Requirement**: All scoring prompts must require justification before the score. Research shows this improves reliability by 15-25% compared to score-first approaches. ### Pairwise Comparison Implementation Pairwise comparison is inherently more reliable for preference-based evaluation but requires bias mitigation. **Position Bias Mitigation Protocol**: 1. First pass: Response A in first position, Response B in second 2. Second pass: Response B in first position, Response A in second 3. Consistency check: If passes disagree, return TIE with reduced confidence 4. Final verdict: Consistent winner with averaged confidence **Prompt Structure for Pairwise Comparison**: ``` You are an expert evaluator comparing two AI responses. ## Critical Instructions - Do NOT prefer responses because they are longer - Do NOT prefer responses based on position (first vs second) - Focus ONLY on quality according to the specified criteria - Ties are acceptable when responses are genuinely equivalent ## Original Prompt {prompt} ## Response A {response_a} ## Response B {response_b} ## Comparison Criteria {criteria list} ## Instructions 1. Analyze each response independently first 2. Compare them on each criterion 3. Determine overall winner with confidence level ## Output Format JSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning. ``` **Confidence Calibration**: Confidence scores should reflect position consistency: - Both passes agree: confidence = average of individual confidences - Passes disagree: confidence = 0.5, verdict = TIE ### Rubric Generation Well-defined rubrics reduce evaluation variance by 40-60% compared to open-ended scoring. **Rubric Components**: 1. **Level descriptions**: Clear boundaries for each score level 2. **Characteristics**: Observable features that define each level 3. **Examples**: Representative text for each level (optional but valuable) 4. **Edge cases**: Guidance for ambiguous situations 5. **Scoring guidelines**: General principles for consistent application **Strictness Calibration**: - **Lenient**: Lower bar for passing scores, appropriate for encouraging iteration - **Balanced**: Fair, typical expectations for production use - **Strict**: High standards, appropriate for safety-critical or high-stakes evaluation **Domain Adaptation**: Rubrics should use domain-specific terminology. A "code readability" rubric mentions variables, functions, and comments. A "medical accuracy" rubric references clinical terminology and evidence standards. ## Practical Guidance ### Evaluation Pipeline Design Production evaluation systems require multiple layers: ``` ┌─────────────────────────────────────────────────┐ │ Evaluation Pipeline │ ├─────────────────────────────────────────────────┤ │ │ │ Input: Response + Prompt + Context │ │ │ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Criteria Loader │ ◄── Rubrics, weights │ │ └──────────┬──────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Primary Scorer │ ◄── Direct or Pairwise │ │ └──────────┬──────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Bias Mitigation │ ◄── Position swap, etc. │ │ └──────────┬──────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────┐ │ │ │ Confidence Scoring │ ◄── Calibration │ │ └──────────┬──────────┘ │ │ │ │ │ ▼ │ │ Output: Scores + Justifications + Confidence │ │ │ └─────────────────────────────────────────────────┘ ``` ### Common Anti-Patterns **Anti-pattern: Scoring without justification** - Problem: Scores lack grounding, difficult to debug or improve - Solution: Always require evidence-based justification before score **Anti-pattern: Single-pass pairwise comparison** - Problem: Position bias corrupts results - Solution: Always swap positions and check consistency **Anti-pattern: Overloaded criteria** - Problem: Criteria measuring multiple things are unreliable - Solution: One criterion = one measurable aspect **Anti-pattern: Missing edge case guidance** - Problem: Evaluators handle ambiguous cases inconsistently - Solution: Include edge cases in rubrics with explicit guidance **Anti-pattern: Ignoring confidence calibration** - Problem: High-confidence wrong judgments are worse than low-confidence - Solution: Calibrate confidence to position consistency and evidence strength ### Decision Framework: Direct vs. Pairwise Use this decision tree: ``` Is there an objective ground truth? ├── Yes → Direct Scoring │ └── Examples: factual accuracy, instruction following, format compliance │ └── No → Is it a preference or quality judgment? ├── Yes → Pairwise Comparison │ └── Examples: tone, style, persuasiveness, creativity │ └── No → Consider reference-based evaluation └── Examples: summarization (compare to source), translation (compare to reference) ``` ### Scaling Evaluation For high-volume evaluation: 1. **Panel of LLMs (PoLL)**: Use multiple models as judges, aggregate votes - Reduces individual model bias - More expensive but more reliable for high-stakes decisions 2. **Hierarchical evaluation**: Fast cheap model for screening, expensive model for edge cases - Cost-effective for large volumes - Requires calibration of screening threshold 3. **Human-in-the-loop**: Automated evaluation for clear cases, human review for low-confidence - Best reliability for critical applications - Design feedback loop to improve automated evaluation ## Examples ### Example 1: Direct Scoring for Accuracy **Input**: ``` Prompt: "What causes seasons on Earth?" Response: "Seasons are caused by Earth's tilted axis. As Earth orbits the Sun, different hemispheres receive more direct sunlight at different times of year." Criterion: Factual Accuracy (weight: 1.0) Scale: 1-5 ``` **Output**: ```json { "criterion": "Factual Accuracy", "score": 5, "evidence": [ "Correctly identifies axial tilt as primary cause", "Correctly explains differential sunlight by hemisphere", "No factual errors present" ], "justification": "Response accurately explains the cause of seasons with correct scientific reasoning. Both the axial tilt and its effect on sunlight distribution are correctly described.", "improvement": "Could add the specific tilt angle (23.5°) for completeness." } ``` ### Example 2: Pairwise Comparison with Position Swap **Input**: ``` Prompt: "Explain machine learning to a beginner" Response A: [Technical explanation with jargon] Response B: [Simple analogy-based explanation] Criteria: ["clarity", "accessibility"] ``` **First Pass (A first)**: ```json { "winner": "B", "confidence": 0.8 } ``` **Second Pass (B first)**: ```json { "winner": "A", "confidence": 0.6 } ``` (Note: Winner is A because B was in first position) **Mapped Second Pass**: ```json { "winner": "B", "confidence": 0.6 } ``` **Final Result**: ```json { "winner": "B", "confidence": 0.7, "positionConsistency": { "consistent": true, "firstPassWinner": "B", "secondPassWinner": "B" } } ``` ### Example 3: Rubric Generation **Input**: ``` criterionName: "Code Readability" criterionDescription: "How easy the code is to understand and maintain" domain: "software engineering" scale: "1-5" strictness: "balanced" ``` **Output** (abbreviated): ```json { "levels": [ { "score": 1, "label": "Poor", "description": "Code is difficult to understand without significant effort", "characteristics": [ "No meaningful variable or function names", "No comments or documentation", "Deeply nested or convoluted logic" ] }, { "score": 3, "label": "Adequate", "description": "Code is understandable with some effort", "characteristics": [ "Most variables have meaningful names", "Basic comments present for complex sections", "Logic is followable but could be cleaner" ] }, { "score": 5, "label": "Excellent", "description": "Code is immediately clear and maintainable", "characteristics": [ "All names are descriptive and consistent", "Comprehensive documentation", "Clean, modular structure" ] } ], "edgeCases": [ { "situation": "Code is well-structured but uses domain-specific abbreviations", "guidance": "Score based on readability for domain experts, not general audience" } ] } ``` ## Guidelines 1. **Always require justification before scores** - Chain-of-thought prompting improves reliability by 15-25% 2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias 3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions 4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective 5. **Include confidence scores** - Calibrate to position consistency and evidence strength 6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance 7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations 8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment 9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model 10. **Design for iteration** - Evaluation systems improve with feedback loops ## Integration This skill integrates with: - **context-fundamentals** - Evaluation prompts require effective context structure - **tool-design** - Evaluation tools need proper schemas and error handling - **context-optimization** - Evaluation prompts can be optimized for token efficiency - **evaluation** (foundational) - This skill extends the foundational evaluation concepts ## References Internal reference: - [LLM-as-Judge Implementation Patterns](./references/implementation-patterns.md) - [Bias Mitigation Techniques](./references/bias-mitigation.md) - [Metric Selection Guide](./references/metrics-guide.md) External research: - [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) Related skills in this collection: - evaluation - Foundational evaluation concepts - context-fundamentals - Context structure for evaluation prompts - tool-design - Building evaluation tools --- ## Skill Metadata **Created**: 2024-12-24 **Last Updated**: 2024-12-24 **Author**: Muratcan Koylan **Version**: 1.0.0


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)