Skill: anderskev/llm-judge (version 1.0.3)

--- name: llm-judge description: "Use when comparing two or more code implementations against a spec or requirements doc. Triggers on \"which repo is better\", \"compare these implementations\", \"evaluate both solutions\", \"rank these codebases\", or \"judge which approach wins\". Also covers choosing between competing PRs or vendor submissions solving the same problem. Does NOT review a single codebase for quality \u2014 use code review skills instead. Does NOT evaluate strategy docs \u2014 use strategy-review. Requires a spec file and 2+ repo paths." disable-model-invocation: true --- # LLM Judge Compare code implementations across multiple repositories using structured evaluation. ## Usage ```bash /beagle-analysis:llm-judge <spec> <repo1> <repo2> [repo3...] [--labels=...] [--weights=...] [--branch=...] ``` ## Arguments | Argument | Required | Description | |----------|----------|-------------| | `spec` | Yes | Path to spec/requirements document | | `repos` | Yes | 2+ paths to repositories to compare | | `--labels` | No | Comma-separated labels (default: directory names) | | `--weights` | No | Override weights, e.g. `functionality:40,security:30` | | `--branch` | No | Branch to compare against main (default: `main`) | ## Workflow 1. Parse `$ARGUMENTS` into `spec_path`, `repo_paths`, `labels`, `weights`, and `branch`. 2. Validate the spec file, each repo path, and the minimum repo count. 3. Read the spec document into memory. 4. Load this skill and the supporting reference files. 5. Spawn one Phase 1 repo agent per repository to gather facts only. 6. Validate the repo-agent JSON results before proceeding. 7. Spawn one Phase 2 judge agent per dimension. 8. Aggregate scores, compute weighted totals, rank repos, and write the report. 9. Display the markdown summary and verify the JSON report. ## Hard gates Sequenced workflow: **do not start the next phase until the current gate passes.** Each pass condition must be checkable (file on disk, non-empty content, or `json.load` succeeds)—not “I reviewed internally.” | Gate | Pass condition | Unblocks | |------|----------------|----------| | **A — Inputs** | `spec_path` is a readable file and non-empty; `len(repo_paths) ≥ 2`; each path contains `.git`. | Phase 1 repo agents | | **B — Phase 1 facts** | For **each** repo agent output: stdin/stdout parses as JSON; required keys/shape match `references/fact-schema.md`. | Phase 2 judge agents | | **C — Phase 2 scores** | **Five** judge outputs (one per dimension) each parse as JSON; each includes a score (and justification) for **every** repo label. | Aggregation | | **D — Report file** | `.beagle/llm-judge-report.json` exists; `python3 -c "import json; json.load(open('.beagle/llm-judge-report.json'))"` exits 0. | Markdown summary to the user | | **E — Consistency** | Summary table and verdict use the same labels, weights, and per-dimension scores as the JSON report. | Mark task complete | Parallelism is allowed **within** a phase (all Phase 1 tasks together; all Phase 2 tasks together), but Phase 2 must not start until Gate B passes, and the user-visible summary must not precede Gate D. ## Command Workflow ### Step 1: Parse Arguments Parse `$ARGUMENTS` to extract: - `spec_path`: first positional argument - `repo_paths`: remaining positional arguments (must be 2+) - `labels`: from `--labels` or derived from directory names - `weights`: from `--weights` or defaults - `branch`: from `--branch` or `main` **Default Weights:** ```json { "functionality": 30, "security": 25, "tests": 20, "overengineering": 15, "dead_code": 10 } ``` ### Step 2: Validate Inputs ```bash [ -f "$SPEC_PATH" ] || { echo "Error: Spec file not found: $SPEC_PATH"; exit 1; } for repo in "${REPO_PATHS[@]}"; do [ -d "$repo/.git" ] || { echo "Error: Not a git repository: $repo"; exit 1; } done [ ${#REPO_PATHS[@]} -ge 2 ] || { echo "Error: Need at least 2 repositories to compare"; exit 1; } ``` ### Step 3: Read Spec Document ```bash SPEC_CONTENT=$(cat "$SPEC_PATH") || { echo "Error: Failed to read spec file: $SPEC_PATH"; exit 1; } [ -z "$SPEC_CONTENT" ] && { echo "Error: Spec file is empty: $SPEC_PATH"; exit 1; } ``` ### Step 4: Load the Skill Load the llm-judge skill: `Skill(skill: "beagle-analysis:llm-judge")` ### Step 5: Phase 1 - Spawn Repo Agents Spawn one Task per repo: ```text You are a Phase 1 Repo Agent for the LLM Judge evaluation. **Your Repo:** $LABEL at $REPO_PATH **Spec Document:** $SPEC_CONTENT **Instructions:** 1. Load skill: Skill(skill: "beagle-analysis:llm-judge") 2. Read references/repo-agent.md for detailed instructions 3. Read references/fact-schema.md for the output format 4. Load Skill(skill: "beagle-core:llm-artifacts-detection") for analysis Explore the repository and gather facts. Return ONLY valid JSON following the fact schema. Do NOT score or judge. Only gather facts. ``` Collect all repo outputs into `ALL_FACTS`. ### Step 6: Validate Phase 1 Results ```bash echo "$FACTS" | python3 -c "import json,sys; json.load(sys.stdin)" 2>/dev/null || { echo "Error: Invalid JSON from $LABEL"; exit 1; } ``` ### Step 7: Phase 2 - Spawn Judge Agents Spawn five judge agents, one per dimension: ```text You are the $DIMENSION Judge for the LLM Judge evaluation. **Spec Document:** $SPEC_CONTENT **Facts from all repos:** $ALL_FACTS_JSON **Instructions:** 1. Load skill: Skill(skill: "beagle-analysis:llm-judge") 2. Read references/judge-agents.md for detailed instructions 3. Read references/scoring-rubrics.md for the $DIMENSION rubric Score each repo on $DIMENSION. Return ONLY valid JSON with scores and justifications. ``` ### Step 8: Aggregate Scores ```python for repo_label in labels: scores[repo_label] = {} for dimension in dimensions: scores[repo_label][dimension] = judge_outputs[dimension]['scores'][repo_label] weighted_total = sum( scores[repo_label][dim]['score'] * weights[dim] / 100 for dim in dimensions ) scores[repo_label]['weighted_total'] = round(weighted_total, 2) ranking = sorted(labels, key=lambda l: scores[l]['weighted_total'], reverse=True) ``` ### Step 9: Generate Verdict Name the winner, explain why they won, and note any close calls or trade-offs. ### Step 10: Write JSON Report ```bash mkdir -p .beagle ``` Write `.beagle/llm-judge-report.json` with version, timestamp, repo metadata, weights, scores, ranking, and verdict. ### Step 11: Display Summary Render a markdown summary with the scores table, ranking, verdict, and detailed justifications. ### Step 12: Verification ```bash python3 -c "import json; json.load(open('.beagle/llm-judge-report.json'))" && echo "Valid report" ``` ### Output Shape The generated report should include: - repo labels and paths - per-dimension scores and justifications - weighted totals and ranking - a verdict explaining the winner ## Reference Files | File | Purpose | |------|---------| | [references/fact-schema.md](references/fact-schema.md) | JSON schema for Phase 1 facts | | [references/scoring-rubrics.md](references/scoring-rubrics.md) | Detailed rubrics for each dimension | | [references/repo-agent.md](references/repo-agent.md) | Instructions for Phase 1 agents | | [references/judge-agents.md](references/judge-agents.md) | Instructions for Phase 2 judges | ## Scoring Model | Dimension | Default Weight | Evaluates | |-----------|----------------|-----------| | Functionality | 30% | Spec compliance, test pass rate | | Security | 25% | Vulnerabilities, security patterns | | Test Quality | 20% | Coverage, DRY, mock boundaries | | Overengineering | 15% | Unnecessary complexity | | Dead Code | 10% | Unused code, TODOs | ## Scoring Scale | Score | Meaning | |-------|---------| | 5 | Excellent - Exceeds expectations | | 4 | Good - Meets requirements, minor issues | | 3 | Average - Functional but notable gaps | | 2 | Below Average - Significant issues | | 1 | Poor - Fails basic requirements | ## Phase 1: Spawning Repo Agents For each repository, spawn a Task agent with: ```text You are a Phase 1 Repo Agent for the LLM Judge evaluation. **Your Repo:** $REPO_LABEL at $REPO_PATH **Spec Document:** $SPEC_CONTENT **Instructions:** Read @beagle:llm-judge references/repo-agent.md Gather facts and return a JSON object following the schema in references/fact-schema.md. Load @beagle:llm-artifacts-detection for dead code and overengineering analysis. Return ONLY valid JSON, no markdown or explanations. ``` Collect all repo-agent outputs into `ALL_FACTS`. ## Phase 2: Spawning Judge Agents After all Phase 1 agents complete, spawn 5 judge agents, one per dimension: ```text You are the $DIMENSION Judge for the LLM Judge evaluation. **Spec Document:** $SPEC_CONTENT **Facts from all repos:** $ALL_FACTS_JSON **Instructions:** Read @beagle:llm-judge references/judge-agents.md Score each repo on $DIMENSION using the rubric in references/scoring-rubrics.md. Return ONLY valid JSON following the judge output schema. ``` ## Aggregation 1. Collect the five judge outputs. 2. Compute each repo's weighted total with the configured weights. 3. Rank repos by weighted total in descending order. 4. Generate a verdict that explains the result and any close calls. 5. Write `.beagle/llm-judge-report.json`. ## Output Display a markdown summary with scores, ranking, verdict, and detailed justifications. ## Verification Before completing (maps to **Hard gates** D and E): 1. **Gate D:** `.beagle/llm-judge-report.json` exists and `json.load` succeeds. 2. **Gate E / completeness:** Every repo label has scores for every dimension; each `weighted_total` equals the sum over dimensions of `(score × weight / 100)` using the configured weights; markdown summary matches the JSON report. ## Rules - Always validate inputs before proceeding - Spawn Phase 1 agents in parallel, then wait before Phase 2 - Spawn Phase 2 agents in parallel, one per dimension - Every score must have a justification - Write the JSON report before displaying the summary

path: references/fact-schema.md
content: # Fact Schema

JSON schema for structured facts gathered by Phase 1 Repo Agents.

## Full Schema

```json
{
  "repo_label": "string - Display name for this repo",
  "repo_path": "string - Absolute path to repo",
  "git_info": {
    "branch": "string - Current branch name",
    "base": "string - Base branch (usually main)",
    "files_changed": "number - Count of changed files",
    "additions": "number - Lines added",
    "deletions": "number - Lines deleted",
    "diff_summary": "string - Brief description of changes"
  },
  "functionality": {
    "spec_requirements": ["array of requirement strings extracted from spec"],
    "implemented": ["array of requirements found implemented"],
    "missing": ["array of requirements not found"],
    "partially_implemented": ["array of requirements with incomplete implementation"],
    "test_results": {
      "ran": "boolean - Whether tests were executed",
      "framework": "string - pytest, jest, go test, etc.",
      "passed": "number",
      "failed": "number",
      "skipped": "number",
      "error_summary": "string - Brief description of failures if any"
    }
  },
  "security": {
    "findings": [
      {
        "file": "string - File path",
        "line": "number - Line number",
        "issue": "string - Description of security issue",
        "severity": "high | medium | low",
        "category": "string - OWASP category if applicable"
      }
    ],
    "patterns_observed": ["array of positive security patterns found"]
  },
  "tests": {
    "test_count": "number - Total test count",
    "coverage_estimate": "none | low | moderate | high",
    "dry_violations": [
      {
        "file": "string",
        "line": "number",
        "description": "string"
      }
    ],
    "mocking_approach": "string - Description of mocking strategy",
    "test_quality_notes": "string - General observations"
  },
  "overengineering": {
    "abstractions": [
      {
        "file": "string",
        "line": "number",
        "issue": "string - Description of over-abstraction"
      }
    ],
    "defensive_code": [
      {
        "file": "string",
        "line": "number",
        "issue": "string"
      }
    ],
    "config_complexity": "low | medium | high"
  },
  "dead_code": {
    "unused_imports": ["array of file:line references"],
    "unused_functions": ["array of file:line references"],
    "unused_variables": ["array of file:line references"],
    "todo_comments": "number - Count of TODO/FIXME",
    "commented_code_blocks": "number - Count of commented code"
  }
}
```

## Example

```json
{
  "repo_label": "Claude",
  "repo_path": "/path/to/repo-a",
  "git_info": {
    "branch": "main",
    "base": "main",
    "files_changed": 42,
    "additions": 1250,
    "deletions": 380,
    "diff_summary": "Adds auth flow and data export features"
  },
  "functionality": {
    "spec_requirements": ["auth flow", "data export", "rate limiting"],
    "implemented": ["auth flow", "data export"],
    "missing": ["rate limiting"],
    "partially_implemented": [],
    "test_results": {
      "ran": true,
      "framework": "pytest",
      "passed": 45,
      "failed": 2,
      "skipped": 1,
      "error_summary": "2 tests fail on edge case validation"
    }
  },
  "security": {
    "findings": [
      {
        "file": "src/api.py",
        "line": 42,
        "issue": "SQL string concatenation instead of parameterized query",
        "severity": "high",
        "category": "Injection"
      }
    ],
    "patterns_observed": ["input validation present", "no secrets in code", "HTTPS enforced"]
  },
  "tests": {
    "test_count": 48,
    "coverage_estimate": "moderate",
    "dry_violations": [
      {
        "file": "tests/test_api.py",
        "line": 15,
        "description": "Setup code repeated in 5 test functions"
      }
    ],
    "mocking_approach": "Mocks at adapter boundary, uses pytest fixtures",
    "test_quality_notes": "Good isolation, some DRY issues"
  },
  "overengineering": {
    "abstractions": [
      {
        "file": "src/factory.py",
        "line": 1,
        "issue": "Factory pattern for single implementation"
      }
    ],
    "defensive_code": [],
    "config_complexity": "low"
  },
  "dead_code": {
    "unused_imports": ["src/utils.py:3"],
    "unused_functions": [],
    "unused_variables": [],
    "todo_comments": 2,
    "commented_code_blocks": 1
  }
}
```


path: references/judge-agents.md
content: # Judge Agent Instructions

Instructions for Phase 2 agents that score implementations on a single dimension.

## Role

You are a scoring judge. You receive facts gathered from ALL repositories and score each one on YOUR specific dimension using the rubrics in [scoring-rubrics.md](scoring-rubrics.md).

## Inputs You Receive

1. **Spec Document**: The original requirements
2. **Facts Array**: JSON facts from all repos (output of Phase 1)
3. **Your Dimension**: One of: functionality, security, tests, overengineering, dead_code

## Your Task

Produce a JSON object with scores and justifications for each repo.

## Output Schema

```json
{
  "dimension": "functionality",
  "scores": {
    "RepoLabel1": {
      "score": 4,
      "justification": "Clear explanation of why this score was assigned",
      "evidence": ["Specific facts that support this score"]
    },
    "RepoLabel2": {
      "score": 5,
      "justification": "...",
      "evidence": ["..."]
    }
  },
  "ranking": ["RepoLabel2", "RepoLabel1"],
  "notes": "Optional comparative notes"
}
```

## Scoring Process

1. Read the rubric for your dimension from [scoring-rubrics.md](scoring-rubrics.md)
2. For each repo's facts:
   - Extract the relevant section (e.g., `facts.functionality` for functionality judge)
   - Apply the rubric criteria
   - Assign a 1-5 score
   - Write a clear justification citing specific evidence
3. Rank the repos by score (highest first)

## Dimension-Specific Instructions

### Functionality Judge

Focus on `facts.functionality`:
- Compare `spec_requirements` to `implemented` and `missing`
- Weight test results heavily (`test_results.passed` vs `failed`)
- Consider `partially_implemented` as half credit

### Security Judge

Focus on `facts.security`:
- Count and weight `findings` by severity
- High severity = major deduction
- Positive `patterns_observed` can offset minor issues

### Tests Judge

Focus on `facts.tests`:
- Evaluate `coverage_estimate`
- Count `dry_violations` (more = worse)
- Consider `mocking_approach` quality
- Raw `test_count` relative to codebase size

### Overengineering Judge

Focus on `facts.overengineering`:
- Count `abstractions` issues
- Count `defensive_code` issues
- Consider `config_complexity`
- FEWER issues = HIGHER score (inverse)

### Dead Code Judge

Focus on `facts.dead_code`:
- Sum all unused items
- Weight `unused_functions` > `unused_imports`
- Count `todo_comments` and `commented_code_blocks`
- FEWER issues = HIGHER score (inverse)

## Important Rules

1. **Use the rubric** - Don't invent criteria
2. **Be consistent** - Apply the same standards to all repos
3. **Cite evidence** - Every score needs justification from facts
4. **Be comparative** - Rankings should reflect relative quality
5. **Valid JSON only** - Output must be parseable


path: references/repo-agent.md
content: # Repo Agent Instructions

Instructions for Phase 1 agents that gather facts from a single repository.

## Role

You are a fact-gathering agent. Your job is to explore a repository and extract structured facts WITHOUT making judgments or assigning scores. Scoring happens in Phase 2 by separate judge agents.

## Inputs You Receive

1. **Spec Document**: The requirements/plan that was given to the LLM to implement
2. **Repo Path**: Absolute path to the repository you're analyzing
3. **Repo Label**: Display name for this repo (e.g., "Claude", "GPT-4")
4. **Branch Info**: Which branch to compare (default: current vs main)

## Your Task

Produce a JSON object following the schema in [fact-schema.md](fact-schema.md).

## Step-by-Step Process

### 1. Gather Git Info

```bash
# Get branch name
git -C $REPO_PATH rev-parse --abbrev-ref HEAD

# Get diff stats
git -C $REPO_PATH diff --stat main...HEAD

# Count files changed
git -C $REPO_PATH diff --name-only main...HEAD | wc -l
```

### 2. Analyze Functionality

1. Read the spec document carefully
2. Extract discrete requirements as a list
3. Explore the codebase to determine which requirements are implemented
4. Run tests if available:

```bash
# Detect and run tests
cd $REPO_PATH

# Python
if [ -f pytest.ini ] || [ -f pyproject.toml ] || [ -d tests ]; then
  pytest --tb=short 2>&1
fi

# JavaScript/TypeScript
if [ -f package.json ]; then
  npm test 2>&1 || yarn test 2>&1
fi

# Go
if [ -f go.mod ]; then
  go test ./... 2>&1
fi
```

### 3. Analyze Security

Look for common vulnerabilities:
- SQL injection (string concatenation in queries)
- Command injection (unsanitized shell commands)
- XSS (unsanitized user input in HTML)
- Hardcoded secrets (API keys, passwords)
- Missing input validation
- Insecure deserialization

Also note positive patterns:
- Input validation present
- Parameterized queries
- Authentication checks
- Rate limiting

### 4. Analyze Tests

- Count test files and test functions
- Look for DRY violations (repeated setup code)
- Assess mocking strategy
- Estimate coverage (file count ratio, critical paths tested)

### 5. Analyze Overengineering

Use patterns from `@beagle:llm-artifacts-detection`:
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding
- Over-configuration

### 6. Analyze Dead Code

- Unused imports (grep for imports, check usage)
- TODO/FIXME comments
- Commented-out code blocks
- Unused functions/variables

## Output Format

Return ONLY the JSON object. No markdown, no explanations. The JSON must be valid and follow [fact-schema.md](fact-schema.md).

## Important Rules

1. **Do not score** - Only gather facts
2. **Be thorough** - Check all changed files
3. **Be specific** - Include file:line references
4. **Be objective** - Report what you find, not opinions
5. **Use the skill** - Load `@beagle:llm-artifacts-detection` for dead code/overengineering


path: references/scoring-rubrics.md
content: # Scoring Rubrics

Detailed rubrics for each of the 5 judging dimensions. Judges use these to assign consistent 1-5 scores.

## General Scoring Scale

| Score | Meaning | General Criteria |
|-------|---------|------------------|
| 5 | Excellent | Exceeds expectations, best practices throughout |
| 4 | Good | Meets all requirements, minor issues only |
| 3 | Average | Functional but notable gaps or issues |
| 2 | Below Average | Significant issues affecting quality |
| 1 | Poor | Fails to meet basic requirements |

---

## Functionality (30% weight)

Evaluates whether the implementation meets the spec requirements and works correctly.

| Score | Criteria |
|-------|----------|
| 5 | All spec requirements implemented. All tests pass. No obvious bugs. |
| 4 | All requirements implemented. Tests pass with minor failures (< 5%). Edge cases may be missing. |
| 3 | Most requirements implemented (> 75%). Some test failures. Core functionality works. |
| 2 | Partial implementation (50-75%). Significant test failures. Core features have bugs. |
| 1 | Minimal implementation (< 50%). Tests fail or don't exist. Core functionality broken. |

**Key Evidence:**
- `functionality.implemented` vs `functionality.spec_requirements`
- `functionality.test_results.passed` vs `functionality.test_results.failed`
- `functionality.missing` and `functionality.partially_implemented`

---

## Security (25% weight)

Evaluates security posture and absence of vulnerabilities.

| Score | Criteria |
|-------|----------|
| 5 | No security findings. Positive security patterns present. OWASP Top 10 addressed. |
| 4 | No high-severity findings. 1-2 low/medium issues. Good security hygiene. |
| 3 | 1-2 medium-severity issues OR 3+ low-severity. Basic security present. |
| 2 | 1+ high-severity issue OR 3+ medium. Security gaps evident. |
| 1 | Multiple high-severity issues. Critical vulnerabilities. No security consideration. |

**Severity Weights:**
- High: SQL injection, command injection, auth bypass, secrets in code
- Medium: XSS, CSRF, insecure deserialization, missing input validation
- Low: Information disclosure, verbose errors, missing security headers

**Key Evidence:**
- `security.findings` (count and severity)
- `security.patterns_observed`

---

## Test Quality (20% weight)

Evaluates test coverage, DRY adherence, and testing practices.

| Score | Criteria |
|-------|----------|
| 5 | High coverage. No DRY violations. Good mock boundaries. Tests are maintainable. |
| 4 | Moderate-high coverage. Minor DRY issues (1-2). Good testing practices. |
| 3 | Moderate coverage. Some DRY violations (3-5). Acceptable mocking. |
| 2 | Low coverage. Significant DRY violations. Poor mock boundaries. |
| 1 | Minimal/no tests. Severe DRY problems. Tests don't follow best practices. |

**Key Evidence:**
- `tests.coverage_estimate`
- `tests.dry_violations` (count)
- `tests.mocking_approach`
- `tests.test_count` relative to codebase size

---

## Overengineering (15% weight)

Evaluates simplicity and absence of unnecessary complexity.

| Score | Criteria |
|-------|----------|
| 5 | Clean, simple code. No unnecessary abstractions. YAGNI followed. |
| 4 | Mostly simple. 1-2 minor over-abstractions. Code is readable. |
| 3 | Some complexity. 3-5 abstraction issues. Config complexity medium. |
| 2 | Significant over-engineering. 6+ abstraction issues. Unnecessary patterns. |
| 1 | Severely over-engineered. Abstractions everywhere. Simple tasks made complex. |

**Key Evidence:**
- `overengineering.abstractions` (count)
- `overengineering.defensive_code` (count)
- `overengineering.config_complexity`

---

## Dead Code (10% weight)

Evaluates cleanliness and absence of unused/obsolete code.

| Score | Criteria |
|-------|----------|
| 5 | No dead code. No TODOs. Clean codebase. |
| 4 | 1-3 minor issues (unused imports). No significant dead code. |
| 3 | 4-6 issues. Some unused functions or TODOs. |
| 2 | 7-10 issues. Unused functions/classes. Multiple TODOs. |
| 1 | 10+ issues. Significant dead code. Many TODOs/commented blocks. |

**Key Evidence:**
- `dead_code.unused_imports` (count)
- `dead_code.unused_functions` (count)
- `dead_code.todo_comments`
- `dead_code.commented_code_blocks`


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)