# 03 — Benchmarks

> As of **2026-07-21**. Prefer Hub card + blog + [trajectories.poolside.ai](https://trajectories.poolside.ai). Harness: **pool** (Harbor fork). Max **500** steps.

## Headline (thinking max)

| Bench | Laguna S 2.1 |
|---|---:|
| Terminal-Bench 2.1 | **70.2** |
| SWE-Bench Multilingual | **78.5** |
| SWE-Bench Pro (Public) | **59.4** |
| DeepSWE v1.1 | **40.4** |
| SWE Atlas (Codebase QnA) | **46.2** |
| Toolathlon Verified | **49.7** |

## Thinking vs not

| Bench | no-think | max think | Δ |
|---|---:|---:|---:|
| Terminal-Bench 2.1 | 60.4 | 70.2 | +9.8 |
| DeepSWE v1.1 | 16.5 | 40.4 | +23.9 |

Blog: largest thinking vs no-thinking quality delta of any Laguna to date.

## Peer table (blog / card excerpt)

| Model | Size | TB 2.1 | SWE Multi | SWE Pro | DeepSWE |
|---|---|---:|---:|---:|---:|
| **Laguna S 2.1** | 118B-A8B | **70.2** | **78.5** | **59.4** | **40.4** |
| Hy3 | 295B-A21B | 71.7 | 75.8 | 57.9 | — |
| Inkling | 975B-A41B | 63.8 | — | 54.3 | — |
| DeepSeek-V4-Pro Max | 1.6T-A49B | 64.0* | 76.2 | 55.4 | 9.0* |
| Kimi K3 | 2.8T-A50B† | 88.3 | — | — | 69.0 |
| Claude Fable 5 | — | 88.0 | — | 80.3 | 70.0 |
| Laguna XS 2.1 | 33B-A3B | 33.4 | — | — | 0.3 |

\* third-party as marked on card.  
† Poolside active listing; monorepo keeps Kimi active **undisclosed** until Moonshot primary docs confirm.

**Read carefully:** S 2.1 sits near Hy3 and above Inkling / DeepSeek-V4-Pro-Max on TB 2.1 while trailing closed frontier leaders. DeepSWE is the more discriminating long-horizon set (scores spread; many 1T+ open models &lt; 10% on vendor tables).

## Method notes

| Item | Detail |
|---|---|
| Metric | Mean pass@1 over k attempts |
| k | 4 for TB2.1 / SWE Multi / SWE Pro; 3 for DeepSWE / Atlas / Toolathlon |
| Internet | On for many evals; anti-reward-hack addendum on SWE family after LLMaaJ flags |
| DeepSWE caveat | Reported in **pool**, not mini-swe-agent (leaderboard default) — less comparable |
| Trajectories | Full final runs public |
| Reward-hack monitoring | LLMaaJ + manual review; rates driven &lt;2% after prompt addendum |

## Case studies (qualitative, blog)

1. **Browser engine** — 50 min / 181 steps from empty folder; no vision; validates via Chromium canvas diffs  
2. **Harness optimize** — +5.2% wall-clock, ~70% lower allocations  
3. **Erdős #397** — offline Perl rediscovery; different construction family from prior GPT-5.2 Pro solution  

## Study rule

Do not invent peer scores. Re-fetch card/blog if numbers move. For matched agentic benches against Inkling/Kimi, prefer shared public leaderboards + released trajectories — not secondary recaps alone.