Solar-JEV-Gemma-embed-2

A calibration-probe model: the frozen text encoder from google/embeddinggemma-2 with three small linear heads trained to decode calibrated decision judgments — noul (yes/no with confidence), choice (best option with probability distribution), and score (0–5 rubric score with distribution) — from compact task/state descriptions.

The heads are distilled from upstage/solar-decide-flash (via the OpenRouter Decisions API), using the paired teacher-probability bank in SargeDev/solar-decisions-corpus-v4.

What this model is (and is not)

This is an experiment, not a production model. The body is fully frozen; only three linear heads were trained. The scientific question it answers:

Is Google's embedding space linearly decodable into calibrated, teacher-distributed judgments?

Answer: yes, substantially. 756k rows of Solar Decide Flash probability distributions were recovered at teacher-level agreement through ~120 KB of trained parameters (three Linear(768→{2,16,6}) heads).

It is not:

• a replacement for Solar Decide Flash
• a full fine-tune (see Limitations)
• wired into any production pipeline

Training

Body	google/embeddinggemma-2, frozen (bf16)
Heads	Linear(768→2) noul, Linear(768→16) choice (padded; masked per-row), Linear(768→6) score
Params trained	11.8k × 3 rows ≈ **12k rows of head weights** (76 KB total)
Data	SargeDev/solar-decisions-corpus-v4 train split: 756,619 rows (= 696,741 v3 rows re-labeled by Solar + 120,000 fresh v4-domain rows — so v3 itself was Jev-1.13-labeled, making this a two-teacher lineage: v3-JEV + v4-Solar)
Teacher	Solar Decide Flash probability distributions (v4 bank)
Loss	per-row KL(teacher ∥ student), soft targets
Input format	`task: classification
Optimizer	AdamW lr 3e-5, wd 0.01, cosine schedule, 200 warmup steps, grad clip 1.0, batch 32 × accum 2
Hardware	1× RTX 3060 12 GB, bf16 autocast
Wall time	~2.8 h @ 74 rows/s
Seed	20261006

KL trajectory (train): 0.428 → 0.0285 — clean convergence, no late drift.

Results (inference-only, run once on each full split)

Overall

Split	n	MAE (sum of per-option abs error)	Argmax agreement	Wall
Validation (full)	18,813	0.668	54.12%	3.9 min
Test (full)	10,000	0.665	55.79%	2.4 min

Per-kind (full test split)

Kind	n	MAE	Argmax agree
noul	4,973	0.568	66.46%
choice	2,619	0.812	45.40%
score	2,408	0.706	45.06%

Context for reading these numbers:

• Solar Decide Flash itself agrees with the v3 Jev-1.13 gold labels at ~58% argmax on choice — a different calibrated teacher is expected to disagree ~40–45% of the time. The probe's 55.8% overall agreement should be read against that backdrop, not against 100%.
• The v4 corpus spans 90 domains and 3 judgment kinds. This is a generalist probe; specialists (e.g. the cascade's production relevance student) reach 81.7% on their single narrow task.

Training dataset

Trained on SargeDev/solar-decisions-corpus-v4 (816,741 rows — v3 states re-labeled by Solar Decide Flash + 120k fresh-domain questions) and, via the v3 lineage, SargeDev/jev-distill-corpus-v3. Two independent teachers over overlapping state/question space (Jev 1.13 + Solar Decide Flash).

Intended use

• Research into whether decoder-style calibrated judgments are recoverable from frozen embedding spaces (what-if experiment).
• Baseline for future full-finetune runs (the 142h/3060 full-FT estimate for this corpus makes the frozen probe the right first experiment).
• Educational reference for KL-distillation of soft distributions onto frozen encoders.

Limitations

• Frozen body — head capacity is the ceiling. Choice/score tasks (8–16-way distributions) show the most headroom; noul (binary) is already near what a frozen-space linear head can express.
• Trained on one epoch; no hyperparameter sweep.
• MAXLEN 384 — long states are truncated.
• Choice vocabulary padded to 16 (the corpus-wide max) — real per-row option counts vary 2–16.
• Not calibrated against out-of-distribution inputs — the ood split was not evaluated (future work).

How to use

from huggingface_hub import snapshot_download
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer, torch.nn

path = snapshot_download("SargeDev/Solar-JEV-Gemma-embed-2")
tok = AutoTokenizer.from_pretrained("google/embeddinggemma-2")
body = AutoModel.from_pretrained("google/embeddinggemma-2", dtype=torch.bfloat16).to("cuda")
heads = torch.load(f"{path}/heads.pt")
heads_n = torch.nn.Linear(768, 2).to("cuda"); heads_n.load_state_dict(heads["noul"])
heads_c = torch.nn.Linear(768, 16).to("cuda"); heads_c.load_state_dict(heads["choice"])
heads_s = torch.nn.Linear(768, 6).to("cuda"); heads_s.load_state_dict(heads["score"])

state = "A server room's cooling unit fails during peak load..."
question = "Given the server state, is immediate operator intervention required?"
enc = tok([f"task: classification | query: {question}\n{state}"],
          padding=True, truncation=True, max_length=384, return_tensors="pt")
with torch.no_grad():
    o = body(input_ids=enc["input_ids"].to("cuda"), attention_mask=enc["attention_mask"].to("cuda"))
    h = o.last_hidden_state
    m = enc["attention_mask"].to("cuda").unsqueeze(-1)
    h = (h * m).sum(1) / m.sum(1).clamp(min=1)
    noul_logits = heads_n(h.float())
    probs = F.softmax(noul_logits, dim=-1)  # [P(no), P(yes)]

Attribution

Corpus and method: SargeDev and contributors. The training bank comes from SargeDev/solar-decisions-corpus-v4 — 85% of the rows are the original SargeDev/jev-distill-corpus-v3 (Jev 1.13-labeled) questions/states re-labeled by Solar, plus 120k fresh Solar-labeled domains — making this model's training data a direct two-teacher lineage (Jev 1.13 states, Solar Decide Flash targets). If you use this model or its training corpus, please credit both prior datasets and link back.

Teacher: Solar Decide Flash (OpenRouter Decisions API, upstage/solar-decide-flash) labels the soft probability targets. Base encoder: google/embeddinggemma-2 (Apache-2.0 / Gemma Terms of Use — see base model license).

License

Apache-2.0. The frozen base encoder inherits its own upstream terms (Gemma); review those if redistributing.