cnfeichen/minimind-jev-v0.1

minimind-jev v0.1 — a typed-decision head you can train in 3 minutes

First public checkpoint of an educational project: train your own miniature Jev-style decision model for Chinese RAG reranking. Given (query, passage), the head predicts P(passage answers query) — a Noul-type calibrated answer in the System One sense, produced by a model you trained yourself.

What was actually trained: a 3-layer MLP head (512 → 256 → 256 → 1, ~660K params), from scratch. The text encoder is frozen BAAI/bge-small-zh-v1.5 (MIT) and is not included in this release — the loader pulls it from the Hub.

Results

Same protocol as jev-rag-zh-benchmark (s1 sample: 100 queries, hard-negative pools, 95% bootstrap CI over queries; DOI 10.5281/zenodo.23322277):

system	nDCG@5	nDCG@10	MRR@10	R@1
hosted Jev score-zh (v1.13)	0.546	0.602	0.477	0.26
hosted Jev noul-zh (v1.13)	0.531	0.588	0.458	0.24
minimind-jev v0.1 (this)	0.488	0.576	0.445	0.23
BM25 (jieba + rank_bm25)	0.413	0.513	0.363	0.11

Paired bootstrap (n=1000): minimind-jev − BM25 nDCG@5 +0.075 [+0.002, +0.154] (CI excludes 0); hosted Jev − minimind-jev +0.059 [−0.040, +0.157] (CI includes 0 — not separable at n=100). Three minutes of training on a laptop RTX 4060 (8GB), zero API cost.

Training

• Data: T2Reranking hard negatives (C-MTEB / DuReader-derived); 15,660 train pairs / 1,730 val pairs; the benchmark's eval queries are excluded from training (zero leakage).
• Objective: BCE on the head, encoder frozen; temperature scaling fitted on val (T = 1.031).
• 5 epochs, seed 42, lr 1e-3, AdamW. Wall-clock ≈ 3 min including encoding 17,390 texts.
• Question wording frozen from the awesome-jev-prompts pattern library.

Usage

import torch, torch.nn as nn
from sentence_transformers import SentenceTransformer

class DecisionHead(nn.Module):
    def __init__(self, dim: int, hidden: int = 256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, hidden), nn.GELU(),
            nn.Linear(hidden, hidden), nn.GELU(),
            nn.Linear(hidden, 1),
        )
    def forward(self, x): return self.net(x).squeeze(-1)

enc = SentenceTransformer("BAAI/bge-small-zh-v1.5")
ckpt = torch.load("head.pt", map_location="cpu")
head = DecisionHead(ckpt["dim"]); head.load_state_dict(ckpt["state"]); head.eval()

QUESTION = "仅凭该段落能回答该查询吗？"
def score(query: str, passage: str) -> float:
    x = torch.tensor(enc.encode([f"{query}\n{QUESTION}\n{passage}"],
                                normalize_embeddings=True))
    with torch.no_grad():
        return torch.sigmoid(head(x) / ckpt["temp"]).item()

Limitations (read before using)

• The frozen encoder sets the ceiling: this is bge-small-zh's understanding plus a trained decision surface, not a from-scratch model (that is the project's v0.3).
• Binary relevance only; separation AUC 0.630 vs hosted Jev's 0.714.
• Output probabilities sit close to the class prior (T = 1.031 barely shifts them): use the scores for ranking, not as calibrated absolute probabilities.
• Educational artifact. Not a production reranker.

Provenance

Eval protocol, logs and the three-way analysis: jev-rag-zh-benchmark · DOI 10.5281/zenodo.23322277. Question design: awesome-jev-prompts. A full Chinese tutorial (from-scratch tiny transformer, distilled multi-task v0.2) is in private build and will be published with chapters.

中文：这是 minimind-jev 项目的首个公开检查点——冻结 bge-small-zh 编码器 + 从零训练的 MLP 决策头，3 分钟笔记本训练即达：显著胜 BM25、与 hosted Jev-1.13 在 n=100 下统计不可区分。 仅用于排序场景（概率贴近类先验）；教育向产物，非生产级重排器。评测协议与日志见 jev-rag-zh-benchmark（带 DOI）。

License

Head weights: MIT (this release). Encoder: BAAI/bge-small-zh-v1.5, MIT, loaded separately. Training data: C-MTEB/T2Reranking (DuReader-derived), per its own terms.