LNTTushar/trynmini-v2-static-7m

TrynMini v2 — a true static embedding model (~7.7M params, ~7.8 MB int8)

A distilled static sentence embedding model. Inference is tokenize → table lookup → Zipf-weighted mean pool → residual → L2-norm — no attention, no transformer layers, CPU-friendly, numpy-only. Supports Matryoshka truncation to 64 / 128 / 256 dims.

This is v2 of the "2D-Matryoshka Static Logic-Constraint" project. v1 marketed itself as static but ran nn.MultiheadAttention at inference; v2 fixes that by distilling a static token table (from v1's trained embeddings: PCA 768→256 + Zipf/SIF weighting) so the model is actually static and small.

Usage

from modeling_trynmini import TrynMiniV2
m = TrynMiniV2.from_pretrained(".")           # downloaded snapshot dir
emb = m.encode(["a man plays guitar", "someone plays a guitar"], dim=256)
sim = float(emb[0] @ emb[1])                   # cosine (vectors are L2-normalized)

Requires only numpy, tokenizers, and safetensors. Run python example.py for a usage demo plus a small built-in STS sanity check. The whole model is a single model.safetensors (int8 static table) + tokenizer.json.

Evaluation (STSBenchmark dev, Spearman)

dim	Spearman
64	0.6358
128	0.6526
256	0.6661

Ablation (paired bootstrap): SIF/Zipf pooling significantly beats plain mean pooling (Δ ≈ +0.20, 95% CI excludes 0, p < 0.001).

Specs

• Params: ~7.7M (30000 × 256 + 256² residual)
• Deployment size: ~7.8 MB (per-row int8 table + tokenizer)
• Tokenizer: 30k WordPiece (tokenizer.json), shared by train/eval/inference

Limitations

Distilled from a modest v1 teacher; below strong baselines like Model2Vec potion-base-8M (~0.75 STSB). Fine-tuning (MNRL + Matryoshka) is implemented in the project repo and can lift these numbers.