Eikos
Collection
Open, calibrated, single-pass typed-decision models (4B & 27B) for finance and trading • 10 items • Updated • 5
Eikos-4B — open typed-decision model (System-1)
Eikos-4B answers typed decisions about a given state in a single forward pass, with calibrated probabilities over a bounded set of options. It does not generate text at decision time.
- Question types:
noul/boolean(yes/no with probability),choice(one of N options) andscore(ordinal levels). - Outputs: the full probability distribution plus a confidence value, so callers can abstain below a threshold.
- Focus: global finance, trading and trade finance.
- Apply a stated rule, policy or rulebook to a case: order limits, margin, wash sales, price bands, payments limits, KYC/AML, lending policies, documentary credits, Incoterms® stage mapping, VAT/GST.
- Entity-level financial sentiment, answer verification over financial tables, temporal and numeric checks.
- Scope limit: it applies rules to the facts it is given. It does not predict prices.
- Languages: trained in English and Portuguese. Spanish was held out entirely and is used as a zero-shot test.
- Runs anywhere: one self-contained checkpoint in the official Qwen layout, with no adapters, no external LLM
and no API.
- vLLM, with prefix caching for many questions over the same state and for agent sessions;
- PyTorch / transformers;
- Apple Silicon via MLX or MPS.
Eikos (εἰκός, "the probable") comes in two sizes: Eikos-4B (this model) and Eikos-27B, each with FP8 and INT4 builds for GPUs and MLX builds for Apple Silicon (4B: 8-bit and 4-bit; 27B: 4-bit). Training data:
caiovicentino1/eikos-decisions. Code:https://github.com/caiovicentino/eikos.Numbers below are from our own evaluation harness (see Evaluation). The official JevBench leaderboard submission is pending.
How it works
Prompt. The state (evidence), the question (criterion) and the lettered options go into a structured prompt, following the SemIf format.
Readout. The model reads the next-token logits only over the option letters. A softmax with a fixed temperature gives the distribution.
Calibration. Temperature T = 1 ships by default (see Calibration).
Training. Distillation from a strong teacher (GLM-5.3-Flash, maximum reasoning effort) on:
- generated and blindly labeled decision items;
- exact programmatic items;
- compositional rules;
- long-context dossiers (6k–32k tokens).
Training-only signals, all switched off at inference:
- soft cross-entropy to the teacher's probabilities, with option-order permutation;
- an auxiliary loss on the item's short rationale (written with the item, explaining the gold answer);
- EN↔PT view consistency (the same item in two languages must give the same answer).
The released 4B is a weight average ("model soup") of two fine-tunes that share data and seed: one with view consistency, and one that also adds a light JEPA-style latent objective (weight 0.2: mid-layer view invariance, latent rationale prediction and a state–option energy head). The average won on our internal dev set and on the held-out family, topic and language.
Quick start
vLLM (production: batching + prefix cache)
Requires vLLM ≥ 0.30.0. Older builds return wrong answers when several long requests are batched together on this hybrid (Gated DeltaNet) architecture; we measured drops of 3–6 points on long, shared-document items with vLLM 0.11. With 0.30 the batched results match the PyTorch reference.
bash serve_vllm.sh <MODEL_DIR> 8001 # vLLM engine: letter readout + hybrid prefix cache (checks vLLM >= 0.30)
python serve.py --model <MODEL_DIR> --vllm-url http://127.0.0.1:8001 --port 8000 # TypeSafe-compatible API on :8000
serve_vllm.sh runs vllm serve with the flags the readout needs: --served-model-name decider,
--enable-prefix-caching --mamba-cache-mode all, --logprobs-mode processed_logprobs --max-logprobs 32,
--dtype bfloat16 and --max-model-len 16384.
HTTP API (TypeSafe-compatible)
curl -s localhost:8000/v1/systemone -d '{
"state": "Order ticket #A-2231. Retail client, moderate risk. BUY 1,500 XYZ at market. Equity USD 48,000. Last price USD 41.20. Rule 4.2: a single order may not exceed 50% of equity without written supervisor approval. Approvals on file: none.",
"questions": {
"allowed": {"type": "noul", "instructions": "Under rule 4.2, can this order be executed as submitted?",
"criteria": {"true": "complies with rule 4.2", "false": "breaches rule 4.2"}},
"action": {"type": "choice", "instructions": "What should the desk do?",
"criteria": {"execute": "send as submitted", "request_approval": "hold and ask a supervisor",
"reduce_size": "cut the order to the allowed size", "reject": "refuse the order"}},
"risk": {"type": "score", "instructions": "How risky is this ticket for the firm?",
"criteria": ["low", "moderate", "high", "critical"]}
}}'
- One pass for the whole request: all questions are answered together, and the shared state is processed once through the prefix cache.
- Response per question:
noul:probability(of "yes"),valueandconfidence;choice:choice,probabilitiesandconfidence;score:score,probabilities,expectedandconfidence.
Agent sessions (incremental state + cache)
POST /v1/sessions {"state": "..."} -> {"session_id"}
POST /v1/sessions/<id>/append {"text": "..."} -> grows the state
POST /v1/sessions/<id>/systemone {"questions": {...}} -> answers over the current state
DELETE /v1/sessions/<id>
On a hybrid (linear-attention) model the state is cached at the end of the context and forked for every question. In our tests a 15-step agent session gave answers identical to sending the full request each time.
Local (Apple Silicon)
python mlx_decide.py <MODEL_DIR> # MLX backend: same readout and calibration as PyTorch
python local_demo.py <MODEL_DIR> mps # PyTorch MPS
Evaluation
All numbers are our own measurements with a single harness. We never trained on any evaluation set. Spanish, one task family (trade-offs) and one topic (healthcare administration) were held out of training entirely.
| Benchmark (our harness) | 4B final | 27B final | Jev (reference) |
|---|---|---|---|
| JevBench public — original / hard | 91.7 / 72.1 | 100.0 / 82.9 | 98.6 / 73.0 (official leaderboard) |
| JevBench hard — ECE (lower is better) | 0.049 | 0.051 | — |
| DecisionBench (OOD) — medium / hard | 77.1 / 66.9 | 88.4 / 78.5 | 89.1 / 69.3 |
| General battery (9 human-labeled tasks) | 76.0 | 82.5 | 84.1 |
| Finance (CUAD, financial sentiment, FinQA-judge) | 74.7 | 85.3 | 79.9 |
| Financial sentiment (Twitter Financial News) | 79.3 | 87.7 | 68.3 |
| Central-bank stance (WCB, balanced acc.) | 38.4 | 44.5 | 58.6 |
| FinDVer (claim verification) | 74.0 | 78.0 | 73.3 |
| Trade rules — seen templates / unseen rules | 73.4 / 76.0 | 85.1 / 88.0 | 78.2 / 79.4 |
| Compositional rules — same type / new domain / rulebooks* | 95.8 / 91.7 / 91.7 | 95.8 / 94.3 / 95.3 | 83.4 / 84.3 / 88.0 |
| Held-out Spanish (never trained), internal | 90.9 | — | — |
| Long context: decision hidden in 64k tokens of text (mean of 3 positions) | 74.2 | 88.3 | — (API limit 32k) |
| Robustness: same answer after option reversal / paraphrase | 89.9 / 89.9 | 94.9 / 97.5 | — |
| RuleArena NBA (balanced acc.) | 0.50 | 0.50 | — |
* generated by the same rule generator family used in training (different rules and cases); see notes.
How to read the table:
- JevBench: public items only, not the official leaderboard run.
- Jev: measured through its API on our batteries, not distilled from.
- Laya (charts only): run with its official package; its JevBench scores are within one item of its official leaderboard results.
- Charts: 7,140 items common to all four systems. The confidence chart shows, for every confidence threshold, how many decisions a system would take on its own and how often those decisions are wrong.
- WCB (central-bank stance): reported as balanced accuracy, because the classes are imbalanced.
- Rules suites: "rules / new domain / rulebooks" come from the same generator family used in training, so
they measure learning of that generator. Transfer to rules the model never saw is measured by "Trade (unseen
rules)" (Incoterms®, documentary-credit presentation, EU VAT) and by RuleArena.
- "New domain" has two parts. 492 items come from a domain absent from training (insurance): 27B 95.1, 4B 91.5. The other 108 items use a rule combination held out of training (an exception over a business-day window): 27B 91.7, 4B 90.7. That combination appears in only 6 of the ~6,000 rule rows used in training.
Long context
The case file is hidden inside unrelated text at the start, middle or end of the prompt (80 public JevBench decisions per point; released checkpoints served with vLLM 0.30).
Release builds
Final evaluation of every published build, run on the exact files in these repos: 7 suites, 7,371 items, vLLM 0.30 with batching and prefix cache on (MLX builds: MLX's CUDA backend on Linux). Batched vLLM is not bit-for-bit deterministic across runs (1–2 items per suite can change), so the bf16 rows differ slightly from the Evaluation table above, which uses our reference harness. "≥0.90" = share of decisions the model would take on its own at ≥90% confidence, and the real error rate among them.
| Build | Size | JevBench orig / hard | DecisionBench med / hard | General | Finance | Unseen trade rules | ECE | ≥0.90: decides / error | Gate vs bf16 |
|---|---|---|---|---|---|---|---|---|---|
| Eikos-27B | 55.6 GB | 100.0 / 82.0 | 89.1 / 78.2 | 82.6 | 85.4 | 87.4 | 0.043 | 44.1% / 2.5% | reference |
| Eikos-27B-FP8 | 31.2 GB | 100.0 / 83.8 | 88.7 / 78.8 | 82.8 | 85.3 | 86.9 | 0.042 | 44.2% / 2.6% | pass |
| Eikos-27B-INT4 | 19.4 GB | 100.0 / 82.9 | 88.4 / 78.5 | 82.5 | 84.9 | 88.6 | 0.041 | 43.6% / 2.4% | pass |
| Eikos-27B-MLX-4bit | 15.1 GB | 100.0 / 83.8 | 88.4 / 78.2 | 81.8 | 84.9 | 85.7 | 0.041 | 42.9% / 1.9% | pass except agreement |
| Eikos-4B | 9.3 GB | 91.7 / 73.9 | 77.1 / 66.6 | 75.7 | 74.7 | 76.0 | 0.033 | 34.7% / 2.5% | reference |
| Eikos-4B-FP8 | 5.8 GB | 90.3 / 73.0 | 78.5 / 66.6 | 75.8 | 75.2 | 76.0 | 0.031 | 34.8% / 2.5% | pass |
| Eikos-4B-INT4 | 4.0 GB | 91.7 / 72.1 | 77.1 / 66.2 | 75.6 | 75.7 | 76.0 | 0.033 | 32.9% / 2.0% | pass except agreement |
| Eikos-4B-MLX-8bit | 4.5 GB | 91.7 / 72.1 | 78.2 / 66.9 | 76.0 | 74.4 | 76.6 | 0.033 | 34.6% / 2.5% | pass |
| Eikos-4B-MLX-4bit | 2.4 GB | 95.8 / 73.0 | 78.8 / 64.8 | 75.3 | 76.0 | 76.0 | 0.021 | 32.5% / 2.7% | pass except agreement |
Release gate, fixed before looking at results: accuracy within 1 point of bf16, ECE within 0.01, and at least 97% of answers unchanged.
| Build | Accuracy (bf16) | Same answer as bf16: all / confident (≥0.9) | ECE (bf16) | Changed answers on items where bf16 was unsure (<0.7) | Gate |
|---|---|---|---|---|---|
| Eikos-27B-FP8 | 83.4 (83.4) | 98.8% / 100.0% | 0.042 (0.043) | 99% | pass |
| Eikos-27B-INT4 | 83.3 (83.4) | 97.8% / 100.0% | 0.041 (0.043) | 96% | pass |
| Eikos-27B-MLX-4bit | 83.1 (83.4) | 96.6% / 99.9% | 0.041 (0.043) | 92% | pass except agreement |
| Eikos-4B-FP8 | 76.9 (76.7) | 97.3% / 100.0% | 0.031 (0.033) | 99% | pass |
| Eikos-4B-INT4 | 76.6 (76.7) | 95.5% / 100.0% | 0.033 (0.033) | 94% | pass except agreement |
| Eikos-4B-MLX-8bit | 76.8 (76.7) | 98.7% / 100.0% | 0.033 (0.033) | 99% | pass |
| Eikos-4B-MLX-4bit | 76.4 (76.7) | 92.5% / 99.8% | 0.020 (0.033) | 93% | pass except agreement |
- Quantization cost: no build loses more than 2 points on any metric of any suite, and calibration (ECE) stays within 0.01 of bf16 or better.
- Eikos-4B-INT4 keeps accuracy and calibration but changes 4.5% of answers, above the 3% the gate allows. Almost all of them (94%) are on items where bf16 itself was unsure (confidence below 0.7); on decisions bf16 takes with confidence ≥0.9, it gives the same answer on 100% of items. We publish it with this note.
- Mac builds (MLX): Eikos-4B-MLX-8bit passes the gate. Eikos-27B-MLX-4bit and Eikos-4B-MLX-4bit keep accuracy and calibration but change 3.4% and 7.5% of answers. Most of those changes (92% and 93%) are on items where bf16 itself was unsure; on decisions bf16 takes with confidence ≥0.9, they agree on 99.9% and 99.8% of items. We publish them with this note. The small JevBench-original split (72 items) moves by a few items between builds.
Calibration
- Why T = 1: our held-out calibration sets turned out easier than real hard data. Every temperature fitted on them was below 1 and made the model over-confident on hard items. Absent a realistic calibration set, we ship T = 1.
- Changing it:
calib.jsonholds the temperature, and callers can re-fit it on their own data.
Parallelism and speed
- Same state, many questions: the hybrid prefix cache processes the state once.
- 200 questions over a 3.3k-token state ran 21× faster with vLLM prefix caching than without it.
- Throughput: batched decisions reach ~51 decisions/s on a shared GPU, with results identical to PyTorch.
- Full GPU (RTX PRO 6000, vLLM, hybrid prefix cache), 200 questions over one state:
- 4B: 161 decisions/s on a 751-token state and 197 decisions/s on a 3.3k-token state;
- 27B: 44 decisions/s on a 3.3k-token state.
- Our own PyTorch cache path: 1,500 questions over one state in 8.7 s (4B).
- Apple M4 16 GB, MLX bf16: ~0.4–0.8 s per decision, and 3 questions over one state in ~1.1 s.
Limitations
- Single pass means no multi-step reasoning. Long chains of arithmetic across many rules are out of reach; on RuleArena (NBA salary-cap rulebooks, ~25k tokens) the model does not discriminate. An optional "verify" mode (a short reasoning budget before the letter) exists in the server; it is off by default and not part of any reported number.
- General world knowledge is bounded by model size. Both sizes trail large frontier systems on knowledge-heavy tasks such as MMLU-Pro.
- Monetary-policy stance (hawkish/dovish) is a known weakness of both sizes: balanced accuracy 38.4 (4B) and 44.5 (27B), against 58.6 for Jev.
- Context: trained on inputs up to 32k tokens (4B) or 12k (27B). The base supports longer inputs, but accuracy beyond the trained range degrades gradually.
- Not advice: it is not legal, tax or investment advice. It applies the rules it is given and does not know your jurisdiction's current law.
Training data and licenses
See NOTICE.
- Model and code: MIT for our contributions. The base model is Apache-2.0 (
LICENSE-Qwen). - Third-party data, with attribution: FinEntity (ODC-BY 1.0), TAT-QA (CC BY 4.0), GSM8K train (MIT).
- Prompt format: SemIf (MIT).
- Trademark: Incoterms® is a trademark of the ICC. No ICC or regulator text was used.
Citation
@misc{eikos2026,
title = {Eikos: open, calibrated, single-pass typed-decision models for finance and trading},
author = {Caio Vicentino},
year = {2026},
url = {https://github.com/caiovicentino/eikos}
}
- Downloads last month
- 15