Jev-Style v3
⚖
0.8B decisions, a probability for every option
chaoliang yan PRO
chaoliangUNSW
AI & ML interests
None yet
Recent Activity
updated a Space about 5 hours ago
chaoliangUNSW/jev-style-v3 updated a model about 5 hours ago
chaoliangUNSW/Jev-Style-0.8B-Decision-v3 updated a model about 5 hours ago
chaoliangUNSW/Jev-Style-0.8B-Decision-v3-MLXOrganizations
None yet
updated a
Space about 5 hours ago
updated 2
models about 5 hours ago
Request to score Jev-Style-0.8B-Decision-v3 (GGUF F16, Qwen3.5-0.8B full fine-tune)
#23 opened about 6 hours ago
by chaoliangUNSW
updated a
model about 6 hours ago
replied to their post about 22 hours ago
Thanks, this was a fair catch. I've rewritten the post around it: the headline is now the out-of-teacher results, and the typed number sits next to a teacher-noise reference.
On your question: an exact ceiling isn't possible with what the dataset releases. The test split ships only the mean of the three teacher samples plus a label_agreement flag, so the card's 73.5% can't be recomputed on the 400 test states. What I can add:
- One draw from the teacher's mean distribution matches the gold argmax 65.9% of the time on the test split.
- Split by teacher margin (gap between its top two probabilities), v3 vs Laya typed on the same rows:
| teacher margin | n | v3 | Laya typed | diff (95% CI) |
|---|---|---|---|---|
| < 0.1 (near-ties) | 315 | 51.7 | 51.7 | +0.0 [−4.4, +4.5] |
| 0.1–0.3 | 555 | 69.7 | 66.1 | +3.6 [+0.2, +7.3] |
| ≥ 0.3 | 1,130 | 91.4 | 88.7 | +2.7 [+0.8, +4.6] |
So v3 isn't gaining by fitting the teacher's coin-flips. That still can't separate task skill from the teacher's consistent biases, since both models trained on its labels. Only an independent gold can, which is why the post now leads with Banking77, tweet_topic and JevBench. The gap to Jev (72.7, zero-shot) shouldn't be read as skill, and the post no longer frames it that way.
The same table and reference are going on the model card.
Bad Answers with Ollama
2
#2 opened about 23 hours ago
by 
gshaxor
liked a
model about 23 hours ago
News: add Jev-Style v3 (0.8B full fine-tune, 25,600-token input)
#16 opened about 24 hours ago
by chaoliangUNSW
reacted to their post with 🔥 1 day ago
Post
64
Jev-style decisions on your laptop: Jev-Style-0.8B-Decision-v3
Give it any text and a question. It returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation, nothing to parse.
On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) too, though that lead is inside the CI
Jev is still ahead on all three. v3 is the open, local option that gets closest: 0.53 GB in 4-bit, runs on CPU.
Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.1 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); the dataset card puts teacher self-agreement at 73.5%
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison
Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); about 44 ms per short decision on an M1 Max; one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.
Try it in your browser, no install:
chaoliangUNSW/jev-style-v3
Model (transformers, with GGUF and MLX builds linked):
chaoliangUNSW/Jev-Style-0.8B-Decision-v3
Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.
Give it any text and a question. It returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation, nothing to parse.
On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) too, though that lead is inside the CI
Jev is still ahead on all three. v3 is the open, local option that gets closest: 0.53 GB in 4-bit, runs on CPU.
Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.1 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); the dataset card puts teacher self-agreement at 73.5%
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison
Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); about 44 ms per short decision on an M1 Max; one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.
Try it in your browser, no install:
chaoliangUNSW/jev-style-v3
Model (transformers, with GGUF and MLX builds linked):
chaoliangUNSW/Jev-Style-0.8B-Decision-v3
Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.
- 3 replies
posted an update 1 day ago
Post
64
Jev-style decisions on your laptop: Jev-Style-0.8B-Decision-v3
Give it any text and a question. It returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation, nothing to parse.
On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) too, though that lead is inside the CI
Jev is still ahead on all three. v3 is the open, local option that gets closest: 0.53 GB in 4-bit, runs on CPU.
Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.1 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); the dataset card puts teacher self-agreement at 73.5%
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison
Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); about 44 ms per short decision on an M1 Max; one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.
Try it in your browser, no install:
chaoliangUNSW/jev-style-v3
Model (transformers, with GGUF and MLX builds linked):
chaoliangUNSW/Jev-Style-0.8B-Decision-v3
Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.
Give it any text and a question. It returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation, nothing to parse.
On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) too, though that lead is inside the CI
Jev is still ahead on all three. v3 is the open, local option that gets closest: 0.53 GB in 4-bit, runs on CPU.
Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.1 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); the dataset card puts teacher self-agreement at 73.5%
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison
Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); about 44 ms per short decision on an M1 Max; one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.
Try it in your browser, no install:
chaoliangUNSW/jev-style-v3
Model (transformers, with GGUF and MLX builds linked):
chaoliangUNSW/Jev-Style-0.8B-Decision-v3
Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.
- 3 replies
updated a
collection 1 day ago
published 2
Spaces 1 day ago
updated a
Space 1 day ago
liked a
model 1 day ago