Laya feels like an embedding/similarity matcher, not a context-understanding model (benchmark included)
I'm building a small voice-controlled hardware project that needs to determine "accelerate / decelerate / stop" from a spoken sentence, so this model looked like a great fit. Downloaded the multilingual variant (mmBERT-base, 322M, 647MB), running on Mac Apple Silicon via MPS.
Speed is genuinely impressive. 35ms median — actually better than the officially quoted T4 numbers. Jev (cloud) takes 1237ms for the same task. That's a real 35x speedup, no exaggeration.
Accuracy is where it falls apart.
20 Chinese voice commands, three-way comparison:
| Method | Accuracy |
|---|---|
| Jev (cloud) | 20/20, 100% |
| 15-line regex | 19/20, 95% |
| Laya (local) | 10/20, 50% |
Examples of misses:
- "停下" (stop) → model says "decelerate" (stop dimension only scored 0.11)
- "太快了" (too fast) → all four dimensions scored 0
- "快一点" (a bit faster) → "wants more intensity" only scored 0.14
The weirder part: following the official docs made it worse.
The docs recommend writing criteria, giving scenario descriptions, and using structured state. I did that, and results collapsed. Ran an ablation:
| Config | Neutral sentence | Pleasant sentence | Genuine-discomfort sentence |
|---|---|---|---|
| Plain string, no criteria, single question | 0.04 | 0.26 | 0.97 ✅ |
| + criteria | 0.90 | 0.85 | 0.94 ❌ |
| + scenario description | 1.00 | 0.65 | 0.96 ❌ |
| + JSON state | 0.69 | 0.76 | 0.69 ❌ |
The moment you add more text to the request, outputs flatten into the 0.6–1.0 range and lose discrimination. After adding scenario context, a neutral opener like "let's start" got scored as "pleading 0.96, wants to stop 0.86."
Jev shows the opposite pattern — more context = more accurate.
It seems to give up entirely on "what should I do" questions. Asked 26 sentences about "how should intensity be adjusted," and every single output landed in the 1.90–2.32 range (full scale 0–4, 2 = no change). "Too fast" and "nice weather today" got the same answer.
It even fails basic sanity checks:
- "今天天气很好" ("nice weather today"), asked "is this Chinese?" → 0.17
- "我爱这个产品，太棒了" ("I love this product, amazing"), asked "is this positive?" → 0.06
- The official README's own English demo — the churn_risk example — also scored wrong (0.13, even though the text explicitly says "or we will cancel")
To be fair, three things it does well:
- Genuinely fast — 35ms
- Fully deterministic — same sentence run 5 times gives identical results
- Confidence is honest — when unsure it gives 0.02–0.34 rather than falsely committing to something high
That third point is actually rare and valuable — much better than a model that guesses wrong and still outputs 0.95.
Bottom line: for short questions asking "what happened" (is this painful? is this positive?), it works and it's fast. For "what should be done," or anything requiring context understanding, it performs worse than a 15-line regex. My read is that this behaves more like an embedding/similarity model matching surface meaning, rather than something that actually understands meaning the way Jev does.
Has anyone else run into this gap between explicit-signal detection and actual context understanding?
这就是一个傻逼模型，你用这个模型是在浪费你的时间，老老实实部署deepseek比什么都简单。
如果你需要小模型可以试试rwkv
lav是傻逼，时间会说明一切。
I thing Laya has a problem with differente languages, it seems to me that the main data used in the training stage is in English.
I'm running a benchmark on a project that i thing this System One Models can be a good tool and i see difference in perfomance between languages even if they say the same thing.
What i suggest to you is to fine tune to your main language or try to add a pre process stage before it enters this model to convert to English.
Or you can use Jev, Jev clearly has a training stage way more robust.
You talk about Context, i dont think the main goal to this models is to give a very deep context, the main goal is to be a classifier or decision type model.
Try multilingual model
@convaiinnovations the OP has explicitly wrote:
Downloaded the multilingual variant (mmBERT-base, 322M, 647MB)
Maybe there should be a specific Chinese variant?
Matches what I measured. With the encoder frozen, a trained head learns what the text states and nothing it has to compute: a payment-risk rule over amount, hour and country trained to 0.42 agreement; the identical rule rewritten as sentences ("first transfer to this payee, larger than anything they sent before") trained to 0.94, same rows. For Snake, an ASCII board trained to 0.73 (the majority move), four relational lines ("food: 3 left, 2 up / safe: up, right") to 0.957. So the practical rule is: give it the relation in words, not the raw fields. Details and the demos: https://github.com/bladedevoff/stuntd
