Catalog selection: low gold-label accuracy and candidate-order sensitivity with Laya-MLX (multilingual included)
Related to #7. We see a similar fast-inference/weak-decision pattern in a private catalog-selection evaluation. We are reporting aggregate measurements only; no customer data or original examples are included.
Fresh reference-label evaluation
100 catalog-selection cases, one item from each of 100 distinct documents. Reference labels were persisted catalog assignments designated approved for this evaluation, not outputs from a newly run comparison model. They were not independently re-adjudicated by humans.
Every model received the same cases and candidate membership, including a no-match option. Laya received a compact task-specific adapter; Jev received the original instructions, so the representations were not byte-identical. Labels were kept separate from model inputs, and no prompt/threshold tuning was performed against this reference set.
| Model | Exact reference matches / 100 | Median inference time per case |
|---|---|---|
| Laya multilingual | 23 | 87 ms |
| Laya English | 24 | 177 ms |
| Laya typed-decisions | 30 | 189 ms |
| Jev via OpenRouter Decisions | 92 | 426 ms |
| First retrieved candidate, no model | 86 | Not measured |
The reference candidate was present in 96/100 shortlists. All four retrieval misses remain in the denominator, so the maximum possible exact-selection score is 96/100. Returning no-match on these cases is not counted as selecting the reference node.
Local timings exclude model loading/warmup and retrieval; Jev includes network/provider time. These are single-run observations, not a controlled cross-hardware speed benchmark.
Environment and input checks
- Apple M4 Pro, 16 GPU cores, 24 GiB unified memory.
- MLX 0.32.2, FP16, explicit GPU execution with synchronization for timing; no CPU inference fallback.
- laya-mlx source revision: fc1df62828a3fedf4d8229fdac1cbd85f1cdf337.
- Checkpoint bundle from convaiinnovations/laya revision: 1c5edc17a7acd8701df6fc341c0d179f1c62c982, using English, multilingual and typed-decisions weights.
- All 300 local outputs were valid. No budget rejections or silent loss of submitted state, instructions or options.
- Multilingual sequences: 304-936 tokens; typed: 311-943. Both fit their shipped 1,024-token total context. Question/option budgets were enlarged where needed, which is a departure from shipped settings.
- English sequences: 311-943 tokens; 71 cases exceeded its shipped 512-token limit using experimental extended context. Its score needs that caveat.
Separate presentation-order experiment
On an earlier, different 100-case catalog sample, reversing the options and corresponding candidate-detail rows changed the selected candidate in:
| Checkpoint | Changed selections / 100 |
|---|---|
| Multilingual | 69 |
| English | 65 |
| Typed-decisions | 76 |
The candidate set and alias-to-candidate mapping stayed fixed; no facts were added or removed. This tests presentation sensitivity, not gold accuracy. It was not rerun on the fresh reference-label sample above.
A separate exact-identity positive-control cohort also showed multilingual agreement dropping from 100/100 to 0/100 when changed from the original input representation to a fuller-context adapter. That transformation changed several aspects at once, so it does not isolate JSON formatting as the cause.
Questions
We already tested multilingual. Its low latency is real, but neither missing GPU acceleration nor total-context overflow explains these observations by itself.
- Is there a recommended input/option representation for selecting among catalog candidates with similar descriptions and pack sizes? Could short answer aliases or enlarging the question/option budget cause this behavior even when the total context fits?
- Are there known candidate-order or context-sensitivity limitations, or checkpoint/runtime fixes relevant to the revisions above?
- What minimal official-runtime versus MLX parity test would you recommend to distinguish adapter/checkpoint limitations from an MLX implementation issue?
We have not yet established identical-input parity with the official runtime, so this is not a claim of a confirmed model defect or a general accuracy ceiling. The original data cannot be published; this report does not include a public reproducer.