A small real-world calibration check on Laya typed-decisions
I ran convaiinnovations/laya-typed-decisions on 24
customer-service decisions while building Second Thought,
an open-source calibration and active-learning layer for
System One / typed-decision models.
On this small sample:
- Accuracy: 87.5% (21/24)
- ECE: 0.62
The sample is far too small to make a general claim about
the checkpoint's calibration, so I'm treating this as a
case study rather than a benchmark.
What I found interesting is that this is directionally
consistent with the model card's own warning that the
checkpoint's probability calibration should be refit on
held-out data before relying on probabilities for
downstream decisions.
Second Thought is designed around exactly that problem:
model prediction
→ probability distribution
→ calibration
→ identify decisions worth reviewing
→ correction
→ training/evaluation data.
Repository:
https://github.com/KNambiarDJsc/second-thought
I'd be interested in whether others have tested the
checkpoint on held-out real-world data.