convaiinnovations

/

laya

Like 735

Thanks for releasing this with the weights and the eval numbers, the Jev-compatible system_one shape made it very easy to try.

Reporting something that looks unintended in the shipped calibration, because it silently removes the confidence signal for any choice with more than ten options.

What happens

temperature_by_options in rl_agent_config.json fits one temperature per option-count bucket:

{'choice:2': 1.906, 'choice:3-5': 1.760, 'choice:6-10': 1.0000, 'choice:11+': 0.1006,
 'score:3-5': 1.251, 'noul:2': 1.983}

choice:11+ is 0.1006. Since the code does z = logits / temperature, that multiplies the logits by about ten and saturates the softmax, so essentially every answer comes back at confidence 1.00.

Minimal reproduction

Same model, same eight support tickets, same instructions. The only thing that changes is how many options the question offers, which moves it across the bucket boundary at ten.

Adding a single option takes mean confidence from 0.715 to 0.968. At twenty options every answer reports 1.000, including the three that are wrong.

It is the temperature, not the option count

Re-scoring the same k=20 forward passes at temperature 1.0 instead of 0.1006 gives mean confidence around 0.80 rather than 1.000, with the same accuracy. So this is not the entropy normalisation in confidence_from_probs reacting to a larger k, and it is not the model: temperature only rescales confidence, and the chosen option never changes.

On a separate 20-option routing task of my own, rescoring at 1.0 also restored a usable gap between confidence-when-right and confidence-when-wrong, where the shipped value left almost none.

Why it seems worth fixing

The main reason to reach for a calibrated decision model is to gate on the probability and ask a human when the model is unsure. Above ten options the returned confidence carries no information, so a gate built on it passes everything through, and the errors arrive looking maximally certain. That is the one failure mode calibration is supposed to prevent.

Worth double-checking how that bucket was fit, or capping it. Happy to share the reproduction script if useful.

Repro script used:

from rl_agent_api import RLAgent
agent = RLAgent('.', device='cpu')
tickets = [("my card was charged twice for one order", "billing"),
           ("the package never arrived", "shipping")]  # ... 8 total
for options in (depts[:10], depts[:11], depts):
    for text, want in tickets:
        out = agent.system_one(text, {"q": {"type": "choice",
            "instructions": "Which department should handle this ticket?",
            "criteria": {d: None for d in options}}})
        print(len(options), out["answers"]["q"]["confidence"])