KhiemGOM/techjam-route-classifier

Dialogue-act router for conversational product search

A six-way dialogue-act classifier used as Node 1 of a conversational e-commerce search agent. It reads what a customer means by a turn when the wording is unfamiliar, so the agent can manage state correctly.

This is a small, narrow, task-specific encoder. It does not retrieve products, rank them, or generate text.

What it is for

The agent it belongs to has a literal recognition gate that matches the message shapes its simulator emits. That gate is a detector, not a router: it reports that a message is unfamiliar, and cannot say what the message means. Two behaviours depend on the meaning:

• override_update / override_opening — the customer changed their mind, so the rejection set must be cleared. Failing to clear it keeps excluding products dismissed under the old intent, which can permanently exclude the right answer.
• no_evidence — the customer stated they have no requirement for an attribute, so the turn carries nothing and must not be mined for evidence.

Why a model rather than a rule

Lexical cues were built first and measured on held-out templates, and they are not enough:

signal	hand-written cues	this model
override → clear rejection state	37.5%	100.0%
no-evidence → skip the turn	0.0%	100.0%
override false positives (of 6,400)	0	2
no-evidence false positives (of 8,000)	0	0

The no-evidence row is decisive. The held-out templates say "indifferent", "nothing to add", "unspecified" where the training templates said "no further preference", "any choice is fine", "use your judgment" — zero shared vocabulary. A lexical rule cannot survive that shift; semantic classification is the only mechanism that transfers when the vocabulary changes, which is the entire threat model.

Overall six-way accuracy under the turn mask: 0.9909 on 9,600 held-out rows. The one material error is buying_opening → override_opening (85/1600), which is benign — it clears a rejection set that is empty at turn 1.

Labels

Positional and alphabetical; id2label in config.json is authoritative.

0 buying_opening      opening turn that also states a requirement
1 constraint_update   a later turn supplying a requirement
2 no_evidence         the customer declines to constrain this attribute
3 override_opening    opening turn of an intent-change scenario
4 override_update     the customer replaces an earlier requirement
5 plain_opening       opening turn with a category and no requirement

Intended use, and the gate around it

Call it only on messages that a literal recognizer has already failed to match. In the host agent this is enforced by control flow, not by a threshold: on clean traffic the gate matches 463 of 463 messages, so the model records 0 loads and 0 inferences and the agent's score is unchanged by construction.

A deterministic turn mask is applied after the model: turn == 1 admits only opening acts, later turns only reply/update acts. That is a released property of the environment, not something learned, and applying it materially raised accuracy in development.

Training

• Base: distilbert-base-uncased
• Data: synthetic dialogue-act templates rendered from a participant-visible product catalogue, with train and test template sets sharing zero strings
• Augmentation: train-only, wrapper-disjoint dialogue-act phrasings
• Selection: epoch chosen on a held-out eval split, then scored once on a fixed test split

Limitations

• English only, and only the dialogue acts listed above.
• Trained on synthetic templates. Train/test are template-disjoint, but both were generated by the same process, so "generalises across held-out templates" is a weaker claim than "generalises to real users". Do not read 0.9909 as an estimate of human traffic.
• Not a general intent classifier. It assumes a product-search conversation with a turn-numbered structure.
• Inference cost measured at ~3.6 ms on a consumer GPU; it is small enough for CPU.

License

Apache 2.0, inherited from distilbert-base-uncased.