AlphaRoute-0.8B-v1.5

AlphaRoute-0.8B-v1.5 is an instruction-conditioned Small Language Model (SLM) engineered specifically for high-speed semantic routing, dynamic ontology classification, and constrained multi-field JSON extraction.

Trained on a curated 17,898-row instruction blend, AlphaRoute v1.5 learns the meta-task of routing: it does not have hardcoded static labels. At runtime, developers supply an arbitrary task instruction, a dynamic ontology (with descriptions), and an arbitrary JSON schema; AlphaRoute outputs valid JSON matching the requested schema at edge-level latencies.

---

Benchmark Evaluation (Deterministic Fixed Splits)

All evaluations are conducted with greedy decoding (temperature = 0.0) on standardized fixed test splits:

Benchmark Dimension	AlphaRoute-0.8B v1.5	AlphaRoute-0.8B v1.1	AlphaRoute-0.8B v1.0	DeepSeek-V4 Flash	Azure GPT-5.4-nano
Golden 300 Enterprise (300 Scenarios)	97.67%	95.67%	89.33%	94.00%	92.33%
Banking77 Intent (500 Fixed Samples)	95.40%	92.60%	91.80%	93.00%	85.40%
CLINC150 Multi-Domain (500 Fixed Samples)	94.00%	73.20%	75.20%	71.40%	54.40%
HWU64 Zero-Shot (1,076 Held-Out)	89.96%	83.18%	83.92%	85.40%	81.20%
Adversarial OOS Rejection (20 Probes)	100.00%	55.00%	35.00%	75.00%	65.00%
Multi-Target Compound (117 Split)	62.39%	52.14%	22.22%	64.10%	58.12%
CLINC150 OOS Recall (94 OOS Queries)	85.11%	96.81%	95.74%	98.94%	94.68%

Understanding the CLINC150 Out-of-Scope (OOS) Recall: Trade-off vs. Regression

At initial glance, AlphaRoute v1.5 records an 85.11% OOS recall on CLINC150, whereas earlier models and frontier APIs scored between 94.68% and 98.94%. A granular breakdown of the 500 benchmark episodes (406 In-Scope user requests vs. 94 Out-of-Scope queries) clarifies why this reflects a deliberate optimization trade-off rather than an architectural regression:

Model Backbone	OOS Recall (94 Out-of-Scope)	In-Scope Intent Accuracy (406 Real Requests)	Overall Benchmark Accuracy (500 Total)
Azure GPT-5.4-nano	94.68% (89/94)	45.07% (183/406)	54.40%
DeepSeek-V4 Flash	98.94% (93/94)	65.02% (264/406)	71.40%
AlphaRoute-0.8B v1.1	96.81% (91/94)	67.73% (275/406)	73.20%
AlphaRoute-0.8B v1.5	85.11% (80/94)	96.06% (390/406)	94.00%

Key Factors Behind This Dynamic:

1. Elimination of False-Rejection Bias:

   • Earlier models achieved high OOS recall by exhibiting an overly defensive bias: when faced with slight semantic ambiguity, they defaulted to out_of_scope: true.
   • Consequently, DeepSeek misclassified or discarded 35.0% of all legitimate user requests (142 real queries falsely rejected), and v1.1 discarded 32.3%.
   • AlphaRoute v1.5 resolved this bias, surging in-scope accuracy from 67.73% to 96.06% (390/406 valid requests correctly routed). This lifted overall CLINC150 accuracy from 73.20% to 94.00% (+20.80% gain).

2. Lexical Proximity in 10-Category Sampling:

   • In the CLINC150 evaluation setup, each query is evaluated against 10 categories sampled from 150 broad conversational intents.
   • Dissection of the 14 unflagged OOS queries revealed that random sampling frequently paired an out-of-scope query with an intent sharing literal keywords (e.g., "what oil is best for chicken" paired with oil_change_how, or "distance between football goal posts" paired with distance). Because v1.5 is trained to route confidently rather than defensively drop requests, it selected the overlapping label.

3. Enterprise Boundary Precision:

   • On production-style ontologies with clear operational scopes (such as the Adversarial OOS benchmark suite and Golden 300 Enterprise), AlphaRoute v1.5 achieves 100.00% out-of-scope rejection.

---

Evolutionary Lineage and Design Methodology

AlphaRoute-0.8B v1.5 was developed through a structured, multi-phase research progression:

1. AlphaRoute-0.8B v1.0 (Baseline Exploration):

   • Established initial proof-of-concept for instruction-conditioned semantic routing on a sub-billion parameter model.
   • Highlighted key behavioral limitations: 0.0% compound accuracy on multi-target nested schemas, susceptibility to keyword distractors (35.0% adversarial out-of-scope accuracy), and inconsistent priority grading.

2. AlphaRoute-0.8B v1.1 (Internal Research Iteration):

   • Rather than continuing from the v1.0 checkpoint, v1.1 was a fresh fine-tune of the base model (Qwen/Qwen3.5-0.8B-Base) designed specifically to address the core behavioral issues identified in v1.0.
   • Introduced multi-field schema extraction (improving compound accuracy to 52.1% and Golden 300 Enterprise routing to 95.67%).
   • Role: Served as the internal research baseline and foundation for subsequent data blend curation.

3. AlphaRoute-0.8B v1.5 (Public Release):

   • Built upon the architectural and training insights of v1.1, introducing a unified canonical prompt formatting pipeline and a curated 17,898-row instruction dataset.
   • Mitigated False-Rejection Bias: Addressed the over-rejection tendency present in earlier models and frontier APIs (which defaulted to out-of-scope on ambiguous queries), achieving 96.06% in-scope accuracy and advancing CLINC150 accuracy to 94.00%.
   • Held-Out Generalization: Achieved 89.96% on HWU64 without any exposure to this dataset during training.
   • Adversarial Precision: Reached 100.00% rejection on adversarial edge-case distractors.

---

Quickstart Inference

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NamanAgnih0tri/AlphaRoute-0.8B-v1.5"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

task = "Triage enterprise infrastructure incident. Disentangle primary root cause, determine responding operational team, and extract urgency."
categories = {
    "db_deadlock": "Postgres/MySQL lock contention, exclusive table locks, query timeouts.",
    "k8s_oom": "Kubernetes OOMKilled exit code 137, cgroup memory exhaustion, pod evicted.",
    "tls_cert_expired": "TLS handshake failure, expired origin cert on ALB/Cloudflare.",
    "iam_privilege_escalation": "Compromised access keys, unauthorized IAM role modification."
}
output_schema = {
    "root_cause": "db_deadlock | k8s_oom | tls_cert_expired | iam_privilege_escalation | null",
    "assigned_team": "dba_sre | k8s_platform | secops | netops",
    "severity": "SEV0_critical | SEV1_high | SEV2_medium",
    "out_of_scope": "boolean"
}
query = "CRITICAL: Checkout service is stuck in CrashLoopBackOff. Logs show: `FATAL: deadlock detected Process 8912 waiting for ExclusiveLock on relation orders`."

# Construct Canonical Routing Prompt (Zero-Shot)
cat_str = "\n".join([f"- {k}: {v}" for k, v in categories.items()])
schema_str = json.dumps(output_schema, indent=2)

prompt = f"""You are a precise semantic routing engine. Analyze the input against the provided categories and output only a valid JSON object matching the requested schema. Do not output conversational text or markdown code fences.

TASK:
{task}

CATEGORIES:
{cat_str}

OUTPUT SCHEMA:
{schema_str}

INPUT:
"{query}"

JSON:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.0,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()

Tip: Guaranteed Schema Adherence with Constrained Decoding (lm-format-enforcer)

For production environments requiring strict schema guarantees, token generation can be physically constrained at each step using a grammar/FSM parser such as lm-format-enforcer:

pip install lm-format-enforcer

from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn

# Formal JSON schema representing your target fields and enums
json_schema = {
    "type": "object",
    "properties": {
        "root_cause": {
            "anyOf": [
                {"type": "string", "enum": ["db_deadlock", "k8s_oom", "tls_cert_expired", "iam_privilege_escalation"]},
                {"type": "null"}
            ]
        },
        "assigned_team": {"type": "string", "enum": ["dba_sre", "k8s_platform", "secops", "netops"]},
        "severity": {"type": "string", "enum": ["SEV0_critical", "SEV1_high", "SEV2_medium"]},
        "out_of_scope": {"type": "boolean"}
    },
    "required": ["root_cause", "assigned_team", "severity", "out_of_scope"],
    "additionalProperties": False
}

parser = JsonSchemaParser(json_schema)
prefix_function = build_transformers_prefix_allowed_tokens_fn(tokenizer, parser)

# Pass prefix_allowed_tokens_fn directly into model.generate
outputs = model.generate(
    **inputs,
    prefix_allowed_tokens_fn=prefix_function,
    max_new_tokens=120,
    temperature=0.0,
    do_sample=False
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
print(response)

Constrained decoding ensures 100% JSON validity, eliminates any markdown formatting leaks, and enforces allowed enum values at runtime.

---

Prompt Specification and Few-Shot In-Context Guidance

AlphaRoute v1.5 is trained on a mixture of zero-shot and dynamic few-shot (1 to 3 exemplars) instruction templates.

Canonical Prompt Structure

You are a precise semantic routing engine. Analyze the input against the provided categories and output only a valid JSON object matching the requested schema. Do not output conversational text or markdown code fences.

TASK:
<task description>

CATEGORIES:
- <category_1>: <description_1>
- <category_2>: <description_2>

OUTPUT SCHEMA:
<json_schema>

[OPTIONAL] EXAMPLES:
Input: "<representative query 1>"
Output: {"target_field": "category_1", ...}
Input: "<representative query 2>"
Output: {"target_field": "category_2", ...}

INPUT:
"<runtime input query>"

JSON:

How Few-Shot Exemplars Improve Accuracy

While AlphaRoute achieves high accuracy out of the box in zero-shot mode, incorporating 1 to 3 few-shot examples is recommended for specialized edge cases:

1. Resolving Ambiguous Domain Boundaries: In complex enterprise domains where two categories share surface terminology (e.g. database connection exhaustion vs backend HTTP socket timeout), providing 1 exemplar for each category sharpens the decision boundary.
2. Multi-Target Schema Value Calibration: When extracting compound fields (such as urgency levels or specialized department queues), few-shot examples anchor the target scale and eliminate bias toward default median values.
3. Internal Acronyms and Technical Vocabulary: Exemplars allow the model to recognize company-specific abbreviations and internal naming conventions without requiring model retraining.

---

Operational Guidelines

• Decoding Temperature: Use temperature = 0.0 (Greedy Decoding / do_sample=False) for deterministic routing decisions and valid JSON schema fidelity.
• Batch Processing: Compatible with batch generation workflows across edge environments (Apple Silicon MPS, NVIDIA CUDA, vLLM).

---

Feedback and Inquiries

If you have any questions, evaluation inquiries, or feedback regarding AlphaRoute, feel free to reach out:

• LinkedIn: Naman Agnihotri