llm-d-sc-complexity

A 384-dimensional sentence embedding model fine-tuned to separate prompt complexity tiers, for use as a routing signal by llm-d-sc, the semantic classification runtime for llm-d.

The model does not emit a class directly. It produces an embedding that is ranked against a set of labelled anchors (anchors.json, shipped with this repository). This keeps the taxonomy as data rather than as a frozen classification head: anchors can be replaced or extended without retraining.

Taxonomy

Tier	Meaning
SIMPLE	Single-fact lookup or a one-step instruction
MEDIUM	One substantive task: a function, an explanation with an example, a short guide
COMPLEX	Multi-component design or build with several interacting concerns
REASONING	Proof, derivation, or formal analysis

Intended use

Selecting a serving tier per request. A SIMPLE prompt does not need a frontier model; a REASONING prompt usually does. The classifier emits ranked evidence only. Routing, endpoint selection, and session affinity remain the caller's responsibility.

Evaluation

Evaluated by llm-d-sc on a held-out set of 80 prompts authored independently of the training corpus, 20 per tier, deliberately drawn from domains outside the training pipeline's domain-transfer list (sailing, agriculture, transit, broadcast, museums, orchestras). 20 of the 80 are boundary cases.

Classification method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.

Model	Accuracy	Macro F1	Boundary cases
llm-d-sc-complexity (this model)	0.9750	0.9749	0.9500
all-MiniLM-L6-v2 (base, same anchors)	0.6250	0.6234	0.6500

Per tier:

Tier	Precision	Recall	F1	Support
SIMPLE	1.000	1.000	1.000	20
MEDIUM	0.909	1.000	0.952	20
COMPLEX	1.000	0.900	0.947	20
REASONING	1.000	1.000	1.000	20

Both errors are COMPLEX predicted as MEDIUM, the same boundary the training-time report identified as the model's only remaining confusion. The base model's confidence is near-uniform (0.25-0.27 across four tiers), which is the expected signature of an embedding space that carries no complexity structure at all.

Latency on CPU (single thread, Apple M-series, embed plus rank): p50 9.2 ms, p99 19.0 ms.

These numbers were produced on a homelab and have not been independently reproduced.

Training

Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and group_by_label batch sampling on 871 synthetic examples generated and cross-verified by two separate LLMs. Pipeline: https://github.com/cnuland/hello-chris-sr-finetuned

Usage

from sentence_transformers import SentenceTransformer
import json, numpy as np

model = SentenceTransformer("cnuland/llm-d-sc-complexity")
anchors = json.load(open("anchors.json"))["anchors"]

def classify(text, top_k=3):
    q = model.encode(text, normalize_embeddings=True)
    scores = {}
    for tier, examples in anchors.items():
        sims = model.encode(examples, normalize_embeddings=True) @ q
        scores[tier] = float(np.sort(sims)[-top_k:].mean())
    return max(scores, key=scores.get), scores

print(classify("Prove that the square root of 3 is irrational."))
# ('REASONING', {...})

Limitations

• English only.
• Trained on synthetic data; no human-labelled validation set exists.
• The MEDIUM / COMPLEX boundary is genuinely ambiguous and is where residual error concentrates.
• Anchor quality directly determines accuracy. Replacing anchors.json changes behaviour without retraining.

License

Apache-2.0.