Gramli/complexity-scorer

Complexity Scorer

Complexity Scorer is a fine-tuned Qwen3-8B model that estimates how difficult a user request is for a general-purpose large language model to complete correctly.

It returns a single complexity score between 0.00 and 1.00 in JSON format.

The goal is simple: help developers make smarter LLM routing decisions without depending on any specific AI provider.

For example, an application might send simple requests to a cheaper model while reserving more capable reasoning models for difficult tasks.

Complexity Scorer doesn't select the model. It provides a complexity score, and the client decides what to do with it.

Complexity levels

Score	Level	Description
0.00	Trivial	Extraction, formatting, or deterministic transformations
0.25	Easy	Straightforward tasks requiring limited interpretation
0.50	Moderate	Multiple steps, domain knowledge, or several constraints
0.75	Hard	Complex reasoning, difficult tradeoffs, or specialized expertise
1.00	Very Hard	Deep reasoning, expert analysis, or complex system interactions

Complexity is based on the reasoning required to produce a correct response, not simply the length of the prompt.

Example

Input:

Review this distributed locking implementation and identify race conditions that could cause duplicate payments.

Example output:

{"complexity":0.75}

The score is a classification label, not a calibrated probability of success or failure.

Model details

Property	Value
Base model	Qwen/Qwen3-8B
Fine-tuning platform	Tinker by Thinking Machines Lab
Training method	Supervised fine-tuning (LoRA)
Training examples	2,500
Complexity classes	5
LoRA rank	32
Learning rate	1e-4
Batch size	16
Epochs	1
Maximum sequence length	4,096 tokens
Evaluation mode	Non-thinking
Evaluation temperature	0

Only the LoRA adapter is provided in this repository. The original Qwen3-8B model is required for inference.

Training dataset

The model was fine-tuned using 2,500 synthetically generated examples, balanced across five complexity levels, with 500 examples per class.

The dataset covers a variety of domains, including software development, APIs, databases, cybersecurity, business, writing, data analysis, and general reasoning.

The examples were designed to include both short but difficult requests and long but simple requests to discourage the model from using prompt length as a shortcut.

Training targets contain only the expected JSON complexity score. Explanations used during dataset generation were not part of the model's target output.

Evaluation results

All models were evaluated against the same 50-example synthetic evaluation dataset, containing 10 examples per complexity level.

Model	Training examples	Exact accuracy	MAE
Base Qwen3-8B	0	50%	0.125
Fine-tuned	50	86%	0.035
Fine-tuned	500	74%	0.065
This model	2,500	76%	0.060

Interestingly, the model trained on only 50 examples achieved the highest accuracy on this evaluation set.

Increasing the dataset to 2,500 examples did not improve the measured accuracy beyond that smaller fine-tune.

Possible explanations include inconsistencies in synthetic labels, differences in training configurations, and the limited size of the evaluation dataset.

Final model performance

For this 2,500-example model:

• Exact accuracy: 76% (38/50)
• Mean absolute error: 0.060
• Valid JSON responses: 100% (50/50)
• Incorrect predictions: 12

All 12 incorrect predictions differed from the expected score by just one adjacent complexity level (0.25).

Limitations of these results

These results are experimental and should not be interpreted as production-level accuracy.

The evaluation dataset is small, synthetic, and was reused during development to compare training configurations. As a result, it is not a fully independent final benchmark.

The model has not been validated against large-scale real-world API traffic or measured for actual cost savings across different LLM providers.

Usage

Install the required packages:

pip install torch transformers peft accelerate

Load the base model and LoRA adapter:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen3-8B"
ADAPTER = "Gramli/complexity-scorer"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    device_map="auto",
    torch_dtype="auto"
)

model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

Run inference:

system_prompt = """
You are a prompt complexity classifier.

Estimate how difficult the user's request is for a capable
general-purpose LLM to answer correctly.

Complexity scores:
0.00 - Trivial
0.25 - Easy
0.50 - Moderate
0.75 - Hard
1.00 - Very Hard

Judge reasoning difficulty, not prompt length.

Return JSON only:
{"complexity":0.50}
"""

messages = [
    {"role": "system", "content": system_prompt},
    {
        "role": "user",
        "content": "Explain distributed transactions in microservices."
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

inputs = tokenizer(
    [text],
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=False
)

generated = outputs[0][inputs["input_ids"].shape[-1]:]

print(tokenizer.decode(generated, skip_special_tokens=True))

Important: For reproducible evaluation results, use the exact system prompt and inference settings from the project's evaluation implementation. The shorter system prompt above demonstrates usage but is not guaranteed to reproduce the published benchmark.

This example runs the model locally after downloading the necessary weights. A suitable machine with sufficient memory is required.

Potential applications

• LLM routing: Help applications decide when to use cheaper or more capable models.
• Cost optimization: Provide an additional signal for deciding how to allocate LLM requests.
• AI gateways: Classify incoming requests before forwarding them to a model provider.
• Usage analytics: Analyze the complexity distribution of application prompts.

The scorer is provider-agnostic. Applications can interpret its scores according to their own available models, pricing, and requirements.

Limitations

• Complexity is subjective, particularly near class boundaries.
• The output