Hanish/lq-decide-1.7b

LQ-Decide 1.7B

A model that answers typed decisions instead of generating text. You give it a state and a fixed option set; it returns a probability distribution over those options from a single forward pass. Nothing is sampled, so the output cannot be malformed and there is nothing to parse or repair.

Built by Hanish Keloth for LocalQuill, a private on-device AI keyboard and offline chat app. Trained from Qwen/Qwen3-1.7B. A smaller 0.6B version is also available.

Results

Two labelled public fixtures from the SemIf project, never seen in training. Balanced accuracy, three options per item, chance is 0.333.

	authored144	perturbations108
LQ-Decide 1.7B, Q8_0 GGUF	0.812	—
LQ-Decide 1.7B, bf16	0.797	0.628
LQ-Decide 1.7B, Q4_K_M GGUF	0.762	—
LQ-Decide 0.6B, Q8_0	0.750	0.586
SemIf, Qwen3.5-4B, published	0.813	—
Jev (TypeSafe, closed), published	0.883 on 102 aligned rows	—

The bf16 row is measured through transformers and the GGUF rows through llama.cpp; the 0.015 spread between bf16 and Q8_0 is within noise on 144 items and should not be read as quantization improving the model.

Per family, bf16, permutation-averaged:

family	authored144	perturbations108
evidence interpretation	0.812	0.667
rule application	0.771	0.556
candidate selection	0.771	0.583

The perturbation set contains modified variants of the same items and is the better test of whether the model reads the distinction or the wording. The gap between 0.797 and 0.628 is the honest measure of how much of this is robust.

Quantization costs real accuracy here

Measured on both sizes, same fixture, same prompts:

model	Q8_0	Q4_K_M	difference	median latency Q8 → Q4
0.6B	0.750	0.703	−4.7	37 ms → 36 ms
1.7B	0.812	0.762	−5.0	67 ms → 59 ms

Because the entire output is a probability over a handful of option tokens, four-bit error lands directly on the thing you read rather than being spread across generated text. Use Q8_0. Q4_K_M buys file size and, at 1.7B, eight milliseconds, for five points of accuracy.

Calibration

A single temperature constant, fitted on a held-out split, shipped in calibration.json.

	expected calibration error
raw	0.161
after T = 2.3	0.039
after T = 2.3, permutation-averaged	0.031

Averaging logits over all option orderings costs nothing in accuracy, improves calibration, and removes position bias in the option list. Apply the temperature before reading probabilities as confidence.

Latency

llama.cpp on an Apple M5 Max, 147-token prompts, one decision per request:

quant	median	p95	size
Q4_K_M	59 ms	72 ms	1,056 MB
Q8_0	67 ms	81 ms	1,749 MB

No phone measurement is published, by us or, as far as we can find, by anyone else working on this pattern. Do not assume desktop numbers transfer to a handset.

Use

import torch, json
from transformers import AutoTokenizer, AutoModelForCausalLM

LETTERS = "ABCDEFGHIJKLMNOP"
SYSTEM = ("Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. "
          "Respond with only its uppercase letter, with no explanation or reasoning.")

tok = AutoTokenizer.from_pretrained("Hanish/lq-decide-1.7b")
model = AutoModelForCausalLM.from_pretrained("Hanish/lq-decide-1.7b", dtype=torch.bfloat16).eval()
T = 2.3   # calibration.json

def decide(state, question, options):
    payload = {"evidence": state, "criterion": question,
               "options": [{"letter": LETTERS[i], "description": d} for i, d in enumerate(options)]}
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(payload, ensure_ascii=False)}]
    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, return_tensors="pt")
    with torch.no_grad():
        logits = model(**ids).logits[0, -1]
    sel = torch.tensor([logits[tok.encode(LETTERS[i], add_special_tokens=False)[0]]
                        for i in range(len(options))]).float()
    return torch.softmax(sel / T, -1).tolist()

print(decide(
    "The optician ordered replacement lenses. The workshop confirms they have not yet been fitted.",
    "Assess the claim: the replacement lenses have been fitted.",
    ["The evidence establishes the claim",
     "The evidence does not establish either",
     "The evidence establishes the opposite"]))

The prompt format matters; the model was trained on exactly this template, with options shuffled and the answer letter re-based so it reads descriptions rather than positions. With llama-server, request one token with logprobs and read the probability mass on the option letters.

Training

LoRA rank 32 on all projections, merged. Loss on the single answer-letter token only. Options permuted and the letter set re-based every epoch. 117,096 rows, two epochs, 19,516 steps, about 2 h 50 m on one L40S.

Data: Hanish/lq-decide-data, 141,038 rows, every source licence-tagged and non-commercial sources excluded. 57,096 rows are synthetic items in three decision families, where the intended answer was fixed before generation so the teacher only wrote surface text and never supplied a label. The evaluation fixtures were never trained on, generated from, or shown to the teacher.

Limitations

• English only.
• Three to sixteen options; not tested beyond that.
• Perturbation robustness is well below clean-fixture accuracy, 0.628 against 0.797.
• Not compared against Laya or other recent decision models, which publish on their own benchmarks; this one has not been run on them. The table above is not a ranking against anything not in it.

Credits

Base model Qwen/Qwen3-1.7B (Apache-2.0, Alibaba). Evaluation fixtures from the SemIf project (MIT). Synthetic data generated with Ornith-1.5-35B-A3B (MIT). The interface pattern follows TypeSafe's Jev, a separate closed product; this model is not affiliated with or endorsed by it.