Aurley/ADM-1

Ministral 3 3B Decision Scorer

An experimental candidate-scoring model built from mistralai/Ministral-3-3B-Base-2512. It scores each supplied answer against an input and question, then normalizes scores across the caller's option set. It returns structured option scores; it does not generate free-form explanations.

Weights and loading

model.safetensors is a single weights file containing the Ministral 3 3B base weights and the trained decision head. config.json and modeling_ministral_decision.py define the custom scoring architecture. The base model is Apache 2.0; retain the dataset attributions below when redistributing this derived model.

The base weights are pre-quantized to 4-bit NF4 with double quantization; the trained decision head stays in floating point. The model is saved as one model.safetensors checkpoint. It was benchmarked with the same NF4 base-weight quantization. A CUDA GPU with enough memory for the 4-bit weights and runtime activations is recommended.

from inference import MinistralDecisionInference

model = MinistralDecisionInference.from_pretrained(".")
result = model.score(
    state="My running shoes arrived in the wrong size. Can I exchange them?",
    question="Which team should handle this?",
    options=[
        {"id": "returns", "label": "Returns", "criteria": "Exchanges, wrong or damaged items"},
        {"id": "shipping", "label": "Shipping", "criteria": "Delivery status, delays, lost packages"},
        {"id": "billing", "label": "Billing", "criteria": "Charges, invoices, payment problems"},
    ],
)
print(result)

The output includes the chosen option ID, per-option probabilities, raw scores, and an abstain flag. Probabilities are temperature-scaled using 205 calibration examples; they still need further calibration before thresholds are used operationally.

Training and evaluation

The base model was frozen and loaded in 4-bit NF4. A scalar MLP scoring head was trained for 12 epochs on 3,072 examples: 1,024 CommonsenseQA training questions and 2,048 BANKING77 training messages. Each question was trained with its correct answer and up to eight sampled negatives. Temperature 1.625 was fit on a separate 205-row calibration split from the official training partitions. Benchmark evaluation rows were held out from both training and calibration.

Benchmark	Evaluation rows	Choices	Top-1 accuracy	Top-k accuracy	Macro-F1	NLL	ECE (10 bins)
CommonsenseQA	1093	5	60.2%	Top-3: 92.8%	0.602	1.104	0.131
BANKING77	231	77	64.5%	Top-5: 95.2%	0.638	1.330	0.165

Compared with the previous 512-example scorer on these same benchmark rows, top-1 accuracy increased by 1.9 percentage points on CommonsenseQA and 6.5 points on BANKING77. BANKING77 uses a held-out sample of three messages per intent (231 total) from the official test split. Random top-1 chance is 20% on five-choice CommonsenseQA and about 1.3% on 77-choice BANKING77. These results are an English task benchmark, not a broad intelligence score; confidence calibration remains limited.

Benchmark charts

Data attribution

• CommonsenseQA, MIT License.
• BANKING77, CC BY 4.0. Attribute PolyAI / Casanueva et al. and link the license when redistributing.

Intended use and limitations

Use for experimentation with bounded, low-risk choice and routing tasks. Candidate lists can be supplied at runtime. The scoring head has only been trained on English commonsense QA and banking intent examples. It may score unfamiliar domains poorly, and probabilities can be miscalibrated. Add an other or defer option where appropriate; validate on representative data before relying on automatic decisions.