Smol-d1

Small attempt to make my own decision model and also make d1-3B and d1-omni-600m (but without multimodal function), but smaller. yknow.

Training code is here.

How to play with it:

pip install --upgrade unsloth

from unsloth import FastDecisionModel
from datasets import load_dataset

model, tokenizer = FastDecisionModel.from_pretrained(
    "FlameF0X/smol-d1",
    load_in_4bit = True,
)
FastDecisionModel.for_inference(model)

answers = FastDecisionModel.predict(
    model,
    tokenizer,
    "Hi, I was charged twice for invoice #4411. Please refund the duplicate today.",
    {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {
                "billing": "invoices, payments, refunds",
                "technical": "bugs, outages, errors",
                "sales": "pricing, new plans",
            },
        },
        "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this?",
            "criteria": ["not urgent", "soon", "today"],
        },
    },
)
for name, result in answers.items():
    print(name, result["answer"], {k: round(v, 3) for k, v in result["probabilities"].items()})

# Expected output:
team technical {'billing': 0.277, 'technical': 0.383, 'sales': 0.34}
refund False {'true': 0.408, 'false': 0.592}
urgency 2 {'0': 0.19, '1': 0.276, '2': 0.534}

---

metrics = FastDecisionModel.calibrate(model, tokenizer, eval_items)
print(metrics) # held-out accuracy, calibration error (ece) and loss

{'accuracy': 0.5225, 'ece': 0.0670787014067173, 'loss': 1.1651001081243157, 'record_accuracy': 0.075, 'fitted_types': [0, 1, 2]}

---

Evaluation?

test = load_dataset("LocalLLaMA/typed-decisions", "all", split = "test")
test_items, _ = FastDecisionModel.build_dataset(test, tokenizer, model)
print(FastDecisionModel.evaluate(model, tokenizer, test_items))

Oputput:

{'accuracy': 0.497, 'ece': 0.06783864939212797, 'loss': 1.1295486646182835, 'record_accuracy': 0.085}