Bazza 0.5.0

Full source, training, and evaluation on GitHub

Bazza is a System 1 decision model implemented in PyTorch. It scores supplied candidate answers using a frozen Gemma 4 12B IT backbone and a separate trained decision head: a bidirectional GRU, recurrent attention, and a gated pretrained readout.

This repository contains the head weights and inference code, not the Gemma backbone. The head is a custom decision module, not a PEFT/LoRA adapter. It does not load through Transformers AutoModel or text-generation pipelines.

Loading

Use Python 3.10 or newer and a suitable PyTorch/CUDA installation. Clone this repository with Git LFS installed, or download it using snapshot_download.

git clone https://huggingface.co/0xdeadbe3f/Bazza
cd Bazza
pip install -r requirements.txt

The loader downloads the head and the exact backbone revision into the Hugging Face cache. The initial backbone download contains the original BF16 weights and is substantially larger than the 115 MiB head; linear weights are quantized to bitsandbytes NF4 during CUDA loading. Embeddings and normalization remain BF16. The evaluated configuration used an RTX 3090 with 24 GiB VRAM.

from systemone import SystemOne

model = SystemOne.from_pretrained("0xdeadbe3f/Bazza", revision="0.5.0")

result = model(
    state={
        "ticket": "Order #1182 arrived broken. The customer requests a refund.",
        "order_age_days": 40,
        "refund_policy_days": 30,
    },
    questions={
        "refund": {
            "type": "noul",
            "instructions": "Is the refund allowed under the policy?",
            "criteria": {
                "true": "The refund is allowed by the stated policy.",
                "false": "The refund is not allowed by the stated policy.",
            },
        },
        "action": {
            "type": "choice",
            "instructions": "Select the next action using the stated policy.",
            "criteria": {
                "refund_order": "Issue a refund because the policy permits it.",
                "explain_policy": "Explain that the order is outside the refund window.",
                "ask_order_age": "Ask the order age because it is unknown.",
            },
        },
    },
)

print(result["answers"]["refund"]["value"])          # boolean decision
print(result["answers"]["refund"]["noul"])           # probability assigned to true
print(result["answers"]["action"]["choice"])         # selected criterion key
print(result["answers"]["action"]["probabilities"])  # scores for all candidates

Pass backbone_path="/path/to/Gemma4-12B-it" to reuse existing local weights. For offline use, call SystemOne.from_pretrained(".", backbone_path=..., local_files_only=True). The bundled Python implementation is imported normally; no remote Python code is downloaded or executed by from_pretrained.

Output types

Type	Criteria	Result
choice	Named candidate descriptions	Selected key and candidate probabilities
noul	Descriptions keyed by "true" and "false"	Probability of true and a boolean value
score	Descriptions keyed by numeric levels	Probability-weighted mean of those levels

Each question uses 2–26 candidates and is scored separately. Returned probabilities are softmax scores; they have not been calibrated. The current API accepts text decisions with JSON-compatible state. Overlong inputs are rejected rather than truncated.

Head-only adaptation

Training updates the decision head while leaving all pretrained backbone weights unchanged. The head is stored separately and can train on reusable token features.

Adaptation cost	Current model
Trainable parameters	30,117,964 (about 0.25% of 12B)
Backbone parameters updated	0
Head weight tensors, FP32	115.3 MiB
One pass over 96,000 cached examples, RTX 3090	8.93 minutes

The training time includes validation, checkpointing, and logging. It excludes feature extraction, model setup, and subsequent testing. Feature extraction is an additional cost that can be reused across head-training experiments. The weight size excludes optimizer state and checkpoint metadata.

Benchmarks

Accuracy (%) on the same 13,499 public-benchmark requests for every model. MMLU-Pro uses a 2,000-example test subset. Bazza gain is relative to stock Gemma; positive values indicate an improvement in percentage points.

Benchmark	Examples	Stock Gemma 12B (NF4)	Bazza 12B (NF4)	Bazza gain (pp)	Winnow 12B (Q8)	PPLX-Decider v1.1 27B (NF4)
When2Call (test)	3,652	63.34	79.41	+16.07	83.19	74.70
ARC-Challenge (test)	1,172	90.27	92.41	+2.13	94.88	97.18
ANLI round 3 (test)	1,200	58.25	61.42	+3.17	62.17	70.17
WinoGrande (validation)	1,267	71.82	71.43	-0.39	73.88	92.34
MMLU-Pro (test subset)	2,000	44.00	48.30	+4.30	54.30	68.15
GSM8K, 4 choices	1,319	50.87	62.32	+11.45	61.18	81.12
GSM8K, 10 choices	1,319	24.34	43.97	+19.64	35.86	71.04
CRUXEval output choice	570	56.67	55.44	-1.23	69.65	85.26
RAGTruth classification	1,000	76.90	82.90	+6.00	82.60	88.40
Eight-dataset macro	—	62.36	68.06	+5.70	71.15	81.54

GSM8K and CRUXEval use adapted multiple-choice formats; RAGTruth uses binary response-level classification accuracy. The macro weights eight datasets equally, combining the two GSM8K views first; they share the same 1,319 source problems.

Bazza uses the frozen Gemma 4 12B IT backbone and the head checkpoint trained on Groundbeef v0.5 at step 2,661 (80,019 training examples), selected on validation. Stock Gemma uses its original answer-letter weights and softcapping on the same cached features and prompts as Bazza. The head adds 16.07 percentage points on When2Call and 4.30 points on MMLU-Pro; WinoGrande drops by 0.39 points.

All runs use an RTX 3090 with no calibration fitted on the test panel. Winnow and PPLX-Decider use their own prompts and readouts. These are decision-format accuracy measurements, not official Decision Index scores or generative benchmark results.

Training data and selection

Groundbeef is our internally curated training dataset: a mixture of existing source datasets with our preprocessing and data splits. This release uses Groundbeef v0.5. The dataset is not included and may be released separately. The published head was selected on validation at step 2,661 (80,019 examples) from a 96,000-example training run. Backbone weights were frozen throughout.

Files and provenance

• model.safetensors: inference head parameters and required readout buffers.
• config.json: architecture, pinned backbone identity, tensor checksum, tokenizer identity, and selected-checkpoint provenance.
• systemone.py, model/, data.py, and utils.py: PyTorch inference source.
• LICENSE, NOTICE, and model/NOTICE.md: Apache 2.0 and third-party notices.

Optimizer state, random state, local paths, data caches, and training records are excluded. Configuration format version 1 is specific to Bazza.

The full implementation, including training, evaluation, and feature-cache support, is at deadbe3f/Bazza.

License

Apache License 2.0. See LICENSE, NOTICE, and third-party attributions.