crabby-devil/zils-mini-decision-r16b

torchcast-lora-r16-fix: torchcast-decision-4b + LoRA through the Zils d3-mini chassis

What this is

A full Qwen3.5-4B checkpoint (language and vision tensors, bf16; no lm_head or mtp tensors) for the Zils (Bittensor SN127) d3-mini soup pool.

• Start: torchcast-ai/torchcast-decision-4b @ 492bfb19662f3c6e601cc9c4bd2911ac95609ed4 (v1.3.1, Apache-2.0), a fine-tune of Qwen/Qwen3.5-4B.
• Change: LoRA rank 16, alpha 32, on the 152 attention projections of the language model (q/k/v/o_proj; in_proj_qkv/z/b/a, out_proj), trained through the zils-ai/d3-mini-4.5b-v1 chassis (native prompts, frozen FP32 readout), merged in FP32 and stored in bf16. Embeddings, norms, MLPs and the vision tower are torchcast's.
• Body sha256 (zils.bodies.body_hash): 5ceb8d11a626a444b5ea776cdacdebd91f5c26744cab329752a26ef6a85b34d1
• Relative change from Qwen/Qwen3.5-4B @ 851bf6e806ef over the language layers: 0.01206

Training

• 11706 training rows (arts 2496, language 2489, reasoning 2524, retrieval 2246, tools 1951); 732 optimizer steps of 16 rows, peak lr 0.0001, seed 7.
• No training or selection row shares an id, a source row or a state text with the Zils public seed files (checked at load; 0 rows removed).
• Distillation anchor: the 1,942 rows of four families (retrieval-known-score-k2, arts-known-noul-k5, reasoning-known-choice-k3, arts-known-score-k3) were trained on torchcast-decision-4b's own answer probabilities instead of the generated labels, keeping the starting model's behaviour there.
• Checkpoint: step 732, chosen on the merged-weights dev score.

Licence

Apache-2.0: see LICENSE and NOTICE (torchcast's notice is kept and the modification is stated).

Use

Zils validators grade it as a soup body (zils.soup.build on the pinned chassis, then zils.d3.DecisionModel). config.json is Qwen/Qwen3.5-4B's (tied embeddings), so transformers also loads it as Qwen3_5ForConditionalGeneration.