ToastyPigeon/gemma-4-12b-full-cpt

Gemma-4-12B — full-CPT prose adapter (QLoRA)

A continued-pretraining LoRA on google/gemma-4-12b (base, non-instruct), trained 2026-08-29 → 2026-09-01 (1740 optimizer steps) on a prose/story-heavy mix.

Approach

• Regular CPT QLoRA: 4-bit NF4 frozen body + bf16 LoRA (r=64, alpha=64, dropout 0.05, dropout-free inference), targeting q/k/v/o/gate/up/down projections, SWA layers only (swa_only: true). Embeddings/head tied and frozen.
• Sequence length 8192 (stride = 8192, non-overlapping packs), shuffled before packing.
• DDP across 2×24GB (micro_bs 1, grad_accum 8, effective batch 16).
• cut_cross_entropy loss path honoring Gemma's final_logit_softcapping = 30.0.
• LR 5e-05 (LoRA), 5% warmup, cosine, weight decay 0, grad clip 1.0.
• Loss: avg 2.42 (first 100 steps) → 2.25 (last 100 steps).

Training data

Mix of story/prose and chat-log corpora (the named sets are public under the ToastyPigeon namespace): erotic-books-clone, new-story-dataset, some-erotica, steve-and-marvin, SpringDragon-chat, disco-chat, counter-signal — plus two other private datasets with a focus on WLW content and chat.

Content note: parts of the mix are adult/erotic fiction. Intended for fiction-writing research and style experimentation by adults; not for production deployment or use by minors.

Intended use

• Text-completion / storywriting experiments on top of the Gemma-4-12B base
• As a starting point (adapter stack/merge) for downstream RP/style finetunes

Notes

• This is a CPT adapter, not an instruct/chat model — expect raw-completion behavior.
• Trained with the box's in-house CPT trainer; see cpt_meta.json and adapter_config.json for exact settings.