brucoder/winter-frost-1-pro

❄️ Winter‑Frost

A small language model, built entirely from scratch — tokenizer, pretraining, and instruction‑tuning — on a single free GPU.

---

Winter‑Frost is not built to compete with production models. It's a from‑first‑principles build: every stage of the pipeline — tokenization, pretraining, instruction tuning, deployment — was implemented and run from scratch on free‑tier hardware, as a hands‑on way to actually understand how LLMs work under the hood.

📋 Model summary

🧠 Architecture	GPT‑2 (decoder‑only transformer)
🔢 Parameters	~111M
✂️ Tokenizer	Custom Byte‑Level BPE, vocab size 32,000
📏 Context length	1,024 tokens
📚 Pretraining data	~4GB custom text corpus
🎯 Instruction tuning	One epoch over the Alpaca dataset (~51,760 examples)
⚙️ Training hardware	Single free‑tier NVIDIA T4 GPU (Google Colab)
🎛️ Precision	fp16 mixed precision

🛠️ How it was built

┌─────────────────┐     ┌──────────────────┐     ┌───────────────────┐     ┌─────────────┐
│  1. Tokenizer    │ ──▶ │  2. Pretraining   │ ──▶ │  3. Instruction    │ ──▶ │  4. Deploy   │
│  Byte‑Level BPE  │     │  Raw next‑token   │     │     tuning         │     │  Push to     │
│  trained from    │     │  prediction on    │     │  Alpaca dataset,   │     │  Hugging     │
│  scratch, 32k    │     │  ~4GB corpus      │     │  teaches Q→A       │     │  Face Hub    │
│  vocab           │     │  (GPT‑2, 111M)    │     │  format            │     │              │
└─────────────────┘     └──────────────────┘     └───────────────────┘     └─────────────┘

1. Tokenizer — A Byte‑Level BPE tokenizer was trained from scratch on the target corpus, rather than reusing an existing model's vocabulary.

2. Pretraining (Phase 1) — The tokenizer streamed-encoded the ~4GB corpus into binary token files, feeding a GPT2LMHeadModel (768 hidden size, 12 layers, 12 heads) trained on next‑token prediction. This teaches raw language patterns — grammar, style, associations — but not instruction‑following.

3. Instruction tuning (Phase 2) — The pretrained checkpoint was fine‑tuned on the Alpaca instruction dataset via trl's SFTTrainer, teaching the model to respond to a prompt instead of just continuing it.

4. Deployment — Final weights and tokenizer were pushed straight to the Hugging Face Hub from the training environment.

🚀 How to use it

Quick start

pip install torch transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_dir = "brucoder/winter-frost"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is a computer?

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(device)
output = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    pad_token_id=tokenizer.pad_token_id,
)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response.strip())

📝 Prompt format

The model expects the Alpaca‑style instruction template it was fine‑tuned on:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{your question or task here}

### Response:

Prompts that don't follow this format will still generate text, but responses tend to drift off‑task.

🎛️ Generation tips

Setting	Effect
repetition_penalty 1.2–1.4	Reduces looping / repeated phrases
temperature 0.5 (lower)	More focused, less rambling output
do_sample=False	Deterministic, greedy (most‑likely‑token) output

⚠️ Limitations — please read before using

This model is small, and was trained on a small amount of data relative to production LLMs. Please calibrate expectations accordingly:

• 🧩 World knowledge is limited and unreliable — it only knows what appeared in its ~4GB training corpus plus Alpaca's instruction patterns. It will confidently generate plausible‑sounding but incorrect information.
• ➗ Math and code are not reliable — training data was not focused on arithmetic or programming.
• 🌀 Topic drift — on ambiguous or open‑ended prompts, the model may default to whatever topics were over‑represented in training, rather than staying on‑topic.
• 🎯 Not instruction‑perfect — it recognizes the instruction → response format but doesn't always answer the actual question asked.
• 🚫 Do not use for factual, medical, legal, financial, or safety‑critical purposes.

🎓 Intended use

Educational and experimental use — exploring from‑scratch LLM training, tokenizer behavior, small‑model generation quality, and the practical realities of training on constrained (free‑tier) hardware.

🔗 Related

A larger follow‑up model, winter-frost-1-pro (~774M parameters, broader pretraining corpus via streaming), is trained under the same from‑scratch philosophy on the same free‑tier hardware. It's meaningfully bigger and broader, but shares the same category of limitations described above.

🙏 Acknowledgements

Built as a solo learning project using Hugging Face transformers, datasets, trl, tokenizers, and Google Colab's free GPU tier.

Made with curiosity, patience, and a lot of Colab reconnects. ❄️