Digilidaahz/Freedream-1B-Instruct-V2

Freedream-1B-Instruct V2

Freedream-1B is a ~977M parameter language model trained completely from scratch.

To be absolutely clear: I am a Grade 8 independent developer, and this model was built, pre-trained, and fine-tuned entirely on a single consumer GPU (RTX 5080 16GB). It is not a fine-tuned wrapper of Llama or Qwen; it is a custom GPT architecture built from the absolute ground up.

Current Versions:

• Freedream-1B-Instruct-V2-Preview: The preview of the ultimate V2. It fixes the tokenizer problem where { was treated as an UNK token, and is trained on a base model that has seen 24.1B tokens. The final V2 release will be based on a 50B tokens seen model.
  • Massive SFT Data Upgrade: Trained on ~220,000 highly curated instruction pairs (Magpie, OpenHermes, Ember literature, and a custom identity dataset).
  • Tokenizer Fixed: Vocabulary expanded to 32,001, successfully hot-swapping the missing { embedding to support code and JSON syntax.
  • Identity Purged: Completely eliminated the "Alex" identity poisoning and the "ARAS-EMBER" literature roleplay leakage from the Ember dataset.

Model Details

• Architecture: Decoder-only Transformer
• Parameters: 977,300,992 (24 layers, 1792 hidden size, 16 heads)
• Context Length: 4096 tokens
• Key Components: SwiGLU, RoPE, Weight-tied embeddings.
• Training Hardware: Single NVIDIA RTX 5080 (16GB VRAM) + 64GB DDR5.

Comprehensive Benchmarks

Evaluated using the standard lm-evaluation-harness (0-shot setting) on a single RTX 5080.

Despite architectural modifications and massive full-parameter SFT in V2, the model successfully avoided catastrophic forgetting and maintained stable general capabilities.

Benchmark	Metric	V1	V1.1	V2 (Preview)	Llama-3.2-1B (Reference)
ARC-Challenge	acc_norm	29.69%	30.46%	30.63%	~59.4%
HellaSwag	acc_norm	45.82%	45.68%	45.93%	~59.0%
PIQA	acc	68.38%	68.55%	67.95%	~74.0%
WinoGrande	acc	52.64%	53.27%	53.67%	~59.0%
Wikitext-2	word_ppl	~26.45	26.31	25.94	~11.8

Engineering Insights on the Scores:

1. No Catastrophic Forgetting: V2 underwent extremely aggressive full-parameter SFT on 220k+ samples. The fact that HellaSwag (45.93%) and WinoGrande (53.67%) scores remained perfectly stable (or slightly improved) proves the high quality and purity of the mixed SFT dataset.
2. Extreme Compute Efficiency: Meta's Llama-3.2-1B was trained on 15 Trillion tokens. Freedream was trained from scratch on only ~24 Billion tokens (~0.1% of Llama's data). Achieving 67.95% on PIQA and 45.93% on HellaSwag with this tiny budget demonstrates the efficiency of the custom architecture and FP8 training pipeline.
3. Perplexity Context: The 25.94 word_ppl on Wikitext-2 (compared to Llama-3.2-1B's ~11.8) is expected due to two physical constraints:
   • Vocabulary Size: Llama uses a 128K vocabulary (massive compression), while Freedream uses a 32K vocab, making the prediction task inherently harder at the token level.
   • Alignment Tax: SFT on conversational and literature data shifts the model's probability distribution away from raw Wikipedia text generation.

Base Model & Training Pipeline

This Instruct version is directly fine-tuned from my CPT release: Digilidaahz/Freedream_1B_BaseModel.

• Pre-training: The base model saw exactly 24.1B tokens of the FineWeb-Edu dataset.
• SFT (Supervised Fine-Tuning): Fine-tuned on 220,000 highly curated samples, including SmolTalk (Magpie, Everyday Chats), OpenHermes (Code/Logic), Ember (Creative Writing), and a custom identity grounding dataset.

Engineering Highlights

Training a model from scratch on consumer hardware requires extreme optimization:

• Tokenizer Hot-swapping: Safely expanded the vocabulary from 32,000 to 32,001 to add the missing { token, inheriting old embedding weights and resetting optimizer states to prevent gradient explosions.
• Data Poisoning Purification: Hand-wrote regex pipelines to sanitize 290MB of SFT text, physically eliminating identity contamination (Alex, ARAS-EMBER) and formatting truncations.
• I/O Deadlock Bypass: Hand-wrote an asynchronous DataPrefetcher (using 16MB chunked random reads + double buffering) to completely bypass disk I/O bottlenecks and WSL deadlocks.
• Extreme Throughput: Achieved a steady 20,300+ tokens/sec during SFT by integrating FP8 mixed precision, chunked Cross-Entropy, and AdamW8bit.
• Hardware Steer-arounds: Successfully navigated the Blackwell sm_120 architecture limitations (abandoned NVFP4 due to missing Triton .rs instructions) and optimized FP8 Tensor Core alignment to prevent gradient collapse.

Known Limitations (Read This!)

As a 1B model trained from scratch on a single consumer GPU, Freedream has physical limitations:

• Math Hallucinations: While it can do basic arithmetic, it may generate mathematical loops under zero-shot abstract prompts. Do not use it as a calculator.
• Code: Since the pre-training phase (FineWeb-Edu) lacked code data, V2's code generation logic is heavily bottlenecked. It can format code perfectly, but the underlying logic may hallucinate. (Scheduled for CPT data mixture in V3).
• Long-form Coherence: As a 1B model, generating long-form stories (like sonnets) might cause logic to drift or break strict formatting constraints.
• Safety: This is an experimental 1B Small Language Model (SLM) trained from scratch. It has NOT undergone RLHF or safety alignment. It will generate hallucinated, biased, or highly inappropriate content if prompted maliciously. This model is for research and educational purposes only.

How to Use (Prompt Format)

This model uses a custom tokenizer and format. Please use the exact format below:

<|system|>
Your name is Freedream. You are a helpful AI assistant.
<|end|>

<|user|>
Hello!
<|end|>

<|assistant|>