Digilidaahz/Freedream-1B-Instruct-V1

Freedream-1B-Instruct V1

Freedream-1B is a ~977M parameter language model trained completely from scratch.

To be absolutely clear: I am a Grade 8 independent developer, and this model was built, pre-trained, and fine-tuned entirely on a single consumer GPU (RTX 5080 16GB). It is not a fine-tuned wrapper of Llama or Qwen; it is a custom GPT architecture built from the absolute ground up.

Current Versions:

• Freedream-1B-Instruct-V1: Fine-tuned on the initial SmolTalk mixture. Learned basic chat, but often hallucinates on facts and frequently breaks formatting.
• Freedream-1B-Instruct-V1.1 (Recommended): Fine-tuned on a highly curated SmolTalk mixture with enhanced formatting alignment. Improved instruction-following and can generate coherent literature/poems. (Note: V1.1 still utilizes the 32000 vocabulary. The missing { tokenizer bug fix is scheduled for the upcoming V2 release). Still exhibits physical limitations in math and code. (Note: The V1.1 weights are available in this repo as Freedream-1B-instruct-V1.1.pt)

Model Details

• Architecture: Decoder-only Transformer
• Parameters: 977,300,992 (24 layers, 1792 hidden size, 16 heads)
• Context Length: 4096 tokens
• Key Components: SwiGLU, RoPE, Weight-tied embeddings.
• Training Hardware: Single NVIDIA RTX 5080 (16GB VRAM) + 64GB DDR5.

Comprehensive Benchmarks (V1 vs V1.1)

Evaluated using the standard lm-evaluation-harness (0-shot setting) on a single RTX 5080.

Despite architectural modifications and full-parameter SFT in V1.1, the model successfully avoided catastrophic forgetting and maintained stable general capabilities.

Benchmark	Metric	V1	V1.1	Llama-3.2-1B (Reference)
ARC-Challenge	acc_norm	29.69%	30.46%	~59.4%
HellaSwag	acc_norm	45.82%	45.68%	~59.0%
PIQA	acc	68.38%	68.55%	~74.0%
WinoGrande	acc	52.64%	53.27%	~59.0%
Wikitext-2	word_ppl	~26.45	26.31	~11.8

Engineering Insights on the Scores:

1. No Catastrophic Forgetting: V1.1 underwent full-parameter SFT. The fact that HellaSwag (45.68%) and PIQA (68.55%) scores remained perfectly stable (and WinoGrande even increased by 0.6%) proves the high quality of our mixed SFT dataset (Magpie, OpenHermes, SystemChats). It successfully injected identity without destroying the base intelligence.
2. Extreme Compute Efficiency: Meta's Llama-3.2-1B was trained on 15 Trillion tokens. Freedream was trained from scratch on only ~20 Billion tokens (~0.1% of Llama's data). Achieving 68.55% on PIQA and 45.68% on HellaSwag with this tiny budget demonstrates the efficiency of the custom architecture and FP8 training pipeline.
3. ARC-Challenge Improvement: V1.1 shows a noticeable improvement on ARC-Challenge (+0.77%), likely due to the better formatting alignment during inference.
4. Perplexity Context: The ~26.31 word_ppl on Wikitext-2 (compared to Llama-3.2-1B's ~11.8) is expected due to two physical constraints:
   • Vocabulary Size: Llama uses a 128K vocabulary (massive compression), while Freedream uses a 32K vocab, making the prediction task inherently harder at the token level.
   • Alignment Tax: SFT on conversational data slightly shifts the model's probability distribution away from raw Wikipedia text generation.

Base Model & Training Pipeline

This Instruct version is directly fine-tuned from my previous release: Digilidaahz/Freedream_1B_BaseModel.

• Pre-training: The base model saw exactly 20B tokens of the FineWeb dataset.
• SFT (Supervised Fine-Tuning): Fine-tuned on a highly curated mixture of SmolTalk (instruction following, reasoning), OpenHermes, and a custom 2MB identity grounding dataset.

Engineering Highlights

Training a model from scratch on consumer hardware requires extreme optimization:

• I/O Deadlock Bypass: Hand-wrote an asynchronous DataPrefetcher (using 16MB chunked random reads + double buffering) to completely bypass disk I/O bottlenecks and WSL deadlocks.
• Extreme Throughput: Achieved a steady 21,000 tokens/sec during pre-training by integrating FP8 mixed precision and AdamW8bit.
• Hardware Steer-arounds: Successfully navigated the Blackwell sm_120 architecture limitations (abandoned NVFP4 due to missing Triton .rs instructions) and optimized FP8 Tensor Core alignment to prevent gradient collapse.

Known Limitations (Read This!)

As a 1B model trained from scratch on a single consumer GPU, Freedream has physical limitations:

• Math Hallucinations: While it can do basic arithmetic, it may generate mathematical loops under zero-shot abstract prompts. Do not use it as a calculator.
• Identity Drift: Due to early-stage over-fitting on specific SmolTalk data in V1, models may occasionally hallucinate names like "Frey" or "Freeman". V1.1 significantly mitigates this but might still drift under extreme prompts.
• Code: Capable of writing basic Python functions, but not complex software systems. (Note: Code generation is heavily bottlenecked by the missing { in the V1/V1.1 tokenizer. This will be resolved in V2).
• Safety: This is an experimental 1B Small Language Model (SLM) trained from scratch. It has NOT undergone RLHF or safety alignment. It will generate hallucinated, biased, or highly inappropriate content if prompted maliciously. This model is for research and educational purposes only.

How to Use (Prompt Format)

This model uses a custom tokenizer and format. Please use the exact format below:

<|system|>
Your name is Freedream. You are a helpful AI assistant.
<|end|>

<|user|>
Hello!
<|end|>

<|assistant|>