Yosua69/one_bit_blm_chimera_causal_fix

One-Bit BLM Chimera — Causal Fix

Byte-level Language Model + Mixture of Experts + 1-bit BitNet, distilled dari Qwen/Qwen2.5-14B.

Training dijalankan dengan 2 GPU terpisah: satu khusus menjalankan teacher (inference only), satu lagi khusus training student (forward, backward, optimizer) -- bukan replikasi data-parallel.

Quick Start

from huggingface_hub import hf_hub_download
exec(open(hf_hub_download("Yosua69/one_bit_blm_chimera_causal_fix", "inference.py")).read())

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model  = load_from_hub(device=device)
print(generate(model, device, prompt="The history of", max_bytes=150))

Arsitektur

Komponen	Detail
Input	Raw bytes (vocab: 257, termasuk EOS)
Sequence mixing	GRU multi-layer, causal
Core	4× blok MoE per-posisi (4 experts masing-masing)
Linear	1-bit AbsMean quantization (BitNet)
Generation	Incremental, hidden state di-cache
Teacher	Qwen/Qwen2.5-14B (4-bit, knowledge distillation)