chienhsinlin/helioslm-toy

HeliosLM toy_v5.13

Char-level toy checkpoint for HeliosLM — a from-scratch DeepSeek-V3/K3-style LLM stack in pure PyTorch (runs on CPU).

Files

File	Description
toy_v5.13.pt	PyTorch state_dict (83 tensors, fp32)
model-f32.gguf	GGUF v3 export, bit-exact fp32
model-f16.gguf	GGUF v3 export, fp16 (load-balancing counters excluded — they overflow f16)
toy_v5.13.json	Training metadata (val_loss, samples, MTP acceptance)

Model

• 8.5M params, char-level vocabulary (token ids = ord(c), vocab 1024)
• Trained in ~10 CPU-minutes on the HeliosLM repo's own source code
• val_loss 2.41; MTP draft acceptance 1.0 on the training distribution
• Architecture (lite config): 2 layers, MLA attention with weight absorption, sigmoid-gated MoE with auxiliary-loss-free load balancing, MTP head
• This is a smoke-test-scale reference model, not a capable text model

Usage

import torch
from helioslm_v5.configs.config_v5 import HeliosLMv5Config
from helioslm_v5.src.model_v5 import HeliosLMv5

ckpt = torch.load("toy_v5.13.pt", map_location="cpu", weights_only=False)
model = HeliosLMv5(HeliosLMv5Config(size="lite"))
model.load_state_dict(ckpt["state_dict"])
model.eval()

out = model.generate(torch.tensor([[ord(c) for c in "# HeliosLM"]]),
                     max_new_tokens=20, temperature=0)
print("".join(chr(t) for t in out[0]))

GGUF exports carry the full helioslm.* hyperparameter metadata (read_gguf in helioslm_v5/src/export/gguf.py reads them back).

License

Apache-2.0 (same as the source repo).