Harmonic GPT 128M Byte Chat

An experimental 127.6M-parameter causal byte transformer trained by the Harmonic GPT project. This repository contains the strongest chat-SFT checkpoint from the 128M transformer control run.

The model uses raw UTF-8 bytes rather than a subword tokenizer. Its decoder has 18 layers, 768 hidden dimensions, 12 attention heads, RoPE, RMSNorm, SwiGLU, tied byte embedding/readout, and PyTorch causal SDPA.

Important limitations

This is a research artifact, not a reliable general assistant. It learned prompt/response framing and usually emits the trained end-of-response sequence, but it frequently gives incorrect or nonsensical factual answers. Do not use it for medical, legal, financial, safety-critical, or production decisions.

The model is released under CC BY-NC 4.0 because its SFT mixture includes HuggingFaceH4/no_robots, which uses that license. Other SFT components include MonumentalSystems historical chat, reasoning, and synthetic salon datasets.

Training snapshot

Pretraining checkpoint: step 57,500
Pretraining exposure: 3.768B bytes
SFT: 500 steps, assistant-only loss
SFT mixture: 70% No Robots, 8% historical chat, 8% reasoning, 4% synthetic salon, 10% pretraining replay
Held-out assistant BPB: 1.191844 (starting value 1.304979)
Matched base-corpus BPB after SFT: 0.966025 (starting value 0.956386)
Learned response delimiter: \n<|end|>\n

Dedicated Inference Endpoint

The repository includes a custom handler.py. Deploy it with Hugging Face Inference Endpoints using the Custom framework. The handler accepts either a plain prompt or role/content messages.

Plain prompt request:

{
  "inputs": "What is DNA?",
  "parameters": {
    "max_new_bytes": 256,
    "temperature": 0.7,
    "top_k": 40,
    "seed": 42
  }
}

Multi-turn request:

{
  "inputs": [
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hello. How can I help?"},
    {"role": "user", "content": "Explain DNA briefly."}
  ],
  "parameters": {"max_new_bytes": 256}
}

Response shape:

{
  "generated_text": "...",
  "ended": true,
  "generated_bytes": 123
}

Generation currently recomputes the causal context for every byte and does not implement a KV cache. GPU serving is strongly recommended.

Local handler usage

from handler import EndpointHandler

model = EndpointHandler(".")
print(model({"inputs": "Ping", "parameters": {"max_new_bytes": 128}}))

Research context

This checkpoint is the conventional-transformer control used alongside spherical and planar geometric architectures in Harmonic GPT. Benchmarks are treated as diagnostics for architecture research rather than claims that this model is competitive with production language models.

Downloads last month: 27

Safetensors

Model size

0.1B params

Tensor type

F32