KronLLM
A compact bilingual (English/Russian) language model whose matrix layers are
built entirely from Kronecker sums of rank-1 factors. Every attention and
FFN projection is a composition of L factors instead of a dense d × d
weight: a layer that would need 102,400 numbers needs 15,753.
This repository hosts the model card, the code and the shipped int8 checkpoints. The training corpus is separate: see Icold21/kron-corpus and the research log at github.com/Icold21/kron-llm.
Main model: Q10 (10M)
Q10 is the model this project is about: 9,938,886 parameters (d=512, 25
layers, 8K bilingual BPE, Kronecker K=10), trained on 2.78B tokens of bilingual
English/Russian text. It is mid-pretraining — step 36,549 of 169,884 (21.5 %)
of the first epoch — and reaches 1.1828 bits/byte mixed (EN 1.4327 /
RU 0.9411) on a held-out bin that never changes; GPT-2-124M scores 1.554 on the
same blocks. The checkpoint lives on Kaggle
(icold21/kron-q10-ckpt)
and will be mirrored here when the epoch finishes.
Experimental checkpoints (the development bench)
The two small checkpoints below are not the product. They are the TinyStories bench that made the 10M configuration cheap to choose: each isolates one decision (tokenizer, gated FFN, depth carry), and their bits/byte is not comparable with Q10's bilingual number.
| file | parameters | size | quality |
|---|---|---|---|
q8g_int8.pt |
2,851,280 | 3.14 MB | 0.7542 bits/byte (TinyStories val, 61.4M tokens seen) |
q2_int8.pt |
2,079,992 | 2.34 MB | previous bench model, same corpus |
Int8 quantisation costs +0.0003 bpb on Q8g. The fp32 checkpoint is 11.4 MB and is not shipped.
Usage (bench checkpoint)
import torch
from kronllm.model import KronTransformer
from kronllm.quantization import load_int8
ckpt = torch.load("q8g_int8.pt", map_location="cpu")
model = KronTransformer(ckpt["cfg"]).eval()
load_int8(model, ckpt["model"])
The bench model is a research artifact: it writes short English text and is not an instruct-tuned assistant.
Architecture of the bench model
| layers / width | 24 / 320, 8 heads, seq 192 |
| composition | product over L=4 factors, K=6 terms, depth carry K_win=2 |
| FFN | gated (SwiGLU-style) |
| tokenizer | BPE-4096, tied factorised embedding (rank 128) |
The main model keeps the same block but scales to d=512, 25 layers, K=10 and the 8K bilingual BPE, with pre-norm QK and a gated FFN.
Measured, not assumed: the depth carry is worth 1.1 % bpb, freezing its scalar 0.8 %, and widening the window to four layers costs 1.4 %. Details in the research log.
Training budget
At the measured 2.552 s/step on a single Kaggle T4 (batch 32, seq 512), one epoch of the 2.78B-token corpus is 120 GPU-hours; the decay pass (30 % of the epoch, quality-selected slice plus 5 % chat) adds 35, and SFT 4,000 steps adds 2.8. Roughly 132 GPU-hours remain from step 36,549 -- about 12 sessions of quota. Derivation in docs/compute_budget.md of the code repository.
Next
Q10 continues on KronCorpus — 2.78B tokens of bilingual English/Russian text with a science accent — on one Kaggle T4. After the epoch: a decay pass (29 % of the epoch, quality-selected, plus 5 % chat) and SFT. The weights will be mirrored here when the epoch finishes; the corpus card documents every source and licence.
Licence
Code and weights: MIT. The corpus keeps its sources' licences (see the dataset card).