vectabass-151m-950mt-base-cooled

Paper cell N1-D1 of the AR square (100.0M non-embedding parameters).

A 151M-parameter base language model (100.0M non-embedding), trained on 0.95B tokens. It is one cell of the Phase 1 AR square: one model ladder from 100M to 1.6B non-embedding parameters, each size trained to several token budgets, released together so the scaling law can be checked.

Seed replicates of this cell, same recipe with a new initialisation and data order, are on branches seed2. Load one with revision="seed2". The paper uses them to measure run-to-run noise.

This is a base model. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-151m-950mt-base-cooled")
model = AutoModelForCausalLM.from_pretrained("VectaBass/vectabass-151m-950mt-base-cooled", trust_remote_code=True, dtype=torch.bfloat16)
ids = tok("The Senedd is", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is Llama-like plus exclusive self-attention (XSA), which Llama does not have. Run it in bf16, as trained.

Model

Non-embedding parameters	99,975,168
Total parameters	151,486,464 (embeddings tied)
Width, layers	1024, 9
Heads	16 query, 8 key-value, dim 64
Feed-forward	SwiGLU, 2592
Norm	RMSNorm, pre-norm
Positions	RoPE, base 10000, context 2048
Attention	causal, GQA 2:1, XSA
Vocabulary	50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Training

Tokens	950,009,856 (10 per non-embedding parameter)
Batch	262,144 tokens
Optimizer	Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Peak LR	0.00243
Warmup	100M tokens
Schedule	warmup, constant, then a 1-sqrt cooldown to 0 over the last 20%
Weight decay	0.105 on matrices
Precision	bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware	1 x NVIDIA GH200 120GB

The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.

Evaluation

Validation loss on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens): 2.6672 nats per token.

This repo's weights reproduce the training code's logits bitwise and score 2.6673 here under PyTorch SDPA (training used FlashAttention-4).

Zero-shot, 500 items per task (seeded shuffle, seed 0), protocol hub-v1. Multiple choice scores log P(continuation | context), normalised accuracy where defined.

Task	Score
ARC-Easy	0.458
ARC-Challenge	0.210
HellaSwag	0.262
PIQA	0.592
WinoGrande	0.470
BoolQ	0.590
MMLU	0.240
LAMBADA	0.248
DQRB (ours)	0.481
LAMBADA bits per byte	1.418

Limitations

• A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
• Context is 2048 tokens.
• Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
• No safety tuning and no filtering of outputs.

Provenance

Run contract ar-square--N1-D1 (revision aaee06abd6be). Source checkpoint sha256 09262ad64d15d9620f224879d3f4fa50487a82965ca1a94cf9dac718d087d62c. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.