vectabass-374m-950mt-base-cooled

Paper cell N3-D1 of the AR square (303.1M non-embedding parameters).

A 374M-parameter base language model (303.1M non-embedding), trained on 0.95B tokens. It is one cell of the Phase 1 AR square: one model ladder from 100M to 1.6B non-embedding parameters, each size trained to several token budgets, released together so the scaling law can be checked.

This is a base model. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-374m-950mt-base-cooled")
model = AutoModelForCausalLM.from_pretrained("VectaBass/vectabass-374m-950mt-base-cooled", trust_remote_code=True, dtype=torch.bfloat16)
ids = tok("The Senedd is", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is Llama-like plus exclusive self-attention (XSA), which Llama does not have. Run it in bf16, as trained.

Model

Non-embedding parameters	303,114,240
Total parameters	373,942,272 (embeddings tied)
Width, layers	1408, 13
Heads	22 query, 11 key-value, dim 64
Feed-forward	SwiGLU, 4112
Norm	RMSNorm, pre-norm
Positions	RoPE, base 10000, context 2048
Attention	causal, GQA 2:1, XSA
Vocabulary	50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Training

Tokens	950,272,000 (3 per non-embedding parameter)
Batch	327,680 tokens
Optimizer	Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Peak LR	0.00174
Warmup	190M tokens
Schedule	warmup, constant, then a 1-sqrt cooldown to 0 over the last 20%
Weight decay	0.0691 on matrices
Precision	bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware	1 x NVIDIA GH200 120GB

The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.

Evaluation

Validation loss on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens): 2.5690 nats per token.

This repo's weights reproduce the training code's logits bitwise and score 2.5690 here under PyTorch SDPA (training used FlashAttention-4).

Zero-shot, 500 items per task (seeded shuffle, seed 0), protocol hub-v1. Multiple choice scores log P(continuation | context), normalised accuracy where defined.

Task	Score
ARC-Easy	0.446
ARC-Challenge	0.222
HellaSwag	0.274
PIQA	0.598
WinoGrande	0.518
BoolQ	0.604
MMLU	0.246
LAMBADA	0.274
DQRB (ours)	0.531
LAMBADA bits per byte	1.384

Limitations

• A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
• Context is 2048 tokens.
• Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
• No safety tuning and no filtering of outputs.

Provenance

Run contract ar-square--N3-D1 (revision de6b4fbf21ef). Source checkpoint sha256 a668281ab713387b4697d01e5d1a560e2edab560742612746105c827d8aaea7e. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.