PanoLM1.5-400M-Base

This repo contains the base PanoLM1.5 model (~400M parameters), a hybrid causal language model. 20 of its 24 layers use KDA linear attention; the other 4 (zero-based layers 5, 11, 17 and 23) use multi-head latent attention (MLA).

Training Data

Pretrained on a weighted mixture of three web-scale English corpora:

Weight	Dataset
0.45	FineWeb-Edu (100B-token subset)
0.30	DCLM (100B-token subset)
0.25	FinePDFs-Edu (100B-token subset)

Requirements

torch==2.12.0
transformers==5.8.1
flash-linear-attention==0.5.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "PanocularAI/PanoLM1.5-400M-Base"
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
).cuda()  # fla's RMSNorm uses Triton kernels that only run on CUDA tensors.
print(model)
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)

prompt = "PanoLM is an edge device friendly language model."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_length=512,
    top_k=10,
    do_sample=True,
    trust_remote_code=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

All scores are 0-shot, evaluated with lm-evaluation-harness. For multi-choice tasks we report length-normalized accuracy (acc_norm), except for commonsense_qa and winogrande, which report plain accuracy (acc).

Commonsense reasoning & general QA

Task	Metric	Value	Stderr
arc_challenge	acc_norm	0.2799	± 0.0131
arc_easy	acc_norm	0.5471	± 0.0102
commonsense_qa	acc	0.2015	± 0.0115
hellaswag	acc_norm	0.4286	± 0.0049
piqa	acc_norm	0.6665	± 0.0110
winogrande	acc	0.5201	± 0.0140

Associative recall (Based suite)

Extraction from a document in the prompt: the model must copy the answer out of the text it was just given. contains counts a generation as correct when the reference answer appears in it. fda, swde and squad_completion are lm-evaluation-harness' own tasks; based_triviaqa, based_drop and based_nq_2048 are the single-document versions of the remaining Based-suite tasks. Greedy decoding, no repetition penalty, prompts left-truncated to the model's 2,048-token training context. Stderr is the binomial standard error.

Task	Metric	Value	Stderr
fda	contains	0.6670	± 0.0142
swde	contains	0.6814	± 0.0140
squad_completion	contains	0.4042	± 0.0090
based_triviaqa	contains	0.5379	± 0.0121
based_drop	contains	0.2415	± 0.0094
based_nq_2048	contains	0.1387	± 0.0062