ObsidianSmall-Base — Multiscreen 8.7M

ObsidianSmall-Base is an 8.7M-parameter English base language model pretrained from scratch on 3,999,989,760 tokens.

It is the first public checkpoint in this model family built entirely from Multiscreen blocks.

This is a base completion model, not an instruction-tuned or chat-aligned assistant.

Model details

Property	Value
Architecture	Pure Multiscreen
Parameters	8,716,506
Blocks	9
Hidden width	256
Query heads	8
Head size	32
Key / value dimensions	16 / 128
Query chunk / window	128 / 256
MLP / FFN	None
Context length	1,024
Vocabulary size	4,096
Training tokens	3,999,989,760
Validation loss / PPL	2.592 / 13.359

Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.

Benchmarks

Zero-shot task results were measured with lm-evaluation-harness.

Benchmark	Accuracy	Normalized
BLiMP	71.891%	—
HellaSwag	26.957%	27.415%
ARC Easy	33.712%	33.291%
ARC Challenge	17.833%	21.843%
PIQA	56.583%	55.114%
SciQ	70.700%	60.000%
ArithMark-2.0	26.160%	—
ArithMark-3.0	28.800%	—

WikiText-2 perplexity

Full WikiText-2 evaluation using rolling log-likelihood with a maximum context length of 1,024 tokens.

Metric	Result
Word perplexity	125.057
Byte perplexity	2.467
Bits per byte	1.303

Install

git lfs install
git clone https://huggingface.co/Dream-W/ObsidianSmall-Base
cd ObsidianSmall-Base

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

For NVIDIA CUDA with Triton:

python -m pip install -r requirements-gpu.txt

For evaluation:

python -m pip install -r requirements-eval.txt

The repository includes the required custom PyTorch/LitGPT runtime. It is not currently compatible with transformers.AutoModelForCausalLM.

Generate text

CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
  --backend triton \
  --device cuda \
  --prompt "Once upon a time" \
  --max-new-tokens 128 \
  --temperature 0.8 \
  --top-k 40

Interactive completion:

CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
  --backend triton \
  --device cuda \
  --interactive

CPU execution is available with MULTISCREEN_BACKEND=torch, --backend torch, and --device cpu.

Evaluate

The bundled evaluator accepts any compatible lm-evaluation-harness task through --tasks.

MODEL="$PWD"
mkdir -p "$MODEL/results"

CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
TOKENIZERS_PARALLELISM=false \
PYTHONPATH="$MODEL/runtime:$MODEL/evaluation" \
python -u "$MODEL/evaluation/lm_eval_obsidian_official.py" \
  --checkpoint-dir "$MODEL" \
  --tokenizer-dir "$MODEL" \
  --tasks hellaswag \
  --batch-size 32 \
  --device cuda \
  --max-length 1024 \
  --progress-every 100 \
  --output "$MODEL/results/hellaswag.json"

Multiple tasks:

--tasks "hellaswag,arc_easy,arc_challenge,piqa,sciq,blimp"

Custom continuation-style multiple-choice datasets can be evaluated with evaluation/custom_mcq.py.

CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u evaluation/custom_mcq.py \
  --model-dir . \
  --hf-dataset AxiomicLabs/Arithmark-3.0 \
  --split train \
  --context-field ctx \
  --choices-field endings \
  --label-field label \
  --group-by difficulty topic \
  --backend triton \
  --device cuda \
  --output results/arithmark-3.json

Backends and decoding

Backend	Setting	Use
Automatic	MULTISCREEN_BACKEND=auto	Select a compatible backend
Triton	MULTISCREEN_BACKEND=triton	Optimized NVIDIA CUDA execution
PyTorch	MULTISCREEN_BACKEND=torch	CPU and compatibility fallback

Incremental Multiscreen KV-cache decoding is not implemented. Generation uses full-sequence decoding and processes the current context again for every generated token.

Training data

The pretraining curriculum included FineWeb-Edu, FinePDFs-Edu, DCLM, FineWiki, NVIDIA Common Crawl Code, and NVIDIA Common Crawl Math.

Limitations

The model has only 8.7M parameters and may produce repetitive, inaccurate, or incoherent text. It is English-focused, has no instruction-following or safety alignment, and was trained for a maximum context length of 1,024 tokens.