LowOnMind-1M

A decoder-only language model with 985,152 parameters, pretrained from scratch on 200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT).

This is the 3.3x scale-up of DedeProGames/LowOnMind-300k, run as a controlled experiment: identical tokenizer, identical dataset, identical token budget, identical schedule, and the same hidden/layers aspect ratio. Parameter count is the only variable, so the two models are directly comparable on validation loss and on downstream benchmarks.

Architecture

	LowOnMind-300k	LowOnMind-1M
parameters	296,960	985,152
hidden_size	64	96
intermediate_size	136 (2.12x)	256 (2.67x)
num_hidden_layers	6	9
heads (q / kv)	4 / 2	6 / 2
head_dim	16	16
aspect ratio	10.7	10.7
vocab_size	1024	1024 (same tokenizer)
context	512	512
tokens seen	200M	200M
tokens/param	673	203

Architecture details, inherited from DynamicMind-Mini with modifications: GQA, SwiGLU, RMSNorm, tied embeddings, QK-Norm per head, precomputed RoPE with automatic re-expansion, and residual projections initialized at std / sqrt(2 * num_layers).

Training

data	HuggingFaceFW/fineweb-edu, sample-10BT
tokens	200M (6,103 steps x 32,768)
sequence length	512
batch size	64
optimizer	AdamW, betas (0.9, 0.95), wd 0.1
lr	1.5e-03 peak, cosine to 1.5e-04, 250 warmup
grad clip	1.0
precision	float16 + GradScaler
hardware	Tesla T4
wall clock	11 min

At 203 tokens per parameter this run still sits far past the Chinchilla-optimal ratio. Train and validation loss tracked each other for the entire run — this model is parameter-limited, not data-limited.

Results

metric	LowOnMind-300k	LowOnMind-1M	delta
validation loss	3.2982	2.9908	-0.3074
validation perplexity	27.06	19.90	-7.16
bits per character	2.030	1.836	-0.194

Perplexity is not comparable across tokenizers, but it is comparable between these two models because they share one. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable figure.

Real-word rate

With a 1024-token byte-level vocabulary, no long word exists as a single token — the model has to assemble every one of them from fragments. The fraction of emitted words that are real English words measures this directly, and neither validation loss nor a multiple-choice benchmark captures it.

	rate
LowOnMind-1M	98.0%
FineWeb-Edu itself (same lexicon)	98.4%

Measured over 64 unconditional samples (5,647 words). The reference lexicon is every lowercase word appearing at least 5 times in a 20k-document sample of the training corpus, so the corpus row is the practical ceiling rather than 100%. At 98.0% against a 98.4% ceiling, the lexicon is essentially saturated — the 300k → 1M scale-up bought almost all of its capacity in spelling and word formation, not in anything downstream of it.

Most frequent non-words: sculieness, lockholm, scul, purpled, paradigmar, prefection, frushing, fullly

BananaMind Base Bench 1.1

Evaluated on BananaMind/BananaMind-Base-Bench-1.1, the same 350-item English continuation-likelihood benchmark used for LowOnMind-300k, with identical scoring: context and each of the four continuations tokenized separately with add_special_tokens=False, no BOS, selection by highest mean conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no context required truncation against the 512-token window.

Category	LowOnMind-1M	LowOnMind-300k	delta	Elo (1M)
language_completion	52.0%	46.0%	+6.0pp	937
logical_reasoning	28.0%	24.0%	+4.0pp	925
context_tracking	24.0%	14.0%	+10.0pp	770
code_completion	20.0%	14.0%	+6.0pp	851
world_knowledge	22.0%	22.0%	+0.0pp	752
quantitative	28.0%	32.0%	-4.0pp	913
commonsense	28.0%	34.0%	-6.0pp	786

	LowOnMind-1M	LowOnMind-300k
Overall Elo	843	833
Chance-level Elo (this item grid)	805	805
Raw accuracy	28.9%	26.6%
95% CI	[24.1%, 33.6%]	[22.0%, 31.2%]
z vs. chance	+1.67	+0.69

Sample output

Prompts sampled at temperature 0.8, top_p 0.95. Unlike the 300k model, output now shifts register with the prompt — a date-led prompt produces bibliography-style formatting, a directive prompt produces policy-style prose — and invented non-words (see the real-word-rate section) are markedly rarer than at 300k parameters. Meaning and referential continuity are still absent.

prompt: "The "
-> The veil-group, where is used to solve solar passwords and vegetables. It
is often very important to reduce their treatments, including checkouts,
labs, maintenance, mental groups, chemicals and maintenance costs. These
literally confirmed elements are the most commonly used in the elementary
front, solar energy, a

prompt: "Photosynthesis is "
-> Photosynthesis is essential to prevent disease infection. These types of
disease include:
• Transformation of Brook Scientific World Organization
• Effectiveness of a diagnosis is commonly diagnosed with diagnosis. The
Cultural Disorder is an important factor in diagnosis. Analysis of this
purpose can be considered.

prompt: "In 1969, "
-> In 1969, 8, 1999, Michael Kerzin, Anderson, 1997, Scotland, 1993, 1,359,
p. 547-414
Our war was chosen in the West of Europe, Papua Party, Ottoman Economy,
Pluto Mountains, and Evans, 1854, 1946, Maine, 1888-81, p

prompt: "Students should "
-> Students should identify their learners and adults.
The United States has free quotations in the world that can help them with
this curriculum and finding problem concerning these funding.
One major solutions for malnutrition in South East and Africa is a way to
limit half its own loop. This is the best way to ensure that the drills of
the state and

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-1M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/LowOnMind-1M", trust_remote_code=True)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))

trust_remote_code=True is required — the architecture ships as custom modeling code in the repository. use_cache=False is required: this implementation has no KV cache and recomputes the full window at each generation step.

Limitations

At ~1M parameters this model learns spelling, morphology, and short-range syntax. It does not produce coherent text, has no reliable factual knowledge, and cannot track state across a passage. It exists to measure the lower end of the scaling curve, not to be used.

The 512-token context, 1024-token vocabulary, and absent KV cache make it unsuitable for any real workload.