NeoLM-Small-Base (61M) — Stage 1: Foundation Continued 30K

NeoLM-Small-Base is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. The main branch publishes the Stage 1 Foundation Continued 30K base-model checkpoint in bfloat16 (safetensors) format. This is a base next-token prediction model, not an instruction-tuned chat assistant.

Available revisions

Revision	Description
main	Current stable Base weights: Foundation continued for 30K additional optimizer steps; BF16 release
stage1-foundation	Historical original Foundation checkpoint, preserved separately
stage1-foundation-continued-30k	Planned immutable tag for this 30K release; created after this README is committed and the release is verified

The original Foundation checkpoint was trained for approximately 202K optimizer steps before a system interruption. The selected continuation added 20K steps from that checkpoint, followed by 10K more steps from the 20K checkpoint (approximately 232K steps total along the selected training trajectory). Independent 5K and 20K continuation experiments were also performed; they should not be summed together as one run. The original planned three-epoch Foundation schedule was not completed.

Architecture

Property	Value
Parameters (unique, tied embeddings)	61,356,032
Layers	12
Hidden dimension	512
Query heads / KV heads	8 / 2 (GQA)
MLP	SwiGLU, intermediate size 1,536
Normalization	RMSNorm, QK Norm
Positional encoding	RoPE
Context length	1,024 tokens
Vocabulary	49,152 (custom ByteLevel BPE)
Special tokens	EOS ID 0, PAD ID 1; no BOS
Published weight format	BF16, safetensors
Model implementation	Custom NeoLMForCausalLM (trust_remote_code=True)

Tokenizer

NeoLM uses a custom ByteLevel BPE tokenizer trained from scratch, not a tokenizer borrowed from another model. It was trained exclusively on the 1,889,952 documents in NeoCorpus / foundation / train; validation and test documents were excluded from tokenizer training.

Property	Value
Algorithm	ByteLevel BPE
Vocabulary size	49,152
Training data	NeoCorpus / foundation / train
Training documents	1,889,952
Pre-tokenization	GPT-style regex with 1–3 digit groups
BOS token	None

Special-token spellings and IDs (kept outside the Markdown table to preserve literal pipe characters):

• EOS: <|endoftext|> — ID 0
• PAD: <|pad|> — ID 1

The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can be loaded without loading model weights:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "MaagDeveloper/NeoLM-Small-Base",
    revision="main",
    trust_remote_code=True,
)
print(tokenizer.encode("Hello, NeoLM!"))

Training data

Foundation pretraining uses the foundation configuration of NeoCorpus, derived from a filtered English-focused subset of Karpathy ClimbMix Shuffle, based on NVIDIA Nemotron-ClimbMix. The training split contains 1,889,952 documents / 1,147,218,931 tokens, counted with this release's tokenizer (including document EOS separators). The validation split contains 5,000 documents / 3,084,132 tokens; the test split contains 20,000 documents / 12,094,582 tokens. Validation and test were held out from pretraining. See the dataset card for preparation, attribution, and licensing details.

The tokenizer is trained on the Foundation training split, and pretraining consumes contiguous uint16 token streams with EOS between documents and 1,024-token windows at stride 1,023 (one-token overlap).

Evaluation

The selected 30K continuation checkpoint was evaluated on held-out Foundation token streams:

Checkpoint	Full validation loss	Full validation PPL	Test loss	Test PPL
Foundation continued 20K	3.2685	26.27	3.2498	25.78
Foundation continued 30K	3.2677	26.25	3.2489	25.76

The 30K continuation improves the reported full validation loss over the 20K checkpoint by approximately 0.0008; this is a small measured improvement, not evidence of a qualitative capability jump. Checkpoint selection should use validation rather than repeatedly consulted test results.

For the BF16 release, the model was exported separately from the FP32 master, reloaded, and used successfully for text generation. A local 256-block quick validation comparison, using the same BF16 autocast compute settings for both weight versions, reported:

Weight version	Quick validation loss	Quick validation PPL
FP32 source	3.236623	25.4476
BF16 export	3.236627	25.4477

The observed difference was +0.000004 loss. This is a quick-subset parity check, not a separate full-validation result or proof that every generated output will be identical. Published weights are BF16; the local FP32 master is retained separately.

Load and generate text

Install compatible versions of torch and transformers. The model requires custom Python code, so review the repository code before enabling trust_remote_code=True. Use revision="stage1-foundation" to load the original historical release instead of the latest stable Base.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True,
    dtype="auto",
).eval()

# For a CPU that does not support BF16 well, use dtype=torch.float32 instead.
prompt = "The future of artificial intelligence"

inputs = tokenizer(
    prompt, return_tensors="pt", truncation=True, max_length=1024
)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Streaming Text Generation

Stream generated text progressively using Hugging Face Transformers' TextStreamer:

from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
    dtype="auto",
).eval()

streamer = TextStreamer(
    tokenizer,
    skip_prompt=False,
    skip_special_tokens=True,
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

_ = model.generate(
    **inputs,
    streamer=streamer,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

Both examples are text completion, not role-based chat. For generation within the trained 1,024-token context, keep the total prompt-plus-generation length within the model's context window.

Capabilities and limitations

The Base checkpoint can generate fluent English passages and frequently follows broad topical cues in its prompt. These observations are not evidence of reliable instruction following, robust reasoning, or factual accuracy. Exploratory generation also exhibits repetition, invented facts, inconsistent entities, topic drift, and occasional malformed endings.

The model is small and English-focused; it can hallucinate, reproduce undesirable patterns, generate harmful or biased material, and fail on multi-step reasoning, factual recall, long-range coherence, or strict output formats. Do not treat generated material as verified information. A future instruction-tuned model, if released, will live in a separate repository.

Provenance and licensing

• Dataset: MaagDeveloper/NeoCorpus, foundation configuration.
• NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. Verify the appropriate model-weight license and source-data obligations before commercial use.
• stage1-foundation preserves the historical original release. The 30K release tag is intended to preserve a separate immutable weights-and-documentation snapshot after the README update.

NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.