Chochmah-350M - TevunahAi From-Scratch Pretrained Base Model

This is our first coded from scratch model. This is a beginning. As we go forward quality will improve. We will always put the highest quality material in our models. A large majority of the material will all be hand selected. This makes it different then a simple put together model on the fly. A 1B model will be next. It will represent the same formula. High quality material pulled together to bring about a model that has potential far past it size.

Chochmah (חָכְמָה, wisdom) is a 359M-parameter decoder-only language model pretrained from scratch by TevunahAi on 30 billion tokens, on a single NVIDIA RTX 5000 Ada (32 GB), in about ten and a half days of wall-clock time. It is the companion term to tevunah (understanding, discernment), from which TevunahAi takes its name; the two appear side by side in Proverbs 3:19.

It is a base model: no instruction tuning, no chat template, no RLHF. The pretraining mix was deliberately weighted toward six registers — case law, philosophy and political philosophy, mathematics, physics, and code — using three hand-built corpora (2.6 million US judicial opinions, a 126-author public-domain philosophy canon, OpenStax textbooks) on top of a broad web / code / encyclopedia foundation, trained through a three-phase curriculum aligned to a warmup–stable–decay learning-rate schedule.

It is released as (a) a clean, fully documented small base model for fine-tuning and quantization research, and (b) a complete worked example of a from-scratch pretraining pipeline — data curation, tokenization, curriculum, schedule, crash-safe training — that fits on one workstation GPU.

Model Details

Property	Value
Developer	TevunahAi
Model Type	Decoder-only transformer, Llama architecture, pretrained from scratch
Parameters	358.7M total (308.3M non-embedding)
Training Tokens	29.99B (57,200 steps x 524,288 tokens)
Context Length	2,048 tokens
Tokenizer	SmolLM2 (49,152 BPE)
Training Precision	bf16 autocast, fp32 master weights
Release Precision	fp32 safetensors (1.43 GB)
Training Hardware	1x NVIDIA RTX 5000 Ada 32 GB
Training Time	~10.5 days wall-clock (August 12–22, 2026)
Training Compute	~6.5 x 10^19 FLOPs
Stage	Base — pretrained only
Loads With	AutoModelForCausalLM / LlamaForCausalLM, no custom code
License	Apache-2.0

Architecture Breakdown

A standard modern decoder, with no exotic choices — every design decision here is the 2024-era default so the model loads everywhere Llama loads.

Layer Composition (24 layers)

• 24 Transformer Blocks, pre-norm residual
  • Multi-Head Attention: 16 heads, head dimension 64, no GQA (16 KV heads), no biases
  • SwiGLU MLP: intermediate size 2,816, down(silu(gate(x)) * up(x)), no biases
  • RMSNorm before attention and before the MLP, eps 1e-5
• Rotary position embeddings (RoPE), theta 10,000, applied to Q and K
• Tied input/output embeddings: the 49,152 x 1,024 embedding matrix is also the LM head
• No dropout, no bias terms anywhere, init std 0.02

Parameter Accounting

Component	Parameters
Attention (24 x 4 x 1024^2)	100.7M
SwiGLU MLP (24 x 3 x 1024 x 2816)	207.6M
Norms	~0.05M
Non-embedding total	308.3M
Tied embedding (49,152 x 1024)	50.3M
Total	358.7M

Why This Matters

• Non-embedding count drives compute. 308M is why this run took ten days on one GPU rather than a month.
• Two-thirds of the parameters are in the MLPs. That is where the model's factual and stylistic knowledge lives, which matters for quantization planning.
• Llama-compatible weights. Conversion from the training implementation is a pure key rename (no weight permutation); logit equivalence between the original PyTorch model and LlamaForCausalLM was verified to ~1e-6 in fp32. Everything downstream — vLLM, GGUF, GPTQModel — works out of the box.

Training

Optimizer and Schedule

Field	Value
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8, fused
Weight Decay	0.1 (none on norms or embeddings)
Peak Learning Rate	4e-4
Schedule	WSD (warmup–stable–decay): 2,000-step linear warmup → constant → decay to ~0 across the final 5,700 steps
Gradient Clipping	1.0
Global Batch	524,288 tokens (8 micro-batch x 32 grad-accum x 2,048)
Total Steps	57,200
Precision	bf16 autocast, fp32 master weights and optimizer state
Kernels	torch.compile, SDPA with the FlashAttention backend, fused AdamW
Throughput	~35,300 tok/s median, peak VRAM 26.4 GB, ~30% MFU
Checkpointing	Atomic writes (temp + rename), full optimizer / RNG / dataloader state; resume verified bit-exact

The learning-rate decay is aligned to the data curriculum on purpose: the LR stays flat while data quality ramps through phases 1 and 2, then decays to zero exactly across the anneal phase. That alignment is what lets the highest-quality 10% of the data set the model's final voice instead of being averaged away by later updates.

Loss trajectory. 10.98 at initialization → 2.24 entering the anneal (step 51,500) → ~1.75 at the end of training (measured on the anneal mix; the phases use different data, so losses are not directly comparable across boundaries). Gradient norm held at ~0.15 through the entire decay with no instability. The run survived one unplanned reboot by resuming from checkpoint with no discontinuity in the loss curve, and both phase transitions occurred automatically without a restart.

Three-Phase Curriculum

Phase	Steps	Tokens	Purpose
1 — Foundation	0 – 34,300	18B	Broad language competence. Deliberately unclever.
2 — Domain build	34,300 – 51,500	9B	The six target domains enter at full weight.
3 — Anneal	51,500 – 57,200	3B	Highest-quality material only, LR decaying to zero. Sets the model's voice.

Each phase was pre-mixed into shards at fixed ratios (no runtime sampling), with a consumption manifest tracking per-source byte offsets and head hashes so that the three phases read disjoint ranges of every source. Phase 1 holds every source under one epoch (0.91). Only the small hand-built corpora repeat in later phases: philosophy 2.0 epochs, textbooks 3.4 epochs; FineWeb-Edu wraps slightly in phase 3. Separate per-domain validation sets were carved from the unused tail of each source.

Phase 1 — Foundation (18B tokens)

Source	Share
FineWeb-Edu	62%
StarCoderData	14%
Wikipedia (en)	8%
Project Gutenberg	7%
OpenWebMath	5%
Cosmopedia v2	4%

Phase 2 — Domain build (9B tokens)

Source	Share
FineWeb-Edu	28%
Math (FineMath-4+, OpenWebMath, AlgebraicStack)	20%
Law (hand-built, broad)	18.8%
StarCoderData	15%
peS2o (science)	8%
Cosmopedia v2	6%
Philosophy (hand-built)	4%
Textbooks (OpenStax)	0.2%

Phase 3 — Anneal (3B tokens, LR → 0)

Source	Share
Cosmopedia v2	37%
FineMath-4+	18%
Philosophy (hand-built)	12%
StarCoderData	12%
Law canon (hand-built, most-cited opinions)	10%
FineWeb-Edu	10%
Textbooks (OpenStax)	1%

Data Sources

Source	Origin	License	Role
FineWeb-Edu	HuggingFaceFW/fineweb-edu (sample-100BT)	ODC-By	Foundation, all phases
StarCoderData	bigcode/starcoderdata	Permissive-licensed code	Code, all phases
Wikipedia	wikimedia/wikipedia (20231101.en)	CC BY-SA 4.0	Phase 1
Project Gutenberg	sedthh/gutenberg_english	Public domain	Phase 1
OpenWebMath	open-web-math/open-web-math	ODC-By	Math, phases 1–2
Cosmopedia v2	HuggingFaceTB/smollm-corpus	Apache-2.0	Synthetic textbook prose, all phases; anneal anchor
FineMath-4+	HuggingFaceTB/finemath	ODC-By	Math, phases 2–3
AlgebraicStack	EleutherAI/proof-pile-2	Mixed permissive	Math/code, phase 2
peS2o	allenai/peS2o	ODC-By	Science (abstracts + introductions), phase 2
Law	CourtListener bulk export, Free Law Project	Public domain	Hand-built; phases 2 (broad) and 3 (canon)
Philosophy	Project Gutenberg, curated	Public domain	Hand-built; phases 2–3
Textbooks	OpenStax CNXML sources (GitHub)	CC BY 4.0	Hand-built; phases 2–3

The Hand-Built Corpora

Law — 2,556,649 opinions (9.92B tokens) broad, 101,688 opinions (0.80B tokens) canon. Built from the 2026-06-30 CourtListener bulk export. Courts were selected by jurisdiction level, not by name (New York's "Supreme Court" is a trial court): the Supreme Court of the United States, the 13 modern federal circuits, and state courts of last resort; state intermediate appellate courts and the pre-1911 circuit trial courts were excluded. Opinions that were too short (1.6M) or carried West headnote markers were rejected. The canon split — every opinion cited 160 or more times, plus all SCOTUS opinions — is what the anneal phase trained on; it is 13% SCOTUS by token count and 73% from the 1980s–2010s. The broad corpus spans 66 courts with 42% of tokens pre-1980. Supply exceeded demand by roughly an order of magnitude: training consumed 0.17 epochs of the broad corpus and 0.37 epochs of the canon, so the law the model saw is a random cross-section, not a skim of the beginning.

Philosophy — 3,072 works (~0.353B tokens), 126 curated authors across seven eras. Ancient through early twentieth century, deliberately broadened to political theory and intellectual history, including Eastern texts, Douglass, Du Bois, Spooner, and the public-domain women's canon (Wollstonecraft, Fuller, Stanton, Harriet Taylor Mill, Anna Julia Cooper, Gilman, Goldman, Addams, de Gouges, Martineau). Nietzsche is present in full (22 works, the complete Levy edition), alongside Voltaire (23 works), Marx, Aristotle, Kant, Machiavelli, Confucius; the largest contributors by volume are William James, Hume, Henry George, T. H. Green, and Augustine. Matched by author, then Library of Congress classification, then subject; Gutenberg license boilerplate stripped; duplicate editions collapsed to the longest. Consumed at 2.0 epochs across phases 2 and 3.

Textbooks — 19 OpenStax STEM and government titles (~14M tokens). Fetched as CNXML from the openstax/osbooks-* source repositories, stripped of MathML, media, and metadata to clean prose. Openly licensed real textbooks are scarce (~100–300M tokens exist in total), so this corpus supplies authenticity while Cosmopedia supplies textbook-style volume. Consumed at 3.4 epochs.

Tokenizer

HuggingFaceTB/SmolLM2-360M — 49,152 BPE with digit splitting and code-friendly merges, chosen for a LaTeX- and code-heavy mix. Documents were concatenated with <|endoftext|> as the separator. No BOS token was prepended during training — do not add one at inference.

Performance Benchmarks

Standard Benchmarks

0-shot, lm-evaluation-harness, fp32, batch size 32. All three models were evaluated with the identical command on the same machine so the columns are directly comparable. Pythia-410M is the closest compute-matched reference (~302M non-embedding parameters vs. 308M here) and saw 10x the tokens; SmolLM2-360M shares this model's tokenizer and saw 133x the tokens.

Task	Chochmah-350M (30B tokens)	Pythia-410M (300B tokens)	SmolLM2-360M (4T tokens)
ARC-Easy (acc / acc_norm)	59.0 / 54.2	51.9 / 45.7	70.4 / 68.0
HellaSwag (acc / acc_norm)	34.8 / 41.8	33.7 / 40.6	43.2 / 56.3
PIQA (acc / acc_norm)	68.6 / 68.3	66.7 / 67.2	72.1 / 72.0
LAMBADA-OpenAI (acc / ppl ↓)	40.2 / 23.8	51.6 / 10.8	53.8 / 9.4

Standard errors: ARC-Easy ±1.0, HellaSwag ±0.5, PIQA ±1.1, LAMBADA ±0.7.

How to read it. With a tenth of Pythia-410M's tokens, Chochmah is clearly ahead on ARC-Easy (+7 points, ~5 standard errors) — the signature of the education-weighted mix (FineWeb-Edu, Cosmopedia, OpenStax), the same effect the SmolLM family demonstrated. HellaSwag and PIQA are on par with Pythia-410M within error. LAMBADA, which tests final-word prediction on modern novel excerpts, trails Pythia by ~11 points: the mix contains ~1.3B tokens of public-domain books in phase 1 and essentially no fiction afterward, a deliberate trade for the target domains. The SmolLM2 column shows what 133x the tokens buys on the same kind of data — most of it on HellaSwag, which scales with token count almost on rails.

Per-Domain Held-Out Loss

Validation loss (nats/token) at the final step on held-out sets carved from the unused tail of each source:

Domain	Loss
FineWeb-Edu	—
StarCoderData	—
Wikipedia	—
Gutenberg	—
OpenWebMath	—
Cosmopedia	—
FineMath	—
Law	—
Philosophy	—
peS2o	—

Sample Outputs

Verbatim, unedited, from the first prompts run on the final checkpoint (temperature 0.7, top-k 50, top-p 0.95, repetition penalty 1.1, 200 new tokens). The first two are shown together deliberately: they are the same model, one prompt apart, and together they are an accurate description of what a 350M base model is.

Prompt: The Supreme Court of the United States has long held that

The Supreme Court of the United States has long held that the Due Process Clause of the Fourteenth Amendment requires that a State provide an opportunity for an impartial adjudication before deprivation occurs. See, e. g., Mullane v. Central Hanover Bank & Trust Co., 339 U. S. 306 (1950); Goldberg v. Kelly, 397 U. S. 254 (1970). This Court has never, however, required the government to provide a hearing where the rights guaranteed by the Constitution are not implicated. In the absence of an express constitutional command, the Government is free to place all or some of those rights in any "general rule" which, when applied to particular cases, would be constitutionally permissible. This general rule is one which we have approved. The Court today adopts a new standard under which the Due Process Clause must be satisfied at the time of the deprivation. The Court finds that a hearing must be provided if the State fails to afford an opportunity

Both citations are real, correctly reported, and actually support the proposition — Mullane and Goldberg are the two cases for pre-deprivation process. Note the Court's own typographic conventions ("U. S.", "See, e. g.,"), learned from the SCOTUS opinions in the canon. Also note the drift: the final paragraph opens in the voice of a dissent.

Prompt: In Marbury v. Madison, Chief Justice Marshall reasoned that

In Marbury v. Madison, Chief Justice Marshall reasoned that the right of a citizen to be tried by his peers, rather than by a jury of his peers, would not serve as a check on the judiciary. The Chief Justice further stated that in order to prevent judicial oppression, the Constitution could not require a trial by jury unless it was provided by statute, but he did not believe that this provision was necessary because the judiciary was so numerous and well endowed that the process of trying an individual is within its jurisdiction. While this court had earlier recognized that Article III, Section 2 of the Constitution gave a right to a jury trial, Chief Justice Marshall concluded that the process of determining guilt or innocence was not constitutionally required. He stated that "the right to a trial by jury is of course limited by the law of the land, and that it does not extend to a trial by a jury of any particular number."

Every substantive claim in this sample is false, including the quotation. Marbury is about judicial review and original jurisdiction; the model followed the wrong clause of Article III § 2 and produced fluent, confident, well-formed legal prose about juries. A repeated citation pattern (Mullane + Goldberg) is memorized; a holding is a concept, and at this scale the model stores forms, not concepts. This is the failure mode to expect from every small base model, and it is why this model must not be used as a source of facts.

More samples — physics and philosophy

Prompt: Newton's second law states that

Newton's second law states that the net external force acting on an object is equal to its mass times its acceleration:
$$\text{net external force}=m\times a_f\backslash text\{net\; external\; force\}\; =\; m\; \backslash times\; a\_f$$
where $m$ is the mass of the object, $a_f$ is the acceleration due to gravity ($9.81 \text{ m/s^2}$), and $a_f$ represents the acceleration of the object's center of mass relative to its center. By rearranging the equation above for the net external force, we can solve for the unknown variable:
$$\text{net external force}=m\times a_f=m\times g\times h\backslash text\{net\; external\; force\}\; =\; m\; \backslash times\; a\_f\; =\; m\; \backslash times\; g\; \backslash times\; h$$
Now that we have solved for $g$, we can substitute it back into our original equation to find the object's acceleration:
$$a_f=\frac{GM}{r^2}a\_f\; =\; \backslash frac\{GM\}\{r^2\}$$
where $G$ is the gravitational constant ($6.67 \times 10^{

"Net external force" is OpenStax's exact phrasing. Every formula and constant is real and the LaTeX is well-formed; the derivation connecting them is not physics.

Prompt: Nietzsche's critique of morality begins with the observation that

[...] So why should middle school students care about Nietzschean thought? One reason is that many of his ideas continue to influence modern philosophy and culture. Nietzsche's emphasis on individual freedom and self-expression has been

The audience address is the fingerprint of Cosmopedia's synthetic textbook prose, which at 37% of the anneal is the model's default expository voice. Prompting in the register of a primary text rather than a summary about one shifts the output toward the philosophy canon's voice.

Usage

Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "TevunahAi/Chochmah-350M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).cuda()

prompt = "The Supreme Court of the United States has long held that"
ids = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **ids,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    repetition_penalty=1.15,
    no_repeat_ngram_size=4,
)
print(tok.decode(out[0], skip_special_tokens=True))

Prompting a Base Model

• Continue, don't instruct. Give it the opening of the document you want; it will not follow a question or a command.
• The prompt's register selects the corpus. A SCOTUS-style opening produces opinion prose with citations; a textbook sentence produces textbook prose; a def produces code. Write the first line the way the target document would be written.
• No BOS token, no chat template. The tokenizer is saved without one; do not add either.
• Sample tightly. Temperature 0.5–0.7, top-p 0.9, repetition penalty 1.1–1.15, no_repeat_ngram_size=4 (or min_p=0.05), and 100–150 new tokens. Base models do not know when to stop, and most drift lives in the second half of a long generation.

vLLM

pip install vllm
vllm serve TevunahAi/Chochmah-350M --max-model-len 2048 --dtype bfloat16

Fine-Tuning

The model is a stock LlamaForCausalLM and works unmodified with TRL, Axolotl, LLaMA-Factory, Unsloth, and PEFT. Its main intended use is as an SFT / DPO base and as a small, clean, fully documented target for quantization research.

Known Issues and Limitations

• It will state false things fluently. At this scale the model learns how legal, philosophical, mathematical, and scientific text is written far better than it learns what is true. It must not be used as a source of legal, medical, financial, or factual answers. It is not a substitute for legal advice, and nothing it writes about a case should be assumed to be in that case.
• Base model. No instruction following, no safety tuning, no chat template, no stop behavior beyond <|endoftext|>.
• Code is the weakest register. Indentation can break after a docstring and logic is unreliable; expect HumanEval near zero. This is expected at 350M / 30B tokens.
• Formatting artifacts from the data. Two are visible in generations: Gutenberg-derived text is reproduced with its original hard line wraps, and case names sometimes appear on their own lines as in CourtListener's HTML-to-text conversion. A re-annealed revision on cleaned phase-3 data is planned and will be pushed to this repository as a new commit with a changelog.
• Narrative fiction is underweighted, which is what the LAMBADA result reflects.
• English only, 2,048-token context. Some Wikipedia-derived knowledge is dated 2023-11.
• Repetition at long generation lengths, as with all small models; see the sampling guidance above.

Memory Requirements

Inference

• fp32 (as released): 1.43 GB weights; ~2 GB VRAM at full 2,048 context
• bf16: ~0.72 GB weights; ~1.2 GB VRAM at full context
• Runs comfortably on CPU; a few hundred tokens per second on a modern GPU without a KV cache, far more with one

Training (Reproduction)

• Used: RTX 5000 Ada (32 GB) + Dual Xeon Max 9480 (128 GB HBM2e + 256 GB DDR5) for the data pipeline
• Peak VRAM: 26.4 GB at micro-batch 8 x 2,048 with torch.compile; micro-batch 16 does not fit in 32 GB
• Tokenized data: ~60 GB (uint16 shards) — fits in page cache on 256 GB of RAM, so the GPU never waits on data
• Raw downloads: ~500 GB, dominated by the 53 GB CourtListener bulk export and FineWeb-Edu

Technical Specifications

Specification	Value
Model Family	TevunahAi Chochmah
Variant	350M (v1.0, base)
Total Parameters	358,663,168
Non-Embedding Parameters	308,331,520
Layers	24
Hidden Size	1,024
Attention Heads	16
KV Heads	16 (MHA)
Head Dimension	64
Intermediate Size	2,816
Activation	SiLU (SwiGLU)
Normalization	RMSNorm, eps 1e-5, pre-norm
Positional Encoding	RoPE, theta 10,000
Biases	None
Embeddings	Tied
Context Length	2,048
Vocab Size	49,152
Tokenizer	SmolLM2
Training Tokens	29.99B
Training Steps	57,200
Global Batch	524,288 tokens
Schedule	WSD (2,000 warmup / stable / 5,700-step decay)
Final Checkpoint	ckpt_57199.pt
Architecture Class	LlamaForCausalLM

Roadmap

• v1.1 — re-anneal of the stable-phase checkpoint on cleaned phase-3 data (hard wraps and citation line breaks removed, Cosmopedia share reduced), pushed to this repo as a new commit with a changelog
• Chochmah-350M-Instruct — supervised fine-tuning on an open instruction set
• TevunahAi Ultra-Hybrid GPTQ variant of the instruct model
• Chochmah-1B — same pipeline and corpora, ~1B parameters, 40–50B tokens

License

Weights: Apache-2.0.

Training data is a mix of ODC-By (FineWeb-Edu, FineMath, OpenWebMath, peS2o), Apache-2.0 (Cosmopedia v2), CC BY 4.0 (OpenStax), CC BY-SA 4.0 (Wikipedia), permissively licensed code (StarCoderData), and public-domain material (Project Gutenberg; United States judicial opinions, which are edicts of government). No data under non-commercial terms was used. Users are responsible for their own compliance with the licenses of the underlying datasets.

Citation

@misc{chochmah350m_2026,
  title  = {Chochmah-350M: a from-scratch pretrained base model on one GPU},
  author = {TevunahAi},
  year   = {2026},
  url    = {https://huggingface.co/TevunahAi/Chochmah-350M},
  note   = {359M parameters, 30B tokens, three-phase WSD curriculum with hand-built case-law and philosophy corpora}
}

Acknowledgments

• Hugging Face / SmolLM team for the SmolLM2 tokenizer, FineWeb-Edu, FineMath, and Cosmopedia — and for the SmolLM reports that made the education-weighted recipe a known quantity
• Free Law Project for the CourtListener bulk data that made the law corpus possible
• Project Gutenberg and the sedthh/gutenberg_english packaging
• OpenStax for publishing textbook sources under CC BY
• BigCode (StarCoderData), Allen AI (peS2o), Wikimedia
• EleutherAI for lm-evaluation-harness, Proof-Pile-2, and the Pythia suite used as the compute-matched reference

---

Pretrained by TevunahAi AI Model Quantization and On-Premises AI Infrastructure From-scratch models, hybrid-architecture quantization (Mamba, MoE, SSM), and enterprise hardware bring-up

https://huggingface.co/TevunahAi