ProCreations/booper-pretrain

This model was asked to be published under my account, not the creators. The compute came from https://huggingface.co/posts/ProCreations/855858308074329

Booper pretrain

A small from-scratch transformer matching the babble / booper architecture, pretrained on a streamed slice of openbmb/Ultra-FineWeb-L1 (filtered English web text, Apache-2.0).

This is stage-1 English pretraining only. It is not the Discord-tuned chatbot. The architecture, tokenizer scheme, and training script come from kowo-co/babble (pretrain_hf.py + the default 34.1M config).

What it is

Parameters	34,096,128
Layers / width / heads	8 / 512 / 8
Context	1024 tokens
Tokenizer	byte-level BPE, 16,384 tokens
Data	openbmb/Ultra-FineWeb-L1
Train split	CC-MAIN-2025-51
Val split	CC-MAIN-2025-47 (disjoint crawl)
Tokens trained	600,206,202
Hardware	1× NVIDIA H200 (Hugging Face Jobs)
Wall clock	51 minutes end-to-end (46 min of training at ~217k tok/s)
Final train loss	2.531
Final val loss	2.472

Val loss fell steadily from 4.23 (step 200) to 2.47 (step 3118). Loss is nats per BPE token, not nats per byte, so it is not comparable to babble's older byte-level numbers without a bits-per-character conversion.

Files

• latest.pt — checkpoint (model state dict, config, optimizer, step/token counts)
• tokenizer.json — BPE merge list, loadable with babble.subword.BPETokenizer.from_json
• loss.jsonl — per-checkpoint train/val loss, throughput, and samples

End-of-run samples

Prompts used by the training script (temperature 0.7, top-k 40):

• the cat → the catastrophic system and its adaptability to manage the catastrophic system is the case with a significant surge
• In the beginning → In the beginning of the New Jersey Law and Law, “law enforcement of the Law and Law and Law in the law
• Scientists have discovered → Scientists have discovered that the current market is expected to take a long way to see how we look at the new market
• The weather today is → The weather today is about to create a more sustainable and sustainable future. It’s a way to make the most of your time and

These are expected to be clumsy: 34M params and 600M tokens is a short English pretrain, not a finished assistant.

Training

Run on Hugging Face Jobs with the repo's self-contained pretrain_hf.py (bf16 AMP, AdamW, cosine LR after warmup). Job: ProCreations/6a893d3e7c5c7dd37923450f.

Source: kowo-co/babble, including PR #21.