deberta-base-75k — BabyLM 2026 strict-small (random-init control)

DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure), 75k byte-level BPE tokenizer, standard random embedding initialization — the matched control for deberta-base-75k-sam: identical data, architecture, tokenizer, and schedule; the only difference is embedding initialization. Training: lr 2e-4, batch 256, grad-acc 4, MLM 15%.

Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM 2024 submission (Edman et al. 2024, "Are BabyLMs Second Language Learners?") — a mixture of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia, Gutenberg, Switchboard). Within the strict-small 10M-word budget.

Intermediate checkpoints: chck_1M … chck_100M and step1000 … step25740; main = final.

Code and analyses: https://github.com/bylinina/augustinian_babylm