deberta-base-75k-s3

Random-initialization baseline for the 75k vocabulary. This is the control that vision-seeded models with the same vocabulary are compared against.

Seed 3 of 3. This configuration was retrained with three random seeds: augustinian-babylm/deberta-base-75k, augustinian-babylm/deberta-base-75k-s2, augustinian-babylm/deberta-base-75k-s3.

Seeded counterparts: augustinian-babylm/deberta-base-75k-sam, -dinov3, -ibot.

Part of the model set for Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model (paper, code).

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("augustinian-babylm/babylm-bpe-75k")
model = AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-75k-s3")

Training

DeBERTa-v3-base masked LM (768 hidden, 12 layers, 12 heads), trained for 10 epochs on bb24.train, the ~9.9M-word corpus of Edman et al. (2024), which mixes LLM-generated paraphrase data with portions of the official BabyLM corpus. This is not the official 2026 strict-small distribution.

Learning rate	2e-4, cosine schedule, 4000 warmup steps
Optimizer	AdamW (beta1 0.9, beta2 0.95), weight decay 0.01
Effective batch	256 (gradient accumulation 4)
Context length	64 to 128 from epoch 5, max 512
Vocabulary	BPE, 75k merges (augustinian-babylm/babylm-bpe-75k)

Only the input-embedding initialization differs between a vision-init model and its baseline. Every other hyperparameter, the data order, and the random seed are held fixed.

Checkpoints

Intermediate checkpoints are stored as branches, saved Pythia-style by step:

from transformers import AutoModelForMaskedLM
AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-75k-s3", revision="step256")

Call huggingface_hub.list_repo_refs("augustinian-babylm/deberta-base-75k-s3") to list the available steps.

Results

Vision seeding produces a consistent object-property gain (COMPS, +1.30 averaged over nine encoder-by-vocabulary configurations, positive in all nine) while grammar-focused benchmarks stay flat. The effect is word-specific: on a corpus-tailored Visual-Property Swap probe the advantage falls entirely on seeded words, with the placebo cell at +0.000. See the paper for full results.

Limitations

Trained on a non-standard corpus (see above), so results are not directly comparable to submissions trained on the official 2026 strict-small data. Single masked-LM architecture. Research artifact, not intended for deployment.

Citation

@inproceedings{bylinina2026augustinian,
  title     = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
               Teach a Small Language Model},
  author    = {Bylinina, Lisa},
  booktitle = {Proceedings of the BabyLM Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=B4TD4XdlwF}
}