NecroDevelopment/roberta-mlm-cc12m-100pct

mlm-roberta-base-cc12m-100pct

RoBERTa-base continually pre-trained on CC12M captions with masked language modeling under the same 100k-step, target-word-marker recipe used by the companion contrastive run. Ships as the text-only control in the WiC semantic-proximity bake-off.

Model details

• Text encoder: roberta-base, all 12 layers unfrozen, MLM head trained on top
• Input: CC12M captions wrapped with <t>...</t> around a randomly chosen word (mark_random_target=True, marker_min_word_len=3); target_pool=False
• Objective: standard subword MLM, 15% masking probability
• Max sequence length: 77, set to match the contrastive run

Training

• Data: pixparse/cc12m-wds (CC12M, webdataset shards), captions branch only
• Steps: 100,000
• Per-device batch size: 128
• Optimizer: AdamW, weight decay 0.01
• Schedule: linear warmup 1,000 steps, then constant
• LRs: encoder 2e-5 with per-layer decay 0.85 from the top; MLM head 2e-4
• Precision: fp32