NanceTide/chinese-babylm-bert-mlm-first

Chinese BabyLM First Model

This repository contains the exported base model package for the first Chinese BabyLM Track 1 masked language model checkpoint in this workspace.

This is a base BertForMaskedLM checkpoint, not a task-specific classifier.

Model summary

• Architecture: BertForMaskedLM
• Hidden size: 768
• Layers: 8
• Attention heads: 12
• Intermediate size: 3072
• Max position embeddings: 512
• Vocab size: 19907
• Weights format: safetensors
• Main weight file size: about 279 MB

Files

• config.json
• model.safetensors
• tokenizer.json
• tokenizer_config.json
• special_tokens_map.json
• vocab.txt
• README.md

Intended use

This checkpoint is suitable for:

• masked language modeling inference
• continued pretraining
• initialization for downstream Chinese classification or NLI finetuning

For downstream classification, the task head still needs to be trained.

Evaluation snapshot

Best local Track 1 candidate scores assembled on 2026-06-11:

• ZhoBLiMP: 75.49
• AFQMC: 69.02
• OCNLI: 67.56
• TNEWS: 53.04
• CLUEWSC2020: 63.82

These results were produced in the local competition workspace and should be treated as project-side reporting rather than an official leaderboard submission.

Quick start

pip install torch transformers safetensors

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "YOUR_HF_USERNAME_OR_ORG/chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
model.eval()

Use as an encoder for finetuning

from transformers import AutoModel, AutoTokenizer

model_id = "YOUR_HF_USERNAME_OR_ORG/chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_id)
encoder = AutoModel.from_pretrained(model_id)
encoder.eval()

If you want to start a classification model from this checkpoint:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "YOUR_HF_USERNAME_OR_ORG/chinese-babylm-bert-mlm-first"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)

The classification head created above is randomly initialized and must be finetuned on the target task.

Notes

• The export excludes intermediate training checkpoints, optimizer states, and scheduler states.
• The package is intended for distribution and reuse, not for exact training-state restoration.