LickyArc/bengali-tokenizer-pilot

Bengali tokenizer transplant pilot

The execution plan is in bengali-3day-pilot.md. The first runnable deliverable is the Day 1 fertility gate.

Run the fertility gate

python3 scripts/fertility_gate.py --output reports/fertility-gate.json

The script loads one shared streaming sample from IndicCorpV2 Bengali, measures all candidate tokenizers against that same sample, and records a PASS, REVIEW, or STOP decision. A gated or unavailable model is recorded as an error without discarding the other measurements.

The live IndicCorpV2 dataset schema uses the indiccorp_v2 config and the ben_Beng Bengali split; these are the script defaults. This differs from the older bn/train example in the pilot note.

Use --model MODEL_ID repeatedly to test a different candidate set. The default sample is 2,000 non-empty text rows, matching the pilot plan.

Prepare the corpus and tokenizer

After the gate passes, prepare the filtered corpus (the default is the pilot’s 1.2M train and 5k validation documents):

python3 scripts/prepare_corpus.py

Then train the 32k tokenizer on the first 200k training documents:

python3 scripts/train_tokenizer.py

Once the tokenizer exists, record the paired fertility and vocabulary metrics required by the pilot:

python3 scripts/measure_tokenizer.py \
  --base-tokenizer meta-llama/Llama-3.2-1B \
  --new-tokenizer bn_tokenizer \
  --input data/indiccorp_bn/train.jsonl \
  --output reports/tokenizer-metrics.json

For FOCUS, train its static fastText space on the same sample first:

python3 scripts/train_fasttext.py \
  --input data/indiccorp_bn/train.jsonl \
  --output reports/bengali-fasttext.bin \
  --max-docs 200000

Then initialize the two transplant arms from the base checkpoint:

python3 scripts/initialize_embeddings.py \
  --base-model meta-llama/Llama-3.2-1B \
  --new-tokenizer bn_tokenizer \
  --method mean \
  --output-dir checkpoints/b3-mean-init

python3 scripts/initialize_embeddings.py \
  --base-model meta-llama/Llama-3.2-1B \
  --new-tokenizer bn_tokenizer \
  --method focus \
  --fasttext-model reports/bengali-fasttext.bin \
  --output-dir checkpoints/b4-focus-init

fasttext is optional until the FOCUS arm is needed; the rest of the Day 1 utilities do not depend on it.

Both commands are intentionally opt-in because the corpus download and output can be large. They refuse to replace existing outputs unless --overwrite is passed.

Sanity gate and training

After both transplant checkpoints exist, run the pre-training gate:

python3 scripts/sanity_check.py \
  --checkpoints checkpoints/b3-mean-init checkpoints/b4-focus-init \
  --base-model meta-llama/Llama-3.2-1B \
  --include-random \
  --val data/indiccorp_bn/val.jsonl \
  --output reports/sanity-check.json

The gate evaluates each checkpoint with its own tokenizer and exits nonzero if the required random > mean > FOCUS loss ordering does not hold. Only launch training after it passes. The control arm uses --stage1-tokens 0:

python3 scripts/train.py \
  --init-checkpoint checkpoints/b4-focus-init \
  --arm-name b4-focus \
  --train data/indiccorp_bn/train.jsonl \
  --val data/indiccorp_bn/val.jsonl \
  --output-dir runs/b4-focus

Use the same --total-tokens for B1, B3, and B4. Set --micro-batch to the largest batch that fits the target GPU; gradient accumulation is calculated to stay near the 500k-token effective batch by default.

Evaluation and plots

Evaluate each saved checkpoint and append to the shared results file:

python3 scripts/evaluate.py \
  --checkpoint runs/b4-focus/ckpt-500M \
  --arm b4-focus \
  --tokens 500000000 \
  --output reports/results.jsonl

This requires lm_eval for MILU and MMLU. Use --skip-lm-eval only for local smoke tests; BPC, fertility, and inference throughput still run. Finally:

python3 scripts/plot_results.py \
  --input reports/results.jsonl \
  --output-dir reports

That writes milu_vs_gpu_hours.png, pilot_metrics.png, and results_table.md.

Offline checks

python3 -m unittest discover -s tests -v