Tanvir106/TokenCal-Q1-Research

TokenCal Q1 — Three modes, three model families, three training seeds

Research repository: https://huggingface.co/Tanvir106/TokenCal-Q1-Research (private)

Research reproduction/extension code for Tokenization Fragmentation and Predictive Reliability in Low-Resource LLMs: A Bengali-Centric Cross-Lingual Study.

Status: The code was syntax-checked and its CPU mathematical/unit tests passed. Full CUDA training, external dataset downloads, and Hugging Face push functionality have not been executed in the authoring environment. Do not report planned results as completed results. Preflight must run successfully on your GPU before full training.

Quick start in Google Colab or Kaggle

1. Open TokenCal_Q1_Notebook.ipynb using a GPU runtime. The notebook is standalone: its cells create the Python runner, requirements and tests.
2. Log in to the Hugging Face account that owns this private repository (Tanvir106). Your access to gated Gemma/Llama checkpoints and ai4bharat/MILU must be enabled separately on Hugging Face.
3. The notebook will rebuild BEnQA from the official GitHub commit. The original exact split assignments and cleaned B-REASO validation set are available from your uploaded tokencal-results.zip; upload the ZIP in the notebook when asked, or use the included reference/ files when running from this full project ZIP. Original data are not re-uploaded to Hugging Face.
4. Choose preflight, then benchmark, then full. Select models and variants in the editable options cell.
5. Enable SYNC_HF=checkpoints for remote crash-recovery checkpoints. The code will upload the source code, manifests, checkpoint artifacts, metrics, predictions, and summary outputs to this private model repository. This can consume significant storage, and uploads may add runtime. Ensure access rights and storage quota are sufficient.
6. Read outputs/summary/ after full training. If interrupted, rerun the same configuration and optionally set --resume-hub.

Experiment matrix (54 full training runs)

Backbone	Variants trained	Seeds	Runs
Qwen2.5-1.5B-Instruct	M0 M1 M2 M3 Focal RDrop Const Shuffle	42,123,456	24
Gemma 3 4B IT	M0 M1 M2 M3 Focal RDrop	42,123,456	18
Llama 3.2 3B Instruct	M0 M1 M2 M3	42,123,456	12
Total			54

Temperature Scaling (TS) is fitted using the M0 validation predictions for each backbone and seed; it requires no additional LLM training. Sarvam-1 is not included.

Dataset protocol

Dataset	Role	Population	Access
BEnQA	train/validation/held-out test	3,553 / 762 / 762 pairs	original official GitHub repo; original split metadata
B-REASO	Bengali OOD evaluation	1,215 labeled validation examples	canonical cleaned reference from original ZIP
MILU	multilingual test only	BN 6,636; GU 4,825; ML 4,321; TE 7,304	gated official ai4bharat/MILU

• Never tune on BEnQA test, B-REASO, or MILU.
• All three backbones recompute their own tokenizer-specific fragmentation differences using training data.
• Locked BEnQA split assignment metadata prevents a silently new split.
• A/B/C/D next-token scoring is validated separately per tokenizer. If any label takes multiple tokens, the runner fails instead of silently using an incorrect scoring rule.
• No overlength prompt is truncated, even on new backbones. You must revise the context policy before training if an entire dataset does not fit.

Core M0–M3 losses (faithful to original notebook)

• M0: mean EN/BN full-vocabulary answer-token cross-entropy.
• M1: M0 + normalized four-class Brier (divide raw score by 2).
• M2: M1 + normalized EN–BN Jensen–Shannon divergence (divide by ln 2).
• M3: M2 + sum_i(weight_i * JS_i) / sum_i(weight_i) / ln(2).
• Fragmentation weight is the 0–1 clipped percentile normalization of EN–BN content fertility difference, using 5th/95th training-only percentiles.
• Const: M2 plus one extra unweighted pair-JS loss. Shuffle: M3 with train-only weights shuffled within the training population.
• Focal: full-vocabulary answer-token focal loss, gamma=2. RDrop: stochastic dual-pass bidirectional KL on four-choice distributions, plus ordinary task loss. These are adaptations for the fixed four-choice experimental protocol, and are not exact replicas of every published implementation.
• TS: validation-only temperature adjustment of M0's four-choice logits (NLL objective).

Crash-resume and validation

• A run ID uniquely identifies backbone/variant/seed/mode.
• Latest checkpoint includes LoRA adapter, optimizer, scheduler, gradient scaler, Python/NumPy/PyTorch/CUDA RNG, epoch, batch cursor, selected validation-Brier best score, and configuration fingerprint.
• Checkpoints are saved at optimizer-step boundaries; staged writes are atomically swapped locally.
• Checkpoint integrity is checked on resume. Do not load trainer.pt from untrusted sources (torch.load(weights_only=False) contains Python pickle objects).
• preflight loads each selected GPU model, executes forward/backward, saves a checkpoint, reloads optimizer/scheduler/RNG, continues an optimizer update, and checks inference.
• benchmark measures short training performance and peak GPU VRAM; it never contributes accuracy to research results.
• full trains for three epochs per selected combination, chooses checkpoint by the average validation Brier, and evaluates the frozen selected checkpoint on held-out BEnQA, B-REASO, and MILU.
• CPU mathematical tests: python -m pytest -q test_tokencal_q1.py.

Limitations and deployment warnings

1. The three non-identical backbones have different parameter sizes and tokenizer vocabularies. Compare within-backbone relative gains, not unadjusted raw model accuracy as evidence of a superior training method.
2. Gemma3 4B IT is multimodal and Llama 3.2 / MILU may require separate gated access. Gemma3 quantized LoRA can be memory-intensive on 2×T4. Two separate T4 cards do not provide a single contiguous 32GB device.
3. Despite successful mathematical/unit tests, no full GPU pipeline was executed here. If any real GPU preflight fails, correct the implementation first and rerun preflight; do not launch all 54 experiments.
4. The current implementation extends the original study rather than claiming to have exactly reproduced the original training bit-for-bit. Different tokenizers, optimizer policy (strict Torch AdamW), environment, seed and data processing decisions can change numbers.
5. The notebook's first upload sends source files, not copyrighted third-party dataset content. The private run repo will store model adapters and evaluation artifacts. Inspect dataset licensing before sharing any derived raw text publicly.
6. Temperature scaling is a post-hoc baseline that generally does not change argmax accuracy. Focal/R-Drop can affect both accuracy and calibration; evaluate all metrics.
7. Original observed strict configuration comparisons were not independent multi-seed evidence. Future results must be statistically audited before Q1 submission.

CLI commands (optional)

python tokencal_q1.py --mode preflight --models qwen --variants M0,M3 --seeds 42 --source-zip /content/tokencal-results.zip
python tokencal_q1.py --mode benchmark --models qwen,gemma,llama --variants M0,M3,RDrop --seeds 42 --source-zip /content/tokencal-results.zip --benchmark-steps 50
python tokencal_q1.py --mode full --models qwen,gemma,llama --variants M0,M1,M2,M3,Focal,RDrop,Const,Shuffle,TS --seeds 42,123,456 --source-zip /content/tokencal-results.zip --sync-hf checkpoints --source-upload

When you restart a Colab runtime, execute the notebook's source-writing cells again, select the same model/seed/method, and set --resume-hub if the latest checkpoint is stored in the HF repo.