fablog-mix-tokenizer-32k

Byte-level BPE tokenizer for machine-level logs (fab tool events, ATE/wafer-sort, burn-in/memory test, GPU/HPC telemetry) with English coverage. Successor to JugsMa/fablog-tokenizer-16k.

Trained (88 MB mixture) on:

• BGL — 58 MB of real Blue Gene/L supercomputer RAS logs (Loghub)
• synthetic fab logs — 18 MB (SECS/GEM-style tool runs, wafer-sort, burn-in, GPU telemetry)
• wikitext-103 — 15 MB English (~17%), so prose is not spelled out byte-by-byte

No confidential data: all three sources are synthetic or public.

Design

• Byte-level BPE, no unicode normalizer → encode→decode is byte-exact, never <unk>
• Digits pre-split in groups of 1–3 (Llama-3 style): 2026 → 202,6; 12.04 → 12,.,04
• = isolated so key= never fuses with its value
• Specials: <|endoftext|> (id 0, doc separator, also BOS), <|pad|> (id 1)
• Vocab 32,768 — fully filled, multiple of 128 (TP-friendly), < 65,536 (tokens fit uint16)

Measured fertility (tokens/byte; lower is better)

corpus	this tokenizer	Llama-3 (128k)	SmolLM2
synthetic fab logs	0.474	0.463	0.634
BGL real machine logs	0.506	0.478	0.760
English (wikitext)	0.218	0.223	0.233

Near-parity with Llama-3 on logs and better on English, at ¼ of the vocabulary — at hidden size 1024, embeddings + LM head cost ~67M parameters instead of ~263M.

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("JugsMa/fablog-mix-tokenizer-32k")
ids = tok("- 1117838570 2005.06.03 R02-M1-N0-C:J12-U11 RAS KERNEL INFO instruction cache parity error corrected")["input_ids"]
assert tok.decode(ids).endswith("corrected")

For nanotron/datatrove: tools/preprocess_data.py --tokenizer-name-or-path JugsMa/fablog-mix-tokenizer-32k --eos-token "<|endoftext|>"; in training configs: vocab_size: 32768, eos_token_id: 0, bos_token_id: 0.

Built with build_log_tokenizer.py (fertility → train → verify gates).