fablog-tokenizer-16k

Byte-level BPE tokenizer for machine-level logs (fab tool events, ATE/wafer-sort, burn-in/memory test, GPU telemetry). Trained with build_log_tokenizer.py on a synthetic fab-log corpus (17.7 MB, 4,000 documents) — retrain on real data before production use.

Design

• Byte-level BPE, no unicode normalizer → encode→decode is byte-exact, no <unk> ever
• Digits pre-split in groups of 1–3 (Llama-3 style): 2026 → 202,6; 12.04 → 12,.,04
• = isolated so field names (pwr=, lot=) never fuse with values
• Specials: <|endoftext|> (id 0, document separator, also BOS), <|pad|> (id 1); vocab padded with <|reserved_k|> tokens to a multiple of 128
• Vocab: 2,944 (< 65,536 → tokens fit in uint16 for datatrove/Nanoset storage)

Measured fertility (tokens/byte on the training-domain sample)

tokenizer	tok/byte	tok/line
HuggingFaceTB/SmolLM2-135M	0.635	69.2
bigcode/starcoder2-3b	0.620	67.6
gpt2	0.524	57.1
this tokenizer	0.475	51.7

(~29% of corpus bytes are digits/hex — entropy-limited floor.)

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("JugsMa/fablog-tokenizer-16k")
ids = tok("<REC> ts=2026-07-03T13:04:15.092Z tool=ETCH07 status=OK")["input_ids"]
assert tok.decode(ids) == "<REC> ts=2026-07-03T13:04:15.092Z tool=ETCH07 status=OK"

For nanotron/datatrove preprocessing, pass the Hub id directly to tools/preprocess_data.py --tokenizer-name-or-path <your-namespace>/fablog-tokenizer-16k --eos-token "<|endoftext|>".