OmarH455/mistral-7b-sp-cs

Mistral-7B SentencePiece, CS vocab expansion

Computer science vocab expansion for the Mistral-7B-v0.1 SentencePiece tokenizer. Words are counted on a CS corpus (arXiv, CodeNet, CornStack, GitHub, StackExchange), gated, ranked, and added as single pieces. Code: contrastors-mosaic-mistral-sp.

Method

• Count words on 9 CS shards (arXiv, CodeNet, CornStack, GitHub, StackExchange), capped at 100,000 docs per shard.
• Gate: seen in 3+ shards, 12+ docs, at most 3 hits per doc counted.
• Rank by count x (pieces - 1). Max piece length 16.
• Add both the space form and the bare form where the bare form is used. Keep bridge pieces for live words.
• Drop targets that never fire on a drop sample, rebuild bridges, repeat.

Folders

• 3839/merged: 3,839 candidate words, 4,514 rows added (3,421 words + 1,093 bridges)
• 8000/merged: 7,780 rows added (6,059 words + 1,721 bridges)
• 12000/merged: 10,028 rows added (7,850 words + 2,178 bridges)

Each has tokenizer.model, tokenizer.json, tokenizer_config.json, special_tokens_map.json, and added_pieces.json.

Results

Held-out: 22,500 cleaned lines and 22,500 raw lines, separate from the drop sample. Stock Mistral: 524,142 tokens cleaned, 524,555 raw.

• 3839: 5.11% fewer tokens cleaned, 5.26% raw
• 8000: 6.21% cleaned, 6.38% raw
• 12000: 6.86% cleaned, 6.98% raw
• Trained tokenizer (13,001 pieces), same held-out: 6.40%

Checks, all budgets: one token 100% (space and bare), round-trip 22,500/22,500, expand-back per text and per token 100%, "The COVID pandemic" unchanged, 0 targets failed one-token.

The curve flattens after 8k. 8000 gets close to the trained tokenizer with 40% fewer rows.

Notes

• 3839 is a word budget, not a row budget. It is not like-for-like with a 3,839 row total.
• No eval-set numbers yet.
• New embeddings need initialization before use. The base model weights are not included.