Tokens indicate unbalanced subsets, that do not represent general English
Only focusing on 5 hundred tokens not present in gemma3 tokenizer. May be OK, for you to decide. Jist FYI
Suboptimal tokens that fail to represent the broader historical English language.
Data Pipeline and Encoding Artifacts (Mojibake)
These tokens are not words; they are the result of character encoding failures (e.g., UTF-8 misinterpreted as Latin-1) during the digitization of historical texts. Their presence in a tokenizer vocabulary is a critical data quality failure.
Abbé (1): Mojibake for "Abbé".
Franç (183): Mojibake for "Franç" (as in François or Française).Aggressive BPE and OCR Fragmentation Glitches
The tokenizer has learned to split common words into nonsensical or highly inefficient subwords. This typically occurs when punctuation, abbreviations, or line breaks in early OCR scans prevent proper word merging.
HAPTER (208): A recurring glitch for "CHAPTER" (e.g., from "C HAPTER" or "Ch. HAPTER").
ESUS (168), ICHARD (229), LIAM (275): Fragments of "JESUS," "RICHARD," and "WILLIAM," caused by the tokenizer splitting after a period or abbreviation (e.g., "J. ESUS", "W. LIAM").
UCTION (509), RIPT (419), RAPH (406), ENERAL (161): Useless suffixes stripped from "RODUCTION/DUCTION," "SCRIPT," "GRAPH," and "GENERAL."
Yanke (532): Truncated form of "Yankee."
TROD (499): While "trod" is a valid archaic verb, its high-frequency tokenization here is likely an artifact of splitting words like "inTRODuced" or "inTRODuction" in poorly formatted texts.Structural and Paratextual Artifacts
These tokens belong to the editorial apparatus of a book, not the natural language of the text. Allocating vocabulary slots to them wastes model capacity.
FINIS (177): Latin end-of-book marker.
FOOTNOTES (182), Footnote (181), Sidenote (452): Editorial metadata.Extreme Genre Skew: Theological and Ecclesiastical Overrepresentation
A massive portion of this vocabulary is dedicated to religious, theological, and sectarian terminology. While these are valid historical words, their density indicates the corpus is with 16th–18th century sermons, biblical commentaries, and polemical tracts (a known issue in early, uncurated digital archives like early Project Gutenberg dumps).
Doctrinal/Sectarian: Antichristian (25), Catholick (74), Catholicks (75), Catholike (76), Congregational (101), Dissenters (144), Hereticks (220), Papal (340), Papist (341), Popery (368), Popish (370), Presbyterians (383), Protestantism (392), Puritans (400), Unitarian (510).
Biblical/Religious Narrative: Apoc (28), Baptisme (43), Catechism (72), Circumcision (83), Devill (136), Gospell (196), Heauen (211), Idolatry (231), Iesu (234), Iesus (235), Iewes (236), Incar (240), Incarnation (241), Ioseph (248), Iudgement (252), Jewes (258), Levites (274), Miracles (301), Psalme (394), Purgatory (399), Sacraments (426), Sathan (431), Sauiour (432), Tabernacle (476).
Note on Archaic Spelling: Tokens like Iesus, Iewes, Ioseph, and Iudgement use the Early Modern English "I/J" interchange. While historically accurate, their high frequency as distinct tokens confirms the corpus is disproportionately weighted toward texts printed before 1700.Classical and Historical Name Fragments
Similar to the previous list, the tokenizer has learned truncated fragments of classical or historical names, likely due to inconsistent OCR or formatting in history books.
Carthag (70): (Carthage/Carthaginian)
Chalde (78): (Chaldean)
Ptole (396): (Ptolemy, despite Ptolemy also being present at 397)
Tusc (504): (Tuscan/Tuscany)
Norwe (323): (Norway/Norwegian)
Neap (315): (Neapolitan)
Somer (461), Seym (446): Likely fragments of "Somerset" and "Seymour."Non-English and Regional Artifacts
The presence of these tokens indicates the inclusion of specific, unbalanced subsets of data that do not represent general English.
Arglwydd (33), Dduw (126): Welsh words ("Lord" and "God"). Their presence strongly suggests the corpus ingested Welsh-English bilingual religious texts or Welsh chapbook archives without language filtering.
Mademoiselle (283), Signor (453): While sometimes used in English literature, their elevation to high-frequency tokens suggests an overrepresentation of specific translated works or travelogues.Literary and Dramatic Skew
Tokens that point to an overrepresentation of specific authors or genres, rather than general language use.
Exeunt (174): A stage direction exclusively found in drama (e.g., Shakespeare).
Shakspe (447), Shakspeare (448): Archaic spelling variants of Shakespeare, indicating heavy ingestion of Shakespearean folios or specific literary critiques.
Jonson (259), Cowper (120), Southey (465), Coleridge (89), Boswell (58), Gibbon (189): High-frequency author names suggest the corpus contains a large amount of literary criticism, biographies, or footnotes about these specific figures.
cat 32522_t-v4.list|ug -vwf <(ug -v '\)' common_with_gemma.22969.list|rg -v '_|\+|\*|\(\||\(\--'|sed 's/---//g;s/-,//g;s/[\}\{\(\[\|]//g;s/\?//g')|rg -v [0-9]|sort|rg [A-Z]|cat -n
1 Abbé
2 Accus
3 Adela
4 Adieu
5 Administr
6 Adul
7 Adversaries
8 Aegypt
9 Affection
10 Affections
11 Affliction
12 Againe
13 Aldermen
14 Allegiance
15 Altar
16 Ambition
17 AMER
18 Amidst
19 Anabapt
20 Ancestors
21 Ancients
22 Angell
23 Angl
24 Answ
25 Antichristian
26 Antiquity
27 Anto
28 Apoc
29 Apostol
30 Appre
31 Apri
32 Archite
33 Arglwydd
34 Armes
35 Armies
36 Assem
37 Assigns
38 Assyria
39 Assyrian
40 Austrians
41 Aven
42 Bacchus
43 Baptisme
44 Baptized
45 Baronet
46 Barons
47 Battel
48 Belie
49 Believers
50 Bellarmine
51 Bessie
52 Bibles
53 Bism
54 Blasphe
55 Blessings
56 Bohem
57 Booke
58 Boswell
59 Brethren
60 Brutus
61 Buonaparte
62 Burnet
63 Cadiz
64 Calam
65 Canons
66 Captaine
67 Captains
68 Captivity
69 Cardinall
70 Carthag
71 Catech
72 Catechism
73 Catharine
74 Catholick
75 Catholicks
76 Catholike
77 Ceremonies
78 Chalde
79 Charlemagne
80 Christe
81 Christes
82 Christs
83 Circumcision
84 Circumstances
85 Citie
86 Claudius
87 Clellan
88 Clergy
89 Coleridge
90 Colledge
91 Commandement
92 Commandements
93 Commandment
94 Companions
95 Concil
96 Concur
97 Condem
98 Confederates
99 Confess
100 Confession
101 Congregational
102 Congregations
103 Conne
104 Conqueror
105 Conscience
106 Consciences
107 Consequence
108 Conspir
109 Constitutions
110 Contempl
111 Contemplation
112 Contempt
113 Cornwallis
114 Councel
115 Councell
116 Counsell
117 Counsels
118 Countenance
119 Countrey
120 Cowper
121 Crosse
122 Crowne
123 Cruelty
124 Cunning
125 Davids
126 Dduw
127 Debts
128 Dece
129 Decrees
130 Defe
131 Defend
132 Desires
133 Desp
134 Despair
135 Deum
136 Devill
137 Devonshire
138 Devoted
139 Devotion
140 Dignity
141 Dioces
142 Disciples
143 Discourses
144 Dissenters
145 Divines
146 Doctr
147 Doctrines
148 Domini
149 Dominions
150 Doubtless
151 Earls
152 Eccl
153 Ecclesiae
154 Ecclesiast
155 Ecclesiastical
156 Ecclesiasticall
157 Elector
158 Emperour
159 Emperours
160 Enacted
161 ENERAL
162 Envy
163 Ephraim
164 Epiph
165 Episcop
166 Epist
167 Epistles
168 ESUS
169 Euangel
170 Euphrates
171 Evangelist
172 Evangelists
173 Everlasting
174 Exeunt
175 Exhort
176 Favour
177 FINIS
178 Flem
179 Florentine
180 Folly
181 Footnote
182 FOOTNOTES
183 Franç
184 Francisc
185 Frenchmen
186 Garrick
187 Generall
188 Geor
189 Gibbon
190 Glimpse
191 Glorious
192 Godhead
193 Godliness
194 Godly
195 Goodness
196 Gospell
197 Goth
198 Goths
199 Governour
200 Governours
201 Graces
202 Gracious
203 Greci
204 Grecian
205 Greeke
206 Guilt
207 Guise
208 HAPTER
209 Heathen
210 Heathens
211 Heauen
212 Heavens
213 Hebr
214 Heirs
215 Hellenic
216 Herein
217 Heres
218 Heresie
219 Heret
220 Hereticks
221 Herodotus
222 Hindoo
223 Hitherto
224 Hopes
225 Humility
226 Humour
227 Husbands
228 Hypocr
229 ICHARD
230 Idolat
231 Idolatry
232 Idols
233 Ierusalem
234 Iesu
235 Iesus
236 Iewes
237 Ignor
238 Ignorance
239 Impost
240 Incar
241 Incarnation
242 INCE
243 Infall
244 Infidels
245 Inhabitants
246 Iniquity
247 Intellect
248 Ioseph
249 Iroquois
250 Isai
251 Iudge
252 Iudgement
253 Iudges
254 Iust
255 Iustice
256 Iustices
257 Jacqu
258 Jewes
259 Jonson
260 Josephus
261 Jove
262 Judea
263 Judg
264 Judgement
265 Judgments
266 Justification
267 Kingdome
268 Kingdomes
269 Laced
270 LADY
271 Latine
272 Lawes
273 Lege
274 Levites
275 LIAM
276 Liturgy
277 Lorde
278 Lordship
279 Loue
280 Lucius
281 Lungs
282 Lusts
283 Mademoiselle
284 Magdalen
285 Magistr
286 Maiest
287 Maiestie
288 Maiesties
289 Maint
290 Maister
291 Majest
292 Majestie
293 Majesties
294 Malice
295 Marry
296 Marseilles
297 Mercies
298 Metap
299 Ministery
300 Minnes
301 Miracles
302 Mischief
303 Misery
304 Mistress
305 Monarchy
306 Monaster
307 Monks
308 Montagu
309 Moone
310 Moorish
311 Muses
312 Musick
313 Mussul
314 Natures
315 Neap
316 Negroes
317 Neighb
318 Neighbours
319 Nerves
320 Newgate
321 Nobility
322 Normans
323 Norwe
324 Oaths
325 Obedience
326 Objections
327 Obligation
328 Occasion
329 Occasions
330 Offence
331 OMAN
332 Ordinances
333 Ordination
334 OUNG
335 OUNT
336 Ovid
337 Pagans
338 Pains
339 Palmerston
340 Papal
341 Papist
342 Parliaments
343 Parson
344 Particulars
345 Passages
346 Passions
347 Pastors
348 Patter
349 Paule
350 Pauls
351 Persec
352 Persecution
353 Pharise
354 Philist
355 Philistines
356 Philosopher
357 Philosophers
358 Phoe
359 Physick
360 Piety
361 Pilate
362 Pity
363 Plaint
364 Pleasures
365 Plutarch
366 Pomp
367 Pompey
368 Popery
369 Popes
370 Popish
371 Posterity
372 Preach
373 Preached
374 Preacher
375 Preachers
376 Preaching
377 Precept
378 Precepts
379 Prejud
380 Prelates
381 Prerogative
382 Presby
383 Presbyterians
384 Presbyters
385 Presbytery
386 Priesthood
387 Priviledges
388 Proclamation
389 Prophecy
390 Prophes
391 Prosper
392 Protestantism
393 Prudence
394 Psalme
395 Psalmist
396 Ptole
397 Ptolemy
398 Publick
399 Purgatory
400 Puritans
401 Pyram
402 Quarters
403 Queene
404 Quixote
405 Rale
406 RAPH
407 Rebell
408 Reformers
409 Regard
410 Remem
411 Repentance
412 Repro
413 Republ
414 Reverence
415 Richelieu
416 Riches
417 Righteous
418 Righteousness
419 RIPT
420 Rites
421 Romane
422 Romish
423 Royall
424 Ruin
425 Rulers
426 Sacraments
427 Sacrifices
428 Sanctification
429 Sarac
430 Saracens
431 Sathan
432 Sauiour
433 Saxons
434 Schism
435 Scipio
436 Scribes
437 Senses
438 Separately
439 Sepulch
440 Serjeant
441 SERMON
442 Sermons
443 Serpents
444 Servants
445 Settle
446 Seym
447 Shakspe
448 Shakspeare
449 Sheriffs
450 Shillings
451 Sickness
452 Sidenote
453 Signor
454 Sinner
455 Sinners
456 Sins
457 Slaves
458 Societ
459 Sodom
460 Solemn
461 Somer
462 Souldier
463 Souldiers
464 Soule
465 Southey
466 Sove
467 Sover
468 Soveraign
469 Spenser
470 Strangers
471 Successors
472 Sufferings
473 Sulphur
474 Sunne
475 Superstition
476 Tabernacle
477 Tacitus
478 Talm
479 Tartar
480 Temperance
481 Temples
482 Temptation
483 Temptations
484 Tenements
485 Territ
486 Tertullian
487 Testimonies
488 Thebes
489 Thence
490 Therfore
491 THis
492 Tithes
493 Tongues
494 Towne
495 Transgress
496 Trave
497 Treason
498 Trist
499 TROD
500 Trumpet
501 Truths
502 Tryal
503 Tumult
504 Tusc
505 Twas
506 Tyranny
507 Tyrant
508 Tythes
509 UCTION
510 Unitarian
511 Usur
512 Veins
513 Vengeance
514 Verses
515 Vertue
516 Vertues
517 Virgins
518 Virtues
519 Visc
520 Volunt
521 Welling
522 Whence
523 Wher
524 Wherein
525 Whereupon
526 Whig
527 Whigs
528 Wickedness
529 Wives
530 Womb
531 Wounds
532 Yanke