Capitalization not normalized
Rather than representing the general historical English language, these tokens are artifacts of specific book formats, publishing metadata, or heavily overrepresented genres.
Below is a categorization of the suboptimal tokens and the specific corpus artifacts they represent.
Structural and Formatting Artifacts
These tokens belong to the physical or digital structure of the books rather than the natural language used within the text. They consume valuable vocabulary slots on non-linguistic elements.
FINIS (261): The traditional Latin marker denoting the end of a book. It is not a natural English word.
FOOTNOTES (389), NOTES (315), NOTE (629): Editorial apparatus and structural elements, not narrative language.
MSS (352), MS (353): Abbreviations for "manuscript(s)," highly specific to scholarly, antiquarian, or archival editions.
XL (274, 599), LXX (507), LX (483): Roman numerals used for chapter or section numbering in older typesetting, rather than standard English words.
VOL (381, 383), PART (415, 434), SEC (384, 597), PAGE (585): Structural divisions of multi-volume works.Title Page and Publishing Metadata Artifacts
Early printed books contain extensive metadata on their title pages. When a corpus is not cleaned of these elements, these terms appear with unnatural frequency.
LONDON (310, 416), YORK (558): While these are legitimate cities, their extreme frequency in the top tokens is an artifact of title pages (e.g., "Printed in London") and specific geographical settings, rather than general vocabulary.
AUTH (630), PUBL (631): Fragments of "Author" and "Publisher/Published" extracted directly from title pages.
HAPTER (28), CHAPTER (39, 96), CHAP (104, 125): Artifacts of chapter headings. HAPTER is a clear tokenization glitch, likely resulting from OCR errors or spacing issues (e.g., "C HAPTER" or "Ch. HAPTER").Genre-Specific Skews (The "Subset of Books" Effect)
The corpus appears heavily weighted toward specific genres, causing the vocabulary to be dominated by terms unique to those subsets rather than representative of the broader language.
Religious Texts: GOD (54, 568), LORD (93, 596), JESUS (396), CHRIST (143), SERMON (591), CHURCH (602). Early digital corpora with Bibles, sermons, and theological tracts - these terms far beyond their natural frequency in general literature. Capitalization in the source texts caused the tokenizer to learn these as distinct, suboptimal tokens rather than normalizing them.
Drama and Epistolary Novels: SCENE (474, 565), LETTER (244, 341). The high frequency of these tokens indicates an overrepresentation of Shakespearean drama and 18th/19th-century epistolary fiction (e.g., works by Samuel Richardson).
Biblical Names: JOHN (146, 460), THOMAS (444), JAMES (422). These are inflated by the Gospels and Epistles rather than general historical naming conventions.
- Subword Tokenization Glitches (BPE Artifacts)
Because the tokenizer was trained on this specific, uncleaned subset of books, it learned inefficient subwords due to specific capitalization, punctuation, or OCR errors present in those texts.
Punctuation and Spacing Splits: ESUS (346), RIST (114), ASHINGTON (563), INGTON (343), ILLIAM (179), ICHARD (538), OMAS (350). These are fragments of "Jesus," "Christ," "Washington," "William," "Richard," and "Thomas." They occur because the tokenizer failed to properly merge common words when they were preceded by abbreviations or punctuation (e.g., "J. ESUS", "W. ASHINGTON", "Mr. ILLIAM").
Suffix/Prefix Over-merging: ENERAL (374), ERAL (295). Fragments of "General," likely split due to the word's frequent use as a title (e.g., "Gen. ENERAL") or structural heading (e.g., "GENERAL INDEX").
Case Artifacts: THe (239), THis (644). Inconsistent capitalization in the source texts caused the tokenizer to learn these as distinct, suboptimal tokens rather than normalizing them.
rg [A-Z][A-Z] 32522_t-v4.list|rg -vwf roman-digits.txt|cat -n
1 ER
2 HE
3 ON
4 IN
5 AN
6 AR
7 OR
8 ES
9 EN
10 THE
11 HA
12 IS
13 OF
14 TER
15 AT
16 AL
17 HAP
18 IT
19 LE
20 IC
21 RE
22 THE
23 OU
24 ND
25 EL
26 ION
27 ING
28 HAPTER
29 IL
30 AS
31 TH
32 OL
33 OD
34 OM
35 ED
36 US
37 AM
38 AND
39 CHAPTER
40 IR
41 ENT
42 AC
43 ET
44 UR
45 OT
46 IN
47 EC
48 ST
49 TO
50 CH
51 RO
52 IG
53 AD
54 GOD
55 AND
56 CH
57 IST
58 ORD
59 ID
60 GE
61 ST
62 AY
63 SE
64 LI
65 RI
66 ATION
67 OW
68 EM
69 UM
70 ERS
71 AP
72 UL
73 UT
74 CE
75 ARD
76 ESS
77 TH
78 DE
79 NE
80 AB
81 ELL
82 OP
83 OND
84 OS
85 AV
86 IM
87 IA
88 OUR
89 CT
90 ANT
91 ART
92 EST
93 LORD
94 OH
95 ECT
96 CHAPTER
97 RE
98 AN
99 PR
100 IF
101 OV
102 UN
103 BER
104 CHAP
105 FR
106 TON
107 FOR
108 OHN
109 AIN
110 ILL
111 EW
112 ITH
113 RY
114 RIST
115 CON
116 AG
117 ALL
118 KE
119 IGH
120 OC
121 OO
122 MS
123 BY
124 AR
125 CHAP
126 QU
127 PH
128 SON
129 LES
130 IES
131 OR
132 END
133 AL
134 OG
135 INE
136 IAN
137 TO
138 TR
139 EAR
140 ON
141 LY
142 ARY
143 CHRIST
144 URE
145 IE
146 JOHN
147 ATE
148 MAN
149 AST
150 AT
151 OUN
152 ATH
153 IGHT
154 ISH
155 VER
156 LAND
157 COM
158 UB
159 ORY
160 SP
161 BE
162 OOK
163 SC
164 THER
165 IP
166 FF
167 UD
168 PER
169 LET
170 AGE
171 TR
172 ACE
173 UC
174 NOT
175 MON
176 ONDON
177 ICH
178 HAT
179 LIAM
180 BY
181 ITY
182 WH
183 TS
184 PR
185 ENE
186 BR
187 AS
188 EN
189 IS
190 ED
191 ENTS
192 OOD
193 WIL
194 LEY
195 EX
196 RO
197 GH
198 MR
199 CL
200 ENCE
201 ICK
202 SE
203 OUS
204 CL
205 UE
206 MAR
207 INIS
208 AD
209 SC
210 RA
211 ISS
212 CHAR
213 NEW
214 CON
215 MY
216 LL
217 PRO
218 LET
219 ICAL
220 HE
221 MR
222 PL
223 OST
224 TE
225 IONS
226 ASS
227 LE
228 PT
229 ICE
230 ORK
231 IND
232 MAN
233 MER
234 FROM
235 LD
236 AK
237 IB
238 ORT
239 THe
240 AU
241 SIR
242 UST
243 WITH
244 LETTER
245 WE
246 OLD
247 HAM
248 MENT
249 OME
250 ORE
251 SIR
252 CC
253 ERE
254 ACK
255 BERT
256 IVE
257 GR
258 EX
259 HEN
260 OF
261 FINIS
262 BE
263 CO
264 ANS
265 HIS
266 ANC
267 FR
268 SER
269 GRE
270 OOT
271 SH
272 AW
273 IRE
274 XL
275 COL
276 ATIONS
277 AMES
278 DE
279 OB
280 OUNT
281 WILLIAM
282 ONE
283 PH
284 KING
285 PL
286 IRST
287 OY
288 AB
289 GR
290 GE
291 DR
292 ERN
293 PAR
294 UN
295 ERAL
296 PE
297 BL
298 ANCE
299 ONG
300 BR
301 MAR
302 ACT
303 MA
304 ITT
305 IUS
306 ENS
307 ABLE
308 NOT
309 SH
310 LONDON
311 IAL
312 FIG
313 IO
314 PP
315 NOTES
316 ITION
317 INC
318 IFE
319 REE
320 HES
321 ORGE
322 ME
323 ACH
324 RED
325 OCK
326 WH
327 UG
328 FOOT
329 CTION
330 BUR
331 SP
332 DEAR
333 SECT
334 QU
335 IEL
336 ANG
337 HAR
338 INGS
339 FN
340 BOOK
341 LETTER
342 ROW
343 INGTON
344 IT
345 PO
346 ESUS
347 INT
348 CAR
349 YN
350 OMAS
351 GL
352 MSS
353 MS
354 AH
355 MY
356 WARD
357 AVE
358 DR
359 ASH
360 HER
361 ISC
362 ONS
363 OUGH
364 TW
365 HIST
366 ERY
367 AMP
368 ORN
369 HEN
370 SECT
371 ISE
372 ATES
373 FOR
374 ENERAL
375 TY
376 EP
377 ADY
378 TERS
379 ALE
380 VE
381 VOL
382 AM
383 VOL
384 SEC
385 ORTH
386 TA
387 AIR
388 MON
389 FOOTNOTES
390 PRE
391 INS
392 SM
393 NO
394 UND
395 ANK
396 JESUS
397 ITE
398 ME
399 ALL
400 IND
401 LIFE
402 ANN
403 VEN
404 WE
405 OUSE
406 FIRST
407 CR
408 ORS
409 ICT
410 IGN
411 BAR
412 UTH
413 ART
414 OSE
415 PART
416 LONDON
417 GEORGE
418 UBL
419 PRO
420 SS
421 AME
422 JAMES
423 IER
424 IDE
425 CAP
426 ENN
427 BUT
428 FORD
429 ATURE
430 OUND
431 FL
432 CHARLES
433 HENRY
434 PART
435 MA
436 OPE
437 ATT
438 URN
439 APH
440 OVE
441 LA
442 IRD
443 FE
444 THOMAS
445 ILE
446 SER
447 PER
448 LI
449 WAR
450 ANE
451 CONT
452 HISTORY
453 DEAR
454 OLOG
455 CHAR
456 BOOK
457 THIS
458 PRES
459 COMP
460 JOHN
461 IZ
462 OTT
463 OE
464 NE
465 AG
466 ENC
467 THAT
468 MRS
469 INA
470 OUT
471 BL
472 OLL
473 UP
474 SCENE
475 YS
476 PHIL
477 IELD
478 FACE
479 AIL
480 DO
481 CO
482 LECT
483 LX
484 EM
485 ICA
486 PAR
487 ATER
488 MRS
489 WORK
490 ATHER
491 ICAN
492 MOR
493 ILD
494 COM
495 YR
496 ULL
497 IM
498 FRI
499 KS
500 AMER
501 BRO
502 PRE
503 END
504 MISS
505 IVER
506 VIL
507 LXX
508 ADE
509 AF
510 SAL
511 APP
512 REET
513 ET
514 COR
515 KING
516 PE
517 STR
518 MEN
519 NOW
520 ISON
521 OLD
522 ESP
523 OPH
524 UCTION
525 URCH
526 AUL
527 ACT
528 ALS
529 MO
530 ENG
531 ATED
532 OUNG
533 WELL
534 NO
535 LIN
536 CONT
537 VENT
538 ICHARD
539 VS
540 ONE
541 DAY
542 SO
543 GL
544 SHA
545 APP
546 NER
547 UP
548 GREAT
549 IDENT
550 DER
551 RET
552 IET
553 NEY
554 DAY
555 IOUS
556 ISM
557 AZ
558 YORK
559 SOU
560 WAL
561 FL
562 FER
563 ASHINGTON
564 REC
565 SCENE
566 SECOND
567 SCH
568 GOD
569 ETT
570 HED
571 OOL
572 REAT
573 ELS
574 REL
575 LED
576 YOU
577 ESC
578 ASE
579 WOR
580 ROBERT
581 URES
582 EAR
583 NESS
584 FRIEND
585 PAGE
586 YM
587 SHE
588 TROD
589 TAIN
590 GU
591 SERMON
592 UEL
593 RIPT
594 SW
595 COUNT
596 LORD
597 SEC
598 CHA
599 XL
600 RON
601 HOL
602 CHURCH
603 SU
604 TIM
605 EP
606 OSS
607 STR
608 BLE
609 ILY
610 ITS
611 LL
612 ICLE
613 AC
614 STREET
615 HOUSE
616 IDGE
617 USTR
618 GER
619 NA
620 ANY
621 ARE
622 NEW
623 CAL
624 ITED
625 ITTLE
626 ANDER
627 BO
628 UTION
629 NOTE
630 AUTH
631 PUBL
632 YOUNG
633 HOW
634 KEY
635 TERN
636 INCE
637 OB
638 RAPH
639 KER
640 IFIC
641 TWO
642 MD
643 GENERAL
644 THis
645 SW
646 ARR
647 LAW
648 OMAN
649 YL
650 USS
651 CA
652 REV
653 DU
654 USE
655 BUR
656 ILL
657 EDWARD
658 HON
659 WILL
660 CAN
661 UNT
662 MED
663 LADY
664 ALD
665 ACTER
666 COMM
667 OGRAPH
668 SECTION
669 ECH
670 DEC
671 DAV
672 POL
673 IANS
674 ERSON