SlowGuess/NAACL_2022_OCR huggingface dataset:
Split Type	SpaCy	BERT-sentence	BERT-token	GPT-J	Control
Aggregate across 26 letters
F1	4.4354	2.9588	3.7989	2.724	4.3973
Best performing ones
s	0.6947	1.2941	0.4853	0.6514	5.5055
y	1.8665	1.6406	0.5697	1.4251	3.2417
e	0.6645	0.8544	0.3245	0.3233	1.8349
t	0.2643	3.4695	0.9129	0.5924	1.7645
p	6.1928	1.1628	0.5669	0.2985	3.7013
i	0.512	1.4392	0.5998	0.4867	5.5685
Worst performing ones
w	4.9794	2.2996	1.9614	1.9536	1.7453
q	2.7071	3.4438	4.5954	4.7932	5.5068
k	2.9332	6.885	2.0885	1.6864	1.6311
o	6.24	1.6009	1.0449	0.463	3.5961
b	4.0455	1.5597	1.4074	2.0701	2.7857
m	7.2995	2.4854	2.1762	1.0948	6.152
Table 15: Standard Deviation of POS/NER labels
implementation is done using PyTorch and Huggingface. Finally for the best hyperparameter, we perform 5 runs with different train/test splits and seeds. Our MLP model is 3-layered with SELU and Tanh activation and 0.1 Dropout before the last layer.
Tables 14 and 15 show the mean and variance of the results over the 4 taggers and control task. We also show the performance over the best-performing and worst-performing characters.
Appendix E Variability of Tokenization
E.1 Quantifying variability in the Pile Corpus
To quantify the variability in the tokenization of frequent words in corpora comparable to the corpora used to train these models, we consider 1/6th of the publicly available Pile Corpus used to train GPT-J (250 GB of text). For our analysis we consider 500 frequent words of  $8+$  characters (as measured using Google Ngrams) since long words are more likely to be the source of variability.
For each target word, we first case-insensively detect each of its occurrences in the sub-corpus. In order to also account for spelling errors, we used case-insensitive fuzzy search, allowing matches for substrings up to 1 Levenshtein distance away. Over these occurrences, we discard those where the substring is part of a bigger word, such as 'differentiation' for the target word 'different' or if the fuzzy match has whitespaces.
Once we have such occurrences, we want to obtain the tokenization of the target word in the context. For each word in the set of matches, if the matched substring ends with a non-valid character for our probing task, we delete the final character.
This allows for matches of [somethin', somethin", somethin] all to be considered as the string 'somethin'. We also account for the factors that leads to differing tokenization, such as preceding whitespaces.
Now, for each of the target words, we have a list of probable tokenization at most 1 Levenshtein distance away. Since two target words such as 'projection' and 'protection' could themselves be at 1 Levenshtein distance, these may act as what we call "pseudo matches" for each other. So we consider only one of these two from our target list, leading to 466 word down from 500 words. Now, for each of these target words, we count the number of possible unique tokenizations.
For each of these 466 target words, we also obtain a list of words from WordNet, which are 1 Levenshtein distance away. We treat this word list as the pseudo-match list. We also consider the number of tokenizations for each target word, excluding their pseudo-match list as well as by excluding all those which are equally close to or closer to a word in the pseudo-match list than they are to the target word. We also compute the statistics of those with exact matches.
Table 16 shows these statistics for the target words. On average, a target word is expected to have 213 different tokenizations depending on the context. We observe that, while one may expect the number of tokenizations to go up with the number of characters in the target word, it doesn't perfectly increase monotonically. This is because the number of occurrences of the target word dictates the number of tokenization it will have. Unsurprisingly, we see a consistent trend that the number
of tokenization greatly increases with increasing occurrences.
We observe three factors contributing to a remarkably large number of tokenizations. First, Case-Sensitive tokenization leads to up to 6 different tokenizations for each of the target words. Second, context-dependent tokenization increases the expected number of different tokenizations to 12.91. The rest of the tokenizations are likely due to misspellings or variants.
Our analyses were sped up using multiprocessing and fuzzy regex. To do so, we split the subcorpus across multiple pieces. These runs take about 3 days across 40 CPU Cores, 60 GB of RAM and less than 600GB hard disk space. We report the mean and standard deviation for the number of tokenizations a word has across the portion of the Pile corpus considered. These are also reported as a function of word length and its frequency of occurrence in the corpus.
Tables 16 and 17 shows these scores. The 'All matches' field considers the unique tokenizations of all matched substrings including those at 1 (case and whitespace insensitive) Levenshtein distance away. These word at 1 Levenshtein distance could be either misspellings or a different English word (for example an occurrence of the word 'projection' for target word 'protection'). The latter of these are identified using the Wordnet dictionary and the statistics recalculated and shown in the column 'Matches except pseudo'. Some of the misspellings contributing to this score could be misspellings of either the target word or of one of the other English words at 1 Levenshtein distance away ('prohesion' could be a misspelling of either 'projection' or 'protection' being at distance 1 from both). Such occurrences are removed, with statistics recomputed for the column 'Matches closer pseudo'. The column 'Exact contain' considers only those occurrences, which contain the exact target word (case-insensitive) in the string ignoring whitespaces. The 'Exact match' column does not consider occurrences involving a preceding whitespace.
Table 18 shows some examples of variation in tokenization.
E.2 Algorithm for increasing tokenization variability
Measure	All Matches	Matches Ex-cluding Pseudo	Matches Closer Pseudo	Exact Con-tain	Exact Match	Num Words
Aggregate	232.90	229.70	213.74	17.91	5.97	466
7 Length words	297.50	271.00	223.50	22.00	6.5	2
8 Length words	332.29	325.68	288.07	25.00	7.89	28
9 Length words	231.48	227.78	206.95	16.94	5.93	190
10 Length words	225.51	222.58	209.53	17.97	5.87	127
11 Length words	213.28	211.02	202.97	17.88	5.85	61
12 Length words	224.14	223.54	218.64	18.25	5.79	28
13 Length words	218.14	217.00	214.76	16.57	5.19	21
14 Length words	238.33	238.33	238.33	16.67	5.00	9
exp(12) occurrence	88.70	86.67	82.11	10.33	5.90	27
exp(13) occurrence	155.78	153.87	146.55	13.61	5.15	74
exp(14) occurrence	210.36	207.51	195.74	16.70	5.75	174
exp(15) occurrence	278.88	275.00	251.69	19.91	5.96	139
exp(16) occurrence	370.02	365.04	336.48	26.62	8.56	52
Table 16: Tokenization variance statistics - mean score.
Measure	All Matches	Matches Excluding Pseudo	Matches Closer Pseudo	Exact Contain	Exact Match
Aggregate	95.12	94.29	91.26	17.91	2.67
7 Length words	155.50	129.00	81.50	13.00	2.50
8 Length words	100.90	99.17	91.19	8.46	2.47
9 Length words	90.97	90.00	86.03	7.34	2.50
10 Length words	88.56	89.04	90.71	7.86	2.75
11 Length words	107.55	107.65	108.46	8.77	2.84
12 Length words	63.25	63.53	62.53	8.26	2.82
13 Length words	81.22	81.30	82.20	7.82	2.59
14 Length words	62.48	62.48	62.48	4.52	1.05
exp(12) occurrence	38.59	37.65	34.60	3.15	1.26
exp(13) occurrence	39.75	39.13	39.36	4.92	2.10
exp(14) occurrence	51.84	52.17	53.73	6.19	2.51
exp(15) occurrence	70.46	70.59	77.22	7.86	2.38
exp(16) occurrence	101.86	100.38	103.83	9.99	3.44
Table 17: Variability across target words in tokenization variance statistics.
String	Tokenization	String	Tokenization
signature	playstation
Exact match case insensitive	Exact match case insensitive
"SIGNATURE"	["SIGN", "ATURE"]	"playstation"	["play", "station"]
"sIGNATURE"	["s", "IGN", "ATURE"]	"PL", "ay", "ST", "ATION"]
"SigNature"	["S", "ig", "Nature"]	"playStation"	["play", "Station"]
"Signature"	["Sign", "ature"]	"PLAYSTATION"	["PLAY", "ST", "ATION"]
"S signature"	["SI", "gn", "ature"]	"Playstation"	["Play", "station"]
"signature"	["sign", "ature"]	"PlayStation"	["Play", "Station"]
Exact match and whitespaces	Exact match and whitespaces
" signature"	["Gsignature"]	"Playstation"	["GPL", "ay", "Station"]
" Signature"	["GSignature"]	"PLStation"	["GPL", "station"]
" SigNature"	["GSign", "Nature"]	"PLAYSTATION"	["GPL", "ST", "ATION"]
" signTure"	["Gsign", "a", "T", "ure"]	"PlayStation"	["GPlayTation"]
" SIGNATURE"	["GSIGN", "AT", "ure"]	"playStation"	["GPlayStation"]
" SiIGNATURE"	["GSi", "GN", "ATURE"]	"playTation"	["Gpl", "AY", "s", "Ta", "T", "ion"]
" SIGNATURE"	["GSIGN", "ATURE"]	"playStation"	["Gplay", "Station"]
" signAture"	["Gsign", "At", "ure"]	"playstation"	["Gplay", "station"]
" SIGNATURE"	["GSIGN", "ature"]	"PLStation"	["GPL", "ay", "station"]
" sSignature"	["Gs", "Ign", "ature"]	"PlaySTation"	["GPL", "ST", "ation"]
Fuzzy match and misspellings	Fuzzy match and misspellings
"SIGNATURE"	["S", "1", "GN", "ATURE"]	"Play-station"	["Play", "-", "station"]
" SIGNATURE"	["GSIGN", "AT", "UT", "RE"]	"PLAY-STATION"	["GPL", "-", "ST", "ATION"]
" signatyure"	["Gsign", "at", "y", "ure"]	"play-station"	["play", "-", "station"]
" signature"	["Gsign", "atre"]	"Play-station"	["GPL", "-", "station"]
" Signiature"	["Sign", "i", "ature"]	"play-station"	["Gplay", "-", "station"]
" signnature"	["Gsign", "nature"]	"Play-Station"	["Play", "-", "Station"]
" signature"	["Gsign", "at", "r", "re"]	"Playstation"	["Play", "t", "station"]
" signature"	["Gsign", "ature"]	"Playst4tion"	["GPL", "st", "4", "tion"]
" Sign(ature"	["GSign", "(", "ature"]	"PlayStatiOn"	["GPL", "St", "ati", "0", "n"]
" signnature"	["sign", "nature"]	"Play-Station"	["GPL", "-", "Station"]
"SIG(NATURE"	["S", "IG", ("", "NAT", "URE"]	"play-Station"	["Play", "st", "aton"]
" Si2nature"	["GSi", "2", "nature"]	"playStation"	["Gplay", ".", "Station"]
" Singnature"	["Sing", "nature"]	"playstaton"	["Gplay", "st", "aton"]
" signature"	["Gsign", "atu", "ure"]	"PLAYSTATION"	["GPL", "T", "ST", "ATION"]
" Signatures"	["GSign", "at", "urs"]	"playstation"	["play", "st", "ati", "om"]
" sigNUTure"	["Gsig", "N", "UT", "ure"]	"playsstation"	["plays", "station"]
Table 18: Some examples of variations in tokenization for two example words.