SlowGuess/NAACL_2022_OCR huggingface dataset:
We test whether pretrained models tend to store more spelling-related information in higher-frequency token types. We focus on RoBERTa-Large, and assign each token in the test set to its frequency quintile according to the number of times it appeared in the pretraining corpus – from the 10000 most frequent token types (top  $20\%$ ) to those ranked 40000-50000 in the vocabulary (bottom  $20\%$ ) – and measure the average performance of SpellingBee within each quintile. Figures 3 and 4 show the results with and without the similarity filter. We observe that SpellingBee is indeed able to extract more information from higher-frequency token types, suggesting that the pretrained model has more information about their character composition.
C Spelling Accuracy by Length
We analyze the effect of token length on the probe's ability to spell. A priori, it is reasonable to assume that it is easier for the probe to spell shorter tokens, since less information needs to be extracted from the embedding and there are less discrete decisions to be made while decoding. Indeed, Figure 5 shows that with the none filter most vocabulary
Figure 3: The EM scores of SpellingBee on RoBERTa-Large for each frequency quintile with the none filter (top) and the similarity filter (bottom).
Figure 4: The chrF scores of SpellingBee on RoBERTa-Large for each frequency quintile with the none filter (top) and the similarity filter (bottom).
tokens with 2-4 characters can be accurately reproduced from their vector representations, while longer tokens are harder to replicate. This trend is particularly sharp when the similarity filter is applied, as the probe is hardly able to spell tokens with 6 or more characters accurately; having said that, the probe is able to generate many partially correct spellings, as measured by chrF (Figure 6). Perhaps a less intuitive result is the probe's failure to spell single-character tokens. A closer look reveals that many of these examples are rare or non-alphanumeric characters (e.g.  $\varsigma$  and $), which are probably very difficult for the probe to generate if it had not seen them during training. While these results show strong trends with respect to length, token length is also highly correlated with frequency, and it is not necessarily clear which of the two factors has a stronger impact on the amount and resolution of character-level information stored in the embedding layer of pretrained models.
Figure 5: The EM scores of SpellingBee on RoBERTa-Large for each token length with the none filter (top) and the similarity filter (bottom). The rightmost column groups together tokens with length of 11 or above.
Figure 6: The chrF scores of SpellingBee on RoBERTa-Large for each token length with the none filter (top) and the similarity filter (bottom). The rightmost column groups together tokens with length of 11 or above.
D Manual Error Analysis
We manually analyze 100 random tokens that SpellingBee spelled incorrectly with the lemma filter to understand the nature of the spelling mistakes. Out of those 100 we display 20 mistakes in Table 4 alongside the spelling prediction of the control baseline. SpellingBee's mistakes vary from single-character typos to completely different words. Having said that, the vast majority of mistakes have significant overlap with the correct spelling, such as shared prefixes and capitalization.
We implement SpellingBee with a 6-layer encoder-decoder model, with 512 model dimensions. The model parameters are optimized with Adam (Kingma and Ba, 2015) for 1000 steps with up to 1024 tokens per batch, a learning rate of 5e-4, and a dropout rate of 0.1. These are the default hyperpa
Token	SpellingBee	Control
_Issa	_Asey	_ kinston
_Rhod	_Rob	_hoedn
Memory	Mathinge	_ entically
_metals	_metrys	_leaved
_Reed	_Redd	_fomparing
_break	_breach	_promoters
_summit	_mosump	Seasons
Catholic	Cravital	_tonversal
_cleanup	_lamed	(paclus
_Winner	_Womer	_purden
_LIM	_LUM	_Send
Copy	Cople	providers
VOICING	_relicing	_walking
_Stab	_Stamb	_hoviders
_356	_353	_budiance
find	wive	_malding
_Psychic	_Syptanc	joacter
_Looking	_Lowing	parging
_CLOSE	DEFIC	_tuldence
_prolific	_promistic	_complexement
Table 4: Sampled SpellingBee errors with the lemma filter alongside the control baseline's spelling for the same tokens. The underscore (_ ) represents a preceding whitespace.
rameters for training a transformer language model in Fairseq (Ott et al., 2019).