SlowGuess/NAACL_2022_OCR huggingface dataset:
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Bofang Li, Aleksandr Drozd, Tao Liu, and Xiaoyong Du. 2018. Subword-level composition functions for learning word embeddings. In Proceedings of the second workshop on subword/character level models, pages 38-48.
Jindrich Libovicky, Helmut Schmid, and Alexander Fraser. 2021. Why don't people use character-level machine translation? arXiv preprint arXiv:2110.08191.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Hans Marchand. 1959. Phonetic symbolism in english wordformation. Indogermanische Forschungen, 64:146.
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Galle, Arun Raja, Chenglei Si, Wilson Y Lee, Benoit Sagot, et al. 2021. Between words and characters: A brief history of
open-vocabulary modeling and tokenization in nlp. arXiv preprint arXiv:2112.10508.
Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975-4989, Florence, Italy. Association for Computational Linguistics.
Sabrina J Mielke and Jason Eisner. 2019. Spell once, summon anywhere: A two-level open-vocabulary language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6843-6850.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space.
Padraig Monaghan, Nick Chater, and Morten H. Christiansen. 2005. The differential role of phonological and distributional cues in grammatical categorisation. Cognition, 96(2):143-182.
Padraic Monaghan, Richard C. Shillcock, Morten H. Christiansen, and Simon Kirby. 2014. How arbitrary is language. Philosophical Transactions of the Royal Society B.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Tiago Pimentel, Arya D. McCarthy, Damian Blasi, Brian Roark, and Ryan Cotterell. 2019. Meaning to form: Measuring systematicity as information. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1751-1764, Florence, Italy. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computational Linguistics.
Yuval Pinter. 2021. Integrating approaches to word representation. arXiv preprint arXiv:2109.04876.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882-1892, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Arij Riabi, Benoit Sagot, and Djamé Seddah. 2021. Can character-based language models improve downstream task performances in low-resource and noisy language scenarios? In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 423-436, Online. Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Jose Carlos Rosales Nunez, Guillaume Wisniewski, and Djamé Seddah. 2021. Noisy UGC translation at the character level: Revisiting open-vocabulary capabilities and robustness of char-based models. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 199-211, Online. Association for Computational Linguistics.
Josh Rozner, Christopher Potts, and Kyle Mahowald. 2021. Decrypting cryptic crosswords: Semantically complex wordplay puzzles as a target for nlp. In Advances in Neural Information Processing Systems, volume 34, pages 11409-11421. Curran Associates, Inc.
Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
F. de Saussure. 1916. Course in general linguistics. Open Court Publishing Company.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152. IEEE.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. Bert is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47-55.
Monica Tamariz. 2008. Exploring systematicity between phonological and context-cooccurrence representations of the mental lexicon. The Mental Lexicon, 3(2):259-278.
Elena Voita, Rico Sennrich, and Ivan Titov. 2021. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1126-1140, Online. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Appendix A Code details
We release our code anonymously at https://github.com/ayushk4/ character-probing-pytorch under MIT License.
The models weights, data and other dependencies required for experiment are at https://github.com/ayushk4/character-probing-pytorch/releases.
The intended use of our code is for academic research. We consider probing publicly available PLMs, which are made available for research as well as end use cases, to be within the intended use of PLMs.
Appendix B Probing for Character Information
We use off-the-shelf APIs for lemmatization and WordNet from NLTK (Apache License 2.0; Bird et al., 2009). Our implementation uses PyTorch (BSD License; Paszke et al., 2019), HuggingFace (Apache License 2.0; Wolf et al., 2019) and custom APIs for GPT-J's embedding.
The probes for each MLP are trained separately starting with random initialization weights. We train the probe via a binary classification task via backpropagation, using the Adam optimizer (Kingma and Ba, 2015) with betas of 0.9 & 0.999 and epsilon of 1e-08 without weight decay, over the standard Binary Cross Entropy loss across the predicted logits  $\hat{y}_i$  and ground truth logits  $y_i$ .
B.1 PLMs considered
Details of the PLMs used along with their modelcard on Huggingface:
- GPT-J: We used the standard GPT-J with 6 Billion parameters and its reversible Byte-Pair encoding based subword tokenizer. We extracted the embeddings and have released it separately. Model Card: 'EleutherAI/gpt-j-6B' under Apache 2.0 License.
- GPT-2: We consider the base model for GPT-2 with 124 Million parameters. The tokenizer used in this model is the exact same as the one used in GPT-3 and is also a subword tokenizer based on reversible Byte-Pair encoding. Model Card: 'gpt2' under Modified MIT License.
- RoBERTa: We again use the Base model for fairer comparison to the GPT-2 model with 125 Million parameters. This model has partially reversible Byte-Pair Encoding based on GPT-2's byte-pair tokenizer but with additional tokens for a BERT-like MLM discriminative pre-training. Model Card: 'roberta-base' under MIT License
- BERT: The BERT-base models have roughly 110 Million parameters. Both the Uncased and Cased versions of this model are considered with their Word-Piece tokenizers. For this tokenizer, we also consider the character ‘##’ while filtering out vocabulary, as it denotes the token continues on the preceding
Model type	Case-Sensitive
PLM	Control
GPT-J	94.35	52.76
GPT-2	84.69	51.05
RoBERTa	83.87	49.00
BERT-Cased	78.47	45.35
BERT-Uncased	77.48	49.37
GloVe 300D	69.40	49.40
GloVe 100D	61.56	49.55
LXMERT	60.30	49.61
Table 6: Results for the main probing experiment, across models.
Model	Case-insensitive	Case-Insensitive
PLM	Control	PLM	Control
GPT-J	0.83	3.12	1.39	2.27
GPT-2	2.01	3.09	2.21	2.75
RoBERTa	2.27	3.13	2.79	2.46
BERT-Cased	2.93	7.46	2.77	5.67
BERT-Uncased	3.32	4.33	3.32	4.33
word. Model Card: 'bert-base-uncased', 'bert-base-cased' under Apache 2.0 License
- GloVe: We experiment with the 100 and 300 dim version of 400K-Vocab GloVe trained on 6B tokens. We consider the 40k most frequent tokens in GloVe, comparable to the vocabulary sizes of the other models. GloVe version used: 'Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download): glove.6B.zip'
- LXMERT: We use the uncased version of LXMERT-base model and, as with the BERT model, filter out '#' preceding symbols. Model Card: 'unc-nlp/lxmert-base-uncased' under
Table 7: Standard deviation in our probing Experiment 1, for the key models considered.
Property	Statistics
Dataset	Tokenizer's VOC for each model
Data-filtered	Tokens having only letters (a-z,A-Z)
GPTs, RoBERTa: Allow preceding G
BERT: Allow preceding '#'
Train-Test split	80-20
Preprocessing	None
Output labels	26 tasks (each with binary label)
Link	Model Card & links in §B.1
Table 8: Dataset Checklist for Experiment 1.
Figure 4: Experiment 2: syntax baselines with BERT-sentence and BERT-token custom taggers.
B.2 Hyperparameter and other Details
Each probe is trained for 5 epochs, with 128 batch-size. The Learning rate is tuned over averaged Macro-F1 in the grid  $\{1e - 5,3e - 5,5e - 5,1e - 4,3e - 4,1e - 3,3e - 3,1e - 2,3e - 2\}$ . We trained the probe on the best hyperparameter settings across 5 different train-test splits and seeds. Table 9 shows the best learning rates and the number of parameters (and frozen-parameters) in the probe. For all the control embedding, we assume the same dimension as the largest model (4096) and considered a maximum vocab of  $100\mathrm{k}$ , even though only the first few thousand might be used. These experiments take less than 20 minutes for each run and require less than  $12\mathrm{GB}$  of GPU memory. They were run on a mix of NVidia Tesla K80, GTX 1080 Ti, P100, V100 GPUs with Dell R740 and Intel Xeon CPUs.
Table 6 shows the result of the probe in a case-sensitive setting. The case-insensitive probe treats both "Cat" and "cat" both as a hit for "c". The case-sensitive probe treats only "cat" (not "Cat") as a hit for "c". Note that performance is the same for BERT-Uncased since it does not distinguish between these conditions.
Appendix C Multilingual Analyses
Model Details: We only consider mBART (Liu et al., 2020) with 610M parameters and 250k vocab size. Its model card in Huggingface is
3Accessible at nlp.stanford.edu/projects/glove/, Apache v2.0 License
Model Probe	Case-insensitive	Case-Sensitive
Lemma	Control	Lemma	Control
LR	# Params	LR	# Params	LR	# Params	LR	# Params
GPT-J	1e-4	240M (206M)	1e-4	443M (410M)	1e-4	240M (206M)	3e-4	443M (410M)
GPT2	3e-4	40M (39M)	1e-4	443M (410M)	3e-4	40M (39M)	3e-4	443M (410M)
RoBERTa	3e-4	40M (39M)	1e-4	443M (410M)	1e-3	40M (39M)	1e-2	443M (410M)
BERT-cased	1e-3	23M (22M)	3e-3	443M (410M)	1e-3	23M (22M)	5e-5	443M (410M)
BERT-uncased	3e-3	25M (23M)	3e-4	443M (410M)	3e-4	25M (23M)	1e-4	443M (410M)
LXMERT	1e-4	24M (23M)	3e-4	443M (410M)	3e-4	24M (23M)	1e-4	443M (410M)
GloVe 100D	1e-4	4.02M (4.00M)	3e-4	12.2M (12.0M)	3e-4	4.02M (4.00M)	3e-4	12.2M (12.0M)
GloVe 300D	3e-4	12.2M (12.0M)	1e-4	12.2M (12.0M)	3e-4	12.2M (12.0M)	3e-5	12.2M (12.0M)
Table 9: Experiment 1 hyperparameters.
Property	Statistics
Train Sentences	14986
Train Tokens	219553
Valid Sentences	3465
Valid Tokens	55043
Test Sentences	3683
Test Tokens	50349
NER Tags	5
PoS Tags	45
Preprocessing	None
Link	github: davidsbatista/NER-datasets
'facebook/mbart-large-cc25', without any mention of its license. Its tokenizer is a reversible one, similar to GPT, except that it encodes preceding space with '\_'.
Languages: For the non-Latin scripts considered, we only consider those characters with more than 250 occurrences in the tokenizer's vocabulary. We consider the experiment case-insensitive (by lower-casing the string) across scripts that have lowercase and uppercase characters.
Hyperparameters: Each probe is trained for 5 epochs, with 128 batch-size. The learning rate is tuned over averaged Macro-F1 in the grid  $\{1e - 5,3e - 5,5e - 5,1e - 4,3e - 4,1e - 3,3e - 3,1e - 2,3e - 2\}$ . We trained the probe on the best hyperparameter settings across 5 different train-test splits and seeds. Table 12 shows these best learning rates and the number of parameters (and frozen parameters) in the probe. For all the control embedding, we assume the same dimension as the largest model (1024) and considered a maximum vocab of  $300\mathrm{k}$ , even though only a few thousand are used. These experiments take less than 20 minutes for each run requiring less than 12 GB of GPU memory and were run on a mix of NVidia Tesla K80, GTX 1080 Ti, P100, V100 GPUs with Dell R740 and Intel Xeon CPUs.
Table 10: Dataset Checklist for training POS/NER CoNLL set.
Script	PLM	Control
Latin (English chars)	3.28	7.21
Devanagari	6.58	5.43
Arabic	10.50	2.99
Cyrillic	3.79	5.31
Table 11: Standard Deviation for Multilingual BART experiment.
Appendix D Syntax Baseline for Character information
D.1 Custom syntax taggers
First we consider an off-the-shelf SpaCy model with 3 features for each token: NER, PoS, and Coarse-Grained PoS tags. Before running this model, we remove the preceding whitespace characters in the token, if present. The resultant features are discrete one-hot feature vectors over labels. The SpaCy tagger is not perfectly suited to our task since it operates at the word level, whereas we are concerned with obtaining a subword token's embeddings. To solve that problem, we also built 3 custom taggers for obtaining PoS and NER labels on subword tokens. These taggers take (a subword) token's model embedding as input and output a vector of probabilities over part of speech and named entity categories.
To build our custom GPT-J-Tagger, we train an MLP to predict PoS and NER label based on GPT-J's static embedding layer for each token. The tagger is trained on the CoNLL 2003 dataset's train and evaluation splits (Sang and De Meulder, 2003), which contains part of speech and named entity information. Unlike the SpaCy tagger, our custom GPT-J-Tagger outputs a probability distribution over categories so we can use this distribution over labels as the vector of interest, rather than a one-hot vector.
Table 13 show the performance of the tagger's performance qua tagger. Table 10 shows the Dataset Checklist for this experiment. To build
Script	PLM	Control
LR	# Params	LR	# Params
Latin	3e-4	258M (256M)	1e-2	309M (307M)
Devanagari	3e-4	258M (256M)	1e-3	309M (307M)
Arabic	3e-4	258M (256M)	3e-3	309M (307M)
Cyrillic	3e-4	258M (256M)	3e-4	309M (307M)
Table 12: Multilingual Hyperparameters. Number of parameters (with frozen parameters in parenthesis) is denoted per probe.
Model Type	# Epochs	Batch Size	LR	Dev F1Wtd	Dev F1Macro	Test F1Wtd	Test F1Macro
BERT-sentence (PoS)	10	32	1e-5	98.17	94.80	93.42	87.40
BERT-token (PoS)	10	32	1e-5	76.42	56.75	77.24	56.74
GPT-J MLP (PoS)	20	64	1e-4	62.90	68.72	60.15	69.14
BERT-sentence (NER)	10	32	1e-5	97.88	93.18	96.02	86.92
BERT-token (NER)	10	32	1e-5	83.50	56.97	81.57	54.88
GPT-J MLP (NER)	20	64	5e-5	85.59	63.56	82.71	57.34
Table 13: Labels from POS/NER labels. LR denotes learning rate
Split Type	SpaCy	BERT-sentence	BERT-token	GPT-J	Control
Aggregate across 26 characters
F1	52.338	55.008	59.7525	61.2395	49.6772
Best performing ones
s	64.5967	60.7179	70.3299	66.8159	40.3154
y	61.9632	60.3871	67.1591	64.8863	48.6838
e	62.0518	57.7531	64.6152	62.3213	47.2712
t	60.6848	54.3826	64.0681	60.7345	48.4873
p	50.235	55.2361	63.9658	60.5067	46.5612
i	60.8024	56.4055	63.3518	61.6032	42.8155
Worst performing ones
w	45.748	52.7235	57.6919	58.2666	48.6947
q	43.7924	56.5274	57.5407	53.5437	49.2841
k	47.7873	49.3832	57.3084	55.9559	46.2371
o	52.9403	53.6138	56.8312	55.6293	43.5871
b	48.9159	56.739	56.3873	55.1265	48.252
m	48.1349	53.4036	56.2846	55.6094	46.1084
Table 14: Syntax baseline: Probing over syntax label distribution.
the BERT sequence-labeling tagger, we fine-tuned a BERT sequence labeling model for the PoS and NER tasks, in order to output a label for each (subword) token in a sentence. When extracting syntactic features for this model, we first do the same pre-processing of removing the special preceding whitespace of GPT's tokens as SpaCy before input into the BERT model. Since BERT's tokenizer could have more than one token for a single GPT-J's token, we consider the average of the logits as the pre-softmax feature vector.
In addition to the BERT sentence-level tagger, we consider a BERT token classifier model finetuned for NER and PoS at token level rather than at sentence level. This token-level model does not leverage context to deduce the label, and is closer to how we use this model later to get features for predicting NER/PoS features.
D.2 Results and Hyperparameters
We use off-the-shelf APIs for lemmatization and WordNet from NLTK. Our implementation uses PyTorch (Paszke et al., 2019), HuggingFace (Wolf et al., 2019) and custom APIs (now released) for GPT-J's embedding. The hyperparameter tuning was done on the dev set for only the learning rate in the grid  $\{1e - 5,3e - 5,1e - 4\}$  for BERT and  $\{1e - 5,3e - 5,5e - 5,1e - 4,3e - 4,1e - 3,3e - 3,1e - 2,3e - 2\}$  for GPT-J. Our MLP model is 3-layered with SELU and Tanh activation and 0.1 Dropout before the last layer. Our BERT-Model is initialized with 'bert-base-cased' from Huggingface with default values of hyperparameters. Our implementation was done using PyTorch and optimized via Adam with betas of 0.9 & 0.999 and epsilon of 1e-08 without weight decay over the standard Cross Entropy loss. We set the batch size to 32 sentences for BERT and 64 for GPT-J. All the experiments can be done within 16GB of GPU memory and no run individually takes more than 2 hours. We release these models along with our codebase with instructions to run them.
Table 13 shows the performance of these NER and PoS models. As expected, the BERT-sentence model performs the best on both the tasks as it leverages the context while tagging. GPT-J slightly outperforms BERT-token on both the tasks. Note that these performances are not comparable as their tokenizations differ and only one of the models leverages context to predict NER and PoS tags.
D.3 Method
Assume we have  $m$  syntactic features. Consider the tokenizer Vocabulary  $V$  (with only alphabetic tokens) and the  $D_{\alpha}$  datapoint pairs for each letter  $\alpha$  of the lowercased English alphabet. For each token-label pair  $(w_i, y_i)$ , we obtain the  $m$  syntactic features of the word  $\{x_i^{(1)}, x_i^{(2)} \ldots x_i^{(m)}\}$  using the trained models to tag the features.
We train a classifier to predict whether a character  $\alpha$  is present in the token  $w_{i}$  using only its syntactic features. Assume randomly initialized 'trainable' embeddings  $\{E_1,E_2\ldots E_m\}$  for each of the  $m$  syntactic features. We predict the logits  $\hat{y}_i$  for token  $w_{i}$  over each letter  $\alpha$  using an MLP classifier over the embeddings:
$$
\hat {y} _ {i} = \sigma (M L P _ {\alpha} ([ E _ {1} ^ {T} x _ {i} ^ {(1)}; \dots ; E _ {m} ^ {T} x _ {i} ^ {(m)} ]))
$$
Each syntactic feature  $x_{i}^{(j)}$  is a vector denoting probability distribution of a token over the corresponding feature labels (including being a one-hot vector), this is crucial because a token (especially subword-token) might have different labels depending on the context.
We train different MLPs and Embeddings from scratch for each alphabet  $\alpha$  with no shared parameters across the (case-insensitive) 26 English characters. We train our model for binary classification via backpropagation over the standard Binary Cross Entropy loss across the predicted logits  $\hat{y}_i$  and ground truth logits  $y_i$ .
As before, for each character we create a balanced dataset consisting of an equal number of positive and negative examples, where each example is made up entirely of either English characters or whitespace. These are randomly divided into training and test split such that we keep words with the same lemmas in the same split. As a control task, we randomly assign the syntactic features for each token. We set the batch size for runs with one-hot vectors as features to 128 and to 64 for others, the learning rate is tuned in  $\{1e - 5,3e - 5,1e - 4,3e - 4,1e - 3,3e - 3,1e - 2\}$  for all the features over the metric of Averaged F1-Scores across the 26 English letters. The best learning rates for SpaCy, BERT-sentence, BERT-token, GPT-J and Control were found to be 1e-3, 1e-3, 3e-3, 1e-4, 1e-2, respectively. Using Adam Optimizer we train each of the 26 models for 5 epochs with betas of 0.9 & 0.999 and epsilon of 1e-8. Our