Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?
Authors: Ahmed Alajrami
Katerina Margatina
Nikolaos Aletras
Venue: EMNLP 2023 Main (2023)
Keywords: LLMs
Interpretability
Pretraining
TL;DR: We examine how different subsets of input token characters affect the performance of pre-trained language models

Abstract: Understanding how and what pre-trained language models (PLMs) learn about language is an open challenge in natural language processing. Previous work has focused on identifying whether they capture semantic and syntactic information, and how the data or the pre-training objective affects their performance. However, to the best of our knowledge, no previous work has specifically examined how information loss in input token characters affects the performance of PLMs. In this study, we address this gap by pre-training language models using small subsets of characters from individual tokens. Surprisingly, we find that pre-training even under extreme settings, i.e. using only one character of each token, the performance retention in standard NLU benchmarks and probing tasks compared to full-token models is high. For instance, a model pre-trained only on single first characters from tokens achieves performance retention of approximately 90% and 77% of the full-token model in SuperGLUE and GLUE tasks, respectively.

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?
Ahmed Alajrami Katerina Margatina Nikolaos Aletras
Department of Computer Science
University of Sheffield, UK
(ajsalaframi,, x marrgatina, a.aletras) het.erasssssteeeeeddaa........
Abstract
Reading all letters vs. only the first three letters
Size and Distance
Size and Distance
Understanding how and what pre-trained lan- guage models (PLMs) learn about language is an open challenge in natural language process- ing. Previous work has focused on identifying whether they capture semantic and syntactic in- formation, and how the data or the pre-training objective affects their performance. However, to the best of our knowledge, no previous work has specifically examined how information loss in input token characters affects the perfor- mance of PLMs. In this study, we address this gap by pre-training language models us- ing small subsets of characters from individual tokens. Surprisingly, we find that pre-traue pre-training even under extreme settings, i.e. using only one character of each token, the performance reten- tion in standard NLU benchmarks and probing tasks compared to toll-token models high. For instance, model pre-trained only on sin- gle first characters from tokens achieves per- formance retention of approximately 90% and 77% of the full-token model in SuperGLUE and GLUE tasks, respectively.'
With a radius 2,106 miles (5 kllomttrs), Mars
With a radius 2,106 miles 3.390 Allomotans),
size
Earth, nickel, Mara would about respberry.
nickel, Mara would be about as big respbery.
Figure 1: Humans able understand natural lan- guage text consisting of words with altered letters (e.g.- part bold, or entiely msig letters). Can language models do the same?
PLMs (Kaplan et al., 2020). Previous studies have mainly focused on exploring how its size affects the performance (Liu et al., 2019b; Yang et al., 2019; Baevski et al., 2019), demonstrating the benefits of using more data together with a larger num- ber of parameters. More recent work has focused studying the relationship between PLMs per- formance and the quality and relevance of the pre- training data (Zhang et al., 2020; Raffel al., 2020; Lee al., 2022; Micallef et al., 2022; Krishna al., 2022; Madasu and Srivastava, 2022). showing that high-quality data leads to improved performance. Interestingly, contrary to these findings, other stud- ies showed that language models can still achieve comparable performance even when they are pre- trained using randomly selected character n-grams (Krishna et al., 2021) or pre-trained using synthetic tasks (Wu et al., 2022).
1 Introduction
The use of pre-trained language models (PLMs), such as BERT (Devlin et al., 2019), T5 (Raffel et al., 2020), GPT-3 (Brown et al., 2020) and many more, has led to significant advances in natural language processing (NLP) tasks, with many achieving state- of-the-art results. However, the exact mechanisms by which these models learn and represent lan- guage are still not fully understood.
However, no previous studies have so far ex- plored the amount of information that PLMs actu- ally need from individual tokens in the pre-training data. Motivated by this, we pre- pre-train language mod- els by introducing partial information loss in the token sequence. This is achieved by using subsets of characters to represent each token in the pre- training data, i.e. keeping only one, two or three characters at a time (Figure 1). To the best of our knowledge, this study is the first to explore how in- formative subsets of tokens are compared to using
The pre-training data evidently plays an im- portant role in the downstream performance of
'Code and models available here: https: /hith ub.com com/aa jrami /mmn p200p- co00o22tttooo-ccaraceeee 5 s-rcle
"We may interchangeably use terms such as 'learn', 'com- prehend' or 'understand" to refer to the ability of PLMs to understand language. These terms are used the context of nnturral language understanding" and 'machine reading com- prehension' as downstrcam tasks, highlighting the capabilities of such models to perform well.

---

full tokens in language models pre-training. Specifically, we draw inspiration from studies in cognitive science and psycholinguistics that have shown that humans possess remarkable flexibility in processing partial word information, such as comprchending text by only reading the first or last characters of each word (McCusker et al., 1981; Rayner et al., 2006; Grainger al., 2004; Johnson and Eisler, 2012). This intriguing ability raises the question of whether computational models can exhibit similar behavior, bringing forth novel av- enues for NLP research. In this paper, we explore the parallels between human reading strategies and language models when presented with partial word information, aiming to uncover common principles and differences underlying both systems.
Pre-training Objectives Another line of work has focused on exploring how pre-training objec- tives affect the performance of PLMs. Saunshi et al. (2020) demonstrated that PLMs pre-trained on masked language modeling (MLM) perform well in downstream tasks due to their ability to learn semantic dependencies between tokens. How- ever, many different pre-training objectives are ef- fective in downstream tasks (Yang al., 2019; Clark et al., 2020; Lan et al., 2020; Di Liello al., 2022). In addition, studies examined how differ- ent types of pre-training tasks help PLMs Iearn, showing that even objectives that are not intuitive for humans, such as tokens grouped in arbitrary random classes, can still help PLMs achieve good performance on downstream tasks and acquire lin- guistic information (Yamaguchi et al., 2021; Ala- jrami and Aletras, 2022). Furthermore, a recent study by Yamaguchi al. (2023) has shown that the performance of PLMs on downstream tasks is significantly influenced by the pre- pre-training objec- tive complexity.
Our main contributions are follows:
1. We empirically demonstrate that even when pre-training using one character from the in- put tokens, PLMs can still effectively learn;
2. We find that the first characters of tokens and consonants are more informative than the last characters and vowels, respectively:
Pre-training Data Closer to our work are studies on investigating the learning process of PLMs by pre-training them on synthetic data. Sinha et al. (2021) found that PLMSs pre-trained on sentences with shuffled word order still perform well on downstream tasks, likely due to their ability to cap- ture higher-order co-occurrence statistics. Krishna et al. (2021) used corpus of randomly selected character n-grams pre-train models on synthetic summarization tasks. They found that the perfor- mance of these models was nearly equivalent to models pre-trained on real data. Ri and Tsuruoka (2022) demonstrated that pre-training on artificial languages that mimic the structural properties of natural language can still result in knowledge trans- ferable to natural language. Even when pre-trained non-linguistic data such music or code, lan- guage models can still perform well on natural language downstream tasks (Pupadimitriou and Ju- rafsky. 2020; Chiang and Lee, 2020). Similarly, hashing semantically unrelated tokens together in the same embedding results in negligible perfor- mance loss (Xue and Aletras, 2022).
3. We show through probing that specific charac- ter positions of tokens assist PLMs in encod- ing better linguistic information.
2 Related Work
Recent efforts in NLP focus on understanding how and what PLMs learn about language including: what information they encode; and how the pre- training objective and data affect what PLMs learn.
Probing for linguistic information A popular area of research is on detecting types of linguistic properties that PLMs encode and to what extent via probing. Typically, model is trained using the representations of language model to predict particular linguistic feature. High accuracy in such a task indicates that the LM effectively encodes that linguistic property (Adi et al., 2016; Hupkes et al., 80; Conneau et al., 2018).
Probing has shown that PLMs encode linguis- tic information, such as syntactic and semantic, in their representations (Liu et al., 2019a; Tenney al., 2019; Lin al., 2019; Rosa and Mareček, 2019; Hewitt and Manning, 2019; Jawahar al., 2019; Kim et al., 2019; Vilares al., 2020; Lim- isiewicz and Mareček, 2021; Müller-Eberstein et al., 2022; Lasri et al., 2022; Davis et al., 2022).
Unlike previous work, we explore how much information loss affects the performance of PLMs inspired by human reading comprehension (Fry, 1968; McCusker et al., 1981; Rayner et al., 2006; Grainger et al., 2004; Johnson and Eisler, 2012).

---

3 Models
Vowels (V) We further expand our investigation by pre-training models where we exclusively retain the vowel characters (if they exist) from each input token. Through this approach, we explore the ca- pacity of PLMs to acquire and preserve meaningful information solely based on vowel characters.
Various studies in cognitive science have attempted to measure the ability of humans to process words and understand sentences when retaining only some of the characters and their relative position in each word while others are masked, i.e. ortho- graphic priming (Fry, 1968; McCusker al., 1981; Grainger et al., 2004; Johnson and Eisler, 2012).
Consonants (C) Finally, to comprehensively plore the impact of different character subsets on pre-training, we also delve into the pre-training of models by exclusively utilizing the consonant characters from each input token.
Inspired by this, we aim to explore how much information PLMs need from individual tokens by pre-training using only small subset of characters in each token. We experiment using three different input token subsets (i.e. one, two and three char- acters) and three relative positions of characters in the token (i.e. first, middle and last). We also experiment with retaining only consonant vowel characters from each input token. Table shows an overview of the inputs to different models.
3.2 Architecture and Pre-training
Our main goal to measure the performance re- tention of different inputs compared to using a full token. For each different input type, we pre-train a BERT-BASE (Devlin et al., 2019) model by modi- fying the embedding matrix according to the var- ious vocabularies. We experiment with two pre- training MLM tasks:
3.1 Input
Single Character First, we experiment with pre- training under extreme settings where would be nearly impossible for humans to guess the con- stituent tokens of a sentence or make associations between the input and labels from downstream task. For this purpose, we only retain single char- acter from each token:
Predicting masked character subsets: First, we only predict the character subsets of each masked token. For example, given the masked tokens 'phone' and 'photo', the model FF should predict 'ph' for both tokens while the model FL should predict 'pe' and 'po' respectively.
Predicting the original full token: We also ex- periment with a more informative pre-training task by predicting the original full masked token. Note that the input to these models is the character sub- sets, e.g. F, LL, FML. The results of fine-tuning the models pre-trained using this task GLUE and SuperGLUE benchmarks are similar predicting masked characters and can be found in Appendix and Appendix E respectively.
First Character (F): we use only the first character of each token in the pre-training data as input. For example, 'cat' and 'computer' are both represented by the same token 'c'.
Middle Character (M): we only keep the middle character of each token: mid len(token)
token| 2
Last Character (L): the last character of each token is retained.
Experimental Setup
4.1 FULL TOKEN Model
Two Characters Further, pre-train models by inereasing the available information using only maximum of two characters in each token: (1) the first and last (FL); (2) the first two (FF); and (3) the last two characters (LL). These models also consider all tokens consisting of a single character.
We pre-train a full token model (Token) using MLM. This model is used as reference to compare the performance of our models that use character subsets as input.
4.2 Implementation Details
Three Characters In addition, we
maxi- mum of three characters from each token as input: (1) the first, middle and last (FML); (2) the first three (FFF); and (3) the last three (LLL.). These models also use all tokens consisting of one or two characters.
We implement all of our models using PyTorch (Paszke et al., 2019) and the Transformers library (Wolf et al., 2020). We pre-train each model for
'Exhaustively testing PLMs of different sizes and types is out of the scope of this paper as well as computationally prohibitive due to limited access to compute.

---

Model
Input Sequence
[MASK] Tokens
FULL TOKEN
mars about [MASK] the [MASK] of earth
half size
F
mars is ahout [MASK] the [MASK] of earth
halr size
M
mars ahout [MASK] the [MASK] of earth
half size
mars about [MASK] [MASK] of earth
half size
FL
mans is about [MASK] the [MASK] of earth
half size
FF
mars is about [MASK] the [MASK] of earth
half size
LL
mars about [MASK] [MASK] of earth
holf size
FML
mors is about [MASK] the [MASK] of earth
holf size
FFF
mar: is about [MASK] the [MASK] of earth
half size
LLL.
mars ahout [MASK] the [MASK) of earth
half size
V
mars ahout [MASK] the [MASK) of earth
half size
C
mirs is about [MASK] the [MASK] of earth
half size
Table I: Example of masked input sequence for each model considered. denotes first, middle, L last, V vowels and C consonants.
Model Category
Vocab Size
#Params
4.4
Tokenization
FULL TOKEN
50,010
124M
In our experiments, we focus full words. Sub- word tokenization such as BPE (Sennrich et al., 2016) results in more fragmented inputs with gen- crally shorter input tokens. Therefore, we apply a whitespace tokenizer to split the input sequences into tokens. We replace numbers with special to- ken <num>, and we filter out some special charac- ters. As result, we obtain approximately 2. to- kens from all training samples. Appendix A shows the distribution of token lengths as well as the num- ber of unique tokens for various token lengths in both the pre-training dataset and the downstream tasks training sets.
ONE CHARACTER
36
86M
TWO CHARACTERS
686
87M
THREE CHARACTERS
17,586
100M
VOWELS
3,0
89M
CONSONANTS
50, 010
124M
Table 2: Statistics for each model category. We use the most frequent 50K tokens in both FULL TOKEN and CONSONANTS.
IM steps using 8 NVIDIA Tesla V100 (SXM2- 32GB). We use learning rate of le-4 and batch size of 16 per GPU. We also of 0.01, attention dropout of 0.1, 10,000 warmup steps, le-8 Adam c, 0.9 Adam 31 and 0.999 Adam 32
weight decay
Table shows the vocabulary size of each model category together with the number of parameters. For instance, the vocabulary of the single character model category consists of the 26 English alphabet letters plus 10 tokens including punctuation marks and special tokens.
4.3 Pre-training Data
The models are pre-trained using the BookCorpus (Zhu et al., 2015) and English Wikipedia datasets from Hugging Face.* 4 We convert text to lowercase in both datasets. For the English Wikipedia, we eliminate headers and extract training samples that are no longer than 512 tokens. For the BookCorpus, sentences concatenated while maintaining maximum total of 512 tokens. The total number of training samples is 8.IM.
4.5 Model Fine-tuning
GLUE We fine-tune our models using the Gen- eral Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) for up to 20 epochs with early stopping tolerance of 5. We use five dif- ferent seeds for each fine-tuning task and report the average. We report matched accuracy for MNLI task (Williams et al., 2018), Matthews correlation for CoLA task (Warstadt al., 2019), Spearman correlation for STS-B B task (Cer et al., 2017), accu- racy for MRPC task (Dolan and Brockett, 2005),
'nttps://gf thub. httpg/ttw..oogggggggg//////////e
ets

---

Fl scores for QQP $ task, and accuracy for all other tasks. For all models, we retain the same type of input used during pre-training.
is expected as increasing the amount of informa- tion available from each token helps models learn better.
The results also show that the models pre-trained first characters achieve better average perfor- mance than the models pre-trained on last charac- ters. For example, the average performance of the and L models 62.0% and 60.1%, respectively. Furthermore, the average performance of the FFF model is about 2.8% higher than the average perfor- mance of the LLL model. While this demonstrates that the first characters are more informative than the last characters, the results also show that the first and last characters the first two or last two characters. For instance, the average performance of the FL. model 2. % and 2.4% better than the average performance of the FF and LL models, respectively. A similar obser- vation has been made for humans suggesting that the first and last letters of a word are more crucial for human reading comprehension than the middle letters (Grainger et al., 2004; Johnson and Eisler, 2012).
SuperGLUE We also fine-tune our models on six tasks from the SuperGLUE benchmark (Wang et al., 2019). The tasks are namely Commitment- Bank (CB) (De Marneffe et al., 2019), Choice of Plausible Alternatives (COPA) (Roemmele et al., 2011), Recognizing Textual Entailment (RTE) (Dagan et al., 2010), Words in Context (WiC) (Pilehvar and Camacho-Collades 2019), The Multi-Sentence Reading Comprehension (Mul- tiR) (Khashabi et al., 2018) and BoolQ (Clark et al., 2019). Similar to fine-tuning on GLUE, we fine-tune our models for up to 20 epochs with early stopping tolerance of 5. We also use five different seeds for each SuperGLUE task and we report the average score. Similar to GLUE tasks, we retain the original model input type.
more informative than
4.6 Probing Settings
Apart from the downstream predictive performance, we also test whether our character subset models acquire linguistic knowledge by probing them. For this purpose, we employ six widely used probing tasks from Conneau et al. (2018) analyze the representation output of our various PLMs at each layer. These tasks assess syntactic and semantic information. More details about the probing tasks can be found in Appendix B.
The results also demonstrate that the model pre- trained using only consonants (C) achieves the best performance across all the GLUE tasks compared to the rest of the models trained on partial tokens, only 2.3% less than the average of the full token model. This is most likely because it has suffered less information loss compared to other models as we discuss in $6.2.
5 Results
Furthermore, we see that the performance reten- tion for the models pre-trained using only one char- acter of the input tokens is high when compared with the performance the model pre-trained us- ing the full tokens. For example, the model re- tains about 77% of the average performance of the model pre-trained using full tokens. Similarly, the model pre-trained using only the middle character achieves approximately 84% of the performance of the model pre-trained using full tokens on the same QNLI task. This might suggest that PLMs have the ability to still learn downstream task-specific associations using very little information from the input tokens.
5.1 GLUE
Table 3 presents the results of fine-tuning the mod- els on GLUE. Overall, we observe that using more characters from the tokens results in better perfor- mance. For example, the model pre-trained using the last two characters (LL) achieves average GLUE performance of 68.2%, while the model pre- trained on the last character only (L. achieves an average GLUE performance of 60. %. Similarly, the performance of the FF model on the MNLI task increases to 71.5% compared 61.4% of the F model. The same can be observed when the number of characters increases from two to three. For in- stance, in the SST task, the performance of the FFF model increases by 9.3% compared the perfor- mance of the FF model. This boost in performance
5.2 SuperGLUE
Table 4 presents the fine-tuning results on Super- GLUE Similar the fine-tuning results on GLUE, we first observe that the performance of the models increases when more characters from the tokens
'https: / /ttp:///gouaattaaae www.uuuurrcoooooo i rst-Quo ra-Dataset- - ease- Saase-Questiine Question-Fall Fairs

---

Model
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
GLUE Avg.
Token
82.5
89.7
86.2
67.8
91.8
86.1
58.0
86.1
81.0 +:: 0.3
F
61.4
76.5
78.8
58.2
64.3
78.0
9.4
69.3
62.0 1: 0.4
57.5
74.8
76.9
56.7
62.3
77.6
11.4
67.3
60.6 1: 0.3
74.3
77.1
64.4
75.9
FL
83.5
58.2
77.8
83.0
27.3
70.6
FF
71.5
83.6
83.2
57.6
76.7
83.0
11.8
80.7
68.5 + 1.4
LL
68.1
81.7
82.4
58.3
75.3
80.7
25.2
74.0
68.2 + 0.2
FML
78.3
87.3
85.4
60.4
87.7
82.5
47.6
83.7
76.6 + 0.2
87.8
86.0
73.1
85.7
83.9
59.2
81.5
82.6
38.3
77.0
72.7
61.8
79.9
80.5
58.3
68.2
79.4
8.7
72.1
63.6 0.5
C
80.7
88.9
85.9
61.4
90.4
84,5
52.1
85.5
78.7 4: 0.4
Table 3: Results on GLUE dev sets with standard deviations over five runs for models pre-trained to predict the partial token. Bold values denote the best performance across all models. Underlined values denote the second best performance.
Model
BoolQ
CB
COPA
RTE
WiC
MultiRC
SuperGI.UE Arg.
Majority
62.2
50.0
55.0
52.7
50.0
59.9
55.0
Taken
73.5
83.5
61.6
66.2
66.5
69.7
70.2 0.8
66.6
73.7
57.9
59.3
65.3
63.2 1.1
70.3
62.4
66.3
68.8
56.8
56.4
57.2
65.0
61.8
FL
70.1
80.8
59.5
58.7
59.3
68.2
66.1 0.6
65.8
69.4
73.1
57.6
57.6
58.6
66.3
63.8 1.1
FMI
72.4
79.5
60.2
59.5
63.7
69.1
67.4 0.7
FFF
70.8
80.4
66.9
LLL
70.1
75.9
58.8
60.1
65.1
V
68.4
68.3
57.5
57.7
59.8
66.8
63.1 0.8
82.9
60.0
62.5
67.7
Table 4: Results on six SuperGLUE task dev sets with standard deviations over five runs. Bold values denote the performance across models. Underlined values denote the second best performance.
are used in pre-training. For instance, the average performance of the model increases from 63.2%, when pre-trained using only the first character, to 65.8% when pre-trained using the first char- acters. A similar pattern can be observed when pre-training the model using the last three charac- ters, we notice an increase 1.3% of the average performance compared with the model pre-trained using only the last two characters when predicting the character parts of the token. That indicates that PLMs learn better when the information from each token increases.
of 66.9% while the LLL model 65.1%. Similarly, the model pre-trained using the first character (F) achieves accuracy of 73.7% while the model pre- trained using the last three characters (L) achieves an accuracy of 68.8% on the CB task.
We also observe, a similar pattern to the fine- tuning results on GLUE, that the model pre-traimed using only consonant characters (C) achieves the second-best average performance of 67.7%. The results also show that the C model achieves 4.6% higher average performance compared to the model pre-trained using only vowel characters (V). A com- parable observation has been made in the context of human reading comprehension, indicating that consonant characters hold greater significance com- pared to vowel characters (Fry. 1968).
The results also show consistent patterns to the results on GLUE where the first characters are more informative than the last characters. For example, the FFF model achieves an average performance

---

Donserenno Vowds Three Chanawa Ches One Chaa F. Token
Model
3,00
2.50
2.25
1.75
2
4
10
12
14
26
Epachs
Figure 2: The loss curves for single character models M I.).
3
5
10
Tolkrn Lungth (Acharattara)
Figure 3: Token length distribution of the pre-training corpus for each model category. 100% of tokens for ONE CHAR have one letter, 99% for Two CHARS have two and 79% for THREE CHARS have three. For Vow- ELS 73% single-character tokens and for CONSO- NANTS 78% of pre-training tokens consist of four char- acters.
Finally, the results also demonstrate substantial performance retention for the models pre-trained using subsets of each input token compared with the performance of the model pre- pre-trained using full input tokens. For instance, the model pre-trained using only the first character retains approximately 90% the average performance the model pre- trained using full tokens. addition, the LL model achieves retention the performance of approxi- mately 94% on the BoolQ task compared with the model pre-trained using the full tokens. However, this might be attributed to the fact that SuperGLUE is more difficult benchmark where the perfor- mance of the full token model not as high in GLUE (average 81.0% and 70.2% respectively).
CHARS and THREE CHARS respectively. In Fig- ure we plot the distribution of token lengths (in terms of number of characters) in each pre-training corpus after tokenization." We observe that the dis- tribution of the FULL TOKEN pre-training corpus more evenly distributed, with 42% of the total pre-training tokens to include either one, two or three characters. When we apply our tokenizers for ONE, Two and THREE CHARS model categories of models, we remove all the extra characters of the words to keep the threshold of the required char- acter length. For instance, for the THREE CHARS every token longer than three letters is trimmed to exact three letters. This way, we can this corpus-based metric to measure how much infor- mation is lost from the original corpus (via its vo- cabulary). For ONE CHAR 95% the original words are trimmed down to a single letter, while in TWo CHARS 79% are turned into double-character tokens. For the THREE CHARS category, we lose around 48% of the original tokens with more than 3 letters, while the resulting distribution is 5% single, 16% double and 79% triple-character tokens.
6 Analysis
6.1
Pre-training Loss Curves
Figure shows the pre-training loss curves for the single character models illustrating the progression and convergence of the loss function throughout the pre-training process. We observe that both the F and M models have very close pre-training losses and that the model L. has the lowest pre-training loss. Overall, we note that the three models have stable pre-training which indicates that the mod- els are able to learn using limited input informa- tion. The pre-training loss curves for the other pre-trained models can be found in Appendix C.
6.2 Corpus-based Token Information Loss
We also measure the distribution of n-grams between the different model categories. Table 6 shows the number of unigrams, bigrams and tri- grams in the pre-training corpus. We observe that the FULL TOKEN and the CONSONANTS models
We aim analyze how much information loss occurs when we different tokenization strate- gis for each of the models considered ($3). We combine all models in Table 1 in three categories depending on how many characters they keep in each word during tokenization: ONE CHAR, Two
"We plot the distribution of all the tokens in the pre-training corpora, not the unique tokens.

---

Model
TreeDepth
TopConst
BShift
Tense (Semantic)
SubjNum
CoordInv
(Syntactic)
(Syntactic)
(Syntactic)
(Semantic)
(Semantic)
Majority
17.9
5.0
50.0
50.0
50.0
50.0
Full-Token
42.7 = 0.1
78.4 + 0.3
88.5 + 0.3
89.2 = 0.8
88.0 = 0.4
73.2 = 0.3
F
31,3 + 0.3
53.3 +: 0.3
56.0 +: 0.4
63.4 +: 0.9
61.2 0.2
59.6 + 0.3
30.6 = 0.2
53.7 0.2
55.2 0.2
63.3 0.3
65.9
58.4 0.4
L
33.2 = 0.2
60.4 # 0.3
57.3 + 0.3
84.1 # 0.5
83.6 = 0.0
60.4 = 0.5
FL
43.3 + 0.6
78.6 + 0.4
72.0 0.8
85.3 +: 0.3
88.1 0.7
68.2 + 0.3
FF
39.9 + 0.6
69.0 + 0.6
67.0 0.4
65.2 + 0.4
72.0 0.5
63.5 + 0.4
LL
41.0 - 0.5
74.9 + 0.4
69.5 + 0.3
- 0.2
89.2
64.5 = 0.3
FML
45.0 0.4
81.4 0.3
84.5# 0.2
87.0 0.3
91.3
68.4 0.8
FFF
41.6 1.0
79.3 # 0.3
80.2
6.12
77.6 0.2
67.7 = 1.2
LLL
44.8 -: 0.2
81.1 0.3
82.0
88.4 0.1
69.1 + 0.5 90.0 1
V
30.4 + 0.4
51.4 0.4
57.6 0.3
68.3 +: 0.1
65.6 0.1
57.9 + 0.2
44.2 = 0.6
80.2 + 0.1
85.0 0.2
87.1 0.4
78.9
71.0 + 0.2
Table 5: Results the probing tasks for the best-performing layer of each model using mean accuracy with standard deviation over three runs. Bold values denote the best performance across each task. Underlined values denote the second best performance.
Category
Unigrams
Bigrams
Trigrams
6.3 Probing
FULL TOKEN
50K
43M
302M
We hypothesize that the probing performance of models pre-trained on partial tokens will be lower to the performance of full tokens. Table presents the results on the six probing tasks using the representations from the models as inputs to the MLP classifier. Note that we evaluate all layers from each model individually and choose the best- performing layer on the test set.
ONE CHAR
34
1K
37K
TWO CHARS
686
300K
20M
THREF CHARS
17K
13M
185M
VOWELS
3K
2M
28M
CONSONANTS
50K
39M
299M
Table 6: The number of unigrams, bigrams and trigrams the pre-training corpus for each model category.
Overall, we find that the predictive performance of the model trained on representations learned us- ing the first, middle and last (FML) characters input tokens achieves the best performance in three out of the six probing tasks. For example, in the Subj probing task, the FML model achieves the best performance of 91.3%, while the one ing only the first character of input tokens achieves 61.2% This is surprising given the large informa- tion loss in the pre-training data ($6.2).
have the largest number of unigrams, bigrams and trigrams compared other model categories. We also see that the tokenized pre-training corpus of the Two CHARS model retains only about 7% of the total number of trigrams in the pre-traing corpus the FULL TOKEN. For the THREE CHARS cate- gory, we observe a reduction of 66% and 70% in the number of unigrams and bigrams, respectively, compared to the FULL TOKEN. This indicates substantial decrease in the occurrence of unique unigrams and bigrams when utilizing only three characters as opposed to full tokens.
We also see that the ability of PLMs to acquire linguistic information increases when more charac- ters from the tokens are used in pre-training. For instance, in the TopConst probing task, the model pre-trained using the first character achieves an accuracy of 53.3%, while the model pre-trained using the first two characters achieves an accuracy of 69.0%. Similarly, in the CoordInv probing task, the model pre-trained using the last three characters achieves 4.6% higher accuracy compared with the model pre- traiieeeed using the last two characters.
Overall our analysis shows that even though we lose very large percentage of the original infor- mation (in terms of token characters) from the orig- inal pre-training corpus for the full token model, surprisingly, fine-tuning the models pre-trained on partial tokens in downstream tasks does not hurt performance in an analogous way (85).
The results also show that models pre-trained

---

using the last characters perform better than the first character models. For example, in the Tense probing task, the L model achieves an accuracy of 84. 84.1% while the model achieves an accu- racy of 63.4%. In the BShift task, the LLL model achieves 1.8% higher accuracy than the FFF model. This might suggest that specific character positions within tokens play crucial role in how PLMs en- code linguistic information in English.
or appear to 'learn' semantic and syntactic infor- mation (i.e. probing) by using only one or two characters from each token. This demonstrates the ability of PLMs to find patterns in seemingly ran- dom or limited data. Additionally, our findings also suggest that these models may have the capability to process language in ways that are fundamentally different from human cognition.
Conclusion
Finally, the results show that even when models are pre-trained using one character, they are still able to encode some linguistic information. For instance, in the Tense probing task, the model pre- trained using only the last character achieves an accuracy of 84.1% while the the model pre-trained using full tokens achieves 89.2%. In the SubjNum task, the difference in accuracy between the L and full-token models is only 1.4%. This indicates similar behavior to downstream tasks in GLUE and SuperGLUE. The MLPs used for modeling the probing tasks are able to find associations in the data using little information from the token.
In this work, we explored how the pre-training of language models using different subsets of charac- ters from the input tokens affect their performance on downstream tasks. To our surprise, we found that even using just one character from each in- put token achieves performance way above chance. Our probing results further revealed that specific character positions within tokens affect how the models capture linguistic knowledge.
Acknowledgments
AA is supported by the Centre for Doctoral Train- ing in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innovation grant EP/S023062/1. NA is supported by EPSRC grant EP/V055712/1 part of the Euro- pean Commission CHIST-ERA programme, call 2019 XAI: Explainable Machine Learning-bast Artificial Intelligence.
7 Discussion
Our results reveal that certain subsets of input to- kens are more informative than others. For in- stance, we found that PLMs perform better when pre-trained using the first and last characters instead of the first or last two characters of input tokens. This appears to align with research in cognitive science and psycholinguistics which suggests that the first and last letters of word are more cru- cial for human reading comprehension than the interior letters and that the positions of letters are not given equal weight in the cognitive representa- tion a word (Grainger et al., 2004; Johnson and Eisler, 2012). Previous studies have also suggested that the first and last letters of a word hold more importance because they pertain to the way the mind organizes and retrieves lexical information (Forster, 1976; Jordan, 1990; Jordan et al., 2003). Other studies have suggested that the first and last letters of word are more easily identified as they are less impacted by external interference or the presence of surrounding letters compared to inner letters (Bouma, 1973; Chambers, 1979; McCusker et al., 1981; van der Heijden, 2003).
Limitations
Our research is currently limited only to English due to computational constraints. How- ever, expanding to other languages with different characteristics presents a potential area for future exploration.
Languages
Models We employed a BERT-like model (De- vlin et al., 2019) for simplicity and due to limited access to compute. Exploring different model sizes or types, i.c. generative models such as GPT-3 (Brown et al., 2020) could be considered in future studies.
Model Usability We experimented with models trained on partial tokens, even only on single char- acters. We do not claim that these models have any practical usability in downstream tasks apart from helping us answering our research questions. Our goal in this study was to test the boundaries of PLMs by pre-training on data containing very limited information.
Surprisingly, our results also highlight that PLMs are still able to learn from data that may be incomprchensible to humans. Our models per- form well on natural language understanding tasks

---

Probing Tasks We experiment with six probing tasks from Conneau et al. (2018) as a representative sample. Expanding probing tasks to cover more lin- guistic characteristics is another avenue for future research.
Human Language Technologies, Volume 1 (Long and Short Pupers), pages 2936, Minneapolis, Min- nesota. Association for Computational I Linguisties.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre- Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- tions.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. line-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv: 1608.04207
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ahmed Alajrami and Nikolaos Aletras. 2022. How does the pre-training objective affect what large language models learn about linguistic properties? In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 131 1-17,, Dublin, Ireland. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single S&!#* vector: Probing sentence emheddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics.
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke 7ettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (FMNL.P-JCNTP)), pages 5360-5369, Hong Kong, China. Association for Computational Linguis- tics.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Ra- tional, evaluation and carroachhe-errattunn Natural Language Engineering, 16(1):105-100
Christopher Davis, Christopher Bryant, Andrew Caines, Marek Rei, and Paula Buttery. 2022. Probing for targeted syntactic knowledge through grammatical error detection. arXiv preprini 2rXiv:2210.162222
H. Bouma. 1973. Visual interference in the parafoveal recognition of initial and final letters of words. Vision Research, 13(4):767-722.
Marie-Cathering De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putatiooal Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
Daniel Cer, Mona Diab, Eneko Agirre, Tãigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-201 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the I 1hth International Workshop on Semantic Evaluation (SemEval-201) 2017),,, pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Susan M. Chambers, 1979. Letter and order information lexical access. Journal of Verbal Learning and Verbal Behavior, 18(2):225-211.
Luca Di Liello. Matteo Gabburo, and Alessandro Mos- chitti. 2022. Effective pre-training objectives for transformer-bast autoencoders. arXiv preprint urXiv:2210.13522
Cheng-Han Chiang and Hung-yi Lee. 2020. Pre- training language model without human language. arXiv preprint arXiv:2012. 19995.
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics:
K. T. Forster. 1976. Accessing the Mental Lexicon. In
New Approaches to Language Mechanisms. A Col- lection of Psycholinguistic Studies, pages 257-287. Horth-Holland Publishing Company.

---

Edward B Fry. 1968. Reading words with vowels miss- ing and with consonants missing.
Kundan Krishna, Jeffrey Bigham, and Zachary C. Lip- ton. 2021. Does pretraining for summarization re- quire knowledge transfer? In Findings of the A Associ- ation for Computational Linguistics: EMNLP 2021, pages 3178-3189 319, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Jonathan Grainger, Carol Whitney, et al. 2004. Does the huamn mnid raed wrods as a wlohe? Trends in cognitive sciences, 8(2)58-59.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesola. Association for Computational Linguistics.
Kundan Krishna, Saurabh Garg, Jeffrey P Bigham, and Zachary Lipton. 2022. Downstream datasets make surprisingly good pretraining corpora. arXiv preprint 2rXiv:2209.144:
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Confer- ence on Learning Representations.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelli- gence Research, 61:907-926.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8818 1818-, Dublin, Ireland. Association for Compu- tational Linguistics.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACI 2019-57th Annual Meeting of the Association for Computational Linguistics.
Rebecca I. Johnson and Morgan E Fisler. 2012. The importance of the first and last letter in words during sentence reading, Acta psychologica, 141(3):336- 351.
Chia-Hsuan Lee, Aditya Siddhant, Viresh Ratnakar, and Melvin Johnson. 2022. DOCmTS: Document-level pretraining of multilingual language models. In Find- ings the Association for Computational Linguistics: NAACI. 2022, pages 425-477, Seattle, United States. Association for Computational Linguisties.
Timothy R Jordan. 1990. Presenting words without
interior letters: Superiority over single letters and influence of postmask boundaries. Journal of Experi- mental Psychology: Human Perception and Perfor- mance, 16(4):893.
Tomasz Limisiewiez and David Mareček. 2021. Intro- ducing orthogonal constraint structural probes. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 428 442, Online. Association for Computational Linguistics.
Timothy R Jordan, Sharon M Thomas, Geoffrey R Patching, and Kenneth C Browt, 2003. As- sessing the importance of letter pairs in initial, ex- terior, and interior positions in reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29)5)8883
Yongjie Lin, Yi Chem Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2079 ACI. Work- shop BlackboxNLP: Analyzing and Interpreting Neu- ral Networks for NLP, pages 241-253, Florence, Italy. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay. and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pa- pers). pages 252-262, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguisties.
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang-goo Lee. 2019. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Conference on Learning Representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907.1

---

Avinash Madasu and Shashank Srivastava. 2022. What do large language models learn beyond language? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6940-6953, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Annual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers), pages 7302- 7315, Dublin, Ireland. Association for Computational Linguisties.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon, 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In 2011 AAAI Spring Symposium Series.
Leo X McCusker, Philip B Gough, and Randolph G Bias. 1981. Word recogmition inside out and outside in. Journal of Experimental Psychology: Human Perception and Performance, 7(3):338.
Rudolf Rosa and David Mareček. 2019. Inducing syn- tactic trees from bert representations. arXiv preprint arXiv: 1111.
Kurt Micallef, Albert Gatt, Mare Tanti, Lonneke van der Plas, and Claudia Borg, 2022. Pre-training data qual- ity and quantity for a low-resouree language: New corpus and bert models for maltese. arXiv preprint arXiv:2205. 10517.
Nikunj Saunshi. Sadhika Malladi, and Sanjeev Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. In Interna- rional Conference on Learning Representations.
Max Huille--Bherseenn, Rob van der Goot, and Barbara Plank. 2022. Probing for labeled dependency trees. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers). pages 7711-7726 Dublin, Ireland. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Isabel Papadimitriou and Dan Jurafsky. 2020. Learn- ing Music Helps You Read: Using transfer to study linguistic structure in language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6829-6839, Online. Association for Computational Linguistics.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, high-per pege-performance deep learning library. Advances in neural information processing systems, 32:8026- 8037.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representa- tions. arxiv preprint 1905. 16.
Mohammad Taher Pilehvar and Jose Camacho- Collados, Camacho-Collados, 2019. WIC: the word-in-context dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Alex HC van der Heijden. 2003. Selective attention in vision. Routledge.
David Vilares, Michalina Strzyz. Anders Sógaard, and Carlos gómez-Rodrígne 2020. Parsing as pretrain- ing. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 34, pages 141111
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael. Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general- purpose language under- standing systems. Advances in neural information processing systems, 32.
Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unitied text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67
Keith Rayner, Sarah J. White, Rebecca L. Johnson, and Simon P. Liversedge. 2006. Raeding wrods with jubmled letrres: There is cost. Psychological Sci- ence, 1993 PMID: 16507057.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowl- edge in language models. In Proceedings of the 60th

---

Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics. 7:625-641 6.11-44..
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Con- ference on Computer Vision (ICCV), pages 19-27.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A
challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yuhuai Wu, Felix Li. and Percy S Liang. 2022. In- sights into pre-training via simpler synthetic tasks. Advances in Neural Information Processing Systems, 35:21844-2188 21857.
Huiyin Xue and Nikolaos Aletras. 2022. HashFormers:
Towards vecabulary-nndepeneent pre-trained trans- formers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7862-7874, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, and Nikolaos Aletras. 2021. Frustratingly simple pretraining alternatives to masked language modeling. In Proceedings the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 31 16-3125, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Atsuki Yamaguchi, Hiroaki Ozaki, Terufumi Morishita, Gaku Morio, and Yasuhiro Sogawa. 2023. How does the task complexity of masked pretraining objectives affect downstream performance? arXiv preprint arXiv:2305. 10992.
Zhilin Yang. Zihang Dai. Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li. Shuailiang 7hang, Xi Zhou, and Xiang Zhou. 2020. Sema bert for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9628-998899 9635.

---

Appendix
A Distribution of Token Lengths
A.1 Pre-training Dataset
Figures show the distribution of the token lengths the pre-training dataset.
A.2 GLUE Tasks
Figures 5 to 12 show the distribution of the token lengths in the training set of each GLUE task.
A.3 SuperGLUE Tasks
Figures 13 to 16 show the distribution of the token lengths in the training set of each SuperGLUUE task.
B Probing Tasks
The syntactic information tasks include: Depth, which examines representations maintain infor- mation about the hierarchical structure of sen- tence by predicting the depth of its parse tree. Top- Const, which predicts the top constituents of a sen- tence's parse tree. BShift evaluates if two neighbor- ing words have been reversed not. The semantic information tasks include Tense, which predicts the verb in the main clause is in present or past form. SubjNum predicts whether the subject of the main clause is singular or plural. CoordIay deter- mines if a sentence consisting of two coordinate clauses has been inverted or not. Each task con- sists of 100k sentences for training, 10k sentences for validation and 10k sentences for testing. Using the recommended hyperparameters in the SentE- val toolkit (Conneau and Kiela, 2018), we train multi-layer perceptron (MLP) classifier for each task.

---

400
10"
250
Number of Unique Tokens
300
100
250
10a
200
150
100
100
50
2
10
10+
Number Characters
Number of Characters
(a)
(b)
Figure The distribution of the token lengths the pre-training dataset.
10a
Number Unique Tokens
2M
1.5M
10
1M
102
500
Number Characters
Namber
(a)
(h)
Figure 5: The distribution of the token lengths in the training of the MNLI GLUE task.
700
10a
Number of Unique Tokers
600
10a
400
Toca
200
100
6
10+ 9
Mumber Characters
Figure 6: The distribution of the token lengths in the training set of the QNLI GLUE task.

---

1.6 M
10"
Number unique Tokens
1.4 M
12M
of
1M
100
800
600 K
10a
400
200
10+
Number of Characters
Namber Characters
(a)
(h)
Figure 7: The distribution the token lengths in the training set of the QQP GLUE task.
20
Total Number of Takens
Number of Unique Tokers
10
5k
4 Number Characters
(a)
(b)
Figure 8: The distribution of the token lengths in the training of the RTE GLUE task.
100 k
Number of Unique Takenss
Z0a
80
40
10
20 K
7
104
6
10+ 9
Number Charactors
Mumber Characters
Figure 9: The distribution of the token lengths in the training set of the SST GLUE task.

---

25
1.2
Total Talal Number of Tokens
20
Number of Unique Tokers
15k
102
4 Number Characters
(a)
(b)
Figure 10: The distribution of the token lengths in the training set of the MRPC GLUE task.

---

C Pre-training Loss
full token, on the six SuperGLUE tasks. Simi- lar GLUE, the the results show that the models do not benefit much when they are pre-trained predict the full token instead of predicting the char- acter parts of the token. For instance, the model achieves an average accuracy of 62.4% when pre- trained to predict the character parts of the token while it achieves average accuracy of 62.1% when pre-trained predict the full token. The same can be observed in the performance the FL models on the MultiRC task, where the difference in performance between the two pre-training tasks is 0.5%.
C.1
Two Characters
Figure 17 shows the pre-training loss curves for the models pre-trained using two characters from each input token. Similar to the loss curves of single character models, we observe that the two characters models have stable pre-training.
C.2 Three Characters
Figure 18 shows the pre-training loss curves for the models pre-trained using three characters from each input token. We also note that the models have stable pre-training.
F Fine-tuning the Full Token Model
C.3 Token, Vowels and Consonants
Tables and 10 show the results of fine-tuning the full token model (Token) using partial input token on both GLUE and SuperGLUE benchmarks, re- spectively. The results show the full token model achieves lower performance when fine-tuned us- ing partial tokens on both GLUE and SuperGLUE benchmarks compared to the models pre-trained and fine-tuned using the same partial input token.
Figure 19 shows the pre-training loss curves for model pre-trained using full tokens in addition to the models pre-trate pre-trained using the vowel and conso- nant characters from each input token.
D GLUE Full Results
Table 7 show the results for all models pre-trained using both pre-training tasks, predicting the char- acter parts of the token and predicting the original full token. The results show that the pre-training task of predicting the character parts of the token slightly helps most models achieve better perfor- mance than the other pre-trae pre-training task of predict- ing the original full token. For instance, the FML model achieves an average performance of 76.6% when pre-trained to predict the character parts of the token, while it achieves an average performance of 76.3% when pre-trained predict the original full token. Similarly, the FL model achieves about 0.3% higher average performance when pre-trained to predict the character parts of the token compared to when pre-trained to predict the original token. This might indicate that the pre-training task to predict the original full token using subsets of the token is hard task for models the size of the full tokens vocabulary (used in the output classification layer of the model) is much higher than the size of the characters subsets vocabulary (used in the input).
E SuperGLUE Full Results
Table 8 show the results for all models pre-trained using both pre- pre-training tasks, predicting the char- acter parts of the token and predicting the original

---

16
10P
TTlenns
12K
Number of Unique Tokens
8
4.K
Number of Characters
Namber Characters
(a)
(h)
Figure The distribution token lengths in the training set of the CoLA GLUE task.
17.5
10a
15
Number of Unique Tokers
Tolens
7.5
10"
5k
2.5
Number Characters
(a)
(b)
Figure 12: The distribution of the token lengths in the training of the STS GLUE task.
140
Number of Unique Tekens
1.2
100
60 k
20
5
7
104
6
10+ 9
Number Charactors
Mumber Characters
Figure 13: The distribution of the token lengths in the training set of the BoolQ SuperGLUE task.

---

400
Number of Unique Tokens
350
2.5
300
of
2k
250
1.5
150
100
300
10+
10+
Number of Characters
Number Characters
(a)
(h)
Figure The distribution of token lengths the training set the CB SuperGLUE task.
200
Number of Unique Tokers
250
200
Number
1.5k
150
1k
100
500
10+
Number of Characters
Number Characters
(a)
(b)
Figure 15: The distribution of the token lengths the training set the COPA SuperGLUE task.
17.5
1.2
15
Number of Unique Tokerts
800
Tocal Number
10
7.5
400
3k
200
2
4
9
10+
1
5
10+
Number Characters
Number of Cheracters
Figure 16: The distribution of the token lengths the training set the WIC SuperGLUB task.

---

Model
MNLI
QNLI
QQP
RTE
SST
MRPC
CoLA
STS
GLUE Avg.
Token
82.5
89.7
86.2
67.8
91.8
86.1
58.0
86.1
81.0 + 0.3
Pre- Pre-training: Predicting the partial token
F
61.4
76.5
78.8
58.2
64.3
78.0
9.4
69.3
62.0 + 01.4
M
57.5
74.8
76.9
56.7
62.3
77.6
11.4
67.3
60.6 + 0.3
L
57.8
74.3
77.1
58.0
64.4
75.9
12,1
61.4
60.1 + 0.3
FL
71.9
83.5
84.0
58.2
77.8
83.0
27.3
79.3
70.6 + 0.4
83.6
83.0
11.8
80.7
68.5 1:
I.I.
68.1
81.7
82.4
58.3
75.3
80.7
25.2
74.0
68.2 +
FML
78.3
87.3
85.4
82.5
47.6
83.7
76.6 1
FFF
77.9
87.8
85.2
60.3
86.0
83.3
39.3
84.6
75.5 + 0.3
LLL
73.1
85.7
83.9
59.2
81.5
82.6
38.3
77.0
72.7 1 0.2
V
61.8
79.9
80.5
58.3
68.2
79.4
8.7
72.1
63.6 + 0.5
C
80.7
88.9
85.9
61.4
90.4
84.5
52.1
85.5
78.7 + 0.4
Pre-training: Predicting the full token
F
61.6
76.3
78.7
55.2
66.2
75.9
11.7
69.5
61.9 + 0.3
M
57.2
75.1
77.4
55.6
63.5
75.4
4.7
67.6
59.6 + 0.6
1.
57.4
74.1
77.6
54.7
65.3
74.5
10.5
62.3
59.5 + 0.7
FL 72.6
84.1
84.3
54.6
78.6
80.7
29.8
77.9
70.3 1 0.4
71.4
83.6
83.3
56.5
77.8
83.3
19.1
81.2
69.5 +
LL
68.3
81.6
82.3
56.6
76.8
81.0
24.1
73.6
68.1 + 0.5
FML
78.0
87.0
59.7
82.8
46.9
82.7
76.3 1
FFF
78.0
87.4
85.4
57.8
86.2
82.7
38.8
83.4
75.0 1 0.2
LLL
73.7
85.7
84.3
57.5
82.6
81.8
38.9
76.9
72.7 1 0.4
V
62.2
80.0
80.6
58
69.2
80.0
9.6
71.4
63.9 + 0.7
C
80.6
88.5
85.7
62.2
90.9
85.5
51.5
85.6
78.8 + 0.2
Table 7: Full results on GLUE dev sets with standard deviations over five runs. Bold values denote the best performance across all models. Underlined values denote the second best performance.
10
Model
Model
FL
FML
FF
5
LL
LLL
1955
Loss
2
2
4
6
10
12
14
16
2
6
10
12
14
16
Epochs
Epochs
Figure 17: The loss curves for characters models.
Figure 18: The loss curves for three character models.

---

Model
BoolQ
CB
COPA
RTE
WiC
MultiRC
Avg.
Majority
62.2
50.0
55.0
52.7
50.0
59.9
55.0
Token
73.5
83.5
61.6
66.2
66.5
69.7
70.2 0.8
Pre-training: Predicting the partial token
F
66.6
73.7
57.9
56.6
59.3
65.3
63.2 1.1
M
66.6
70.3
57.2
57.9
57.4
64.7
62.4 = 0,9
66.3
68.8
56.8
56.4
57.2
65.0
61.8 0.8
FL
70.1
80.8
59.5
58.7
59.3
68.2
66.1 0.6
FF
69.8
80.1
58.1
57.9
60.7
68.3
65.8 0.9
69.4
73.1
57.6
57.6
58.6
66.3
63.8 1.1
FML
72.4
79.5
60.2
59.5
63.7
69.1
67.4 - 0.7
FFF
70.8
80.4
59.4
59.8
61.9
68.8
66.9 0.9
LLL
70.1
75.9
58.8
57.9
60.1
67.8
65.1 0.9
V
68.4
68.3
57.5
57.7
59.8
66.8
63.1
0.8
C
72.1
82.9
60.0
60.8
62.5
68.0
67.7 0.7
Pre-training: Predicting the full token
F
67.8
72.1
58.3
55.4
58.6
65.8
63.0 0.6
M
67.6
71.7
55,2
56,1
57.0
65.0
62.1 = 0.8
67.1
71.2
55.0
57.9
65.1
62.4 0.9
FL
69.4
70.5
59.7
57.1
60.1
67.7
64.1 + 0,9
FF
69.5
72.3
58.9
56.4
59.2
68.0
64.1 0.8
LL
68.4
70.6
58.5
56.7
58.7
67.1
63.3 1.0
FML
72.3
79.1
62.8
59.6
61.1
68.2
67.2 0.8
FFF
71.6
78.4
60.8
59.8
60.9
66.8 0.9
LLL
69.8
71.9
60.1
58.4
60.3
67.6
64.7 = 1.1
68.2
70.6
60.8
57.3
65.9
63.7 0.7
C
72.6
80.4
59.7
61.0
64.1
69.3
67.9 0.9
Table 8: Full results on six SuperGLUE task dev with standard deviations over five runs. Bold values denote the best performance across all models. Underlined values denote the second best performance.
Partial Token
MNLI
QNLI
QQP
RTE
SST
MRPC
CoL.A
STS
GLUE Avg.
59.3
74.3
77.8
56.2
64.4
76.5
7.3
66.4
60.3 1 0.4
M
55.3
71.8
75.0
55.5
60.5
74.6
8.1
65.0
58.2 = 0.9
55.7
75.9
60.1
74.8
10.3
57.5 +
FL
66.2
78.3
81.4
59.4
71.2
77.6
16.1
73.9
65.5 + 0.6
FF
66.3
79.1
81.3
59.4
70.3
79.5
12.0
77.9
65.7 + 0.4
LL
63.0
77.3
80.3
58.6
68.7
79.7
10.6
69.8
63.5 = 1.0
FML
69.5
80.1
82.4
59.7
80.3
79.1
12.9
77.6
67.7 + 0.4
FFF
59.9
77.7
80.3
11.9
81.4
LLL
67.0
79.6
81.6
60.4
73.6
79.4
13.7
73.5
66.1 + 0.4
58.1
77.8
79.
56.6
64.3
77.1
6.5
69.5
61.1 + 0.5
69.1
80.2
82.3
59.7
79.9
79.5
15.4
78.1
68.0 + 0.2
Table Results fine-tuning the Token model on GLUE dev sets using partial input tokens. Bold values denote the best performance across all models. Underlined values denote the second best performance.

---

Partial Token
BoolQ
CB
CОРA
RTE
WiC
MultiRC
Avg.
Majority
62.2
50.0
55.0
52.7
50.0
59.9
55.0
F
66.0
68.8
52.0
54.1
55.6
60.4
59.5 + 1.1
M
66.4
64.7
57.0
56.6
55.8
61.1
60.3 + 0.9
L
66.0
71.9
56.4
55.0
57.7
62.3
61.6 0.8
FL
67.5
73.2
55.5
59.7
56.6
64.0
62.8 + 1.2
FF
67.5
70.5
55.2
60.7
58.4
63.7
62.7 + 1.0
LL
67.4
73.2
61.5
57.9
59.6
61.0
63.4 0.8
FML.
68.4
76.8
62.8
60.6
60.5
66.4
65.9 + 0.9
FFF
67.2
69.2
56.0
59.7
60.9
64.4
62.9 1.1
LLL
67.1
75.4
55.0
62.6
60.3
64.7
64.2 1.2
66.4
69.2
56.5
56.2
57.1
63.2
61.4 1.1
67.2
73.7
54.2
60.6
60.3
64.1
63.4 0.7
Table 10: Results for fine-tuning the Full Token model on six SuperGLUE task dev sets using partial input tokens. Bold values denote the best performance across all models.

---

Model
Token Vowels
8
Consonants
Loss
Epochs
Figure 19: The loss curves for Full Token, Consonants and Vowels models.


Reviews:
[{"review_id": "x8FeMz4Hsx", "reviewer": "Reviewer_77VM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive feedback. \n\n1. **Experimental protocol:** Thank you for the suggestion. The same protocol used for pre-training is also used on downstream tasks for each model. For example, when fine-tuning the first character model (F), we use the first character of each token in the downstream tasks. We are going to include extra experiments in the camera-ready version where we fine-tune the full token model on downstream tasks using partial input tokens (i.e. using single, two and three characters). \n\n2. **LMs of smaller sizes:** This is an interesting idea and can be explored in future work. \n\nRegarding the experiment with other LM architectures, we agree that our experiments can be extended to cover other LM models such as GPT in future work. The suggestion of using smaller datasets to pre-train other models sounds interesting. We will consider that in our future work.\n\nThank you for pointing out the missing references. We will add them in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons to Accept:\n\nNovelty: This paper studies a novel (insightful) question that is quite relavant to the NLP community.\n\nSupporting experiments: The authors did an excellent job at performing supporting experiments to explore the proposed study.\n\nWell written paper: The paper is well-written and easy to follow.\n\nFuture improvements suggestions:\n1. If possible, the authors can also experiment with the same protocol on downstream tasks i.e using single, two and three characters and measure the performance on them. This can bring out additional insights.\n\n2. Smaller sizes of LMs: The authors can also study for smaller sizes of LMs and evaluate how much size of a LM matter.", "reasons_to_reject": "There are some weaknesses that are major (needs additional experiments) weaknesses.\n\nGeneralization: The authors experiment with just BERT for the proposed study on LMs. Findings on one model may not be used to generalize across all the LMs. Although the authors argue this in limitations, there is a work around for this. Instead of using a big corpus like BookCorpus, the authors can pre-train on a subset of this data which wouldn't be computationally costly. In addition, the authors may consider smaller datasets used in [1].", "questions_to_authors": "", "paper_topic_and_main_contributions": "What is this paper about:\nThis paper is poses an interesting question about LMs: Do they need all the characters in a word to understand the meaning? To explore this, the authors pre-train BERT with one two and three characters of the entire word. The different pre-trained models are then fine-tuned on GLUE, SuperGLUE and linguistic tasks to evaluate their performance. \n\n\nMain Contributions:\nFocuses on important questions: This paper explores an interesting question related to language models: How much information in a word is required for LMs? \n\nSufficient methodology: The methodology introduced in the paper is sufficient to understand the problem this paper deals with.\n\nUsefulness to the NLP community: The findings that LMs perform well even with single and two characters of a word is an interesting finding to the community.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Madasu, A. and Srivastava, S., 2022, December. What do Large Language Models Learn beyond Language?. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 6940-6953).\n\n2. Wu, Yuhuai, Felix Li, and Percy S. Liang. \"Insights into pre-training via simpler synthetic tasks.\" Advances in Neural Information Processing Systems 35 (2022): 21844-21857. "}}, {"review_id": "zRBpoXVrtE", "reviewer": "Reviewer_tAA1", "summary": "", "questions": "A) Are there exactly 50,010 unique words in the corpus or is this truncated somehow? What happens to words not seen in pretraining but seen in finetuning on GLUE/SuperGLUE?\nB) Do any of GLUE/SuperGLUE tasks have different vocabularies in output space or are they all classification tasks like NLI? If the former, then are not most comparisons unfair, similar to how comparing perplexities across tokenizers with different vocabularies is unfair?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive feedback.\n\nWe experimented only with English due to limited computational resources. Expanding our experiments to cover other languages is an important avenue for future work.\n\nWe appreciate your suggestion to enhance the Discussion section by considering the practical downstream tasks that can make use of the outcomes observed in our experiments. We will extend the Discussion section based on the results observed given the extra space in the camera-ready version.\n\n**Response to Q A:** This is truncated as we use the most frequent 50,000 unique words in addition to 10 tokens for punctuation marks and special tokens.\nAny words not in the vocab are replaced with the unknown “<unk>” special token.\n\n**Response to Q B:** The GLUE/SuperGLUE benchmarks cover different types of tasks, and they exhibit varying characteristics in terms of the output space vocabulary. While some of these tasks are indeed classification tasks similar to NLI, there are also tasks that involve different output vocabularies. We will bring the full results from the appendix to the main body of the paper, as you suggested, given the extra space in the camera-ready version. \n\nThank you for the suggestion to show the majority class baselines. We will add them in all tables given the extra space in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Despite the proposed variants not occurring naturally (nor obviously outperforming the baseline), this is an interesting experiment that sheds light on how English language non-uniformly (first > middle, last) and redundantly, which can mostly be comprehended even when most of the characters are removed.\nOne way to interpret this paper is as a simulation of several (below average) human learners (i.e., BERTs) being trained to learn English and take tests (GLUE/SuperGLUE) but with access to only a small set of characters in each word.\n\nAlternatively, their stance of \"testing the limits of PLMs of learning with partial information\" is also a worthwhile research question for which we find valuable answers to, albeit only on English.", "reasons_to_reject": "I contest the fairness of the evaluation leading to their \"main results\" since the output space is different for different settings on GLUE/SuperGLUE where the answer is natural language.\nThankfully, the corresponding fair results are in the Appendix (the setting where output vocabulary is kept constant).\n\nTheir claims are exaggerated to properties \"natural language\" in general, even though the experiments are only on English.\n\nThe motivation is not very strong, the reader may be left wondering \"so what\"? Discussion section can be expanded to consider practical downstream (even if far fetched) ways to exploit the results observed in these experiments.", "questions_to_authors": "A) Are there exactly 50,010 unique words in the corpus or is this truncated somehow? What happens to words not seen in pretraining but seen in finetuning on GLUE/SuperGLUE?\nB) Do any of GLUE/SuperGLUE tasks have different vocabularies in output space or are they all classification tasks like NLI? If the former, then are not most comparisons unfair, similar to how comparing perplexities across tokenizers with different vocabularies is unfair?", "paper_topic_and_main_contributions": "The authors pretrain BERT-base on English corpora with different schemes to trim the input vocabularies, e.g., representing every word with its first character, or just the last character, or both, or first-middle-last, or only vowels, or only consonants, etc. This is inspired by studies showing how humans can read very well with partially boldened words, even with mispellings.\n\nThey pretrain and finetune each model in two settings: (1) the output is also collapsed into the resulting vocabulary from the corresponding changes to input, or (2) the output vocabulary is left intact. Their baseline is not BPE or WordPiece but a word-level BERT which includes all words in the vocabulary.\n\nTheir major contribution is to show that despite losing a lot of the information by replacing words with only a subset of characters (as few as one character per word), they can achieve close to baseline performance, not only on MLM intrinsic task but also on GLUE and SuperGLUE (even the original task setting, where the output vocabulary stays same).\n\nThey also show how first characters are more informative than the middle or last, and that their variants capture more syntactic information than even the baseline.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be helpful to show % of baseline alongside most results on BLEU, etc."}, "extra_text": {}}, {"review_id": "wOJXjUSbTD", "reviewer": "Reviewer_Ed3M", "summary": "", "questions": "A. For Vowels (V), what happens if a word contains no vowel?\n\nB. Is there any explanation for why L has the lowest pre-training loss?\n\nC. I would like to see the majority baseline for the probing experiments.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your constructive feedback.\n\n1. Regarding Section 6.2, we are going to rewrite the section to make it clearer as suggested. “cut all words to keep the threshold” means that we keep only the specified characters for each token and remove any other characters if they exist.\n\n2. In our experiments, we pre-train two types of models using two different pre-training objectives. The first one is to pre-train on partial tokens and predict the same partial tokens. The second one is to pre-train on partial tokens and predict the full tokens. We agree that this is slightly different from the cognitive science motivation but we consider it as a more simple setting that allows us to explore the predictive capacity of LLMs under information loss. Thank you for the suggestion to fine-tune the full token model on downstream tasks using partial tokens {F, M, L, FF, FL, etc.}. This is really an interesting experiment and it is easy to implement. Intuitively, we expect the full token model to achieve very good results when fine-tuned on downstream tasks with partial input tokens. We are going to include this and report the results in the camera-ready version of the paper.\n\n3. We agree that when keeping three characters (FFF, FML and LLL) of the input tokens, some pronouns and auxiliary verbs do not suffer any information loss. However, many other verbs and nouns contain more than three characters and therefore suffer from information loss. In addition, the probing tasks we utilized in our paper cover a number of syntactic and semantic information tasks such as SubjNum which predicts whether the subject of the main clause is singular or plural. The suggestion to include error analysis for the probing experiments sounds interesting. We can address that in the camera-ready given the extra space.\n\n**Response to Q A:** For Vowels (V) models, if a word contains no vowel, it will be dropped completely from the sentence. However, this is very rare as the vast majority of words in English contain vowels. We will better clarify this in the camera-ready.\n\n**Response to Q B:** This is most likely due to the initialization processes as all three models (F, M and L) have the same architecture and pre-training setup.  The combination of starting parameter values and random initialisation might help the L model achieve a slightly lower pre-training loss compared to the F and M models. However, when we finetune the L model on downstream tasks we observe that it achieves the lowest average score compared to the F and M models. We will make this more clear in the discussion of the results.\n\n**Response to Q C:** Thank you for the suggestion. The majority baseline for the probing experiments will be added in the camera-ready version of the paper.\n\nThank you for pointing out the missing references for each dataset in GLUE and SuperGLUE. This will be addressed in the camera-ready version.\n\nThe presentation of the Figures in the Appendices will be enhanced in the camera-ready version. Thanks for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper has a clear takeaway, and the topic is never explored before. They show that we can pre-train a PLM that still works reasonably on downstream tasks even when the input tokens contain a few characters.\n2. The paper is well-written", "reasons_to_reject": "1. Some parts of the paper are difficult to understand. Precisely, the whole Section 6.2 is unclear to me when I first read it. The phrase \"`cut all words to keep the threshold`\" is odd. I thought what this paper does is dropping characters from a word, but \"`cut all words`\" sounds like you are removing words from the sentences that are longer than the threshold.\n2. The results of the paper may not be directly related to how current PLMs learn, so it does not really push our knowledge of existing PLMs a step further. The results in this paper can only tell us PLMs can still learn something when input characters are dropped. However, since existing PLMs do not pre-train or fine-tune on character-dropped datasets, we still do not know if PLMs trained on full tokens can comprehend character-dropped inputs. The experiments in this paper are quite different from the cognitive science motivation. In cognitive science, the takeaway is that humans learning from full tokens can understand partial words, but the experiments in this paper show that models pre-train on partial words can comprehend partial words. To address this issue, a simple experiment is fine-tune the full token model on downstream tasks whose inputs are {F,M,L, FF, FL, etc.}.\n3. The results of the probing experiments are not as surprising as the paper states. This is because when keeping the first, middle, and last characters (FML) in the sentence, a lot of information for syntactic and semantics information is kept. This is because many pronouns are less than three characters (e.g., he, she, it, we), so FML does not really drop any characters in those words. This is also the case for the verb be (e.g., is, are, was) and auxiliary (do, did). I think the papers should conduct a more detailed analysis of the error pattern of the probing experiments. Is the probing accuracy higher for sentences whose verbs and pronouns are not character-dropped?\n\n", "questions_to_authors": "A. For Vowels (V), what happens if a word contains no vowel?\n\nB. Is there any explanation for why L has the lowest pre-training loss?\n\nC. I would like to see the majority baseline for the probing experiments.", "paper_topic_and_main_contributions": "Topic\n===\nThis paper studies if pre-trained language models (PLM) can work even if characters in the tokens are dropped. They pre-train different PLMs by keeping different numbers of characters (one, two, or three) and different positions (first, last, or middle) in the input token. After pre-training, they fine-tune those models on GLUE and SuperGLUE, and the inputs during fine-tuning only contain some characters of the original word, as in pre-training. They find that even when only keeping a few ($\\leq 3$) characters of the word, the PLM still performs quite well on downstream tasks. Probing experiments reveal that even when most characters are dropped from a word, the hidden representation of the PLM still encodes some linguistic information. \n\nContribution types\n===\nComputationally-aided linguistic analysis, NLP engineering experiment", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Each dataset in GLUE and SuperGLUE should be cited in Section 4.5.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Words in the Figures in the Appendices are too small to read.\n- Line 361: trained **on** partial tokens"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores how robust language models are to missing input token characters. Their results are surprising: they find that language models retain high performance when pre-trained with extreme character loss. The reviewers agree that this is a novel research question, and, on the whole, that the methodology is sound. There is a concern about the reliability of the results given that only BERT is studied, but BERT remains a very commonly used model (particularly in computational humanities and social science), so I think this concern is outweighed by the computational costs of additional experiments, as acknowledged by the reviewer who raised this concern. I do agree that it is important to revise the text to clarify the scope of the findings (English).
Decision: Accept-Main