Text Rendering Strategies for Pixel Language Models
Authors: Jonas F. Lotz
Elizabeth Salesky
Phillip Rust
Desmond Elliott
Venue: EMNLP 2023 Main (2023)
Keywords: Pixel-based language modelling
isotropy
word frequency bias
TL;DR: Compressing the input space of a pixel-based language models enables better semantic performance

Abstract: Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling. However, recent approaches use text renderers that produce a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks, due to redundancy in the input representations.
In this paper, we investigate four approaches to rendering text in the PIXEL model (Rust et al., 2023), and find that simple character bigram rendering brings improved performance on sentence-level tasks without compromising performance on token-level or multilingual tasks.
This new rendering strategy also makes it possible to train a more compact model with only 22M parameters that performs on par with the original 86M parameter model.
Our analyses show that character bigram rendering leads to a consistently better model but with an anisotropic patch embedding space, driven by a patch frequency bias, highlighting the connections between image patch- and tokenization-based language models.

Text Rendering Strategies for Pixel Language Models
Jonas E. Lotz Elizabeth Salesky* Phillip Rust Desmond Elliott
*Department of Computer Science, University of Copenhagen 'ROCKWOOL Foundation Research Unit *Johns Hopkins University jonasf lotzēdi. ku. dk
Abstract
(a) Continuous rendering (CONTINUOUS):
ust
be
growi
ng
small again.
Pixel-based language models process text ren- dered images, which allows them to han- dle any script, making them promising ap- proach open vocabulary language modelling. However, recent approaches use text renderers that produce large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks, due to redundancy in the input representations. this paper, we in- vestigate four approaches to rendering text in the PIXEL model (Rust et al., 2023), and find that simple character bigram rendering brings improved performance on sentence- level tasks without compromising performance on token- level or multilingual tasks. This rendering strategy also makes it possible to train more compact model with only 22M parameters that performs par with the original 86M param- eter model. Our analyses show that character bigram rendering leads to a consistently better model but with an anisotropic patch embedding space, driven by a patch frequency bias, high- lighting the connections between image patch- and tokenization-boe based language models.
(b) Structured rendering (BIGRAMS):
must
st
be
ow
sm al ag ai
(c) Structured rendering (MONO):
mu
st
b
jow
in
|sm al
ag
(d) Structured rendering (WORDS):
must be
growingg sm
aggiin.
Examples of rendering strategies for the sen- tence "7 must be growing small again." " from Carroll (1865). Black patches mark the end of sequence, fol- lowing Rust et al. (2023).
Figure I:
sliding window (Salesky et al., 2021, 2023) in speech processing. Although the proposed ap- proaches have the appealing properties of yielding compact and transferable representations, they also result in a very large input space because there is unique way to represent lexical units. As a consequence, pixel-based models could observe set of image representations with every sentence, which adds redundancy in the input space and is sub-optimal for developing contextual language representations. We refer to these unstructured rendering strategies as CONTINUOUS and illustrate the point qualitatively in Figure and Figure 2, and quantitatively in Figure 3. In this work, we ask whether structuring the input, which leads more frequent parameter updates through now-unique word representations, would enable pixel-based models develop deeper understand- ing of context and semantics. We then propose rendering strategies structured around providing the model with a compressed input space.
1 Introduction
There is growing movement in NLP towards tokenization-free methods (Clark et al., 2022; Xue et al., 2022; Yu et al., 2023) including pixel-based representations of text (Salesky et al., 2021, 2023; Rust al., 2023; Tschannen al., 2023). It has been shown that these tokenization-free methods can readily handle unseen languages and that they are more robust to noise attacks than tokenization- based models. In addition, pixel-based approaches can effectively exploit visual similarities between characters and scripts because they allow for com- plete parameter sharing across all inputs, making them promising direction for multilingual NLP. Previous work on pixel-based models segments the rendered text into either consecutive patches (Rust et al., 2023; Tschannen et al., 2023) or with
We demonstrate how enforcing BIGRAMS- structured rendering strategy leads to both more capable and data-efficient model: when evaluated on semantic sentence-level tasks, we find that a 22M parameters model performs

---

Most frequent patches with CONTINUOUS rendering:
continueus
bigrams
meno
the
the
the
the
the
the
he
thé
the
(b) Most frequent patches with BIGRAMS rendering:
umique patches
200k
d
150k
Figure 2: A continuous rendering strategy results in
100k
many mmiquely-valuee image patches for similar inputs, while structured rendering (here, BIGRAMS) regularises and compresses the potential input space.
50K
6k
2k
62 Sentences observed
4k
8k
10k
competitively with the unstructured original at 86M parameters, and that scaling back up to 86M parameters narrows the performance gap to BERT (Devlin al., 2019) trained on the same data. In subsequent analyses, find that the added input structure provokes clear visual token frequency bias in the learned embedding space. While also found BERT, frequency biases have been shown to degrade the quality of embedding spaces when word representations are not only determined by semantic relations but also by the number of model updates (Gong et al., 2018; Gao et al., 2019; Fuster Baggetto and Fresno, 2022). We show that frequent words have more context-specifie representations than infrequent words, especially in the upper layers. Finally, we show that PIXEL models acquire non-trivial semantic understanding during pretraining, but that their sentence representations are easily influenced by this frequency bias. We release all models and 2ode2 for pretraining and finetuning.
Figure 3: Number of unique image patches observed as a function of training data sequences. Structured rendering results in greater representational efficiency.
the closest benchmark for PIXEL. By using the Google Noto font family which supports the major- ity of Unicode codepoints," 4 the renderer supports all languages that can currently be typeset.
Before the first layer of the PIXEL model, image patches are linearly projected obtain sequence of patch 'embeddings", During pretraining, 25% of embeddings are masked in spans of up to patches and only the unmasked patches with a prepended CLS embedding are passed through the encoder. Af- ter replacing the masked embeddings amidst the encoder outputs, relying on fixed sinusoidal posi- tion embeddings for ordering information, the de- coder predicts the pixel values of solely the masked patches. To later finetune the encoder on a classi- fication task, the decoder can be replaced with a cusksspccfiic head and the masking ratio set to 0%.
2 Background: modelling text as images
We build upon the general-purpose language en- coder framework presented in Rust et al. (2023): PIXEL is text autoencoder which builds the Masked Autoencoding Vision Transformer (ViT- MAE; He al., 2021) and is similarly pretrained with masked reconstruction objective. However, instead of patches from natural images of objects (Deng et al., 2009), the patches now contain im- ages of text. To go from text to images of text, PIXEL relies on rendering library (PangoCairo)) produce sequence-level image which is sliced into image patches of size 16 x 16 pixels. The sequence-length maximum of 529 patches approxi- mately equals the memory requirements BERT,
Structured rendering
Previously proposed approaches rendering text as images render full sequences of text and segment into either consecutive patches (Rust et al., 2023; Tschannen et al., 2023) or with a sliding window (Salesky et al., 2021, 2023). These CONTINUOUS strategies result in significant number of uniquely-valued patches, many of which may be observed only once during training. We depict this redundancy in Figure 2 and quantify in Figure 3, showing how similar text inputs result in unique visual representations.
We compare four rendering strategies: the orig- inal unstructured (CONTINUOUS), and three struc- tured (WORDS, MONO, BIGRAMS), as depicted in Figure 1. To render WORDS we separate
"httos://hugginggfee waaeee.c///eeeeeeeeee co/Team-PIXEL TextRenderingStratteiis /tttss://docs..o http://bbcs..gkkkgggagggggaaaa
/nttps://fonss.gonn moogle..co/nonn

---

ments with additional whitespace" such that new segments begin the beginning of the next im- age patch, regulating possible spatial variation. BI- GRAMS, rendering two characters per image patch, is chosen to be widely applicable, without knowl- edge of word or morphemic segmentation (Mielke et al., 2021; Keren et al., 2022). More specifically- consider the word pairs ("grow". "growing") and ("growing", "walkingg)) BIGRAMS renderer will produce an overlap of image patches (under- lined) for both pairs while the same extent is not guaranteed with WORDS-level rendering as it is reg- ulated by character width. The choice of character (n = 2) -grams is motivated by what generally fits within 16 16 pixels image patch in the setup from Rust et al. (2023). MONO instead applies monospaced fonts where each character is fixed width; depending on font size, this may result in character bigram patches without breaks within characters, but this is not guaranteed. The main difference between BIGRAMS and MONO is that MONO simply slides across the sentence, two char- acters at the time, yielding two ways to represent a word whereas BIGRAMS renders the words and then pads with whitespace, ensuring unique inputs." 6 As seen in Figure 3, the structured rendering strategies result in greatly compressed input space as measured by the number of unique im- age patches processed by the model, but Figure 1 reveals that it comes at the cost of longer sequence lengths. While the rendering strategies we propose were not specifically designed for English, they may not equally generalise to other languages or scripts. We further discuss the representational effi- ciencies of these strategies in A.l and limitations to generalisability under Limitations.
Model
Enez-Dect
Hid
MLP
Alt
[0]
BASE
12-8
768
3072
12
86M
SMALL
12-4
384
1536
6
22M
TINY
12-2
192
768
3
5.5M
Table I: Details PIXEL model scale variants.
ing performance. While "Base" models and larger ones are widely used for their strong performance, proven scaling laws (Touvron et al.. 2021; Zhai et al., 2021) enable greater experimentation and model development at smaller scale (lvgi et al., 2022), which is both more envirommentally friendly (Strubell al., 2019; Bender et al., 2021; Hersh- covich et al., 2022) and facilitates contributions with limited computational resources.
With this in mind, we propose two smaller architectures which we will compare across down- stream tasks in $ 5. Our BASE model architecture is directly adopted from ViT (Dosovitskiy et al., 2021) and PIXEL, and we add two more compact SMALL and TINY model variants, as described in Table 1. The configurations of the smaller models are based on the ViT variants presented in Zhai et al. 12 Following the scaling experiments in He et al. (2021), indicating that shallow decoders of as small as layers can be sufficient for ViT-MAEs, we apply a scheme of halving the number of decoder layers every scale reduction.
Experiments
We pretrain SMALL models with the proposed ren- dering strategies. The models are then evaluated on dependency parsing (UDP) with data from Uni- versal Dependencies v2.10 treebanks (Zeman et al.. 2022; Nivre et al., 2020) and GL.UE (Wang et al., 2018), exploring the models' capabilities syn- tactic processing on the word level and semantic processing on the sentence level.
4 Model scale variants
Recall from Figure 3 that CONTINUOUS rendering produces significantly larger set of unique image patches compared to other approaches. A conse- quence of this is that models must learn to encode many almost-identical visual representations, which may be wasteful, both in terms of parameters and training efficiency. Therefore, we hypothesise that PIXEL models that operate over fewer unique image patches can be scaled down without sacrific-
5.1 Prefraining
We pretrain all models on the English Wikipedia and Bookcorpus (Zhu et al., 2015) data used by Rust et al. (2023) for direct comparison with PIXEL and BERT, which results in ~16.8M training ex- amples. We follow the suggested hyperparameters used for PIXEL with the exception of batch size. The smaller architectures of SMALL and TINY al- low for larger batch sizes, which we double from 256 examples to 512 and 1024, respectively. We then halve the number of pretraining steps accord-
"e render whitespace minimum 3 pixels wide, sometimes resulting in blank patch between tokens structured inputs. As an example, "be" in Figure is split into 2 image patches with MONO rendering. Depending on the context, it could also be represented in a single image patch.

---

Structure
Scale
UDP Avg.
GLUE
UDP
GLUB
TyDiQA WDDiA--GoldP
Renderer
Avg.
Variant
[0]
Ang.
Ap
Avg.
Ay
Avg.
CONTINUOUS
76.2
71.0
TINY
5.5M
72.0
0.3
66.5
+12.7
41.6
+4.9
RIGRAMS
76.1
75.4
SMALL
22M
76.1
0.1
75.4
+4..4
50.8
+2.0
75.9
74.4
86M
75.5
-0.6
78.0
+3.9
52.8
+0.5
WORDS
76.6
74.7
BERT
110M
50.5
80.0
51.5
Table 2: Structure (left): averaged results for SMALL- comparing downstream performance on UDP and GLUE following the different rendering strategies. Scale (right): averaged results across model scales using the BIGRAMS rendering structure. Aμ is the difference in average performance between BIGRAMS and CONTINUOUS rendering for given model scale, BERT results are marked grey visually distinguish from pixel-based models.
ingly from 1M to 500k and 250k in order to train for the same number of epochs as PIXEL (~16 epochs, but varying slightly due to differing se- quence lengths per rendering strategy).
5.3 Rendering strategies
We present averaged results comparing the render- ing strategies in the left part of Table 2. Detailed results for each downstream task presented in Table 4 and Table 5 in the appendix. For UDP we find that the WORDS structure slightly outper- forms BIGRAMS and MONO on this word-level task. When comparing the WORDS and CONTINUOUS strategies we get first hint as to the importance of including structure during pretraining as well, keep- ing in mind that the rendering structure is the same for both strategies when finetuning UDP. For GLUE we large increase in performance when rendering with any structure and especially BI- GRAMS. We attribute the difference performance between BIGRAMS and MONO to the unique word representations with BIGRAMS, as discussed in $ 3. We find that BIGRAMS is the best performing structure on average, even slightly outperforming the 86M parameters PIXEL (average UDP: 76.1; average GLUE: 74.1) with only 1/4 its model parameters. We provide an investigation into the mechanisms that enable this improved performance on GLUE in $ 6.4. Next we pretrain TINY and BASE model variants with BIGRAMS rendering to evaluate performance at different model scales.
Pretraining BASE takes days on 8 x 40GB Nvidia A100 GPUs, while comparison, pretrain- ing SMALL takes less than 48 hours on x 40GB Nvidia A100 GPUs, and TINY less than 24 hours. Loss trajectories for the different rendering strategies are in line with their representational efficiency (Figure 3), indicating that structured rendering may make the masked reconstruction task more data-efficient, achieving low loss in fewer steps (see & A.2: Figure 10).
5.2 Finetuning
To finetune our models for classification tasks we replace the decoder used for pretraining with a task-specific classification head. We do not search for more optimal hyperparameters than those used for PIXEL with the exception of the learning rate; we find that the more compact architectures often benefit from slightly higher learning rate.?
We follow the same protocol during finetuning as done for PIXEL: for word-level tasks we obtain the rendered image patch indices for every word and as a consequence, the CONTINUOUS strategy becomes identical to the WORDS structure when finetuning UDP. $ 6.1 further investigates the consequence of mismatch between how the data is structured during pretraining and finetuning. When finetuning on GLUE the structure follows what was seen during pretraining for all rendering strategies. Reported performances for BERT and PIXEL are taken from Rust et al. (2023).
5.4 Model scaling
The right part of Table 2 compares the different model scales all following a BIGRAMS rendering strategy. Detailed results are likewise presented in Table 4, Table 5, and Table 6 in the appendix. We find that the TINY configuration performs competitively the word-level tasks considering its only 5.5M parameters. but has larger gap up to SMALL and BASE on the sentence-level GLUR tasks. SMALL proves be a good trade-off between scale and performance where is not far behind BASE on GLUE and even slightly
WWe search the space (le-5, 3,-5, 5,-5, 7e-5, 9e-5) and report the average over soeds.

---

outperforms on UDP. BASE comes step closer closing the gap performance up BERT on GLUE. Comparing to the performance following a CONTINUOUS rendering strategy, summarised as the difference average performance (Aj1), it is clear that the more compact the model size, the greater the benefit from structured rendering.
RENDERER
GLUE
Pretraining
Finetuning
Avg.
BIGRAMS
BIGRAMS
75.4
CONTINUOUS
CONTINCOUS
71.0
CONTINUOUS
BIGRAMS
61.1
BIGRAMS
CONTINCOUS
53.0
To verify that BIGRAMS rendering does not degrade the performance on multilingual sentence- level tasks across different scripts and morpholo- gies, we also include results on TyDiQA-GoldP (Clark et al., 2020).' 9 Again we find that SMALL performs competitively considering its size.
Table 3: Rendering strategy combinations between pre- training and finetuning with SMALL models. For GLUE, matching pretraining structure is most effective.
demonstrate that BASE-BIGRAMS outperforms PIXEL by 3.6 points on average on MasakhaNER (Adelani et al., 2021), named entity recognition benchmark for 10 African languages. This further illustrates the potential of PIXEL-based models for modelling low-resource languages. Detailed results are presented in Table in the appendix. We next turn our attention to how BIGRAMS rendering enables better performance on semantic tasks.
6 Ablations and supplementary analyses
this section we investigate how BIGRAMS rendering changes the model compared to CON- TINUOUS. For clarity in what follows, we refer to the BASE model with BIGRAMS rendering from $ 5.4 as BASE-BIGH BIGRAMS and keep referring to the original model from Rust et al. (2023) PIXEI..
6.2 Contextual representations
6.1 When does rendering structure matter?
The extent which language models capture se- mantic information is partly determined by their ability to contextualise text (Peters et al., 2018). We therefore analyse how capable LASE-BIGRANN is at producing contextualised word representations. We use the Words in Context dataset (WiC; Pile- hvar and Camacho-Collados, 2019) of sentences that contain target words (noun verb) in either similar (True) different (False) context across sentence pairs. We compute the mean hidden state output over all tokens associated with the target word to obtain a representation. We infer that there is contextualisation the model gener- ates representations of a target word from different contexts with a low cosine similarity compared to target words in similar contexts. We report this indication of contextuality for each layer of the model, including the input layer, better under- stand the properties of the different layers. Similar- ities between randomly chosen words from random examples (Random) are included as a baseline." Figure 4a plots the resulting distributions of similarities. We see that representations of target words from similar contexts have higher cosine similarity than from different contexts, though with
Having established that a structured rendering strat- egy leads to improved downstream performance, we further investigate when it is needed: is it suf- ficient to finetune with structure or does the model develop strategy-speciff features during pretrain- ing? We analyze this by comparing rendering strategies between pretraining and finetuning.
The results in Table 3 for GLUE show that mis- match leads to lower downstream performance for both strategies, with BIGRAMS - CONTINUOUS being the most harmful, perhaps unsurprisingly. This result does not align with the finding for UDP in 5.3 where CONTINUOUS overeomes the change to WORDS-structured rendering.
may indicate that the lower-level UDP tasks are easier for Pbxet-based models than the high-level GLUE tasks (Lauscher et al., 2020). This is in line with the relatively good performance for TINY-BIGRAMS on UDP.
To emphasize the increase in performance semantic tasks with BIGRAMS rendering,
*W expect that could prevail and would benefit from wider search for optimal hyperparameters during finetuning. "With the CONTINUOUS rendering strategy, answer spans are extracted such that the answer may include leading or trailing characters when there is no exact mapping from a word to an image patch index. Therefore, we did not include TyDiQA- GoldP in the comparison in $ 5.3. More details can be found in Rust et al. (2023). We discuss limitations to answer span extraction with BIGRAMS rendering in $ A.4.
Target words are not necessarily identical across sentence pairs can vary e.g. in conjugation or number. It is not possible to obtain an exact mapping from words to neat image patch indices following the CONTINUOOS ren- dering strategy so we do not present this analysis for PIXEL.

---

0.8
0.8
Cosine similarity
0.6
0.4
0.4
Similar context
0.2
True
0.2
False
Random
1
10 11
2
11
Layer depth
Layer depth
(a)
(b) BERT
Figure 4: Distributions of cosine similarities for verbs and nouns from the WIC dataset across model layers 0-12. layer () being the input layer. Every example presents a target word either a similar or different context across sentence pair. The representation of the target word is computed the mean hidden state output the corresponding tokens. We generally see that BASE-BIGRAMS encodes target words in similar context more similar. The median cosine similarity between random words from random sentences are shown as a baseline.
considerable overlap, and higher for different con- texts than for random. When comparing to BERT in Figure 4b, there is clear difference in the similar- ity compared to random words. The difference in similarity between similar and random words grad- ually increases throughout the BASE-BIGRAME model, until the final layers, whereas the difference steadily decreases throughout the model for BERT. Given the shared image patch embedding layer in Paxel-based models, random words are more similar each other the input layer when modelled images than entries vocabulary.
an expressive word embedding space (Arora et al., 2016; 6; Mu and Viswanath, 2018)."
Recent work has shown that Transformer- based language models can develop a representation bias driven by token frequency, where Now-frequency to- kens are clustered together in the embedding space, leading to anisotropy in the model (Gao et al.. 2019; Fuster Baggetto and Fresno, 2022; Jiang et al., 2022). This bias leads poor word contex- tualisation because the learned vector positions of low frequency words have not moved far from their random initialisation. Thus, their embeddings are not sufficiently distinct from unrelated words with similarly low token frequency (Gong et al., 2018; Cai et al., 2021). Tokens with a higher frequency, and thus more parameter updates, can move further in the embedding space from their initialisation and become more semantically meaningful. Conse- quently, hypothesise that compressing the input space in the form of structured rendering allows the model to build more contextualised word represen- tations through more frequent parameter updates.
Taken together, these plots suggest that PIXEL- based language model is capable of forming con- textualised word representations and that these more sontext-specific in upper layers, though not fine-graine -reigra as seen for BERT.
6.3 Token frequency and similarity
The degree of cosine similarity between random words observed Figure 4a encourages us to assess the isotropic nature of the model (Etha- yarajh, 2019; Rajaee and Pilehvar, 2021). The high cosine similarities suggest that the word representations are not evenly distributed with respect to direction in the embedding space, but instead appear be anisotropic. When learned vector representations populate narrow cone in the embedding space, this geometric alignment leads to an overestimation of their similarity (Gao et al., 2019), which is not an expected property of
We investigate this by sampling inputs that were seen during pretraining with high and low fre- quency. Specifically, we take the 100 most fre-
Following Cai et al. (2021) this global estimate of ansiotropy does not rule out the possibility distinct and locally isotropic clusters in the embedding space. Ding et al. (2022) show that isotropy calibration methods (Gao et al., 2019; Wang et al., 2020; Li et al., 2020) do not lead to consistent improvements on downstream tasks when models already henefit from local isotropy. We leave this direction for PIXEL to future rescarch.

---

Comparing lligh,
Low; Low
Denity
High.
4-
2-
0
0.6 Cosine similanity
1.0
0.2
1.0
Cosine similarity
Cosine similarity
(a) Words isolation, PIXEL
(b) Words in isolation,
(c) Words context, BASE-BIGRAMS
words (High), low- cowffreuenccy words (Low), or between the two samples. Rendering with BIGRAMS structure leads to less directionally aligned vector representations of frequent words that have seen more updates during pretraining compared to infrequent words.
Figure 5: Distributions of cosine similarities within samples of
quently occurring words from the Wikipedia cor- pus that was seen during pretraining and 100 words that occur around 1000 times (rank x 50k). 13 We first render each word from the two frequency sam- ples in isolation. We then include a comparison to words in context across 100 unique sentences per word with BASE-BIGRAMS.
aforementioned challenges in aligning patch embeddings with CONTINUOUS rendering.
6.4 Frequency bias and semantic modelling
While there is less evidence of representation degeneration with CONTINUOUS rendering, it is likely that the poorer performance on GLUE in $ 5,4 is caused by PIXEL seeing too many different patches too few times. This is direct consequence of the multitude of ways that similar inputs can be rendered by the CONTINUOUS approach. However, the drop in performance when mismatching the rendering strategies in $ 6.1 for CONTINUOUS + BIGRAMS demonstrates that the model has devel- oped a set of strategy-specific expectations and features that are not easily updated. In fact, the new rendering strategy for finetuning introduces set of patches that likely never escape the low-frequeenc domain and therefore remain poorly contextualised. Signs of token frequency bias has also been found in BERT (Fuster Baggetto and Fresno, 2022).
We plot the distributions of cosine similarities between representations from the last encoder layer, where we expect embeddings from both models to be contextualised. Comparing the plots from the two rendering strategies, summarised in Figure 5, the effect of pretraining with smaller set of unique tokens becomes clear: for PIXEL the distribution appears mixtures with a larger distribution mass at higher values of cosine similarity from comparing high-frequency words to other feggefrreueecy (excluding self-similarity for now) than when comparing low-frequency to other low-frequency. For BASE-BIGR AMS the frequent words both in isolation and in-context are less directionally aligned with each other compared to the infrequent, which is in line with the represen- lation degeneration problem from Gao et al. (2019) and more frequent updates leading to better con- textualisation. Figure visualises the in-context representations in 2 dimensions using t-SNE (van der Maaten and Hinton, 2008) and provides an additional indication of more frequent words having less locally compact representations. 15
We lastly assess the connection between visual token frequency and downstream semantic performance. With BERT, high-frequeney words have the most context-specific representations (Bthayarajh, 2019), and upper-layer represtttios of low-frequency words are influenced more by their context than frequent words (Voita et al., 2019). Following Ethayarajh (2019), we see that this applies to BASE-BIGRAMS as well (illustrated in Figure 7 and discussed in greater detail in A.5). We expect that sentences that only vary in being cased or uncased would result in different representations when lowercase appears more frequently (for most words). This demonstrates the impact of observed token frequency on semantic modelling and in line with observed biases BERT'S embedding space (Jiang et al., 2022).
We expect that in-context representations from PIXEL also qualitatively resembles Figure 5a but cannot easily demonstrate this due the
Excluding punctuation and numbers.
14 Recall from that CONTINUOUS rendering strategy by design makes an exact mapping from words sentence to neat image patch indices unattainable. 15 "Plotting the first 2 singular values from a singular value decomposition gives the same qualitative indications.

---

1.0
Spearman's correlation
1.0
High teq.
Continuous
Low
Cosine similarity
Bigrems
Bigrams, ubcased
0.5
0.5
Similarity
Swif
Intra-senlatcon
High freg
High Ereq Low freq
Low frwg.
0.0
0.0
10
0
10
-200
-200
200
Layer depth
Layer depth
Figure 6: t-SNE plot of the output embeddings of high- and low- frequency words in context from BASE-BIGRAME Low-frequency words cluster tightly in this space.
Figure 7: Self- and intra-sentence similarity from BASE-BIGRAMS, High-frequeng words are the most context-specific; low-frequency words influenced by their context.
Figure 8: Evaluation performance on STS-B. Uncased sentences yield better performance than the original with BASE-BIGRAME the effect is less clear for PIXEL (not shown).
We rely on the Semantic Textual Similarity Benchmark (STS-B; Cer et al., 2017) also found in GLUE for this assessment. We measure the cosine similarity between sentence representations" and plot its correlation with the gold standard similarity scores as the measure of performance. Figure 8 proves that both CONTINUOUS and BIGRAMS ren- dering during pretraining lead to non-trivial seman- tic modelling capabilties. At peak performance, around the middle layers, the increase from sim- ply ensuring that all words are uncased is roughly the same as the increase from PIXEL to BASE- BIGRAMS. This resembles how frequent and in- frequent tokens have unequal influence their context in BERT (Voita et al., 2019).
related to ours, after Rust et al. (2023), the work on machine translation with pixel representations (Salesky et al., 2021, 2023). A detailed discussion of previous pixel-based approaches can be found in Rust et al. (2023, $ 5). Where PIXEL laid the foun- dation for general purposs language encoding with pixel-based representations, this work takes the first step towards hypothesis-driven improvements without adding additional data (Yang et al., 2019) or scaling up the model (Conneau and Lample, 2019). Though is possible that competitive performance could be achieved by model with CONTINUOUS rendering by pretraining on more data for more steps (Liu al., 2019).
Our addition of BIGRAMS structure resembles the addition of optional but hugely beneficial (n 4)-grams in the character-based CANINE model (Clark et al., 2022). While character-level n- n-gram models (Wicting et al., 2016; Bojanowski et al., 2017) have been succeeded by Transformer- based language models, character-level features remain valuable they are less sparse and more robust to misspellings than word n-grams, and remain useful for especially morphologically rich languages (Garrette and Baldridge, 2013; Kulmizev et al., 2017). Previous work have hypothesised that character-level models would be more suitable than subword-based based for modelling morphologically-rict languages (Tsarfaty et al., 2020; Keren et al., 2022), but a semantically capable design has proven non-obvious (Ma et al., 2020; Keren et al., 2022; Nzeyimana and Niyongabo Rubungo, 2022; Sun al., 2023). We see potential for future work with pixel-based language models exploring appropriate strategies for learning morphological patterns (Klein and Tsarfaty, 2020; Seker and Tsarfaty, 2020; Soulos al., 2021).
Seeing that BASL-BIGRAMS exhibits similar representational traits to that of BERT, future work could aim for more semantically capable PIXEL- based models by generalising advances found for tokenizer-basse based models (Gao et al., 2021).
7
Related work
Recent work on pixel-based language modelling has demonstrated how visual language under- standing can be achieved through pixels only (Lee et al., 2022), observed that the visual similarity of languages plays an important role in cross-lingual transfer (Rahman et al., 2023), and shown how unifying the modalities for text and images allow single encoder to perform multimodal tasks (Tschannen al., 2023). By relying on bytes directly, the unification of modalities can be taken even further (Jaegle et al., 2021; Horton et al., 2023; Yu et al., 2023). The work most closely
16 Mean hidden state output across all tokens in sentence, excluding the CLS token and black cnd-of-soquence token.

---

8 Conclusion
Acknowledgements
Jonas F. Lotz funded by the ROCKWOOL Foun- dation (grant 1242). Elizabeth Salesky is supported by the Apple Scholars in AI/L fellowship. Phillip Rust is funded by the Novo Nordisk Foundation (grant NNF 20SA0066568). This work was sup- ported by a research grant (VIL53122) from VIL- LUM FONDEN.
We evaluate four text rendering strategies to ad- dress the problem of redundancy in the input space of PIXEL-based language models. Consequently, more frequent parameter updates lead to better contextualised language representations. We find that rendering two characters per image patch (BIGRAMS) a good trade-off between efficiency and generalisability, resulting in substantial im- provements on downstream semantic and sentence- level task;; contributing to open-vocabulary NLP with limited computational resources.
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big, Daniel D' souza, Julia Kreutzer, Constantine Lig- nos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sehastian Ruder, Stephen Mayhew, Is- rael Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue. Joyce Maikkktuba-Naabette Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhic Yi- mam, Tajuddeen Rabiu Gwadabe. Ignatius Ezeani, Ruhungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende, Troro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi. Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- ode, Deborah Nabagereka, Maurice Katusiim, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Abia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thiemno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Further analyses reveal how the added ren- dering structure provokes clear representational similarities to what has been found in BERT. We see potential in future work generalising improvements found for tokenization-bant based masked language models to PiXEL-based masked language models. Furthermore, considering that the Vision Transformer has also been applied to speech modelling (Huang et al., 2022), and that patch representation has been suggested to be critical component for the success of ViTs (Trockman and Kolter, 2023), see potential for image patches as the basis for unifying modalities.
Limitations
While the rendering strategies we propose here well-suited English, not all equally generalise to other languages or scripts. WORDS rendering relies on word boundaries which may not be readily avail- able or well-defined for many languages which do not mark word or sentence boundaries with whites- pace such as Thai or polysynthetic languages such Inuktitut. MONO and BIGRAMS are more gen- eral approaches, but may affect the rendering of positional characters such diacritics or correct contextual forms based on where boundaries are created. For both approaches, it may be necessary to modulate font size across languages to ensure character pairs fit into single patch, especially when rendering with diacritics. MONO provides further representational efficiency compared to BI- GRAMS by fixing character width, but comes at the cost of more limited language coverage; many scripts cannot be made fixch-width and fewer than 10 have mono fonts available. CONTINUOUS ren- dering provides a more general approach which must be balanced with learning efficiency.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transac- tions of the Association for Computational Linguis- rics, 4:385-399.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability; and Trans- parency, FAceT '21, page 610 623, New York, NY. USA. Association for Computing Machinery.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Con- ference on Learning Representations.

---

Lewis Carroll. 1865. Alice's Adventures in Wonderland. Macmillan.
Hong Kong, China, Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2011 task I: Semantic textual similarity multilingual and Proceedings
Alejandro Fuster Baggetto and Victor Fresmo. 2022. Is anisotropy really the cause of BERT embeddings not being semantic? In Findings of the Asvociation for Computational Linguistics: EMNLP 2022, pages Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
crosslingual focused evaluation. of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada, Association for Computational Linguisties.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration prob- lem in training natural language generation models. In International Conference on Learning Representa- tions.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeee -eeeantiii--eiinnngggeeg seeking question answering typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, I0, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-frit encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Dan Garrette and Jason Baldridge. 2013. Learning part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 138-147, Atlanta, Georgia. Association for Computational Linguistics.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2018. Frage: Frequency-agnostic word representation. arXiv preprint.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- Doll' ar, and Ross B. Girshick. 2021. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15979-15988
Daniel Hershcovich, Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold. 2022. Towards climate awareness in NLP research. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2480- 2494, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yue Ding, Karolis Martinkus, Damian Pascual, Si- mon Clematide, and Roger Wattenhofer. 2022. On
isotropy calibration of transformer models. In Pro- ceedings of the Third Workshop on Insights from Neg- ative Results in NLP, pages 1-9, Dublin, Ireland. Association for Computational Linguistics.
Maxwell Horton, Sachin Mehta, Ali Farhadi, and Mo- hammad Rastegari. 2023. Bytes are all you need: Transformers operating directly on file bytes. arXiv preprint.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Po-Yao Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. Masked autoen- coders that listen. In NewriPS.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 55-65,
Maor Ivgi, Yair Carmon, and Jonathan Berant, 2022. Scaling laws under the microscope: Predicting trans- former performance from small scale experiments. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 7354-7371, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.

---

dataset for evalu- Andrew Jaegle, Sebastian Borgeaud, Jean-Bay Cann-aptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H' enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. 2021. Perceiver io: A general architecture for structured inputs & oulputs. arXiv preprint.
Wentao Ma, Yiming Cui, Chenglei Si. Ting Liu, Shijin Wang, and Guoping Hu. 2020. CharBERT: Character- aware pre-trained language model. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 39-50, Barcelona. Spain (Online). International Committee on Computational Linguistics.
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang. Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi 7hang. 2022. Prompt- BERT: Improving BERT sentence embeddings with prompts. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 0826-8837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sabrina J. Mielke, Zaid Alyafeai, Flizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Sam- son Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tok- enization in nlp, arXiv preprint.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-oop: Simple and effective postprocessing for word repre- sentations. In International Conference on Learning Representations.
Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character; Are subwords good enough for mrls after all? arXiv preprint.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Stav Klein and Reut Tsarfaty. 2020. Getting the #ilife out of living: How adequate are word-pieces pieces for mod- elling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 204-209,, Online. Association for Computa- tional Linguistics.
Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan van Noord, Barbara Plank, and Mar- tijn Wieling. 2017. The power of character n-grams native language identification. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 382-389, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: a morphology-aves aware Kin- yarwanda language model. In Proceedings of the 60th Annual Meeting of the Association for Compu- rational Linguistics (Volume I: Long Papers), pages 5347-5363, Dublin, Ireland. Association for Compu- tational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Anne Lauscher. Vinit Ravishankar, Ivan Vulić, and Goran Glavaš, 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483 4499, On- line. Association for Computational Linguistics.
Kenton Lee, Mandar Joshi, Tulia Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wel Chang, and Kristina Toutanova. 2022. Pix2struct: Screenshot parsing as pretraining for visual language understand- ing. arxiv preprint.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267-1273, Minneapolis. Minnesota. Association for Computa- tional Linguistics.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 91 19-9130, Online. Association for Computa- tional Linguistics.
Md Mushfiqur Rahman, Fardin Ahsan Sakib, Fahim Faisal, and Antonios Anastasopoulos. 2023. To token or not to token: comparative study of text repre- sentations for cross-lingual transfer. arXiv preprint.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint.
Sara Rajace and Mohammad Taher Pilehvar. 2021. A cluster-based approach for improving isotropy in con- textual embedding space. In Proceedings of the 59th

---

Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 575-584, Online. Association for Computational Linguistics.
Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically- rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7396-7408, Online, Association for Computational Linguistics.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliza- beth Salesky, Miryam de Lhoneux, and Desmond EI- liott. 2023. Language modelling with pixels. ICLR.
Michael Tschannen, Basil Mustafa, and Neil Houlsby. 2023. Image-an sonage-ad--oe aad--and-dddaaa language understanding from pix- els only. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Elizaheth Salesky, David Eiterr, and Matt Post. 2021. Robust open- vocabulaay translation from visual text representations. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7235-7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey E. Hinton. 2008
Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the trans- former: A study with machine translation and lan- guage modeling objectives. In Proceedings of the 2019 Conference
Elizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. 2023. Multilingual pixel representations for translation and effective cross-lingual transfer. arXiv preprint.
Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLLP-JCCNPP), pages 4396-4406, Hong Kong, China. Association for Computational Linguistics.
Amit Seker and Reut Tsarfaty. 2020. A pointer net- work architecture for joint morphological segmen- tation and tagging. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4368-4378, Online. Association for Computational Linguisties.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Paul Soulos, Sudha Rao, Caitlin Smith, Eric Rosen, Asli Celikyilmaz. R. Thomas McCoy, Yichen Jiang. Coleman Haley, Roland Fernandez, Hamid Palangi, Jianfeng Gao, and Paul Smolensky. 2021. Structural biases for improving transformers on translation into morphologically rich languages. In Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), pages 52-67, Virtual. Association for Machine Translation in the Americas.
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2020. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representa- tions.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 3645-3650, Florence, Italy. Asso- ciation for Computational I. Linguistics.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1504-1515, Austin, Texas, Association for Computational Linguistics.
Jimin Sun, Patrick Fernandes, Xinyi Wang, and Gra- ham Neubig. 2023. A multi-dimensional evaluation of tokenizer-free multilingual pretrained models. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1725-1735, Dubrovnik, Croatia. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami AL- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Compututional Linguis- tics, 10:291-306.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Hervé Jé- gou. 2021. 1. Training data-cfficient image transform- ers & distillation through attention. In JCML, pages 10347-10357.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in Neural Infor- mation Processing Systems, volume 32, Curran Asso- ciates, Inc.
Asher Trockman and Zico Kolter. 2023. Patches are all you need? Transactions on Machine Learning Research. Featured Certification.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Agha- janyan, Luke Zettlemayer, and Mike Lewis. 2023.

---

Megabyte: Predicting million-by byte sequences with multiscale transformers. arXiv preprint.
ung. Petter Hohle, Jena Hwang, Takumi Ikeda, An- ton Karl Ingason, Radu lon, Elena Irimia, Olájidé Ishola, Kaoru Ito, Siratun Jannat, Tomás Jelínek, Apoorva Jha, Anders Johannsen, Hildur Jónsdóttir, Fredrik Jørgensen, Markus Juutinen, Sarveswaran K, Hüner Kaşikara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Neslihan Kara, Ritván Karahóğa, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Elena Klementieva, Elena Klyachko, Amee Köhn, Abdullatif Köksal, Kamil Kopacewicz, Timo Korkiakangas, Mehmet Köse, Natalia Kotsyba, Jolanta Kovalevskaite, Si- mon Krek, Parameswari Krishnamurthy, Sandra Kübler, Oğuzhan Kuyrukçu, Ash Kuzgun, Sooky- oung Kwak, Veronika Laippala, Lucia Lam, Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phuong L. Hông, Alessandro Lenci, Saran Lertpradit, Herman Leung. Maria Levina, Cheuk Ying Li, Josie Li. Keying Li, Yuan Li, Kyung Lim, Bruna Lima Padovani, Kris- ter Lindén, Nikola Ljubešić, Olga Loginova, Ste- fano Lusito, Andry Luthfi, Mikko Luukko, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Menel Mahamdi, Jean Maillard, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Ma- nurung. Büşra Marşan, Cätälina Märânduc, David Mareček, Katrin Marheinecke, Stella Markamtonatou, Héctor Martínez Alonso, Lorena Martín Rodríguez, André Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Alessandro Mazzei, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça. Tatiana Merzhevich, Niko Miekka, Karina Mischenkova, Margarita Misirpashayeva, Anna Missila, Cătälin Mititelu, Maria Mitrofan, Yusuke Miyao, AmirHos- sein Mojiri Foroushani, Judit Molnár, Amirsaeid Moloodi, Simonetta Montemagni, Amir More, Laura Moreno Romero, Giovanni Moretti, Keiko Sophie Mori, Shinsuke Mori. Tomohiko Morioka, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müärisep. Pinkey Nainwani, Mariam Nakhlé, Juan Ignacio Navarro Horfiaace, Anna Nedoluzhko, Gunta Nešpore Beroor-Brrz Manuela Nevaci. Luong Nguyēn Thi, Huyên Nguyēn Thi Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adédayò Olúökun, Mai Omura, Emeka Onwueg- buzia, Noam Ordan, Petya Osenova, Robert Östling, Lilja Øvrelid, Şaziye Betül Övateş, Merve Öxçelik, Arzucan Özgür. Balkız Öztürk Başaran, Teresa Pac- cosi, Alessio Palmero Aprosio, Hyunji Hayley Park, Niko Partanen, Elena Pascual, Marco Passarotti, Ag- nieszka Patejuk, Guilherme Paulino-Passos, Giulia Pedonese, Angelika Peljak-t.apifsska Siyao Peng. Cenel-Augusto Perez, Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Andrea Peverelli, Jason Phelan, Jussi Pittulainen, Tommi A Pirinen, Emily Pitler, Barbara Plank, Thierry Poibeau, Larisa Pono- mareva, Martin Popel. Lauma Pretkalnipa. Sophie Prévost, Prokopis Prokopidis, Adam Przepiórkowski, Tiina Puolakainen, Sampo Pyysalo, Peng Qi, An- dricla Rääbis, Alexandre Rademaker, Mizanur Ra-
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Zeljko Agić, Amir Ahmadi, Lars Ahrenberg. Chika Kennedy Ajede, Gabrielé Aleksandravičiite, Ika Alfina, Avner Algom, Erik Andersen, Lene Antonsen, Katya Aplonova, Angelina Aquino, Carolina Aragon, Glyd Aranes, Maria Jesus Aranzabe, Bilge Nas Arican, Hórunn Arnardóttir, Gashaw Arutie, Jes- sica Naraiswari Arwidarasti, Masayuki Asahara, Deniz Baran Aslan, Cengiz Asmazoğlu, Luma Ateyah, Furkan Atmaca, Mohammed Attia, Aitz- iber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Starkaõur Barkarson, Rodolfo Basile, Victoria Bas- mov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Kepa Bengoetxea, Yifat Ben Moshe. Gözde Berk, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ah- mad Bhat, Erica Biagetti, Eckhard Bick, Agnè Bielinskiené. Kristín Bjarnadóttir, Rogier Blok- land. Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Anouck Brag- gaar, Kristina Brokaité, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Lauren Cassidy. Tatiana Cavalcanti, Gülşen Cebiroğlu Ery- igit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Neslihan Cesur, Savas Cetin, Özlem Çetinoğlu, Fabricio Chalub, Shweta Chauhan, Ethan Chi, Taishi Chika, Yongseok Cho, Jinho Choi Jayeol Chun. Juyeon Chung, Alessan- dra T. Cignarella, Silvie Cinková, Aurelie Collomb, Çağm Çöltekin, Miriam Connor, Daniela Corbetta, Marine Courtin, Mihacla Cristescu, Philemon Daniel, Elizabeth Davidson, Mathieu Dehouck, Martina de Laurentiis, Marie-Catherine de Marneffe, Vale- ria de Paiva, Mehmet Oguz Derin, Elvis de Souza, Arantza Diaz de Ilarraza, Carly Dickerson, Arawinda Dinakaramani, Elisa Di Nuovo, Bamba Dione, Pe- ter Dirix Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puncet Dwivedi, Hanne Eckhoff. Sandra Riche, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomaž Frjavec, Aline Etienne, Wograine Evelyn, Sidney Facundes, Richárd Farkas, Feder- ica Favero, Jannatul Ferdaousi, Manília Fernanda, Hector Fermandez Alcalde. Jennifer Foster, Cláu- dia Freitas, Kazunori Fujita, Katarína Gajdošová, Daniel Galbraith. Federica Gamba, Marcos Gar- cia, Moa Gärdenfors, Sebastian Garza, Fabrício Fer- raz Gerardi, Kim Gerdes, Filip Ginter, Gustavo Godoy, Takes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldherg, Xavier Gómez Guino- vart, Berta González Saavedra, Bernadeta Griciūté, Matias Grioni, Loic Grobol, Normunds Grüzitis, Bruno Guillaume. Céline Guillot-Barbance Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Ha- jić, Jan Hajič jr., Mika Hämäläinen, Linh Hà Mý, Na- Rae Han, Muhammad Yudistira Hanifmuti, Takahiro Harada, Sam Hardwick, Kim Harris, Dag Haug, Johannes Heinecke, Oliver Hellwig, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hoci-

---

homan, Taraka Rama, Loganathan Ramasamy, Carlos Ramisch, Fam Rashel, Mohammad Sadegh Rasooli, Vinit Ravishankar, Livy Real, Petru Rebeja, Siva Reddy, Mathilde Regnault, Georg Rehm, Ivan Ri- abov, Michael Rießler, Erika Rimkuté, Larissa Ri- naldi, Laura Rituma, Putri Rizgiyah, Luisa Rocha, Firíkur Rögnvaldsson, Mykhailo Romanenko, Rudolf' Rosa, Valentin Rosca, Davide Rovati, Ben Ro- zonoyer, Olga Rudina, Jack Rueter, Kristján Rú- narsson, Shoval Sadde, Pegah Safari, Benoît Sagot, Alcksi Sahala, Shadi Salch, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela San- guinetti, Ezgi Samyar, Dage Särg, Baiba Saulite, Yanin Sawanakunanon, Shefali Saxena, Kevin Scan- nell, Salvatore Scarlata, Nathan Schneider, Sebastian Schuster, Lane Schwartz, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji. Syeda Shahzadi, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Yana Shishk- ina, Muh Shohibussirri, Dmitry Sichinava, Janine Siewert, Einar Freyr Siguròsson, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Maria Skachedubova, Aaron Smith, Isabela Soares- Bastos, Shafi Sourov, Carolyn Spadine, Rachele Sprugnoli, Vivian Stamou, Steinhór Steingrímsson, Antonio Stella, Milan Straka, Emmett Strickland, Jana Str- nadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Daniel Swanson, Zsolt Szántó, Chihiro Taguchi. Dima Taji, Yuta Takahashi, Fabio Tamburini, Mary Ann C. Tan, Takaaki Tanaka, Dipta Tanaya, Mirko Tavoni, Samson Tella, Is- abelle Tellier, Marinella Testori, Guillaume Thomas, Sara Tonelli, Liisi Torga, Marsida Toska. Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sumire Uematsu, Roman Untilov, Zdeňka Urešová, Larraitz Uria. Hans Uszkoreit, drius Utka, Elena Vagnoni, Sowmya Vajjala, Rob der Goot, Martine Vanhove, Daniel van Nick- erk, Gertjan van Noord, Viktor Varga, Uliana Ve- denina, Eric Villemonte de la Clergerie, Veronika Vineze, Natalia Vlasova, Aya Wakasa, Joel C. Wal- lenberg, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Paul Widmer, Shira Wigderson, Sri Hartati Wi- jono, Seyi Williams, Mats Wirén, Christian Wit- tern. Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan. Koichi Yasuoka, Marat M. Yavrumyan, Arife Betüil Yenice, Olcay Taner Yıldız, Zhuoran Yu, Arlisa Yuliawati, Zdenčk Zabokrtský, Shorouq Zahra, Amir Zeldes. He Zhou, Hanzhi Zhu, Anna Zhuravleva, and Rayan Ziane. 2022. Universal dependencies 2. 10. LINDAT/CLARIIH--C digital library at the Institute of Formal and Applied Linguis- tics (UFAL), Faculty of Mathematics and Physics, Charles University.
Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEF International Con- ference on Computer Vision (ICCV).
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2021. Scaling vision transformers. 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognitioon (CVPR), pages 1204-1213.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja

---

A Appendix
A.1 Representational efficiency
0,40
Renderer
continuous
0.35
bigrams
mono
0.30
words
Proportion
0.25
0.20
0.15
0.10
0.00
0
500
1000
1500
Length [Patches]
Figure 9: Distributions of sequence lengths (in patches) resulting from different rendering strategies.
As seen in Figure 1, structured rendering compresses the input space by reducing the positions characters may be observed in. This dramatically affects the number of unique inputs observed in fixed number of sequences, as quantified in Figure 3. Concretely, the 10 most frequently observed image patches after processing 100,000 sequences from English Wikipedia are shown in Figure 2; with continuous rendering all are positional variants of the same subword, while with structured rendering each represents different words morphemes. However, instituting word- or subword-level structure with whitespace padding increases sequence lengths compared unstructured rendering as quantified in Figure
A.2 Prefraining loss curves
1.0
Renderer
0.9
continuous
bigrams
0.8
mono
words
0.7
Loss
0.6
0.5
0.4
0.3
100k
200k
300k
400k
500k
Training steps
Figure Pretraining loss for SMALL models with different rendering strategies, indicating that structured rendering may make the masked reconstruction task more data efficient, reaching low loss in fewer steps.

---

A.3 Detailed experimental results
ENG
ARA
COP
HIN
JPN
KOR
TAM
VIE
ZHO
AVG
BERT
90.6
77.7
13.0
75.9
73.8
30.2
15.2
49.4
28.8
50.5
PIXEL
88.7
77.3
83.5
89.2
90.7
78.5
52.6
50.5
73.7
76.1
TINY-CONTINUOUS
78.9
74.6
80.0
87.9
89.9
75.1
48.3
46.2
69.5
72.3
Structure
SMALL-CONTINUOUS
87.2
77.2
83.4
88.9
91.0
78.8
53.8
51.9
73.5
76.2
SMALL-BIGRAMS
87.9
75.4
84.1
88.9
90.8
79.4
53.9
50.9
73.9
76.1
SMALL-MONO
88.3
76.8
83.4
88.9
91.0
79.0
50.5
51.3
73.8
75.9
88.0
77.2
83.9
89.3
91.2
78.7
53.7
53.3
74.2
76.6
Scale
TINY-BIGRAMS
82.9
70.6
86.2
76.2
44.9
47.6
69.8
72.0
AMALL-BIGRAMS
87.9
75.4
84.1
88.9
90.8
79.4
53.9
50.9
73.9
76.1
BASE-BIGRAMS
89.6
77.7
81.4
88.6
90.8
78.1
49.8
49.4
73.9
75.5
Table 4: Test set LAS results for dependency parsing selection of Universal Dependencies treebanks (UDP).
MNLI-M/MM
QQP
SST-2
COLA
STS-B
MRPC
RTB
WNLI
AVG
BERT
84.0/84.2
87.6
91.0
92.6
60.3
88.8
90.2
69.5
51.8
80.0
PIXEL
78.1/
84.5
87.8
89.6
38.4
81.1
88.2
60.5
53.8
74.1
TINY-CONTINUOUS
36.7 / 37.0
76.6
72.9
87.2
2.1
25.1
82.4
58.5
59.2
53.8
Structure
SMALL-CONTINUOUT
72.2 73.6
84.8
86.2
88.3
19.1
81.7
84.6
61.4
57.7
71.0
SMALL-BIGRAMM
77.3 / 78.1
85.7
87.8
90.4
42.3
84.3
87.8
63.5
56.3
75.4
SMALL-MONNO
77.4 77.6
84.7
89.4
42.3
82.4
86.9
57.5
58.9
74.4
SMALL-WORDS
76.7 77..3
84.5
86.6
89.9
44.6
80.5
87.4
62.8
56.3
74.7
Scale
TINY-BIGRAMS
60.8 61.9
79.6
81.7
87.2
77.9
83.0
59.4
57.7
66.5
SMALL-BIGRAMS
77.3 / 78.1
85.7
87.8
90.4
42.3
84.3
87.8
63.5
56.3
75.4
81.1 / 81.4
87.6
89.7
90.4
53.3
86.6
90.2
63.5
56.3
78.0
Table 5: Validation set performance on GLUE, The reported metrics are Fi score for QQP and MRPC, Matthew's correlation for COLA. Spearman's p for STS-B, and accuracy for the rest.

---

ENG
ARA
BEN
FIN
IND
KOR
RUS
SWA
TEL
AVG
BERT
68.5
58.0
43.2
58.3
67.1
12,4
53.2
71.3
48.2
51.5
PIXEL
59.6
57.3
36.3
57.1
63.6
26.1
50.5
65.9
61.7
52.3
IINY-CONTINUOUS
42.6
45.0
12.4
45.3
48.1
13.2
36.7
46.8
45.7
36.6
SMALL-CONTNUOOLS
57.1
53.3
20.3
57.5
62.9
22.3
51,1
65.3
58.1
48.8
Scale
TINY-RIGRAMS
43.3
45.5
19.0
50.3
48.2
14.9
45.4
52.7
56.4
41.6
BIGRAMS
50.8
53.2
37.1
59.1
57.5
20.1
52,8
62.4
64.2
50.8
BASE-BIGRAMS 53.8
53.1
59.6
60.3
18.8
54.1
64.1
65.7
52.8
Table 6: Validation set Fi scores for TyDiQA-GoldP Average (AVG) scores exclude ENG (Clark et al., 2020). With some rendering structures, answer span extraction adversely affects results (see discussion at $ A.4).
KIN
LUG
LUO
AVG
BERT
86.6
83.5
72.0
78.4
73.2
87.0
83.3
62.2
73.8
62.7
PIXEL
47.7
82.4
79.9
64.2
76.5
66.6
78.7
79.8
59.7
70.7
70.6
BASE- BIGRABS
50.1
85.6
82.2
68.4
78.4
72.5
82.8
82.4
64.4
74.8
74.2
Table 7: Test set Fi scores on MasakhaNER (Adelani et al., 2021). (2023) and render each word at start a new image patch.
We follow the implementation of Rust et al.
A.4 TyDiQa-GoldP
The CONTINUOUS rendering strategy used for PIXEL, in which words often overlap in image patch, leads to extracted answer spans that potentially include leading or trailing characters that should not be part of the answer. BIGRAMS rendering adressess this issue by yielding clear word boundaries in the input representations.
However, the BIGRAMS rendering strategy poses new challenges to extracting answer spans for TyDiQA- GoldP. While the task is simplified compared to the primary task by removing language tracks that lack whitespace, !7 we find that a surprisingly high number of "words" are string of comma secaratd words concatenations of characters and letters that should be delimited by whitespace. By design we consider and render these one unit when only split by whitespace. An example of a single "unit" from the training split highlights this issue more clearly: "oikeudet[1) L Vilna523,0500 180113877nnn''' where (ilnin))
the expected answer is "Vilna" and highlighted in bold. In such instance, PIXEL BIGRAMS model will predict the whole unit, resulting in a lower performance. Furthermore, some of these "words" the training data are more than a thousand characters long and therefore do not fit within the maximum sequence length of 529 patches.
I /https://github. http:://ss..ccoooaeeeeeeeeeaaaaa 1id = h 0380277799988888899977747

---

A.5 Measuring self-similarity and intra-sentence similarity
We follow Ethayarajh (2019) and measure the degree self-similarity and intra-sentence similarity for the words in the two frequency samples from $ 6.3. Self-similarity is computed as the cosine similarity between the same word in different sentences and a high degree therefore indicates that representations vary little across contexts. For intra-sentence similarity we compute the cosine similarity between word representation and the sentence representation (mean hidden state output across all tokens excluding the CLS token and black end-of-sequence token). 19 This captures how aligned the representation of word is with the sentence similarity, we infer that the word has context-specific representation that still distinct from the other words in that sentence. self-similarity is low but intra-sentence similarity is high, this alludes the word simply being contextualised by aligning its representation with the other words in that sentence. We summarise these two measures in Figure 7 and find that, just like in Figure 4a, the upper layers produce more context-specific representations as seen by the lower self- self-similarity, and that high-frequene words are the most context-specific. This is in line with Ethayarajh (2019) who finds that stopwords, being some of the most frequently observed words in the pretraining data, have some of the most context-specific representations. The measure of intra-sentence similarity reveals that the contextualised representation of eew-frequnncy words more similar to that its context, with nighffreuenccy words having more nuance where words do not necessarily mean the same just because they appear in the same sentence.
whole. If word has both low degree of self-similarity and intra-sentence
""thhyyrrahh (2019) average over every word-sentence combination for a given sentence, not just a single word.


Reviews:
[{"review_id": "bMG1enxYUq", "reviewer": "Reviewer_ocCw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and positive feedback on our work. We are grateful to see that you appreciate the thoroughness of our analyses, and that you agree with the importance of building compact and resource-friendly models. We will definitely highlight the performance of the pixel-based models on low-resource languages in the main text. We hope to see the field of pixel-based language modelling mature just as tokenization-based models have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper analyses its points thoroughly. The bigram representation proposed in the paper achieves competitive results with fewer parameters than previous pixel-based representations. It is important to encourage such efforts for more efficient NLP given the environmental impact of resource-hungry models. ", "reasons_to_reject": "The advantages of pixel-based language models over tokenization-based ones are only explicitly highlighted in the introduction, and the proposed pixel-based models do not perform better than tokenization-based ones yet in settings with sufficient language resources.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This papers focuses on the comparison of four rendering strategies for pixel-based language models. It contributes an empirical comparison of said rendering strategies on three different datasets (UDP, GLUE, and TyDiQA-GoldP), finding character bigram-based structure to be considerably more computationally efficient than previous work (the continuous structure). The authors further provide an ablation study and an investigation of the embedding space that reveals an anisotropy (low-frequency tokens being clustered together).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "My suggestion would be to highlight the performance of pixel-based models (at least for BIGRAMS) on low-resource languages in the main text, not just in the appendix."}, "extra_text": {}}, {"review_id": "Oa3ds5fFUy", "reviewer": "Reviewer_u9Mz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and feedback. To address your concerns:\n\n> The methods proposed in this paper are relatively simple and lack novelty.\n\nWhile character (n=2)-gram models are not novel, as discussed in the Related work section (L535-546), hypothesis-driven improvements through new text rendering strategies are. All previous work on pixel-based language modelling [1,2,3,4] used the continuous rendering strategy without considering any alternative approaches. We see it as a strength that the downstream task improvements can be realised from a straightforward modification of the rendering strategy that generalises across scripts and languages. \n\n> While the rendering strategies this paper proposed are suit to English, those strategies may not generalize to other languages.\n\nWe agree that the rendering strategies may appear to be suited to English but they were not designed with latin script writing systems in mind. The paper includes both a discussion of this in the Limitations section (L574-596) and experiments on TyDiQa-GoldP (L305-310) to specifically address the concern that the bigrams rendering strategy may not generalise to other languages. For languages that do not mark word or sentence boundaries with whitespace such as Thai or polysynthetic languages such as Inuktitut, the bigrams text rendering strategy would still result in a compressed input space (increase in observed token frequency) compared to a continuous rendering strategy. \nThe TyDiQa-GoldP benchmark already includes typologically diverse languages: Arabic, Bengali, Finnish, Indonesian, Kiswahili, Korean, Russian, and Telugu (4/8 being Latin scripts). We find that the bigrams rendering strategy leads to better performance compared to the PIXEL baseline (52.8 vs 52.3) and with the smaller model scales benefitting even more (50.8 vs 48.8 for SMALL; 41.6 vs 36.6 for TINY).\n\nAdditionally, in response to your concern, we have also finetuned the BASE-BIGRAMS model for NER on MasakhaNER [5] during this response period. We find that the proposed bigrams rendering strategy leads to consistent improvements across the 10 non-English languages (Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Luo, Nigerian-Pidgin, Swahili, Wolof, Yorùbá) compared to PIXEL following a continuous rendering strategy (74.2 vs 70.6). We will include the results of this experiment when we prepare the final version of the paper.\n|               | Amharic  | Hausa  | Igbo  | Kinyarwanda  | Luganda  | Luo  | Nigerian-Pidgin  | Swahili  | Wolof  | Yorùbá  | AVG  |\n|---------------|------|------|------|------|------|------|------|------|------|------|------|\n| BASE-BIGRAMS   | 50.1 | 85.6 | 82.2 | 68.4 | 78.4 | 72.5 | 82.8 | 82.4 | 64.4 | 74.8 | 74.2 |\n| PIXEL          | 47.7 | 82.4 | 79.9 | 64.2 | 76.5 | 66.6 | 78.7 | 79.8 | 59.7 | 70.7 | 70.6 |\n\n\n> Reproducibility.\n\nWe aim for our work to be fully reproducible. We emphasise that we will make all code and models available to the public (L87-88). All details about (hyper)parameter values can be found in [3], since we aimed for a clean comparison by varying as few aspects as possible. The exception being the learning rates, which are stated in footnote 5 on page 4. \n\n[1]\tElizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. *EMNLP*.\n\n[2]\tElizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. 2023. Pixel representations for multilingual translation and data-efficient cross-lingual transfer. *arXiv preprint*. \n\n[3]\tPhillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. *ICLR*.\n\n[4]\tMichael Tschannen, Basil Mustafa, and Neil Houlsby. 2023. Image-and-language understanding from pixels only. *CVPR*.\n\n[5]\tDavid Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. MasakhaNER: Named Entity Recognition for African Languages. *TACL*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper evaluates four text rendering strategies to address the problem of redundancy in the input space of PIXEL-based language models. Pixel-based language models process text rendered as images, making them a promising approach to open vocabulary language modelling.", "reasons_to_reject": "1. The methods proposed in this paper are relatively simple and lack novelty.\n2. While the rendering strategies this paper proposed are suit to English, those strategies may not generalize to other languages. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper evaluates four text rendering strategies to address the problem of redundancy in the input space of PIXEL-based language models. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7HzRDXm31k", "reviewer": "Reviewer_Tumj", "summary": "", "questions": "1.\tThe authors mentioned that they performed experiments on UD 2.10 and the experimental results are given in the Appendix. However, the authors’ method is to add a simple task-specific MLP to the language model. This is not suitable for parsing tasks on UD, because directly using classification MLP for parsing will lead to illegal structures (such as unrooted trees or ring structures), and additional post-processing is required in parsing Steps to remove this illicit influence. Generally speaking, post-processing will reduce the performance of the model. Is the author's results in Table 4 in the appendix post-processing? If not, then this performance is not of reference significance.\n2.\tTo achieve parsing by classification, two different classifier heads are generally required because two different label results of head and relation are to be generated. Neither the traditional transition-based method nor the graph-based method can use a classification head to achieve parsing. I am very curious how the author achieved it (The simplest structure of parsing using the masked language model still requires a specially designed Biaffine structure header and chu-liu post-processing algorithm, see [1] for details)? In addition, corresponding to the accuracy of head and relation, two statistical indicators, UAS and LAS, are generally used in parsing, but the results in Table 4 do not indicate which indicator is used at all. In addition, the parsing task generally uses 12 fixed languages in UD 2.0 or UD 2.2 (Bulgarian, Catalan, Czech, German, English, Spanish, French, Italian, Dutch, Norwegian, Romanian, Russian[2]). There are many baselines for reference in these languages, but the author chose UD 2.10, and did not use the common 12 languages, so it is impossible to make an accurate judgment on the performance comparison in Table 4.\n[1] Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT. Han He and Jinho D.Choi. AAAI 2020.\n[2] Stack-pointer Networks for Dependency Parsing. Xuezhe Ma, Xecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. ACL 2018.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and feedback. We are pleased to see that you appreciate the detailed analyses, and that you agree with the importance of building compact and resource-friendly models. To address your concerns:\n\n> The novelty of this paper is slightly insufficient. The new rendering strategy proposed by the author is an improvement of the rendering strategy of the previous pixel-based language model.\n\nWhile character (n=2)-gram models are not novel, as discussed in the Related work section (L535-546), hypothesis-driven improvements through new text rendering strategies are. All previous work on pixel-based language modelling [1,2,3,4] used the continuous rendering strategy without considering any alternative approaches. To the best of our knowledge this is also the first work to investigate the embedding space of pixel-based language models and identify representational similarities to tokenization-based language models. We regard this new insight as valuable also for future work in this direction because it opens up new avenues for further improving pixel-based models, given the accumulated knowledge from techniques that have proven successful from tokenization-based models (L506-510; L570-575).\n\n> The experimental results in this paper, especially the relevant parts of UDP, may have some problems (see the Questions section for details)\n\nWe acknowledge the reviewer’s concern. To be clear, we follow the same protocol during finetuning as done in PIXEL [3]. This is stated on line 248-249. From [3, page 4]:\n“*For dependency parsing, we render text as above but obtain word-level representations by mean pooling over all corresponding image patches of a word and employ a biaffine parsing head (Dozat & Manning, 2017), following the implementation from Glavaš & Vulic (2021).*”\n\n> However, the authors’ method is to add a simple task-specific MLP to the language model.\n\nWe understand this concern, which is directly related to the question above. We will rework section 5.2 to make our approach clear and avoid confusion about how the model is finetuned for the dependency parsing task. We also acknowledge that we have a typo and that line 241-243 should read: “*To finetune our models for classification tasks we replace the decoder used for pretraining with a task-specific classification head.*”, dropping the erroneous mention of MLP.\n\n> To achieve parsing by classification, two different classifier heads are generally required because two different label results of head and relation are to be generated. Neither the traditional transition-based method nor the graph-based method can use a classification head to achieve parsing. I am very curious how the author achieved it\n\nSee response above. We use a biaffine parsing head from Dozat & Manning (2017), following the implementation from Glavaš & Vulic (2021).\n\n> In addition, corresponding to the accuracy of head and relation, two statistical indicators, UAS and LAS, are generally used in parsing, but the results in Table 4 do not indicate which indicator is used at all.\n\nWe report test set LAS scores. This will be clearly stated in the final version of the paper. \n\n> In addition, the parsing task generally uses 12 fixed languages in UD 2.0 or UD 2.2 (Bulgarian, Catalan, Czech, German, English, Spanish, French, Italian, Dutch, Norwegian, Romanian, Russian[2]). \n\nWe agree that the choice of languages for multilingual evaluation is an important issue.\nIn this paper, we re-used the same set of languages as [3] for the sake of direct comparison. There is no straightforward agreement on which languages should be used for multilingual evaluations of language models [5], so re-using the languages from [3] means that our experiments cover 8 different writing scripts: Latin, Chinese, Japanese, Korean, Arabic, Devanagari, Coptic, and Tamil; as opposed to only covering Latin and Cyrillic in the 12 fixed languages. Nevertheless, if the reviewer still thinks this would strengthen the contribution of the paper, we can include the results on the 12 fixed languages in the appendix of final version of the paper. We are currently not able to present these results due to limited computing resources during the author response period.\n\n> The last line in Table 2 is displayed in light gray instead of normal black. I didn't find the corresponding explanation or emphasis from authors. This seems to be a formatting error.\n\nWe thank the reviewer for highlighting this problem, which we will address when we revise the paper.\n\n> Reproducibility.\n\nWe aim for our work to be fully reproducible. We emphasise that we will make all code and models available to the public (L87-88). All details about (hyper)parameter values can be found in [3], since we aimed for a clean comparison by varying as few aspects as possible. The exception being the learning rates, which are stated in footnote 5 on page 4. \n\n[1]\tElizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. *EMNLP*.\n\n[2]\tElizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. 2023. Pixel representations for multilingual translation and data-efficient cross-lingual transfer. *arXiv preprint*.\n\n[3]\tPhillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. *ICLR*.\n\n[4]\tMichael Tschannen, Basil Mustafa, and Neil Houlsby. 2023. Image-and-language understanding from pixels only. *CVPR*.\n\n[5]\tEthan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding Universal Grammatical Relations in Multilingual BERT. *ACL*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe rendering strategy proposed by the author can make the small model achieve the same effect as the large model, and can be used as a means of compressing model parameters\n2.\tThe authors’ ablation experiment design is very detailed, and the experimental results are analyzed in terms of model parameter magnitude, structure comparison, token frequency, etc., which is very detailed.", "reasons_to_reject": "1.\tThe novelty of this paper is slightly insufficient. The new rendering strategy proposed by the author is an improvement of the rendering strategy of the previous pixel-based language model.\n2.\tThe experimental results in this paper, especially the relevant parts of UDP, may have some problems (see the Questions section for details)", "questions_to_authors": "1.\tThe authors mentioned that they performed experiments on UD 2.10 and the experimental results are given in the Appendix. However, the authors’ method is to add a simple task-specific MLP to the language model. This is not suitable for parsing tasks on UD, because directly using classification MLP for parsing will lead to illegal structures (such as unrooted trees or ring structures), and additional post-processing is required in parsing Steps to remove this illicit influence. Generally speaking, post-processing will reduce the performance of the model. Is the author's results in Table 4 in the appendix post-processing? If not, then this performance is not of reference significance.\n2.\tTo achieve parsing by classification, two different classifier heads are generally required because two different label results of head and relation are to be generated. Neither the traditional transition-based method nor the graph-based method can use a classification head to achieve parsing. I am very curious how the author achieved it (The simplest structure of parsing using the masked language model still requires a specially designed Biaffine structure header and chu-liu post-processing algorithm, see [1] for details)? In addition, corresponding to the accuracy of head and relation, two statistical indicators, UAS and LAS, are generally used in parsing, but the results in Table 4 do not indicate which indicator is used at all. In addition, the parsing task generally uses 12 fixed languages in UD 2.0 or UD 2.2 (Bulgarian, Catalan, Czech, German, English, Spanish, French, Italian, Dutch, Norwegian, Romanian, Russian[2]). There are many baselines for reference in these languages, but the author chose UD 2.10, and did not use the common 12 languages, so it is impossible to make an accurate judgment on the performance comparison in Table 4.\n[1] Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT. Han He and Jinho D.Choi. AAAI 2020.\n[2] Stack-pointer Networks for Dependency Parsing. Xuezhe Ma, Xecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. ACL 2018.", "paper_topic_and_main_contributions": "The author proposes that for the current pixel-based language model, suboptimal almost-equivalent input patches may be generated when processing downstream tasks. The author further found through research that a simple character bigram structure can effectively improve the performance of the sentence-level task. The new rendering strategy proposed by the author can also make the 22M language model achieve performance equivalent to the 86M language model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The last line in Table 2 is displayed in light gray instead of normal black. I didn't find the corresponding explanation or emphasis from authors. This seems to be a formatting error."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comparative analysis of various rendering algorithms utilized in pixel-based language models. The reviewers acknowledged the level of detail in the study and commended the efficiency of the methods employed. They also emphasized the need for further research in this area, considering the growing interest in multimodal language models.
Decision: Accept-Main